Join 34,000+ subscribers and receive articles from our blog about software quality, testing, QA and security.
 

Splitting the log file


#1

Hi,

We ran a test for 5 days, 700Mb log file, but even a 2GB RAM machine can’t open it (“Out of Memory” error). There’s at least 3 million records. Please, how can we split the binary log file we have? We really can’t afford to throw away 5 days of testing!

(yes, we’ve already set it to hourly splits for future tests)

thanks,
Daniel


#2

Never mind… we used the known format of the file to write a simple splitter application. Thanks!


#3

Hello Daniel,

Thanks for your interest in SmartInspect.

Writing a tool for splitting the log file should indeed be the best way to handle this. Such a tool should normally require less than 100 lines of code. Please note that there’s no need to fully read and extract all properties of the individual packets. It is sufficient to read the packet header, look at the packet size, read the packet body and then write the entire packet to an output file. In pseudo code this could look like:

[code]MAXIMUM_SIZE = 50 * 1024 * 1024 { 50 MB }

OpenInput()
try
if (!ReadSILF()) then { Read log file header }
begin
Error(“Not a SmartInspect log file”);
end

OpenOutput()
try
WriteSILF() { Write log file header }

currentSize = 0
while ReadHeader(out header) do
begin
  if currentSize + header.Size > MAXIMUM_SIZE then
  begin
    SwitchOutput() { Open the next output file }
    currentSize = 0
  end
  currentSize += header.Size
  if ReadBody(header, out body) then
  begin
    WritePacket(header, body)
  end
end

finally
CloseOutput()
end
finally
CloseInput()
end
[/code]

Please let me know if you have any questions. You can also contact me via email (simply click on the my name below).