Peter Stuifzand

Log parsing speed

Now that we have two programs that parse log files, we can start to take a look at how many lines the program parses per second. First we have to make the two programs as similar as possible. In pseudocode it looks likes this.

  1. Load all modules
  2. Take the start time using Time::HiRes
  3. Put the code of the program here
  4. Set line_count = 0
  5. Using stdin: loop through all lines
    1. Parse the line
    2. Set line_count++
  6. Find the time difference
  7. Divide and line_count / time as n lines/s

In Perl this looks like:

use Time::HiRes 'gettimeofday', 'tv_interval';

my $start = [gettimeofday];

# Your program

my $line_count = 0;

while (<>) {
    # Parse one line using your software

my $diff = tv_interval($start);
printf "%.2f lines/s\n", $line_count / $diff;

Now run the two programs a few times and look at the parsing speed. In my case there was a big difference between the speed of the two programs. I expect a difference in your run as well.

© 2023 Peter Stuifzand