Problems while measuring performance

While browsing around StackOverflow I found a page about Square root function implementation. One of the answer points to an article containing different implementations of square root.

One of the things I like to do is pointing out obvious errors in benchmarks. It’s not that I don’t have an intimate understanding of these things, but if I can point out the error, is has to be obvious. One of the obvious errors in time measurement can be shown using a short pseudo code example.

Begin = Time();
For (I = N; I > 0; I--) {
     F();
}
End = Time();

AvgTime = (End - Begin) / N

This program starts by retrieving the time at the start. Then the function F is called N times. At the end we set End to the current time. The absolute difference between Begin and End is the time it took to call the function F, N times. We divide by N to find the time it took to call the function once.

A benchmark should be structured like this. It reduces the error of your measurement. Benchmarks that are nog structured like this are wrong.

Now back to the code in the article. On the surface the code looks a lot like my pseudo code. I will show why it isn’t the same.

for (int j = 0; j  < AVG; j++) {
    dur.Start();
    for (int i = 1; i < M; i++) {
        RefTotalPrecision+=sqrt((float) i);
    }
    dur.Stop();
    Temp+=dur.GetDuration();
}
RefTotalPrecision/=AVG;
Temp/=AVG;
RefSpeed=(float)(Temp)/CLOCKS_PER_SEC;

In this code the variable AVG is set to 10 and M is set to 10.000. After running the variable RefTotalPrecision will contain the average precision of M sqrt calls. The variable RefSpeed should contain the average speed of M sqrt calls. It – however – doesn’t.

The M loop makes it look like we average over many calls. We don’t because the value we average with is AVG, which is used in the outer loop. To show what happens we remove the code that is measured. I also removed the averaging code at the end.

for (int j = 0; j < AVG; j++) {
    dur.Start();
    // insert code to be measured
    dur.Stop();
    Temp+=dur.GetDuration();
}

The code measures the time 2 * AVG times, while it should only be measured twice, once at the start, and once at the end. To fix this, we need to move two lines and remove one character.

dur.Start();
for (int j = 0; j < AVG; j++)
{
    // insert code to be measured
}
dur.Stop();
Temp = dur.GetDuration();

We move the Start and Stop calls outside the loop. The error introduced by the measuring functions will now be divided by AVG, which is AVG times smaller. I also removed the +, it wasn’t needed anymore.

Peter Stuifzand

Problems while measuring performance