public static double testMethod(double dfl)
        {       
        for (int i = 0; i < 100; ++i)
            {
            dfl = Math.log(dfl);
            }
        return dfl;
        }

The content of the method is not important, in fact we should not assume to know the implementation, all we know is that it is likely to complete in well under the system’s clock resolution. Typically we’d micro benchmark this during development time by simply calling it a large number of times in a tight loop and computing the average execution time:

    public static Result testClassic(int cIters)
        {
        long   ldtStart  = System.currentTimeMillis();
        double dflResult = 123.456;
        for (int i = 0; i < cIters; ++i)
            {
            dflResult = testMethod(dflResult);
            }
        long   cMillis        = System.currentTimeMillis() - ldtStart;
        double dflMillisPerOp = ((double) cMillis) / cIters;
        return new Result(cIters, cMillis, dflMillisPerOp, dflResult);
        }

One of the primary reasons we do the loop is that as we expect testMethod to return in well under the clock resolution, and thus without the loop we’d be left with an arguably meaningless value of “0″. So instead we forcefully run it long enough that we are sure the clock will have “ticked” many times. This is all well and good, but what if we weren’t in a position to call it in a tight loop. What if our interest was in measuring the cost of periodic calls to this function inside a working application, where we can’t simply time a loop over thousands of calls?

At this point you may be asking why would you want to have the application measure the cost, isn’t that what profilers are for? Well yes, this isn’t meant to replace a profiler, but it can be useful in collecting runtime statistics. For instance in Oracle Coherence we’ve consider measuring the average serialization time for user data objects. These are not classes we have control over, or the ability to easily profile, but it would be nice to know track their serialization cost as it is a crucial part of our overall performance. As Coherence still supports Java 1.4 we are without the higher resolution System.nanoTime(), and so an alternative is needed.

So what do we do with a millisecond resolution clock, when we want to measure sub millisecond operations? Well we just do the obvious.

        // ... other application code ...

        long  ldtStart = System.currentTimeMillis();
        testMethod(dflInpuut);
        cMillisTotal += System.currentTimeMillis() - ldtStart;
        ++cIters;

        // ... other application code ...
        // at some point compute the average cost
        double dflMillisPerOp = ((double) cMillisTotal) / cIters;    

That is it, we just need to do is cross over this code a few thousand times and we’re good. I’d imagine some might be wondering what the use in this is as the per-operation measurement should yield zero each time. We’ll lets give it a try and see what we get. To simulate the application crossing over the above code we’ll introduce a loop, which between calls into the above code, burns a random amount of CPU, i.e. our fake application logic. We’ll call this approach the “discrete” approach as compared to the “classic” approach shown above.

    public static Result testDiscrete(int cIters)
        {
        int     nFactor   = rnd.nextInt(10);
        long   ldtStart  = System.currentTimeMillis();
        long   cMillis   = 0;
        double dflResult = 123.456;
        for (int i = 0; i < cIters; ++i)
            {
            long ldtIter = System.currentTimeMillis();
            dflResult = testMethod(dflResult);
            cMillis  += System.currentTimeMillis() - ldtIter;

            burnCPU(nFactor); // simulate lots of other code running
            }
        long   cMillisOuter   = System.currentTimeMillis() - ldtStart;
        double dflMillisPerOp = ((double) cMillis) / cIters;
        return new Result(cIters, cMillisOuter, dflMillisPerOp, dflResult);
        }

The results for testing multiple runs with 100,000 iterations each are as follows:

Classic: Average execution in 0.0092 ms; after measuring 100000 over 920 ms
Discrete: Average execution in 0.00938 ms; after measuring 100000 over 17192 ms

Classic: Average execution in 0.00883 ms; after measuring 100000 over 883 ms
Discrete: Average execution in 0.00898 ms; after measuring 100000 over 908 ms

Classic: Average execution in 0.0089 ms; after measuring 100000 over 890 ms
Discrete: Average execution in 0.00905 ms; after measuring 100000 over 19128 ms

Classic: Average execution in 0.00885 ms; after measuring 100000 over 885 ms
Discrete: Average execution in 0.0089 ms; after measuring 100000 over 5435 ms

Classic: Average execution in 0.00884 ms; after measuring 100000 over 884 ms
Discrete: Average execution in 0.0094 ms; after measuring 100000 over 12287 ms

Classic: Average execution in 0.00887 ms; after measuring 100000 over 887 ms
Discrete: Average execution in 0.00899 ms; after measuring 100000 over 3173 ms

As you can see both approaches yield surprisingly similar results, the cost of our testMethod is around .01 ms. The variability in the total duration of the discrete test is intentional and caused by the burnCPU() method, which is configured to burn a random amount of CPU per test run simulating the frequency of calls to testMethod().

But how does the discrete approach yield apparently accurate results if always measures 0ms per call? Well I suppose the obvious answer is that it doesn’t always measure 0ms, sometimes is measures 1ms. The fact that the method always completes in well under a millisecond doesn’t mean we don’t occasionally cross a clock edge during the measurement and get an even more incorrect 1ms result. Remember that we aren’t in control of the clock, and thus we can’t assume that we are at the beginning of a clock-cycle when we record the start time, thus we could end up recording the start time when we are just a few nanoseconds from the next tick. Ok, but still how does this yield such good results? This can be answered is in the question, “What is the probability of us crossing the clock edge?”. Using the result from the classic method we can see that the probability of crossing a one ms clock boundary is around 1%. This comes straight from our measurement of ~0.01ms. In the classic case with back to back method calls, for every 100 or so 0.01ms calls the clock would tick. This cost of testMethod() doesn’t change with the discrete approach, and thus the probability of us encountering a clock tick is still 1%. So 99% of our samples result in 0ms, and 1% result in 1ms. Meaning that the accumulated milliseconds for out 100,000 samples will be around one second, which pretty much lines up with what we get in the classic case. So this is how we come to the title, we are measuring relying upon the small probability of crossing a clock edge, and the result of adding the results from a large number of samples.

I find it really interesting to watch this work. Even though the math makes sense it is still surprising to see the results occur which such accuracy. The results are not the result of a carefully crafted test, change the iterations, take out or change the “burn”, change the test function, it all still holds up quite well. It also reliant on this particular clock resolution, for instance the same thing will work on Windows where the precision of System.currentTimeMillis() is actually ~16ms (resolution is still 1ms). In such a case the probability of seeing a clock edge becomes 1 in 1600, but the cost of that one, is now 16ms, so the totals and averages will still hold.

Now there are a few small caveats:

Of course the biggest caveat of all is that with higher resolution clocks such as Java 1.5′s System.nanoTime() this technique isn’t as necessary, though that doesn’t change the fact that it is still quite interesting.

And for those that are interested here is the full source:

import java.util.Random;

/**
* Scratch test
*/
public class Scratch
    {
    public static void main(String[] asArg)
            throws Exception
        {
        int cRuns  = asArg.length > 0 ? Integer.parseInt(asArg[0]) : 100;
        int cIters = asArg.length > 1 ? Integer.parseInt(asArg[1]) : 100000;

        for (int i = 0; i < cRuns; ++i)
            {
            System.out.println("Classic:  " + testClassic(cIters));
            System.out.println("Discrete: " + testDiscrete(cIters));
            System.out.println();
            }
        }

    /**
    * Some non-trivial method which is still very short in duration.
    */
    public static double testMethod(double dfl)
        {
        for (int i = 0; i < 100; ++i)
            {
            dfl = Math.log(dfl);
            }
        return dfl;
        }

    /**
    * Perform "classic" performance test measuring how long it takes to run
    * the operation many times, and then computing an average.
    */
    public static Result testClassic(int cIters)
        {
        long   ldtStart  = System.currentTimeMillis();
        double dflResult = 123.456;
        for (int i = 0; i < cIters; ++i)
            {
            dflResult = testMethod(dflResult);
            }
        long   cMillis        = System.currentTimeMillis() - ldtStart;
        double dflMillisPerOp = ((double) cMillis) / cIters;
        return new Result(cIters, cMillis, dflMillisPerOp, dflResult);
        }

    /**
    * Perform &quot;non-classic&quot; performance measurement where timings are recorded
    * for individual executions, and then computing an average.
    */
    public static Result testDiscrete(int cIters)
        {
        int    nFactor   = rnd.nextInt(10);

        long   ldtStart  = System.currentTimeMillis();
        long   cMillis   = 0;
        double dflResult = 123.456;
        for (int i = 0; i &lt; cIters; ++i)
            {
            long ldtIter = System.currentTimeMillis();
            dflResult = testMethod(dflResult);
            cMillis  += System.currentTimeMillis() - ldtIter;

            burnCPU(nFactor); // simulate lots of other code running
            }
        long   cMillisOuter   = System.currentTimeMillis() - ldtStart;
        double dflMillisPerOp = ((double) cMillis) / cIters;
        return new Result(cIters, cMillisOuter, dflMillisPerOp, dflResult);
        }

    /**
    * Data structure for recording resuls.
    */
    static class Result
        {
        public Result(int cIters, long cMillisTotal, double dflMillisPerOp,
                double dflResult)
            {
            m_cIters         = cIters;
            m_cMillisTotal   = cMillisTotal;
            m_dflMillisPerOp = dflMillisPerOp;
            m_dflResult      = dflResult;
            }

        public String toString()
            {
            return &quot;Average execution in &quot; + m_dflMillisPerOp +
                   &quot; ms; after measuring &quot; + m_cIters + &quot; over &quot; +
                   m_cMillisTotal + &quot; ms&quot;;
            }

        private int    m_cIters;
        private long   m_cMillisTotal;
        private double m_dflMillisPerOp;
        private double m_dflResult;
        }

    /**
    * Helper method to burn a random amount of CPU in order to simulate
    * spreading out the individual measurements.
    */
    public static int burnCPU(int nFactor)
        {
        int r = 0;
        for (int j = 0; j &lt; nFactor * 1000; ++j)
            {
            r = 1 + rnd.nextInt();
            }
        return r;
        }

    static final Random rnd = new Random();
    }

I hope you’ve found this as interesting as I have

public class Main
    {
    public static class Test
        {
    	public int getVirtual()
            {
            return m_i;
            }

        public int m_i = 1;
        }

    public static void main(String[] asArg)
	{
    	int cIters = Integer.parseInt(asArg[0]);
    	int cOps   = Integer.parseInt(asArg[1]);

        Test test = new Test();
        for (int i = 0; i < cIters; ++i)
            {
            long ldtStart = System.currentTimeMillis();
            for (int j = 0; j < cOps; ++j)
                {
                test.getVirtual();
                }

            long cMillis = System.currentTimeMillis() - ldtStart;
            System.out.println(cMillis);
            }
        }
    }

And running 10 iterations of 1 billion virtual calls yields the following millisecond timings:

java14 Main 10 1000000000
1646
1636
1650
1677
1645
1669
1689
1668
1676
1724

java15 Main 10 1000000000
1775
1752
1698
1688
1759
1732
1758
1709
1670
1733

java16 Main 10 1000000000
8 <— WTF?
1409
1370
1347
1357
1341
1346
1349
1350
1354

Ok, so on 1.6 the first iteration of billion calls took 8ms and each subsequent iteration took nearly 200 times as long, how could that be?

Q> I screwed up my measurements

A> Nope looks ok, double check, still looks ok

Q> Was is the garbage collector

A> Shouldn’t be, other then printing the results the test doesn’t generate any garbage.  I reran with GC logging just to be sure, and no GCs were logged (not surprising).

Q> Did I need to let the test run longer so HotSpot could do its thing?

A> Nope, ran for much longer, results held steady at ~1350ms

Q> Is the optimizer broken or deoptimizing after the first iteration?

A> Sure looks like it

Ok so time to start thinking about how to optimizer is going to change my test.  The first thing to notice is that it could identify that there is no reason to actually call my getVirtual() method, it can see there are no side effects from calling it, and can see that the result is discarded.  So lets modify to do something with the result, just looking at the test loop now.

for (int i = 0; i < cIters; ++i)
    {
    long ldtStart = System.currentTimeMillis();
    int  cTotal   = 0;
    for (int j = 0; j < cOps; ++j)
        {
        cTotal += test.getVirtual();
        }

    long cMillis = System.currentTimeMillis() - ldtStart;
    System.out.println(cMillis + ", " + cTotal);
    }

And the results:

java16 Main 10 1000000000
46, 1000000000
1934, 1000000000
1875, 1000000000
1897, 1000000000
1924, 1000000000
1938, 1000000000
1884, 1000000000
1890, 1000000000
1895, 1000000000
1943, 1000000000

A few observations:

• We can see the getVirtual() method (or least its logic) is getting run
• the cost of both the fast iteration and the slow iteration increased
• the performance difference between fast and slow runs is down to ~40

Still what’s the deal, running the test for longer doesn’t yield any additional fast iterations.  So at this point I get a few other people involved, and they work through many of the same suggestions and assumptions I’d listed above.  We also try the following:

• pull the logic out of main() and put it in a non-static method
• try running the test loop in parallel on multiple threads
• try first warming up the JVM by running some random but heavy code prior to running the test
• try recording the results into an array rather then printing during the test

All of these yield essentially the same results as above.  We do randomly trigger the loss of the fast iteration, but never trigger multiple fast iterations.  So yipee we figured out how to make things go slower.  As a side note it was intersting what would trigger the loss of the fast iteration, which was triggered by recording the results into an array as follows:

long[] acMillis = new long[cIters];
int[] acTotal  = new int[cIters];
for (int i = 0; i < cIters; ++i)
    {
    long ldtStart = System.currentTimeMillis();
    int  cTotal   = 0;
    for (int j = 0; j < cOps; ++j)
        {
        cTotal += test.getVirtual();
        }

    acMillis[i] = System.currentTimeMillis() - ldtStart;
    acTotal [i] = cTotal;
    }

for (int i = 0; i < cIters; ++i)
    {
    System.out.println(acMillis[i] + ", " + acTotal[i]);
    }

java16 Main 10 1000000000
1328, 1000000000
1346, 1000000000
1338, 1000000000
1308, 1000000000
1343, 1000000000
1335, 1000000000
1324, 1000000000
1354, 1000000000
1336, 1000000000
1344, 1000000000

And even odder we could “fix” this and get an initial fast result by recording the time measurements int an Object array rather then a long array. No kidding, dynamic object allocation improved results.

Object[] acMillis = new Object[cIters];
int[] acTotal  = new int[cIters];
for (int i = 0; i < cIters; ++i)
    {
    long ldtStart = System.currentTimeMillis();
    int  cTotal   = 0;
    for (int j = 0; j < cOps; ++j)
        {
        cTotal += test.getVirtual();
        }

    acMillis[i] = new Long(System.currentTimeMillis() - ldtStart);
    acTotal [i] = cTotal;
    }

for (int i = 0; i < cIters; ++i)
    {
    System.out.println(acMillis[i] + ", " + acTotal[i]);
    }

java16 Main 10 1000000000
46, 1000000000
1806, 1000000000
1791, 1000000000
1831, 1000000000
1775, 1000000000
1779, 1000000000
1806, 1000000000
1811, 1000000000
1789, 1000000000
1783, 1000000000

Ok, so clearly this is nuts. But then things start to come into focus, lets go back to a version where the optimizer can optimize out the getVirtual() call, in fact we’ll do it for them.

for (int i = 0; i < cIters; ++i)
    {
    long ldtStart = System.currentTimeMillis();
    for (int j = 0; j < cOps; ++j)
        {
        // cTotal += test.getVirtual();
        }
    long cMillis = System.currentTimeMillis() - ldtStart;
    System.out.println(cMillis);
    }

java16 Main 10 1000000000
6
865
875
859
856
969
875
872
861
877

Ok, so we’ve so removed the thing we were originally trying to test, and performance improves across the board. We get a 40% savings on the slow runs. So we see not too surprisingly that of the time we were measuring nearly half was in the inner ‘for’ loop not in the thing we’d intended to test. And now we the ah ha moment, how does the optimizer look at optimizing the test “infrastructure”, i.e. our main method, and we make this change.

public static void main(String[] asArg)
    {
    int cIters = Integer.parseInt(asArg[0]);
    int cOps   = Integer.parseInt(asArg[1]);

    for (int i = 0; i < 10; ++i)
        {
        runTest(cIters, cOps);
        System.out.println("pass " + i + " complete");
        }
    }

public static void runTest(int cIters, int cOps)
    {
    Test test = new Test();
    for (int i = 0; i < cIters; ++i)
        {
        long ldtStart = System.currentTimeMillis();
        int  cTotal   = 0;
        for (int j = 0; j < cOps; ++j)
            {
            cTotal += test.getVirtual();
            }
        long cMillis = System.currentTimeMillis() - ldtStart;
        System.out.println(cMillis + ", " + cTotal);
        }
    }

java16 Main 10 1000000000
46, 1000000000
1886, 1000000000
1894, 1000000000
1902, 1000000000
1877, 1000000000
1870, 1000000000
1862, 1000000000
1910, 1000000000
1861, 1000000000
1878, 1000000000
pass 0 complete
1869, 1000000000
1899, 1000000000
1897, 1000000000
1866, 1000000000
1882, 1000000000
1855, 1000000000
1895, 1000000000
1890, 1000000000
1903, 1000000000
1880, 1000000000
pass 1 complete
38, 1000000000
41, 1000000000
40, 1000000000
43, 1000000000
39, 1000000000
42, 1000000000
39, 1000000000
42, 1000000000
61, 1000000000
44, 1000000000
pass 2 complete
40, 1000000000
41, 1000000000
41, 1000000000
42, 1000000000
39, 1000000000
39, 1000000000
42, 1000000000
….

Eureka, finally we can get lots sustained fast runs. So it appears that we need to pop out of our test code in order for the optimizer to undo the apparent deoptimization it had made during the second iteration. Note that it was not simply that we’d run more total iterations, because we’d tried that earlier, it is in fact popping out of the test function, which apparently allows an optimized version to be slipped in. This ends up being an important bit to remember to account for in doing these types of micro benchmarks.

At this point I’d assumed the “fast” runs had inlined the body of the getVirtual() method into the ‘for’ loop, and that the “slow” runs did call the function. Changing the code to manually inline however don’t back this up, leaving me with more questions. At least the mystery of the broken optimizer are solved though, that is enough for now.