benchmarking - Create quick/reliable benchmark with java?

Question

I'm trying to create a benchmark test with java. Currently I have the following simple method:

public static long runTest(int times){
    long start = System.nanoTime();     
    String str = "str";
    for(int i=0; i<times; i++){
        str = "str"+i;
    }       
    return System.nanoTime()-start;     
}

I'm currently having this loop multiple times within another loop that is happening multiple times and getting the min/max/avg time it takes to run this method through. Then I am starting some activity on another thread and testing again. Basically I am just wanting to get consistent results... It seems pretty consistent if I have the runTest loop 10 million times:

Number of times ran: 5
The max time was: 1231419504 (102.85% of the average)
The min time was: 1177508466 (98.35% of the average)
The average time was: 1197291937
The difference between the max and min is: 4.58%

Activated thread activity.

Number of times ran: 5
The max time was: 3872724739 (100.82% of the average)
The min time was: 3804827995 (99.05% of the average)
The average time was: 3841216849
The difference between the max and min is: 1.78%

Running with thread activity took 320.83% as much time as running without.

But this seems a bit much, and takes some time... if I try a lower number (100000) in the runTest loop... it starts to become very inconsistent:

    Number of times ran: 5
    The max time was: 34726168 (143.01% of the average)
    The min time was: 20889055 (86.02% of the average)
    The average time was: 24283026
    The difference between the max and min is: 66.24%

    Activated thread activity.

    Number of times ran: 5
    The max time was: 143950627 (148.83% of the average)
    The min time was: 64780554 (66.98% of the average)
    The average time was: 96719589
    The difference between the max and min is: 122.21%

    Running with thread activity took 398.3% as much time as running without.

Is there a way that I can do a benchmark like this that is both consistent and efficient/fast?

I'm not testing the code that is between the start and end times by the way. I'm testing the CPU load in a way (see how I'm starting some thread activity and retesting). So I think that what I'm looking for it something to substitute for the code I have in "runTest" that will yield quicker and more consistent results.

Thanks

Answer 1

In short:

(Micro-)benchmarking is very complex, so use a tool like the Benchmarking framework http://www.ellipticgroup.com/misc/projectLibrary.zip - and still be skeptical about the results ("Put micro-trust in a micro-benchmark", Dr. Cliff Click).

In detail:

There are a lot of factors that can strongly influence the results:

• The accuracy and precision of System.nanoTime: it is in the worst case as bad as of System.currentTimeMillis.
• code warmup and class loading
• mixed mode: JVMs JIT compile (see Edwin Buck's answer) only after a code block is called sufficiently often (1500 or 1000 times)
• dynamic optimizations: deoptimization, on-stack replacement, dead code elimination (you should use the result you computed in your loop, e.g. print it)
• resource reclamation: garbace collection (see Michael Borgwardt's answer) and object finalization
• caching: I/O and CPU
• your operating system on the whole: screen saver, power management, other processes (indexer, virus scan, ...)

Brent Boyer's article "Robust Java benchmarking, Part 1: Issues" ( http://www.ibm.com/developerworks/java/library/j-benchmark1/index.html) is a good description of all those issues and whether/what you can do against them (e.g. use JVM options or call ProcessIdleTask beforehand).

You won't be able to eliminate all these factors, so doing statistics is a good idea. But:

• instead of computing the difference between the max and min, you should put in the effort to compute the standard deviation (the results {1, 1000 times 2, 3} is different from {501 times 1, 501 times 3}).
• The reliability is taken into account by producing confidence intervals (e.g. via bootstrapping).

The above mentioned Benchmark framework ( http://www.ellipticgroup.com/misc/projectLibrary.zip) uses these techniques. You can read about them in Brent Boyer's article "Robust Java benchmarking, Part 2: Statistics and solutions" ( https://www.ibm.com/developerworks/java/library/j-benchmark2/).