如何用Java编写(并运行)正确的微基准测试?
我正在寻找一些代码示例和注释,以说明需要考虑的各种事情。
示例:基准测试应该测量时间/迭代还是迭代/时间,为什么?
相关:秒表基准测试是可接受的?
如何用Java编写(并运行)正确的微基准测试?
我正在寻找一些代码示例和注释,以说明需要考虑的各种事情。
示例:基准测试应该测量时间/迭代还是迭代/时间,为什么?
相关:秒表基准测试是可接受的?
当前回答
基准应该测量时间/迭代还是迭代/时间,为什么?
这取决于你要测试什么。
如果您对延迟感兴趣,则使用时间/迭代,如果您对吞吐量感兴趣,则使用迭代/时间。
其他回答
Java基准测试的重要事项是:
Warm up the JIT first by running the code several times before timing it Make sure you run it for long enough to be able to measure the results in seconds or (better) tens of seconds While you can't call System.gc() between iterations, it's a good idea to run it between tests, so that each test will hopefully get a "clean" memory space to work with. (Yes, gc() is more of a hint than a guarantee, but it's very likely that it really will garbage collect in my experience.) I like to display iterations and time, and a score of time/iteration which can be scaled such that the "best" algorithm gets a score of 1.0 and others are scored in a relative fashion. This means you can run all algorithms for a longish time, varying both number of iterations and time, but still getting comparable results.
我正在写一篇关于。net基准测试框架设计的博客。我有一些以前的帖子,也许能给你一些想法——当然,不是每件事都合适,但其中一些可能是合适的。
我知道这个问题已经被标记为已回答,但我想提到两个帮助我们编写微基准的库
来自谷歌的卡尺
入门教程
http://codingjunkie.net/micro-benchmarking-with-caliper/ http://vertexlabs.co.uk/blog/caliper
来自OpenJDK的JMH
入门教程
避免JVM上的基准测试陷阱 使用JMH进行Java微基准测试 JMH简介
为了补充其他优秀的建议,我还会注意以下几点:
For some CPUs (e.g. Intel Core i5 range with TurboBoost), the temperature (and number of cores currently being used, as well as thier utilisation percent) affects the clock speed. Since CPUs are dynamically clocked, this can affect your results. For example, if you have a single-threaded application, the maximum clock speed (with TurboBoost) is higher than for an application using all cores. This can therefore interfere with comparisons of single and multi-threaded performance on some systems. Bear in mind that the temperature and volatages also affect how long Turbo frequency is maintained.
也许您可以直接控制的一个更根本的重要方面是:确保您在测量正确的东西!例如,如果您正在使用System.nanoTime()对特定代码进行基准测试,请将对赋值的调用放在有意义的位置,以避免测量您不感兴趣的内容。例如,不要做:
long startTime = System.nanoTime();
//code here...
System.out.println("Code took "+(System.nanoTime()-startTime)+"nano seconds");
问题是,当代码完成时,您不能立即得到结束时间。相反,试试下面的方法:
final long endTime, startTime = System.nanoTime();
//code here...
endTime = System.nanoTime();
System.out.println("Code took "+(endTime-startTime)+"nano seconds");
在Java中编写微基准测试有许多可能的陷阱。
首先:你必须计算各种随机占用时间的事件:垃圾收集、缓存效果(OS对文件的缓存效果和CPU对内存的缓存效果)、IO等。
第二:对于非常短的时间间隔,您不能相信测量时间的准确性。
第三:JVM在执行时优化代码。因此,在同一个jvm实例中不同的运行将变得越来越快。
My recommendations: Make your benchmark run some seconds, that is more reliable than a runtime over milliseconds. Warm up the JVM (means running the benchmark at least once without measuring, that the JVM can run optimizations). And run your benchmark multiple times (maybe 5 times) and take the median-value. Run every micro-benchmark in a new JVM-instance (call for every benchmark new Java) otherwise optimization effects of the JVM can influence later running tests. Don't execute things, that aren't executed in the warmup-phase (as this could trigger class-load and recompilation).
http://opt.sourceforge.net/ Java Micro Benchmark -确定不同平台上计算机系统的比较性能特征所需的控制任务。可用于指导优化决策和比较不同的Java实现。