Synthetic v's Application benchmarks [1]
Benchmarks are designed to mimic a particular type of workload on a component or system. Synthetic benchmarks do this by specially created programs that impose the workload on the component. Application benchmarks run real-world programs on the system. Whilst application benchmarks usually give a much better measure of real-world performance on a given system, synthetic benchmarks are useful for testing individual components, like a hard disk or networking device.
Synthetic Benchmarks:
Whetstone benchmark is based on floating point algorithm performance tests [2].
Dhrystone benchmarking is based entirely on integer based algorithm performance tests.
The SPECint (Standard Performance Evaluation Corporation) suite evolved from the Dhrystone benchmark.
Application Benchmarks:
Joslin [5] emphasizes three characteristics of the application benchmark.
1. The routines are actually run on the proposed systems.
2. The total throughput time is considered.
3. The routines are aimed at specific applications.
Unrepresentative workloads being tested will provide unreliable benchmark results. Workload modelling is concerned with establishing workloads.
Joslin suggests Four factors to consider:
1. Types of processing — logic, computation, housekeeping, etc.
2. Time requirements — compile, execute, tapes, card units, printers, etc.
3. Equipment requirements — core, I/O channels,printers, etc.
4. Order of the benchmark programs (priority).
The effort and cost involved in designing and applying Application benchmarks gave rise to the Synthetic benchmarks, the most high profile of which include SPECint and TPC benchmarks.
References
[1] http://en.wikipedia.org/wiki/Benchmark_%28computing%29
[2] http://www.roylongbottom.org.uk/whetstone%20results.htm#anchorjava
[3] http://www.spec.org/
[4] http://www.spec.org/cpu2000/results/cint2000.html
[5] The Evolution of Benchmarking as a Computer Performance Evaluation Technique, MIS Quarterly; Mar85, Vol. 9 Issue 1, p7-16, 10p.
Benchmarks are designed to mimic a particular type of workload on a component or system. Synthetic benchmarks do this by specially created programs that impose the workload on the component. Application benchmarks run real-world programs on the system. Whilst application benchmarks usually give a much better measure of real-world performance on a given system, synthetic benchmarks are useful for testing individual components, like a hard disk or networking device.
Synthetic Benchmarks:
Whetstone benchmark is based on floating point algorithm performance tests [2].
Dhrystone benchmarking is based entirely on integer based algorithm performance tests.
The SPECint (Standard Performance Evaluation Corporation) suite evolved from the Dhrystone benchmark.
Application Benchmarks:
Joslin [5] emphasizes three characteristics of the application benchmark.
1. The routines are actually run on the proposed systems.
2. The total throughput time is considered.
3. The routines are aimed at specific applications.
Unrepresentative workloads being tested will provide unreliable benchmark results. Workload modelling is concerned with establishing workloads.
Joslin suggests Four factors to consider:
1. Types of processing — logic, computation, housekeeping, etc.
2. Time requirements — compile, execute, tapes, card units, printers, etc.
3. Equipment requirements — core, I/O channels,printers, etc.
4. Order of the benchmark programs (priority).
The effort and cost involved in designing and applying Application benchmarks gave rise to the Synthetic benchmarks, the most high profile of which include SPECint and TPC benchmarks.
References
[1] http://en.wikipedia.org/wiki/Benchmark_%28computing%29
[2] http://www.roylongbottom.org.uk/whetstone%20results.htm#anchorjava
[3] http://www.spec.org/
[4] http://www.spec.org/cpu2000/results/cint2000.html
[5] The Evolution of Benchmarking as a Computer Performance Evaluation Technique, MIS Quarterly; Mar85, Vol. 9 Issue 1, p7-16, 10p.
No comments:
Post a Comment