JavaBenchEmbedded
From JopWiki
Running benchmarks is problematic, both generally and especially in the case of embedded systems. The best benchmark would be the application that is intended to run on the system being tested. To get comparable results SPEC provides benchmarks for various systems. However, the one for Java, the SPECjvm98, is usually too large for embedded systems.
Due to the absence of a standard Java benchmark for embedded systems, a small benchmark suit that should run on even the smallest device is provided here. It contains several micro-benchmarks for evaluating the number of clock cycles for single bytecodes or short sequences of bytecodes, one kernel benchmark, and several application benchmarks.
Contents |
[edit] Benchmark Methodology
As we will see, there is a great variation in processing power across different embedded systems. To cater for this variation, all benchmarks are `self adjusting'. Each benchmark consists of an aspect that is benchmarked in a loop. The loop count adapts itself until the benchmark runs for more than a second. The number of iterations per second is then calculated, which means that higher values indicate better performance.
All the benchmarks measure how often a function is executed per second. In the Kfl benchmark, this function contains the main loop of the application that is executed in a periodic cycle in the original application. In the benchmark the wait for the next period is omitted, so that the time measured solely represents execution time. The UDP benchmark contains the generation of a request, transmitting it through the UDP/IP stack, generating the answer and transmitting it back as a benchmark function. The iteration count is the number of received answers per second.
[edit] Benchmark Types
[edit] Micro Benchmarks
The micro-benchmarks are intended to give insight into the implementation of the JVM. As almost all bytecodes manipulate the stack, it is not possible to measure the execution time for a single bytecode. As a minimum requirement, a second instruction is necessary to reverse the stack operation. For compiling versions of the JVM, these micro-benchmarks do not produce useful results. The compiler performs optimizations that make it impossible to measure execution times at this fine a granularity.
[edit] Benchmark Kernels
At the moment only Sieve is available in this category. It would be interesting to port some of the kernels from the WCET benchmark project to Java.
[edit] Application Benchmarks
All applications consist of a main loop that gets executed periodically.
[edit] Kfl
To provide a realistic workload for embedded systems, a real-time application was adapted to create the first application benchmark (Kfl). The application is taken from one of the nodes of a distributed motor control system \cite{jop:wises03}. Kfl is one node of a distributed RTS to tilt the line over a train for easier loading and unloading of goods wagon. A simulation of both the environment (sensors and actors) and the communication system (commands from the master station) forms part of the benchmark, so as to simulate the real-world workload.
[edit] UdpIp
The second application benchmark is an adaptation of a tiny TCP/IP stack for embedded Java. This benchmark contains two UDP server/clients, exchanging messages via a loopback device.
[edit] Lift
Lift is a lift controller used in an automation factory.
[edit] Configuration and Usage
The benchmark framework needs only two system functions: One to measure time in millisecond resolution and one to print the results. These functions are encapsulated in LowLevel.java and can be adapted to environments, where the full Java library is not available. For example, the leJOS system has very limited output capabilities and and a special LowLevel.java exists for this device. The benchmark is straight-forward to use. An example on a standard JVM is:
javac jbe/DoAll.java java jbe.DoAll
[edit] Versions
- jbe 1.0 Used for the JOP thesis and various papers. Main applications are Kfl and UdpIp
- jbe 1.1 2007-04-11
- cleanup of LowLevel.java - other devices are included in the single LowLevel.java in comments
- less verbose output
- Print clock cycles when LowLevel.FREQU is set
- Smaller packet sizes in ejip for memory constraint devices
- Added Lift to the application benchmarks
[edit] Results
This section contains benchmark results from various embedded Java systems. If you have access to an embedded Java devices please run the benchmark and add your results. Thanks to Philipp Wasmayr the benchmark is also available as MIDLet. So feel free to test your Java enabled mobile phone and add the results.
