Benchmark Results for Toba

Note: these are very dated, but they're still interesting.

We have run several benchmarks to test the performance of Toba-generated applications. We have used both the UCSD Java Microbenchmarks and several applications to compare our performance against existing systems. We actually use a modified version of the UCSD benchmarks that includes additional tests for more thorough testing of synchronization and exception handling. Benchmarks include:

Various microbenchmarks, derived from the UCSD Java Microbenchmarks
javac, Sun's Java bytecode compiler
espresso, a free Java bytecode compiler
Toba, our Java-to-C translator
JavaLex, a lexical-analyzer generator written in Java
JavaCup, an LALR(1) parser generator written in Java
JHLZip and JHLUnzip, compression programs written in Java

Benchmark Environment

We ran each benchmark on a Sun SparcStation 20 with 2 processors running Solaris 2.5.1. Where appropriate, we compiled all programs and benchmarks with the Sun's commercial SunPro cc. We used full optimization (-xO4 -xcg92) for all compilation. We compared code generated using Toba-generated code to using Sun's Interpreter (JDK Version 1.0.2), the Kaffe Interpreter, and the Guava Java JIT compiler (Version 1.0 Beta 1).

Microbenchmarks (UCSD + others)

Description of Benchmarks

Arithmetic operations
- Add 1,000,000 integers in a loop.
- Multiply 1,000,000 integers in a loop.
- Add 1,000,000 doubles in a loop.
- Multiply 1,000,000 doubles in a loop.
Class Field and Method Access
- Read Integer integer field 1,000,000 times.
- Call method defined in the same object 1,000,000 times.
- Call method defined in another object 1,000,000 times.
- Invoke an interface method 1,000,000 times.
Exceptions Handling
- Throw an exception and catch it in the same method 1,000,000 times.
- Throw an exception and catch it in a the caller 1,000,000 times.
- Throw an exception and catch it after unwinding past one intervening catch clause 1,000,000 times.
- Throw an exception and catch it in a procedure 10 levels back in the call chain 1,000,000 times.
Synchronization
- Enter a synchronized block 1,000,000 times in a single-threaded program.
- Enter a synchronized method 1,000,000 times in a single-threaded program.
- Enter a synchronized block 1,000,000 times in a multi-threaded program.
- Enter a synchronized method 1,000,000 times in a multi-threaded program.
- Note: We test synchronization (entering and exiting monitors) both before and after multiple threads exist, because it is possible to optimize monitor access in single-threaded programs.
Miscellaneous Benchmarks
- 1,000,000 iterations of an empty loop.
- 1,000,000 assignments to an integer array.
- 1 call to Runtime.gc().
- 3 Threads each calling yield 10,000 times.
- 100,000 1-byte write calls.
- 1 100,000-byte write call.

Results

Benchmark	JDK 1.0.2	Kaffe	Guava	Toba
Arithmetic	(sec)	(sec)	(sec)	(sec)
Add ints	3.236	18.765	0.150	0.117
Mult Ints	3.496	17.803	0.216	0.185
Add Doubles	4.123	23.729	0.508	0.274
Mult Doubles	4.156	23.105	0.488	0.279
Class Access
Read integer instance variable	2.796	15.294	0.172	0.102
Call to method in same object	3.119	17.884	0.354	0.235
Call to method in another object	3.741	22.400	0.410	0.268
invoke interface method	4.130	24.726	1.397	1.405
Throw Exceptions
local	5.026	19.424	17.634	1.325
non-local	7.315	34.722	33.268	6.616
Catch exception past 1 intervening catch clause	11.516	37.006	39.609	8.079
Catch exception 10 calls away	41.245	392.769	124.182	8.177
Synchronization - Single-threaded program
synchronized blocks	16.088	12.400	6.349	1.683
synchronized methods	15.914	19.852	6.587	2.546
Synchronization - Multi-threaded program
synchronized blocks	16.135	11.867	6.259	3.519
synchronized methods	16.114	20.092	6.518	4.596
Miscellaneous Benchmarks
Empty Loop	1.115	4.547	0.071	0.033
Assign to Integer Array	2.781	12.357	0.294	0.186
Runtime.gc()	0.022	0.025	abort	0.403
3 Threads doing 10,000 yields	2.724	3.284	1.952	0.485
100,000 1-byte `writes`	4.671	20.039	3.898	3.125
1 100,000 byte `write`	0.009	0.008	0.009	0.006

Discussion of Results

In virtually all cases, programs compiled with Toba run much faster than the other systems. In particular,

For arithmetic and class access, code generated using Toba is reliably much faster than the using JDK 1.0.2 and Kaffe. This is because both the JDK and Kaffe are interpreted systems. In addition, Toba-generated code is also at least as fast and usually faster than Guava's.
Exception handling in Toba-generated code is very fast .
Synchronization in Toba-generated code is much faster in single-threaded programs because the Toba runtime optimizes monitor accesses while a program is single-threaded.
Runtime.gc() is noticeably slower when using Toba.

Application Benchmarks

Description of Benchmarks

javac - Compile the Toba source files using Sun's javac compiler
espresso - Compile the Toba source files using the Espresso compiler
Toba - Compile the Toba class files into C source files
JavaLex - Use JavaLex to generate a lexical analyzer for the Tiger.lex grammer distributed with JavaLex.
java_cup - Use java_cup to generate a parser for the example grammar distributed with java_cup
JHLZip - Use JHLzip to package 4 files from /usr/dict/
JHLUnzip - Use JHLUnzip to unpack 4 file originally from /usr/dict/

Results

Benchmark	JDK 1.0.2 (sec)	Guava (sec)	Toba (sec)
javac	45.01	17.85	12.8
espresso	24.9	abort	4.9
Toba	59.8	36.8	20.1
JavaLex	176.3	80.8	38.8
JavaCup	8.3	6.2	2.9
JHLZip	50.7	abort	5.8
JHLUnzip	52.2	abort	5.9

Discussion

Applications compiled using Toba run significantly faster than when using the JDK. Programs generated using Toba are also faster than running with the Guava JIT system, and Toba programs run to completion in several applications where the Guava system fails.

Toba: A Java-to-C Translator
Sumatra Project
Department of Computer Science
Last modified: Thu Nov 7 07:47:12 MST 1996