MATSim Benchmark

Overview

The performance of MATSim depends on a lot of different factors:

• CPU-speed (although by far not always the limiting factor!)
• Memory-Bus / -Controller (we're moving huge amounts of memory, the faster the better)
• Java Virtual Machine (JVM 1.6 is usually faster than JVM 1.5)
• File system (local Hard drive vs. RAID vs. NFS vs ...)

To get a better understanding, under which circumstances MATSim performs best, we created a simple benchmark (performance test) that runs 20 iterations of a sample scenario with different settings. If you run the benchmark on your machine, we would be happy if you could send us your results.

Download and Installation

Download the following zip-file: benchmark.zip [35MB]

Unzip the downloaded file.

Running the benchmark

java -Xmx500m -jar Benchmark.jar

This will generate a directory output with some files in it from the run. The test will usually run between 25 and 40 minutes. The benchmark requires Java 1.5 or newer and 150MB free disk space.

If you want to re-run the benchmark, rename or delete the output directory and run the test again.

Submitting benchmark results

Please send an email to benchmark AT matsim DOT org containing:

• the file output/stopwatch.txt
• the file output/logfile.log
• a description of your benchmark environment, including:
  • vendor of machine (e.g. Sun, Dell, Apple, ...)
  • processor-type (vendor (AMD, Intel, ...), model-number, clock-speed, cache-size, number of processors and cores, ...)
  • memory (bus-speed, memory-controller, ...)
  • storage system (rpm, cache size, ..., type: e.g. local hard drive, RAID, ...)
  • operation system
  • java virtual machine
  • any other information you think might be of interest to us

Benchmark results analysis

We collected some results and did a short analysis on them. Have a look at the results. If you want your results included as well or have some interesting findings yourself, please submit us our benchmark results.

MATSim Benchmark Results

The benchmark contains parts of the code running in parallel (the replanning part, using 4 threads) and other parts running single-threaded.

Speed comparison

The benchmark was run on some of our production servers with different versions of Java Virtual Machines. The servers have 2 Single-Core "AMD Opteron Processor 248" running at 2.2 GHz (date of purchase: fall 2004, so they have to be considered as "old").
The Java Virtual Machines tested were:

• IBM Java 5 64bit: The default Java 5 version on the servers. The JVM identified itself as "J2RE 1.5.0 IBM J9 2.3 Linux amd64-64 j9vmxa6423ifx-20080811"
• Sun 6u5 64bit: The default Java 6 version on the servers, identifying itself as "1.6.0_05; Sun Microsystems Inc.; mixed mode; 64-bit"
• Sun 5u19 64bit: The latest (as of writing this text) Java 5 version from Sun, Java 5 update 19 64-bit.
• Sun 5u19 32bit: The latest Java 5 version from Sun, Java 5 update 19 32-bit.
• Sun 6u14 64bit: The latest Java 6 version from Sun, Java 6 update 14 64-bit.
• Sun 6u14 32bit: The latest Java 6 version from Sun, Java 6 update 14 32-bit.
• Sun 6u14 64bit COP: Sun's Java 6 update 14 64-bit, started with the extra argument "-XX:+UseCompressedOops". Compressed Object Pointers should "improve performance of the 64-bit JRE when the Java object heap is less than 32 gigabytes in size" (see Java SE 6 Update 14 Release Notes). As an additional advantage, memory consumption should also be a bit lower when using only 32bits for object pointers.
• Sun 6u14 64bit COP + AO: Sun's Java 6 update 14 64-bit, started with the extra argument "-XX:+UseCompressedOops -XX:AggressiveOpts". In addition to using Compressed Object Pointers, also try out the new "experimental implementation of java.util.TreeMap that can improve the performance" as MATSim makes heavy use of TreeMaps (although not necessarily that often iterating over them)
• Sun 6u14 32bit AO: Sun's Java 6 update 14 32-bit, started with the extra argument "-XX:AggressiveOpts". Just for comparison, also start the 32-bit version of the JVM with the aggressive optimization option.

The shown number in above graph are the average of two runs of the benchmark for each configuration (Yeah, two isn't that big a sample, we know... but it should still be valid to demonstrate some findings).
What can be observed is the huge difference of execution time in general between 32-bit and 64-bit versions of the virtual machines. The "compressed object pointers" (COP) feature of Suns JVM 6u14 seems to compensate for this nicely, making the 64-bit version about the same speed than the 32-bit version. The aggressive optimization options (AO) on the other hand doesn't seem to influence the performance of MATSim drastically.
While the difference between Suns Java 5 and Java 6 versions in the total execution time seem more or less random, there seems to be a performance improvement for the multithreaded replanning part by changing from Java 5 to Java 6. Interestingly, despite IBM's worse multithreaded performance, it is able to catch up in the single-threaded parts to come in with a similar total execution time than Sun's 64-bit JVMs.

The benchmark was also run on other server machines:

• Servers with two Dual-Core AMD Opteron 2222 processors, running at 3.0 GHz; date of purchase winter 2007/2008 (very similar architecture than the previously tested AMD Opteron 248 systems, just "newer" with higher clocked CPU)
• Servers with Intel Xeon X5355 Quad-Core processors, clocked at 2.66 GHz, 8 MB L2 cache, built with 65 nm technology
• Servers with Intel Xeon E5430 Quad-Core processors, clocked at 2.66 GHz, 12 MB L2 cache, built with 45 nm technology
• Servers with Intel Xeon E5530 Quad-Core processors, clocked at 2.4 GHz, 8 MB L3 cache, 45 nm technology, Nehalem architecture
• Servers with Intel Xeon E7540 Hex-Core processors, clocked at 2.0 GHz, 18 MB L3 cache, 45 nm technology, Nehalem architecture, DDR3 RAM

Comparing the AMD servers running at 2.2 GHz and 3.0 GHz, the most obvious difference is the time for the replanning, explainable by the different number of cores the machines have. Interestingly, the remaining execution time didn't really improve by the change in CPU speed, leading to the guess that the performance of the memory controller or the memory bus is limiting the speed of MATSim (both AMD servers seem to have a front side bus of 1000 MHz).

The Intel servers were massively faster than the AMD machines. We do not yet know if it's the different memory controller, faster memory bus, or if Sun's JDK is just more optimized for Intel processors. Anyway, the difference is striking. And each newer generation of Intel processors seems to deliver a real performance upgrade, even when running at a lower clock-speed.

At last, the benchmark was also run on some of our laptop machines: An Apple MacBook Pro with a Intel Core 2 Duo processor clocked at 2.33 GHz (model from fall 2006), running Mac OS X 10.5.7, and an IBM/Lenovo Laptop with the same Intel Core 2 Duo processor, 2.33 GHz, running a Gentoo 64-bit Linux (Kernel 2.6.28). Both laptops have a front side bus of 667 MHz, so from a technical view point they are very similar.

On the Apple MacBook Pro, the following Java Virtual Machines were used:

• Apple JVM 5u16 32bit: The default Java 5 version on Mac OS provided by Apple.
• Apple JVM 6u7 64bit: The default Java 6 version on Mac OS provided by Apple.
• Soylatte 6u3 32bit: An early port of OpenJDK 6, identifying itself as "1.6.0_03-p3; Sun Microsystems Inc.; mixed mode; 32-bit", provided by Landon Fuller.

On the Lenovo Laptop, an OpenJDK 6u0 64-bit JVM was used.

 

Surprisingly, Apple pulled the trick to make their 64-bit Java 6 a lot faster than the older 32-bit Java 5—well, it could also mean that their Java 5 offering is just very slow... The Mac-port of OpenJDK 6 ("Soylatte") is even a bit faster, but that may be likely due to the difference between 64-bit and 32-bit.

Are you able to run the MATSim benchmark even faster? Please tell us so! We're very interested in your benchmark results.

Memory usage comparison

MATSim writes out information about memory usage from time to time into the logfile. Plotting this information gives a jagged line running from left to right. Heights and lows in the plot can be explained with the Java Garbage Collector, only freeing up the memory from time to time. Still, one can guess the absolute minimum of memory required by MATSim by looking at the lower parts of the curve (that's then when a Garbage Collection just ran, showing all the memory that could not be collected).

Comparing the memory consumption in Sun's (currently) latest Java VM version holds no real surprises.

It can be clearly seen that the 64-bit JVM uses the largest amount of memory, due to the fact that each object pointer takes up 8 bytes. The 32-bit JVM uses the least memory. The 64-bit JVM with compressed object pointers seems to lie somewhere in between—although I would have expected it to be comparable to the 32-bit JVM, it seems that it still uses a bit more memory for unknown reasons. Anyway, it comes in handy to know that one can load now larger scenarios on a 32GB (or less) machine. The memory savings, compared to a 64-bit JVM without compressed object pointers, should be even bigger the larger the scenario is or the more details the simulated network has, so this feature really looks promising.
 

Benchmark: Parallel Events Handling and JDEQSim

The following shows two different benchmarks using jdeqsim, parallel events handling, or both.

The first benchmark uses the ivtch-osm network (~60k links), while the second one uses a navteq network with much more links.

QueueSim vs. JDEQSim; parallel Events Handling

Zrh 10%, ivtch-osm network; computing times per iteration (computer = cluster4 = 2x "Dual-Core AMD Opteron Processor 2222", 3.0 GHz, 1000 MHz FSB).  Runs 669, 676, 678, 679

QueueSim vs. JDEQSim; parallel Events Handing

Computing times per Iterations.  Scenario = navteq network of Switzerland; computer = cluster4 = servers with 2x "Dual-Core AMD Opteron Processor 2222", 3.0 GHz, 1000 MHz FSB.

How To Speed Up MATSim Runs

The following list gives some hints on how to speed up MATSim simulation runs.

• Use an up-to-date Java version: Java 6 usually performs better than Java 5.
• Use Compressed Object Pointers in 64-bit JVM (available since Sun Java 6 Update 14). If you use a 64-bit JVM and use at most 32 GB of RAM, start Sun's Java virtual machine with the argument "-XX:+UseCompressedOops". This reduces the size of object pointers to 32 bit, making many pointer operations a lot faster than when they were 64 bit long, while still supporting heap sizes larger than 2 GB (what a regular 32-bit JVM would do). See the Java SE 6 Update 14 Release Notes for more details. In addition, we observed notable speed-ups in the region of 10% in the run time of MATSim.
• Limit writing events. Writing the simulation events to files in each iteration not only consumes a lot of disk space, but also a considerable amount of time. Add the following parameter to your configuration to restrict writing events to certain iterations.
  

  <module name="controler">
    <param name="writeEventsInterval" value="10" />
  </module>

  This will instruct the MATSim Controler to write events only every 10th iteration, which is usually enough for analyses.

• Use faster routing algorithms. By default, MATSim uses a routing algorithm based on Dijkstra's shortest path algorithm. But MATSim also includes a faster routing algorithm, based on the A* algorithm with landmarks. To use this routing algorithm, add the following configuration parameter to your configuration file:
  

  <module name="controler">
    <param name="routingAlgorithmType" value="AStarLandmarks" />
  </module>