Test Cases

Why Tests?

Our main reason to have tests is to ensure we do not loose existing functionality when adding new functionality. Another reason, hopefully getting less important, is to figure out what influences bugs had, respectively what is influenced by a bugfix.

Possible Examples of Tests

The following list is not complete and likely never will. It is more to give you an idea, what could be tested:

  • readers and writers for data (Generating data, writing data to file. Reading the file and writing it out again should produce the same file as before).
  • calculating link travel times based on events
  • events-generation, e.g. from the transport simulation
  • calculation of plans' scores
  • are parameters correctly reflected in the execution (e.g. when replanning?)
  • ...

Automatic Run of the Tests

The tests are automatically run every night. When you log in on matsim.org, you'll find a link in the left navigation area ("developer info") where you can look at the nightly status of the tests. If errors or failures occur, an email is sent to Marcel Rieser who will then inform people as necessary.

More Information

Running the Tests

General Remarks

  • The tests currently require around 400 MB of RAM to run successfully on a 32-bit JVM (600 MB on 64-bit JVM).

In Eclipse

To run all tests, create a new Run-Configuration. Select to run all Tests in src/test/java with a JUnit 4 runner.

In the Arguments tab, do not forget to increase the available memory for the JVM (-Xmx600m), then click "Run".

With Maven on the Command Line

export MAVEN_OPTS=-Xmx600m
mvn test

Maven should output a summary, listing the number of all tests as well as the number of failed tests (if any).

Guidelines for Writing Tests

(The following list has been updated for JUnit 4.7. Many existing tests written for JUnit 3.8 still follow different guidelines.)

  • Name your class containing the test-methods such that it starts or ends with "Test", e.g. MyModuleTest.
    Otherwise, your class may not be recognized as test on the Build server.
  • Annotate your test methods with @Test and name your methods that contain your tests with a starting "test", e.g.: @Test public void testGetParameter();
  • Do not use the regular assert keyword from the Java language to check conditions, but use the methods provided by JUnit: Assert.assertEquals(x, y); Assert.assertTrue(x); Assert.assertNull(x); Assert.assertNotNull(y);
  • Provide a short description in the assert*-statements of what is being checked or what failed. JUnit will display that info when an assertion failes. This helps to find problems faster. Example:
    Assert.assertEquals("different events files.", checksum1, checksum2);
    Assert.assertNotNull("did not find person 1.", population.getPersons().get(1));
  • Note that when you compare two double values, also provide the precision. Example:
    double d1, d2;
    Assert.assertEquals(d1, d2); // this matches to assertEquals(Object, Object); likely not what you wanted...
    Assert.assertEquals(d1, d2, MatsimTestUtils.EPSILON); // this is the right way to test for double values
  • If your test reads or writes data from/to files, use @Rule MatsimTestRule, see Mini-Introduction to JUnit 4.7.
    You have then the following methods at your convenience:
    • Config loadConfig(String filename);
      automatically sets the correct output-directory for your tests.
    • getOutputDirectory(); getInputDirectory(); getClassInputDirectory(); getPackageInputDirectory();
      return the corresponding filesystem paths to easily read and write test-data.
    • Reset the global state of MATSim (Gbl, MatsimRandomNumber, ...) so that your test is not influenced by other tests.
    • Provide the constant EPSILON for comparing double values (see above).
    • Cleans the output-directory before running the test, so you're test doesn't get mixed up by old data.
  • Load config files with MatsimTestUtils.loadConfig().
  • Write any output to the directory specified by MatsimTestUtils.getOutputDirectory().
  • Files containing test-data must be located in test/input/<package>/<class>/<test>/. Example:
    test/input/org/matsim/mymodule/MyModuleUtils/testUtilMethod/config.xml
    is a configuration file for the test org.matsim.mymodule.MyModuleUtils.testUtilMethod().
    If some files are used by more than one test, they can be located in the directory corresponding to the class or the package. Test files in your playgrounds are organized the exactly same way.
  • Do not use test-classes from other projects. While Eclipse may happily compile your code, Maven does not allow that test-classes are used outside their own project.

[deprecated] Mini-Introduction to JUnit 3.8

This information is out-dated. Newer tests should be written with JUnit 4.7.

JUnit is a framework for writing tests in Java. We currently use JUnit 4.7 for our tests, but many older tests are still written as JUnit 3.8. Thus, the following information is retained only for better understanding of existing tests.

JUnit 3.8 uses three main concepts:

  • Tests: the effective Tests, implemented as methods in a class. The method-name must start with "test".
  • TestCases: group several tests, implemented as a class containing severeal test-methods
  • TestSuites: group several testcases or other testsuites

A small example of a test and testcase:

 public class MyTests extends TestCase {
     protected void setUp() throws Exception {
         super.setUp();
         // your code here...
     }
     protected void tearDown() throws Exception {
         super.tearDown();
         // your code here...
     }
     public final void testOne() {
         // your code here...
         assertEquals(expected, actual);
     }
     public final void testTwo() {
         // your code here...
         assertNotNull(someObject);
     }
 }

This code defines two tests (testOnetestTwo). Additionally, there are two methods setUp() and tearDown(), which are automatically called by JUnit when executing the TestCase. Make sure to call Gbl.reset() in tearDown() when you use the class Gbl or Config within your tests, so that the following TestCases are not influenced by your TestCase.
JUnit offers many different assert-statements which should be used to verify the results of your tests. Do not use the standardassert() offered by Java, as this must especially be enabled to be executed!

When writing Tests for MATSim, do not extend the class TestCase, but extend MatsimTestCase, as that one offers some convenient methods related to MATSim (see Guidelines for more details).

Mini-Introduction to JUnit 4.7

JUnit is a framework for writing tests in Java. We currently use JUnit 4.7 for our tests, which still supports the older JUnit 3.8 syntax.

A simple example of a testcase:

 import org.junit.Assert;
 import org.junit.Test;

 public class MyTests {
     @Test
     public final void testOne() {
         // your code here...
         Assert.assertEquals(expectedValue, actualValue);
     }
     @Test
     public final void testTwo() {
         // your code here...
         Assert.assertNotNull(someObject);
     }
     @Test @Ignore("not yet fully implemented")
     public final void testThree() {
         // your code here...
         // TODO complete test
     }
 }

This code defines three tests (testOne, testTwo, testThree). JUnit offers many different assert-statements which should be used to verify the results of your tests. Do not use the standard assert() offered by Java, as this must especially be enabled to be evaluated! If a test is not yet fully implemented but you still want to commit the code, add the annotation @Ignore to the test. In that case, the test will not be executed. Note that your code must still compile when committing ignored tests.

If you need to read in or write out files to disk, use MatsimTestUtils as a rule:

 import org.junit.Assert;
 import org.junit.Rule;
 import org.junit.Test;

 public class MyTests {
     @Rule public MatsimTestUtils utils = new MatsimTestUtils();
     @Test
     public final void testOne() {
         String inputFile = utils.getInputDirectory() + "myFile.txt";
         // your code here...
         String outputFile = utils.getOutputDirectory() + "myFile.txt";
         Assert.assertEquals(expectedValue, actualValue);
     }
 }

Please note that many methods of MatsimTestUtils can only be used in the actual test (marked with @Test), but not in a constructor or other initialization methods (e.g. in @Before).