The Order of Tests in JUnit

1. Overview

By default, JUnit runs tests using a deterministic but unpredictable order (MethodSorters.DEFAULT).

In most cases, that behavior is perfectly fine and acceptable. But there are cases when we need to enforce a specific order.

2. Test Methods Ordering in JUnit 5

In JUnit 5, we can use @TestMethodOrder to control the execution order of tests.

We can use our own MethodOrderer, as we’ll see later.

Or we can select one of three built-in orderers:

1. Alphanumeric Order
2. @Order Annotation
3. Random Order

2.1. Using Alphanumeric Order

JUnit 5 comes with a set of built-in MethodOrderer implementations to run tests in alphanumeric order.

For example, it provides MethodOrderer.MethodName to sort test methods based on their names and their formal parameter lists:

@TestMethodOrder(MethodOrderer.MethodName.class)
public class AlphanumericOrderUnitTest {
    private static StringBuilder output = new StringBuilder("");
    
    @Test
    void myATest() {
        output.append("A");
    }
    
    @Test
    void myBTest() {
        output.append("B");        
    }
    
    @Test
    void myaTest() {
        output.append("a");
    }
 
    @AfterAll
    public static void assertOutput() {
        assertEquals("ABa", output.toString());
    }
}

Similarly, we can use MethodOrderer.DisplayName to sort methods alphanumerically based on their display names.

Please keep in mind that MethodOrderer.Alphanumeric is another alternative. However, this implementation is deprecated and will be removed in 6.0.

2.2. Using the @Order Annotation

We can use the @Order annotation to enforce tests to run in a specific order.

In the following example, the methods will run firstTest(), then secondTest() and finally thirdTest():

@TestMethodOrder(OrderAnnotation.class)
public class OrderAnnotationUnitTest {
    private static StringBuilder output = new StringBuilder("");
    
    @Test
    @Order(1)    
    void firstTest() {
        output.append("a");
    }
    
    @Test
    @Order(2)    
    void secondTest() {
        output.append("b");
    }
 
    @Test
    @Order(3)    
    void thirdTest() {
        output.append("c");
    }
 
    @AfterAll
    public static void assertOutput() {
        assertEquals("abc", output.toString());
    }
}

2.3. Using Random Order

We can also order test methods pseudo-randomly using the MethodOrderer.Random implementation:

@TestMethodOrder(MethodOrderer.Random.class)
public class RandomOrderUnitTest {

    private static StringBuilder output = new StringBuilder("");

    @Test
    void myATest() {
        output.append("A");
    }

    @Test
    void myBTest() {
        output.append("B");
    }

    @Test
    void myCTest() {
        output.append("C");
    }

    @AfterAll
    public static void assertOutput() {
        assertEquals("ACB", output.toString());
    }

}

As a matter of fact, JUnit 5 uses System.nanoTime() as the default seed to sort the test methods. This means that the execution order of the methods may not be the same in repeatable tests.

However, we can configure a custom seed using the junit.jupiter.execution.order.random.seed property to create repeatable builds.

We can specify the value of our custom seed in the junit-platform.properties file:

junit.jupiter.execution.order.random.seed=100

2.4. Using a Custom Order

Finally, we can use our custom order by implementing the MethodOrderer interface.

In our CustomOrder, we’ll order the tests based on their names in a case-insensitive alphanumeric order:

public class CustomOrder implements MethodOrderer {
    @Override
    public void orderMethods(MethodOrdererContext context) {
        context.getMethodDescriptors().sort(
         (MethodDescriptor m1, MethodDescriptor m2)->
           m1.getMethod().getName().compareToIgnoreCase(m2.getMethod().getName()));
    }
}

Then we’ll use CustomOrder to run the same tests from our previous example in the order myATest(), myaTest(), and finally myBTest():

@TestMethodOrder(CustomOrder.class)
public class CustomOrderUnitTest {

    // ...
 
    @AfterAll
    public static void assertOutput() {
        assertEquals("AaB", output.toString());
    }
}

2.5. Set Default Order

JUnit 5 provides a convenient way to set a default method orderer through the junit.jupiter.testmethod.order.default parameter.

Similarly, we can configure our parameter in the junit-platform.properties file:

junit.jupiter.testmethod.order.default = org.junit.jupiter.api.MethodOrderer$DisplayName

The default orderer will be applied to all tests that aren’t qualified with @TestMethodOrder.

Another important thing to mention is that the specified class must implement the MethodOrderer interface.

3. Test Methods Ordering in JUnit 4

For those still using JUnit 4, the APIs for ordering tests are slightly different.

Let’s go through the options to achieve this in previous versions as well.

3.1. Using MethodSorters.DEFAULT

This default strategy compares test methods using their hash codes.

In case of a hash collision, the lexicographical order is used:

@FixMethodOrder(MethodSorters.DEFAULT)
public class DefaultOrderOfExecutionTest {
    private static StringBuilder output = new StringBuilder("");

    @Test
    public void secondTest() {
        output.append("b");
    }

    @Test
    public void thirdTest() {
        output.append("c");
    }

    @Test
    public void firstTest() {
        output.append("a");
    }

    @AfterClass
    public static void assertOutput() {
        assertEquals(output.toString(), "cab");
    }
}

When we run the tests in the class above, we will see that they all pass, including assertOutput().

3.2. Using MethodSorters.JVM

Another ordering strategy is MethodSorters.JVM.

This strategy utilizes the natural JVM ordering, which can be different for each run:

@FixMethodOrder(MethodSorters.JVM)
public class JVMOrderOfExecutionTest {    
    // same as above
}

Each time we run the tests in this class, we get a different result.

3.3. Using MethodSorters.NAME_ASCENDING

Finally, this strategy can be used for running tests in their lexicographic order:

@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class NameAscendingOrderOfExecutionTest {
    // same as above
    
    @AfterClass
    public static void assertOutput() {
        assertEquals(output.toString(), "abc");
    }
}

When we run the tests in this class, we see they all pass, including assertOutput(). This confirms the execution order that we set with the annotation.

4. Test Classes Ordering Using @TestClassOrder

We can also control the execution order of test classes using the @TestClassOrder.

Junit 5 comes with a ClassOrderer interface, similar to MethodOrderer. We can use it in the same manner as the method ordering in the previous sections. ClassOrderer supports the following ways to order tests:

• ClassName – sorts classes alphanumerically, based on their names
• DisplayName – sorts classes alphanumerically based on their display names
  
• OrderAnnotation – sorts classes based on the @Order annotation
• Random – sorts classes randomly
• Custom Order – sorts classes based on the custom sorting order

4.1. Using ClassName

It enables the execution of classes in alphabetical order according to their fully qualified class names. This is the default ordering mechanism based on the actual class name in the code.

Let’s define some test classes that we’ll use to run different tests:

public class TestA {
    @Test
    void testA() {
        System.out.println("Running TestA");
    }
}

public class TestB {
    @Test
    void testB() {
        System.out.println("Running TestB");
    }
}

public class TestC {
    @Test
    void testC() {
        System.out.println("Running TestC");
    }
}

To use @TestClassOrder, the classes need to be part of the same hierarchy. For that, by using @Nested and inheritance, we create a flexible structure that allows us to group the tests logically within a suite:

@TestClassOrder(ClassOrderer.ClassName.class)
public class ClassNameOrderUnitTest {
    @Nested
    class C extends TestC {}

    @Nested
    class B extends TestB {}

    @Nested
    class A extends TestA {}
}

In the above example, we created a parent test suite, ClassNameOrderUnitTest, which acts as a container for the test classes.

The @TestClassOrder(ClassOrderer.ClassName.class) allows us to enforce alphabetical order. After running the program, the output will be:

Running TestA
Running TestB
Running TestC

4.2. Using DisplayName

The ClassOrderer.DisplayName orders classes by their display names if explicitly set. If we haven’t set any display name, JUnit falls back to the class name.

In the below example, we’ll set the @DisplayName annotations on the nested wrapper classes and not on the original test class TestA, TestB, and TestC:

@TestClassOrder(ClassOrderer.DisplayName.class)
public class DisplayNameOrderUnitTest {
    @Nested
    @DisplayName("Class C")
    class Z extends TestC {}

    @Nested
    @DisplayName("Class B")
    class A extends TestA {}

    @Nested
    @DisplayName("Class A")
    class B extends TestB {}
}

For the above program, JUnit will sort these display names alphabetically. The sorted order of display names will be “Class A”, “Class B”, and “Class C”, which leads us to the below output:

Running TestB
Running TestA
Running TestC

4.3. Using OrderAnnotation

The @Order annotation allows us to explicitly define the order in which test classes are executed within a test suite. It’s useful when we want to have some explicit control over execution order or need to run tests in a specific sequence.

In the below example, the @Order annotation specifies the priority for the nested classes:

@TestClassOrder(ClassOrderer.OrderAnnotation.class)
public class OrderAnnotationUnitTest {
    @Nested
    @Order(3)
    class A extends TestA {}

    @Nested
    @Order(1)
    class B extends TestB {}

    @Nested
    @Order(2)
    class C extends TestC {}
}

The classes with lower @Order values will be executed first. Hence, we get the below output:

Running TestB
Running TestC
Running TestA

4.4. Random

Sometimes, we want to run tests in random order to detect dependencies between tests. This allows us to ensure that our tests don’t rely on the order of execution.

ClassOrderer.Random.class allows us to configure the suite to randomize the execution of test classes. This randomness will be applied each time the test suite run:

@TestClassOrder(ClassOrderer.Random.class)
public class RandomOrderUnitTest {
    @Nested
    class C extends TestC {}

    @Nested
    class B extends TestB {}

    @Nested
    class A extends TestA {}
}

When the above test is is executed, the output will be in a random order, varying with each run.

Run 1 output:

Running TestA
Running TestC
Running TestB

Run 2 output:

Running TestC
Running TestA
Running TestB

4.5. Using a Custom Order

We can implement our own ClassOrderer to define ordering logic based on criteria such as metadata, class name length, configuration, etc. The ClassOrderer interface allows us to define custom logic to sort test classes based on a specific criteria.

Let’s implement a custom ClassOrderer that orders test classes based on the length of their class names:

public class CustomClassOrderer implements ClassOrderer {
    @Override
    public void orderClasses(ClassOrdererContext context) {
        context.getClassDescriptors().sort(
            Comparator.comparingInt(descriptor ->
                descriptor.getTestClass().getSimpleName().length()
            )
        );
    }
}

Let’s use CustomClassOrderer to sort test classes based on the length of their class names. The sorting order will be shortest to longest:

@TestClassOrder(CustomClassOrderer.class)
public class CustomOrderUnitTest {
    @Nested
    class Longest extends TestA {}

    @Nested
    class Middle extends TestB {}

    @Nested
    class Short extends TestC {}
}

The output is ordered by the length of the class names i.e. Longest, Middle and Short:

Running TestC
Running TestB
Running TestA

Since Short has the shortest name among these classes, the test runner executes it first, resulting in the output “Running TestC” appearing first. Subsequently, the test runner executes the Middle and Longest class in order.

5. Conclusion

In this quick article, we went through the ways of setting the execution order available in JUnit.