Guide to Dynamic Tests in Junit 5

1. Overview

Dynamic testing is a new programming model introduced in JUnit 5. In this article, we’ll have a look at what exactly dynamic tests are and how to create them.

If you’re completely new to JUnit 5, you might want to check the preview of JUnit 5 and our primary guide.

2. What Is a DynamicTest?

The standard tests annotated with @Test annotation are static tests which are fully specified at the compile time. A DynamicTest is a test generated during runtime. These tests are generated by a factory method annotated with the @TestFactory annotation.

A @TestFactory method must return a Stream, Collection, Iterable, or Iterator of DynamicTest instances. Returning anything else will result in a JUnitException since the invalid return types cannot be detected at compile time. Apart from this, a @TestFactory method cannot be static or private.

The DynamicTests are executed differently than the standard @Tests and do not support lifecycle callbacks. Meaning, the @BeforeEach and the @AfterEach methods will not be called for the DynamicTests.

3. Creating DynamicTests

First, let’s have a look at different ways of creating DynamicTests.

The examples here are not dynamic in nature, but they’ll provide a good starting point for creating truly dynamic ones.

We’re going to create a Collection of DynamicTest:

@TestFactory
Collection<DynamicTest> dynamicTestsWithCollection() {
    return Arrays.asList(
      DynamicTest.dynamicTest("Add test",
        () -> assertEquals(2, Math.addExact(1, 1))),
      DynamicTest.dynamicTest("Multiply Test",
        () -> assertEquals(4, Math.multiplyExact(2, 2))));
}

The @TestFactory method tells JUnit that this is a factory for creating dynamic tests. As we can see, we’re only returning a Collection of DynamicTest. Each of the DynamicTest consists of two parts, the name of the test or the display name, and an Executable.

The output will contain the display name that we passed to the dynamic tests:

Add test(dynamicTestsWithCollection())
Multiply Test(dynamicTestsWithCollection())

The same test can be modified to return an Iterable, Iterator, or a Stream:

@TestFactory
Iterable<DynamicTest> dynamicTestsWithIterable() {
    return Arrays.asList(
      DynamicTest.dynamicTest("Add test",
        () -> assertEquals(2, Math.addExact(1, 1))),
      DynamicTest.dynamicTest("Multiply Test",
        () -> assertEquals(4, Math.multiplyExact(2, 2))));
}

@TestFactory
Iterator<DynamicTest> dynamicTestsWithIterator() {
    return Arrays.asList(
      DynamicTest.dynamicTest("Add test",
        () -> assertEquals(2, Math.addExact(1, 1))),
      DynamicTest.dynamicTest("Multiply Test",
        () -> assertEquals(4, Math.multiplyExact(2, 2))))
        .iterator();
}

@TestFactory
Stream<DynamicTest> dynamicTestsFromIntStream() {
    return IntStream.iterate(0, n -> n + 2).limit(10)
      .mapToObj(n -> DynamicTest.dynamicTest("test" + n,
        () -> assertTrue(n % 2 == 0)));
}

Please note that if the @TestFactory returns a Stream, then it will be automatically closed once all the tests are executed.

The output will be pretty much the same as the first example. It will contain the display name that we pass to the dynamic test.

4. Creating a Stream of DynamicTests

For the demonstration purposes, consider a DomainNameResolver which returns an IP address when we pass the domain name as input.

For the sake of simplicity, let’s have a look at the high-level skeleton of our factory method:

@TestFactory
Stream<DynamicTest> dynamicTestsFromStream() {

    // sample input and output
    List<String> inputList = Arrays.asList(
      "www.somedomain.com", "www.anotherdomain.com", "www.yetanotherdomain.com");
    List<String> outputList = Arrays.asList(
      "154.174.10.56", "211.152.104.132", "178.144.120.156");

    // input generator that generates inputs using inputList
    /*...code here...*/

    // a display name generator that creates a 
    // different name based on the input
    /*...code here...*/

    // the test executor, which actually has the 
    // logic to execute the test case
    /*...code here...*/

    // combine everything and return a Stream of DynamicTest
    /*...code here...*/
}

There isn’t much code related to DynamicTest here apart from the @TestFactory annotation, which we’re already familiar with.

The two ArrayLists will be used as input to DomainNameResolver and expected output respectively.

Let’s now have a look at the input generator:

Iterator<String> inputGenerator = inputList.iterator();

The input generator is nothing but an Iterator of String. It uses our inputList and returns the domain name one by one.

The display name generator is fairly simple:

Function<String, String> displayNameGenerator 
  = (input) -> "Resolving: " + input;

The task of a display name generator is just to provide a display name for the test case that will be used in JUnit reports or the JUnit tab of our IDE.

Here we are just utilizing the domain name to generate unique names for each test. It’s not required to create unique names, but it will help in case of any failure. Having this, we’ll be able to tell the domain name for which the test case failed.

Now let’s have a look at the central part of our test – the test execution code:

DomainNameResolver resolver = new DomainNameResolver();
ThrowingConsumer<String> testExecutor = (input) -> {
    int id = inputList.indexOf(input);
 
    assertEquals(outputList.get(id), resolver.resolveDomain(input));
};

We have used the ThrowingConsumer, which is a @FunctionalInterface for writing the test case. For each input generated by the data generator, we’re fetching the expected output from the outputList and the actual output from an instance of DomainNameResolver.

Now the last part is simply to assemble all the pieces and return as a Stream of DynamicTest:

return DynamicTest.stream(
  inputGenerator, displayNameGenerator, testExecutor);

That’s it. Running the test will display the report containing the names defined by our display name generator:

Resolving: www.somedomain.com(dynamicTestsFromStream())
Resolving: www.anotherdomain.com(dynamicTestsFromStream())
Resolving: www.yetanotherdomain.com(dynamicTestsFromStream())

5. Improving the DynamicTest Using Java 8 Features

The test factory written in the previous section can be drastically improved by using the features of Java 8. The resultant code will be much cleaner and can be written in a lesser number of lines:

@TestFactory
Stream<DynamicTest> dynamicTestsFromStreamInJava8() {
        
    DomainNameResolver resolver = new DomainNameResolver();
        
    List<String> domainNames = Arrays.asList(
      "www.somedomain.com", "www.anotherdomain.com", "www.yetanotherdomain.com");
    List<String> outputList = Arrays.asList(
      "154.174.10.56", "211.152.104.132", "178.144.120.156");
        
    return inputList.stream()
      .map(dom -> DynamicTest.dynamicTest("Resolving: " + dom, 
        () -> {int id = inputList.indexOf(dom);
 
      assertEquals(outputList.get(id), resolver.resolveDomain(dom));
    }));       
}

The above code has the same effect as the one we saw in the previous section. The inputList.stream().map() provides the stream of inputs (input generator). The first argument to dynamicTest() is our display name generator (“Resolving: ” + dom) while the second argument, a lambda, is our test executor.

The output will be the same as the one from the previous section.

6. Additional Example

In this example, we’re further exploring the power of the dynamic tests to filter the inputs based on the test cases:

@TestFactory
Stream<DynamicTest> dynamicTestsForEmployeeWorkflows() {
    List<Employee> inputList = Arrays.asList(
      new Employee(1, "Fred"), new Employee(2), new Employee(3, "John"));
        
    EmployeeDao dao = new EmployeeDao();
    Stream<DynamicTest> saveEmployeeStream = inputList.stream()
      .map(emp -> DynamicTest.dynamicTest(
        "saveEmployee: " + emp.toString(), 
          () -> {
              Employee returned = dao.save(emp.getId());
              assertEquals(returned.getId(), emp.getId());
          }
    ));
        
    Stream<DynamicTest> saveEmployeeWithFirstNameStream 
      = inputList.stream()
      .filter(emp -> !emp.getFirstName().isEmpty())
      .map(emp -> DynamicTest.dynamicTest(
        "saveEmployeeWithName" + emp.toString(), 
        () -> {
            Employee returned = dao.save(emp.getId(), emp.getFirstName());
            assertEquals(returned.getId(), emp.getId());
            assertEquals(returned.getFirstName(), emp.getFirstName());
        }));
        
    return Stream.concat(saveEmployeeStream, 
      saveEmployeeWithFirstNameStream);
}

The save(Long) method needs only the employeeId. Hence, it utilizes all the Employee instances. The save(Long, String) method needs firstName apart from the employeeId. Hence, it filters out the Employee instances without firstName.

Finally, we combine both the streams and return all the tests as a single Stream.

Now, let’s have a look at the output:

saveEmployee: Employee 
  [id=1, firstName=Fred](dynamicTestsForEmployeeWorkflows())
saveEmployee: Employee 
  [id=2, firstName=](dynamicTestsForEmployeeWorkflows())
saveEmployee: Employee 
  [id=3, firstName=John](dynamicTestsForEmployeeWorkflows())
saveEmployeeWithNameEmployee 
  [id=1, firstName=Fred](dynamicTestsForEmployeeWorkflows())
saveEmployeeWithNameEmployee 
  [id=3, firstName=John](dynamicTestsForEmployeeWorkflows())

7. Conclusion

The parameterized tests can replace many of the examples in this article. However, the dynamic tests differ from the parameterized tests as they do not support full test lifecycle, while parametrized tests do.

Moreover, dynamic tests provide more flexibility regarding how the input is generated and how the tests are executed.

JUnit 5 prefers extensions over features principle. As a result, the main aim of dynamic tests is to provide an extension point for third party frameworks or extensions.

You can read more about other features of JUnit 5 in our article on repeated tests in JUnit 5.