A Quick JUnit vs TestNG Comparison

1. Overview

JUnit and TestNG are undoubtedly the two most popular unit-testing frameworks in the Java ecosystem. While JUnit inspires TestNG itself, it provides its distinctive features, and unlike JUnit, it works for functional and higher levels of testing.

In this post, we’ll discuss and compare these frameworks by covering their features and common use cases.

2. Test Setup

While writing test cases, often we need to execute some configuration or initialization instructions before test executions, and also some cleanup after completion of tests. Let’s evaluate these in both frameworks.

JUnit offers initialization and cleanup at two levels, before and after each method and class. We have @BeforeEach, @AfterEach annotations at method level and @BeforeAll and @AfterAll at class level:

class SummationServiceTest {

    private static List<Integer> numbers;

    @BeforeAll
    static void initialize() {
        numbers = new ArrayList<>();
    }

    @AfterAll
    static void tearDown() {
        numbers = null;
    }

    @BeforeEach
    void runBeforeEachTest() {
        numbers.add(1);
        numbers.add(2);
        numbers.add(3);
    }

    @AfterEach
    void runAfterEachTest() {
        numbers.clear();
    }

    @Test
    void givenNumbers_sumEquals_thenCorrect() {
        int sum = numbers.stream().reduce(0, Integer::sum);
        assertEquals(6, sum);
    }
}

Note that this example uses JUnit 5. In the previous JUnit 4 version, we would need to use the @Before and @After annotations which are equivalent to @BeforeEach and @AfterEach. Likewise, @BeforeAll and @AfterAll are replacements for JUnit 4’s @BeforeClass and @AfterClass.

Similar to JUnit, TestNG also provides initialization and cleanup at the method and class level. While @BeforeClass and @AfterClass remain the same at the class level, the method level annotations are @BeforeMethod and @AfterMethod:

@BeforeClass
public void initialize() {
    numbers = new ArrayList<>();
}

@AfterClass
public void tearDown() {
    numbers = null;
}

@BeforeMethod
public void runBeforeEachTest() {
    numbers.add(1);
    numbers.add(2);
    numbers.add(3);
}

@AfterMethod
public void runAfterEachTest() {
    numbers.clear();
}

TestNG also offers, @BeforeSuite, @AfterSuite, @BeforeGroup and @AfterGroup annotations, for configurations at suite and group levels:

@BeforeGroups("positive_tests")
public void runBeforeEachGroup() {
    numbers.add(1);
    numbers.add(2);
    numbers.add(3);
}

@AfterGroups("negative_tests")
public void runAfterEachGroup() {
    numbers.clear(); 
}

Also, we can use the @BeforeTest and @AfterTest if we need any configuration before or after test cases included in the <test> tag in TestNG XML configuration file:

<test name="test setup">
    <classes>
        <class name="SummationServiceTest">
            <methods>
                <include name="givenNumbers_sumEquals_thenCorrect" />
            </methods>
        </class>
    </classes>
</test>

Note that the declaration of @BeforeClass and @AfterClass method has to be static in JUnit. By comparison, TestNG method declaration doesn’t have these constraints.

3. Ignoring Tests

Both frameworks support ignoring test cases, though they do it quite differently. JUnit5 offers the @Disabled annotation:

@Disabled
@Test
void givenEmptyList_sumEqualsZero_thenCorrect() {
    int sum = numbers.stream().reduce(0, Integer::sum);
    Assert.assertEquals(6, sum);
}

With JUnit4, we used the @Ignore annotation:

@Ignore
@Test
public void givenNumbers_sumEquals_thenCorrect() {
    int sum = numbers.stream().reduce(0, Integer::sum);
    Assert.assertEquals(6, sum);
}

On the other hand, TestNG uses @Test with a parameter “enabled” with a boolean value true or false:

@Test(enabled=false)
public void givenNumbers_sumEquals_thenCorrect() {
    int sum = numbers.stream.reduce(0, Integer::sum);
    Assert.assertEquals(6, sum);
}

4. Running Tests Together

Running tests together as a collection is possible in both JUnit and TestNG, but they do it in different ways.

We can use the @Suite, @SelectPackages, and @SelectClasses annotations to group test cases and run them as a suite in JUnit 5. A suite is a collection of test cases that we can group together and run as a single test.

If we want to group test cases of different packages to run together within a Suite we need the @SelectPackages annotation:

@Suite
@SelectPackages({ "org.baeldung.java.suite.childpackage1", "org.baeldung.java.suite.childpackage2" })
class SelectPackagesSuiteUnitTest {

}

If we want specific test classes to run together, JUnit 5 provides the flexibility through @SelectClasses:

@Suite
@SelectClasses({Class1UnitTest.class, Class2UnitTest.class})
class SelectClassesSuiteUnitTest {

}

Previously using JUnit 4, we achieved grouping and running multiple tests together using @RunWith and @Suite annotations:

@RunWith(Suite.class)
@Suite.SuiteClasses({ RegistrationTest.class, SignInTest.class })
public class SuiteTest {

}

In TestNG we can group tests by using an XML file:

<suite name="suite">
    <test name="test suite">
        <classes>
            <class name="com.baeldung.RegistrationTest" />
            <class name="com.baeldung.SignInTest" />
        </classes>
    </test>
</suite>

This indicates RegistrationTest and SignInTest will run together.

Apart from grouping classes, TestNG can group methods as well using the @Test(groups=”groupName”) annotation:

@Test(groups = "regression")
public void givenNegativeNumber_sumLessthanZero_thenCorrect() {
    int sum = numbers.stream().reduce(0, Integer::sum);
    Assert.assertTrue(sum < 0);
}

Let’s use an XML to execute the groups:

<test name="test groups">
    <groups>
        <run>
            <include name="regression" />
        </run>
    </groups>
    <classes>
        <class 
          name="com.baeldung.SummationServiceTest" />
    </classes>
</test>

This will execute the test method tagged with group regression.

5. Testing Exceptions

The feature for testing for exceptions using annotations is available in both JUnit and TestNG.

Let’s first create a class with a method that throws an exception:

public class Calculator {
    public double divide(double a, double b) {
        if (b == 0) {
            throw new DivideByZeroException("Divider cannot be equal to zero!");
        }
        return a/b;
    }
}

In JUnit 5 we can use the assertThrows API to test exceptions:

@Test
void whenDividerIsZero_thenDivideByZeroExceptionIsThrown() {
    Calculator calculator = new Calculator();
    assertThrows(DivideByZeroException.class, () -> calculator.divide(10, 0));
}

In JUnit 4, we can achieve this by using @Test(expected = DivideByZeroException.class) over the test API.

And with TestNG we can also implement the same:

@Test(expectedExceptions = ArithmeticException.class) 
public void givenNumber_whenThrowsException_thenCorrect() { 
    int i = 1 / 0;
}

This feature implies what exception is thrown from a piece of code, that’s part of a test.

6. Parameterized Tests

Parameterized unit tests are helpful for testing the same code under several conditions. With the help of parameterized unit tests, we can set up a test method that obtains data from some data source. The main idea is to make the unit test method reusable and to test with a different set of inputs.

In JUnit 5, we have the advantage of test methods consuming data arguments directly from the configured source. By default, JUnit 5 provides a few source annotations like:

• @ValueSource: we can use this with an array of values of type Short, Byte, Int, Long, Float, Double, Char, and String:

@ParameterizedTest
@ValueSource(strings = { "Hello", "World" })
void givenString_TestNullOrNot(String word) {
    assertNotNull(word);
}

• @EnumSource – passes Enum constants as parameters to the test method:

@ParameterizedTest
@EnumSource(value = PizzaDeliveryStrategy.class, names = {"EXPRESS", "NORMAL"})
void givenEnum_TestContainsOrNot(PizzaDeliveryStrategy timeUnit) {
    assertTrue(EnumSet.of(PizzaDeliveryStrategy.EXPRESS, PizzaDeliveryStrategy.NORMAL).contains(timeUnit));
}

• @MethodSource – passes external methods generating streams:

static Stream<String> wordDataProvider() {
    return Stream.of("foo", "bar");
}

@ParameterizedTest
@MethodSource("wordDataProvider")
void givenMethodSource_TestInputStream(String argument) {
    assertNotNull(argument);
}

• @CsvSource – uses CSV values as a source for the parameters:

@ParameterizedTest
@CsvSource({ "1, Car", "2, House", "3, Train" })
void givenCSVSource_TestContent(int id, String word) {
	assertNotNull(id);
	assertNotNull(word);
}

Similarly, we have other sources like @CsvFileSource if we need to read a CSV file from classpath and @ArgumentSource to specify a custom, reusable ArgumentsProvider.

In JUnit 4, the test class has to be annotated with @RunWith to make it a parameterized class and @Parameter to use the denote the parameter values for unit test.

In TestNG, we can parametrize tests using @Parameter or @DataProvider annotations. While using the XML file annotate the test method with @Parameter:

@Test
@Parameters({"value", "isEven"})
public void 
  givenNumberFromXML_ifEvenCheckOK_thenCorrect(int value, boolean isEven) {
    Assert.assertEquals(isEven, value % 2 == 0);
}

and provide the data in the XML file:

<suite name="My test suite">
    <test name="numbersXML">
        <parameter name="value" value="1"/>
        <parameter name="isEven" value="false"/>
        <classes>
            <class name="baeldung.com.ParametrizedTests"/>
        </classes>
    </test>
</suite>

While using information in the XML file is simple and useful, in some cases, you might need to provide more complex data.

For this, we can use the @DataProvider annotation which allows us to map complex parameter types for testing methods.

Here’s an example of using @DataProvider for primitive data types:

@DataProvider(name = "numbers")
public static Object[][] evenNumbers() {
    return new Object[][]{{1, false}, {2, true}, {4, true}};
}

@Test(dataProvider = "numbers")
public void givenNumberFromDataProvider_ifEvenCheckOK_thenCorrect
  (Integer number, boolean expected) {
    Assert.assertEquals(expected, number % 2 == 0);
}

And @DataProvider for objects:

@Test(dataProvider = "numbersObject")
public void givenNumberObjectFromDataProvider_ifEvenCheckOK_thenCorrect
  (EvenNumber number) {
    Assert.assertEquals(number.isEven(), number.getValue() % 2 == 0);
}

@DataProvider(name = "numbersObject")
public Object[][] parameterProvider() {
    return new Object[][]{{new EvenNumber(1, false)},
      {new EvenNumber(2, true)}, {new EvenNumber(4, true)}};
}

In the same way, any particular objects that are to be tested can be created and returned using data provider. It’s useful when integrating with frameworks like Spring.

Notice that, in TestNG, since @DataProvider method need not be static, we can use multiple data provider methods in the same test class.

7. Test Timeout

Timed out tests means, a test case should fail if the execution is not completed within a certain specified period. Both JUnit and TestNG support timed out tests. In JUnit 5 we can write a timeout test as:

@Test
void givenExecution_takeMoreTime_thenFail() throws InterruptedException {
    Assertions.assertTimeout(Duration.ofMillis(1000), () -> Thread.sleep(10000));
}

In JUnit 4 and TestNG we can the same test using @Test(timeout=1000)

@Test(timeOut = 1000)
public void givenExecution_takeMoreTime_thenFail() {
    while (true);
}

8. Dependent Tests

TestNG supports dependency testing. This means in a set of test methods, if the initial test fails, then all subsequent dependent tests will be skipped, not marked as failed as in the case for JUnit.

Let’s have a look at a scenario, where we need to validate email, and if it’s successful, will proceed to log in:

@Test
public void givenEmail_ifValid_thenTrue() {
    boolean valid = email.contains("@");
    Assert.assertEquals(valid, true);
}

@Test(dependsOnMethods = {"givenEmail_ifValid_thenTrue"})
public void givenValidEmail_whenLoggedIn_thenTrue() {
    LOGGER.info("Email {} valid >> logging in", email);
}

9. Order of Test Execution

There is no defined implicit order in which test methods will get executed in JUnit or TestNG. The methods are just invoked as returned by the Java Reflection API. Since JUnit 4 it uses a more deterministic but not predictable order.

To have more control, we will annotate the test class with @TestMethodOrder annotation and mention a method orderer:

@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
class SortedUnitTest {
    
    @Test
    @Order(2)
    void a_givenString_whenChangedtoInt_thenTrue() {
        assertTrue(Integer.valueOf("10") instanceof Integer);
    }

    @Test
    @Order(1)
    void b_givenInt_whenChangedtoString_thenTrue() {
        assertTrue(String.valueOf(10) instanceof String);
    }

}

The MethodOrderer.OrderAnnotation.class parameter enables the use of the @Order annotation. Then, JUnit 5 will run the tests from the lowest to the highest order value. In JUnit4, there was nothing so easy. However, we could run tests in lexicographic order thanks to the @FixMethodOrder(MethodSorters.NAME_ASCENDING) annotation:

@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class SortedUnitTest {

    @Test
    public void a_givenString_whenChangedtoInt_thenTrue() {
        assertTrue(Integer.valueOf("10") instanceof Integer);
    }

    @Test
    public void b_givenInt_whenChangedtoString_thenTrue() {
        assertTrue(String.valueOf(10) instanceof String);
    }

}

While TestNG also provides a couple of ways to have control in the order of test method execution. We provide the priority parameter in the @Test annotation:

@Test(priority = 1)
public void givenString_whenChangedToInt_thenCorrect() {
    Assert.assertTrue(
      Integer.valueOf("10") instanceof Integer);
}

@Test(priority = 2)
public void givenInt_whenChangedToString_thenCorrect() {
    Assert.assertTrue(
      String.valueOf(23) instanceof String);
}

Notice, that priority invokes test methods based on priority but does not guarantee that tests in one level are completed before invoking the next priority level.

Sometimes while writing functional test cases in TestNG, we might have an interdependent test where the order of execution must be the same for every test run. To achieve that we should use the dependsOnMethods parameter to @Test annotation as we saw in the earlier section.

10. Custom Test Name

By default, whenever we run a test, the test class and the test method name is printed in console or IDE. JUnit 5 provides a unique feature where we can mention custom descriptive names for class and test methods using @DisplayName annotation.

This annotation doesn’t provide any testing benefits but it brings easy to read and understand test results for a non-technical person too:

@ParameterizedTest
@ValueSource(strings = { "Hello", "World" })
@DisplayName("Test Method to check that the inputs are not nullable")
void givenString_TestNullOrNot(String word) {
    assertNotNull(word);
}

Whenever we run the test, the output will show the display name instead of the method name.

Right now, in TestNG there is no way to provide a custom name.

11. Conclusion

Both JUnit and TestNG are modern tools for testing in the Java ecosystem.

In this article, we had a quick look at various ways of writing tests with each of these two test frameworks.