Why does JUnit only report the first failure in a single test?
Why does JUnit only report the first failure in a single test?
(Submitted by: J. B. Rainsberger)
Reporting multiple failures in a single test is generally a sign that the test does too much, compared to what a unit test ought to do. Usually this means either that the test is really a functional/acceptance/customer test or, if it is a unit test, then it is too big a unit test.
JUnit is designed to work best with a number of small tests. It executes each test within a separate instance of the test class. It reports failure on each test. Shared setup code is most natural when sharing between tests. This is a design decision that permeates JUnit, and when you decide to report multiple failures per test, you begin to fight against JUnit. This is not recommended.
Long tests are a design smell and indicate the likelihood of a design problem. Kent Beck is fond of saying in this case that "there is an opportunity to learn something about your design." We would like to see a pattern language develop around these problems, but it has not yet been written down.
Finally, note that a single test with multiple assertions is isomorphic to a test case with multiple tests:
One test method, three assertions:
public class MyTestCase {
@Test
public void testSomething() {
// Set up for the test, manipulating local variables
assertTrue(condition1);
assertTrue(condition2);
assertTrue(condition3);
}
}
Three test methods, one assertion each:
public class MyTestCase {
// Local variables become instance variables
@Before
public void setUp() {
// Set up for the test, manipulating instance variables
}
@Test
public void testCondition1() {
assertTrue(condition1);
}
@Test
public void testCondition2() {
assertTrue(condition2);
}
@Test
public void testCondition3() {
assertTrue(condition3);
}
}
The resulting tests use JUnit's natural execution and reporting mechanism and, failure in one test does not affect the execution of the other tests. You generally want exactly one test to fail for any given bug, if you can manage it.
