Getting @BeforeAll and @AfterAll behaviour with Cucumber JVM and JUnit

Posted by Johannes F. Knauf on March 10, 2020

tl;dr

Cucumber JVM does not feature a @BeforeAll or @AfterAll lifecycle hook to run setup and teardown procedures. This article shows how to achieve global setup and teardown with

All variants are demonstrated by working code examples on github.

BDD connects tech and business

At metamorphant we often face the challenge of bringing tech and business closer together. One method from our toolbox are BDD-style feature tests. A popular approach is to phrase tests in Gherkin – a human-readable, domain-specific language for scenario testing. Users of Java or other JVM languages will typically choose either JBehave or Cucumber for that purpose.

Cucumber currently has no @BeforeAll or @AfterAll

When testing complex scenarios, sometimes you need to setup an expensive resource before starting the first test and tear it down after the last test.

Example: Test Database

One of our customers had a dependency on a MongoDB NoSQL Database. For tests they wanted to adopt flapdoodle embedded MongoDB. In general, non-Java DBs will run as a separate OS process. This rule equally applies when packaging the component in a Docker container. Startup and teardown take at least some seconds. So, you do not want to afford it for each test. Sharing a DB instance for multiple tests is feasible, if you have a way of cleanly resetting the DB before the individual tests.

Train yourself a hamster

For the purpose of this article I will switch to a simpler scenario. My little test scenario revolves around hamster training (thanks for the inspiration, Michael Keeling). It features the bare minimum of a BDD test.

Feature: Hamster training

  Scenario: Hamster repeats trick immediately after reward
    Given a trained hamster
    When I make the hamster jump through a burning loop
    And I reward him with a honey cracker
    Then the hamster will be happy
    And the hamster will be ready to just do it again

  Scenario: Missing reward disappoints the hamster
    Given a trained hamster
    When I make the hamster jump through a burning loop
    And I do not react
    Then the hamster will be sad
    And the hamster will decline another jump

Like setting up an embedded DB, the training of a hamster takes a long time and is expensive in terms of resources. I will simulate that in the hamster code by a simple Thread.sleep(5000).

No standard solution in Cucumber JVM

It turns out, that this kind of setup/teardown scenario is currenty not well supported by Cucumber JVM. Cucumber JVM brings its own JUnit Runner and controls its internal lifecycle independently. Users define the details like glue code Java packages and feature file paths by annotation of the test class. It is not unusual to end up with an empty test class:

...

@RunWith(Cucumber.class)
@CucumberOptions(
    strict = true,
    features = "src/test/features",
    glue = "de.metamorphant.blog.my.steps"
)
public class MyTest {}

The execution is controlled mostly by the step implementations. There is a fixed set of supported steps:

  • @Given, @When, @Then steps are executed when used in a scenario
  • @Before is executed before a scenario
  • @After is executed before a scenario

There is no equivalent of @BeforeAll and @AfterAll. No step runs before the whole feature or even before the whole test suite. The corresponding github issue 515 about adding @BeforeAll and @AfterAll hooks is highly popular. It obviously struck a nerve. But, despite the high interest, it is unclear when – if ever – the Cucumber upstream project will release such new features.

Hence, my customer asked for help in implementing a workaround with current Cucumber versions. Issue 515 already describes some workarounds but lacks complete, workable examples.

Other frameworks’ solutions

How is the same problem solved in other test frameworks?

I will compare the solutions of the most common Java test toolkits:

I do not have a clue about other frameworks like Concordion or FitNesse. For a library like jgiven the answer is simple: it piggybacks on other test frameworks like JUnit 4 as an embedded fluid DSL and inherits their test structure and lifecycle.

Standard solution in plain JUnit

Plain JUnit 4 structures tests by methods and classes. Logically, the hook annotations are

  • @BeforeClass and @AfterClass for wrapping a test class’s lifecycle and
  • @Before and @After for wrapping each test method.

Resp. for JUnit 5 the annotations have been renamed to

  • @BeforeAll and @AfterAll for test classes and
  • @BeforeEach and @AfterEach for test methods.

I will use JUnit annotations for one of our proposed workarounds later.

Standard solution in Spock

The awesome Spock Framework follows a similar approach as JUnit and uses magic method names

  • setupSpec() and cleanupSpec() for test classes and
  • setup() and cleanup() for test methods.

As a special gimmick you can extend base test classes and chain setup and cleanup methods superclass-first.

Standard solution in JBehave

JBehave uses a slightly different lingo than cucumber to structure the test hierarchy:

  • A JBehave scenario comprises multiple JBehave steps. It is equivalent to a Cucumber scenario.
  • A JBehave story comprises multiple JBehave scenarios. It is equivalent to a Cucumber feature.
  • Multiple JBehave stories can be combined to a collection of stories.

JBehave’s lifecycle concept provides detailed hook-in points to attach logic to each element. You can either call steps from your story files using a special syntax or programmatically using JBehave’s annotations.

  • @BeforeScenario and @AfterScenario wrap a scenario.
  • @BeforeStory and @AfterStory wrap a single story
  • @BeforeStories and @AfterStories wrap a collection of stories.

Looking for workarounds in Cucumber

I will demonstrate the workarounds using the hamster scenarios described above. The hamster implementation in class de.metamorphant.blog.hamster.Hamster is the same for all 4 examples.

package de.metamorphant.blog.hamster;

public class Hamster {
    String port;
    Boolean happy;
    
    private Hamster(String port) {
        this.port = port;
        this.happy = true;
    }
    
    public void makeJump() {
        System.out.println("Trying to make the hamster jump through a burning loop");
        if (this.happy == true) {
            this.say("And it burns, burns, burns, The ring of fire!");
        } else {
            this.say("Go jump yourself");
            throw new NotInTheMoodException("I am mad at you!");
        }
    }
    
    public void reward() {
        System.out.println("Rewarding the hamster with a honey cracker");
        this.say("Yay! This honey cracker is yummy!");
        this.happy = true;
    }
    
    public void ignore() {
        System.out.println("Ignoring the hamster");
        this.say("You make me sad.");
        this.happy = false;
    }
    
    public Boolean isHappy() {
        return this.happy;
    }
    
    public Boolean isSad() {
        return !this.isHappy();
    }
    
    private void say(String message) {
        String output = String.format("Hamster on port %s: %s", this.port, message);
        System.out.println(output);
    }
    
    public static Hamster trainedHamster(String port) {
        System.out.println("Providing a trained hamster");
        return new Hamster(port);
    }
}

All examples except the maven failsafe approach contain a class de.metamorphant.blog.hamster.HamsterUtil in addition. It provides the reusable hamster training logic:

package de.metamorphant.blog.hamster;

import java.util.Random;

public class HamsterUtil {  
    public static String performExpensiveHamsterTraining() {
        System.out.println("Hamster training initiated");
        System.out.println("Expensive magic is happening in the background.");
        try {
            Thread.sleep(5000);
        } catch (InterruptedException ex) {
            ;;
        }
        System.out.println("Hamster training completed");
        String port = HamsterUtil.randomPort();
        return port;
    }
    
    public static String randomPort() {
        Random rng = new Random();
        Integer basePortOffset = 30000;
        Integer port = basePortOffset + rng.nextInt(2000);

        return port.toString();
    }
}

The challenges to be solved for our tests are 2-fold:

  • How can I tap into the test execution lifecycle to run setup before all and teardown after all Cucumber tests?
  • How can I inject results from the setup (e.g. DB connection data, random ports, …) into the object under test?

Injecting results

Injection becomes necessary, whenever the hooks are separate from the step definition class. Possible injection strategies are:

  • as configuration e.g. by JVM system properties, environment variables, files, …
  • through static variables and static methods

The Cucumber JVM documentation warns about the use of static variables and recommends using other state management mechanisms. In the case of emulating @BeforeAll/@AfterAll, the static behaviour is exactly what we need and the warning does not apply.

Tapping into Cucumber’s lifecycle

In a typical Maven project, Cucumber runs in a specific context:

  • Maven performs whatever is configured until the relevant test phase (either test or integration-test)
  • Maven starts the JUnit test (by default in a new JVM, see Maven Fork option documentation)
  • JUnit calls its @BeforeClass hooks
  • JUnit delegates to the Cucumber Runner
  • Cucumber executes all scenarios from all features. During the feature execution, Cucumber reports about each internal lifecycle transition by events. For each scenario Cucumber calls (in that order):
    1. Before hooks
    2. Background steps
    3. Scenario steps
    4. After hooks
  • Cucumber finishes execution
  • JUnit calls its @AfterClass hooks
  • Maven performs whatever is configured after the test phase

You probably already noticed the available hook-in-points. I will demonstrate them one by one. All examples use Cucumber’s Java 8 flavour.

Use a Before and a shutdown hook

The infamous issue 515 starts with a recommendation to

  • use a Before step,
    • check if initialization already happened,
    • lazily initialize if not and
  • register a JVM Runtime shutdown hook to perform the shutdown.

The global hook solution in my examples looks like this:

public class HamsterSteps implements En {
  ...
  private static String port;
  
  public HamsterSteps() {
    ...

    Before(() -> {
      if (port == null) {
        System.out.println("Cucumber Before hook called; starting to train a hamster");

        String port = HamsterUtil.performExpensiveHamsterTraining();
        HamsterSteps.injectPort(port);
      
        java.lang.Runtime.getRuntime().addShutdownHook(new Thread(() -> {
          System.out.println("JVM shutdown hook called; gracefully shutting down hamster");
        }));
      }
    });
        
    ...
  }
    
  ...
}

Looks kinda clean, doesn’t it? … Buzzinga! It doesn’t.

You can see the true lifecycle in the test’s console output:

[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ cucumber-global-hook-demo ---
[INFO] Surefire report directory: /shared/cucumber-lifecycle-demo/cucumber-global-hook-demo/target/surefire-reports

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running de.metamorphant.blog.hamster.HamsterCucumberTest
Cucumber Before hook called; starting to train a hamster             <============= Look HERE
Hamster training initiated
Expensive magic is happening in the background.
...

2 Scenarios (2 passed)
10 Steps (10 passed)
0m5.126s


Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.286 sec
JVM shutdown hook called; gracefully shutting down hamster           <============= and HERE

Results :

Tests run: 2, Failures: 0, Errors: 0, Skipped: 0

Watch out for the lines

  • Cucumber Before hook called; starting to train a hamster
  • JVM shutdown hook called; gracefully shutting down hamster

When running with Maven this works out fine, as the Maven Surefire Plugin forks a separate JVM for the test execution. If you e.g. run your tests in an IDE inside a reused, long-lived JVM process, your teardown will not be called after test completion.

⊕⊕⊕ Pros ⊕⊕⊕:

  • Works for most use cases

⊖⊖⊖ Cons ⊖⊖⊖:

  • Dirty trick: java.lang.Runtime.getRuntime().addShutdownHook(...) is a JVM feature and in no way related to Cucumber.
  • Asymmetric: Setup and Teardown hook into different layers
  • Teardown will only happen on JVM termination
  • No per-feature lifecycle possible

Variant: Use a Feature Background

You might think, that the Hamster training is business relevant in our scenarios. In that case, you can express the setup more explicitly in a Feature’s Background steps. Lifecycle-wise it is mostly equivalent to the Before hook and I will skip a separate demo here.

Implement a Cucumber EventListener

As a next variant, I want to explore Cucumber’s Lifecycle Event notifications. For that purpose, I will create and register a plugin PortSetupLifecycleHandler of type EventListener.

package de.metamorphant.blog.hamster.cucumberlifecycle;

import de.metamorphant.blog.hamster.HamsterUtil;
import de.metamorphant.blog.hamster.steps.HamsterSteps;
import io.cucumber.plugin.EventListener;
import io.cucumber.plugin.event.EventPublisher;
import io.cucumber.plugin.event.TestRunFinished;
import io.cucumber.plugin.event.TestRunStarted;

public class PortSetupLifecycleHandler implements EventListener {
  @Override
  public void setEventPublisher(EventPublisher publisher) {
    publisher.registerHandlerFor(TestRunStarted.class, event -> {
      System.out.println("Caught TestRunStarted event; starting to train a hamster");
      
      String port = HamsterUtil.performExpensiveHamsterTraining();
      HamsterSteps.injectPort(port);
    });
    
    publisher.registerHandlerFor(TestRunFinished.class, event -> {
      System.out.println("Caught TestRunFinished event; gracefully shutting down hamster");
    });
  }
}

I register it by annotating the test class:

package de.metamorphant.blog.hamster;

import org.junit.runner.RunWith;

import io.cucumber.junit.Cucumber;
import io.cucumber.junit.CucumberOptions;

@RunWith(Cucumber.class)
@CucumberOptions(
  strict = true,
  features = "src/test/features",
  glue = "de.metamorphant.blog.hamster.steps",
  plugin = {"de.metamorphant.blog.hamster.cucumberlifecycle.PortSetupLifecycleHandler"}
)
public class HamsterWithCucumberLifecycleEventTest {}

The output shows the result:

[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ cucumber-eventlistener-demo ---
[INFO] Surefire report directory: /shared/cucumber-lifecycle-demo/cucumber-eventlistener-demo/target/surefire-reports

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running de.metamorphant.blog.hamster.HamsterWithCucumberLifecycleEventTest
Caught TestRunStarted event; starting to train a hamster                   <============= Look HERE
Hamster training initiated
...
Caught TestRunFinished event; gracefully shutting down hamster             <============= and HERE

2 Scenarios (2 passed)
10 Steps (10 passed)
0m5.102s


Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.264 sec

Results :

Tests run: 2, Failures: 0, Errors: 0, Skipped: 0

The setup and teardown are bound to the event types TestRunStarted and TestRunFinished now. Both are under the control of Cucumber. No outside party is involved. This will also work when using a different driver, e.g. Cli.main or an IDE’s custom implementation.

Unfortunately, there are no event types for binding by feature. You either choose the coarse grained event type for the whole test run or you choose the fine grained event type for an individual scenario. The middle ground is not covered.

⊕⊕⊕ Pros ⊕⊕⊕:

  • Cleanly uses Cucumber’s lifecycle events
  • Works even with non-JUnit drivers like Cli.main

⊜⊜⊜ Neutral ⊜⊜⊜:

  • Test infrastructure tightly coupled to Cucumber

⊖⊖⊖ Cons ⊖⊖⊖:

  • No per-feature lifecycle possible

Use JUnit’s lifecycle to wrap Cucumber’s

When running with JUnit only, you can leverage JUnit’s @BeforeClass and @AfterClass annotations. Additionally, my JUnit wrapper example includes @Before and @After annotations: They will not be called, as after class initialization Cucumber’s test runner takes over.

package de.metamorphant.blog.hamster;

import java.util.Random;

import org.junit.After;
import org.junit.AfterClass;
import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.runner.RunWith;

import de.metamorphant.blog.hamster.steps.HamsterSteps;
import io.cucumber.junit.Cucumber;
import io.cucumber.junit.CucumberOptions;

@RunWith(Cucumber.class)
@CucumberOptions(
  strict = true,
  features = "src/test/features",
  glue = "de.metamorphant.blog.hamster.steps"
)
public class HamsterWithJunitWrapperCucumberTest {
  // A method with annotation @BeforeClass runs at test class initialization time, i.e. before the whole bunch of all tests
  @BeforeClass
  public static void setupClass() {
    System.out.println("JUnit BeforeClass hook started; starting to train a hamster");
    String port = HamsterUtil.performExpensiveHamsterTraining();
    HamsterSteps.injectPort(port);
  }
  
  // A method with annotation @Before runs at test method initialization time, i.e. before every single test
  // for Cucumber Runner it will never appear
  @Before
  public void setupMethod() {
    System.out.println("JUnit Before hook called");
  }

  // A method with annotation @After runs at test method completion time, i.e. after every single test
  // for Cucumber Runner it will never appear
  @After
  public void teardownMethod() {
    System.out.println("JUnit After hook called");
  }

  // A method with annotation @AfterClass runs at test class completion time, i.e. after the whole bunch of all tests
  @AfterClass
  public static void teardownClass() {
    System.out.println("JUnit AfterClass hook started; gracefully shutting down hamster");
  }
}

One typical problem with the JUnit-only assumption are IDEs. For example, IntelliJ’s Cucumber plugin uses the Cucumber Command Line Runner Cli.main and will not run the tests with a JUnit driver. If you go that path, you are in for the ‘works on my machine’ effect.

The output of this variant is:

[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ junit-wrapper-demo ---
[INFO] Surefire report directory: /shared/cucumber-lifecycle-demo/junit-wrapper-demo/target/surefire-reports

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running de.metamorphant.blog.hamster.HamsterWithJunitWrapperCucumberTest
JUnit BeforeClass hook started; starting to train a hamster                <============= Look HERE
Hamster training initiated
...

2 Scenarios (2 passed)
10 Steps (10 passed)
0m0.086s


JUnit AfterClass hook started; gracefully shutting down hamster            <============= and HERE
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.314 sec

Results :

Tests run: 2, Failures: 0, Errors: 0, Skipped: 0

⊕⊕⊕ Pros ⊕⊕⊕:

  • Cleanly wraps the Cucumber test
  • Easy to read
  • Easy to write
  • Easy to reason about

⊖⊖⊖ Cons ⊖⊖⊖:

  • No per-feature lifecycle possible
  • Depends on JUnit runner
    • implies: breaks IntelliJ IDE usage resp. demands additional setup

Leverage Maven lifecycle’s integration test features

A totally different approach is to acknowledge the tests’ nature as integration tests and set up all dependencies completely from the outside. For Maven projects this is done by using the Maven Failsafe Plugin instead of the Maven Surefire Plugin. Typically, this means configuring the Maven Build Lifecycle phases

  • pre-integration-test,
  • integration-test,
  • post-integration-test,

which are all covered by the higher-level verify phase. The pom.xml of my example project shows an example configuration. It uses the Maven AntRun Plugin as a simple way to produce effects and the Build Helper Maven Plugin to generate random ports.

<project xmlns="http://maven.apache.org/POM/4.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
  ...

  <build>
    <plugins>
      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>build-helper-maven-plugin</artifactId>
        <version>3.0.0</version>
        <executions>
          <execution>
            <id>generate-hamster-port</id>
            <phase>pre-integration-test</phase>
            <goals>
              <goal>reserve-network-port</goal>
            </goals>
            <configuration>
              <portNames>
                <portName>hamster.port</portName>
              </portNames>
            </configuration>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <artifactId>maven-antrun-plugin</artifactId>
        <version>1.8</version>
        <executions>
          <execution>
            <id>hamster-init</id>
            <phase>pre-integration-test</phase>
            <goals>
              <goal>run</goal>
            </goals>
            <configuration>
              <target>
                <echo>Maven pre-integration-test phase called: Starting to train a hamster</echo>
                <echo>The hamster will listen on port ${hamster.port}</echo>
                <echo>Hamster training initiated</echo>
                <echo>Expensive magic is happening in the background.</echo>
                <sleep seconds="5" />
                <echo>Hamster training completed</echo>
              </target>
            </configuration>
          </execution>
          <execution>
            <id>hamster-shutdown</id>
            <phase>post-integration-test</phase>
            <goals>
              <goal>run</goal>
            </goals>
            <configuration>
              <target>
                <echo>Maven post-integration-test phase called: Gracefully shutting down hamster</echo>
              </target>
            </configuration>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <artifactId>maven-failsafe-plugin</artifactId>
        <executions>
          <execution>
            <goals>
              <goal>integration-test</goal>
              <goal>verify</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <systemPropertyVariables>
            <hamster.port>${hamster.port}</hamster.port>
          </systemPropertyVariables>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

The heavy lifting is now moved from the Java test code into the Maven configuration. The injection of parameters into the test runtime is done using system properties. Notice that all inputs are generated beforehand and passed by property name convention. Even the pre-integration-test phase takes the random hamster.port as input from outside.

This approach works smoothly, but only as long as you find proper Maven plugins to achieve your setup and teardown goals.

mvn verify produces:

[INFO] --- build-helper-maven-plugin:3.0.0:reserve-network-port (generate-hamster-port) @ maven-failsafe-demo ---
[INFO] Reserved port 34817 for hamster.port
[INFO] 
[INFO] --- maven-antrun-plugin:1.8:run (hamster-init) @ maven-failsafe-demo ---
[INFO] Executing tasks

main:
     [echo] Maven pre-integration-test phase called: Starting to train a hamster   <==== Look HERE
     [echo] The hamster will listen on port 34817
     [echo] Hamster training initiated
     [echo] Expensive magic is happening in the background.
     [echo] Hamster training completed
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-failsafe-plugin:3.0.0-M4:integration-test (default) @ maven-failsafe-demo ---
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running de.metamorphant.blog.hamster.HamsterCucumberIT
Providing a trained hamster
Trying to make the hamster jump through a burning loop
...

2 Scenarios (2 passed)
10 Steps (10 passed)
0m0.079s


[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.325 s - in de.metamorphant.blog.hamster.HamsterCucumberIT
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] 
[INFO] --- maven-antrun-plugin:1.8:run (hamster-shutdown) @ maven-failsafe-demo ---
[INFO] Executing tasks

main:
     [echo] Maven post-integration-test phase called: Gracefully shutting down hamster   <== and HERE
...

As a side effect, you can now run your test in other environments in exactly the same way, e.g. after deploy to a Quality Assurance stage. Just create a different build profile that provides the necessary system properties from a different source.

⊕⊕⊕ Pros ⊕⊕⊕:

  • Clean separation between setup / teardown and test logic
  • Battle-proven pattern
  • Test implementation technology agnostic
  • Can be used to test against live environments without changing the test code

⊖⊖⊖ Cons ⊖⊖⊖:

  • Only feasible, if Maven plugins for setup / teardown are available (or you are willing to create them)
  • No per-feature lifecycle possible

Summary

You have seen 4 ways to work around the missing solution for issue 515. None of them enables per-feature lifecycle hooks. However, all of them enable per-test-run setup and teardown.

Which one you choose depends on your circumstances.

Post header background image by S. Hermann & F. Richter from Pixabay.

Contact us