In this article, I will show how teams at Zalando Marketing Services are using functional tests. We will follow the idea of functional tests: the main concept and the attributes of a good functional test. Then, we will discuss an example based on the TestContainers library used in the Spring environment.

You can find an introduction to the TestContainers library in my previous article Integration tests with Testcontainers, because that is out of the scope of this one.

Definition of functional test

There are many definitions of functional testing. For example, the definition found on Wikipedia is:

Functional testing is a quality assurance (QA) process and a type of black-box testing that bases its test cases on the specifications of the software component under test. Functions are tested by feeding them input and examining the output, and internal program structure is rarely considered (unlike white-box testing). Functional testing is conducted to evaluate the compliance of a system or component with specified functional requirements. Functional testing usually describes what the system does.”

Functional tests answer the fundamental question: Do the features work as intended? Functional tests are not answering the question of HOW it works internally, but rather WHAT the result should be.

Non-functional vs. functional testing

What is the key difference between non-functional software testing and functional testing?

The answer is relatively simple: non-functional testing is concerned with how, and functional testing is concerned with what. Functional testing verifies what the system should do, and non-functional testing tests how well the system works. The intention of functional testing is to verify software actions, and non-functional testing validates the behavior of the application.

Another comparison you might see when discussing this is black-box testing vs white-box testing. Black-box testing looks at the functionality of the software without looking at the internal structures. White-box testing is aware of the internal structures.

Concept

Testcontainers.org is a JVM library that allows users to run and manage docker images and control them from Java code. Zalando uses it mainly for integration and functional tests.

The main purpose of functional tests with the Testcontainers library is to set up a black-box test, by using an environment closest to the production one. To achieve this:

• package and run your service in a docker container;
• run all its dependencies, like: database, queues, streams, as separate docker containers;
• make your service connect to locally run dependencies;
• make your testing code independent of implementation;

The structure of invocation can look like below.

Your entire production code needs to be packaged and run as a docker image. If your service needs to communicate to the database, you need to run the database as a docker image as well. Your functional tests will test your code ran as a docker image, so your testing code does not have any connection to production code.

You also need to remember that a proper pyramid of tests is (when sorted from the highest to the lowest amount of tests):

• unit tests
• component tests
• integration tests
• functional tests
• system tests

It is very nice to have functional tests, but it cannot dominate your testing structure.

Packaging your application into a docker container

Packaging your application into a docker image is pretty simple. In the root of your repository, just define Dockerfile like:

FROM openjdk:17-alpine
COPY service/target/application-exec.jar application.jar
EXPOSE 8080
ENTRYPOINT java ${ADDITIONAL_JAVA_OPTIONS} -jar application.jar

As an alternative solution, I would suggest using Jib

Code separation

I recommend organizing code into a multi-module maven project with two modules: service and functional-tests. The functional-tests module cannot have any dependency on the service module.

.
├── service
│   └── pom.xml
├── functional-tests
│   └── pom.xml
├── Dockerfile
└── pom.xml

Because we don’t have access to the service code, we cannot use any DTO objects, database repositories, etc.

• We should operate on the simplest possible interfaces. For example, if we call a REST endpoint, send plain JSON and read JSON. Don’t create any internal DTOs. It would place you in the position of a real client of your service.
• I recommend using only official interfaces to create resources, e.g. create entities via the REST interface. We could create the entity directly inside the database and inside the test to just retrieve it, but it would not be a black-box test then. If there are changes to the storage of the service in the future, we would need to change our tests.

AbstractFunctionalTests

All functional tests extend the AbstractFunctionalTest class where all needed docker images are run. In our example, I will run my microservice which is connected to the database.

public class AbstractFunctionalTest {
  private static final int HTTP_PORT = 8080;
  private static final int DEBUG_PORT = 5005;
  private static final Logger LOGGER =
      LoggerFactory.getLogger("Docker-Container");
  private static final Network network = Network.newNetwork();

  public static final PostgreSQLContainer postgreSQLContainer =
    (PostgreSQLContainer) new PostgreSQLContainer("postgres:14.2")
    .withUsername("username")
    .withPassword("password")
    .withDatabaseName("databaseName")
    .withNetwork(network)
    .withNetworkAliases("postgres");

  private static final GenericContainer<?> backendContainer;

  static {
    postgreSQLContainer.start();
    backendContainer = ofNullable(System.getenv("CONTAINER_VERSION"))
      .map(version ->
          new ServiceContainer("docker-repository/application", version))
      .orElseGet(() -> new ServiceContainer(".", Paths.get("../")))
      .withExposedPorts(HTTP_PORT, DEBUG_PORT)
      .withFixedExposedPort(DEBUG_PORT, DEBUG_PORT)
      .withEnv("SPRING_PROFILES_ACTIVE", "functional")
      .withEnv("ADDITIONAL_JAVA_OPTIONS",
          "-agentlib:jdwp=transport=dt_socket,"
        + "server=y,suspend=n,address=0.0.0.0:" + DEBUG_PORT)
      .withNetwork(network)
      .withCreateContainerCmdModifier(cmd -> cmd.withName("application"))
      .withLogConsumer(new Slf4jLogConsumer(LOGGER)
          .withPrefix("Service"))
      .waitingFor(Wait.forHttp("/actuator/health").forPort(HTTP_PORT)
      .withStartupTimeout(Duration.ofMinutes(2)));
            backendContainer.start();
            Runtime.getRuntime().addShutdownHook(new Thread(() -> {
              backendContainer.stop();
              postgreSQLContainer.stop();
            }));
          }
  }

As an alternative solution, I would suggest the creation of a Junit5 extension. In this case, we would use an annotation instead inheritance, with the same logic.

Logging

When running the docker image with our service, it is critical to add logging. Without it, you are loosing visibility on errors. Don't forget adding a logger to the container code:

.withLogConsumer(new Slf4jLogConsumer(LOGGER).withPrefix("Service"))

Stopping images

One of the biggest advantages of the TestContainers library is the fact that there is a Ryuk container that stops all other containers when an initial JVM process is terminated. It protects us from unwanted zombie containers (and networks, volumes) in the system. But if you run docker images from multiple maven modules, the Ryuk image can be too slow and the build can crash. That’s why I additionally specify shutdownHook, which stops all docker images when test execution finishes.

Runtime.getRuntime().addShutdownHook(new Thread(() -> {
  backendContainer.stop();
  postgreSQLContainer.stop();
}));

Example of a functional test

An example functional test can look like below. The testing method uses many helper methods to simplify the test. Helper methods are key to make the code readable.

public class AccountFunctionalTest extends AbstractFunctionalTest {

  @Test
  void shouldUpdateAccount() throws JSONException {
    // given
    createAccount();

    // when
   ResponseEntity<String> response = updateAccount();

    // then
   assertThat(response.getStatusCodeValue())
       .isEqualTo(HttpStatus.NO_CONTENT.value());
    var actual = getAccount("00000000-0000-0000-0000-000000000001");
    var expected = readFromResources("get_account_dto.json");
    JSONAssert.assertEquals(expected, actual, JSONCompareMode.LENIENT);
  }

  private void createAccount() {
    var json = readFromResources("create_account_dto.json");
    ResponseEntity<String> response = getTestRestTemplate()
        .exchange("/accounts",
            HttpMethod.POST,
            new HttpEntity<>(json, getPostHeaders()),
            String.class);
    assertThat(response.getStatusCodeValue())
        .isEqualTo(HttpStatus.CREATED.value());
  }

  private ResponseEntity<String> updateAccount() {
   return getTestRestTemplate()
      .exchange("/accounts/00000000-0000-0000-0000-000000000001",
      HttpMethod.PATCH,
      new HttpEntity<>(readFromResources("patch_account_dto.json"),
        getPatchHeaders(etag)),
      String.class);
  }

  private String getEtag(String id) {
    ResponseEntity<String> response = getTestRestTemplate()
      .getForEntity("/accounts/{id}", String.class, id);
    return response.getHeaders().getETag();
  }

  private String getAccount(String id) {
    ResponseEntity<String> response = getTestRestTemplate()
      .getForEntity("/accounts/{id}", String.class, id);
    return response.getBody();
  }

  private HttpHeaders getPostHeaders() {
    HttpHeaders headers = new HttpHeaders();
    headers.setContentType(MediaType.APPLICATION_JSON);
    return headers;
  }

  private HttpHeaders getPatchHeaders(String etag) {
    HttpHeaders headers = new HttpHeaders();
    headers.setContentType(
        new MediaType("application", "merge-patch+json"));
    headers.add(HttpHeaders.ETAG, etag);
    return headers;
  }
}

Advantages of functional tests

The biggest advantages of functional tests are:

• We force engineers to think about the API first principle.
• We are able to test the service as black-box, meaning that when you have a good functional tests coverage, you are able to make a deep refactoring without changing functional tests.
• It gives developers a lot of confidence that the code does what it should do.
• You are sure that your application is correctly packed as a docker image, so another layer of application is tested.
• Functional tests give you a lot of confidence that the application works as expected. I find it very useful during code refactoring.

Disadvantages of functional tests

• Writing functional tests can be time-consuming. Especially when something doesn’t work as expected, debugging becomes much harder. From a different point of view, if you have well-written helper classes you can speed up this process.
• Because functional tests are running services and dependencies (like database, queues) as docker images, we need to run it at least once. Usually, it is slow. For example: PostgreSQL as a docker image needs around 4 seconds to start on my machine, Localstack which emulates AWS components, can take much longer to start, even 20 seconds.
• In an ideal world, we should run new containers for each test, but it would be way too slow. So, we need to run it once for all tests. If functional tests are written in a bad way, they can make tests interfere with each other. It is critical that tests use different object identifiers and that there is a clean state after the test.

Summary

Unit tests force developers to think about methods. Functional tests do the same for applications/components.

I find functional tests to be an interesting concept. The TestContainers library makes it possible to use this concept inside the Java world. It can be pretty expensive to implement it, but it also gives you big confidence that a system still works during deep refactoring.

Functional tests implemented in this way are not for everybody. I would suggest having it in the systems where microservice contracts are not changing very fast. Besides of high cost of development, it gives us a very high confidence level that the delivered applications are working as intended.

Code example

You can find examples of usages in my GitLab project.

If you would like to help us improve our tests and thus help shipping high-quality features for our customers, please consider joining our Engineering Teams at Zalando Marketing Services (ZMS).