Contract testing vs functional testing
May 7, 2024
Understanding the Difference Between Different Types of Tests and Establishing an Appropriate Testing Strategy
In today's software development landscape, the adoption of distributed architectures is prevalent, where each service handles specific functionality and communicates with others through network message exchanges, typically via REST requests. In this context, the need arises to ensure compatibility between services, ensuring they interact as expected and that changes in one do not negatively affect others. This is where Contract Testing, particularly the Pact tool, plays a crucial role.
But what sets contract testing apart from integration and functional testing? That's what we'll explore in this article.
Why It Matters
Understanding the difference between different types of tests allows you to establish an appropriate testing strategy that optimizes the software development process.
Being able to differentiate the objective and, above all, the scope of each type of test will make your testing suite much more reliable and faster. This is because it will be easier to detect errors in early stages, be more precise in identifying the source of the problem, avoid blockages between teams, and improve communication among them.
Digging Deeper
When it comes to testing APIs, let's now examine the fundamental differences between the types of tests:
Contract Testing | Functional Testing | End-to-End Testing | |
---|---|---|---|
What is it? | Validates the interaction between two services based on an agreed-upon "contract." | Verifies the integration and proper functioning between modules or services. | Tests that simulate complete user scenarios from start to finish. |
Test Objective | Ensure that services can communicate correctly according to defined contracts. | Verify that different modules or services work together as expected. | Validate the entire system, including integrations with third parties. |
How it works? | Uses tools like Pact to simulate calls between services, with specific examples to verify that contracts are met. | Generally requires all components to be operational to test their interaction. | Involves comprehensive system tests, simulating workflows of an end user. |
Test Scope | Focuses on specific interaction between services based on defined contracts. | Covers multiple components or services, but not necessarily the entire system. | Involves the entire system and all its operational dependencies. |
SDLC Stage | During development, before full integration. | After development of individual modules, before final delivery. | Generally in the final stages of the development cycle or during user acceptance. |
Test Benefits | Fast and efficient for detecting integration problems in early stages. | Helps identify issues and edge cases in systems. | Ensures the entire system works correctly for the end user. |
Limitations | Does not verify complete functionality or system performance. | Does not capture errors that may only appear in the complete interaction of all components. | Requires more time and resources; can be challenging to fully automate. |
Now that we've seen the differences, and it's clear that contract testing is based on compatibility between services, let's see in which scenarios it's interesting to apply it and when it's not.
✅ Validating that the message body is appropriate
This is one of the main advantages of using contract testing because, through the specification of concrete examples ("expectations") and the mocking of network requests, we can automatically verify that both producer and consumer exchange information as agreed upon.
This is not simply checking the data schema; the main differences are:
-
Only the fields that actually affect the consumer are taken into account, potentially ignoring the rest of the response data.
-
It allows the use of "wildcards" or regular expressions to validate the expected data type (email, date, number, string, etc.). This avoids fragility in checks since it is not tied to the exact value.
-
It's not necessary to have the producer part deployed; validations are done based on the defined contract and through mocked applications that generate real network requests.
✅ Validating that the client will be able to process the agreed response
To avoid getting into functional testing territory, we must understand that the scope of this check is solely to validate that the consumer is able to process the provider's response properly. We're not trying to check the side effects that occur afterward.
For example, if our client application requests data of a student to calculate their average grade, the test on the client side would be aimed at ensuring that we indeed receive the student's data and that we're capable of processing it properly. In practice, in most cases, it will consist of checking that we have a class that maps this response correctly.
Here's the resulting code:
//Expectation
@Pact(consumer = "student-consumer", provider = "student-provider")
public V4Pact getStudentWithId1(PactDslWithProvider builder) {
return builder.given("student 1 exists")
.uponReceiving("get an existing student with ID 1")
.path("/students/1")
.method("GET")
.willRespondWith()
.status(200)
.headers(Map.of("Content-Type", "application/json"))
.body(newJsonBody(object -> {
object.stringType("id", "1");
object.stringType("name", "Fake name");
object.stringType("email", "some.email@sngular.com");
object.numberType("studentNumber", 23);
}).build())
.toPact().asV4Pact().get();
}
//Validation
@Test
@PactTestFor(pactMethod =
"getStudentWithId1")
void getStudentWhenStudentExist(MockServer mockServer) {
Student expected = Student.builder()
.id("1")
.name("Fake name")
.email("some.email@sngular.com")
.studentNumber(23).build();
RestTemplate restTemplate = new RestTemplateBuilder().rootUri(mockServer.getUrl()).build();
//Check that our service can perform and process the response properly
Student student = new StudentService(restTemplate).getStudent("1");
assertEquals(expected, student);
}
✅ Validating that the endpoint routes are correct
Following the previous code example, we can see how we're specifying the route and specific parameters with which the request should be made ("students/1"). This will enforce both parties to comply with what's agreed upon regarding the definition of endpoints.
✅ Checking that the response codes are appropriate
Also, in the code example, we can see how the expected response code is specified (200). Similar to the previous case, we're ensuring that the provider will send the agreed-upon response code.
✅ Validating that request headers and parameters are appropriate
In the code example, we can also see how the expected request headers are specified. Therefore, if either party fails to comply, the test will fail.
❌ Validating different request scenarios
Suppose we want to validate that when making "POST" requests, the provider returns "400 Bad Request" under certain cases.
In this case, as mentioned earlier, through contract testing, we should check that there is indeed the possibility for the provider to return the 400 code and that the client can process such a response type.
However, it wouldn't be appropriate to use contract testing to check all possible scenarios where the service might return "Bad Request." Doing so would make our tests too fragile because they would largely depend on the specific examples we're using to generate them. We must consider that validation or business rules may change, and this shouldn't affect the communication between services.
For example, this wouldn't be a good contract definition because if the maximum allowed length is changed, the tests would fail even if both systems are compatible:
When "creating a user with a blank username"
POST /users { "username": "", email: "...", ... }
Then
Expected Response is 400 Bad Request
Expected Response body is { "error": "username cannot be blank" }
When "creating a user with a username with 21 characters"
POST /users { "username": "thisisalooongusername", email: "...", ... }
Then
Expected Response is 400 Bad Request
Expected Response body is { "error": "username cannot be more than 20 characters" }
When "creating a user with a username containing numbers"
POST /users { "username": "us3rn4me", email: "...", ... }
Then Expected Response is 400 Bad Request
Expected Response body is { "error": "username can only contain letters" }
In this case, a single example where a generic "Bad Request" is specified would be sufficient. This way, we should also be more generic when specifying the expected return message, avoiding specifying the exact text expected.
Finally, it could have a case like this:
When "creating a user with an invalid username"
POST /users { "username": "bad_username_that_breaks_some_rule_that_you_are_fairly_confident_will_not_change", ... }
Then
Response is 400 Bad Request
Response body is { "error": "<any string>" }
❌ Validating side effects
By side effects of a test, we can understand system changes such as database changes, calls to third-party services, file modifications, notification sending, etc.
Although technically, these types of situations can be verified using Pact, it's not the purpose of contract tests. The main disadvantages are that it makes these tests extremely fragile and dependent on other systems.
Therefore, while possible, it would be counterproductive since we would lose the main advantages that Pact offers, such as execution speed, test independence, and avoiding blockages.
Hence, these types of validations should be postponed to later stages: integration or end-to-end.
❌ Validating workflows (sequence of steps)
Pact allows specifying an initial system state before the test execution. This "setup" is essential in any testing framework. However, we shouldn't confuse this with executing a series of actions sequentially and checking the final state.
That is, preparing the system to have a certain user on which we request data is not the same as modifying a field on the same user and then checking the consequences of that change.
For example, through contract testing, it wouldn't be appropriate to propose a test like this:
Given a VIP user with ID 1
When
PUT /users/1 { VIP: "false" ... }
Then
Expected Response is 200 OK
And
GET /discounts?user=1
Then
Expected Response is 200 OK
Expected Response body {"No discounts for user 1"}
In Summary
In a world where microservices architecture and distributed applications are the norm, the importance of a solid and efficient testing strategy cannot be underestimated. In this article, we've explored the key distinctions between Contract Testing, Integration Testing, and End-to-End Testing, underscoring their relevance and applicability in different software development scenarios.
Contract Testing, facilitated by tools like Pact, focuses on ensuring that services interact correctly according to defined contracts, offering an efficient way to detect communication problems early in the development cycle. Integration Testing verifies cooperation between modules or services, identifying issues that only arise when these components are deployed and interact with each other. Finally, End-to-End Testing evaluates the entire system in an environment that simulates end user interaction, ensuring that all parts of the system work together smoothly.
By understanding these differences and applying the appropriate type of test at the right time, teams can significantly improve the efficiency of their development workflow, detect errors in early stages, and eliminate blockages, all while ensuring that each component works as intended.
Incorporating these practices not only optimizes the development process but also enhances collaboration between teams, ensuring that changes in one service do not affect others. Ultimately, choosing the right testing strategy is crucial for high-quality software development.
Our latest news
Interested in learning more about how we are constantly adapting to the new digital frontier?
November 26, 2024
The digital ecosystem with which Vítaly reinvents healthcare
November 21, 2024
Contract Testing as a Service: Support your clients
October 14, 2024
PactFlow & Contract Testing: A Business Case Study
October 7, 2024
Custom Lint Task Configuration in Gradle with Kotlin DSL