Thank you Bas Dijkstra for the presentation and Valentina Cupać (Валентина Цупаћ) for hosting it! It was a great introduction to contract testing! Here is a summary Pre-Conditions Before thinking about using contract testing you need to meet some conditions: Unit test & testable architecture - Before implementing contract testing you need to have a solid testing strategy. Micro service / distributed system - Contract testing is used to test the communication between components of a distributed system. It makes little sense to use it on a monolithic architecture. Painful E2E and Integration tests - You can start by having classic E2E and integration tests. Once they start getting unstable or painful, you can think of contract testing. Problems With Testing Distributed System When maintaining a distributed system, you might have thousands of loosely coupled components. All these components might belong to different teams, departments or organisations. Several issues might occur: Breaking change - How to make sure to not introduce breaking change? Synchronous testing - Every component should be up at the same time, with the right state and the right data in the same testing environment. Testing environment - The testing environment itself should be maintained. Scalability - The more the component the harder it is to maintain and test. It takes a long time and slows you down. Tracking integration error - Some integration errors are hard to find using conventional methods. Contract testing Contract testing works by making sure two components of the distributed system are able to communicate by following a contract. Contract testing has 3 main components: Consumer - It's the component that consumes a service. Provider - It's the component that provides a service: you will need a running instance (you can choose to have it connected to a live database or have it use an in memory database). Pact Broker - It's the component that keeps the contract and verification results. You can use OSS pact-broker or Pactflow. They're available online or on premise. Contract testing is: Focused - Narrow the scope of integration testing to two components only per test. Asynchronous - The Consumer and the Provider don't have to be in the same environment. They just have to fulfill their contract as a part of their own build, deployment cycle and pipeline. Testing the communication - The expectations should be on the model of response but not on the actual values. The provider should have acceptance tests to ensure that you'll receive the right business data. We're only checking that we're receiving the right type of data and don't care about implementation details. There are 3 types of contract testing: Consumer Driven Contract Testing (CDCT) - Consumers have expectations of the behavior of the provider. CDCT formalizes the expectation as a contract. The Provider meets the expectation of the Consumers. Provider Driven Contract Testing (PDCT) - The Provider has expectations of how to be used and the Consumers meet the expectation (ie: Open Api specification as a form of contract). Bi-Directional Contract Testing (BDCT) - Uses existing tests for the Consumer and existing Open Api for the Provider to create the contract. Choose the right approach according to your context. Consumer Driven Contract Testing You CDCT by following the steps: Consumer generates the contract and publishes it to the Pact Broker - The contract can be generated using Pact. It's generated when the tests run. The Consumer and Provider are given mocks along with the contract for testing purposes. Provider downloads the contract and verifies compliance. The Provider publishes the verification result. The Consumer and Provider can check the verification results as part of their pipeline (using tools like can_i_deploy) Advantages: Integration issues are easily found using contract testing. Consumers can trust providers' behavior. Providers can refactor without fear of regressions. Drawbacks: Costly - Teams should write more tests and pact implementations hence spending more time and money. Coupling consumer-provider - Don't scale to a large number of consumers. Conflict between consumers may have to be resolved (one consumer expects a 204 and another a 200 status code) Getting providers team on board - It's a lot of work to validate every consumer, it's easier to just have the consumers respect the API. Getting testers on board - It can add a lot of code and tests to maintain. Bi-directional Contract Testing BDCT is only available on Pactflow since March 2022. It's not a full-blown pact integration as CDCT. Consumer contract - You generate the consumer contract using your pre-existing integration test. For example wireMock in JAVA has a built-in WireMockPactGenerator that will help generate the contract. Provider contract - As in PDCT the contract will be created by using the Open Api (swagger file). You can use multiple sources to create contracts: Postman collections. Cypress tests. Open Api. Rest tests. Many other and more to come Conclusion Contract testing can help you ease the pain of integration testing by testing the communication between components by pairs. They have the advantage to be asynchronous and can be an integral part of the deployment pipeline of each component independently. However, they can be verbose and costly. Use contract testing when you need them and be sure to already have a testing culture in place with acceptance tests and a testable architecture first.
I'm not sure I agree on never testing specific values. .. We will test this in unit test but with pact I'd like to be able to check the consumer 'copes' with a range of domain values. Agree that the 'plumbing' ie communication/shared data structures are absolutely highly valuable to test OFC.
Thank you Bas Dijkstra for the presentation and Valentina Cupać (Валентина Цупаћ) for hosting it! It was a great introduction to contract testing!
Here is a summary
Pre-Conditions
Before thinking about using contract testing you need to meet some conditions:
Unit test & testable architecture - Before implementing contract testing you need to have a solid testing strategy.
Micro service / distributed system - Contract testing is used to test the communication between components of a distributed system. It makes little sense to use it on a monolithic architecture.
Painful E2E and Integration tests - You can start by having classic E2E and integration tests. Once they start getting unstable or painful, you can think of contract testing.
Problems With Testing Distributed System
When maintaining a distributed system, you might have thousands of loosely coupled components.
All these components might belong to different teams, departments or organisations.
Several issues might occur:
Breaking change - How to make sure to not introduce breaking change?
Synchronous testing - Every component should be up at the same time, with the right state and the right data in the same testing environment.
Testing environment - The testing environment itself should be maintained.
Scalability - The more the component the harder it is to maintain and test. It takes a long time and slows you down.
Tracking integration error - Some integration errors are hard to find using conventional methods.
Contract testing
Contract testing works by making sure two components of the distributed system are able to communicate by following a contract.
Contract testing has 3 main components:
Consumer - It's the component that consumes a service.
Provider - It's the component that provides a service: you will need a running instance (you can choose to have it connected to a live database or have it use an in memory database).
Pact Broker - It's the component that keeps the contract and verification results. You can use OSS pact-broker or Pactflow. They're available online or on premise.
Contract testing is:
Focused - Narrow the scope of integration testing to two components only per test.
Asynchronous - The Consumer and the Provider don't have to be in the same environment. They just have to fulfill their contract as a part of their own build, deployment cycle and pipeline.
Testing the communication - The expectations should be on the model of response but not on the actual values. The provider should have acceptance tests to ensure that you'll receive the right business data. We're only checking that we're receiving the right type of data and don't care about implementation details.
There are 3 types of contract testing:
Consumer Driven Contract Testing (CDCT) - Consumers have expectations of the behavior of the provider. CDCT formalizes the expectation as a contract. The Provider meets the expectation of the Consumers.
Provider Driven Contract Testing (PDCT) - The Provider has expectations of how to be used and the Consumers meet the expectation (ie: Open Api specification as a form of contract).
Bi-Directional Contract Testing (BDCT) - Uses existing tests for the Consumer and existing Open Api for the Provider to create the contract.
Choose the right approach according to your context.
Consumer Driven Contract Testing
You CDCT by following the steps:
Consumer generates the contract and publishes it to the Pact Broker - The contract can be generated using Pact. It's generated when the tests run. The Consumer and Provider are given mocks along with the contract for testing purposes.
Provider downloads the contract and verifies compliance.
The Provider publishes the verification result.
The Consumer and Provider can check the verification results as part of their pipeline (using tools like can_i_deploy)
Advantages:
Integration issues are easily found using contract testing.
Consumers can trust providers' behavior.
Providers can refactor without fear of regressions.
Drawbacks:
Costly - Teams should write more tests and pact implementations hence spending more time and money.
Coupling consumer-provider - Don't scale to a large number of consumers. Conflict between consumers may have to be resolved (one consumer expects a 204 and another a 200 status code)
Getting providers team on board - It's a lot of work to validate every consumer, it's easier to just have the consumers respect the API.
Getting testers on board - It can add a lot of code and tests to maintain.
Bi-directional Contract Testing
BDCT is only available on Pactflow since March 2022.
It's not a full-blown pact integration as CDCT.
Consumer contract - You generate the consumer contract using your pre-existing integration test. For example wireMock in JAVA has a built-in WireMockPactGenerator that will help generate the contract.
Provider contract - As in PDCT the contract will be created by using the Open Api (swagger file).
You can use multiple sources to create contracts:
Postman collections.
Cypress tests.
Open Api.
Rest tests.
Many other and more to come
Conclusion
Contract testing can help you ease the pain of integration testing by testing the communication between components by pairs. They have the advantage to be asynchronous and can be an integral part of the deployment pipeline of each component independently. However, they can be verbose and costly. Use contract testing when you need them and be sure to already have a testing culture in place with acceptance tests and a testable architecture first.
Thank you Ilias for your summary!
I'm not sure I agree on never testing specific values. ..
We will test this in unit test but with pact I'd like to be able to check the consumer 'copes' with a range of domain values.
Agree that the 'plumbing' ie communication/shared data structures are absolutely highly valuable to test OFC.
this kind of plumbing checks is kinda like extended type safety checks
What about other languages?
Pact supports many languages. See the full list here docs.pact.io/implementation_guides/cli