Maintaining Test Suite and Reference Implementation using Continuous Integration

Guide on how to change either project

It is imperative to keep Test Suite and Reference Implementation in sync as they are vital tools in OpenActive specification implementations, and rely on each other to ensure conformance. Therefore there are some processes in place to make sure they don't fall out of step.

This guide covers the infrastructure around Test Suite and Reference Implementation. For more information on making code changes to Test Suite (such as writing a test) see here.

Continuous Integration (CI)

Continuous Integration for Test Suite and Reference Implementation is run using GitHub Actions on GitHub. For more information about what CI workflows are, and how they are created using GitHub Actions, see here. When a change to a branch is pushed to Github, the CI workflow is triggered.

The CI workflow that is triggered is different depending on the branch name of the implemented change. For more information about the different between coverage/ and non-coverage/ branches, please see below.

The files for the CI workflows can be found in the .github folder in the root of the project folder. It is rare that they will need to be edited as the processes have now been set up.

coverage/ branches: Making a change to both Test Suite and Reference Implementation

In many cases making a change to one of Test Suite or Reference Implementation will necessitate a change being made to the other. For example if a test was created to check for the existence of a new property, that property would have to be added into Reference Implementation to keep the project in sync, and to allow the new test to pass.

By default CI for any PR created in Test Suite will run the current branch of Test Suite against the master branch of Reference Implementation; and visa versa.

However if we have created branches in both Test Suite and Reference Implementation, how do we ensure that CI runs these branches against each other instead of against master?

We must ensure that both branches match a specific pattern:

• Both branches must be called exactly the same name

• Both branches must start with coverage/

For example, both branches must be called coverage/ifus.

PR in Test Suite called coverage/ifus

Equivalent PR in OpenActive.Server.NET for Reference Implementation

This would result in CI pulling and building matching branches from both repositories.

In Test Suite, when the CI workflow is triggered, the matching Reference Implementation branch is built and the tests are run against that newly created Reference Implementation.

Triggered CI job in Test Suite for coverage/ifus

In Reference Implementation, when the CI job is triggered, it pulls and runs the matching Test Suite branches' tests against Reference Implementation. The images below shows the processes that are triggered.

Triggered CI job in OpenActive.Server.NET for coverage/ifus

Apart from the changes stated here, the process for development of OpenActive projects should be following as outlined here.

Making a change to either Test Suite or Reference Implementation

In the cases where making a change to either project would not result in an equivalent change in the other project - such as fixing a bug, the normal development process for changes can be followed. That process can be found here.

For these branches it is assumed that the changes do not affect the other project. In Test Suite this could be adding tests that the Reference Implementation already caters for. In Reference Implementation this could be adding properties not yet tested, or fixing data generation logic. In either project, there will be always be a need for improved documentation, bug fixes, or framework changes that do not necessarily affect the other project.

In these cases, the CI job does not look for a matching branch, but instead pulls and builds the main branch of the other project.

In OpenActive.Server.NET, checkout of matching Test Suite coverage/ branch is not run, and default branch is fetched