This document describes the design of all jobs running in the S-CORE Communication repository. In places where the current implementation does not match the design, a ticket to change the implementation is attached. There shall be no job running in the CI that is not stated in this document.
For each job running in the CI, it shall be possible for a developer to run it locally. It shall be possible to run it fully or partially by providing specific targets. This ensures efficient use of resources because developers have an easy way to test jobs in advance. When a job fails, apart from the relevant logs, it shall clearly state how to reproduce it locally. All jobs must follow the same mechanism to communicate that, a developer should not get that information in different ways depending on the job. In all jobs, the command to reproduce should show the fastest way of reproduction. For example, if a target fails to build, the command line should be such that only builds that target.
Any change to this document needs to be proposed via pull request, and approved by the codeowners by consensus.
It is requested to add the motiviation of the change in the pull request description and explain what goals the change is trying to fulfill.
If the change request has not been discussed in advance with any of the codeowners, we encourage to present the change on Monday's communication sync.
Our CI infrastructure is there to support us on fulfilling the project goals. Each CI job should support achieving at least one of them. Here we list the goals, and we add a shortcut to them to be able to refer to them below. The number or the order could be random. For the purpose of this design we consider all goals equal.
- SCOM-PG-01: Deliver high quality SW
- SCOM-PG-02: Deliver stable SW
- SCOM-PG-03: Deliver functionally safe SW
- SCOM-PG-04: Use resources efficiently
- SCOM-PG-05: Satisfy all functional requirements
- SCOM-PG-06: Satisfy all none-functional requirements
- SCOM-PG-07: Satisfy our processes and enhancement proposals
S-CORE Communication supports multiple architectures, compilers, and operating systems. Some of them are only supported for development and some are supported for production. In the CI we do not test all combinations in all pipelines, we try to find a combination that helps us to find most of the issues without duplication. If we believe that most issues are dected with certain combinations, the rest of combinations might not run in the check and merge pipeline but only in the nightly pipeline.
Below is the list of supported configurations:
| GCC 12.2 + libstdc++ | GCC 15 + libstdc++ | QCC (GCC 12.2) + libc++ | Clang 22 + libc++ | |
|---|---|---|---|---|
| x86-64 linux | Only for development | Only for development | NA* | Only for development |
| ARM64 linux | Only for development | Only for development | NA* | Only for development |
| x86-64 QNX | NA* | NA* | Yes | NA* |
| ARM64 QNX | NA* | NA* | Yes | NA* |
Note: Configuration marked as "NA" are considered as not applicable because the architecture/compiler/operating system configuration is in general not supported.
And this is the list of bazel --config= for the supported configurations:
| GCC 12.2 + libstdc++ | GCC 15 + libstdc++ | QCC (GCC 12.2) + libc++ | Clang 22 + libc++ | |
|---|---|---|---|---|
| x86-64 linux | linux_x64_gcc_12 | (default) / linux_x64_gcc_15 / linux_x64 | NA | linux_x64_clang_22 /clang |
| ARM64 linux | linux_arm64_gcc_12 | linux_arm64_gcc_15 / linux_arm64 | NA | linux_arm64_clang_22 |
| x86-64 QNX | NA | NA | qnx_x64_qcc_12 / qnx_x64 | NA |
| ARM64 QNX | NA | NA | qnx_arm64_qcc_12 / qnx_arm64 /qnx | NA |
Note: Not all configs exist right now, that will be fixed in #792.
Check and merge are designed to ensure certain quality of the SW before a change request can be merged to the main branch. Any job in check and merge queue must be robust against flakiness. That is crucial for the developers to gain confidence in the CI. At the same time, we do not want to have unstable software, however, this is not the task of the check and merge. Preventing flakiness is achieved with the periodic pipelines.
Additionally, check and merge jobs must be relatively fast, otherwise the feedback cycle becomes too long. For that reason, no job running on check and merge should take longer than 20 minutes (daily average).
When creating merge commits, conflicts might arise. If that is the case, and conflicts need to be resolved, it might be hard to see in a pull request what was change. Because of that, to reduce the chance that unintentional changes are merged, we forbid merge commits in pull requests.
Note: Not yet implemented, it will be done with #800.
Time and focus is lost in a PR review when files do not have exactly one new line at the end of the file. To save time from everyone we want this to be enforced in checks and not rely on the reviewers.
Note: Not yet implemented for all file types, it will be done with #801.
Ensure that the Markdown files are formatted following our decided style. The goal is not to ensure a specific format but to have consistency in the project. With a linter, we also want to reduce the possibility of errors and maximize compatibility across different Markdown viewers.
Note: Not yet implemented, it will be done with #786.
Ensure that all Bazel files (BUILD, *.bzl, WORKSPACE, etc.) are formatted in the same way.
Also, it lints the files to ensure the basic Bazel best practices.
Ensure that the modified C++ files are formatted following our decided style. The goal is not to ensure a specific format but to have consistency in the project.
Ensure that the modified Python files are formatted following our decided style. The goal is not to ensure a specific format but to have consistency in the project.
Ensure that the modified Rust files are formatted following our decided style. The goal is not to ensure a specific format but to have consistency in the project.
Builds everything and runs all unit tests for all languages (C++, Rust, Python, etc.).
A Bazel unit test target should not take more than 10 seconds to execute. Short time ensures several things:
- that these are really unit tests
- that we do not have big targets with too many unit tests on it
- that the total time for running all unit test is not longer than necessary
- that we have a fast job that developers can run locally to test API breaking changes in common libraries
- that when we run test targets multiple times to detect flakiness, the running times will not explode
To ensure that the job is fast, the targets should be built with the right combination of flags that provides a fast enough compilation but still keeping a fast enough execution. To ensure that the job is robust, a failing test will be retried up to 3 times.
Runs all C++ unit tests under address, undefined behavior (UB), and leak sanitizers.
Each unit test target also has a maximum run time as the job for unit tests without sanitizers and the same stability design where a test is rerun 3 times in case of failure.
This job cannot be combined with any of the other sanitizer jobs due to how sanitizers work. If the tool used for sanitizers is ever changed, this decision could be revisited.
Runs all C++ unit tests under thread sanitizer.
Each unit test target also has a maximum run time as the job for unit tests without sanitizers and the same stability design where a test is rerun 3 times in case of failure.
This job cannot be combined with any of the other sanitizer jobs due to how sanitizers work. If the tool used for sanitizers is ever changed, this decision could be revisited.
Note: Not yet implemented, it will be done with #863.
Running integration tests on x86-64 QNX in addition to ARM64 QNX reduces the risk that the QNX only code that we have in our codebase, behaves different in an x86-64 architecture compared to an ARM64 architecture.
Note: Not yet implemented, it will be done with #788.
Note: Not yet implemented, it will be done with #789.
Runs the clang-tidy checks for the host.
Runs the clang-tidy checks for the target.
Note: Not yet implemented, it will be done with #790.
Runs the default clippy checks for the host.
There are many linters for Python with no clear recommendation. We chose ruff because it is fast, and it has a good set of default checks. Runs the ruff checks for the host.
Make sure that the publicly visible targets are part of the public API specification. Also makes sure that we do not break the API (source-code compatibility) accidentally.
Makes sure that S-CORE Communication can be integrated as a module, and that it can work with different versions of Bazel and Bazel modules.
Note: Not yet implemented, it will be done with #791.
Ensures that our SW is stable by running each unit tests 20 times. If a test fails at least one time, it is considered a bug and a ticket is created.
The setup of this job is exactly the same as the one running in check and merge
but with the addition of the flag --runs_per_test=20.
Ensures that our SW is stable by running each unit tests 20 times. If a test fails at least one time, it is considered a bug and a ticket is created.
The setup of this job is exactly the same as the one running in check and merge
but with the addition of the flag --runs_per_test=20.
To be able to have fast feedback, all our CI job use Bazel cache. To avoid permanent cache poisoning and cache size exploding, the cache is recreated nightly. More details can be found in the cache strategy design document.
Ensures that specific manual actions are performed by the reviewers of a pull request. Reviewers must acknowledge that they have performed the actions in the checklist before a pull request can be merged. During merge to the base branch, the acknowledgements are stored in the git history as evidence. The checklists are defined in the review checklist configuration document.
It is acknowledged that this design is not perfect, and experience will teach us that this design will have to be adapted. Most changes to this document should be triggered by new requirements or because of a follow-up action from a post-mortem analysis.
Also remember that the decision of moving a job from a late pipeline to an earlier pipeline should not be made based on singularities and without considering testing strategies of affected components. For example, some post-mortem might conclude that an issue could a) have been prevented by moving a job from "nightly" to "check and merge" or b) by creating a unit test. In such case the latter option shall alway be the preferred one.