Nightly Releases and Continuous Integration for DUNE DAQ Software¶

JCF, Jan-6-2024: the CI dashboard below hasn't worked for some time. It would be good to reinstate a functioning dashboard showing each repo's build status

New Features¶

🆕 Checkout the CI Dashboard to get an overview of recently CI build status for DAQ repositories in DUNE-DAQ organization.

Introduction¶

Continuous Integration (CI):
- developers integrate code into share repository frequently;
- each integration will be verified by automated build and tests;
CI configuration is easy in the case of one or a few packages; however, things can become quickly complicated when number of packages grows:
- PRs for same feature affecting multiple packages need to be built together;
- Each repo to have CI jobs building the whole release is just too expensive;
Nightly release becomes necessary.
- at the end of day, the develop release will be re-made;
- it will be the base for each repo's CI in the next day;
- releated PRs in multiple repos should be merged to develop on the same day.

Nightly build DAQ releases¶

As of Feb-11-2024, there are nightly builds both for the longer-term v5-focused development and for shorter-term v4-focused development used for data taking at np04
Built off the HEAD of the develop or production/v4 branches of the relevant DAQ repositories
Builds in GitHub Actions every night, scheduled not to interfere with American or European working hours
Published (deployed) to /cvmfs/dunedaq-development.opensciencegrid.org

How the nightly releases are made¶

The GitHub Workflows which perform the build-and-publish are defined in .github/workflows/build-nightly-release-alma9.yml and build-nightly-release-sl7.yml; note that they make use of scripts/spack/build-release.sh script to do so, which can also be run at the command line
Analogous to how frozen and candidate releases are made, the nightly release's packages are defined via release.yaml files in configs/coredaq/coredaq-develop,configs/fddaq/fddaq-develop and configs/nddaq/nddaq-develop
The Workflows will use the most recent versions of daq-buildtools and daq-cmake to build all the DUNE DAQ repos off of their develop or production/v4 branches, and upload the build as artifacts (an example).
The build is performed in an image which contains the externals packages and can be found in the daq-docker repo; for the Alma9 build this image is ghcr.io/dune-daq/alma9-slim-externals:v2.0 and for the SL7 build it's ghcr.io/dune-daq/sl7-slim-externals:v1.1
The resulting builds are saved in the images ghcr.io/dune-daq/nightly-release-alma9:latest (Alma9 build on Alma9 OS), ghcr.io/dune-daq/nightly-release-sl7:latest (SL7 build on SL7 OS) and ghcr.io/dune-daq/nightly-release-c8:latest (SL7 build on Centos Stream 8 OS)
The workflow also publishes the builds automatically to /cvmfs/dunedaq-development.opensciencegrid.org

A snapshot of the Actions page

Per-repository CI¶

This is a build of the repository which runs inside ghcr.io/dune-daq/nightly-release-alma9:latest; i.e. the build is done against the latest nightly build
CI build is triggered by:
- Push to develop;
- Pull request to develop;
- Every night after the full nightly build is performed;
- Manual trigger.
Defined in the .github repository
Any change to this Workflow can be published to every individual repo using this script

Some details of the CI workflow¶

It sets up a working directory using the develop release first;
it unsetup the package itself before building;
the build log and pre-build package is saved for 90 days on GitHub;
You can add additional tests after the building step;
Almost all DAQ packages currently included in the release have CI workflow enabled.

CI workflow triggered by Pull request¶

CI workflow status for latest commit merged into develop¶

Manually trigger a CI workflow¶

Last git commit to the markdown source of this page:

Author: John Freeman

Date: Wed Apr 10 13:31:38 2024 -0500

If you see a problem with the documentation on this page, please file an Issue at https://github.com/DUNE-DAQ/daq-release/issues