CI/CD pipelines and self-hosted runners¶

For maintainers operating the GitHub Actions pipelines. This is the source of truth for the workflow set, their triggers and jobs, the reusable build workflow, and self-hosted runner configuration. The release process these pipelines run is described in docs/releasing.md; the branch and PR rules that drive CI triggers are in CONTRIBUTING.md.

Workflow set¶

The table below is the complete inventory of everything under .github/ — the core build/test/release pipeline, the scanners, and the self-contained supporting workflows, plus the repo's composite actions. Every workflow has a section below.

ci.yml and release.yml share one build definition (_build.yml), so the suites that run on a PR and the suites that gate a release can never drift.

ci.yml¶

ci.yml builds on push only for the long-lived branches (main, develop); every other branch is exercised through its pull request into develop or main.

Triggers¶

on:
  push:
    branches: ["main", "develop"]
  pull_request:
    branches: ["develop", "main"]

Only main and develop build on push. Feature branches build via their pull_request into develop/main (the pull_request trigger targets those two branches). A WIP push to a branch that has no open PR runs no CI at all — and a push to a branch that does have a PR is already covered by the PR run, so this also avoids the push+PR double-run.

Concurrency is keyed on the PR head branch (or the ref for branch pushes) with cancel-in-progress: true, so a new push supersedes an in-flight run for the same PR or branch.

Jobs¶

Docs-only gating. build-and-test, e2e-ui, matter-bridge, version-check, and docker-verify are all gated on changes reporting code == 'true', so a docs-only change skips the whole build/test matrix while ci-status still reports success. branch-name, i18n-catalogs, and platform-macros run regardless (cheap, source-only checks).

e2e-ui modes. The Playwright runs mirror the modes encoded in playwright.config.ts, selected by environment variable. The three identity specs each run in their own step because they have incompatible auth-store preconditions (auth.spec needs an empty store for the first-run wizard; admin.spec self-seeds an admin; guest.spec configures the Guest scope), so each must boot against a fresh --auth-state-dir:

The remaining modes (TLS, bootstrap admin, open bind, MQTT, and the persistence modes) run in automated-codescanning.yml's coverage sweep, which is a superset of this table — see automated-codescanning.yml.

docker-verify checks. It builds the ci target, then the runtime target (tagged aqualink-automate:verify), runs docker run --rm -e MATTER_ENABLED=false aqualink-automate:verify --version, and confirms the bundled sidecar is present (node --version plus a test for /opt/matter-bridge/dist/index.js). Both Docker builds are wrapped in a bounded retry so a transient Docker Hub base-image timeout does not fail CI. This cold, from-source build is kept deliberately as the gate that the multi-stage Dockerfile + vcpkg toolchain still build end to end — the release path's docker-publish assembles its image from prebuilt install trees and skips this, so docker-verify is its safety net.

_build.yml¶

_build.yml is the single definition of how the C++ project is configured, built, tested, and optionally packaged on each platform. It is a workflow_call reusable workflow — it has no triggers of its own and only runs when another workflow calls it.

Inputs¶

The matrix¶

The job runs a four-row matrix with fail-fast: false, so one platform failing does not cancel the others:

These are the same presets you use locally — see INSTALL.md.

The arm64 row builds natively on an aarch64 runner (no cross-compile / QEMU) so the .deb/.rpm/.tgz install on a Raspberry Pi and other arm64 hosts; CPack derives the package architecture from the target, so the packages are correctly labelled arm64/aarch64. Both Linux rows can upload their install tree (installtree-linux-gcc / installtree-linux-gcc-arm64, gated on the upload_installtree input): CI uploads only the x64 tree (for e2e-ui), while the release path uploads both so docker-publish can assemble the multi-arch image without recompiling.

Runner selection per row is vars.RUNNER_LINUX / vars.RUNNER_LINUX_ARM / vars.RUNNER_WINDOWS with a GitHub-hosted fallback (the arm64 row falls back to ubuntu-24.04-arm); macOS always runs on macos-latest. See Self-hosted runners.

Test scope¶

The Test step runs the full test preset with no -L label filter, so unit and integration tests run together in one pass. There is no separate fast/slow split in CI.

Packaging (release path only)¶

When do_package is true:

1. On Linux, the whole configure/build/test/cpack sequence for the release path runs inside a gcc:15-bookworm container (glibc 2.36 — the Debian Bookworm / Raspberry Pi OS baseline), natively per arch, so the produced packages run on a stock Pi and every newer distro; the bundled gcc-15 libstdc++/libgcc keep the older-glibc base safe for the C++23 code. Windows and macOS run cpack --preset=<package_preset> on the host after the normal build.
2. Smoke-tests the package on a clean target — installs the .deb in a fresh debian:bookworm container (Linux), extracts the ZIP and runs the exe (Windows), or extracts the TGZ and runs the binary (macOS). Each asserts aqualink-automate --version succeeds. The Linux test deliberately uses a clean debian:bookworm rather than the build image: it catches both a missing runtime dependency and a regression to a too-new glibc/libstdc++ baseline (which a newer-distro test would silently pass) before the package is published.
3. Uploads the packages as an artifact named packages-<configure_preset> (for example packages-config-linux-gcc), with retention-days: 30.

Version stamping¶

The version is threaded explicitly into the configure step: _build.yml passes -DDERIVED_VERSION_OVERRIDE=${{ inputs.version_tag || '0.0.0-dev' }} (so a CI build with no tag stamps 0.0.0-dev rather than emitting a noisy git describe warning on the shallow checkout), and cmake/GitVersionDerivation.cmake honors DERIVED_VERSION_OVERRIDE over git describe (it is also how the Docker image is versioned, since the build context omits .git). Release callers pass the resolved tag as version_tag, so a workflow_dispatch build stamps the right version even though its tag does not exist yet (github-release pushes it at the very end) — no local pre-tagging is needed.

release.yml¶

release.yml produces a release from a version tag or a manual dispatch.

Triggers¶

on:
  push:
    tags: ["v*"]
  workflow_dispatch:
    inputs:
      version:     # e.g. v1.0.0, v1.0.0-beta.1
      prerelease:  # boolean
      dry_run:     # boolean — build packages but create no release

There is no push to main trigger — pushing to main does not release. A release is cut either by pushing a v* tag or by running the workflow manually. Concurrency is grouped on release-<version> with cancel-in-progress: false, so an in-flight release is never cancelled by a second run of the same version.

Job graph¶

resolve-version ──> build-packages (_build.yml) ──> docker-publish ──┬─> homeassistant-addon-publish
                                                                     └─> github-release ──> publish-repos (publish-repos.yml)

Every job after build-packages is skipped on a dry run (is_dry_run == 'true'); build-packages still runs so the pipeline can be validated end to end without publishing. A cleanup-failed-prerelease job additionally watches the graph with if: always() (below).

resolve-version guards¶

resolve-version rejects a bad release before any package is built:

• Tag format. The tag must match ^v[0-9]+\.[0-9]+\.[0-9]+(-(alpha|beta|rc)\.[0-9]+)?$ (for example v1.2.3 or v1.2.3-rc.1).
• Cut from main. Real releases must be contained in origin/main (git merge-base --is-ancestor against the fetched origin/main). develop is the integration branch; only main holds released code. Dry runs are exempt so the pipeline can be validated from any branch.
• Existing tag. On a workflow_dispatch (non-dry-run), the job fails fast if the tag already exists, so a collision is caught before building every package rather than at the final push.

docker-publish¶

This job runs only on Linux when not a dry run, with packages: write (GHCR push) plus id-token: write + attestations: write (attestation signing).

1. Logs in to GHCR, and to Docker Hub only if DOCKERHUB_USERNAME is set (so fork PRs and credential-less runs still build anonymously).
2. Derives tags from docker/metadata-action: type=sha, the full semver, major.minor, a raw latest tag enabled only when the release is not a prerelease, and a raw edge tag enabled only for prereleases (the floating prerelease channel — see the tag table in releasing.md).
3. Downloads the per-arch install trees uploaded by build-packages and assembles the multi-arch (linux/amd64 + linux/arm64, via QEMU) runtime-assembled target from them — no C++ recompile; only the Matter sidecar's TypeScript builds — pushing to GHCR (and Docker Hub when credentials are present), wrapped in a bounded retry (up to 3 attempts) so a transient Docker Hub CDN timeout does not fail the release. --metadata-file records the pushed image-index digest. (CI's from-source docker-verify remains the cold-build gate for the runtime target itself.)
4. Attests the image by digest: a keyless build-provenance attestation (actions/attest-build-provenance, Sigstore/OIDC) and an SPDX SBOM attestation (anchore/sbom-action → actions/attest-sbom), both push-to-registry: true so they attach to the image in GHCR. Attesting by digest (not tag) binds the proof to the exact bytes pushed.
5. Runs a post-push smoke test: pulls the GHCR image and asserts --version contains the resolved version, for both architectures (arm64 under QEMU). Because buildx build --push pushes straight to the registry, this genuinely exercises the published artifact rather than a local cache.

github-release¶

This job runs only when not a dry run, with contents: write (tag + release) plus id-token: write + attestations: write (attestation signing).

1. For a workflow_dispatch build, it pushes the resolved tag to origin at the end — the tag is created here, not at the start.
2. Downloads every packages-* artifact from build-packages (merge-multiple: true).
3. GPG-signs the artifacts (gated on REPO_GPG_PRIVATE_KEY, the repo-signing key): a detached .asc per binary, a signed SHA512SUMS manifest, and the exported public key, all added to the release. No-ops if the key is unset.
4. Attests the packages: a keyless build-provenance attestation and an SPDX SBOM attestation over the shipped binaries (the SBOM is also uploaded as a release asset). Verified by consumers with gh attestation verify <file> --repo iainchesworthlabs/aqualink-automate.
5. Runs gh release create <tag> --generate-notes, adding --prerelease when the release is a prerelease, and attaches the downloaded package files, signatures, checksums, and SBOM.

Both attestation mechanisms are described end-to-end for consumers in SECURITY.md > Verifying build authenticity.

See docs/releasing.md for the operator-facing release walkthrough.

Release helpers (auto-tag.yml, suggest-version.yml, publish-repos.yml)¶

Three small workflows surround release.yml. None is part of the PR gate.

suggest-version.yml¶

Propose-only. On a develop → main release PR (pull_request into main, types opened/synchronize/reopened) it runs scripts/next-version.sh --markdown against the full history + tags and upserts the result as a sticky PR comment plus a job summary. It never creates or pushes a tag — the human still decides the version and pushes it (see the pre-release checklist in releasing.md). The same script is runnable locally: scripts/next-version.sh.

auto-tag.yml¶

Opt-in tag advancement. It triggers on every push to main, but the single job runs only when the repo variable AUTO_TAG_ENABLED is "true" — by default it does nothing and versions are tagged by hand. When enabled, it advances only the prerelease counter (vX.Y.Z-<channel>.N → .N+1, channel from AUTO_TAG_CHANNEL, default beta) and pushes the tag, which fires release.yml exactly like a manual tag push. Everything else stays a human decision by design: it refuses to invent a first tag, bump the base version, or switch channels, and it skips when there are no new commits since the last tag or when the delta is docs-only (same classifier as ci.yml's changes job — a docs merge must not trigger a redundant binary release).

publish-repos.yml¶

Publishes the GPG-signed APT (reprepro) and DNF (createrepo_c) package repositories to the gh-pages branch, so users can apt install / dnf install and receive upgrades. It is a reusable workflow invoked by release.yml's publish-repos job right after the GitHub release is created — a release: published trigger would never fire, because GitHub suppresses events for actions performed with the default GITHUB_TOKEN — and it is also dispatchable with a tag input to (re)publish an existing release. It downloads the release's .deb/.rpm assets, signs each .rpm in place (rpm --addsign; the DNF repo advertises gpgcheck=1, a per-package check), builds both repos, signs the repo metadata, and publishes with keep_files: true so previously published versions accumulate. It no-ops until the REPO_GPG_PRIVATE_KEY secret is configured — the one-time setup and operator walkthrough live in releasing.md.

docs.yml shares the gh-pages branch with this workflow: the rendered docs site owns the root index.html and page tree, the package machinery owns apt/, rpm/, key.gpg, and the install-*.sh scripts. The filename sets are disjoint and both publish with keep_files: true, so neither clobbers the other (see Docs site).

automated-codescanning.yml¶

Code scanning is slow (each scanner does a full compile) and was previously the heaviest thing on the PR critical path, so it does not run on PRs at all: it runs nightly, matrixed over main and develop, on the org's shared self-hosted fleet (also used by other repos — see ci-self-hosted-runners.md). ci.yml's build/test/e2e matrix remains the per-PR correctness gate; this workflow is detection, not a merge gate.

Triggers¶

on:
  schedule:
    - cron: '7 4 * * *'   # nightly, 04:07 UTC
  workflow_dispatch:

04:07 UTC is deliberately off the top of the hour (dodges cron-minute congestion) and offset from the ac3forge repo's 02:00-04:00 UTC nightly window on the same shared self-hosted fleet. A nightly run against main on its own ref also keeps the default-branch security baseline current (the Security tab / CodeQL "tool status"): the old dedicated push: [main] trigger existed only for that — a pull_request SARIF upload is recorded against the throwaway refs/pull/N/merge ref and never updates main's baseline — and is redundant now that a nightly scan touches refs/heads/main directly every night.

None of these jobs is a required status check on develop or main (verified via gh api .../branches/<branch>/protection), so removing them from pull_request/push stranded no PR.

Branch matrix¶

Every job below carries a strategy.matrix.branch: ['main', 'develop'] axis instead of reacting to whatever ref triggered the workflow. This matters because schedule (and workflow_dispatch) always report github.sha/github.ref as the default branch's tip, regardless of what a job actually checks out — so each job:

1. Checks out ref: ${{ matrix.branch }} explicitly.
2. Captures the real commit via a Resolve checked-out SHA step (git rev-parse HEAD) immediately after checkout.
3. Passes that branch and SHA explicitly to whatever uploads results — ref/sha on codeql-action/upload-sarif and codeql-action/analyze, -Dsonar.branch.name=${{ matrix.branch }} on the SonarCloud scan — instead of relying on the (always-wrong-for-develop) ambient GITHUB_REF.

Artifacts shared between jobs in the same run (e2e-coverage-xml, plus the debug sarif-results/sarif-file artifacts) are suffixed -${{ matrix.branch }} so the two branch legs don't collide on upload.

Jobs¶

Coverage = unit + e2e. The SonarCloud new-code coverage gate reflects what both test layers exercise. The unit/integration binary's coverage alone misses every line only the running app reaches (HTTP routes, WebSocket, MQTT, bootstrap), which read as uncovered and depressed the gate. CodeScanning_E2ECoverage instruments the app, drives the Playwright suite against it (the app exits cleanly on Playwright's SIGTERM — see gracefulShutdown in playwright.config.ts — so gcov flushes .gcda), and emits a second report. CodeScanning_SonarCloud needs: it and merges the two; if the e2e job fails, the scan still runs with unit-only coverage (it never drops the whole gate).

The coverage mode sweep. The job runs the six e2e-ui modes from ci.yml plus seven more that exist to reach code the default modes cannot. Every mode step is continue-on-error — this job harvests coverage, it is not a functional gate (e2e-ui is), so a spec flake must not discard coverage already written:

Supply-chain scanning (trivy.yml, osv-scanner.yml, scorecard.yml)¶

Three lightweight workflows cover the supply-chain surfaces the compile-heavy automated-codescanning.yml does not. Unlike the code scanners, they run on GitHub-hosted ubuntu-latest (free for public repos, no self-hosted runner), need no C++ build, and publish to Security > Code scanning. None fails a PR — a finding surfaces in the Security tab rather than blocking the merge.

Division of labour with the existing tooling: CodeQL / SonarCloud / MSVC scan first-party source; Dependabot updates the Actions + npm manifests (and raises its own vulnerability alerts for them); OSV-Scanner closes the vcpkg gap Dependabot cannot see; Trivy covers the image layers no source or manifest scanner reaches. The vcpkg-built C++ libraries inside the image carry no package metadata for Trivy to match, so OSV-Scanner (reading the manifest) is what covers them — the two do not overlap.

The two PR-facing workflows (trivy.yml, osv-scanner.yml) still run on pull_request (unlike automated-codescanning.yml, which moved to nightly-only — see above) and carry their own promotion-PR skip and fork-PR gating, the same pattern automated-codescanning.yml used before that move. scorecard.yml does not run on pull_request at all (its checks are repository-level and need a token a fork PR lacks). The weekly crons are staggered (22:17 / 22:27 / 22:37 UTC) so they do not all contend at once.

Scorecard hardening applied¶

Two Scorecard checks are satisfied structurally rather than per-finding:

• Token permissions — every workflow declares permissions: {} (deny-all) at the top level and re-grants the minimum write scope on the single job that needs it (e.g. contents: write on the tag/publish jobs, actions: write on the cache-cleanup job, security-events: write on the SARIF-upload jobs). A top-level write is what Scorecard penalizes; a narrowly job-scoped write is the recommended pattern and the residual per-job grants are the least privilege each job genuinely requires.
• Pinned dependencies — the external base images in the root Dockerfile (ubuntu:26.04, node:26-bookworm-slim) are pinned by @sha256: digest. .github/dependabot.yml carries a docker ecosystem entry so Dependabot advances those digests weekly — a digest pin without an update path would otherwise freeze the image onto a stale, unpatched base. (Internal FROM <stage> references and the GitHub-hosted runner toolchains are not digest-pinnable and are not flagged as real findings.)

Fuzzing (fuzzing.yml)¶

fuzzing.yml runs the libFuzzer harnesses over the app's untrusted-input parsers — the RS-485 protocol decoders (Jandy + Pentair message deserialisers) plus the schedule/WebSocket JSON, MQTT payload, config, query-string, JWT, duration, and replay-line parsers — the one supply-chain surface the posture scanners above cannot reach, because it is a runtime property of first-party parsing code. It builds the config-linux-llvm-fuzzing preset (Clang + -fsanitize=fuzzer,address), seeds each harness's corpus (the decoder corpora come from the recorded test/fixtures/**/*.cap captures), and fuzzes each harness for a bounded time; a crash fails the job and uploads the reproducer as an artifact. This gives the OpenSSF Scorecard Fuzzing check a genuine signal rather than a posture tick.

• Triggers: weekly cron (47 22 * * 1, staggered after the supply-chain crons); pull_request touching a fuzzed parser's source (src/jandy/**, src/pentair/**, and the fuzzed src/core/ subsystems) or the fuzzing scaffolding (fuzz/**, cmake/Fuzzing.cmake); and workflow_dispatch (with a max_total_time input). It is not a required gate — a bounded run is best-effort reassurance, but any crash it does find is a real bug.
• Matrix: one job per harness — ten of them, fuzz-jandy-message through fuzz-replay-line (the full list lives in fuzzing.md) — on the GitHub-hosted ubuntu-latest (or vars.RUNNER_LINUX), with the same fork-PR gating as the build jobs.
• A crash is handled per the bug-fix discipline: fix the decoder + add a regression test, never weaken the parser. Full harness/corpus/run docs: fuzzing.md.

cleanup-branch-caches.yml¶

This workflow keeps the Actions cache from filling up with stale per-PR entries.

• Trigger: pull_request with types: [closed].
• Action: On ubuntu-latest with actions: write, it installs the gh-actions-cache extension and deletes every cache key scoped to the closed PR's merge ref (refs/pull/<number>/merge). Deletion failures do not fail the workflow.

Dependabot auto-merge (dependabot-auto-merge.yml)¶

Routine dependency bumps merge themselves. .github/dependabot.yml raises one grouped weekly PR per ecosystem (GitHub Actions, npm root, npm matter-bridge/, Docker digests) against develop; dependabot-auto-merge.yml runs on each Dependabot PR and — for non-major updates — enables GitHub auto-merge (squash) via gh pr merge --auto. GitHub then performs the merge only once every required status check passes ("Branch Name", which exempts dependabot/** heads, and the aggregated "CI Status"), so nothing lands without the full build/test matrix going green.

• Major bumps stay manual. dependabot/fetch-metadata reports the highest update type in a grouped PR, so any group containing a semver-major update is left open for review. Digest-only bumps (the Docker group) carry no semver level and auto-merge.
• SonarCloud is expected to fail on Dependabot PRs (Dependabot-triggered runs receive Dependabot secrets, not repo secrets such as SONAR_TOKEN). It is not a required check, so it does not hold up the merge.
• The squash commit takes the PR title, which Dependabot already emits in Conventional Commit form (ci:/build: prefixes configured in .github/dependabot.yml).
• Requires the repository setting Allow auto-merge (Settings > General > Pull Requests), which is enabled on this repo.

Docs site (docs.yml)¶

The documentation site is built with Zensical — the Rust-core successor to MkDocs + Material (which entered maintenance mode ahead of its 2026-11-05 EOL); Zensical reads the same mkdocs.yml, including the nav that selects the published subset. It is built from the in-repo Markdown and published to the root of the gh-pages branch, so the docs home is the GitHub Pages landing page.

• Triggers: push to main, and pull requests into develop/main, touching docs/**, overrides/**, mkdocs.yml, scripts/zensical-prune-excluded.py, README.md, or the workflow itself; plus workflow_dispatch. A pull request runs the strict build only — the Publish step is skipped for it — so a broken cross-link fails the PR that introduced it. (Until this was added, --strict first ran at the develop → main promotion, which is how a bad link reached a release PR before anything could reject it.)
• Build: pip install "zensical==<pinned>" (pinned in the workflow — an unpinned install let a Zensical release change link-resolution behaviour between the PR that last touched the docs and the promotion that published them; bump it deliberately and run zensical build --strict locally), then zensical build --strict — broken internal links/anchors and other warnings fail the build instead of shipping a broken page. theme.variant: classic in mkdocs.yml keeps the Material appearance the brand skin (overrides/, docs/stylesheets/aqualink.css) is built for; Zensical's own default is a different, newer theme.
• exclude_docs and Zensical. As of 0.0.60 Zensical did not honour exclude_docs: it built every page under docs_dir, so design/, refactoring/, alwin32/ and the other entries in mkdocs.yml's exclude_docs block were built as real, unlinked pages, and the Prune docs excluded from the published site step (scripts/zensical-prune-excluded.py) deleted them from site/ afterwards. As of 0.0.70 (the pinned version) Zensical honours exclude_docs natively, so the prune step reports Pruned 0/N and is a defensive no-op; it can be dropped (with the script and its paths entry) once the pin is known to stay on a version with native support.
• Linking to files outside docs/, or into excluded content. --strict resolves every relative link against the documentation tree, not the repository. A link into content held back by exclude_docs (for example design/ha-companion-package.md) fails the build with page does not exist, because the excluded page is not part of the build — this is exactly what failed the first publish after the Zensical migration (#216). A link to a repository file outside docs/ (such as ../docker-compose.yml) likewise cannot resolve on the published site even though it resolves on GitHub. Use an absolute https://github.com/iainchesworthlabs/aqualink-automate/blob/main/… URL for both. With the strict build now running on pull requests, the PR fails instead of the release.
• No social-preview cards. The old social plugin (Open-Graph/Twitter card generation) isn't supported by Zensical yet; it was dropped along with the MkDocs Material dependency rather than kept as dead config. Revisit if Zensical adds native support.
• Publish: peaceiris/actions-gh-pages with keep_files: true, preserving the APT/DNF package tree that publish-repos.yml owns on the same branch (apt/, rpm/, key.gpg, install-*.sh); the filename sets are disjoint, so neither workflow deletes the other's files. Its publish-docs concurrency group is separate from publish-repos, so a docs build and a package publish can run at the same time.

Home Assistant add-on validation (homeassistant-addon.yml)¶

Validates the Home Assistant add-on wrapper — the repo-root repository.yaml plus the stable aqualink-automate/ and generated aqualink-automate-edge/ channels (see homeassistant-addon.md) — whenever it changes: push to main/develop and pull_request into develop/main, path-filtered to the add-on files and their generator/sync scripts.

One validate job (GitHub-hosted, no third-party actions to pin) checks, in order:

1. YAML parses — repository.yaml, both channels' config.yaml, and every translation catalog.
2. The Edge channel is generated, not hand-edited — re-runs scripts/gen-homeassistant-edge-addon.ps1 and fails on any diff, so the two channels can never drift by hand.
3. Options translations cover config.yaml — every locale must document exactly the schema keys (configuration:) and ports (network:), no missing and no extra (mirrors the web UI's i18n key check).
4. run.sh passes shellcheck.
5. Version present and readable — each channel's config.yaml carries a parseable version:, via scripts/sync-homeassistant-addon-version.ps1 -Check. The Dockerfile derives its base-image tag from that version through BUILD_VERSION; there is no build.yaml. Pinning that version to the release version happens in release.yml (-Check -Channel <c> -Version <v>), not here. The same script is the single writer in set mode, so CI and release can never disagree about the format.

The add-on wrapper images are published by release.yml's homeassistant-addon-publish job, not by this workflow.

Home Assistant companion package validation (ha-companion.yml)¶

Validates the Home Assistant companion package — the blueprints, helpers package, and dashboard under homeassistant/companion/ (see homeassistant-companion.md) — on push to main/develop and pull_request into develop/main, path-filtered to the companion tree, scripts/check-ha-companion-entities.ps1, and the workflow itself.

One validate job (GitHub-hosted) checks, in order:

1. YAML lint — yamllint with the relaxed profile and line-length off (Home Assistant YAML carries custom tags like !input and long Jinja lines).
2. Entity references match the manifest — scripts/check-ha-companion-entities.ps1: every active *.aqualink_automate_* reference in companion YAML must exist in entity-manifest.json; install-specific (panel-label) entities may only appear in commented examples or as blueprint inputs. The manifest itself is locked to src/core/mqtt/ha_discovery.cpp by TestSuite_HaDiscovery_CompanionManifest in the unit suite, closing the loop code → manifest → shipped YAML.
3. Real schema validation — assembles a scratch config (the helpers package, every blueprint instantiated by the test-harness/ automations, and the documented history_stats recipes) and runs hass --script check_config in the official Home Assistant container. The step fails on a non-zero exit or any logged ERROR line: check_config exits 0 when a blueprint expands into an invalid automation, so the exit code alone is not a sufficient gate.

The companion bundle (aqualink-automate-homeassistant-companion-<version>.zip) is shipped by release.yml, not by this workflow — see releasing.md.

Self-hosted runners¶

Every Linux and Windows job that can build on the self-hosted fleet reads its runner labels from a shared check-runners job (_check-runners.yml), which live-checks whether a matching self-hosted runner is actually available right now — not a static "self-hosted is configured" switch. macOS always runs on macos-latest (GitHub-hosted); the arm64 Linux row of _build.yml stays on the static RUNNER_LINUX_ARM override (no self-hosted arm64 runner exists in the current fleet to check against).

check-runners emits four label arrays, not two. linux_runner / windows_runner are the ordinary shared-fleet outputs. linux_runner_big / windows_runner_big additionally require the big label — the oversized runners (8 vCPU / 32 GB, 48 GB data disks) in the org-scoped Labs CI Big Runners group, provisioned because the heavy legs OOM-killed and filled the disks on the shared shape. Only CodeScanning_CodeQL (c-cpp) and CodeScanning_MSVCCodeAnalysis use them; routing every heavy leg there measures out slower than leaving them parallel on GitHub-hosted. The Linux side is currently a single machine, so anything that lands two jobs on it (e.g. automated-codescanning.yml's main/develop branch matrix on CodeScanning_CodeQL) serialises; the Windows side currently has two machines, so the equivalent MSVC branch matrix runs in parallel. This is fleet state, not a fixed property of the label — re-check gh api orgs/<org>/actions/runners before relying on either count.

Full design, the fork-PR gating, the two-part live check (repo-level, then optional org-level via ORG_RUNNERS_TOKEN), the big routing arithmetic and its two known sharp edges are documented in ci-self-hosted-runners.md — this section is the short summary.

vcpkg disk hygiene¶

vcpkg keeps deps/vcpkg/buildtrees and deps/vcpkg/packages after every port build, and nothing prunes them — _build.yml's self-hosted Clean workspace step does rm -rf build install and deliberately leaves deps/vcpkg alone. On a persistent runner they therefore grow monotonically (~6.2 GB measured; OpenSSL's buildtree alone ~1.0 GB, boost-test next at 425 MB), until a build dies:

ld.bfd: final link failed: No space left on device            # Linux, at [723/726]
fatal error C1085: Cannot write compiler generated file ...   # MSVC
  : No space left on device

Note the Linux case: all 726 objects compiled and it died at the link — so this is disk, not the memory pressure described above.

The core preset therefore sets:

"VCPKG_INSTALL_OPTIONS": "--clean-buildtrees-after-build;--clean-packages-after-build"

It must be a preset cacheVariable, never a -D argument. VCPKG_INSTALL_OPTIONS is a CMake list, so its elements are ;-separated. Passing -DVCPKG_INSTALL_OPTIONS=a;b through lukka/run-cmake's configurePresetAdditionalArgs fails, because that action invokes cmake via a shell, which splits on ; — macOS/Windows Configure died at exit 127 with /bin/sh: --clean-packages-after-build: command not found. That is exactly how an earlier attempt was reverted. A cacheVariable never reaches a shell, so the ; survives as a list separator — the same channel already carries "CMAKE_CONFIGURATION_TYPES": "Debug;Release" without trouble.

scripts/check-vcpkg-cleanup.ps1 (CI job vcpkg Disk Hygiene, gated by CI Status) enforces all three parts: the option is present on core, every concrete preset still inherits core, and no workflow passes it via -D.

Reclaiming what already accumulated. The cacheVariable only prevents future growth, and the volumes were already full. Every self-hosted Clean workspace step therefore also does:

rm -rf deps/vcpkg/buildtrees deps/vcpkg/packages

deps/vcpkg persists between jobs on a self-hosted runner — that is why the clean step exists at all — so those two directories carry residue from every previous run. Both are pure intermediates: the binary cache lives in $HOME/.cache/vcpkg/archives and is untouched, so ports still restore from cache rather than rebuilding. downloads/ is deliberately not pruned — on self-hosted it is redirected to the size-capped persistent cache, so clearing it would force re-downloads for no disk win.

This is self-healing across the fleet: a job reclaims whichever host it lands on, so all runners get cleaned as work naturally distributes, with no fleet-side operation and no manual intervention.

Still out of scope: a prune on the runner supervisor's recycle would reclaim even on runners this repo never schedules onto. That lives in iainchesworthlabs/ci-runners.

Why SonarCloud is pinned to GitHub-hosted¶

CodeScanning_SonarCloud does not go through check-runners. It exhausts the shared Linux guests during the scan phase, and the failure names its own axis:

##[error]Action failed: ENOSPC: no space left on device, write

Disk, not memory — an earlier reading of this attributed it to the CFamily analyzer's memory footprint scaling with changeset size. That was wrong; the log is explicit.

The controlled comparison holds the changeset constant and varies only the runner (same PR, 41 files, same workflow):

Critically, that failing run already contained the vcpkg reclaim, so this is not residual accumulation. Confirmed step-by-step on a 12 GB shared guest:

[success] Clean workspace (self-hosted)   <- the reclaim ran
[success] Configure
[success] Build with build-wrapper        <- completed, reached [724/725]
[failure] Run SonarCloud scan             <- ENOSPC

This is a peak-demand problem, not a growth problem, and that distinction is why more reclaiming cannot fix it. The reclaim runs at the start of a job. SonarCloud's peak disk demand is at the end: build-wrapper's output, compile_commands.json, .scannerwork and the analysis cache all land after the reclaim has already happened. Freeing buildtrees up front does not help a stage that fills the volume later — and that is true of any reclaim-once-up-front approach, not just this one. The prune in #162 stops the cache growing across runs, which is real and still needed; this is a different constraint that the same volume cannot satisfy within one run. The shared guests' 18 GB data volume is shared with the size-capped persistent caches (ccache, vcpkg archives, vcpkg downloads), so the working space actually available to a job is materially less than the raw figure, and less than a hosted runner's dedicated disk. The precise breakdown has not been measured.

Deliberately not routed to linux_runner_big. The reflex is to send it after CodeScanning_CodeQL, but big is one machine per OS that CodeQL already occupies for ~100 min per run; serialising the scan behind it would push Code Scanning's Linux path from ~96 min to ~140+. Hosted already works, costs nothing from the fleet, and leaves big for the one leg that completes nowhere else.

Note this is the opposite conclusion to CodeQL's, from the same kind of evidence — which is the point. CodeQL earns big because it completes nowhere else; SonarCloud does not, because a cheaper shape already works.

What this does not establish: the largest changeset measured successfully anywhere is 41 files (77 min on hosted). A 71-file changeset on hosted is untested. If that fails too, big becomes the answer after all — at which point the serialisation cost above is a known price rather than a reason against.

Build parallelism on the shared Linux fleet¶

Ninja defaults to nproc + 2 and no preset in CMakePresets.json sets jobs, so an 8-vCPU guest builds at -j10. Measured per-TU cost for this codebase is ~1.09 GiB of cc1plus RSS (from an oom-killer dump: 10 concurrent cc1plus holding 10.9 GiB anon RSS, page cache evicted to 17 MB, swap 0). Against the 11.7 GiB usable on a 12 GB shared Linux guest that leaves ~7% headroom, which the heaviest Boost/C++23 translation units consume — the observed failure being a runner shutdown / exit 143 deep into the build.

Note the fleet's RAM:vCPU ratio did not change across the 2026-08-27 right-size (1.5 GB per vCPU before and after). What broke is that Ninja's +2 is additive, so it overshoots by 25% at 8 cores against 6% at 32 — taking effective RAM-per-job from ~1.38 GiB to ~1.17 GiB.

Every Linux build leg therefore pins CMAKE_BUILD_PARALLEL_LEVEL: 6 (_build.yml including its in-container package build, fuzzing.yml, and the coverage builds in automated-codescanning.yml). This is a CI-only cap — CMakePresets.json is untouched, so developer machines keep Ninja's default. The variable is set to the empty string on non-Linux legs, which CMake documents as "use the native tool's default".

CodeScanning_CodeQL is the one leg whose parallelism is conditional, because it must track the runner it actually landed on:

--parallel ${{ contains(needs.check-runners.outputs.linux_runner_big, 'big') && '8' || '4' }}

linux_runner_big falls back to ubuntu-latest whenever the single big Linux runner is unavailable, and on a public repo that is 4 vCPU / 16 GB. CodeQL's extractor wraps every compile and roughly doubles per-TU RSS (~2.2 GiB), so a fixed 8 is ~17.6 GiB — comfortable on the big runner's 32 GB and an OOM on the hosted fallback. Worse, a fixed 8 is higher than the nproc + 2 = 6 Ninja would have chosen there unaided, so hardcoding it made the fallback path strictly worse than leaving it unset. That configuration already killed this job twice on hosted (108 min at object 721/762, and 2h33m). 4 on the fallback is ~8.8 GiB.

CodeScanning_MSVCCodeAnalysis needs no equivalent: it sets no explicit parallelism at all, so it self-corrects to nproc + 2 on whichever runner it lands on.

Inside the Docker build, the cap has to be passed in. Dockerfile's own cmake --build --preset build-linux-gcc never sees a workflow-level environment variable, so it ran at nproc + 2 = 10 on the 12 GB guests and OOM-killed the runner — observed deterministically at object 721/762 (integration_test_scenarios.cpp) on two different hosts, which is what distinguishes it from per-host disk exhaustion. The Dockerfile therefore takes ARG CMAKE_BUILD_PARALLEL_LEVEL="" (empty = the native tool's default, so local builds are unaffected) and every caller that builds the ci stage passes it:

The HA add-on build in release.yml uses a different Dockerfile and does not build the C++ stage, so it needs nothing.

The general rule the conditional encodes: an explicit --parallel must be justified against every runner the job can resolve to, not just the intended one. Where that is awkward, leaving it unset is safer, because Ninja's default already tracks the hardware.

Repository variables¶

Set these under Settings > Variables > Actions.

Both _MODE variables govern the _big outputs too — there is no separate mode for the big pair. Forcing github-hosted sends the heavy legs to GitHub-hosted as well, which is where CodeQL c-cpp has been killed mid-build. | RUNNER_LINUX_ARM | JSON label array, e.g. ["self-hosted","linux","arm64"] | The arm64 Linux row of _build.yml only — unaffected by check-runners. Falls back to the GitHub-hosted ubuntu-24.04-arm (free for public repos). |

RUNNER_LINUX / RUNNER_WINDOWS (the old static label-array variables) are retired — the exact self-hosted labels (self-hosted, Linux, X64, shared / self-hosted, Windows, X64, shared, plus the big variants) are now hardcoded inside _check-runners.yml, since they must match what the ci-runners fleet actually registers with. The shared label exists so an ordinary leg cannot be scheduled onto the big pair — runs-on matches any runner whose labels are a superset and has no NOT operator.

Repository secret¶

ORG_RUNNERS_TOKEN (optional) — an org-scoped PAT or GitHub App token with "Self-hosted runners: read" on iainchesworthlabs, letting check-runners see the org-level fleet in iainchesworthlabs/ci-runners rather than just runners registered directly against this repo. Unset, that half of the check is simply skipped.

Self-hosted jobs also run extra steps the hosted jobs skip — they clean the workspace, point vcpkg at a persistent ~/.cache/vcpkg, and (on Windows) load the MSVC environment with the local ./.github/actions/setup-msvc-env composite action (pure PowerShell; sources vcvars64.bat and exports the env delta to $GITHUB_ENV).

Provisioning¶

Runner VM images (Packer templates, provisioning scripts) live in iainchesworthlabs/ci-runners, shared across every repo in the org — not in this repo's cicd/.

The Linux image is hardened against OS-disk creep: Docker's data-root and containerd's root live on the dedicated /data disk, ~/.cache and ~/.sonar are symlinked into the size-capped /data/cache, unattended-upgrades is removed (background dpkg runs also raced CI's own apt-get calls), and the ephemeral supervisor's per-job reset additionally deletes superseded self-updated runner-agent versions and runs apt hygiene (apt-get clean, wipe /var/lib/apt/lists). See cicd/packer/README.md ("Disk layout & the pristine ephemeral model").

Caching¶

vcpkg binary cache¶

The setup-vcpkg-cache composite action restores the vcpkg binary cache. It is keyed by runner OS, not by job:

vcpkg-${{ runner.os }}-<vcpkg-commit-hash>-<hashFiles(vcpkg.json, vcpkg-configuration.json, triplets, overlays)>

Because OS maps 1:1 to the triplet in use, jobs that share a triplet — build-and-test, CodeQL, and SonarCloud all use x64-linux-gcc — share one cache instead of keeping four disjoint copies. The action still accepts a cache-key-prefix input for backward compatibility, but it is deprecated and ignored: it no longer affects the binary cache key.

Self-hosted runners use a persistent on-disk ~/.cache/vcpkg/archives instead of the actions/cache restore, set up by the inline "Setup vcpkg cache (self-hosted)" steps.

Compiler cache (ccache)¶

ccache is used inside the Docker builds (the in-Dockerfile cache mounts a docker buildx retry reuses) and is pre-installed on the self-hosted runner images (5 GB, compression enabled — see cicd/packer/README.md). The host-side CMake build steps in _build.yml rely on the vcpkg binary cache rather than a separate Actions-cached ccache.

setup-cpp-toolchain behavior¶

The toolchain action installs only what each platform needs:

• Linux: installs the named compiler (gcc-15 in CI) plus CMake and Ninja via aminya/setup-cpp.
• Windows: installs only CMake and Ninja; MSVC is assumed present (baked into the runner image).
• macOS: brew install llvm cmake ninja and prepends the Homebrew llvm to PATH, so the macOS row targets Clang 21 from Homebrew.

On self-hosted runners this action is skipped entirely — the toolchain is provisioned into the VM image.