CI/CD Pipeline

How the Pipeline Fits Together

Steno uses GitHub Actions to build and test contributions, then prepare approved app releases. Continuous integration (CI) checks each change. Continuous delivery (CD) takes a tested source revision through signing, notarization, and release review.

For contributors, the path is pull request → checks → review → merge. For maintainers, it is accepted source and tag → checks → signing and notarization → draft review → publication. Merging a pull request does not publish an app release.

The workflow files define the jobs. GitHub settings control required merge checks, release approvals, and access to signing credentials. Maintainers must configure and verify those protections separately. This page covers the app pipeline; the website deploys separately to Vercel.

Workflows and Triggers

The workflow definitions live in the app repository's .github/workflows directory.

A newer push to the same pull request cancels its obsolete CI and Security runs. Runs on main, scheduled scans, manual dispatches, and release calls are never canceled by a newer run; each completes and reports for its own commit. Release, Publish release, and signing recovery share a lock that does not cancel an in-progress release. Release rehearsal has its own concurrency group and cannot replace a pending release. There are no path filters that leave a documentation-only contribution waiting indefinitely for a required check.

A feature-branch push updates its pull request checks without starting a duplicate branch run. Before opening a pull request, use manual CI dispatch if you need a hosted preview. Main keeps its post-merge validation.

Hosted recognition tests use a thread count measured on the hosted runner, which cut CI validation from about 70 minutes to about 13 with identical recognized text.

What CI Checks

Validation jobs run independently so contributors can see failures in more than one area during the same run. The table describes full validation; documentation-only changes use the narrower selection below.

Within the runtime and distribution job, the native build, short native checks, retained-engine checks, benchmark, and preview DMG run first. The preview is uploaded before the 26-case protocol matrix and prompt-scoring suites start, so it can appear before those suites have passed. CI Gate still fails unless every step of the job succeeds.

CI Gate requires the policy job to succeed and checks the other jobs against the selected scope. Failed, cancelled, or unexpectedly skipped jobs cannot satisfy it. Coverage artifacts support inspection; the pipeline does not use an arbitrary coverage-percentage gate.

Documentation-only changes. Only a pull request can select the shorter path, and only when every changed file in the complete Git diff is a recognized root documentation file, Markdown under docs/, or a supported documentation image. Both paths of a rename must qualify, and symbolic links do not qualify. Unknown paths, missing comparison commits, empty diffs, and changes to workflows, scripts, or app source require full validation. Classification uses the complete diff, so a GitHub file-list limit cannot hide a changed source file.

Documentation-only runs keep workflow policy and automation tests, local Markdown file-target checks, artifact upload/download compatibility, CodeQL Actions analysis, and dependency review on pull requests. They skip package and hosted tests, native runtime inference and preview packaging, and Swift/C++ CodeQL. The aggregate gates accept those skips only after successful documentation-only classification. Both CI and Security still report their required checks.

The Markdown check covers local inline file targets. It does not check remote URLs, heading anchors, prose, or command correctness; reviewers still check those. Every other event uses full validation: pushes to main, merge groups, release calls, manual dispatches, and weekly security scans. A documentation-only commit on main, such as a release's changelog commit, therefore still gets the complete tests and a preview. This selection does not reduce runtime timeouts or reuse results from a different source revision.

The native builder preserves the pinned upstream checkout and applies the reviewed corrections to a separate source directory. A manifest records the upstream revision, patch hash, and file hashes. Every reuse verifies that source; a mismatch requires investigation and a fresh build directory. Allocation-failure tests run in separate executables, so fault injection is absent from the shipped helper.

Runtime tests check cancellation during computation and safe backend reuse afterward. CPU cancellation is checked between compute nodes; Metal retains its command-buffer boundaries. VAD worker tests cover the hardware-count limit: up to four workers, with one when the count is unavailable. These checks do not promise immediate GPU interruption or a particular dictation speed.

The native lane drives the Swift speech-engine client against the real helper, including streaming the public sample through the live-transcription client to one final transcript, and fails if the command-line fallback answers instead of the helper. Structural tests check how both required gates are wired and that security analysis uses the same pinned whisper.cpp revision as the runtime lock.

Package regressions cover editor-setup ownership during cancellation, helper pipe reads that leave Swift's cooperative executor available, and complete output collection when a child process exits. Distribution checks reject private build paths while allowing legitimate product names regardless of checkout capitalization.

The runtime build disables automatic backend-plugin discovery and uses its linked CPU and Metal backends. A regression checks that automatic entry points cannot load a test plugin, while an explicit-load control still works. This change does not remove the library's explicit loading API. An artifact upload/download check also verifies file layout and checksums.

What Security Checks

CodeQL analyzes GitHub Actions and, for full-validation runs, Swift and the native C++ build. Verified documentation-only diffs skip the native scans while retaining Actions analysis. The scans that run use full-source queries, including findings outside changed lines. Workflow policy rejects missing or overridden full-scan configuration.

By default, the gate reads the machine-readable scan results (SARIF) and blocks security findings rated 7.0 or higher, plus non-security findings marked as errors. Missing or invalid scan evidence fails the gate. The severity check includes existing and suppressed findings; lower-severity results remain available for review.

The C++ gate has explicit reviews for four local-file findings: the CLI response file and the helper's audio, VAD-model, and transcription-model reads. They remain severity-7.5 findings. Review classified them as not actionable under the inspected launch contracts, where the app selects local inputs and communicates with its own helper through anonymous pipes. This does not establish parser safety or cover every signed-app permission combination; changes to the launcher, transport, privileges, or path sources require another review.

Each review binds the finding's rule, severity, location, message, full trace, and relevant source hashes. The gate prints the original severity and rationale and leaves the scanner report and GitHub alerts unchanged. Missing evidence, changed source or traces, duplicate matches, and unmatched high-severity findings fail. Review hashes come from the retained original SARIF; GitHub's API export omits some trace fields. Reviews are never refreshed automatically. Swift and Actions retain the ordinary severity gate.

Native analysis builds target arm64 explicitly. Blocking findings include source locations and scanner messages in the job log. Read those results even when GitHub's summary shows no new alerts; an empty uploaded alert list does not override a failed severity gate.

On pull requests, dependency review blocks newly introduced dependencies with high or critical vulnerabilities. Security Gate requires every applicable scan and severity check to succeed. Dependabot proposes updates; it does not approve or merge them.

Contributor jobs run on disposable GitHub-hosted runners with read-only repository-content access and no Apple signing credentials. CodeQL separately receives permission to upload security results. Actions are pinned to full commit SHAs, checkout removes persisted Git credentials, and downloaded tools and model assets are verified against expected hashes. The workflows do not use self-hosted PR runners or privileged pull_request_target execution.

Review the native runtime and model files separately; ordinary Swift dependency advisories do not cover them. Static analysis cannot prove the absence of vulnerabilities. Code review, runtime tests, secret protection, and records of where release artifacts came from address different risks.

Reading Results and Artifacts

After opening a pull request, inspect its Checks tab. Contributions from outside the repository may need a maintainer to approve the workflow run first. The contribution still needs review before it can be merged or released.

Start with the first failing step in each failed job. The workflow publishes evidence that helps reproduce the failure:

• Package and app build logs, hosted-test .xcresult bundles, coverage JSON, and synthetic interface renders.
• Runtime protocol results and benchmark receipts.
• An unsigned-preview-arm64-<SHA> artifact containing Steno-<version>-<short-SHA>-preview.dmg when packaging succeeds.

Test and runtime evidence is retained for 14 days; preview DMGs are retained for seven days. Download the evidence you need before it expires.

Preview DMGs are ad-hoc signed. They have no Developer ID identity or notarization and may be blocked by Gatekeeper. A preview from a pull request contains that contribution's code; only run contributions you have reviewed. A downloadable preview is not an official release.

The runtime and distribution job has a 75-minute overall limit for native tests, the public benchmark, app packaging, and DMG verification. Individual test deadlines and gate requirements remain unchanged.

Protocol receipts record the backend, inference deadlines, and network-observation coverage. Explicit CPU tests allow 180 seconds per inference response; default and Metal tests allow 30 seconds. Model loading, acknowledgements, cancellation, and shutdown have separate deadlines. These are test limits, not app latency claims.

Network observation requires two complete startup scans. Inspection errors, timeouts, and network descriptors fail the check. Intentional-crash cases mark the termination boundary and confirm the owned helper was killed and reaped; an incomplete final scan never counts as completed observation. If the protocol suite fails, bounded ASR, VAD, and stream diagnostics help identify the cause while preserving the original failure.

For orderly shutdown, observation stops after the final operational response and before the valid shutdown request. Invalid shutdown requests remain observed. The receipt does not establish network behavior during the valid shutdown request, its acknowledgement, or native teardown. Protocol correlation, acknowledgement, and exit checks still run with their existing deadlines.

Reproducing Checks Locally

Before pushing app or package changes, run scripts/check.sh from the app repository. It regenerates the project with the pinned XcodeGen, confirms every hosted test file is in it, runs the package tests, builds the app, and runs the hosted tests, then fails if a test helper process it started is still running. Use scripts/check.sh --clean after a shared struct or enum changes shape. It does not run the native runtime suites, the benchmark, or packaging.

To run the steps individually, use an Apple silicon Mac with Xcode 26.3, Python 3.11 or later, CMake, and Git. Run these commands from the app repository, using the scripts from the source revision you are testing. The Development Setup page covers the app and native runtime prerequisites.

Generate the project with the pinned tool, then run policy and workflow checks:

bash scripts/ci/tools.sh xcodegen
build/ci-tools/xcodegen/xcodegen/bin/xcodegen generate

python3 scripts/ci/check-policy.py
python3 -m unittest discover -s scripts/ci/tests -p 'test_*.py' -v
bash scripts/ci/tools.sh actionlint
build/ci-tools/actionlint/actionlint -shellcheck= -pyflakes=

Run the package and app tests:

swift test --package-path StenoKit
xcodebuild build -project Steno.xcodeproj -scheme Steno \
  -destination 'platform=macOS' CODE_SIGNING_ALLOWED=NO
xcodebuild test -project Steno.xcodeproj -scheme Steno \
  -configuration Debug -destination 'platform=macOS' \
  -derivedDataPath "$(mktemp -d /private/tmp/steno-hosted-tests.XXXXXX)" \
  CODE_SIGNING_ALLOWED=NO

The temporary DerivedData directory keeps test builds outside protected Desktop folders, where the test host can have trouble loading libraries. It does not change macOS permissions. project.yml remains the project configuration source; do not commit the generated Xcode project.

Native verification downloads roughly 489 MB of pinned model assets, plus upstream source. Use fresh output directories; the scripts refuse to overwrite evidence or repair a mismatched checkout:

bash scripts/ci/prepare-runtime.sh --root "$PWD/build/runtime-sources"
bash scripts/ci/runtime-checks.sh --backend cpu \
  --root "$PWD/build/runtime-sources" --output "$PWD/build/runtime-checks-cpu"

GGML_METAL_DEVICES=0 \
STENO_WHISPER_ROOT="$PWD/build/runtime-sources" \
STENO_WHISPER_BUILD_DIR="$PWD/build/runtime-checks-cpu/build-steno" \
STENO_CI_BENCHMARK_OUTPUT="$PWD/build/benchmark-cpu" \
  bash scripts/ci/benchmark.sh

runtime-checks.sh runs every check by default. --stage fast builds the runtime and runs the short checks into a new output directory; --stage slow then runs the protocol matrix and prompt scoring against that same directory. For native GPU verification, use --backend metal with a new runtime-check output directory. That mode removes inherited GPU suppression and requires observed Metal execution. To validate existing runtime dependencies without downloading or modifying them, run prepare-runtime.sh --verify-only --root <path>.

These commands run checks on your Mac. Check the hosted run and repository settings separately to confirm that GitHub ran and enforced them.

What Still Needs App Testing

The hosted runtime lane uses CPU inference with GGML_METAL_DEVICES=0. The shipped helper is compiled with Metal support, but a CPU receipt does not verify production GPU behavior. Running on Apple silicon does not by itself prove the GPU was used.

The public JFK benchmark is a single-fixture smoke regression. It checks recognition and cleanup on that recording; it is not a general accuracy score or a comparison against the previous release. Broader recognition evaluation remains a separate requirement for relevant changes.

Before release, test the accepted source in the app: microphone capture, text insertion into real editors, permission handling, supported media apps, native Metal inference, and macOS 13 compatibility. Unit tests, synthetic renders, and a successful build do not replace those checks. Record the result against the exact source revision using the native-app acceptance list in the 1.0 release checklist.

Preparing GitHub for Releases

The GitHub activation guide contains the repository setup procedure. Complete it before relying on merge and release gates.

Require the observed CI Gate and Security Gate checks from GitHub Actions on main. Confirm the actual check names after a hosted run, including any reusable-workflow prefix. Protect release tags, preserve review requirements, keep workflow tokens read-only by default, and verify that a failing check prevents a merge.

Keep review policy separate from required checks. A sole maintainer may need a review exception for their own pull request because GitHub does not allow self-approval. That exception must leave CI and Security enforced. Require independent review when another trusted maintainer is available.

Configure three protected environments. Restrict each to main and require approval from a release reviewer:

• release unlocks signing and notarization. Its enablement variable is RELEASE_ENABLED=true; Apple credentials belong only in this environment.
• release-draft permits draft creation after artifact verification. Its variable is RELEASE_DRAFT_ENABLED=true; it needs no Apple credentials.
• release-publish permits publication of the reviewed draft. Its variable is RELEASE_PUBLISH_ENABLED=true; it needs no Apple credentials.

Set the enablement variables after verifying reviewers and branch restrictions. Creating an environment with the right name does not enforce approval. A sole maintainer must be able to approve their own deployment request. When another trusted maintainer is available, use independent release approval.

Signing requires a Developer ID Application certificate and private key, plus an App Store Connect notary API key. Enter credentials through GitHub's trusted environment-secret UI using the exact names in the activation guide. Do not put them in source, logs, or repository-wide contributor secrets.

From Accepted Source to Download

Before dispatching a release, complete the source and native-app acceptance checks, integrate the intended source into main, finalize release notes and version metadata, and create the approved vX.Y.Z tag. The workflow does not create tags or bump versions automatically. The 1.0.1 release procedure lists the ordering rules no workflow enforces, including an unchanged main between merge and dispatch.

A version tag is permanent: a repository rule prevents moving or deleting it. Before creating one, check the hand-edited release facts at the intended commit with python3 scripts/ci/release-guard.py plan X.Y.Z. The check needs no credentials. It compares project.yml, CHANGELOG.md, and the distribution entitlements at the current commit with the latest published release, and requires the requested version, a larger build number, one dated changelog heading with no leftover [Unreleased] entries, a tag that is absent or already at the current commit, unchanged bundle identifiers, and unchanged entitlements unless a reviewed change is acknowledged.

Release rehearsal. Actions → Release rehearsal → Run workflow on main exercises signing before the tag exists. It runs the plan check, then waits at the same protected release environment as a real release. Mode signing checks the signing and notary credentials in about a minute without building anything. Mode full also builds, signs, and notarizes the app once, staples and verifies Gatekeeper, then deletes the signed files on the runner and uploads only a notary receipt and a hash summary. Its permissions are read-only and it cannot create an attestation, so it cannot tag, draft, or publish. It cannot exercise draft creation, attestation verification against release.yml, publication, or acceptance testing of the final signed download; those first run in the real release.

Open Actions → Release → Run workflow, select main, and supply the stable version and full 40-character source SHA. The version must match project.yml and the existing tag; the source must match main at dispatch. Confirm manual acceptance only after completing the checks for that source. Leave publish_release false to stop at a draft.

The workflow proceeds through these stages:

1. Validate the source. Check the tag, version, source, and remote main ancestry, then run the shared validation and security workflows.
2. Approve signing. Wait for the protected release environment. Recheck the source after approval and build a fresh distribution.
3. Sign and notarize. Import the Developer ID certificate into a temporary keychain, sign the distribution, submit it once to Apple, and wait for Accepted. Staple the notarization ticket and verify Gatekeeper. Remove temporary signing material afterward.
4. Record the artifact's origin. Generate SHA-256 checksums and a manifest from the final stapled DMG, then create an attestation linking it to the build workflow. Only approved distribution artifacts leave the signing job; credentials and notary diagnostic logs are excluded.
5. Create a verified draft. Wait at release-draft, verify provenance and the remote source/tag, then create one draft with the verified assets. Existing releases and drafts are not overwritten.
6. Review and publish. If publication was requested, wait at release-publish. Review the actual DMG, complete installation/distribution acceptance, and replace placeholder notes with approved release notes before approving. The workflow does not assess note completeness or perform those manual tests.

Draft creation discovers the unpublished release through authenticated, paginated release listings. It requires exactly one matching draft, verifies its numeric ID, source, and assets, then refetches that ID. Missing or ambiguous results stop without creating another draft. Publication uses the retained ID instead of looking up an unpublished draft through the public tag endpoint. This repair applies to draft creation after 1.0; it does not rerun or change the historical 1.0 recovery.

Steno 1.0.1 went through this standard Release workflow. Run 37050792045 validated tag v1.0.1 at commit 184d261c8bea5213b5fcc58c7492568fc91a3f5b, signed and notarized the installer, created the draft, and published it in the same run. The 1.0.1 release did not use the signing recovery. Its manual acceptance was completed for media pausing and insertion on macOS 26; microphone disconnection, fresh-account permissions, and installation on macOS 13 to 15 were not checked by hand.

Before publication, the workflow rechecks the exact draft ID, source, tag, asset digests, and version progression. It rejects prereleases and versions that do not advance beyond every published stable version. After publishing, it verifies that both the release and GitHub's latest-release pointer refer to the same expected release and assets.

For an ordinary Release draft-only run, finish its review and use Actions → Publish release → Run workflow with the version, source SHA, exact numeric draft ID, and publication acceptance. This reruns validation and security, waits for approval, and verifies the existing assets and their original attestation. It does not rebuild the DMG or resubmit it to Apple. The requested source must still equal main at this new dispatch; reconcile an older draft if main has advanced.

1.0 signing recovery. The recovery procedure preserves the existing v1.0.0 app source while running reviewed workflow controls from a separate commit on main. It verifies and reuses the successful app, native-runtime, distribution-preview, and security jobs from the original release run. It checks the reviewed signing failures and the absence of prior notarization or release evidence before signing. The recovery controls have separate focused tests.

Signing, draft creation, and publication retain separate protected environments. Apple credentials remain in the signing environment. The final DMG and its manifest are both attested. Attestations identify the workflow revision; the verified manifest separately records the app revision, tag, DMG checksum, producing workflow, and reused validation jobs. Check both revisions against the release evidence.

Request publication at recovery dispatch if the same run should continue to publication review. Review the installer, draft, and both attestations before approval. The ordinary Publish release workflow cannot promote a recovery draft unchanged because it expects a different relationship between app and workflow revisions.

For the published 1.0.0 release, recovery signed and notarized the installer and created the draft. The draft job then failed when its public tag lookup could not find the unpublished draft. The existing draft and its assets were independently reverified and published once by release ID through the reviewed publication guard. The workflow did not complete successfully end to end, and its protected publication job did not publish the release. No second installer build or Apple submission was used.

Live capture, transcription, and teardown were observed with the bundled helper. The documented manual acceptance remains incomplete for live media interruption, independently verified insertion with that helper, VoiceOver, and macOS 13 installation. Publication does not mark those checks as passed; see the acceptance checklist.

The latest stable download follows each newly published stable release and now points to Steno 1.0.1. Publishing a release does not update installed copies of Steno; users replace the app manually.

Failure Recovery and Maintenance

Check what completed before retrying a failed operation:

• Build, test, or security failure: fix the cause and rerun validation. Missing or failed evidence must not become a passing gate.
• Dependency hash mismatch: inspect the expected pin and upstream artifact. Do not accept a replacement hash just because a download returned it.
• Interrupted notarization: inspect the notary-recovery-<run-id>-<attempt> artifact and query the existing submission with notarytool info or log. If no submission ID was recorded after a network failure, treat the outcome as unknown before considering another upload.
• Interrupted draft creation or publication: look up the exact release ID and compare its tag, source, and asset digests. Check whether the first operation succeeded before retrying.
• A defective public release: prepare a new patch version. Preserve the existing release tag and published bytes.

Recovery rejects automatic reruns. A previous recovery that reached signing blocks another dispatch until its outcome has been investigated and a separate recovery reviewed. Query any existing Apple submission and preserve any existing draft.

For a DMG produced by the ordinary Release workflow, such as Steno 1.0.1, use GitHub CLI to verify its build attestation. For another version, replace the file name with the one you downloaded:

gh attestation verify Steno-1.0.1.dmg --repo Ankit-Cherian/steno \
  --signer-workflow Ankit-Cherian/steno/.github/workflows/release.yml

For Steno 1.0.0, verify both downloaded files against the recovery workflow:

gh attestation verify Steno-1.0.0.dmg --repo Ankit-Cherian/steno \
  --signer-workflow Ankit-Cherian/steno/.github/workflows/recover-release.yml
gh attestation verify release-manifest.json --repo Ankit-Cherian/steno \
  --signer-workflow Ankit-Cherian/steno/.github/workflows/recover-release.yml

Compare the verified manifest's app revision, tag, final DMG checksum, producing workflow revision, and reused validation records with the accepted release evidence. Verifying the workflow identity alone does not establish which app source the installer contains.

Review Dependabot's monthly Action and Swift updates: minor and patch updates arrive grouped, and each major update arrives as its own pull request. Keep the tool, Xcode, runtime, and model pins current. If a runner image drops the pinned Xcode version, review and test a toolchain update instead of silently selecting another version. Adjust timeouts based on actual workflow runs.

For the full maintainer procedure, use the app repository's CI/CD operating guide, GitHub activation guide, 1.0 release checklist, and 1.0.1 release procedure from the source revision you are releasing.

The CI activation corrections record the source changes and regression evidence behind the runtime, process-lifecycle, and packaging checks, and the thread-count measurement that shortened hosted validation.