CI

Pull requests run workflows/ci.yml on the self-hosted general runner pool. PRs also run the desktop smoke test and native UI tests when they touch the files listed in those workflows' paths: filters. Pushes to main run workflows/main-pipeline.yml, whose verification, stage deployments, integration tests, browser tests, desktop smoke test, ARM native compilation, advisory x86 UI tests, and mobile OTA release use persistent self-hosted runners. An explicit [client-release] main commit also releases every changed client from that exact SHA after both production suites pass. Native builds finish before OTA publication; ordinary commits retain the weekly release schedule.

Both verification jobs run turbo run lint check-types test build and validate workflow YAML.

Beta and production test jobs explicitly require a successful deployment and use !cancelled() so GitHub does not suppress them when an earlier optional chat validation was skipped. Requested chat validation must succeed before promoting the next stage; only genuinely optional chat jobs may be skipped.

Verify Cloud Rollout Completion requires production deployment and both production test suites to succeed whenever the cloud rollout was selected. It also requires chat activation when requested. Unexpected skips fail this final check; all client releases wait for it. A green cloud pipeline therefore cannot stop after beta and still publish clients. Client-only runs with the cloud rollout deliberately disabled keep this check skipped.

Runner Pools

Deep-thought has three runner services:

General CI, deployment, browser, desktop, and release jobs use the general pool. Native Android UI tests require [self-hosted, Linux, X64, android], reserving that runner for the emulator. Only the Android runner should have the android label, and it must not have the general label. Each service needs its own installation and workspace.

All services run as fb. The general services have systemd drop-ins that set HOME and XDG_CACHE_HOME to their separate persistent directories. This isolates concurrent tool installation and user-level configuration while retaining each service's caches. The Android service retains its Android SDK, Gradle, Maestro, and AVD directories. Its emulator uses port 5556, so a single Android runner prevents simultaneous native jobs from competing for that port.

The existing workflow dependencies still enforce dev → beta → prod promotion. The main pipeline remains serialized across pushes; independent jobs within a pipeline can now use separate runners.

Caching on self-hosted runners

The runner account's home directory must persist between jobs. npm and Playwright use their existing directories there; their caches are not restored from or uploaded to GitHub on self-hosted jobs.

• npm: npm ci --prefer-offline uses the runner's npm download cache (normally ~/.npm on Linux), fetching missing packages as needed. It still installs a clean dependency tree from the lockfile for every job. actions/setup-node has package-manager caching explicitly disabled on self-hosted jobs, including the automatic caching enabled by packageManager in the root package.json. Node itself still uses the runner's tool cache.
• Playwright: browser binaries remain in ~/.cache/ms-playwright on Linux. Every self-hosted E2E job runs playwright install chromium to ensure the current package's required browser revision exists. Existing revisions are reused. System dependency installation runs on both warm and cold caches.
• Turbo: verification, deployment, and desktop smoke jobs set TURBO_CACHE_DIR to $HOME/.cache/$GITHUB_REPOSITORY/turbo/$RUNNER_OS-$RUNNER_ARCH/$(node --version). This shares task outputs between jobs and runs on the same host, with separate directories per repository, OS, architecture, and Node version. Keeping the cache outside the checkout lets actions/checkout continue cleaning the workspace without deleting cached task outputs.

No runner provisioning change is needed when the existing home directory persists. A new runner or empty cache warms up automatically. If the runner is containerized, these directories must live on a persistent volume to retain the benefit. Monitor their disk usage; old Turbo version directories can be removed when no jobs are using them, at the cost of rebuilding on the next run.

No job uses GitHub's Actions cache, hosted or self-hosted, because artifacts and caches share the account's small Actions storage quota (#598, #679). Hosted desktop builds rely on the Turborepo remote cache instead. Artifact uploads are failure reports kept for 1–2 days, and dry-run desktop installers for one day. The Microsoft Store APPX and Windows review screenshots/validation reports go to the existing private, seven-day S3 submission prefix instead (see apps/desktop/README.md). npm run lint:workflows fails on an upload without retention-days of 7 or less, on actions/cache, and on a setup-* cache. Every Monday, Client Release's Report Actions Storage job lists this repository's live artifacts and caches in its summary. It warns above 300 MB (see scripts/actions-storage-report.mjs).

Turborepo Remote Cache

Turbo also uses Vercel's remote cache when TURBO_TOKEN and TURBO_TEAM are configured. This shares task outputs with hosted PR jobs and developers, while the persistent self-hosted cache allows reuse directly from disk. There is no actions/cache archive of .turbo.

Required GitHub configuration

Without remote cache configuration, Turbo uses its local cache. Self-hosted jobs still share it across runs; a fresh hosted runner has no previous local task cache.

Linking a local checkout

To get the same speedup locally:

npx turbo login          # opens a browser to vercel.com
npx turbo link           # picks the team/project; writes .turbo/config.json

.turbo/ is gitignored, so each contributor runs this once on their machine.

Cache invalidation

Turbo hashes task inputs, declared environment variables, dependency state, and task configuration to decide which outputs can be reused. Moving its cache directory does not change those rules; tasks with cache: false, including OpenAPI/client generation and live tests, still execute normally.

To check the performance change, compare completed run step timings for Node setup, dependency installation, Playwright installation, and Turbo tasks. Self-hosted Node setup should no longer log npm cache downloads, and the Playwright cache action should be skipped. Actual savings depend on cache warmth and changes in dependencies or build inputs.

Native UI tests

workflows/maestro-native.yml requires current-source ARM native compilation with Expo's default Hermes V1 on [self-hosted, Linux, X64, android]. It builds both armeabi-v7a and arm64-v8a, then checks the APK has exactly those native ABIs and both React Native and Hermes libraries for each. Main Pipeline requires this job to succeed.

The same Intel runner also runs the Maestro launch/navigation/permissions and screenshot flows on an x86_64 emulator as an advisory job for PRs/manual runs. Main Pipeline skips this unsupported-architecture runtime job. The known Hermes/GWP-ASan x86 startup issue (#713) must not force an engine downgrade or block ARM releases. This job can fail while the workflow succeeds; its result is not ARM device acceptance. Maestro E2E provides a manual entry point. No ARM runner is registered, so ARM emulator/device UI acceptance is currently a separate release checklist item; a cross-build cannot execute ARM code.

CI loads the host's KVM module when needed and grants the runner account read/write access to /dev/kvm using a per-user ACL. This applies immediately to the running service without restarting it or making the device world-writable. The setup requires passwordless sudo, installs acl if missing (waiting up to ten minutes for another package installation to finish), and verifies access with the Android emulator's -accel-check. The host must expose CPU virtualization; containerized runners must mount /dev/kvm. SDK tools and Gradle caches live under ~/.cache and survive checkout cleanup. Maestro stays in ~/.maestro and is only downloaded when the pinned version is missing. CI installs the Android SDK and Java, then creates a disposable emulator. Reports, screenshots and the build log are uploaded even on failure. See apps/mobile/e2e/README.md for local Android and iOS commands. The Linux runner cannot execute Apple's iOS simulator.

Android Build Disk Space

Both Android jobs run scripts/prepare-android-space.mjs before installing workspace dependencies. A build needs at least 20 GiB free on the checkout's filesystem. When space is low, the script removes only the repository's generated Gradle version/transform caches and build-cache-* task outputs. If that is insufficient, it also removes the repository's sibling turbo/ cache in that Android runner's own home namespace. These task outputs are regenerable; the next task run rebuilds them. Other repositories' caches and the separate general-runner home namespaces are outside cleanup's scope. Downloaded Maven modules, Gradle distributions, the Android SDK, AVDs, settings and credentials are retained. The script rejects cache symlinks/misconfigured roots and fails before compilation if cleanup still cannot meet the minimum. The ARM build remains required; disk pressure never bypasses compilation or ABI verification. If cleanup is insufficient, the job log includes filesystem capacity and sizes of the current runner's repository caches, default Gradle cache, Android SDK, AVDs, npm cache and checkout. This read-only report works even when GitHub's artifact quota is full. API startup measurement collection and artifact uploads are also advisory; integration test failures still block stage promotion. An explicit native recovery option (local_android: true, or the main merge markers [client-release] [local-android]) also uses this Android runner to build and sign a production AAB with EAS locally. It verifies the bundle, signature and ARM libraries before waiting for Google Play submission; iOS builds on EAS cloud independently. Both builds still gate OTA and native release tagging. See mobile release recovery.

Beta Native Build Acceptance

The manual EAS Build workflow accepts preview-simulator for beta Android APK and iOS simulator builds with the preview channel. Expo Doctor runs before cloud builds; the workflow waits for both requested platforms and checks the returned build status. See the mobile release guide for archive inspection and the separate device acceptance gate.

Expense Chat Model Investigation

Manually dispatch Expense Chat Validation with the default model_only: true to investigate the live model against the 14 fictional acceptance cases, with the existing USD 0.10 budget and no paid retries. This mode skips deployed API tests, settings changes and serving-cost queries. It prints only fixed field types/counts/lengths and token usage, including usage from responses rejected by the adapter. It retains its report in private S3. It cannot activate production; normal reusable dev/beta release calls still require the complete validation suite and fresh serving evidence. See expense chat.

The live API integration suite now verifies expense create/edit/mention/delete history through the deployed stream worker, actor exclusion, muted-history retention, and removal of device registrations/token claims on account deletion. It uses disposable accounts and groups. Those tests do not prove device display or notification-tap navigation.

Integration Fixture Recovery

The deployed integration suite retries only DNS/connect failures known to occur before a request is sent, with at most three attempts. Established-socket failures, read/header timeouts and HTTP errors remain visible; ambiguous writes are never replayed by this helper. App fixture requests separately retain their existing single retry after an explicit HTTP 429. Developer requests expose 429 responses directly so allowance assertions remain meaningful. Helper safety checks run without AWS under npm test --workspace api-integration-tests.

The developer role-revocation test creates a fresh group per attempt. A test retry cannot inherit an already-approved member or demoted role and obscure the original failure with an unrelated join-request conflict. Production integration and route-coverage gates remain mandatory.

Avoiding Unnecessary AWS Work

Every main push still runs repository verification, desktop smoke and native UI validation. The cloud rollout can be skipped only when all changes since the last successful main pipeline are under apps/mobile/ or apps/desktop/, excluding published Markdown documentation. Unknown/shared inputs, lockfiles, backend, infrastructure, web, documentation (published by ApiDocs) and workflow changes retain the full stage rollout. Missing GitHub history or diff errors also retain the full rollout. Comparing against success rather than the previous commit prevents a failed backend rollout from being forgotten by a later client-only push.

Explicit [client-release], [mobile-ota], [expense-chat-release], and [deploy-cloud] markers force the cloud rollout. Manual Deploy remains available for external configuration changes. Actual cloud deployments retain dev → beta → prod integration/E2E gates. This reduces API calls and DynamoDB fixture/rate-limit writes from redundant client-only deployments without removing request limits.

TAILSPEND_WEB_RELEASE is the latest commit touching conservative browser build inputs, resolved with full Git history. Backend-only/client-only/docs-only commits no longer change web bundle bytes just to stamp the main SHA; New Relic source maps use the same stable release ID. Shared packages, infra/config and lock/build changes still advance it. Normal S3 sync can then reuse identical assets.