# hotlane - complete reference for agents

This is the full operating contract for hotlane. Reading this document once is sufficient to operate every feature without trial and error. Shorter overview: /llms.txt.

## What hotlane is

Validation-first deployment for apps on a single machine you own (Linux/macOS with Docker and git). The deploy unit is a verified running fork: push a git delta, hotlane forks the live container, applies the delta, boots the fork on a private port, runs verify hooks against it, and only then atomically flips traffic. Rollback flips a pointer to a kept previous version. A background "archivist" rebuilds every promoted version from source into a container image (optionally pushed to a registry) and periodically diffs its behavior against live to detect drift; drifted or too-deep fork chains self-heal by rebuilding from the clean image on the next push. hotlane does NOT run on ECS/Fargate/Cloud Run/Kubernetes - it commands the host's Docker daemon and replaces that layer.

## Install

    curl -fsSL https://hotlane.dev/install.sh | sh     # latest release, /usr/local/bin or ~/.local/bin
    brew install StefanIancu/hotlane/hotlane           # Homebrew tap (macOS/Linux)
    npm install -g hotlane                             # fetches the binary at install time
    pip install hotlane                                # fetches the binary on first run, caches ~/.cache/hotlane

One Go binary is both daemon and CLI. Linux/macOS, amd64/arm64.

In GitHub Actions, use the marketplace action instead of a manual install step: `uses: StefanIancu/hotlane-action@v1` with `daemon`/`token` inputs runs `hotlane push` (or any client command via `command:`/`args:`) and surfaces the verify verdict as outputs (`version`, `promoted`, `json`) and a job summary. Checkout needs fetch-depth: 0.

## Concepts and invariants

- Versions are monotonic and NEVER reused. A rejected push still consumes a version number. Container names: hotlane-<app>-v<N>.
- The daemon records its BASELINE COMMIT (git HEAD when it first snapshotted the source). All client diffs are computed FROM that baseline TO the working tree (committed + uncommitted changes). New files must be tracked (`git add -N`) to appear. CI checkouts need full history (fetch-depth: 0) so the baseline commit exists locally.
- The RING keeps `ring` (default 5) superseded versions as stopped containers for instant rollback. Since v0.7.1 pruning retains the most-RECENTLY-LIVE versions (persisted per-app live history), not the highest-numbered: after a rollback the known-good version you just left stays rollbackable even as newer pushes land. Pruned versions lose container and image both.
- HELD FORKS (from `test`) are running-but-not-serving instances: max 3 concurrently, TTL 15m (HOTLANE_HOLD_TTL duration on the daemon). Promote flips traffic to the exact held instance - nothing is rebuilt between tested and live.
- LAYER REBASE: every promoted fork adds a docker layer; past depth 40 (HOTLANE_REBASE_DEPTH) pushes automatically fork from the archivist's clean image, resetting the chain. Also happens when drift is flagged.
- DRIFT: the archivist cold-boots its from-source clean image and compares behavior with live - every verify hook must pass on the cold boot and every `http:` hook path must serve equivalent bodies. Since v0.4.2 the comparison tolerates dynamic content: volatile patterns (ISO-8601/RFC-1123 timestamps, UUIDs, 16-64-char hex ids, 10-19-digit numbers) are masked before diffing, and each instance is sampled twice - a path whose normalized body differs between two requests to the SAME instance is treated as dynamic and compares status code only. Status codes always compare. Drift checks run after each archive build, every 6h, and on demand.
- Pushes/tests are serialized daemon-side PER APP; concurrent calls queue. Clean builds across apps share a 2-slot semaphore.
- TRAFFIC REPLAY (since v0.6): with a `replay:` block the proxy records a rolling IN-MEMORY slice of live traffic (request + the response live served; memory only, dies with the process, max 512 exchanges / 64KB bodies). After verify hooks pass, each push replays the newest `last:` entries against the fork and diffs answers against the recorded ones using the drift normalizer; a path whose live answers differed WITHIN the buffer is self-dynamic and compares status only. Zero extra load on live. Results land in the push/test response as `replay:{replayed,matched,dynamic,mismatched,buffered,budget_hit,ms,mismatches[<=5]}`; the categories are disjoint (replayed = matched + dynamic + mismatched; a self-dynamic entry whose status agrees counts as dynamic only). `mode:gate` turns any mismatch into a 422 rejection (fork destroyed, live untouched); `hotlane test` holds always get the report and are never gated - the caller reads the evidence and decides. Note: an INTENDED user-visible content change will mismatch by definition - that is what report mode is for; gate suits apps whose captured paths should never change silently.
- REPLAY BUFFER LIFECYCLE (since v0.6.1): every traffic flip - promote, held-promote, rollback - RESETS the buffer, because recorded exchanges describe the version that served them and replaying them against a successor false-positives. Consequence: the push immediately after a promote replays nothing (`replay.replayed: 0`) until live traffic refills the buffer; this is correct, not a bug. Drift checks also replay the slice against the COLD BOOT (phase 2): behavioral drift is caught on any endpoint users recently exercised, not just verify-hook paths - detail reads "replayed traffic differs on METHOD /path".
- MULTI-APP (since v0.5): one daemon can serve several apps (`serve -apps DIR`, one *.yml per app). Each app keeps its own pool/ring/held forks/archivist/drift verdict. App traffic routes by Host header matching each config's `domain:`; an unknown Host gets 421, NEVER a fall-through to another app. The X-Hotlane-Fork header resolves only within the Host-matched app, and its token must match. The set of apps is the directory contents at startup - add/remove = edit files + restart (restarts adopt running containers; invisible).

## Configuration: hotlane.yml (one file, in the app repo root)

    app: api                    # required; names containers, ^[a-z0-9][a-z0-9-]*$
    image: node:22-alpine       # required; base image for the first boot
    build: npm run build        # optional; chained before run on every container boot (incremental against warm caches)
    run: node dist/server.js    # required; must listen on `port`
    port: 3000                  # required; container-internal port
    workdir: /app               # optional, default /app
    verify:                     # optional; gate every promote
      - http: /health == 200    # polls until expected status (warming = not-yet, not failed); default budget 15s
        timeout: 5s             # optional per-hook budget override (any hook; needs a unit, e.g. 90s, 2m)
      - run: ./smoke.sh         # exec in the fork's workdir, exit 0 passes; default budget 60s
    ring: 5                     # optional; rollback targets kept
    archive: ghcr.io/acme/api   # optional; archivist pushes clean images as <ref>:vN (docker login AS THE USER THE DAEMON RUNS AS)
    notify: ${HOTLANE_NOTIFY_URL}  # optional; webhook (Slack/Discord compatible)
    src: /srv/api               # multi-app mode: REQUIRED - the checkout to snapshot/diff against
                                # (single-config mode: optional; defaults to the config file's directory)
    domain: api.example.com     # multi-app routing key (Host header) + TLS cert subject; bare hostname only
    replay:                     # optional; traffic-replay verification (since v0.6)
      last: 200                 # newest buffered live requests replayed against each fork; 0/absent = off
      mode: report              # report (annotate the push; default) | gate (mismatch rejects like a failing hook)
      methods: [GET, HEAD]      # captured+replayed methods; default is reads-only ON PURPOSE (fork state is
                                #   isolated, external side effects - payments, email - are NOT)
      exclude: [/metrics]       # path prefixes never captured
      budget: 5s                # replay wall-clock cap per push; partial coverage is reported, never silent

Since v0.4.2, `notify` and `archive` interpolate `${VAR}` from the daemon's environment at load - keep secrets out of committed config. An unset variable fails the load (loud, at startup). ONLY these two fields interpolate: `build`/`run`/verify `run` scripts keep their `${VAR}` literally (those belong to the shell inside the container).

## Daemon

    hotlane serve [-config hotlane.yml | -apps /etc/hotlane/apps] [-addr ADDR] [-proxy :7480] [-token T] [-tls | -tls-domain deploy.example.com]

- Without TLS: API on -addr. Since v0.6.2 the daemon REFUSES to start if -addr binds beyond loopback without a token (the API deploys code; an open one is remote code execution). Since v0.7.1 the -addr default follows the token: 127.0.0.1:7433 without one (bare `hotlane serve` just works, locally and safe), :7433 (all interfaces) once -token/HOTLANE_TOKEN is set - so tokened deployments keep a remote API with no flags. The refusal now only fires on an EXPLICIT beyond-loopback -addr with no token. App traffic on -proxy, both plain HTTP. Multi-app plain mode still routes by Host on -proxy (curl -H "Host: app.domain"). GET / on the API answers with plain-text directions (no auth), useful to confirm which listener you reached.
- With -tls (REQUIRES a token): one shared HTTPS listener on :443 with automatic Let's Encrypt, one certificate per configured `domain:` (TLS-ALPN; only 443 must be open). Each APP owns https://its-domain/ ; the daemon API lives under the /-/ prefix; :80 redirects to https. The private -addr listener keeps working with bare /v1/* path aliases. -tls-domain D is single-app shorthand for `domain: D` + -tls.
- -apps DIR: serve every *.yml in DIR (multi-app). Validation is all-or-nothing at startup: duplicate app names, duplicate domains, or missing src refuse the whole load with every problem listed.
- Auth: every API route except healthz requires `Authorization: Bearer <token>` when a token is set. Tokens compare constant-time.
- systemd: since v0.4.2 no HOME gymnastics needed - when $HOME is unset the daemon stores state under /var/lib/hotlane instead of $HOME/.hotlane (pristine + shadow source, held fork sources, autocert cache, baseline-commit). A ready unit file ships in the repo at packaging/systemd/hotlane.service (StateDirectory=hotlane, EnvironmentFile=/etc/hotlane/env for HOTLANE_TOKEN and any ${VAR} config refs).
- Restarts adopt the newest running version without dropping traffic and keep the archivist's promoted-source snapshot.

## CLI (client commands)

Environment: HOTLANE_DAEMON (default http://127.0.0.1:7433), HOTLANE_TOKEN (bearer), HOTLANE_APP (app name on a multi-app daemon). Note: env files written as KEY=value need `set -a; source file; set +a` to export for the binary. Every command below accepts `-daemon URL` and `-app NAME`; those marked [json] accept `-json` to print the daemon's raw JSON response (exit code still reflects success). App resolution order: -app flag > HOTLANE_APP > the app: named by ./hotlane.yml > bare single-app path. Against a multi-app daemon a client with no app name gets 400 with directions - loud, never the wrong app.

    hotlane init [-force]            # detect app (Node/TS, FastAPI, Flask, Django, Go), write starter hotlane.yml
    hotlane push [-from REF] [json]  # diff -> fork -> verify -> promote. Exit 0 promoted / 1 rejected (with hook verdicts + fork logs)
    hotlane test [-from REF] [json]  # like push but HOLD the verified fork; prints version, TTL, and the poke header
    hotlane promote <n> [json]       # flip traffic to held fork n (verify re-runs as gate; failure leaves live untouched)
    hotlane discard <n> [json]       # destroy held fork n
    hotlane rollback [n] [json]      # flip to previous (or specific) kept version; restarts it if stopped
    hotlane status [json]            # live version, ring, held forks, archive/drift, baseline_commit
    hotlane status -all [json]       # one line per app the daemon serves (multi-app)
    hotlane logs [-n N]              # tail live container output
    hotlane drift [json]             # run a drift check NOW; exit 1 if drifted
    hotlane mcp                      # MCP server over stdio (see below)

## HTTP API

Canonical paths use the /-/ prefix (work on BOTH listeners); bare /v1/* aliases exist on the private port only. All requests: `Authorization: Bearer <token>` when the daemon has a token.

APP SCOPING: on a multi-app daemon every app-scoped route below lives under `/-/v1/apps/<app>/...` (e.g. POST /-/v1/apps/api/push); the unscoped forms return 400 with directions. On a single-app daemon BOTH forms work - the unscoped routes are full aliases, so pre-0.5 clients and CI keep working unchanged. Clients getting 404 on the /apps form (daemon older than 0.5) should retry the unscoped path once.

    GET  /-/v1                 -> {"service":"hotlane","version":...,"docs":...,"routes":[...]}  (self-describing index)
    GET  /-/v1/apps            -> [{app,domain,version,live,drift}, ...]  (apps served by this daemon)
    GET  /-/healthz            -> "ok app=<name> version=<v>" single-app; "ok apps=<n> version=<v>" multi-app (always unauthenticated; never enumerates app names)
    GET  /-/v1/status          -> {app, live, version, backend, baseline_commit, last_fork, ring:[{version,container,status,live}], held:[{version,backend,expires_at}], archive:{image,last_version,building,drift,detail,checked_at}}
    POST /-/v1/push            body: raw git diff (text/x-diff, <=10MB)
                               200 -> {container,version,from_clean?,snapshot_ms,patch_ms,boot_ms,total_ms,verify:[{hook,ok,ms,detail?}],verify_ms,promoted:true,replay?}
                               422 -> same shape, promoted:false, plus logs (the fork's last output). Fork destroyed; live untouched.
                               (replay? appears when a replay: block is configured; a gate-mode mismatch is a 422 with promoted:false and the replay verdict)
    POST /-/v1/test            same body; 200 -> {fork:<push-shape>, held:true, expires_in:"15m0s", header:"X-Hotlane-Fork: <version>-<token>"}  (the token is unguessable; the fork is reachable on the PUBLIC app listener, so the header value IS the credential); 422 like push; 409 if 3 forks already held
    POST /-/v1/promote         {"version":N} -> 200 with the fork result; 422 if unknown/expired or verify fails (live untouched)
    POST /-/v1/discard         {"version":N} -> 204
    POST /-/v1/rollback        {"version":N} or empty -> 200 {container,backend,version,restarted,total_ms}; 422 if no target/not ready
    POST /-/v1/drift-check     -> 200 archive status (drift: "clean"|"drifted"|"unknown", detail on drifted)
    GET  /-/v1/logs?tail=N     -> text (N<=10000, default 100)

Reaching a held fork: send requests to the APP url (https://domain/ or the -proxy port) with the header value returned by `test` - `X-Hotlane-Fork: <version>-<token>` (since v0.6.2; a bare version number does NOT work, and never did anything but leak unreleased code to anyone who could count). Authenticated status also returns it per held fork. Unknown/expired fork -> 421 with an explanatory body - never a silent fall-through to live. On a multi-app daemon the fork resolves within the Host-matched app only; app traffic to an unknown Host is also 421.

## MCP server

`hotlane mcp` speaks MCP (2024-11-05) over stdio: newline-delimited JSON-RPC. Run it from the app's repo (push/test compute the git diff from the working directory); HOTLANE_DAEMON and HOTLANE_TOKEN as usual, plus HOTLANE_APP or `-app NAME` when the daemon serves several apps. Tools (all return the daemon's JSON as text content; isError mirrors HTTP failure): hotlane_status, hotlane_push{from?}, hotlane_test{from?}, hotlane_promote{version}, hotlane_discard{version}, hotlane_rollback{version?}, hotlane_drift, hotlane_logs{tail?}.

Client config example:

    {"mcpServers": {"hotlane": {"command": "hotlane", "args": ["mcp"], "env": {"HOTLANE_DAEMON": "https://deploy.example.com", "HOTLANE_TOKEN": "..."}}}}

## Errors you may see and what they mean

- `push REJECTED` / promoted:false -> your change failed a verify hook; response carries which hook and the fork's logs. Fix and push again. (Failing http hooks poll their full budget - 15s default - before giving up; set per-hook `timeout:` to fail faster.)
- `baseline commit X is not in this clone` -> shallow checkout; fetch full history (fetch-depth: 0 / git fetch --unshallow).
- `applying diff: patch does not apply` -> your diff base doesn't match the daemon's pristine source; use -from or check baseline_commit in status.
- 404 with an app-shaped body on push -> your CLI predates v0.2.0 and hit the app instead of the API; upgrade the CLI.
- 400 `this daemon serves multiple apps` -> name the app: /-/v1/apps/<app>/... paths, or -app / HOTLANE_APP on the CLI.
- promoted:false with replay.mismatched > 0 and all verify hooks ok -> gate-mode replay rejection: the fork answered recorded live traffic differently. If the change was intentional, that mismatch is expected - use mode:report for apps whose captured paths change on purpose.
- 421 `no app for host H` -> the Host header matches no configured domain: - check DNS and the app's `domain:` field.
- 421 `no held fork N` -> the fork expired (TTL), was promoted, or was discarded (multi-app: forks resolve within the Host-matched app only).
- 409 `3 forks already held` -> promote or discard one first.
- `rollback ... not ready` -> the target failed verify hooks on restart; live was left untouched.
- drift `drifted` with a behavior diff -> live no longer matches a from-source rebuild; the NEXT push automatically rebuilds from the clean image and heals it.

## Recommended agent workflow

1. `hotlane status` (or MCP hotlane_status) - learn live version, baseline_commit, drift state.
2. Edit code. `hotlane test` - get a held fork + header.
3. Validate YOUR change against the fork (requests with the X-Hotlane-Fork header). Users are still on live.
4. `hotlane promote <n>` - traffic flips to the exact instance you validated. Or `discard` and iterate.
5. On any mistake after promote: `hotlane rollback` (sub-second).
Straight `hotlane push` collapses steps 2-4 when standing hooks are sufficient validation.

## Facts

- Site: https://hotlane.dev · Repo: https://github.com/StefanIancu/hotlane (MIT) · CI guide: /docs and repo docs/ci.md · Benchmark: repo docs/benchmark.md (median 1.2-1.7s push-to-verified-live on small real apps vs 493s classical baseline; your build+boot time is the floor).
- State on the daemon host: $HOME/.hotlane/<app>/ (pristine, shadow, held/, autocert/, baseline-commit).
- Defaults: API 127.0.0.1:7433 (:7433 all-interfaces once a token is set; since v0.7.1), app proxy :7480, shared TLS :443/:80, ring 5, hold TTL 15m, max held 3 PER APP, rebase depth 40, drift ticker 6h (apps checked sequentially), clean-build semaphore 2.