# Hanalyx Documentation

> Full documentation for OpenWatch, Kensa, and Specter, concatenated as one file for agents. Source: https://www.hanalyx.com/docs

---

# OpenWatch: Quickstart guide
URL: https://www.hanalyx.com/docs/openwatch/quickstart

**Last Updated:** 2026-06-22 · **Applies to:** OpenWatch 0.2.0-rc series (Go single-binary)

Go from a freshly installed package to your first host under automatic
compliance monitoring. This guide assumes OpenWatch is already installed and
running. If it is not, follow
[docs/guides/INSTALLATION.md](/docs/openwatch/installation) first, then
return here.

OpenWatch is a single Go binary (`/usr/bin/openwatch`) that serves both the REST
API and the embedded React UI over HTTPS on port `8443`. It is managed by
`systemd` as `openwatch.service` and stores all state in PostgreSQL. There is no
container runtime, no separate web tier, no Redis, and no Celery. Compliance
checks run over SSH via the embedded Kensa engine.

## Before you begin

- OpenWatch is installed and `openwatch.service` is active.
- You have the `admin` account created during install
  (`openwatch create-admin`).
- You have a Linux host reachable over SSH (TCP/22) from the OpenWatch server.
- You have SSH credentials for that host (username plus either an SSH private
  key or a password).

| Surface | URL |
|---------|-----|
| Web UI and API | `https://<your-host>:8443/` |
| API base path | `https://<your-host>:8443/api/v1` |

The TLS certificate lives under `/etc/openwatch/tls/`. With a self-signed
certificate, pass `-k` to `curl` and accept the browser warning.

## Step 1: Confirm the service is healthy

Check that the service is running:

```bash
systemctl status openwatch
```

Query the anonymous health endpoint:

```bash
curl -sk https://localhost:8443/api/v1/health | jq .
```

A healthy response looks like this:

```json
{
  "status": "healthy",
  "db_connected": true,
  "version": "0.2.0-rc.13"
}
```

`status` is `healthy` or `degraded`; `db_connected` is `false` when the database
is unreachable. If the request fails or `status` is `degraded`, inspect the logs
before continuing:

```bash
journalctl -u openwatch --no-pager -n 50
```

Common causes are an unreachable database (check
`/etc/openwatch/secrets.env` for `OPENWATCH_DATABASE_DSN`) or pending
migrations. To verify the resolved configuration without starting the server:

```bash
openwatch check-config --config /etc/openwatch/openwatch.toml
```

To confirm the schema is current:

```bash
openwatch migrate --config /etc/openwatch/openwatch.toml
```

`migrate` is idempotent; if everything is applied it prints the current version
and exits.

## Step 2: Sign in

Open `https://<your-host>:8443/` in a browser and sign in with the `admin`
username and the password you set during `openwatch create-admin`. You land on
the dashboard.

OpenWatch does not ship a hard-coded default password. The first admin is
created out-of-band by the CLI, so there is no factory `admin`/`admin`
credential to rotate. If you need an additional admin, run
`openwatch create-admin` again with a different `--username`.

The equivalent API call returns an access token, a refresh token, and the
calling identity:

```bash
TOKEN=$(curl -sk -X POST https://localhost:8443/api/v1/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"<your-password>"}' \
  | jq -r '.access_token')
```

Send the token as `Authorization: Bearer $TOKEN` on subsequent requests. The
browser UI uses HttpOnly session cookies instead of the bearer token; the
bearer flow shown here is for scripting.

## Step 3: Add a host

In the UI, go to **Hosts** and add a host. Only `hostname` and `ip_address` are
required; `port` defaults to the SSH port and the rest are optional.

| Field | Required | Notes |
|-------|----------|-------|
| `hostname` | Yes | 1–256 characters |
| `ip_address` | Yes | 1–64 characters |
| `port` | No | 1–65535 |
| `display_name` | No | Up to 256 characters |
| `environment` | No | Up to 64 characters; namespaces the hostname uniqueness check |
| `tags` | No | Array of strings, each up to 64 characters |
| `username` | No | Default SSH username for this host |

The equivalent API call:

```bash
HOST_ID=$(curl -sk -X POST https://localhost:8443/api/v1/hosts \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"hostname":"web-01","ip_address":"192.168.1.10","port":22}' \
  | jq -r '.id')
```

Host IDs are UUIDs. A duplicate `hostname` within the same `environment` returns
`409`.

## Step 4: Add SSH credentials

OpenWatch needs SSH access to reach the host. Add a credential scoped to the
host (or a `system`-scoped default that applies to every host without its own).

| Field | Required | Notes |
|-------|----------|-------|
| `scope` | Yes | `system` or `host` |
| `scope_id` | When `scope=host` | The host UUID |
| `name` | Yes | 1–256 characters |
| `username` | Yes | SSH login user |
| `auth_method` | Yes | `ssh_key`, `password`, or `both` |
| `private_key` | When key-based | PEM private key; stored encrypted |
| `password` | When password-based | Stored encrypted |
| `is_default` | No | Mark a `system` credential as the fleet default |

Secret material (`private_key`, `password`) is encrypted at rest with the
credential data-encryption key loaded at startup
(`[identity].credential_key_file`). It is never returned by the API; read
responses expose only metadata such as the key fingerprint.

The equivalent API call for a host-scoped key credential:

```bash
curl -sk -X POST https://localhost:8443/api/v1/credentials \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"scope\": \"host\",
    \"scope_id\": \"$HOST_ID\",
    \"name\": \"web-01 key\",
    \"username\": \"openwatch\",
    \"auth_method\": \"ssh_key\",
    \"private_key\": \"$(cat ~/.ssh/id_ed25519 | sed ':a;N;$!ba;s/\n/\\n/g')\"
  }"
```

## Step 5: Verify connectivity

Confirm OpenWatch can reach the host over SSH before relying on automatic
checks. In the UI this is the host's connectivity action; via the API:

```bash
curl -sk -X POST "https://localhost:8443/api/v1/hosts/$HOST_ID/connectivity:check" \
  -H "Authorization: Bearer $TOKEN" | jq .
```

A failure here means SSH cannot connect — wrong credentials, an unreachable
address, or a firewall blocking TCP/22. Fix that before expecting compliance
results.

## Step 6: Let automatic compliance checks run

Compliance checking is primarily scheduler-driven: background loops in the
service discover each host's OS, then run Kensa compliance checks over SSH on an
adaptive cadence and write the results into PostgreSQL. The `serve` process runs
these schedulers; the `openwatch worker` subcommand drains the PostgreSQL job
queue (`SKIP LOCKED`) for queued background work.

You do not need to do anything to start the first check beyond adding the host
and working credentials. The first OS-discovery and compliance cycle runs once
the host is due.

You can also trigger an on-demand scan instead of waiting for the schedule: use
the **Run scan** action on the host detail page in the UI, or
`POST /api/v1/hosts/{id}/scans`, which enqueues an HMAC-signed scan job and
returns `202` with the new scan id (the scan itself runs asynchronously on the
worker):

```bash
curl -sk -X POST "https://localhost:8443/api/v1/hosts/$HOST_ID/scans" \
  -H "Authorization: Bearer $TOKEN" | jq .
```

To watch progress:

```bash
journalctl -u openwatch --no-pager -f
```

## Step 7: View compliance results

Once a check has run, the host's compliance roll-up is available on the host
detail view. In the UI, open the host from **Hosts**. Via the API,
`GET /api/v1/hosts/{id}` returns `compliance_summary` (and a `liveness`
sub-object):

```bash
curl -sk "https://localhost:8443/api/v1/hosts/$HOST_ID" \
  -H "Authorization: Bearer $TOKEN" | jq '.compliance_summary'
```

The summary counts are integers:

```json
{
  "passing": 0,
  "failing": 0,
  "skipped": 0,
  "error": 0,
  "total": 0
}
```

All-zero counts mean no compliance check has completed against the host yet —
give the scheduler time, and confirm Step 5 passed.

For a fleet-wide view, the dashboard aggregates across hosts. The backing
endpoints include:

| Endpoint | Returns |
|----------|---------|
| `GET /api/v1/fleet/score` | Fleet-wide passing/total score |
| `GET /api/v1/fleet/liveness` | Host counts by reachability |
| `GET /api/v1/fleet/top-failing-rules` | Rules failing across the fleet |
| `GET /api/v1/fleet/top-failing-hosts` | Hosts with the most failures |
| `GET /api/v1/fleet/recent-changes` | Recent pass/fail transitions |
| `GET /api/v1/intelligence/state/{host_id}` | Per-host intelligence state |

Most fleet endpoints accept an optional `framework` query parameter to scope the
counts to a single framework key.

## Step 8: Next steps

| Task | Where |
|------|-------|
| Full install and configuration reference | [docs/guides/INSTALLATION.md](/docs/openwatch/installation) |
| Roles and permissions | [docs/engineering/rbac_registry.md](../engineering/rbac_registry.md) |
| Kensa ↔ OpenWatch boundary | [docs/KENSA_OPENWATCH_BOUNDARY.md](../KENSA_OPENWATCH_BOUNDARY.md) |
| API contract (source of truth) | `api/openapi.yaml` (paths under `/api/v1`) |

## Troubleshooting

The service runs as a single `systemd` unit against PostgreSQL. The recipes
below cover the common first-run failures.

**The UI or API does not respond on 8443.** Check the unit and recent logs:

```bash
systemctl status openwatch
journalctl -u openwatch --no-pager -n 50
```

A boot failure is usually a missing JWT signing key
(`[identity].jwt_private_key`), a missing credential key
(`[identity].credential_key_file`), or an empty
`OPENWATCH_DATABASE_DSN`. Run `openwatch check-config` to see the resolved
configuration with secrets redacted.

**Health reports `db_connected: false`.** The database is unreachable. Verify
PostgreSQL is up and the DSN is correct:

```bash
systemctl status postgresql
psql "$OPENWATCH_DATABASE_DSN" -c "SELECT 1;"
```

If the schema is behind, run `openwatch migrate`.

**Connectivity check fails for a host.** SSH cannot connect. Confirm the address
and port, that the credential username and key/password are correct, and that
TCP/22 is open from the OpenWatch host to the target.

**A host shows all-zero compliance counts.** No compliance cycle has completed
yet. Confirm the connectivity check passes (Step 5), then watch
`journalctl -u openwatch -f` for discovery and compliance activity. The
schedulers run on an adaptive cadence, so the first result is not instantaneous.

**Background work appears stalled.** Confirm the `serve` process is running (it
hosts the schedulers) and, if you run a dedicated worker, that
`openwatch worker` is up. Inspect the PostgreSQL job queue directly:

```bash
psql "$OPENWATCH_DATABASE_DSN" -c \
  "SELECT status, count(*) FROM job_queue GROUP BY status;"
```

## Runbooks

These are first-response steps for the single binary on `systemd` plus
PostgreSQL. Replace `<dsn>` with the value from
`/etc/openwatch/secrets.env`.

### Service down

1. Check the unit and why it stopped:
   `systemctl status openwatch` and
   `journalctl -u openwatch --no-pager -n 100`.
2. If it crash-loops, look for the boot-time fatal log line (missing key, bad
   DSN, validation error). Fix the config, then `systemctl restart openwatch`.
3. Validate config out-of-band before restarting:
   `openwatch check-config --config /etc/openwatch/openwatch.toml`.
4. Confirm the dependency is up: `systemctl status postgresql`.
5. After restart, confirm recovery:
   `curl -sk https://localhost:8443/api/v1/health`.

### Disk full

1. Find what filled up: `df -h` then `du -xhd1 /var | sort -h | tail`.
2. Inspect the journal footprint (logs go to the journal):
   `journalctl --disk-usage`. Vacuum old logs with
   `journalctl --vacuum-time=7d` or `journalctl --vacuum-size=500M`.
3. Check PostgreSQL data growth and look for table bloat:
   `psql "<dsn>" -c "SELECT pg_size_pretty(pg_database_size(current_database()));"`.
4. If audit or history tables dominate, apply your retention policy rather than
   deleting rows ad hoc — these back compliance evidence.
5. After freeing space, confirm the service recovered:
   `systemctl status openwatch` and the health endpoint.

### High CPU

1. Identify the offender: `top` or `pidstat 1`. Confirm whether it is the
   `openwatch` process or `postgres`.
2. If it is `postgres`, look for long-running or stuck queries:
   `psql "<dsn>" -c "SELECT pid, state, now()-query_start AS age, query FROM pg_stat_activity ORDER BY age DESC LIMIT 10;"`.
3. If it is `openwatch`, correlate with scheduler activity in the journal:
   `journalctl -u openwatch --no-pager -n 200`. A burst of discovery or
   compliance cycles across many hosts can drive CPU; the cadence is adaptive
   and self-limits.
4. Check the job queue depth:
   `psql "<dsn>" -c "SELECT status, count(*) FROM job_queue GROUP BY status;"`.
5. If load is sustained and harmful, lower scheduler pressure via the runtime
   config endpoints (`PUT /api/v1/system/intelligence/config`,
   `PUT /api/v1/system/discovery/config`) — for example by enabling the
   maintenance pause — rather than killing the process.

### Security incident

1. Preserve evidence first. Do not wipe logs. Snapshot the journal:
   `journalctl -u openwatch --since "-24h" > /var/tmp/openwatch-incident.log`.
2. Review the audit trail. Audit events are queryable via
   `GET /api/v1/audit/events` and stored in PostgreSQL; export the relevant
   window for analysis.
3. If credentials may be exposed, rotate them: revoke or replace the affected
   SSH credentials (`/api/v1/credentials`) and rotate any user passwords.
   Active sessions can be ended via logout; force re-authentication for
   affected users.
4. If the host itself is compromised, isolate it at the network layer and stop
   the service to halt outbound SSH:
   `systemctl stop openwatch`.
5. Protect the secrets: `/etc/openwatch/secrets.env`,
   `/etc/openwatch/tls/`, and the identity keys referenced by the TOML config.
   If any may have leaked, rotate the JWT signing key and credential key, then
   re-encrypt or re-enter affected secrets.
6. After containment, document the timeline from the preserved journal and audit
   export before restarting.

---

# OpenWatch: OpenWatch install guide (native packages)
URL: https://www.hanalyx.com/docs/openwatch/installation

**Last Updated:** 2026-06-22 · **Applies to:** OpenWatch 0.2.0-rc series (Go single-binary)

This guide takes an administrator from a fresh Linux host to a running,
logged-in OpenWatch: install the package, point it at PostgreSQL, create the
first admin user, start the service, and sign in to the web UI.

## What you get

Installing the package gives you a single `systemd`-managed service that serves
both the REST API and the web UI over HTTPS on port 8443. One binary contains
everything — the API, the embedded React UI, and the Kensa compliance engine; no
separate web tier, no container runtime, no external cache.

After the steps below you have:

- The OpenWatch UI and API at `https://<host>:8443/`, behind session login.
- An `admin` account you create during install.
- A PostgreSQL database holding hosts, scans, transactions, and audit events.
- Kensa ready to run SSH-based compliance checks against the hosts you add.

---

## At a glance

| Step | What | Command |
|------|------|---------|
| 1 | Install PostgreSQL | `dnf install postgresql-server` / `apt install postgresql` |
| 2 | Provision the database | `createdb` + role (see below) |
| 3 | Install the packages | `dnf install ./openwatch-*.rpm ./kensa-rules-*.rpm` / `apt install ./openwatch_*.deb ./kensa-rules_*.deb` |
| 4 | Configure the database secret | edit `/etc/openwatch/secrets.env` |
| 5 | Run migrations | `openwatch migrate` |
| 6 | Create the first admin | `openwatch create-admin --username admin --email …` |
| 7 | Start the service | `systemctl enable --now openwatch` |
| 8 | Sign in | open `https://<host>:8443/` |

On a host that already runs PostgreSQL, this takes about five minutes.

---

## Requirements

- **OS:**
  - RPM: CentOS Stream 9, RHEL 9, Rocky Linux 9, AlmaLinux 9, Oracle Linux 9
  - DEB: Ubuntu 24.04 LTS, Debian 12 (or a compatible `systemd` derivative)
- **Architecture:** `x86_64`/`amd64` or `aarch64`/`arm64` (packages ship for both).
- **CPU/RAM:** 1 vCPU / 512 MB for the service itself; size up for large fleets.
- **Disk:** 500 MB for the binary plus database growth sized to your retention.
- **PostgreSQL:** 14 or newer. The package depends on the PostgreSQL client/server
  but does **not** create a database — you do that in Step 2.
- **Network:**
  - TCP/8443 inbound for the API and UI.
  - TCP/22 outbound from this host to every managed host (Kensa scans over SSH).
- **A browser** to reach the UI, and `sudo`/root for the install steps. The
  service itself runs as the unprivileged `openwatch` user the package creates.

> Download the `.rpm`/`.deb`, the `SHA256SUMS`, `SHA256SUMS.asc`, and `KEYS`
> from the GitHub release. To verify authenticity before installing:
> `gpg --import KEYS && gpg --verify SHA256SUMS.asc SHA256SUMS`, then
> `sha256sum -c SHA256SUMS`. RPMs are also signed in-header — import `KEYS`
> with `rpm --import KEYS` and check with `rpm -K openwatch-*.rpm`.

---

## Install on RHEL family (RPM)

### Step 1 — Install PostgreSQL

```bash
sudo dnf install -y postgresql-server postgresql-contrib
sudo postgresql-setup --initdb
sudo systemctl enable --now postgresql
```

### Step 2 — Provision the database

Create the role and database:

```bash
sudo -u postgres psql <<'SQL'
CREATE ROLE openwatch WITH LOGIN PASSWORD 'replace-with-a-strong-password';
CREATE DATABASE openwatch OWNER openwatch;
SQL
```

Allow password auth from localhost. Edit `/var/lib/pgsql/data/pg_hba.conf` and
ensure these lines exist near the top of the host rules, then reload:

```
host    openwatch    openwatch    127.0.0.1/32    scram-sha-256
host    openwatch    openwatch    ::1/128         scram-sha-256
```

```bash
sudo systemctl reload postgresql
PGPASSWORD='replace-with-a-strong-password' \
  psql -h 127.0.0.1 -U openwatch -d openwatch -c '\conninfo'
```

### Step 3 — Install the packages

```bash
sudo dnf install -y ./openwatch-*.x86_64.rpm ./kensa-rules-*.noarch.rpm
```

Install **both** files in one transaction. `openwatch` declares a hard
dependency on `kensa-rules` — the rule corpus the scan engine loads from
`/usr/share/kensa/rules`. Installing `openwatch` alone fails the dependency
check (by design: a corpus-less node cannot scan). `kensa-rules` is `noarch`
and versioned on the Kensa content line (e.g. `0.5.0`), independent of the
platform version, so the rules can update without re-releasing OpenWatch.

Use the filenames you downloaded (`aarch64` for the arm64 openwatch RPM; the
`kensa-rules` package is the same `noarch` file for every arch). Installing the
packages:

1. Creates the `openwatch` system user and group (idempotent).
2. Installs the binary at `/usr/bin/openwatch`, config under `/etc/openwatch/`
   (`openwatch.toml` plus a self-signed TLS cert/key), the `systemd` unit, and
   the `/var/lib/openwatch` and `/var/log/openwatch` data directories. The
   `kensa-rules` package installs the rule corpus to `/usr/share/kensa/rules`.
3. Generates the identity keys the server requires in production —
   `/etc/openwatch/keys/jwt_private.pem` (RSA-2048 JWT signing key) and
   `/etc/openwatch/keys/credential.key` (AES-256 credential DEK). This is
   generate-if-absent: a reinstall or upgrade never overwrites existing keys
   (regenerating them would invalidate sessions and make stored SSH/MFA
   secrets undecryptable). The server does **not** auto-generate these — it
   exits if they are missing — so the package lays them down at install time.
4. Reloads `systemd`. It does **not** start the service — you do that in Step 7,
   after the database and admin user exist.

Confirm the install:

```bash
rpm -q openwatch
openwatch --version
```

### Step 4 — Configure the database secret

The service reads its database connection string from
`/etc/openwatch/secrets.env` so the password stays out of the world-readable
config. The `systemd` unit loads this file automatically.

```bash
sudo tee /etc/openwatch/secrets.env >/dev/null <<'EOF'
OPENWATCH_DATABASE_DSN=postgres://openwatch:replace-with-a-strong-password@127.0.0.1:5432/openwatch?sslmode=disable
EOF
sudo chown root:openwatch /etc/openwatch/secrets.env
sudo chmod 0640 /etc/openwatch/secrets.env
```

> Use `sslmode=require` (or stronger) for any PostgreSQL that is not on the
> loopback interface.

### Step 5 — Run database migrations

This creates the schema (hosts, scans, transactions, audit events, the job
queue, and more). Run it as the `openwatch` user with the same DSN the service
uses:

```bash
sudo -u openwatch env $(cat /etc/openwatch/secrets.env | xargs) \
    openwatch migrate
```

The command applies every pending migration and reports the version it reached.
Re-running it when the schema is current is a safe no-op.

### Step 6 — Create the first admin user

This is the account you sign in with. The admin password policy requires **at
least 15 characters**; pick a single line with no spaces.

```bash
sudo -u openwatch env $(cat /etc/openwatch/secrets.env | xargs) \
    openwatch create-admin --username admin --email admin@example.com
# Type the admin password at the prompt and press Enter.
```

`create-admin` reads the password from stdin when `--password` is omitted, which
keeps it out of your shell history. For automation, pipe it instead:

```bash
printf '%s' "$ADMIN_PASSWORD" | sudo -u openwatch env $(cat /etc/openwatch/secrets.env | xargs) \
    openwatch create-admin --username admin --email admin@example.com
```

On success it prints `created admin user admin (admin@example.com) with id=…` and
assigns the built-in `admin` role.

### Step 7 — Start the service

```bash
sudo systemctl enable --now openwatch
sudo systemctl status openwatch
```

### Step 8 — Sign in

Confirm the API is healthy, then open the UI:

```bash
curl -k https://localhost:8443/api/v1/health
# {"status":"healthy","db_connected":true,"version":"<your installed version>"}
```

In a browser, go to **`https://<host>:8443/`**. The browser warns about the
self-signed cert — accept it (or install a CA cert first; see
[Replace the demo TLS cert](#replace-the-demo-tls-cert)) — then sign in with the
admin username and password from Step 6.

The `-k` flag and the browser warning both come from the bundled self-signed
cert. Replace it before any non-loopback use.

---

## Install on Ubuntu and Debian (DEB)

The flow is identical to the RPM path; only Steps 1–3 differ.

### Step 1 — Install PostgreSQL

```bash
sudo apt update
sudo apt install -y postgresql postgresql-contrib
sudo systemctl enable --now postgresql
```

### Step 2 — Provision the database

```bash
sudo -u postgres psql <<'SQL'
CREATE ROLE openwatch WITH LOGIN PASSWORD 'replace-with-a-strong-password';
CREATE DATABASE openwatch OWNER openwatch;
SQL
```

Ubuntu's default `pg_hba.conf` already allows `scram-sha-256` for
`host all all 127.0.0.1/32`, so no edit is needed unless you customized it.
Verify:

```bash
PGPASSWORD='replace-with-a-strong-password' \
  psql -h 127.0.0.1 -U openwatch -d openwatch -c '\conninfo'
```

### Step 3 — Install the packages

```bash
sudo apt install -y ./openwatch_*_amd64.deb ./kensa-rules_*_all.deb
```

Install **both** files together — `openwatch` `Depends` on `kensa-rules` (the
scan engine's rule corpus at `/usr/share/kensa/rules`), so installing the
openwatch `.deb` alone fails the dependency check by design. The `kensa-rules`
package is `Architecture: all` (one file for every arch). Use the openwatch
filename you downloaded (`arm64` for aarch64). If `apt` reports missing
dependencies, add `-f`. The packages create the `openwatch` user, install the
same files as the RPM plus the corpus, and reload `systemd` without starting
the service.

```bash
dpkg -l openwatch
openwatch --version
```

### Steps 4–8

Follow Steps 4 through 8 from the RPM section above — configure
`/etc/openwatch/secrets.env`, run `openwatch migrate`, run
`openwatch create-admin`, `systemctl enable --now openwatch`, and sign in at
`https://<host>:8443/`. The commands are the same.

---

## First steps as an administrator

Once you are signed in:

1. **Add a host.** Provide the hostname/IP and an SSH credential (key or
   password). OpenWatch checks reachability and discovers the OS.
2. **Confirm the credential.** The host's liveness and intelligence panels
   populate once the credential works.
3. **Run a Kensa scan** and read the compliance posture, then drift and
   exceptions over time.
4. **Add more administrators or scoped roles** from Settings as needed.

For the day-to-day workflows, see the operator guides under
[`docs/guides/`](../guides/) (hosts and remediation, scanning and compliance,
user roles). For the API, see [`api/openapi.yaml`](../../api/openapi.yaml).

---

## Common operations

### Service control

```bash
sudo systemctl start openwatch        # start
sudo systemctl stop openwatch         # stop
sudo systemctl restart openwatch      # restart
sudo systemctl status openwatch       # current state
sudo systemctl enable openwatch       # start at boot
sudo systemctl disable openwatch      # don't start at boot
```

### Logs

The service logs JSON to journald:

```bash
sudo journalctl -u openwatch -f                  # tail live
sudo journalctl -u openwatch --since '5 min ago' # recent
sudo journalctl -u openwatch -o cat | jq .       # pretty-print JSON
```

### Inspect the resolved config

```bash
sudo -u openwatch env $(cat /etc/openwatch/secrets.env | xargs) \
    openwatch check-config
```

### Replace the demo TLS cert

The package ships a self-signed cert. Replace it with one from your CA for any
non-loopback use:

```bash
sudo cp /path/to/your-cert.pem /etc/openwatch/tls/cert.pem
sudo cp /path/to/your-key.pem  /etc/openwatch/tls/key.pem
sudo chown root:openwatch       /etc/openwatch/tls/cert.pem
sudo chown openwatch:openwatch  /etc/openwatch/tls/key.pem
sudo chmod 0644                 /etc/openwatch/tls/cert.pem
sudo chmod 0600                 /etc/openwatch/tls/key.pem
sudo systemctl restart openwatch
```

The server reads the cert on every TLS handshake, so swapping the files takes
effect for new connections without a restart; restart anyway to cover existing
keep-alive connections.

### Configuration layering

Config values resolve in this order, highest precedence first:

1. CLI flags (`--listen`, `--log-level`)
2. Environment variables (`OPENWATCH_<SECTION>_<KEY>`)
3. The TOML file (`/etc/openwatch/openwatch.toml`)
4. Built-in defaults

Recognized environment variables:

| Variable | Effect |
|----------|--------|
| `OPENWATCH_SERVER_LISTEN` | Override `[server].listen` (default `:8443`) |
| `OPENWATCH_SERVER_TLS_CERT` | Override `[server].tls_cert` |
| `OPENWATCH_SERVER_TLS_KEY` | Override `[server].tls_key` |
| `OPENWATCH_DATABASE_DSN` | Override `[database].dsn` |
| `OPENWATCH_DATABASE_MAX_CONNECTIONS` | Override `[database].max_connections` |
| `OPENWATCH_LOGGING_LEVEL` | `debug` / `info` / `warn` / `error` |
| `OPENWATCH_LOGGING_FORMAT` | `json` / `text` |

---

## Troubleshooting

### Service won't start

```bash
sudo systemctl status openwatch
sudo journalctl -u openwatch --since '1 min ago' -p err
```

| Symptom | Cause | Fix |
|---------|-------|-----|
| `config: env override: OPENWATCH_DATABASE_DSN: …` | Malformed DSN in `secrets.env` | Use `postgres://user:pass@host:port/db?sslmode=…` |
| `db: ping: … password authentication failed` | Wrong DSN password, or `pg_hba.conf` rejects scram | Recheck Step 2; reload PostgreSQL after edits |
| `db: ping: … connection refused` | PostgreSQL not running | `sudo systemctl status postgresql` |
| `server: … no such file: cert.pem` | TLS cert path or perms wrong | Ensure `/etc/openwatch/tls/cert.pem` is readable by `openwatch` |

### `migrate` fails

`connection refused` means PostgreSQL isn't running; `password authentication
failed` means the DSN or `pg_hba.conf` is wrong (recheck Step 2).

### Can't sign in

- Confirm you created the admin: re-run `openwatch create-admin` (it reports if
  the username already exists).
- The password must be at least 15 characters and was read as a single line —
  re-create the admin if you're unsure what was stored.
- Make sure you're using `https://` (not `http://`) and accepted the cert.

### Health endpoint returns 503

```bash
curl -k https://localhost:8443/api/v1/health
# {"error":{"code":"server.unavailable",…}}
```

The database ping inside `/health` failed. Check `journalctl -u openwatch` for
the underlying error.

---

## Upgrading

Upgrading is one command. Download the newer `openwatch` package (and the newer
`kensa-rules` package if the rule corpus moved) and install it the same way you
did originally:

```bash
# RHEL family
sudo dnf install -y ./openwatch-<new>.x86_64.rpm ./kensa-rules-<new>.noarch.rpm

# Debian / Ubuntu
sudo apt install -y ./openwatch_<new>_amd64.deb ./kensa-rules_<new>_all.deb
```

On an upgrade (and only on an upgrade — never on a fresh install) the package
post-install step runs the upgrade helper, which:

1. Checks the database is reachable. If it is not, it leaves the service alone,
   prints how to finish later (`openwatch migrate && systemctl restart
   openwatch`), and does **not** fail the package transaction.
2. Stops the service so the old binary never runs against a half-migrated
   schema.
3. Takes a full `pg_dump` restore point into `/var/lib/openwatch/backups/`
   before touching the schema. If the backup fails, it aborts **without**
   migrating (fail-closed) — your data is untouched.
4. Applies any pending migrations, then starts the service again.

If a migration fails, the helper leaves the service **stopped** and exits
non-zero so the package manager surfaces the problem, and it prints the restore
path. Your data is intact (each migration runs in its own transaction and rolls
back on error). After fixing the cause:

```bash
openwatch migrate            # re-apply; reads the same DSN from secrets.env
sudo systemctl start openwatch
```

To preview what an upgrade would apply without changing anything:

```bash
sudo -u openwatch openwatch migrate --status
```

Tunables live in `/etc/openwatch/upgrade.conf` (a `noreplace` config file):
`AUTO_BACKUP=yes|no` toggles the pre-migration dump, and
`BACKUP_RETENTION_DAYS` controls pruning. A `systemd` timer
(`openwatch-backup-cleanup.timer`) prunes old dumps daily but **always keeps the
most recent one** regardless of age.

> Scope: this automates the OpenWatch **application** schema only. A PostgreSQL
> **engine** major-version upgrade (for example PostgreSQL 15 -> 16) is a
> separate, operator-supervised `pg_upgrade` and is never triggered from a
> package scriptlet. See `specs/release/upgrade.spec.yaml` for the full
> contract.

---

## Uninstall

### RPM

```bash
sudo systemctl stop openwatch
sudo dnf remove -y openwatch
```

Config under `/etc/openwatch/` is preserved (`%config(noreplace)`). Remove it
manually if you won't reinstall:

```bash
sudo rm -rf /etc/openwatch /var/lib/openwatch /var/log/openwatch
sudo userdel openwatch && sudo groupdel openwatch
```

### DEB

```bash
sudo systemctl stop openwatch
sudo apt remove openwatch          # leaves /etc/openwatch in place
sudo apt purge openwatch           # also removes the packaged config
```

`apt purge` removes the packaged `openwatch.toml` but leaves `secrets.env` and
the TLS material; remove those manually if needed.

### The database

Removing the package does **not** touch PostgreSQL. To reclaim that space:

```bash
sudo -u postgres psql <<'SQL'
DROP DATABASE openwatch;
DROP ROLE openwatch;
SQL
```

---

## Where to go next

- **Operator guides:** [`docs/guides/`](../guides/) — hosts and remediation,
  scanning and compliance, user roles.
- **API contract:** [`api/openapi.yaml`](../../api/openapi.yaml) — every endpoint
  with its required permission, license gate, and audit events.
- **Behavioral specs:** [`specs/`](../../specs/).
- **Release process:** [`docs/runbooks/RELEASING.md`](../runbooks/RELEASING.md).

---

## Quick reference card

```
UI + API       https://<host>:8443/        (API under /api/v1/…)
TLS cert       /etc/openwatch/tls/{cert,key}.pem   (self-signed by default)
Config         /etc/openwatch/openwatch.toml
DB secret      /etc/openwatch/secrets.env          (OPENWATCH_DATABASE_DSN)
Service unit   /etc/systemd/system/openwatch.service
Binary         /usr/bin/openwatch
Data / logs    /var/lib/openwatch  /var/log/openwatch  (journald is primary)
User/group     openwatch:openwatch
Migrate        sudo -u openwatch env $(cat /etc/openwatch/secrets.env | xargs) openwatch migrate
Create admin   sudo -u openwatch env $(cat /etc/openwatch/secrets.env | xargs) openwatch create-admin --username admin --email you@example.com
Logs           journalctl -u openwatch -f
Restart        sudo systemctl restart openwatch
```

---

# OpenWatch: Host Management and Remediation
URL: https://www.hanalyx.com/docs/openwatch/hosts-and-remediation

**Last Updated:** 2026-06-22 · **Applies to:** OpenWatch 0.2.0-rc series (Go single-binary)

This guide covers adding and managing hosts, organizing them into groups,
understanding server intelligence data, and using automated remediation to fix
compliance findings. Most of these tasks are performed in the web UI.

---

## Adding a Host

### From the UI

1. Navigate to **Hosts** in the left sidebar.
2. Click **Add Host**.
3. Fill in the host details:

| Field | Required | Example |
|-------|----------|---------|
| Hostname | Yes | `web-01` |
| IP Address | Yes | `192.168.1.10` |
| SSH Port | Yes | `22` |
| Display Name | No | `Web Server 01` |
| Operating System | No | `RHEL 9` |
| Environment | No | `production` |

4. Click **Save**.

![Add Host dialog](https://raw.githubusercontent.com/Hanalyx/OpenWatch/main/docs/guides/../images/hosts/add-host.png)

The host appears in the host list immediately after creation.

### Bulk Import

For adding many hosts at once:

1. Navigate to **Hosts** and click **Bulk Import**.
2. Download the CSV template.
3. Fill in the template with your host data.
4. Upload the CSV file.
5. Review the auto-detected field mappings.
6. Confirm the import.

![Bulk import CSV mapping review](https://raw.githubusercontent.com/Hanalyx/OpenWatch/main/docs/guides/../images/hosts/bulk-import.png)

Set **Dry Run** to validate the file without creating hosts. Set **Update
Existing** to overwrite hosts that match by hostname or IP address.

---

## Configuring SSH Credentials

OpenWatch connects to hosts over SSH to run compliance checks. No agent is
installed on target hosts.

### From the UI

1. Navigate to the host detail page.
2. Go to the **Credentials** section.
3. Select an authentication method:

| Method | When to Use |
|--------|-------------|
| **SSH Key** (recommended) | Paste or upload the private key. Stored encrypted. |
| **Password** | Enter the SSH password. Stored encrypted with AES-256-GCM. |
| **System Default** | Uses the credential configured in Settings > System Credentials. |

4. Enter the SSH username.
5. Click **Save**.

![Credential configuration form](https://raw.githubusercontent.com/Hanalyx/OpenWatch/main/docs/guides/../images/hosts/credentials.png)

### Testing Connectivity

After saving credentials, click **Test Connection** to verify that OpenWatch
can reach the host via SSH. The test checks:

- Network reachability
- SSH port availability
- Authentication success

Fix any connection issues before running a scan.

### System Credentials

For organizations where all hosts share the same SSH credentials, configure a
system-wide default:

1. Go to **Settings > System Credentials**.
2. Add the shared SSH key or password.
3. When adding hosts, select **System Default** as the auth method.

### Credential Security

All credentials are encrypted with AES-256-GCM before being stored in the
database. Decryption happens only at scan time, in memory. Plaintext credentials
are never written to disk or logs.

---

## Host Groups

Host groups let you organize hosts into logical collections for group-level
compliance reporting and batch scanning.

### Creating a Group

1. Navigate to **Host Groups** in the sidebar.
2. Click **Create Group**.
3. Enter a name, description, OS family, and compliance framework.
4. Click **Save**.

![Create host group dialog](https://raw.githubusercontent.com/Hanalyx/OpenWatch/main/docs/guides/../images/hosts/create-group.png)

### Assigning Hosts

1. Open the group detail page.
2. Click **Add Hosts**.
3. Select hosts from the list.
4. Click **Confirm**.

Each host can belong to one group at a time.

### Smart Group Creation

Select multiple hosts and click **Smart Group**. OpenWatch analyzes their OS,
architecture, and compliance profile to recommend group settings automatically.

### Group Scanning

From the group detail page, click **Scan Group** to start a compliance scan
for all hosts in the group simultaneously. Monitor progress on the group's
scan session page.

---

## Host Discovery

### OS Detection

OpenWatch automatically detects the operating system for hosts during scans.
You can also trigger manual OS discovery from the host detail page by clicking
**Discover OS**.

A scheduled task runs daily at 02:00 UTC to discover the OS for all active
hosts that have not been identified yet.

### Connectivity Monitoring

Host connectivity is checked every 30 seconds automatically. Each check
verifies ICMP reachability, SSH port availability, and SSH authentication.
Host status (online, offline, degraded) updates in the host list.

---

## Server Intelligence

During compliance scans, OpenWatch collects detailed information about each host.
This data is available on the host detail page under the **Intelligence** tab.

![Server intelligence overview](https://raw.githubusercontent.com/Hanalyx/OpenWatch/main/docs/guides/../images/hosts/server-intelligence.png)

### Data Collected

| Category | What It Contains |
|----------|------------------|
| Packages | Installed packages, versions, sources |
| Services | Running services, listening ports, enabled state |
| Users | User accounts, groups, shell, last login |
| Network | Interfaces, IP addresses, firewall rules |

### System Information

The host detail page also shows:

- OS name, version, and kernel release
- CPU model, core count, and architecture
- Total and available memory
- SELinux or AppArmor status
- Firewall status and active service

This data helps operators understand the security surface of each host without
needing to SSH in manually.

---

## Remediation Overview

OpenWatch can automatically fix compliance findings through Kensa's 23
remediation mechanisms. All changes are made over SSH -- nothing is installed
on target hosts.

### What Remediation Can Fix

| Category | Examples |
|----------|----------|
| Boot configuration | GRUB settings, boot parameters |
| Authentication | PAM modules, password policies |
| Filesystem | fstab mount options, file permissions |
| Kernel | sysctl parameters, module blacklisting |
| Services | systemd service management, cron restrictions |
| Audit | auditd rules, log configuration |
| Network | SSH daemon settings, firewall rules |

---

## Starting a Remediation

### From the UI

1. Navigate to the host detail page and view the scan results.
2. Select the failing findings you want to remediate (use checkboxes).
3. Click **Remediate Selected**.

![Selecting findings for remediation](https://raw.githubusercontent.com/Hanalyx/OpenWatch/main/docs/guides/../images/hosts/select-remediation.png)

4. Review the proposed changes. Each finding shows what will be modified.
5. Click **Start Remediation** to confirm.

![Remediation confirmation dialog](https://raw.githubusercontent.com/Hanalyx/OpenWatch/main/docs/guides/../images/hosts/confirm-remediation.png)

For organizations that require an approval step:

1. A user with `remediation:request` (`ops_lead`, `security_admin`, or `admin`)
   selects findings, clicks **Request Remediation**, and enters a justification.
2. A **different** user with `remediation:approve` (`security_admin` or `admin`)
   reviews and approves or rejects it. You cannot approve your own request
   (separation of duties; self-approval returns `409 self_review`).
3. Once approved, a user with `remediation:execute` clicks **Fix** to apply the
   change. Execution is operator-initiated, not automatic.

See [Remediation & Exception Governance](../engineering/remediation_exception_governance.md)
for the full role matrix. Single-operator workspaces cannot self-approve today;
see the [governance ADR](../engineering/remediation_governance_adr.md).

---

## Monitoring Remediation Progress

After starting a remediation, track its progress on the host detail page
under the **Remediation** tab.

![Remediation progress view](https://raw.githubusercontent.com/Hanalyx/OpenWatch/main/docs/guides/../images/hosts/remediation-progress.png)

The progress view shows:

- **Job status**: pending, running, completed, failed, partial, cancelled
- **Progress percentage**: how many rules have been processed
- **Per-rule results**: which fixes succeeded, failed, or were skipped
- **Execution log**: timestamps and details for each step

---

## Rollback

Pre-state snapshots are captured automatically before any remediation changes.
If a remediation causes problems, you can roll back to the pre-change state.

### From the UI

1. Go to the **Remediation** tab on the host detail page.
2. Find the remediation job you want to roll back.
3. Click **Rollback**.
4. Enter a reason for the rollback (logged for audit purposes).
5. Click **Confirm Rollback**.

![Rollback confirmation](https://raw.githubusercontent.com/Hanalyx/OpenWatch/main/docs/guides/../images/hosts/rollback.png)

Rollback requires the `remediation:rollback` permission (`ops_lead`,
`security_admin`, or `admin`).

### After Rolling Back

After a rollback completes, run a follow-up compliance scan to verify the host
returned to its previous state.

---

## Required Permissions

Built-in roles, least to most privilege: `viewer` → `auditor` → `ops_lead` →
`security_admin` → `admin` (`admin` holds every permission). The permission
source of truth is `auth/permissions.yaml`; see
[Remediation & Exception Governance](../engineering/remediation_exception_governance.md)
for the complete matrix.

| Operation | Permission | Roles that hold it |
|-----------|------------|--------------------|
| View hosts | `host:read` | viewer, auditor, ops_lead, security_admin, admin |
| Add / edit hosts | `host:write` | ops_lead, security_admin, admin |
| Delete hosts | `host:delete` | security_admin, admin |
| Request remediation | `remediation:request` | ops_lead, security_admin, admin |
| Approve / reject remediation | `remediation:approve` | security_admin, admin |
| Execute remediation (Fix) | `remediation:execute` | ops_lead, security_admin, admin |
| Rollback remediation | `remediation:rollback` | ops_lead, security_admin, admin |
| View server intelligence | `host:read` | viewer, auditor, ops_lead, security_admin, admin |

---

## Best Practices

1. **Test credentials before scanning.** Use the Test Connection button to
   confirm SSH access before running a compliance scan.
2. **Use SSH keys, not passwords.** Key-based authentication is more secure
   and works reliably with automated scanning.
3. **Start remediation on a single host.** Test changes on one host before
   applying to a group.
4. **Review findings before remediating.** Understand what each rule checks
   and what the fix changes.
5. **Monitor compliance score after remediation.** The adaptive scheduler will
   automatically scan again, but you can force a scan for immediate results.
6. **Use groups for consistent scanning.** Hosts in the same group share OS
   family, framework, and scan schedule settings.

---

## What's Next

- [Scanning and Compliance](/docs/openwatch/scanning-and-compliance) -- understanding scan results and posture
- [User Roles](/docs/openwatch/user-roles) -- role permissions and what each role can access
- [API Guide](/docs/openwatch/api-guide) -- REST API for automation

---

## Appendix: API Automation

For operators who want to script host management or integrate with CI/CD
pipelines, here are the key API endpoints. OpenWatch serves the REST API over
HTTPS on port `8443`; every path lives under `/api/v1`. The contract source of
truth is `api/openapi.yaml`. Replace `openwatch.example.com` with your host.

### Add a Host

```bash
curl -s -X POST https://openwatch.example.com:8443/api/v1/hosts \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"hostname": "web-01", "ip_address": "192.168.1.10", "port": 22, "environment": "prod"}'
```

`hostname` and `ip_address` are required; `port` defaults to 22. Other optional
fields: `display_name`, `description`, `tags`, `group_id`, `username`. There is
no bulk-import or CSV-export API endpoint. Import many hosts from a CSV in the
web UI (Hosts, Import), which validates each row before insert.

### Create a Group

```bash
curl -s -X POST https://openwatch.example.com:8443/api/v1/groups \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "Production Web Servers", "kind": "site", "membership": "manual"}'
```

`kind` is `site` or `os_category`; `membership` is `manual` or `auto` (an `auto`
group also needs `match_family`). Add a host to a manual group via
`POST /api/v1/groups/{id}/members`.

### Request and Execute Remediation

Remediation is a request lifecycle, not a single call: request a fix for a
failing rule on a host, then execute it (free-core single-rule fixes
auto-approve on request; the licensed bulk track keeps the approve/reject step).

```bash
# 1. Request a fix for one failing rule on one host.
RID=$(curl -s -X POST https://openwatch.example.com:8443/api/v1/remediation/requests \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"host_id": "HOST_UUID", "rule_id": "sshd-disable-root-login"}' \
  | jq -r '.id')

# 2. Execute it (mutates the host; runs serialized per host).
curl -s -X POST "https://openwatch.example.com:8443/api/v1/remediation/requests/${RID}:execute" \
  -H "Authorization: Bearer $TOKEN"
```

### Roll Back

```bash
curl -s -X POST "https://openwatch.example.com:8443/api/v1/remediation/requests/${RID}:rollback" \
  -H "Authorization: Bearer $TOKEN"
```

See the [API Guide](/docs/openwatch/api-guide) for authentication and the complete endpoint
reference.

---

# OpenWatch: Scanning and Compliance
URL: https://www.hanalyx.com/docs/openwatch/scanning-and-compliance

**Last Updated:** 2026-06-22 · **Applies to:** OpenWatch 0.2.0-rc series (Go single-binary)

This guide covers how OpenWatch performs compliance scanning, how to read
results and posture scores, and how to use drift detection, alerts, and
audit exports. Most of these tasks are performed in the web UI.

---

## How Scanning Works

When a scan runs, OpenWatch uses the Kensa compliance engine to connect to the
target host over SSH, execute each rule's check command, and return a pass/fail
result with machine-verifiable evidence.

```
Operator clicks "Run Scan" (or adaptive scheduler triggers)
        |
        v
Scan job enqueued on the PostgreSQL job queue (SKIP LOCKED)
        |
        v
Kensa retrieves SSH credentials from OpenWatch's encrypted store
        |
        v
SSH connection to target host
        |
        v
538 YAML rules evaluated (check commands, config values, file permissions)
        |
        v
Each rule returns: pass/fail, severity, detail, evidence
        |
        v
Write-on-change: changed verdicts -> transactions; current state -> host_rule_state
        |
        v
Daily posture snapshot rolls up; drift alerts generated if thresholds met
```

Key points:

- **No agent on targets.** Kensa connects over SSH, runs commands, and
  disconnects. Nothing is installed on the scanned host.
- **One scan, many frameworks.** A single scan produces results that map to
  CIS, STIG, NIST, PCI-DSS, and FedRAMP simultaneously.
- **Evidence captured.** Each check records the command executed, the raw
  output, the expected value, and the actual value found.

---

## Available Frameworks

| Framework | Mapping ID | Rules |
|-----------|------------|-------|
| CIS RHEL 9 v2.0.0 | cis-rhel9-v2.0.0 | 271 |
| STIG RHEL 9 V2R7 | stig-rhel9-v2r7 | 338 |
| NIST 800-53 Rev 5 | nist-800-53-r5 | 87 |
| PCI-DSS v4.0 | pci-dss-v4.0 | 45 |
| FedRAMP Moderate | fedramp-moderate | 87 |

Framework mappings are carried per-rule as Kensa's normalized `framework_refs`
(a multi-valued framework_id -> control-ids map, since one rule can satisfy
several controls within a framework). They are stored on each `host_rule_state`
row as `framework_refs` JSONB and projected into the lens views at query time --
there is no separate sync service in the Go rebuild.

---

## Running a Scan

### From the UI

1. Navigate to **Hosts** and select the host you want to scan.
2. On the host detail page, click **Run Scan**.

A scan always runs the **full applicable rule corpus** -- you do not pick a
framework to scan. Frameworks (CIS, STIG, NIST, PCI) are **reporting lenses**
applied at view time on the Compliance tab: one scan, viewed through any
framework. The lens bar only offers frameworks compatible with the host's
detected OS (a RHEL 8 host does not show CIS/STIG RHEL 9 or 10 lenses);
OS-neutral frameworks (NIST, PCI, SRG) always appear.

![Running a scan from the host detail page](https://raw.githubusercontent.com/Hanalyx/OpenWatch/main/docs/guides/../images/scanning/run-scan.png)

The scan runs in the background. A progress indicator shows the scan status.
Results appear on the host's compliance tab once the scan completes
(typically 1--5 minutes).

### Automatic Scanning

Most hosts are scanned automatically by the adaptive scheduler. You do not need
to trigger scans manually unless you want immediate results. See the
[Adaptive Scheduling](#adaptive-scheduling) section below.

---

## Reading Scan Results

After a scan completes, the results are displayed on the host detail page under
the **Compliance** tab.

![Scan results page with findings table](https://raw.githubusercontent.com/Hanalyx/OpenWatch/main/docs/guides/../images/scanning/scan-results.png)

### What You See

- **Compliance score** -- percentage of rules passing (e.g., 85.0%)
- **Summary bar** -- pass, fail, error, and skipped counts
- **Severity breakdown** -- counts by critical, high, medium, low
- **Findings table** -- sortable, filterable list of all findings

### Finding Details

Click any finding row to expand it. Each finding shows:

| Field | Description |
|-------|-------------|
| Rule ID | Kensa rule identifier (e.g., `sshd-disable-root-login`) |
| Title | Human-readable description |
| Severity | critical, high, medium, or low |
| Status | pass, fail, error, or skipped |
| Detail | Explanation of the check result |
| Evidence | Command executed, expected value, actual value |

### Filtering Results

Use the filter controls above the findings table to narrow results:

- **By severity** -- show only critical and high findings
- **By status** -- show only failures
- **By search** -- search rule titles and descriptions

---

## Compliance Posture

### What the Score Means

The compliance score is the percentage of evaluated rules that passed:

```
compliance_score = (passed_rules / total_rules) * 100
```

A score of 85.0 means 85% of rules passed. Skipped rules are excluded from
the total.

### Viewing Posture in the Dashboard

Navigate to the **Dashboard** from the sidebar. The posture overview shows:

- **Aggregate score** across all hosts
- **Per-host scores** in the host list
- **Trend chart** showing score changes over time
- **Framework breakdown** with per-framework compliance percentages

![Compliance posture dashboard](https://raw.githubusercontent.com/Hanalyx/OpenWatch/main/docs/guides/../images/scanning/posture-dashboard.png)

### Historical Posture

OpenWatch captures daily posture snapshots at 00:30 UTC. To view historical
posture:

1. Navigate to the host detail page.
2. Select the **Posture History** tab.
3. Choose a date range to view the compliance trend.

Historical posture queries with specific `as_of` dates require OpenWatch+.

---

## Drift Detection

Drift occurs when a rule's status changes between two points in time. A rule
that was passing and now fails is a **regression**. A rule that was failing
and now passes is an **improvement**.

### Viewing Drift in the UI

1. Navigate to the host detail page.
2. Select the **Drift** tab.
3. Choose a date range (start date and end date).

![Drift detection showing regressions and improvements](https://raw.githubusercontent.com/Hanalyx/OpenWatch/main/docs/guides/../images/scanning/drift-view.png)

The drift view shows:

- **Score delta** -- how much the compliance score changed
- **Drift type** -- stable, minor, major, or improvement
- **Rules improved** and **rules regressed** -- counts with expandable lists
- **Timeline** -- when each drift event occurred

### Field-Level Value Drift

Enable **Include value drift** to see rules where the underlying configuration
value changed even though the pass/fail status did not. For example, a password
minimum length changing from 14 to 12 while both still pass the threshold.

### What to Do When Drift Is Detected

1. Review the regressed rules and their evidence.
2. Investigate the root cause on the host (configuration change, package update).
3. Remediate the finding, or create a compliance exception if the risk is accepted.

---

## Adaptive Scheduling

The compliance scheduler automatically scans hosts at intervals based on their
compliance state. You do not need to trigger manual scans for routine monitoring.

### How It Works

A host is classified into one of five score bands (plus Unknown for
never-scanned hosts) after every scan, and the next scan is scheduled from the
band's interval. The intervals below are the **defaults** -- they are
operator-editable per band under **Settings -> Scanning & monitoring ->
Compliance scanner**, clamped to a 5-minute floor and a 48-hour ceiling.

| Compliance State | Score Range | Default Interval |
|------------------|-------------|------------------|
| Critical | < 20%, or any critical finding | Every 4 hours |
| Non-compliant | 20--49% | Every 8 hours |
| Partial | 50--69% | Every 12 hours |
| Mostly compliant | 70--89% | Every 24 hours |
| Compliant | >= 90% | Every 48 hours |
| Unknown | Never scanned | Every 6 hours (due immediately on first sight) |

The maximum interval is 48 hours. No active host goes unscanned longer than
that. A per-host or fleet-wide maintenance flag pauses scheduled scans without
affecting on-demand Run Scan.

### Viewing a Host's Schedule

On the host detail page, the **Scheduling** section shows:

- Current scan interval
- Next scheduled scan time
- Compliance state driving the interval
- Whether the host is in maintenance mode

### Maintenance Mode

To pause scanning for a host (during planned maintenance):

1. Go to the host detail page.
2. Click **Maintenance Mode**.
3. Set the duration (1--168 hours).
4. Click **Enable**.

Maintenance mode expires automatically. You can disable it early from the same page.

### Force Scan

To trigger an immediate scan outside the normal schedule, click **Force Scan**
on the host detail page. This bypasses the schedule and runs at highest priority.

---

## Compliance Exceptions

A compliance exception is a documented, operator-approved waiver for a failing
rule on a host -- accepted risk ("rule X on host Y is accepted because Z, until
date D"). Exceptions are governed through a request and approval workflow.

**Overlay model (important):** an exception **never changes a rule's scan
verdict**. A failing rule with an active exception still shows as failing in the
raw results and still counts against the raw compliance score -- Kensa's verdict
is authoritative. The exception is a governance annotation that marks the
failure as accepted risk wherever it surfaces. This keeps the score honest and
keeps the audit trail showing both "the control failed" and "the failure was
formally accepted."

### Lifecycle

```
requested -> approved  -> (active until expiry) -> revoked | expired
          -> rejected
```

### Who can do what (separation of duties)

| Action | Permission | Roles |
|--------|------------|-------|
| Request an exception | exception:request | ops_lead, auditor, security_admin, admin |
| Approve / reject a request | exception:approve | auditor, security_admin, admin |
| Revoke an active exception | exception:revoke | security_admin, admin |
| View exceptions | exception:read | all roles above + viewer |

The requester **cannot** approve their own request.

### Requesting an exception

1. On the host's **Compliance** tab, find a failing rule.
2. Click **Request exception** in its row.
3. Enter the reason (required) and an optional expiry date.
4. Submit. The rule now shows a **Pending** badge.

### Approving / managing exceptions

The fleet queue lives at **Settings -> Compliance policies -> Exception
workflow**. Filter by status (Pending / Active / Rejected / Revoked / Expired),
and on a pending request an approver can **Approve** or **Reject**; on an active
exception, **Revoke** ends it before its expiry. Once approved, the rule shows a
**Waived** badge everywhere it appears (the host's Compliance tab, the Watchlist
tile, the Server-intelligence Open-exceptions count). Approved exceptions whose
expiry passes are swept to **expired** automatically.

---

## Alert Management

Alerts are generated automatically when scan results meet configured thresholds.

### Alert Categories

| Category | Alert Types |
|----------|-------------|
| Compliance | Critical finding, high finding, score drop, non-compliant, degrading trend |
| Operational | Host unreachable, scan failed, scheduler stopped, scan backlog |
| Exception | Exception expiring, exception expired, exception requested |
| Drift | Configuration drift, unexpected remediation, mass drift |

### Viewing Alerts

Navigate to **Alerts** in the sidebar. The alert list shows all active,
acknowledged, and recently resolved alerts.

![Alert management page](https://raw.githubusercontent.com/Hanalyx/OpenWatch/main/docs/guides/../images/scanning/alerts.png)

Use filters to narrow by:

- **Status** -- active, acknowledged, resolved
- **Severity** -- critical, high, medium, low
- **Category** -- compliance, operational, exception, drift

### Alert Lifecycle

```
Active --> Acknowledged --> Resolved
```

- **Active**: Alert generated, requires attention.
- **Acknowledged**: Click **Acknowledge** to indicate you are investigating.
- **Resolved**: Click **Resolve** after the issue is fixed or accepted.

### Configuring Thresholds

Navigate to **Settings > Alert Thresholds** to customize when alerts fire.

| Setting | Default | Meaning |
|---------|---------|---------|
| Score drop threshold | 20 points | Alert if score drops 20+ points in 24h |
| Non-compliant threshold | 80% | Alert if score falls below 80% |
| Degrading trend scans | 3 | Alert after 3 consecutive declining scans |
| Max scan age | 48 hours | Alert if host not scanned in 48 hours |
| Exception expiry warning | 7 days | Warn 7 days before exception expires |
| Mass drift threshold | 10 hosts | Alert if 10+ hosts drift simultaneously |

---

## Exporting for Audits

OpenWatch provides audit query and export tools for generating compliance
evidence.

### Creating a Saved Query

1. Navigate to **Compliance > Audit Queries**.
2. Click **New Query**.
3. Define filter criteria (severities, statuses, date range, hosts).
4. Name the query and set visibility (private or shared).
5. Click **Save**.

### Previewing Results

Before generating a full export, click **Preview** to see a sample of matching
findings and the total count.

### Generating an Export

1. From a saved query, click **Export**.
2. Choose a format: **CSV**, **JSON**, or **PDF**.
3. The export generates in the background. A download link appears when ready.

![Audit export download](https://raw.githubusercontent.com/Hanalyx/OpenWatch/main/docs/guides/../images/scanning/audit-export.png)

Exports include a SHA-256 checksum for integrity verification. Exports expire
after 7 days by default.

---

## What's Next

- [Hosts and Remediation](/docs/openwatch/hosts-and-remediation) -- managing hosts and fixing findings
- [User Roles](/docs/openwatch/user-roles) -- role-based access control
- [API Guide](/docs/openwatch/api-guide) -- REST API for automation and scripting

---

## Appendix: API Automation

For operators who want to script scanning workflows or integrate with CI/CD
pipelines, here are the key API endpoints.

### Start a Scan

Enqueue an on-demand scan for one host. The scan runs the full Kensa rule corpus
(frameworks are lenses applied to the results, not a scan parameter). The call
returns `202` with the new scan id; the scan itself runs asynchronously on the
worker.

```bash
curl -k -X POST https://localhost:8443/api/v1/hosts/HOST_UUID/scans \
  -H "Authorization: Bearer $TOKEN"
```

### Query Posture

```bash
curl -k "https://localhost:8443/api/v1/compliance/posture?host_id=HOST_UUID" \
  -H "Authorization: Bearer $TOKEN"
```

### Query Historical Posture

```bash
curl -k "https://localhost:8443/api/v1/compliance/posture?host_id=HOST_UUID&as_of=2026-02-15" \
  -H "Authorization: Bearer $TOKEN"
```

### Query Drift

```bash
curl -k "https://localhost:8443/api/v1/compliance/posture/drift?host_id=HOST_UUID&start_date=2026-02-01&end_date=2026-02-28" \
  -H "Authorization: Bearer $TOKEN"
```

### List Alerts

```bash
curl -k "https://localhost:8443/api/v1/compliance/alerts?status=active" \
  -H "Authorization: Bearer $TOKEN"
```

### Acknowledge an Alert

```bash
curl -k -X POST "https://localhost:8443/api/v1/compliance/alerts/ALERT_ID/acknowledge" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" -d '{}'
```

### Create an Export

```bash
curl -k -X POST https://localhost:8443/api/v1/compliance/audit/exports \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query_definition": {"severities": ["critical"], "statuses": ["fail"]}, "format": "csv"}'
```

See the [API Guide](/docs/openwatch/api-guide) for the complete endpoint reference.

---

# OpenWatch: Compliance Control Mapping
URL: https://www.hanalyx.com/docs/openwatch/compliance-controls

**Last Updated:** 2026-06-22 · **Applies to:** OpenWatch 0.2.0-rc series (Go single-binary)

This document maps OpenWatch's security controls to industry frameworks, providing evidence for compliance audits.

## Framework Coverage

| Framework | Controls Mapped | Coverage |
|-----------|----------------|----------|
| NIST SP 800-53 Rev 5 | 42 | Moderate baseline |
| CIS Controls v8 | 18 | Implementation Group 2 |
| CMMC Level 2 | 28 | Practice-level mapping |
| FedRAMP Moderate | 42 | Inherited from NIST |
| ISO 27001:2022 | 15 | Annex A controls |

## NIST SP 800-53 Control Mapping

### Access Control (AC)

| Control | Title | OpenWatch Implementation | Evidence |
|---------|-------|-------------------------|----------|
| AC-2 | Account Management | User CRUD with RBAC (admin, analyst, viewer) | `internal/users/`, user audit events |
| AC-3 | Access Enforcement | Role-based permission checks on each route | `internal/auth/`, generated permission registry |
| AC-6 | Least Privilege | Three-tier role model, default viewer role | `internal/auth/permissions.gen.go` |
| AC-7 | Unsuccessful Logon Attempts | Rate limiting (100/min per user, 1000/min per IP) | `internal/server/` middleware |
| AC-8 | System Use Notification | Configurable login banner | Frontend login page |
| AC-11 | Session Lock | Inactivity timeout (default 15 min, configurable 1-480) | `internal/systemconfig/` (session-timeout) |
| AC-12 | Session Termination | Session cookie and JWT expiration (30 min access, 7 day refresh) | `internal/auth/` |
| AC-17 | Remote Access | SSH with NIST SP 800-57 key validation | `internal/ssh/` |

### Audit and Accountability (AU)

| Control | Title | OpenWatch Implementation | Evidence |
|---------|-------|-------------------------|----------|
| AU-2 | Event Logging | Structured audit events for auth/scan/admin actions | `internal/audit/` |
| AU-3 | Content of Audit Records | User, timestamp, action, resource, outcome | `internal/audit/` |
| AU-6 | Audit Record Review | Audit query API (`/api/v1/audit/events`) | `internal/audit/`, `api/openapi.yaml` |
| AU-9 | Protection of Audit Information | Audit events stored append-only in PostgreSQL | `audit_events` table (`internal/db/migrations/`) |
| AU-12 | Audit Record Generation | API routes generate audit events | `internal/server/`, `internal/audit/` |

### Configuration Management (CM)

| Control | Title | OpenWatch Implementation | Evidence |
|---------|-------|-------------------------|----------|
| CM-2 | Baseline Configuration | Kensa YAML rules define expected configurations | Kensa rules (338 native YAML rules) |
| CM-3 | Configuration Change Control | SQL migration tracking, git version control | `internal/db/migrations/` (run via `openwatch migrate`) |
| CM-6 | Configuration Settings | Configuration validation at startup | `internal/config/`, `openwatch check-config` |
| CM-8 | System Component Inventory | Host management with discovery and metadata | `internal/host/`, `internal/intelligence/` |

### Identification and Authentication (IA)

| Control | Title | OpenWatch Implementation | Evidence |
|---------|-------|-------------------------|----------|
| IA-2 | Identification and Authentication | Session cookie and JWT auth with username/password | `internal/auth/` |
| IA-2(1) | MFA for Privileged Accounts | TOTP-based MFA with backup codes | `internal/auth/` |
| IA-5 | Authenticator Management | Argon2id hashing (64MB, 3 iterations), 8-char minimum (15 for admin) | `internal/users/` |
| IA-5(1) | Password-Based Authentication | Complexity requirements (upper, lower, digit, special) | `internal/auth/` (password policy) |

### Risk Assessment (RA)

| Control | Title | OpenWatch Implementation | Evidence |
|---------|-------|-------------------------|----------|
| RA-5 | Vulnerability Monitoring | Automated compliance scanning via Kensa | `internal/kensa/` |
| RA-5(2) | Update Vulnerabilities | Kensa rule updates via rule sync | `internal/kensa/` |

### System and Communications Protection (SC)

| Control | Title | OpenWatch Implementation | Evidence |
|---------|-------|-------------------------|----------|
| SC-8 | Transmission Confidentiality | TLS 1.2/1.3 for all connections | `internal/server/` (HTTPS listener) |
| SC-8(1) | Cryptographic Protection | FIPS-approved cipher suites | `internal/config/` |
| SC-10 | Network Disconnect | Configurable session timeout | `internal/systemconfig/` |
| SC-12 | Cryptographic Key Establishment | AES-256-GCM with environment-sourced keys | `internal/secretkey/`, `internal/credential/` |
| SC-13 | Cryptographic Protection | FIPS via OpenSSL 3.x FIPS provider | `internal/config/` |
| SC-23 | Session Authenticity | Session cookie plus JWT, HttpOnly cookies | `internal/auth/` |
| SC-28 | Protection of Information at Rest | AES-256-GCM encryption for credentials | `internal/credential/`, `internal/secretkey/` |

### System and Information Integrity (SI)

| Control | Title | OpenWatch Implementation | Evidence |
|---------|-------|-------------------------|----------|
| SI-2 | Flaw Remediation | Single Go binary built from a maintained Go toolchain | `go.mod`, native RPM/DEB packages |
| SI-4 | System Monitoring | Health checks and fleet monitoring endpoints | `/api/v1/health`, `internal/liveness/` |
| SI-10 | Information Input Validation | Request validation at the API boundary, parameterized SQL | `internal/server/`, `sqlc`-generated queries |

## CIS Controls v8 Mapping

| CIS Control | Title | OpenWatch Implementation |
|-------------|-------|-------------------------|
| 1.1 | Enterprise Asset Inventory | Host management with system info collection |
| 2.1 | Software Inventory | Server intelligence (package collection) |
| 3.3 | Data Encryption | AES-256-GCM at rest, TLS 1.2+ in transit |
| 4.1 | Secure Configuration | Kensa compliance scanning (538-rule corpus) |
| 4.2 | Baseline Network Configuration | Network discovery and topology mapping |
| 5.2 | Unique Passwords | Argon2id hashing, 8-char minimum (15 for admin), breached-password screening |
| 5.4 | MFA | TOTP-based MFA with backup codes |
| 6.1 | Audit Log Management | Structured JSON audit logs, audit query API |
| 6.3 | Centralized Log Collection | JSON logging, configurable log aggregation |
| 8.2 | Audit Logging | All authentication and authorization events logged |
| 8.5 | Access Control Logs | JWT validation events, RBAC enforcement logged |
| 8.11 | Audit Log Retention | Configurable retention, export to CSV/JSON/PDF |
| 9.1 | Email Security | SMTP TLS for notifications |
| 10.1 | Anti-Malware | File upload validation, no executable uploads |
| 13.1 | Network Monitoring | Health check endpoints, Prometheus metrics |
| 16.1 | Application Security | Request validation at the API boundary, parameterized SQL (no raw SQL) |
| 16.9 | Security Headers | CSP, X-Frame-Options, HSTS, X-Content-Type-Options |
| 16.11 | Web Application Firewalls | Built-in rate limiting, request size limits |

## CMMC Level 2 Practice Mapping

| Practice | Domain | OpenWatch Implementation |
|----------|--------|-------------------------|
| AC.L2-3.1.1 | Access Control | RBAC with three-tier role model |
| AC.L2-3.1.2 | Access Control | Transaction-level access enforcement |
| AC.L2-3.1.5 | Access Control | Least privilege (viewer default role) |
| AC.L2-3.1.7 | Access Control | Prevent non-privileged users from executing privileged functions |
| AC.L2-3.1.8 | Access Control | Unsuccessful logon attempt limiting |
| AC.L2-3.1.10 | Access Control | Session lock after inactivity |
| AC.L2-3.1.12 | Access Control | Remote access session termination |
| AU.L2-3.3.1 | Audit | System-level audit records |
| AU.L2-3.3.2 | Audit | User accountability through audit trails |
| CA.L2-3.12.1 | Assessment | Compliance posture assessment |
| CA.L2-3.12.3 | Assessment | Continuous monitoring via scheduled scans |
| CM.L2-3.4.1 | Configuration | Baseline configurations (Kensa rules) |
| CM.L2-3.4.2 | Configuration | Security configuration enforcement |
| CM.L2-3.4.5 | Configuration | Access restrictions for configuration changes |
| IA.L2-3.5.1 | Identification | User identification and authentication |
| IA.L2-3.5.2 | Identification | Device authentication (SSH host verification) |
| IA.L2-3.5.3 | Identification | Multi-factor authentication |
| IA.L2-3.5.7 | Identification | Minimum password complexity |
| IA.L2-3.5.8 | Identification | Password reuse prevention |
| IA.L2-3.5.10 | Identification | Cryptographically-protected passwords |
| MP.L2-3.8.6 | Media Protection | Data encryption at rest |
| RA.L2-3.11.2 | Risk Assessment | Vulnerability scanning |
| RA.L2-3.11.3 | Risk Assessment | Vulnerability remediation |
| SC.L2-3.13.1 | System/Comms | Boundary protection (network segmentation) |
| SC.L2-3.13.8 | System/Comms | Cryptographic mechanisms for CUI |
| SC.L2-3.13.11 | System/Comms | FIPS-validated cryptography |
| SI.L2-3.14.1 | System Integrity | Flaw identification and remediation |
| SI.L2-3.14.6 | System Integrity | System monitoring |

## Compliance Evidence Collection

To generate evidence for an audit:

1. **Access control evidence**: Export user list and role assignments from the Users API
2. **Audit log evidence**: Use the Audit Query API to export logs for the audit period
3. **Scan evidence**: Export compliance scan results showing configuration assessment
4. **Encryption evidence**: Document FIPS mode configuration and cipher suite settings
5. **Monitoring evidence**: Export fleet health and liveness data from the monitoring endpoints

OpenWatch serves the REST API over HTTPS on port 8443. Authenticate with a session
cookie obtained from `/api/v1/auth/login`, or with a Bearer token.

```bash
# Query audit events for a date range
curl "https://localhost:8443/api/v1/audit/events?date_from=2026-01-01&date_to=2026-02-17" \
  -H "Authorization: Bearer $TOKEN" > audit_evidence.json

# Export fleet compliance score
curl https://localhost:8443/api/v1/fleet/score \
  -H "Authorization: Bearer $TOKEN" > fleet_score_evidence.json
```

> Note: dedicated compliance-posture and Kensa-framework export endpoints are pending
> a Go-era rewrite. See `api/openapi.yaml` for the current endpoint surface and `specs/`
> for the behavioral contracts.

---

# OpenWatch: User roles and permissions
URL: https://www.hanalyx.com/docs/openwatch/user-roles

**Last Updated:** 2026-06-22 · **Applies to:** OpenWatch 0.2.0-rc series (Go single-binary)

This guide describes the role-based access control (RBAC) system in the Go-era
OpenWatch. It covers the five built-in roles, the permissions they grant, and how
you create users and assign roles from the single `openwatch` binary.

OpenWatch runs as one Go binary that serves the REST API and the embedded React
UI over HTTPS on port `8443`. All RBAC state lives in PostgreSQL. There is no
separate web tier, container runtime, or Python service.

## Source of truth

RBAC is registry-driven. Do not hand-edit role or permission lists; they are
generated from one file.

| Artifact | Path | Role |
|----------|------|------|
| Permission and role registry | `auth/permissions.yaml` | The single source you edit |
| Generated permission constants | `internal/auth/permissions.gen.go` | Typed Go constants (do not edit) |
| Generated role definitions | `internal/auth/roles.gen.go` | Built-in roles with wildcards expanded (do not edit) |
| Design reference | `docs/engineering/rbac_registry.md` | Rationale, codegen workflow, custom-role design |
| API contract | `api/openapi.yaml` | `x-required-permission` per operation, paths under `/api/v1` |

When this guide and the registry disagree, the registry wins. Regenerate the Go
code with `make generate-rbac` after changing `auth/permissions.yaml`.

## Built-in roles

OpenWatch ships five built-in roles. They form a single privilege ladder from
read-only to full administration; there is no parallel "compliance officer" or
"guest" track. Built-in roles are loaded into the `roles` table by migration with
`is_built_in = true`, so the API rejects attempts to modify them.

| Role ID | Description | Permission count |
|---------|-------------|------------------|
| `viewer` | Read-only access across the platform | 16 |
| `auditor` | Read-only plus exception authority and audit export | 20 |
| `ops_lead` | Day-to-day operations: hosts, scans, alerts | 32 |
| `security_admin` | Full security operations including dangerous and license-gated actions | 56 |
| `admin` | Full system administration | All permissions (bare `*` wildcard) |

A user may hold more than one role. Their effective permission set is the union
of every assigned role's permissions.

### `viewer`

Read-only across every domain. Grants `*:read`-style permissions for hosts,
scans, scan templates, compliance state, baselines, exceptions, alerts,
notifications, license, policy, remediation, integrations, audit, system, and
roles, plus `auth:read` for the user's own profile.

Cannot write, execute, export, approve, or administer anything.

### `auditor`

Everything `viewer` has, plus the exception workflow authority an auditor needs:
`exception:request`, `exception:comment`, and `exception:approve`. Adds
`audit:export` (license-gated by the `audit_export` feature) and `auth:write` so
the auditor can manage their own password, MFA, and sessions.

Cannot create or modify hosts, run scans, or touch system configuration.

### `ops_lead`

The day-to-day operator. Adds write and execute authority over the operational
surface: `host:write`, `host:connectivity_check`, `host:intelligence_refresh`,
`credential:read`, `scan:execute`, `scan:cancel`, `scan_template:write`,
`baseline:write`, alert `acknowledge`/`resolve`, `notification:test`,
`remediation:request`, and the exception request/comment verbs.

Cannot delete hosts, manage credentials beyond reading them, approve
remediations, install licenses or policies, or manage users.

### `security_admin`

Full security operations. Grants category wildcards (`host:*`, `credential:*`,
`scan:*`, `scan_template:*`, `baseline:*`, `exception:*`, `alert:*`,
`notification:*`, `remediation:*`, `integration:*`, `audit:*`) plus
`user:read`, `user:write`, `license:install`, and the policy
`reload`/`install` verbs. This includes the dangerous and license-gated actions
`remediation:execute` and `remediation:rollback` (both gated by the
`remediation_execution` feature).

Cannot perform the high-privilege `admin:*` bundle: managing other users' roles,
SSO providers, retention policy, system settings, or `user:delete`.

### `admin`

Full system administration. Holds the bare `*` wildcard, which is reserved
exclusively for this built-in role and cannot be granted to a custom role. Adds
the `admin:*` bundle (`user_manage`, `role_manage`, `retention_policy`,
`sso_provider`, `system_setting`), `user:delete`, `role:assign`, `role:write`,
`license:revoke`, and `system:config_write`.

## Permission model

Permissions are named `resource:action`, both lowercase
(for example `host:read`, `scan:execute`, `remediation:rollback`). The registry
defines 19 categories. Two attributes affect enforcement:

- `dangerous: true` marks destructive or high-impact actions (for example
  `host:delete`, `license:install`, `user:delete`). The UI uses this for
  confirmation prompts and the audit middleware records denials at high priority.
- `license_gated: <feature>` makes a permission inert unless the active license
  enables that feature. A role may grant the permission, but the combined
  RBAC-plus-license middleware denies the call with `402` until the license
  enables it. Today this applies to `audit:export` (`audit_export`) and
  `remediation:execute` / `remediation:rollback` (`remediation_execution`).

Enforcement happens in middleware generated from the OpenAPI `x-required-permission`
extension, so handlers never check RBAC inline. A request with a missing or
insufficient permission returns `403` with `error.code = "authz.permission_denied"`
and emits an `authz.permission_denied` audit event.

## Permissions matrix

`Y` = granted, `-` = not granted. License-gated permissions are marked `(LG)`;
they are granted by the role but require the matching license feature at runtime.

| Permission | viewer | auditor | ops_lead | security_admin | admin |
|------------|:------:|:-------:|:--------:|:--------------:|:-----:|
| `auth:read` | Y | Y | Y | Y | Y |
| `auth:write` | - | Y | Y | Y | Y |
| `user:read` | - | - | - | Y | Y |
| `user:write` | - | - | - | Y | Y |
| `user:delete` | - | - | - | - | Y |
| `host:read` | Y | Y | Y | Y | Y |
| `host:write` | - | - | Y | Y | Y |
| `host:delete` | - | - | - | Y | Y |
| `host:connectivity_check` | - | - | Y | Y | Y |
| `host:intelligence_refresh` | - | - | Y | Y | Y |
| `credential:read` | - | - | Y | Y | Y |
| `credential:write` | - | - | - | Y | Y |
| `credential:delete` | - | - | - | Y | Y |
| `scan:read` | Y | Y | Y | Y | Y |
| `scan:execute` | - | - | Y | Y | Y |
| `scan:cancel` | - | - | Y | Y | Y |
| `scan_template:read` | Y | Y | Y | Y | Y |
| `scan_template:write` | - | - | Y | Y | Y |
| `scan_template:delete` | - | - | - | Y | Y |
| `compliance:read` | Y | Y | Y | Y | Y |
| `baseline:read` | Y | Y | Y | Y | Y |
| `baseline:write` | - | - | Y | Y | Y |
| `baseline:delete` | - | - | - | Y | Y |
| `exception:read` | Y | Y | Y | Y | Y |
| `exception:request` | - | Y | Y | Y | Y |
| `exception:comment` | - | Y | Y | Y | Y |
| `exception:approve` | - | Y | - | Y | Y |
| `exception:revoke` | - | - | - | Y | Y |
| `alert:read` | Y | Y | Y | Y | Y |
| `alert:acknowledge` | - | - | Y | Y | Y |
| `alert:resolve` | - | - | Y | Y | Y |
| `alert:write` | - | - | - | Y | Y |
| `notification:read` | Y | Y | Y | Y | Y |
| `notification:write` | - | - | - | Y | Y |
| `notification:delete` | - | - | - | Y | Y |
| `notification:test` | - | - | Y | Y | Y |
| `license:read` | Y | Y | Y | Y | Y |
| `license:install` | - | - | - | Y | Y |
| `license:revoke` | - | - | - | - | Y |
| `policy:read` | Y | Y | Y | Y | Y |
| `policy:reload` | - | - | - | Y | Y |
| `policy:install` | - | - | - | Y | Y |
| `remediation:read` | Y | Y | Y | Y | Y |
| `remediation:request` | - | - | Y | Y | Y |
| `remediation:approve` | - | - | - | Y | Y |
| `remediation:execute` (LG) | - | - | - | Y | Y |
| `remediation:rollback` (LG) | - | - | - | Y | Y |
| `integration:read` | Y | Y | Y | Y | Y |
| `integration:write` | - | - | - | Y | Y |
| `integration:execute` | - | - | - | Y | Y |
| `audit:read` | Y | Y | Y | Y | Y |
| `audit:export` (LG) | - | Y | - | Y | Y |
| `system:read` | Y | Y | Y | Y | Y |
| `system:config_write` | - | - | - | - | Y |
| `role:read` | Y | - | - | - | Y |
| `role:write` | - | - | - | - | Y |
| `role:assign` | - | - | - | - | Y |
| `admin:user_manage` | - | - | - | - | Y |
| `admin:role_manage` | - | - | - | - | Y |
| `admin:retention_policy` | - | - | - | - | Y |
| `admin:sso_provider` | - | - | - | - | Y |
| `admin:system_setting` | - | - | - | - | Y |

`security_admin` grants `audit:*`, which includes `audit:export`; the `auditor`
row grants `audit:export` explicitly. Both depend on the `audit_export` license
feature at runtime.

## Creating the first admin

The first admin is created from the CLI, not the API. The `create-admin`
subcommand creates the user and assigns the built-in `admin` role in one step. It
requires `--username` and `--email`; the password is read from stdin when
`--password` is omitted, and is held to the 15-character admin policy.

```bash
sudo -u openwatch env $(cat /etc/openwatch/secrets.env | xargs) \
  openwatch --config /etc/openwatch/openwatch.toml \
  create-admin --username admin --email admin@example.com
```

The command connects to PostgreSQL using `OPENWATCH_DATABASE_DSN` from
`/etc/openwatch/secrets.env` and exits non-zero if the user is created but the
role assignment fails, so you can detect a partial state. See
`docs/guides/INSTALLATION.md` for the full install sequence
(`openwatch migrate`, `create-admin`, `systemctl enable --now openwatch`).

## Managing users and roles through the API

Once an admin exists, manage users over HTTPS at `https://<host>:8443` under
`/api/v1`. Authenticate at `POST /api/v1/auth/login` to obtain a bearer token,
then call the user and role endpoints. The required permission for each is below.

| Operation | Method and path | Required permission |
|-----------|-----------------|---------------------|
| List users | `GET /api/v1/users` | `user:read` |
| Fetch a user | `GET /api/v1/users/{id}` | `user:read` |
| Create a user | `POST /api/v1/users` | `user:write` |
| Soft-delete a user | `DELETE /api/v1/users/{id}` | `user:delete` |
| Assign a role | `POST /api/v1/users/{id}/roles:assign` | `role:assign` |
| Remove a role | `POST /api/v1/users/{id}/roles:unassign` | `role:assign` |
| List built-in roles | `GET /api/v1/roles` | `role:read` |
| Create a custom role | `POST /api/v1/roles:create` | `role:write` |
| Effective permissions for the caller | `GET /api/v1/auth/me/permissions` | authenticated |
| Full RBAC registry | `GET /api/v1/auth/permissions:registry` | authenticated |

Creating a user does not assign a role. `POST /api/v1/users` takes only
`username`, `email`, and `password`; role assignment is a separate
`roles:assign` call. Among the built-in roles, only `admin` holds `role:assign`
and `user:delete`.

Assign a role by posting the role id:

```bash
curl -sS -X POST "https://<host>:8443/api/v1/users/<user_id>/roles:assign" \
  -H "Authorization: Bearer <admin_token>" \
  -H "Content-Type: application/json" \
  -d '{"role_id": "ops_lead"}'
```

A `roles:assign` call with an unknown role id returns `400`. The
`roles:unassign` call is idempotent and returns `204` whether or not the role
was present.

## Custom roles

The registry supports custom, DB-stored roles created at runtime via
`POST /api/v1/roles:create` (requires `role:write`). A custom role may grant any
registry permission and category wildcards such as `host:*`, but not the bare
`*` wildcard, which is reserved for the built-in `admin` role. Every permission a
custom role lists is validated against the registry; unknown permissions are
rejected with `400`.

For the custom-role design, validation rules, and the relationship between
wildcards and newly added permissions, see `docs/engineering/rbac_registry.md`.

## Related documentation

- `docs/engineering/rbac_registry.md` — RBAC design, codegen workflow, custom roles
- `docs/guides/INSTALLATION.md` — install, `migrate`, `create-admin`, service start
- `api/openapi.yaml` — API contract and `x-required-permission` per operation
- `auth/permissions.yaml` — the editable permission and role registry

---

# OpenWatch: OpenWatch monitoring and operations guide
URL: https://www.hanalyx.com/docs/openwatch/monitoring-setup

**Last updated**: 2026-06-10

This guide describes how you monitor a running OpenWatch deployment and how you
respond to common operational incidents. OpenWatch ships as a single Go binary
(`/usr/bin/openwatch`) that serves the REST API and the embedded React UI over
HTTPS on port `8443`, backed by PostgreSQL and managed by systemd. There is no
container runtime, no separate web tier, and no message broker.

For installation, configuration layering, and first-run setup, see
`docs/guides/INSTALLATION.md`. This guide does not repeat those steps; it
focuses on observing the service and running it day to day.

## Contents

1. [What you can observe today](#1-what-you-can-observe-today)
2. [Health and version endpoints](#2-health-and-version-endpoints)
3. [Logs via journald](#3-logs-via-journald)
4. [Audit events](#4-audit-events)
5. [Fleet and connectivity signals](#5-fleet-and-connectivity-signals)
6. [Service lifecycle](#6-service-lifecycle)
7. [Operational runbooks](#7-operational-runbooks)
8. [Not yet implemented](#8-not-yet-implemented)

## 1. What you can observe today

OpenWatch exposes operational signals through four channels:

| Channel | Source | Authentication |
|---------|--------|----------------|
| Health probe | `GET /api/v1/health` | None |
| Version metadata | `GET /api/v1/version` | None |
| Structured logs | systemd journal (`journalctl -u openwatch`) | Host access |
| Audit and fleet APIs | `GET /api/v1/audit/events`, `/api/v1/fleet/*`, `/api/v1/system/connectivity/status` | Bearer token |

OpenWatch does not currently expose a Prometheus `/metrics` endpoint and does
not ship a Prometheus, Grafana, Jaeger, or exporter stack. See
[Not yet implemented](#8-not-yet-implemented).

## 2. Health and version endpoints

### Health probe

The health endpoint is anonymous and is the right target for an external uptime
check or a load-balancer probe. It is implemented in
`internal/server/handlers.go` (`GetHealth`) and pings PostgreSQL with a
two-second timeout.

```bash
curl -k https://localhost:8443/api/v1/health
```

A healthy response returns `200 OK`:

```json
{"status": "healthy", "db_connected": true, "version": "0.2.0-rc.13"}
```

When the database ping fails, the endpoint returns `503 Service Unavailable`
with an error envelope. Treat a non-`200` status, or a connection failure, as
service-down.

The response schema (`status`, `db_connected`, `version`) is defined in
`api/openapi.yaml` under `HealthResponse`. The current contract reports only a
binary `healthy`/`degraded` status driven by database reachability.

### Version metadata

The version endpoint is also anonymous and reports build metadata sourced from
ldflags and Go build info (`internal/server/handlers.go`, `GetVersion`):

```bash
curl -k https://localhost:8443/api/v1/version
```

```json
{
  "openwatch": "0.2.0-rc.13",
  "kensa": "<embedded engine version>",
  "go": "<go toolchain>",
  "commit": "<abbrev commit>",
  "build_time": "<ISO-8601 build timestamp>"
}
```

Use this to confirm which build is running after an upgrade. The same metadata
prints from the CLI with `openwatch --version`.

## 3. Logs via journald

The systemd unit (`packaging/common/openwatch.service`) sends both stdout and
stderr to the journal. With `format = "json"` in `[logging]` (the packaged
default in `packaging/common/openwatch.toml`), every line is a structured JSON
record that carries a correlation ID.

```bash
sudo journalctl -u openwatch -f                    # tail live
sudo journalctl -u openwatch --since '15 min ago'  # recent window
sudo journalctl -u openwatch -o cat | jq .         # pretty-print JSON
sudo journalctl -u openwatch -p err --since today  # errors only
```

To trace one request or one boot across log lines, filter on its correlation
ID:

```bash
sudo journalctl -u openwatch -o cat | jq 'select(.correlation_id == "<id>")'
```

Set `level = "debug"` in `[logging]` (or pass `--log-level debug`, or set
`OPENWATCH_LOGGING_LEVEL=debug`) to raise verbosity, then restart the service.
Log level precedence follows the standard config layering documented in
`docs/guides/INSTALLATION.md`.

## 4. Audit events

Every server action that mutates state, authenticates, or authorizes emits a
row to the `audit_events` PostgreSQL table (migrations `0001_initial.sql` and
`0002_audit_events_taxonomy.sql`). This is the durable record for security
review; the journal is the operational record.

Query audit events through the API (requires a bearer token with the
appropriate permission):

```bash
curl -k -H "Authorization: Bearer $TOKEN" \
  'https://localhost:8443/api/v1/audit/events'
```

The endpoint (`getAuditEvents` in `api/openapi.yaml`) is cursor-paginated. For
direct inspection during an incident you can also read the table with `psql`:

```bash
psql "$OPENWATCH_DATABASE_DSN" -c \
  "SELECT recorded_at, action, severity, actor_type, actor_id, outcome
   FROM audit_events
   ORDER BY recorded_at DESC
   LIMIT 50;"
```

Indexed columns include `recorded_at`, `action`, `severity`, and
`(actor_type, actor_id)`, so filtered queries on those fields stay fast. For the
event taxonomy (action names and severities), see
`docs/engineering/audit_event_taxonomy.md`.

## 5. Fleet and connectivity signals

OpenWatch continuously probes managed hosts (the liveness loop wired in
`cmd/openwatch/main.go`). These endpoints expose the resulting fleet state and
require a bearer token:

| Endpoint | Reports |
|----------|---------|
| `GET /api/v1/fleet/liveness` | Counts: `reachable`, `unreachable`, `unknown`, `never_probed` |
| `GET /api/v1/fleet/connectivity/breakdown` | 4-state breakdown: `online`, `degraded`, `critical`, `down`, `never_probed` |
| `GET /api/v1/system/connectivity/status` | In-process connectivity-monitor metrics and the maintenance flag |

```bash
curl -k -H "Authorization: Bearer $TOKEN" \
  https://localhost:8443/api/v1/fleet/liveness
```

A rising `unreachable`/`down` count is a useful early signal that either the
monitored fleet or the OpenWatch host's network path is degrading. The schemas
(`FleetLiveness`, `ConnectivityBreakdown`) are defined in `api/openapi.yaml`.

## 6. Service lifecycle

OpenWatch runs as the `openwatch.service` systemd unit, which executes
`openwatch serve --config /etc/openwatch/openwatch.toml`.

```bash
sudo systemctl status openwatch     # current state
sudo systemctl restart openwatch    # restart
sudo systemctl stop openwatch       # stop
sudo systemctl enable --now openwatch  # start now and at boot
```

Before restarting after a config change, validate the resolved configuration:

```bash
sudo -u openwatch openwatch check-config --config /etc/openwatch/openwatch.toml
```

Other CLI subcommands (`cmd/openwatch/main.go`): `migrate` applies pending
database migrations, `create-admin` bootstraps the first admin user, and
`worker` runs the background scan-job loop. The packaged systemd unit runs only
`serve`; the in-process schedulers and liveness loop run inside the `serve`
process.

## 7. Operational runbooks

These runbooks assume the single binary on systemd with a PostgreSQL backend.
Run the commands from the OpenWatch host unless noted.

### SERVICE_DOWN

The service is unreachable or `GET /api/v1/health` does not return `200`.

1. Check the unit state and recent errors:

   ```bash
   sudo systemctl status openwatch
   sudo journalctl -u openwatch --since '10 min ago' -p err
   ```

2. Confirm the local probe:

   ```bash
   curl -k https://localhost:8443/api/v1/health
   ```

3. If the journal shows a database ping failure (for example
   `db: ping: ... connection refused`), check PostgreSQL:

   ```bash
   sudo systemctl status postgresql
   psql "$OPENWATCH_DATABASE_DSN" -c 'SELECT 1;'
   ```

4. If the config is suspect, validate it before restarting:

   ```bash
   sudo -u openwatch openwatch check-config --config /etc/openwatch/openwatch.toml
   ```

5. Restart and confirm recovery:

   ```bash
   sudo systemctl restart openwatch
   curl -k https://localhost:8443/api/v1/health
   ```

The unit is configured with `Restart=on-failure` and `RestartSec=5s`, so a
crashing process restarts automatically; persistent restart loops show up in
`systemctl status` as repeated restarts and warrant the steps above.

### DISK_FULL

Disk pressure on the OpenWatch or PostgreSQL data volume.

1. Find what is full:

   ```bash
   df -h
   sudo du -xh /var/log/openwatch /var/lib/openwatch | sort -h | tail
   ```

2. The journal is a common consumer. Inspect and cap it:

   ```bash
   journalctl --disk-usage
   sudo journalctl --vacuum-time=7d      # drop entries older than 7 days
   sudo journalctl --vacuum-size=500M    # or cap total size
   ```

3. Check the PostgreSQL data directory and database size:

   ```bash
   psql "$OPENWATCH_DATABASE_DSN" -c \
     "SELECT pg_size_pretty(pg_database_size(current_database()));"
   ```

   The `audit_events` table grows over time. Confirm its size before pruning,
   and follow your retention policy:

   ```bash
   psql "$OPENWATCH_DATABASE_DSN" -c \
     "SELECT pg_size_pretty(pg_total_relation_size('audit_events'));"
   ```

4. After freeing space, confirm the service is healthy
   (`curl -k https://localhost:8443/api/v1/health`).

OpenWatch does not currently rotate or prune `audit_events` automatically; apply
your own retention if the table dominates database size.

### HIGH_CPU

The OpenWatch process is consuming excessive CPU.

1. Confirm which process and how much:

   ```bash
   top -b -n1 | head -20
   sudo systemctl status openwatch     # shows the main PID
   ```

2. Correlate with request and scan activity in the journal:

   ```bash
   sudo journalctl -u openwatch --since '15 min ago' -o cat | jq -r '.msg' | sort | uniq -c | sort -rn | head
   ```

3. Check whether background work is driving load. The liveness loop and the
   intelligence and discovery schedulers run inside `serve`. If a scheduler is
   misconfigured, pause it via its config endpoint, for example:

   ```bash
   curl -k -H "Authorization: Bearer $TOKEN" \
     https://localhost:8443/api/v1/system/intelligence/config
   ```

4. Check PostgreSQL for long-running or stuck queries:

   ```bash
   psql "$OPENWATCH_DATABASE_DSN" -c \
     "SELECT pid, now()-query_start AS runtime, state, left(query,80)
      FROM pg_stat_activity
      WHERE state <> 'idle'
      ORDER BY runtime DESC NULLS LAST LIMIT 10;"
   ```

5. If the process is wedged rather than merely busy, capture the journal context
   first, then `sudo systemctl restart openwatch`.

### SECURITY_INCIDENT

Suspected unauthorized access, credential misuse, or anomalous authorization
failures.

1. Pull recent authentication and authorization events from the audit log. These
   are the durable security record:

   ```bash
   psql "$OPENWATCH_DATABASE_DSN" -c \
     "SELECT recorded_at, action, severity, actor_type, actor_id, outcome
      FROM audit_events
      WHERE severity IN ('warning','critical')
         OR action LIKE 'auth.%'
      ORDER BY recorded_at DESC
      LIMIT 100;"
   ```

2. Cross-reference with the journal for the same window, filtering by correlation
   ID where you have one:

   ```bash
   sudo journalctl -u openwatch --since '1 hour ago' -o cat | jq 'select(.level=="WARN" or .level=="ERROR")'
   ```

3. If you must contain immediately, stop the service to halt all access while you
   investigate:

   ```bash
   sudo systemctl stop openwatch
   ```

4. Rotate any potentially exposed secrets in `/etc/openwatch/secrets.env` (for
   example `OPENWATCH_DATABASE_DSN`) and the keys under `/etc/openwatch/`, then
   restart. Preserve the journal and a copy of relevant `audit_events` rows
   before you prune anything.

For role and permission definitions referenced by audit `action`/`actor` fields,
see `docs/engineering/rbac_registry.md`.

## 8. Not yet implemented

The following observability capabilities described in earlier (Python-era)
documentation do not exist in the current Go build. They are recorded here so
operators do not look for them:

- **Prometheus `/metrics` endpoint** — not exposed. The only health signal is
  `GET /api/v1/health`.
- **Bundled monitoring stack** (Prometheus, Grafana, Jaeger, Alertmanager,
  node/redis/postgres exporters, cAdvisor) — not shipped. There is no
  `monitoring/` Compose stack and no container runtime in this architecture.
- **Distributed tracing** — not implemented. Correlation IDs in the JSON logs
  are the current mechanism for following a request across log lines.
- **Detailed authenticated health endpoints** (per-service, content, history) —
  not implemented. The current health contract is a single binary
  `healthy`/`degraded` status.

If and when metrics or tracing land, this section and the contract in
`api/openapi.yaml` will be updated together.

---

# OpenWatch: Production deployment guide
URL: https://www.hanalyx.com/docs/openwatch/production-deployment

**Last Updated:** 2026-06-22 · **Applies to:** OpenWatch 0.2.0-rc series (Go single-binary)

This guide covers running OpenWatch in production: a single Go binary that serves
the REST API and the embedded React UI over HTTPS, backed by PostgreSQL, managed
by `systemd`. There is no container runtime, no separate web tier, no Redis, and
no Celery — those belonged to the archived Python stack and are gone.

For first-time install and database provisioning, follow the canonical
[install guide](/docs/openwatch/installation). This document does **not**
repeat those steps; it focuses on production concerns the install guide only
touches lightly: process layout, TLS, the background worker, backups, upgrades,
and incident runbooks.

> Verify the version you deploy. The current line is a pre-release
> (`0.2.0-rc.13` per `packaging/version.env`), not a GA build. Treat it
> accordingly until a GA tag ships.

---

## Architecture in production

One binary, `/usr/bin/openwatch`, provides every runtime role through
subcommands (`cmd/openwatch/main.go`):

| Subcommand | Role | Long-running |
|------------|------|--------------|
| `serve` | HTTPS API + embedded UI, schedulers (liveness, intelligence, discovery), event bus, alert router | Yes — this is the service unit |
| `worker` | Scan-job claimer/dispatcher loop; drains the PostgreSQL job queue and runs Kensa scans | Yes — optional separate unit |
| `migrate` | Apply pending database migrations, then exit | No |
| `create-admin` | Create the first admin user, then exit | No |
| `check-config` | Validate and print the resolved config (secrets redacted), then exit | No |

The packaged `openwatch.service` runs `openwatch serve`
(`packaging/common/openwatch.service`). The `serve` process also runs the
in-process schedulers, so a minimal single-node deployment needs only that one
unit. The `worker` subcommand exists for separating scan execution onto its own
process or host; it is HTTP-free and shares the same boot prerequisites
(config, DB pool, audit, license, JWT key, credential DEK) as `serve`. There is
no packaged `worker` unit yet — running `worker` as its own service is a manual
step today (write a unit that runs `ExecStart=/usr/bin/openwatch worker`).

| Component | Where | Notes |
|-----------|-------|-------|
| API + UI | `https://<host>:8443/` | UI embedded via `go:embed`; API under `/api/v1/` |
| Database | PostgreSQL 14+ | The only datastore. Not provisioned by the package. |
| Job queue | PostgreSQL table, `SKIP LOCKED` | No external broker. Drained by `serve`/`worker`. |
| Compliance engine | Kensa (Go), in-process | SSH-based, native YAML rules. See [the boundary doc](../KENSA_OPENWATCH_BOUNDARY.md). |


---

## Prerequisites

See the [install guide requirements](/docs/openwatch/installation#requirements)
for the authoritative list. In short:

- A supported RHEL-family or Debian-family host with `systemd`.
- PostgreSQL 14 or newer, reachable from the OpenWatch host.
- TCP/8443 inbound (API + UI); TCP/22 outbound to every managed host (Kensa scans
  over SSH).
- A CA-signed TLS certificate for any non-loopback use.

---

## Install and first run

Follow the [install guide](/docs/openwatch/installation) end to end:

1. Install and provision PostgreSQL.
2. Install the signed `.rpm`/`.deb` (verify `SHA256SUMS.asc` against `KEYS` first).
3. Set `OPENWATCH_DATABASE_DSN` in `/etc/openwatch/secrets.env`.
4. Run `openwatch migrate`.
5. Run `openwatch create-admin --username admin --email you@example.com`.
6. `systemctl enable --now openwatch`.
7. Sign in at `https://<host>:8443/`.

Before starting the service, validate the resolved configuration:

```bash
sudo -u openwatch env $(cat /etc/openwatch/secrets.env | xargs) \
    openwatch check-config
```

It prints the effective config with secrets redacted and exits non-zero if the
config is invalid.

---

## Configuration

Config layers, highest precedence first (`cmd/openwatch/main.go`):

1. CLI flags (`--listen`, `--log-level`, `--config`)
2. Environment variables (`OPENWATCH_<SECTION>_<KEY>`)
3. The TOML file (`/etc/openwatch/openwatch.toml`)
4. Built-in defaults

The TOML file has four sections (`internal/config/config.go`,
`packaging/common/openwatch.toml`):

| Section | Key | Default | Purpose |
|---------|-----|---------|---------|
| `[server]` | `listen` | `0.0.0.0:8443` | Bind address and port for API + UI |
| `[server]` | `tls_cert` | `/etc/openwatch/tls/cert.pem` | TLS certificate |
| `[server]` | `tls_key` | `/etc/openwatch/tls/key.pem` | TLS private key |
| `[database]` | `dsn` | — | PostgreSQL DSN (set via `secrets.env` in production) |
| `[database]` | `max_connections` | `25` | Connection-pool ceiling |
| `[logging]` | `level` | `info` | `debug` / `info` / `warn` / `error` |
| `[logging]` | `format` | `json` | `json` / `text` |

Two more values come from `[identity]` and must be set for `serve`/`worker` to
boot — the JWT signing key (`jwt_private_key`) and the credential DEK file
(`credential_key_file`). Without them the process exits at startup rather than
running with a silent fallback (see `cmdServe` in `cmd/openwatch/main.go`).

Keep the database password out of the world-readable TOML by putting the DSN in
`/etc/openwatch/secrets.env`, which the `systemd` unit loads via
`EnvironmentFile`:

```
OPENWATCH_DATABASE_DSN=postgres://openwatch:STRONG_PW@db.internal:5432/openwatch?sslmode=require
```

Set the file to `0640`, owner `root:openwatch`. Use `sslmode=require` or stronger
for any PostgreSQL not on the loopback interface.

---

## TLS

The package ships a self-signed certificate so the service starts out of the box.
Replace it before any non-loopback use:

```bash
sudo cp your-cert.pem /etc/openwatch/tls/cert.pem
sudo cp your-key.pem  /etc/openwatch/tls/key.pem
sudo chown root:openwatch      /etc/openwatch/tls/cert.pem
sudo chown openwatch:openwatch /etc/openwatch/tls/key.pem
sudo chmod 0644 /etc/openwatch/tls/cert.pem
sudo chmod 0600 /etc/openwatch/tls/key.pem
sudo systemctl restart openwatch
```

The server reads the certificate on each TLS handshake, so new connections pick
up a swapped cert without a restart; restart anyway to cover existing keep-alive
connections.

---

## Service hardening

The packaged unit (`packaging/common/openwatch.service`) already applies
`systemd` sandboxing. Keep these in place:

| Directive | Value | Effect |
|-----------|-------|--------|
| `User` / `Group` | `openwatch` | Runs unprivileged |
| `NoNewPrivileges` | `true` | No setuid escalation |
| `ProtectSystem` | `strict` | Read-only filesystem except `ReadWritePaths` |
| `ProtectHome` | `true` | No access to `/home`, `/root` |
| `PrivateTmp` | `true` | Private `/tmp` |
| `ReadWritePaths` | `/var/lib/openwatch /var/log/openwatch` | Only writable paths |
| `RestrictAddressFamilies` | `AF_INET AF_INET6 AF_UNIX` | No raw sockets unless granted |
| `Restart` | `on-failure` (`RestartSec=5s`) | Auto-restart on crash |

If you front OpenWatch with a reverse proxy or load balancer, terminate or
pass through TLS to 8443 — there is no separate HTTP listener to target.

---

## Health, version, and logs

OpenWatch exposes two anonymous endpoints for probes (`api/openapi.yaml`):

```bash
curl -k https://localhost:8443/api/v1/health
# 200 {"status":"healthy","db_connected":true,"version":"..."}
# 503 when the database ping fails (status "degraded"/unavailable)

curl -k https://localhost:8443/api/v1/version
# {"openwatch":"...","kensa":"...","go":"...","commit":"...","build_time":"..."}
```

`/api/v1/health` returns `200` with `db_connected:true` when healthy and `503`
when the database ping inside the handler fails. Use it as your liveness and
readiness probe.

The service logs structured JSON to journald:

```bash
sudo journalctl -u openwatch -f                   # tail live
sudo journalctl -u openwatch --since '5 min ago'  # recent
sudo journalctl -u openwatch -o cat | jq .        # pretty-print
```

> Prometheus-style `/metrics` scraping is **not yet implemented** — there is no
> HTTP metrics endpoint. The in-process connectivity-monitor metrics are exposed
> only through the authenticated `GET /api/v1/system/connectivity/status`
> endpoint, not via an open-text scrape target. For production observability
> today, rely on `/api/v1/health` and the journald logs.

---

## Background scan worker

The `serve` process drains the job queue on its own, so a single-node install
needs nothing extra. To run scan execution as a dedicated process (separate
resource limits, or a separate host), run the `worker` subcommand:

```bash
sudo -u openwatch env $(cat /etc/openwatch/secrets.env | xargs) \
    openwatch worker --poll-interval 1s
```

`--poll-interval` controls the empty-queue sleep between dequeue attempts
(`worker.DefaultPollInterval`, 1s default, 5s max — `cmd/openwatch/worker.go`).
The worker needs the same `secrets.env`, JWT key, and credential DEK as `serve`,
because it decrypts host credentials and derives the queue HMAC key from the DEK.
There is no packaged worker unit; if you split it out, model a unit on
`openwatch.service` with `ExecStart=/usr/bin/openwatch worker`.

---

## Upgrades

```bash
# RHEL family
sudo dnf upgrade ./openwatch-<new-version>.rpm

# Debian family
sudo apt install ./openwatch_<new-version>_amd64.deb
```

After the package upgrade:

```bash
sudo -u openwatch env $(cat /etc/openwatch/secrets.env | xargs) openwatch migrate
sudo systemctl restart openwatch
```

`openwatch migrate` applies any new migrations from
`internal/db/migrations/` and is a safe no-op when the schema is already current.
Restart the service (and the worker, if you run one separately) to load the new
binary. Take a database backup before upgrading (see below). Config under
`/etc/openwatch/` is preserved across package upgrades.

---

## Backup and restore

OpenWatch keeps all durable state in PostgreSQL. Back up the database with the
standard PostgreSQL tooling — there is no OpenWatch-specific backup command.

```bash
# Backup
pg_dump -h 127.0.0.1 -U openwatch -d openwatch -Fc -f openwatch-$(date -u +%Y-%m-%dT%H-%M-%SZ).dump

# Restore into a freshly created, empty database
pg_restore -h 127.0.0.1 -U openwatch -d openwatch --clean --if-exists openwatch-<timestamp>.dump
```

Also back up `/etc/openwatch/` — it holds the TLS material, the JWT signing key,
the credential DEK, and `secrets.env`. Losing the credential DEK makes stored SSH
credentials and MFA secrets unrecoverable. Test restores periodically; a backup
you have never restored is a hypothesis, not a backup.

---

## Operational runbooks

Concise, single-binary runbooks follow. Diagnose with `systemctl`, `journalctl`,
`psql`, `df`, and `top` — not `docker`. The standalone files under
[`docs/runbooks/`](../runbooks/) still describe the archived Python/Docker stack
and are pending a Go-era rewrite; prefer the steps below until they are updated.

### SERVICE_DOWN — service unavailable

Symptoms: `https://<host>:8443/` refuses connections, or `/api/v1/health` times
out or returns `503`.

```bash
sudo systemctl status openwatch
sudo journalctl -u openwatch --since '10 min ago' -p err
curl -k https://localhost:8443/api/v1/health
```

1. If the unit is `failed`/`inactive`, read the journal for the boot error.
   Common causes: malformed `OPENWATCH_DATABASE_DSN`, unreadable TLS cert, or a
   missing `jwt_private_key` / `credential_key_file`.
2. If `/api/v1/health` returns `503` with `db_connected:false`, treat it as a
   database problem (see DATABASE_ISSUES below).
3. Confirm the config is valid, then restart:
   ```bash
   sudo -u openwatch env $(cat /etc/openwatch/secrets.env | xargs) openwatch check-config
   sudo systemctl restart openwatch
   ```
4. Verify recovery: `curl -k https://localhost:8443/api/v1/health` returns `200`.

### DATABASE_ISSUES — database connectivity

Symptoms: `/api/v1/health` reports `db_connected:false`; journal shows
`db: ping:` errors.

```bash
sudo systemctl status postgresql
PGPASSWORD=... psql -h 127.0.0.1 -U openwatch -d openwatch -c 'SELECT 1;'
```

1. `connection refused` → PostgreSQL is down: `sudo systemctl restart postgresql`.
2. `password authentication failed` → the DSN in `secrets.env` or `pg_hba.conf`
   is wrong. Recheck both, then `sudo systemctl reload postgresql`.
3. Pool exhaustion under load → raise `[database].max_connections` (and the
   server's `max_connections`), then restart OpenWatch.
4. Re-test with `curl -k https://localhost:8443/api/v1/health`.

### DISK_FULL — disk space exhausted

Symptoms: writes fail; the journal shows `no space left on device`; the service
may crash-loop.

```bash
df -h
sudo du -xh /var/log/openwatch /var/lib/openwatch 2>/dev/null | sort -rh | head
sudo du -xh /var/lib/pgsql /var/lib/postgresql 2>/dev/null | sort -rh | head
```

1. Reclaim journald space if logs dominate:
   ```bash
   sudo journalctl --vacuum-size=500M
   ```
2. If PostgreSQL data is large, check table bloat and run maintenance:
   ```bash
   psql -h 127.0.0.1 -U openwatch -d openwatch -c "\
     SELECT relname, pg_size_pretty(pg_total_relation_size(relid)) AS size \
     FROM pg_catalog.pg_statio_user_tables ORDER BY pg_total_relation_size(relid) DESC LIMIT 10;"
   psql -h 127.0.0.1 -U openwatch -d openwatch -c "VACUUM (ANALYZE);"
   ```
3. The transaction-log/write-on-change model bounds growth (`transactions` plus a
   fixed-size `host_rule_state`), so unbounded growth usually means audit/event
   retention or PostgreSQL WAL — not scan results. Investigate before deleting.
4. After freeing space: `sudo systemctl restart openwatch` and re-check `df -h`.

### HIGH_CPU — sustained high CPU

Symptoms: load high; UI/API latency up.

```bash
top -b -n1 | head -20
sudo journalctl -u openwatch --since '15 min ago' | jq -r 'select(.level=="WARN" or .level=="ERROR")'
psql -h 127.0.0.1 -U openwatch -d openwatch -c "\
  SELECT pid, state, wait_event_type, left(query,80) FROM pg_stat_activity \
  WHERE datname='openwatch' ORDER BY state;"
```

1. If `postgres` backends dominate, look for long-running or stuck queries in
   `pg_stat_activity` and for a missing `VACUUM`/`ANALYZE`.
2. If the `openwatch` process dominates, a scan burst or a tight scheduler loop is
   the usual cause. The intelligence and discovery schedulers can be paused
   without a restart by setting `maintenance_global=true`:
   ```bash
   # via the API, as an admin token:
   # PUT /api/v1/system/intelligence/config   {"maintenance_global": true}
   # PUT /api/v1/system/discovery/config      {"maintenance_global": true}
   ```
3. If a separate `worker` is saturating the host, raise `--poll-interval` toward
   its 5s ceiling, or move the worker to its own host.
4. Re-check `top` and API latency after each change; re-enable maintenance flags
   when load normalizes.

### SECURITY_INCIDENT — suspected compromise

1. **Preserve evidence first.** Do not wipe the host. Capture the journal and the
   audit trail:
   ```bash
   sudo journalctl -u openwatch --since '24 hours ago' > /tmp/openwatch-journal.log
   ```
   OpenWatch writes an immutable audit event stream to PostgreSQL (see
   `docs/engineering/audit_event_taxonomy.md`); export the relevant range via the
   audit query API or `psql` before any remediation.
2. **Contain.** Stop accepting traffic without destroying state:
   ```bash
   sudo systemctl stop openwatch
   ```
   Or block 8443 at the firewall if you need the process alive for forensics.
3. **Rotate secrets** in `/etc/openwatch/` — the JWT signing key, credential DEK,
   and database password. Rotating the JWT key invalidates all sessions
   (everyone re-authenticates). Rotating the DEK requires re-encrypting stored
   credentials; plan that change deliberately.
4. **Review access.** Audit admin accounts and roles
   (`docs/engineering/rbac_registry.md`) and revoke anything unexpected.
5. **Restore from a known-good backup** only after the root cause is understood
   (see Backup and restore above).
6. Document the timeline and follow your organization's incident process.

---

## Quick reference

| Item | Value |
|------|-------|
| UI + API | `https://<host>:8443/` (API under `/api/v1/`) |
| Binary | `/usr/bin/openwatch` |
| Service unit | `openwatch.service` (runs `openwatch serve`) |
| Config | `/etc/openwatch/openwatch.toml` |
| DB secret | `/etc/openwatch/secrets.env` (`OPENWATCH_DATABASE_DSN`) |
| TLS | `/etc/openwatch/tls/{cert,key}.pem` |
| Data / logs | `/var/lib/openwatch`, `/var/log/openwatch`, journald |
| User / group | `openwatch:openwatch` |
| Health probe | `GET /api/v1/health` |
| Migrate | `openwatch migrate` |
| Restart | `sudo systemctl restart openwatch` |
| Logs | `journalctl -u openwatch -f` |

---

## See also

- [Install guide](/docs/openwatch/installation) — canonical install and provisioning.
- [Kensa ↔ OpenWatch boundary](../KENSA_OPENWATCH_BOUNDARY.md) — compliance engine integration.
- [RBAC registry](../engineering/rbac_registry.md) — roles and permissions.
- [API contract](../../api/openapi.yaml) — every endpoint, its permission, and audit events.
- [Releasing runbook](../runbooks/RELEASING.md) — building and signing releases.

---

# OpenWatch: Scaling guide
URL: https://www.hanalyx.com/docs/openwatch/scaling-guide

**Last Updated:** 2026-06-22 · **Applies to:** OpenWatch 0.2.0-rc series (Go single-binary)

This guide covers how OpenWatch behaves as you add hosts, run more scans, and
push more concurrent API traffic, and what you can tune today. It describes the
current Go-era stack: a single `openwatch` binary that serves the REST API and
the embedded React UI over HTTPS on port `8443`, backed by PostgreSQL, with the
Kensa compliance engine built in. There is no separate web tier, no container
runtime, no Redis, and no Celery.

For first-time install and configuration, follow
`docs/guides/INSTALLATION.md` — this guide assumes a working install and
focuses only on capacity and tuning.

## What scales, and how

OpenWatch has two long-lived processes and one database:

| Component | What it does | How you scale it today |
|-----------|--------------|------------------------|
| `openwatch serve` | HTTPS API + embedded UI + in-process schedulers (liveness, intelligence, discovery) **and an in-process worker that drains the scan-job queue** | Raise `[server].scan_concurrency` (how many scans run at once in this process); then vertical CPU/RAM. Stateless apart from PostgreSQL. |
| `openwatch worker` | An **optional, additional** process that also drains the scan-job queue and runs Kensa scans over SSH | Run one or more for extra/off-box capacity. The queue uses `SELECT ... FOR UPDATE SKIP LOCKED`, so the serve worker and any `openwatch worker` processes cooperate without double-claiming a job. |
| PostgreSQL | All state: hosts, scans, transactions, audit events, queue | Vertical first (CPU, RAM, faster disk), then tune `max_connections` and the OpenWatch pool size. |

`openwatch serve` runs an in-process worker that **does** drain the scan-job
queue — the single-binary deployment scans with no extra process. By default it
runs **`scan_concurrency` (4) scans concurrently** (`internal/worker/worker.go`,
wired in `internal/server/server.go`). A separate `openwatch worker` is
optional, for additional or off-box capacity.

## Scaling the scan workers

Scans are the most resource-intensive work OpenWatch does: each one opens an SSH
session to a target host and runs Kensa's native YAML checks. Worker throughput
is the usual first bottleneck.

### Scan concurrency (the first knob to turn)

The in-process worker runs `[server].scan_concurrency` scan loops at once
(default `4`). Each loop independently claims a job with `SKIP LOCKED`, so up to
that many **different hosts** scan in parallel; a per-host advisory lock still
prevents two scans of the **same** host from overlapping. This is the simplest
way to clear a large queue — one config value, no extra processes:

```toml
# /etc/openwatch/openwatch.toml
[server]
scan_concurrency = 8
```

Restart `openwatch` to apply. Sizing: scans are SSH/IO-bound (they spend most of
their time waiting on the remote host), so concurrency can comfortably exceed
CPU core count. Mind two ceilings — the PostgreSQL pool (`[database].max_connections`
/ pool size: each in-flight scan uses a connection plus the advisory-lock
transaction) and how many simultaneous SSH sessions your targets and network
tolerate. `8`–`16` is a reasonable range for a few dozen to a few hundred hosts;
set it to `1` to restore strictly one-at-a-time draining.

### Run more worker processes

The scan queue is PostgreSQL-native and claims one job at a time per worker with
`SKIP LOCKED`. To increase scan throughput, run additional `openwatch worker`
processes pointed at the same database and config:

```bash
openwatch worker --config /etc/openwatch/openwatch.toml
```

Each worker claims one scan job at a time (`internal/worker/scan_worker.go`).
Within a single worker, a per-host `pg_advisory_xact_lock` serializes work so
two jobs for the same host never run concurrently. Across workers, the queue's
`SKIP LOCKED` semantics prevent any two workers from claiming the same job.

The package ships only the `openwatch.service` unit, which runs `serve`
(`packaging/common/openwatch.service`). There is no packaged worker unit yet, so
run the worker under your own `systemd` unit or process supervisor. A minimal
unit mirrors the shipped one but changes the `ExecStart` command:

```ini
[Unit]
Description=OpenWatch scan worker
After=network.target postgresql.service
Wants=postgresql.service

[Service]
Type=simple
User=openwatch
Group=openwatch
EnvironmentFile=-/etc/openwatch/secrets.env
ExecStart=/usr/bin/openwatch worker --config /etc/openwatch/openwatch.toml
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target
```

Use a systemd template (for example `openwatch-worker@.service`) if you want to
run several workers on one host. The worker shares the same configuration,
database DSN, JWT key, and credential key as `serve`, so no extra config is
required.

### Poll interval

Each worker sleeps between dequeue attempts when the queue is empty. The
`--poll-interval` flag controls this; it defaults to `1s` and is capped at `5s`
(`internal/worker/scan_worker.go`, `DefaultPollInterval` / `MaxPollInterval`):

```bash
openwatch worker --poll-interval 2s
```

A shorter interval lowers scan-pickup latency on an idle queue at the cost of
more empty database round-trips. A longer interval does the reverse. The cap
exists because raising it further only adds latency without a corresponding
benefit. Worker concurrency comes from running more processes, not from a
per-worker concurrency knob — there is no `--concurrency` flag.

## Scaling PostgreSQL

PostgreSQL holds all OpenWatch state and is the shared coordination point for
the queue. Tune it before reaching for anything else.

### Connection pool

OpenWatch opens one pgx pool per process (`internal/db/db.go`,
`db.NewPool`). The pool size is the `max_connections` value under `[database]`
in `/etc/openwatch/openwatch.toml`; it defaults to `25`
(`packaging/common/openwatch.toml`, `internal/config/config.go`):

```toml
[database]
dsn             = "postgres://openwatch@localhost/openwatch?sslmode=require"
max_connections = 25
```

You can also override it with the environment variable
`OPENWATCH_DATABASE_MAX_CONNECTIONS` (set it in `/etc/openwatch/secrets.env` or
the unit's `EnvironmentFile`). Each running process — `serve` and every
`worker` — opens its own pool of up to `max_connections`. Size PostgreSQL's
server-side `max_connections` to cover the sum across all OpenWatch processes
plus headroom for `psql`, backups, and monitoring. As a rule of thumb:

```
postgres max_connections  >=  (1 serve + N workers) * openwatch max_connections + slack
```

### Server tuning

Standard PostgreSQL tuning applies; OpenWatch does nothing unusual here. Start
from your host's RAM and adjust `shared_buffers`, `effective_cache_size`,
`work_mem`, and `max_wal_size` to match. Keep the database on fast local or
network-attached SSD storage — the transaction log and audit-event tables are
the highest-write paths.

### Migrations

Schema changes ship as ordered migrations in `internal/db/migrations/`, applied
with `openwatch migrate`. Run migrations once per upgrade against a single
database before starting the new binary; the command is safe to re-run and
reports the resulting version. Multiple processes can then connect to the
already-migrated schema.

## Capacity planning

OpenWatch has no fixed sizing matrix, and the scan cadence — not raw host
count — drives load. The intelligence and liveness schedulers run on
operator-tunable intervals, and scans are enqueued on a per-host schedule, so a
large fleet scanned infrequently can be lighter than a small fleet scanned
aggressively.

Plan capacity from these levers rather than a host-count table:

- **Scan throughput** — add `openwatch worker` processes until the scan queue
  drains as fast as you enqueue. Watch for jobs sitting in the queue.
- **API/UI responsiveness** — give the `serve` host enough CPU and RAM; it is a
  single process today, so vertical sizing is the lever.
- **PostgreSQL** — size RAM and connections to the combined pool demand above;
  this is usually the first thing to upgrade for a large fleet.

Measure on your own workload before committing hardware. The numbers that matter
are queue depth, scan duration, API latency, and PostgreSQL connection count and
query latency — all observable with the tools below.

## Observing load

There is no Prometheus endpoint and no Grafana stack in the current build (see
"Not yet implemented"). What you have today:

- **Health** — `GET /api/v1/health` returns `200` when the service and its
  database connection are healthy, `503` when degraded
  (`api/openapi.yaml`, `internal/server/handlers.go`). Use it for load-balancer
  and uptime probes.
- **Version** — `GET /api/v1/version` returns build metadata
  (`api/openapi.yaml`). `openwatch --version` prints the same locally.
- **Logs** — both processes emit structured JSON logs to `journald`. Follow them
  with `journalctl`:

  ```bash
  journalctl -u openwatch -f
  ```

  The worker logs a periodic `worker.loop.tick` line (roughly every 60s) with
  idle/claimed/in-flight/completed counters — a lightweight way to confirm a
  worker is alive and draining (`internal/worker/scan_worker.go`).
- **Audit and queue state** — query PostgreSQL directly:

  ```bash
  psql "$OPENWATCH_DATABASE_DSN" -c \
    "SELECT status, count(*) FROM job_queue GROUP BY status;"
  ```

  A growing count of non-terminal jobs means workers are not keeping up; add
  worker processes.

## Not yet implemented

Be explicit about what this stack does *not* offer today, so you do not plan
around features that are absent:

- **Horizontal API scaling is not packaged.** The `serve` process is stateless
  apart from PostgreSQL (it uses stateless JWT auth), so running replicas behind
  a load balancer is architecturally possible, but there is no shipped unit,
  load-balancer config, or supported procedure for it. Treat `serve` as a single
  vertically-scaled process for now.
- **No packaged worker unit.** Only `openwatch.service` (running `serve`) ships
  in the RPM/DEB. Running additional scan workers requires the operator-authored
  unit shown above.
- **No Prometheus/Grafana/metrics endpoint.** There is no `/metrics` route and
  no bundled monitoring stack. Observability is `GET /api/v1/health`, the JSON
  logs in `journald`, and direct PostgreSQL queries.
- **No PgBouncer integration, read replicas, or built-in connection proxy.** You
  can place standard PostgreSQL tooling in front of the database yourself;
  OpenWatch only knows the single DSN it is configured with.
- **No Redis, Celery, or message broker.** Background work is the PostgreSQL
  `SKIP LOCKED` queue only. Anything that referenced these in older docs is from
  the archived Python stack and does not apply.

## Related documentation

| Topic | Document |
|-------|----------|
| Install and configuration | `docs/guides/INSTALLATION.md` |
| Roles and permissions | `docs/engineering/rbac_registry.md` |
| Kensa scanning boundary | `docs/KENSA_OPENWATCH_BOUNDARY.md` |
| API contract | `api/openapi.yaml` (paths under `/api/v1`) |
| Worker behavior spec | `specs/system/worker-subcommand.spec.yaml` |

---

# OpenWatch: OpenWatch security hardening guide
URL: https://www.hanalyx.com/docs/openwatch/security-hardening

**Applies to:** OpenWatch 0.2.0 pre-release (rc series; Go single-binary build)
**Audience:** System administrators, security engineers, compliance officers

This guide covers the security controls you operate when you deploy OpenWatch
as a native package: one Go binary (`/usr/bin/openwatch`) that serves the REST
API and the embedded React UI over HTTPS on port 8443, backed by PostgreSQL and
managed by `systemd`. It documents what the current build enforces, what you
configure at the host level, and what is not yet implemented.

For installation, database provisioning, the admin bootstrap, and TLS-cert
replacement, follow [`docs/guides/INSTALLATION.md`](/docs/openwatch/installation).
This guide does not repeat those steps; it focuses on hardening the result.

Verify any claim here against the source before you rely on it. The grounding
files are cited in each section.

---

## 1. Architecture and trust boundaries

OpenWatch is a single process. There is no separate web tier, container runtime,
cache, or message broker. Background work (scans, discovery, intelligence
cycles) runs either in-process or under `openwatch worker`, draining a
PostgreSQL-native job queue with `SELECT ... FOR UPDATE SKIP LOCKED`.

| Component | What it is | Exposure |
|-----------|------------|----------|
| `openwatch serve` | HTTPS API + embedded UI | TCP/8443 inbound |
| PostgreSQL | All persistent state | You provision and bind it (loopback by default) |
| Kensa (in-process, Go) | SSH-based compliance engine | TCP/22 outbound to managed hosts |

Source: `cmd/openwatch/main.go`, `packaging/common/openwatch.service`,
`internal/server/server.go`.

The compliance engine is Kensa, which connects to managed hosts over SSH and
runs native YAML checks.
See
[`docs/KENSA_OPENWATCH_BOUNDARY.md`](../KENSA_OPENWATCH_BOUNDARY.md).

---

## 2. Network exposure

The binary listens on `0.0.0.0:8443` by default (`[server].listen`, override
with `OPENWATCH_SERVER_LISTEN`). It opens no other listening socket.

Source: `internal/config/config.go` (`Defaults()`), `cmd/openwatch/main.go`.

Hardening steps you perform at the host level:

- Restrict inbound TCP/8443 to operator networks with `firewalld`, `nftables`,
  or `ufw`. OpenWatch does not implement IP allowlisting itself.
- Bind PostgreSQL to the loopback interface and require `scram-sha-256` from
  `127.0.0.1`/`::1`, as the install guide's Step 2 sets up. The package does
  not manage PostgreSQL for you.
- Allow only the outbound TCP/22 that Kensa needs to reach managed hosts.
- The `systemd` unit sets `RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX`,
  so the process cannot open raw or exotic sockets even if compromised.

Source: `packaging/common/openwatch.service`,
`docs/guides/INSTALLATION.md` (Step 2).

### Outbound SSH to managed hosts

When OpenWatch connects to a managed host it validates the host key on a
trust-on-first-use basis and **persists** the accepted key in PostgreSQL
(`ssh_known_hosts`, migration 0036). The first connection records the key; every
later connection compares against it and is rejected if the key changed
(`ErrHostKeyMismatch`). Because the store is durable, a service restart does not
re-trust hosts, so an attacker cannot MITM the first scan after a restart to
harvest credentials. Presented keys are also strength-validated per NIST SP
800-57 (RSA >= 2048, Ed25519 always accepted). To rotate a host's key
intentionally (a re-provisioned host), delete its row from `ssh_known_hosts` so
the next connection re-learns it. Source: `internal/knownhosts/store.go`,
`internal/ssh/`.

---

## 3. Transport security (TLS)

The HTTPS server is configured with:

| Setting | Value | Source |
|---------|-------|--------|
| Minimum TLS version | TLS 1.2 | `internal/server/server.go` (`tls.Config{MinVersion: tls.VersionTLS12}`) |
| Cipher suites | Go standard-library defaults (not pinned) | `internal/server/server.go` |
| Certificate loading | `GetCertificate` callback, read per handshake | `internal/server/server.go`, `internal/server/tls.go` |
| Cert / key paths | `/etc/openwatch/tls/cert.pem`, `/etc/openwatch/tls/key.pem` | `internal/config/config.go` |

Because the cert is read on every handshake, you can replace the files and new
connections pick up the new cert without a restart; restart anyway to drop
existing keep-alive connections.

The package ships a self-signed cert. Replace it with a CA-issued cert before
any non-loopback use, following the "Replace the demo TLS cert" section of the
install guide. Set the key file to mode `0600`, owned `openwatch:openwatch`.

> The build does not pin an explicit cipher-suite list, so TLS 1.2 negotiation
> follows the Go runtime's secure defaults for the toolchain it was built with.
> If your accreditation requires a documented, pinned cipher list, raise it as a
> requirement; it is not configurable today.

Source: `internal/server/server.go`, `internal/server/tls.go`,
`docs/guides/INSTALLATION.md`.

---

## 4. Cryptography and FIPS

| Function | Algorithm | Source |
|----------|-----------|--------|
| Password hashing | Argon2id, t=3, m=64 MiB, p=1, 128-bit salt, 256-bit key | `internal/identity/password.go` |
| Credential / MFA-secret encryption at rest | AES-256-GCM (DEK) | `internal/secretkey/secretkey.go` |
| JWT signing | RS256 (RSA ≥ 2048) | `internal/identity/jwt.go`, `cmd/openwatch/main.go` |
| Breach-corpus lookup | SHA-1 prefix (HaveIBeenPwned k-anonymity; not authentication) | `internal/identity/password.go` |

### FIPS builds

A FIPS build target exists and uses the Go-native FIPS 140-3 module, not an
OpenSSL provider. Build it with:

```bash
make build-fips
```

This sets `GOFIPS140=v1.0.0` and produces `dist/openwatch-fips`. The resulting
binary reports its FIPS status:

```bash
openwatch --version
# ... fips: true ...
```

Source: `Makefile` (`build-fips`), `internal/version/version.go`.

> The standard package build is not FIPS-validated. If you require FIPS 140-3,
> deploy the `-fips` artifact and confirm `fips: true` from `openwatch --version`.
> The legacy "RHEL OpenSSL FIPS provider" / `fips-mode-setup` approach from the
> archived Python stack does not apply.

---

## 5. Cryptographic key material

`openwatch serve` refuses to start unless both identity keys are present. There
is no silent fallback to ephemeral keys.

| Config key | Default path | Contents | Required mode |
|------------|--------------|----------|---------------|
| `[identity].jwt_private_key` | `/etc/openwatch/keys/jwt_private.pem` | PEM RSA private key, ≥ 2048-bit | `0600` |
| `[identity].credential_key_file` | `/etc/openwatch/keys/credential.key` | 32-byte raw AES-256 key | `0600` |

Source: `internal/config/config.go` (`IdentityConfig`), `cmd/openwatch/main.go`
(`cmdServe` key-loading).

Hardening steps:

- Set both key files to mode `0600`, owned by the `openwatch` user.
- Keep the database password out of the world-readable config: put
  `OPENWATCH_DATABASE_DSN` in `/etc/openwatch/secrets.env` (mode `0640`, owner
  `root:openwatch`), which the `systemd` unit loads via `EnvironmentFile=`.
- `openwatch check-config` prints the resolved config with the DSN password
  redacted, so it is safe to capture in tickets.

Source: `packaging/common/openwatch.service`, `internal/config/config.go`
(`RedactDSN`, `Summary`), `docs/guides/INSTALLATION.md` (Step 4).

---

## 6. Authentication

| Control | Value | Source |
|---------|-------|--------|
| Access-token lifetime | 30 minutes | `internal/identity/jwt.go` (`AccessTokenWindow`) |
| Refresh-token lifetime | 7 days, rotated on use (reuse is detected and revokes the chain) | `internal/identity/refresh.go` (`RefreshTokenWindow`) |
| Session inactivity timeout | 15 minutes | `internal/identity/sessions.go` (`SessionInactivityWindow`) |
| Session absolute timeout | 12 hours | `internal/identity/sessions.go` (`SessionAbsoluteWindow`) |
| Password policy | Length only — 8 chars (regular), 15 chars (admin), max 128; NIST SP 800-63B | `internal/identity/password.go` |
| Breach check | Always-on in production: new passwords are screened against an embedded common/breached corpus (airgap-safe); point `OPENWATCH_BREACH_CORPUS_FILE` at a full HIBP list to extend it | `internal/identity/password.go`, `internal/identity/breach_corpus_default.go` |
| MFA | TOTP enrollment and verification | `internal/identity/mfa.go` |

The password policy is deliberately length-based with no character-class rules,
per NIST SP 800-63B. The first admin is created out-of-band with
`openwatch create-admin`, which enforces the 15-character admin minimum.

Source: `internal/identity/`, `cmd/openwatch/main.go` (`cmdCreateAdmin`).

`/api/v1/auth/login` and `/auth/mfa:verify` are rate-limited per client IP
(Section 10), which throttles online guessing in addition to the Argon2id cost
(~50-100 ms per verification). There is still no per-account lockout after N
failed attempts, so for an internet-facing 8443 a reverse proxy or network ACL
is still worthwhile as defense in depth. Source:
`internal/server/auth_handlers.go`, `internal/server/ratelimit.go`.

---

## 7. Authorization (RBAC)

Authorization is a permission registry, not free-form strings. Every protected
operation declares `x-required-permission` in `api/openapi.yaml`; the handler
middleware checks the caller's effective permission set; built-in roles grant
permissions from the same registry. A misspelled permission anywhere is a build
error.

Source of truth: `auth/permissions.yaml` →
`internal/auth/permissions.gen.go` and `internal/auth/roles.gen.go`. Enforcement:
`internal/auth/middleware.go` (`EnforcePermission`, `RequirePermission`).
Design doc: [`docs/engineering/rbac_registry.md`](../engineering/rbac_registry.md).

Built-in roles, least to most privileged:

| Role ID | Purpose |
|---------|---------|
| `viewer` | Read-only across the platform |
| `auditor` | Read-only plus exception authority and audit export |
| `ops_lead` | Day-to-day operations — hosts, scans, alerts |
| `security_admin` | Full security operations, including dangerous and license-gated actions |
| `admin` | Full system administration (user/role/SSO/system-setting management) |

Source: `internal/auth/roles.gen.go` (`BuiltInRoles`).

Hardening steps:

- Assign the least-privileged role that satisfies each user's job. `auditor` is
  read-only except for exception workflow and audit export.
- Keep `admin` accounts to the minimum. Only `admin` holds the
  `admin:user_manage`, `admin:role_manage`, `admin:sso_provider`, and
  `admin:system_setting` permissions.
- License-gated permissions (for example `remediation:execute`) are enforced in
  the same middleware pass as RBAC; you cannot use them without the entitlement.

---

## 8. Audit logging

Security-relevant events are written to a durable audit store in PostgreSQL with
a stable taxonomy (`actor`, `resource`, `action`, correlation ID). The taxonomy
is the single naming source so events do not drift across components.

The writer initializes at startup (`audit.Init`) and `system.startup` is emitted
synchronously before the server accepts traffic. Operational logs are separate:
the process logs JSON to `journald`.

Source: `cmd/openwatch/main.go` (`audit.Init`, `audit.EmitSync`),
`internal/audit/`, `internal/db/migrations/0002_audit_events_taxonomy.sql`,
[`docs/engineering/audit_event_taxonomy.md`](../engineering/audit_event_taxonomy.md).

Representative event codes (taxonomy):

| Category | Examples |
|----------|----------|
| Authentication | `auth.login.success`, `auth.login.failure`, `auth.logout` |
| System lifecycle | `system.startup`, `system.shutdown` |
| Hosts / scans / users / roles | `host.*`, `scan.*`, `user.*`, `role.*` |
| Licensing | license load/result events |

Hardening steps:

- Ship `journald` to a central collector (`systemd-journal-remote`, or a
  shipper such as Vector or rsyncd) so operational logs survive host loss.
- The durable audit trail lives in PostgreSQL; back up the database (Section 11)
  to retain audit evidence for your compliance window.
- View live operational logs:

  ```bash
  sudo journalctl -u openwatch -f
  sudo journalctl -u openwatch -o cat | jq .   # pretty-print the JSON
  ```

> The audit-event taxonomy is the authoritative list. Treat the table above as a
> sample, not a complete enumeration — read `internal/audit/` and
> `docs/engineering/audit_event_taxonomy.md` for the full set.

---

## 9. Process hardening (systemd)

The packaged `systemd` unit runs the service unprivileged and confined:

| Directive | Value | Effect |
|-----------|-------|--------|
| `User` / `Group` | `openwatch` | Runs as a dedicated unprivileged account |
| `NoNewPrivileges` | `true` | Process cannot gain privileges via setuid/setgid |
| `PrivateTmp` | `true` | Isolated `/tmp` |
| `ProtectSystem` | `strict` | Filesystem is read-only except declared paths |
| `ProtectHome` | `true` | No access to `/home`, `/root`, `/run/user` |
| `ProtectKernelTunables` | `true` | Cannot write `/proc/sys`, `/sys` |
| `ProtectKernelModules` | `true` | Cannot load kernel modules |
| `ProtectControlGroups` | `true` | Cgroup hierarchy is read-only |
| `RestrictAddressFamilies` | `AF_INET AF_INET6 AF_UNIX` | No raw/packet sockets |
| `LockPersonality` | `true` | Cannot change execution domain |
| `ReadWritePaths` | `/var/lib/openwatch /var/log/openwatch` | Only these are writable |

Source: `packaging/common/openwatch.service`.

Hardening steps:

- Do not loosen `ProtectSystem=strict` or widen `ReadWritePaths` unless you have
  a verified need.
- Confirm the effective sandbox after any unit edit:

  ```bash
  systemd-analyze security openwatch
  ```

- Keep the config and key files owned away from the service account where the
  service only needs read access (cert/JWT/credential keys at `0600`, owned by
  `openwatch`; `secrets.env` at `0640`, owner `root:openwatch`).

---

## 10. Rate limiting, CSRF, and security headers

The HTTP server sets request-hardening timeouts and size limits:

| Control | Value | Source |
|---------|-------|--------|
| `ReadHeaderTimeout` | 10 s | `internal/server/server.go` |
| `ReadTimeout` | 30 s | `internal/server/server.go` |
| `WriteTimeout` | 60 s | `internal/server/server.go` |
| `IdleTimeout` | 120 s | `internal/server/server.go` |
| `MaxHeaderBytes` | 64 KiB | `internal/server/server.go` |

The single binary serves the SPA and the API from one origin with no required
edge proxy, so the perimeter controls run in the application itself:

| Control | Behavior | Source |
|---------|----------|--------|
| Auth rate limiting | Per-client-IP sliding window on `POST /api/v1/auth/login` and `/auth/mfa:verify`; over the limit returns `429` + `Retry-After` and skips the credential check. The key is the direct connection address (`RemoteAddr`), not a client-supplied `X-Forwarded-For`. | `internal/server/ratelimit.go` |
| CSRF | Double-submit token: login and refresh set a non-HttpOnly `XSRF-TOKEN` cookie, and unsafe (POST/PUT/PATCH/DELETE) cookie-authenticated requests must echo it in `X-CSRF-Token` (constant-time compare) or get `403 authz.csrf_invalid`. Bearer/token requests and `/api/v1/auth/*` are exempt. | `internal/server/csrf.go` |
| Security headers | Every response carries HSTS (>=1 year, includeSubDomains), a Content-Security-Policy that denies framing (`frame-ancestors 'none'`, `default-src 'self'`; `/docs` relaxes script/style for Swagger but still denies framing), `X-Content-Type-Options: nosniff`, `X-Frame-Options: DENY`, and `Referrer-Policy: no-referrer`. | `internal/server/security_headers.go` |

> Scope notes: auth rate limiting covers the credential-guessing surface, not
> every route — there is still no general per-route HTTP limiter, and no request
> body-size cap (`http.MaxBytesReader`) on JSON endpoints. If you expose 8443
> publicly, an upstream reverse proxy is still useful for global rate limiting
> and body-size enforcement. The scheduler `RateLimit` constants are unrelated
> (they bound how many hosts the intelligence and discovery schedulers enqueue
> per tick), see `internal/intelligence/scheduler/service.go`.

---

## 11. Database and backups

OpenWatch stores all persistent state — hosts, credentials (AES-256-GCM
encrypted), scans, transactions, the job queue, and the audit trail — in
PostgreSQL. The package does not manage PostgreSQL and does not implement an
in-product backup tool.

Schema is applied with `openwatch migrate`, which runs the migrations in
`internal/db/migrations/` (goose).

Source: `cmd/openwatch/main.go` (`cmdMigrate`), `internal/db/migrations/`.

Hardening steps:

- Require TLS to PostgreSQL when it is not on the loopback interface: use
  `sslmode=require` (or `verify-full` with a CA) in `OPENWATCH_DATABASE_DSN`.
  Source: `docs/guides/INSTALLATION.md` (Step 4).
- Back up with the standard PostgreSQL tooling on a schedule that meets your
  retention requirement, and store backups encrypted off-host:

  ```bash
  sudo -u postgres pg_dump -Fc openwatch > openwatch-$(date -u +%Y-%m-%dT%H%M%SZ).dump
  ```

- Restrict the database role to the `openwatch` database only; do not reuse a
  superuser DSN for the service.

---

## 12. Operational runbooks

These are first-response procedures for the single binary on `systemd` with
PostgreSQL. They assume you have shell access on the OpenWatch host.

### SERVICE_DOWN — the service is not responding

```bash
sudo systemctl status openwatch
sudo journalctl -u openwatch --since '5 min ago' -p err
curl -k https://localhost:8443/api/v1/health
# healthy: {"status":"healthy","db_connected":true,"version":"…"}
```

1. If `status` shows the unit failed, read the error lines from `journalctl`.
2. Common causes (from the install guide's troubleshooting table): malformed
   `OPENWATCH_DATABASE_DSN`, wrong DB password or `pg_hba.conf`, PostgreSQL not
   running, or an unreadable TLS cert. Validate config without starting:

   ```bash
   sudo -u openwatch env $(cat /etc/openwatch/secrets.env | xargs) openwatch check-config
   ```

3. If `/api/v1/health` returns 503, the database ping failed — check
   PostgreSQL: `sudo systemctl status postgresql`.
4. Restart once the cause is fixed: `sudo systemctl restart openwatch`. The unit
   uses `Restart=on-failure` with a 5 s delay, so transient crashes self-heal.

Source: `docs/guides/INSTALLATION.md` (Troubleshooting),
`internal/server/server.go`, `packaging/common/openwatch.service`.

### DISK_FULL — the host is out of disk

```bash
df -h
sudo du -xh /var/log/openwatch /var/lib/openwatch 2>/dev/null | sort -h | tail
sudo journalctl --disk-usage
```

1. Identify the consumer. The service writes only to `/var/lib/openwatch` and
   `/var/log/openwatch` (`ReadWritePaths`); `journald` and PostgreSQL data are
   the other large consumers.
2. Vacuum `journald` to a bound:

   ```bash
   sudo journalctl --vacuum-size=500M
   ```

3. If PostgreSQL's data directory is the consumer, reclaim space there (vacuum,
   prune old backups) — do not delete files under PostgreSQL's data directory by
   hand.
4. A full disk can wedge the audit writer and database. After reclaiming space,
   confirm health with the `SERVICE_DOWN` check.

Source: `packaging/common/openwatch.service` (`ReadWritePaths`),
`internal/db/migrations/`.

### HIGH_CPU — sustained high CPU

```bash
top -b -n1 | head -20
sudo journalctl -u openwatch --since '10 min ago' | jq -r 'select(.level=="ERROR")' 2>/dev/null
```

1. Identify whether the `openwatch` process or `postgres` backends dominate.
2. If `postgres` dominates, look for slow queries:

   ```bash
   sudo -u postgres psql -d openwatch \
     -c "SELECT pid, state, now()-query_start AS runtime, left(query,80) \
         FROM pg_stat_activity WHERE datname='openwatch' AND state<>'idle' \
         ORDER BY runtime DESC LIMIT 10;"
   ```

3. Background scan/intelligence/discovery load is operator-tunable. Reduce the
   scheduler rate or pause it via the system-config API
   (`PUT /api/v1/system/intelligence/config`,
   `PUT /api/v1/system/discovery/config`) — the boot logs name these knobs when a
   scheduler is paused.
4. If the API process itself is hot with no DB pressure, capture logs and
   restart as a containment step: `sudo systemctl restart openwatch`.

Source: `cmd/openwatch/main.go` (scheduler wiring and maintenance knobs),
`internal/intelligence/scheduler/`, `internal/intelligence/discovery/scheduler/`.

### SECURITY_INCIDENT — suspected compromise or credential exposure

1. **Contain.** Block inbound 8443 at the host firewall, or stop the service if
   you must take it offline:

   ```bash
   sudo systemctl stop openwatch
   ```

2. **Preserve evidence.** Snapshot operational logs and the audit trail before
   changing anything:

   ```bash
   sudo journalctl -u openwatch --since '24 hours ago' > /tmp/openwatch-journal.log
   sudo -u postgres pg_dump -Fc openwatch > /tmp/openwatch-evidence.dump
   ```

3. **Review authentication and authorization events** in the audit trail —
   `auth.login.success`, `auth.login.failure`, role and user changes — for the
   incident window. Query the audit tables directly with `psql` or via the audit
   API.
4. **Rotate secrets.** Rotate the database password (update
   `/etc/openwatch/secrets.env`), and replace the JWT signing key
   (`/etc/openwatch/keys/jwt_private.pem`) and TLS cert/key as warranted.
   Replacing the JWT key invalidates all outstanding access tokens.
5. **Revoke or reset affected accounts.** Reset compromised users' passwords and
   reassign roles as needed; review `admin`-role membership.
6. **Restart and re-verify** once contained: `sudo systemctl start openwatch`,
   then the `SERVICE_DOWN` health check. File the incident per your
   organization's process.

Source: `cmd/openwatch/main.go` (JWT key loading, audit), `internal/identity/`,
`docs/engineering/audit_event_taxonomy.md`.

---

## 13. Hardening checklist

Network and transport

- [ ] Inbound TCP/8443 restricted to operator networks at the host firewall.
- [ ] PostgreSQL bound to loopback (or TLS-required) with `scram-sha-256` auth.
- [ ] CA-issued TLS cert installed at `/etc/openwatch/tls/`; key mode `0600`.
- [ ] If 8443 is reachable beyond a trusted network, an upstream reverse proxy
      provides rate limiting and security response headers.

Cryptography and keys

- [ ] `[identity].jwt_private_key` present, RSA ≥ 2048, mode `0600`.
- [ ] `[identity].credential_key_file` present, 32 bytes, mode `0600`.
- [ ] `OPENWATCH_DATABASE_DSN` in `secrets.env` (mode `0640`, `root:openwatch`),
      not in the world-readable TOML.
- [ ] For FIPS environments: the `-fips` build is deployed and
      `openwatch --version` reports `fips: true`.

Identity and access

- [ ] Admin accounts minimized; users hold the least-privileged role.
- [ ] MFA (TOTP) enrolled for privileged users.
- [ ] Breach-corpus check enabled for production password validation.

Process and platform

- [ ] `systemd-analyze security openwatch` reviewed; sandbox not loosened.
- [ ] `ProtectSystem=strict` and `ReadWritePaths` unchanged unless justified.

Audit and durability

- [ ] `journald` forwarded to a central collector.
- [ ] Scheduled, encrypted, off-host PostgreSQL backups meeting the retention
      window (audit trail lives in the database).

Source for every checklist item is cited in the section above that introduces it.

---

## Related documentation

- Install, configure, TLS replacement, uninstall:
  [`docs/guides/INSTALLATION.md`](/docs/openwatch/installation)
- RBAC registry and permission model:
  [`docs/engineering/rbac_registry.md`](../engineering/rbac_registry.md)
- Audit event taxonomy:
  [`docs/engineering/audit_event_taxonomy.md`](../engineering/audit_event_taxonomy.md)
- Kensa ↔ OpenWatch boundary:
  [`docs/KENSA_OPENWATCH_BOUNDARY.md`](../KENSA_OPENWATCH_BOUNDARY.md)
- API contract (per-operation required permission, license gate, audit events):
  [`api/openapi.yaml`](../../api/openapi.yaml)
- Behavioral specs: [`specs/`](../../specs/)

---

# OpenWatch: Secret rotation procedures
URL: https://www.hanalyx.com/docs/openwatch/secret-rotation

**Last Updated:** 2026-06-22 · **Applies to:** OpenWatch 0.2.0-rc series (Go single-binary)

This guide describes how to rotate each secret used by OpenWatch on the current
single-binary stack: one `/usr/bin/openwatch` process that serves the REST API
and embedded UI over HTTPS on port `8443`, backed by PostgreSQL and run under
the `openwatch.service` systemd unit. There is no separate web tier, no
container runtime, and no Redis or message broker.

For install and first-time configuration, see
[`docs/guides/INSTALLATION.md`](/docs/openwatch/installation); this
guide assumes the service is already installed and running.

## Secrets at a glance

OpenWatch reads its secrets from three places: the TOML config
(`/etc/openwatch/openwatch.toml`), the systemd `EnvironmentFile`
(`/etc/openwatch/secrets.env`), and on-disk key/cert files under
`/etc/openwatch/`. The config layering order, highest precedence first, is CLI
flags, then `OPENWATCH_<SECTION>_<KEY>` environment variables, then the TOML
file, then built-in defaults.

| Secret | Where it lives | Loaded at | Rotation impact |
|--------|----------------|-----------|-----------------|
| Database DSN (incl. password) | `OPENWATCH_DATABASE_DSN` in `/etc/openwatch/secrets.env` | Service start, `migrate`, `create-admin` | Brief restart |
| JWT signing key (RSA private key) | `[identity].jwt_private_key` file (default `/etc/openwatch/keys/jwt_private.pem`) | Service start | Invalidates all sessions; users re-authenticate |
| Credential DEK (AES-256 key) | `[identity].credential_key_file` file (default `/etc/openwatch/keys/credential.key`) | Service start | Stored SSH credentials and MFA secrets become unreadable unless re-encrypted |
| TLS certificate and key | `[server].tls_cert` / `[server].tls_key` (default `/etc/openwatch/tls/{cert,key}.pem`) | Read on each TLS handshake | New connections pick up the new cert; restart to drop keep-alives |

> The credential DEK and JWT key fields are verified in
> [`internal/config/config.go`](../../internal/config/config.go)
> (`IdentityConfig`). The server refuses to start if either path is empty or the
> file fails to load — see
> [`cmd/openwatch/main.go`](../../cmd/openwatch/main.go) (`cmdServe`).

There is no separate "master key" or second "encryption key" on this stack. The
single credential DEK encrypts every at-rest secret (SSH credentials and MFA
secrets) with AES-256-GCM. The previous Python build's
`OPENWATCH_SECRET_KEY` / `OPENWATCH_MASTER_KEY` / `OPENWATCH_ENCRYPTION_KEY` /
`REDIS_PASSWORD` variables no longer exist.

## Before you rotate

1. Schedule a maintenance window. Every rotation here requires a service restart.
2. Back up the database with `pg_dump` before rotating the credential DEK or the
   JWT key, so you can recover if re-encryption goes wrong.
3. Record the current and new secret values in a secrets manager, not a plaintext
   file on the host.
4. Confirm the service is healthy first:

   ```bash
   curl -k https://localhost:8443/api/v1/health
   # {"status":"healthy","db_connected":true,"version":"<version>"}
   ```

## Rotate the database password

Impact: a brief restart while the service reconnects. The DSN lives in
`/etc/openwatch/secrets.env`, which the systemd unit loads via
`EnvironmentFile=-/etc/openwatch/secrets.env`.

1. Choose a new password and set it on the PostgreSQL role:

   ```bash
   sudo -u postgres psql -c "ALTER ROLE openwatch WITH PASSWORD 'new-strong-password';"
   ```

2. Update the DSN in `/etc/openwatch/secrets.env` (keep the file mode at `0640`,
   owner `root:openwatch`):

   ```bash
   sudo tee /etc/openwatch/secrets.env >/dev/null <<'EOF'
   OPENWATCH_DATABASE_DSN=postgres://openwatch:new-strong-password@127.0.0.1:5432/openwatch?sslmode=disable
   EOF
   sudo chown root:openwatch /etc/openwatch/secrets.env
   sudo chmod 0640 /etc/openwatch/secrets.env
   ```

   Use `sslmode=require` or stronger for any PostgreSQL that is not on the
   loopback interface.

3. Validate the resolved config before restarting:

   ```bash
   sudo -u openwatch env $(cat /etc/openwatch/secrets.env | xargs) \
       openwatch check-config
   ```

   `check-config` prints the config with the DSN password redacted and exits
   non-zero on a malformed DSN.

4. Restart and verify:

   ```bash
   sudo systemctl restart openwatch
   sudo systemctl status openwatch
   curl -k https://localhost:8443/api/v1/health
   ```

## Rotate the JWT signing key

Impact: all active sessions are invalidated and users must sign in again. The
key is an RSA private key in PEM form (PKCS#1 or PKCS#8), and the loader rejects
keys smaller than 2048 bits
([`internal/identity/jwt.go`](../../internal/identity/jwt.go), `LoadJWTKey`).
Access tokens have a 30-minute lifetime, but rotating the key invalidates the
refresh tokens too, so plan for a full re-login.

1. Generate a new 2048-bit (or larger) RSA key as the `openwatch` user, mode
   `0600`:

   ```bash
   sudo install -d -m 0750 -o root -g openwatch /etc/openwatch/keys
   sudo -u openwatch openssl genpkey -algorithm RSA \
       -pkeyopt rsa_keygen_bits:2048 \
       -out /etc/openwatch/keys/jwt_private.pem
   sudo chmod 0600 /etc/openwatch/keys/jwt_private.pem
   ```

   Write to a new path and update `[identity].jwt_private_key` if you prefer to
   keep the old key around for rollback.

2. Point the config at the key. Either set it in `/etc/openwatch/openwatch.toml`:

   ```toml
   [identity]
   jwt_private_key = "/etc/openwatch/keys/jwt_private.pem"
   ```

   or set `OPENWATCH_IDENTITY_JWT_PRIVATE_KEY` in `/etc/openwatch/secrets.env`.

3. Restart and verify:

   ```bash
   sudo systemctl restart openwatch
   sudo journalctl -u openwatch --since '1 min ago' | grep -i jwt
   curl -k https://localhost:8443/api/v1/health
   ```

   If the key is missing, unparseable, or under 2048 bits, the service logs
   `load jwt key failed` and exits — `journalctl -u openwatch` shows the reason.

4. Confirm users can sign in. Existing tokens are no longer accepted.

There is no dual-key (old + new) verification on this stack, so there is no
zero-downtime overlap window. Rotate during low usage to limit the number of
forced re-logins.

## Rotate the credential DEK

Impact: high. The DEK is a single 32-byte AES-256 key that directly encrypts
every stored SSH credential and every MFA secret with AES-256-GCM
([`internal/secretkey/secretkey.go`](../../internal/secretkey/secretkey.go)).
There is no per-credential wrapped key, so changing the DEK without
re-encrypting every row makes those secrets permanently unreadable.

> **Not yet implemented.** OpenWatch does not ship a re-encryption or rekey
> command. The CLI subcommands are `serve`, `worker`, `migrate`,
> `create-admin`, and `check-config`
> ([`cmd/openwatch/main.go`](../../cmd/openwatch/main.go)) — none re-wraps
> stored secrets. Rotating the DEK in place therefore requires either
> re-entering the affected secrets by hand or a one-off migration written for
> your deployment. An online rotation command is roadmap work; until it lands,
> treat DEK rotation as a manual, planned operation.

### Option A — re-enter secrets (no custom tooling)

This is the supported path when you have a manageable number of credentials.

1. Back up the database (`pg_dump`) so you can roll back to the old DEK.
2. Generate a new 32-byte key, mode `0600` (the loader rejects any file readable
   by group or other):

   ```bash
   sudo -u openwatch sh -c 'umask 077; head -c 32 /dev/urandom > /etc/openwatch/keys/credential.key'
   sudo chmod 0600 /etc/openwatch/keys/credential.key
   ```

3. Point `[identity].credential_key_file` (or
   `OPENWATCH_IDENTITY_CREDENTIAL_KEY_FILE`) at the new key and restart:

   ```bash
   sudo systemctl restart openwatch
   ```

4. Re-create the SSH credentials and re-enroll MFA through the UI or API
   (`/api/v1/...`); secrets created before the swap will fail to decrypt and must
   be replaced. Keep the old key file until you have confirmed every secret is
   re-entered, in case you need to roll back.

### Option B — offline re-encryption (custom)

For a large credential set, write a one-off program that opens the database,
decrypts each ciphertext column with the old DEK, re-encrypts it with the new
DEK, and updates the row, then swaps the key file and restarts. This is
deployment-specific code; there is no in-tree tool for it. Always run it against
a `pg_dump` restore first.

> Loss warning: if you change `credential_key_file` without re-encrypting and
> without keeping the old key, all stored SSH credentials and MFA secrets are
> unrecoverable. Back up before rotating.

## Rotate the TLS certificate

Impact: minimal. The server reads the cert and key on each TLS handshake, so new
connections use the new material immediately; restart to drop existing
keep-alive connections.

```bash
sudo cp /path/to/new-cert.pem /etc/openwatch/tls/cert.pem
sudo cp /path/to/new-key.pem  /etc/openwatch/tls/key.pem
sudo chown root:openwatch      /etc/openwatch/tls/cert.pem
sudo chown openwatch:openwatch /etc/openwatch/tls/key.pem
sudo chmod 0644                /etc/openwatch/tls/cert.pem
sudo chmod 0600                /etc/openwatch/tls/key.pem
sudo systemctl restart openwatch
```

See the "Replace the demo TLS cert" section of the install guide for the same
procedure in install context.

## Suggested rotation schedule

These intervals are guidance for compliance-driven environments, not values
enforced by the software.

| Secret | Suggested interval | Reference |
|--------|--------------------|-----------|
| Database password | 90 days | NIST SP 800-53 IA-5 |
| JWT signing key | 180 days, or on suspected compromise | Organization policy |
| Credential DEK | 365 days, or on suspected compromise | NIST SP 800-57 |
| TLS certificate | Before expiry | CA/Browser Forum (398-day maximum) |

## Post-rotation checklist

- [ ] `/health` reports healthy: `curl -k https://localhost:8443/api/v1/health`.
- [ ] The unit is active: `sudo systemctl status openwatch`.
- [ ] No startup errors: `sudo journalctl -u openwatch --since '5 min ago' -p err`.
- [ ] For a JWT rotation: a fresh sign-in succeeds and old tokens are rejected.
- [ ] For a DEK rotation: an SSH-backed action (host liveness or a Kensa scan)
      succeeds against a host whose credential you re-entered.
- [ ] The `system.startup` audit event recorded the restart (visible in the
      audit log / `journalctl -u openwatch`).
- [ ] The new secret value is stored in your secrets manager and the rotation
      date and next-due date are recorded.

---

# OpenWatch: Backup and recovery
URL: https://www.hanalyx.com/docs/openwatch/backup-recovery

**Last Updated:** 2026-06-22 · **Applies to:** OpenWatch 0.2.0-rc series (Go single-binary)

This guide covers backup, restore, and disaster recovery for an OpenWatch
deployment. OpenWatch is a single Go binary (`/usr/bin/openwatch`) that serves
the REST API and the embedded React UI over HTTPS on port `8443`, backed by
PostgreSQL and managed by `systemd`. There is no container runtime, no Redis,
and no separate web tier to back up.

For install and first-run setup, see
[`docs/guides/INSTALLATION.md`](/docs/openwatch/installation). This
document assumes the layout that guide produces.

## What you need to back up

Two things must be backed up together. A database dump alone is not a complete
backup.

| Item | Path | Why it matters | Recoverable without backup? |
|------|------|----------------|-----------------------------|
| PostgreSQL database | external PostgreSQL server | Hosts, scans, transactions, findings, users, roles, encrypted credentials, audit events, job queue, system config | No |
| Credential encryption key | `/etc/openwatch/keys/credential.key` | AES-256 key that encrypts stored SSH credentials and MFA secrets in the database | No |
| JWT signing key | `/etc/openwatch/keys/jwt_private.pem` | Signs auth tokens; losing it invalidates all sessions (recoverable by re-issuing) | Partially |
| Database secret | `/etc/openwatch/secrets.env` | Holds `OPENWATCH_DATABASE_DSN` | No |
| Configuration | `/etc/openwatch/openwatch.toml` | Server, database, and logging settings | Re-creatable by hand |
| TLS certificate and key | `/etc/openwatch/tls/cert.pem`, `/etc/openwatch/tls/key.pem` | Serves HTTPS on `8443` | Re-issuable from your CA |

> The `credential.key` is the most important non-database item. SSH credentials
> and MFA secrets in the database are encrypted with it. If you restore a
> database dump but lose `credential.key`, those secrets are unrecoverable and
> you must re-enter every host credential. Back up `credential.key` and the
> database together, and store the key with at least the same protection as the
> database.

The default key paths above come from the shipped configuration
(`internal/config/config.go`); confirm yours with
`sudo -u openwatch openwatch check-config`, which prints the resolved
`jwt_private_key` and `credential_key_file` paths.

### What you do not need to back up

- **Application logs.** Logs go to the `systemd` journal (`journalctl -u
  openwatch`); `/var/log/openwatch` exists but the journal is primary. Back up
  the journal only if your retention policy requires it.
- **The job queue.** Background jobs use a PostgreSQL-native queue
  (`SKIP LOCKED`) inside the same database, so the database dump already covers
  it. There is no separate queue store to back up.
- **Compliance scan content.** Kensa rules are native YAML bundled with the
  install; they are not user data.

## Backup procedure

OpenWatch connects to an external PostgreSQL instance. Run `pg_dump` against
that server. The DSN is in `/etc/openwatch/secrets.env` as
`OPENWATCH_DATABASE_DSN`.

### Database dump

Use a compressed custom-format dump. It restores faster and supports selective
restore.

```bash
source /etc/openwatch/secrets.env   # sets OPENWATCH_DATABASE_DSN

pg_dump "$OPENWATCH_DATABASE_DSN" \
    --format=custom \
    --file="/var/backups/openwatch/openwatch_$(date -u +%Y%m%dT%H%M%SZ).dump"
```

The timestamp uses UTC (ISO 8601). For a plain-text dump you can inspect, drop
`--format=custom` and redirect to a `.sql` file.

### Configuration and keys

Back up the encryption keys and secrets alongside the database dump. These are
secrets — store them encrypted and restrict access.

```bash
tar czf - \
    /etc/openwatch/keys/ \
    /etc/openwatch/secrets.env \
    /etc/openwatch/openwatch.toml \
    /etc/openwatch/tls/ \
  | openssl enc -aes-256-cbc -salt -pbkdf2 \
      -out "/var/backups/openwatch/config_$(date -u +%Y%m%dT%H%M%SZ).tar.gz.enc"
```

### Verify a backup

A backup you have not verified is not a backup. List the contents of a dump
without restoring it:

```bash
pg_restore --list /var/backups/openwatch/openwatch_<timestamp>.dump >/dev/null \
  && echo "dump readable"
```

For a stronger check, restore into a throwaway database and compare row counts:

```bash
createdb "$RESTORE_DSN_DB"
pg_restore --dbname="$RESTORE_DSN" --no-owner --no-privileges \
    /var/backups/openwatch/openwatch_<timestamp>.dump
psql "$RESTORE_DSN" -c \
    "SELECT 'hosts' AS t, count(*) FROM hosts
     UNION ALL SELECT 'scans', count(*) FROM scans
     UNION ALL SELECT 'users', count(*) FROM users;"
dropdb "$RESTORE_DSN_DB"
```

Confirm table names against your installed schema before relying on them; the
authoritative list is `internal/db/migrations/`.

### Scheduling

Run the database dump and config backup on a schedule that meets your recovery
point objective. A `systemd` timer or `cron` entry that calls a wrapper script
covering both the dump and the encrypted config archive is sufficient. Apply a
retention policy (for example, `find /var/backups/openwatch -name '*.dump'
-mtime +30 -delete`) and copy backups off-host.

## Restore procedure

### Restore the database

1. Stop the service so nothing writes while you restore:

   ```bash
   sudo systemctl stop openwatch
   ```

2. Restore into the OpenWatch database. With a custom-format dump:

   ```bash
   source /etc/openwatch/secrets.env

   pg_restore "$OPENWATCH_DATABASE_DSN" \
       --clean --if-exists --no-owner --no-privileges \
       /var/backups/openwatch/openwatch_<timestamp>.dump
   ```

   `--clean --if-exists` drops existing objects first, so the restore replaces
   current contents. If you restore into a fresh, empty database instead, omit
   those flags.

3. Apply any migrations newer than the dump (safe no-op if the schema is
   already current):

   ```bash
   sudo -u openwatch env $(cat /etc/openwatch/secrets.env | xargs) \
       openwatch migrate
   ```

4. Start the service and confirm health:

   ```bash
   sudo systemctl start openwatch
   curl -k https://localhost:8443/api/v1/health
   # {"status":"healthy","db_connected":true,"version":"<installed version>"}
   ```

### Restore configuration and keys

Restore `credential.key` from the same backup generation as the database dump.
A mismatched key cannot decrypt stored credentials.

```bash
openssl enc -aes-256-cbc -d -pbkdf2 \
    -in /var/backups/openwatch/config_<timestamp>.tar.gz.enc \
  | sudo tar xzf - -C /

sudo chown -R root:openwatch /etc/openwatch/keys
sudo chmod 0640 /etc/openwatch/keys/credential.key
sudo systemctl restart openwatch
```

## Disaster recovery (rebuild on a new host)

1. Install the OpenWatch package on the new host (`dnf install` or `apt
   install`) per [`INSTALLATION.md`](/docs/openwatch/installation). This
   creates the `openwatch` user, the binary, `/etc/openwatch/`, and the
   `systemd` unit.
2. Provision PostgreSQL and create the database. The package does not provision
   PostgreSQL.
3. Restore `/etc/openwatch/keys/`, `/etc/openwatch/secrets.env`,
   `/etc/openwatch/openwatch.toml`, and `/etc/openwatch/tls/` from the encrypted
   config backup.
4. Restore the database dump into the new PostgreSQL database (see above).
5. Run `openwatch migrate` to apply any pending migrations.
6. Validate config, then start:

   ```bash
   sudo -u openwatch openwatch check-config
   sudo systemctl enable --now openwatch
   curl -k https://localhost:8443/api/v1/health
   ```

### Recovery objectives

| Scenario | Procedure | Recovery point |
|----------|-----------|----------------|
| Service crash / bad config | `systemctl restart openwatch`; fix config; `openwatch check-config` | None (no data loss) |
| Database corruption | Restore latest dump; `openwatch migrate` | Last dump |
| Full host loss | Rebuild on new host (above) | Last dump + last key backup |
| Lost `credential.key` | No recovery for stored secrets; re-enter host credentials after restore | Credentials lost |

Measure your actual recovery time against these scenarios; the numbers depend
on database size and your storage.

## Operational runbooks

These cover the common operational alarms for the single binary on `systemd`
with PostgreSQL.

### SERVICE_DOWN — the API is unreachable

```bash
sudo systemctl status openwatch
journalctl -u openwatch -n 100 --no-pager
```

Common causes and checks:

- **Database unreachable.** The log shows `failed to open db pool`. Verify
  `OPENWATCH_DATABASE_DSN` in `/etc/openwatch/secrets.env` and that PostgreSQL
  is up: `psql "$OPENWATCH_DATABASE_DSN" -c 'SELECT 1;'`.
- **Missing signing or credential key.** The log shows
  `identity.jwt_private_key is empty` or a key-load failure. Confirm the key
  files exist at the paths from `openwatch check-config`.
- **TLS cert or key missing/unreadable.** The log mentions `cert.pem`. Confirm
  `/etc/openwatch/tls/` files exist and the `openwatch` user can read the key.
- **Invalid config.** Run `sudo -u openwatch openwatch check-config`; it
  validates and prints the resolved config with secrets redacted.

After fixing the cause: `sudo systemctl restart openwatch`, then
`curl -k https://localhost:8443/api/v1/health`.

### DISK_FULL — a filesystem is out of space

```bash
df -h
journalctl --disk-usage
du -sh /var/lib/openwatch /var/log/openwatch /var/backups/openwatch 2>/dev/null
```

Likely sources and actions:

- **Journal growth.** Vacuum old logs: `sudo journalctl --vacuum-time=7d` (or
  `--vacuum-size=500M`).
- **Old backups.** Prune per your retention policy under
  `/var/backups/openwatch`.
- **Database growth on the PostgreSQL host.** Inspect with
  `psql "$OPENWATCH_DATABASE_DSN" -c "SELECT pg_size_pretty(pg_database_size(current_database()));"`.
  OpenWatch uses a write-on-change transaction model (one row per host×rule plus
  change records), so steady-state growth is bounded; sudden growth usually
  means the audit-event or job-queue tables. Investigate before deleting rows —
  do not hand-edit OpenWatch tables.

If the service stopped because the disk filled, restart it after freeing space:
`sudo systemctl restart openwatch`.

### HIGH_CPU — the host is CPU-saturated

```bash
top -b -n1 | head -20
systemctl status openwatch
journalctl -u openwatch -n 200 --no-pager | grep -iE 'scheduler|worker|scan'
```

- Confirm whether the `openwatch` process or PostgreSQL is the consumer. Scan
  fan-out and the intelligence/discovery schedulers drive most OpenWatch CPU
  use.
- On the PostgreSQL host, look for expensive queries:
  `psql "$OPENWATCH_DATABASE_DSN" -c "SELECT pid, state, query_start, left(query,80) FROM pg_stat_activity WHERE state <> 'idle' ORDER BY query_start;"`.
- The schedulers honor a maintenance switch. To pause intelligence collection
  while you investigate, an admin can `PUT /api/v1/system/intelligence/config`
  with `maintenance_global=true` (and the discovery equivalent at
  `/api/v1/system/discovery/config`). The startup log notes when either is
  paused.
- As a last resort, `sudo systemctl restart openwatch` clears any runaway
  in-process loop without losing data (queued jobs resume).

### SECURITY_INCIDENT — suspected compromise

1. **Preserve evidence first.** Capture the journal and audit trail before
   changing anything:

   ```bash
   journalctl -u openwatch --since "-24h" > /var/backups/openwatch/incident_journal.txt
   ```

   OpenWatch writes structured audit events (auth, authz, system lifecycle) to
   the database; export the relevant rows for the incident window before any
   restore.

2. **Contain.** Stop the service to halt active sessions and scans:
   `sudo systemctl stop openwatch`. If only network exposure is the concern,
   firewall port `8443` instead.

3. **Rotate secrets.** If a key may be exposed:
   - Rotate the database password and update
     `OPENWATCH_DATABASE_DSN` in `/etc/openwatch/secrets.env`.
   - Replace the TLS certificate and key in `/etc/openwatch/tls/`.
   - Rotating the JWT signing key (`/etc/openwatch/keys/jwt_private.pem`)
     invalidates all existing sessions and forces re-login.
   - The credential DEK (`/etc/openwatch/keys/credential.key`) cannot be rotated
     by swapping the file alone — stored credentials are encrypted under it. Do
     not replace it without a migration path, or stored host credentials become
     undecryptable.

4. **Review access.** Audit user accounts and role assignments. Roles and
   permissions are defined in
   [`docs/engineering/rbac_registry.md`](../engineering/rbac_registry.md).

5. **Recover.** If integrity is in doubt, rebuild on a clean host from a
   known-good backup using the disaster-recovery procedure above, then rotate
   all credentials again.

## Not yet implemented

The following are not part of OpenWatch today. Do not script against them.

- **No built-in backup command.** There is no `openwatch backup` or
  `openwatch restore` subcommand. The subcommands are `serve`, `worker`,
  `migrate`, `create-admin`, and `check-config` (`openwatch --help`). Use
  `pg_dump`/`pg_restore` and file copies as shown above.
- **No continuous WAL archiving or point-in-time recovery shipped by
  OpenWatch.** If you need PITR, configure it on your PostgreSQL server
  independently; it is a PostgreSQL feature, not an OpenWatch one.
- **No automated off-site replication.** Copying backups off-host is your
  responsibility.

## Reference

| Item | Value |
|------|-------|
| Binary | `/usr/bin/openwatch` |
| Service unit | `openwatch.service` (`User=openwatch`) |
| Config | `/etc/openwatch/openwatch.toml` |
| DB secret | `/etc/openwatch/secrets.env` (`OPENWATCH_DATABASE_DSN`) |
| Encryption keys | `/etc/openwatch/keys/` (`jwt_private.pem`, `credential.key`) |
| TLS | `/etc/openwatch/tls/{cert,key}.pem` |
| Data / logs | `/var/lib/openwatch`, `/var/log/openwatch` (journal is primary) |
| Health probe | `GET https://<host>:8443/api/v1/health` |
| Migrate | `sudo -u openwatch env $(cat /etc/openwatch/secrets.env \| xargs) openwatch migrate` |
| Logs | `journalctl -u openwatch -f` |

See also: [`INSTALLATION.md`](/docs/openwatch/installation),
[`rbac_registry.md`](../engineering/rbac_registry.md), and the API contract in
[`api/openapi.yaml`](../../api/openapi.yaml).

---

# OpenWatch: Upgrade procedure
URL: https://www.hanalyx.com/docs/openwatch/upgrade-procedure

**Last Updated:** 2026-06-22 · **Applies to:** OpenWatch 0.2.0-rc series (Go single-binary)

This guide covers upgrading an OpenWatch deployment to a newer version. OpenWatch
ships as a single Go binary (`/usr/bin/openwatch`) that serves both the REST API
and the embedded web UI over HTTPS on port 8443, managed by the `openwatch.service`
systemd unit and backed by PostgreSQL. There is no container runtime, no separate
web tier, and no Redis/Celery to coordinate, so an upgrade is: install the new
package, apply migrations, restart the service.

There are two paths. The **automatic** path (below) is one command: the package
scriptlet backs up and migrates the database and restarts the service. The
**controlled (manual)** path gives you a step at a time and is the right choice
for production change windows. Both are documented here.

For first-time install and configuration, see
[`INSTALLATION.md`](/docs/openwatch/installation). For the database backup and restore
commands referenced below, see [`BACKUP_RECOVERY.md`](/docs/openwatch/backup-recovery). For
migration mechanics, see [`DATABASE_MIGRATIONS.md`](/docs/openwatch/database-migrations).

> Version note: the current release line is a pre-release (the `0.2.0-rc`
> series). Treat upgrades between pre-release builds as potentially breaking and
> always back up first.

## Quick upgrade (automatic, recommended)

On a single-instance install an upgrade is **one command**. The package
post-install scriptlet applies any pending database migrations automatically —
taking a backup restore point first — and restarts the service.

```bash
# RHEL / CentOS / Rocky / Alma / Fedora
sudo dnf update -y 'openwatch*' 'kensa-rules*'

# Debian / Ubuntu
sudo apt update && sudo apt install --only-upgrade openwatch kensa-rules
```

### What happens automatically (on upgrade)

The scriptlet runs **only on upgrade**, never on a fresh install, and does:

1. **Checks the database is reachable.** If it isn't, migrations are skipped
   with a warning (the upgrade doesn't fail) — run `openwatch migrate` manually
   once the DB is back, then `systemctl restart openwatch`.
2. **Stops the service** — so the new binary never runs against an old schema.
3. **Backs up the database** with `pg_dump` to `/var/lib/openwatch/backups/`
   (your restore point; the password is passed via the environment, never on the
   command line).
4. **Applies pending migrations.** Each runs in a transaction, so a failure
   rolls back atomically — data is never left half-migrated.
5. **On success → starts the service** on the new version.
   **On failure → leaves the service stopped**, prints the restore path, and
   exits non-zero so `dnf`/`apt` flag that the upgrade needs attention.

Preview what would change before upgrading:

```bash
sudo openwatch migrate --status
# -> "up to date — no migrations pending"  OR  "PENDING: N migration(s) ..."
```

### If an automatic migration fails

The service is left **stopped** and your data is intact (the failed migration
rolled back). Recover with:

```bash
# 1. read the error in the dnf/apt output or:  journalctl -u openwatch
# 2. fix the cause, then re-apply:
sudo openwatch migrate
sudo systemctl start openwatch
# on Debian, also clear the half-configured state:
sudo dpkg --configure -a
```

To restore the pre-upgrade dump instead, see [Rollback](#rollback).

### Backups: location, retention, opt-out

- Dumps live in `/var/lib/openwatch/backups/`.
- A systemd timer (`openwatch-backup-cleanup.timer`, daily) prunes dumps older
  than `BACKUP_RETENTION_DAYS` (default 30) but **always keeps the most recent
  one**, so you never lose your last restore point.
- Tune in `/etc/openwatch/upgrade.conf`: `AUTO_BACKUP=yes|no` (set `no` only if
  you run your own verified pre-upgrade backups), `BACKUP_DIR`,
  `BACKUP_RETENTION_DAYS`.

## Controlled (manual) upgrade

The remaining sections are the manual, step-at-a-time path — for production
change windows, multi-step validation, or when `AUTO_BACKUP=no`.

## Before you upgrade

Run through this checklist on the running host:

- [ ] Read the release notes for the target version.
- [ ] Confirm the service is healthy:
      `curl -k https://localhost:8443/api/v1/health`
      (expect `{"status":"healthy","db_connected":true,"version":"<version>"}`).
- [ ] Record the current version: `openwatch --version`.
- [ ] Record the current migration version (printed at the end of
      `openwatch migrate`, or query `goose_db_version` — see
      [Record the migration version](#record-the-migration-version)).
- [ ] Take a full PostgreSQL backup (see [`BACKUP_RECOVERY.md`](/docs/openwatch/backup-recovery)).
- [ ] Back up `/etc/openwatch/` (config, `secrets.env`, and `tls/`).
- [ ] Confirm free disk space with `df -h /var/lib/openwatch /var`.
- [ ] Schedule a maintenance window and notify users.

### Record the migration version

Migrations are tracked in the `goose_db_version` table. Capture the current
version so you know what the database looked like before the upgrade:

```bash
sudo -u openwatch env $(cat /etc/openwatch/secrets.env | xargs) \
  psql "$OPENWATCH_DATABASE_DSN" \
  -c "SELECT max(version_id) FROM goose_db_version;"
```

## How migrations work

`openwatch migrate` applies every pending up-migration in
`internal/db/migrations/` using goose, then prints the resulting version and the
list of migration files. The command is idempotent: it applies only migrations
not yet recorded in `goose_db_version`, so it is safe to re-run.

There is no down-migration or `downgrade` subcommand. Migrations are forward-only.
To revert a schema change you restore the pre-upgrade database backup (see
[Rollback](#rollback)). Plan upgrades accordingly: the database backup is your
rollback path, not a reverse migration.

## Standard upgrade

These steps assume the OpenWatch user is `openwatch` and the database DSN is in
`/etc/openwatch/secrets.env` as `OPENWATCH_DATABASE_DSN`, matching the install
guide.

### Step 1 — Back up the database

Take a fresh dump immediately before the upgrade (commands in
[`BACKUP_RECOVERY.md`](/docs/openwatch/backup-recovery)). Do not skip this: it is the only
rollback path for schema changes.

### Step 2 — Stop the service

```bash
sudo systemctl stop openwatch
```

Stopping the service quiesces the API, the embedded worker loops, and the
PostgreSQL-native job queue before the schema changes.

### Step 3 — Install the new package

On RHEL-family hosts (RPM):

```bash
sudo dnf upgrade ./openwatch-<new-version>.<arch>.rpm
```

On Debian/Ubuntu hosts (DEB):

```bash
sudo apt install ./openwatch_<new-version>_<arch>.deb
```

Both packages replace `/usr/bin/openwatch`, refresh the systemd unit, and run
`systemctl daemon-reload` in their post-install scripts. The config files under
`/etc/openwatch/` are marked as config files and are not overwritten on upgrade;
review the new package's default `openwatch.toml` against yours for new keys.

Confirm the binary version:

```bash
openwatch --version
```

### Step 4 — Validate the resolved config

Catch missing or renamed config keys before starting the server:

```bash
sudo -u openwatch openwatch check-config --config /etc/openwatch/openwatch.toml
```

This prints the resolved configuration with secrets redacted and exits non-zero
if validation fails. Config layering, highest precedence first: CLI flags > env
vars (`OPENWATCH_<SECTION>_<KEY>`) > the TOML file > built-in defaults.

### Step 5 — Apply migrations

```bash
sudo -u openwatch env $(cat /etc/openwatch/secrets.env | xargs) \
  openwatch migrate --config /etc/openwatch/openwatch.toml
```

The command prints the current version, the count of migration files, and each
filename, then `migrations applied`. If it fails, the service is still stopped —
fix the cause or restore the backup (see [Rollback](#rollback)) before starting.

### Step 6 — Start the service

```bash
sudo systemctl start openwatch
sudo systemctl status openwatch
```

### Step 7 — Verify the upgrade

```bash
# Health and reported version.
curl -k https://localhost:8443/api/v1/health
curl -k https://localhost:8443/api/v1/version

# Watch the structured logs for the startup line and any errors.
sudo journalctl -u openwatch -n 100 --no-pager

# Confirm the database is reachable from the host.
sudo -u openwatch env $(cat /etc/openwatch/secrets.env | xargs) \
  psql "$OPENWATCH_DATABASE_DSN" -c "SELECT 1;"
```

The `version` field in both `/api/v1/health` and `/api/v1/version` should report
the new version. Sign in at `https://<host>:8443/` and confirm the UI loads.

## Rollback

Because migrations are forward-only, rolling back a release that changed the
schema means restoring the pre-upgrade database backup and reinstalling the
previous package.

### Code-only rollback (no migration ran)

If the upgrade failed before Step 5, or the target version applied no new
migrations, reinstall the previous package and restart:

```bash
sudo systemctl stop openwatch
# RHEL family:
sudo dnf install ./openwatch-<old-version>.<arch>.rpm
# Debian/Ubuntu:
sudo apt install ./openwatch_<old-version>_<arch>.deb
sudo systemctl start openwatch
curl -k https://localhost:8443/api/v1/health
```

### Full rollback (migrations ran)

If Step 5 applied new migrations, restore the pre-upgrade database backup, then
reinstall the previous binary:

```bash
# 1. Stop the service.
sudo systemctl stop openwatch

# 2. Restore the pre-upgrade database dump
#    (exact pg_restore/psql commands: BACKUP_RECOVERY.md).

# 3. Reinstall the previous package (see Code-only rollback above).

# 4. Start and verify.
sudo systemctl start openwatch
curl -k https://localhost:8443/api/v1/health
```

Keep the pre-upgrade dump until you have validated the upgrade in production
(at least several days).

## Updating Kensa compliance rules

Kensa is the SSH-based compliance engine, integrated as a Go dependency
(`internal/kensa/`); its native YAML rules are compiled into the `openwatch`
binary. Rules therefore travel with
the binary — installing a new OpenWatch package is what updates the bundled
rule set. There is no separate rule-pull or out-of-band rule-sync step. For the
Kensa/OpenWatch responsibility boundary, see
[`docs/KENSA_OPENWATCH_BOUNDARY.md`](../KENSA_OPENWATCH_BOUNDARY.md).

## Upgrading PostgreSQL

PostgreSQL is provisioned and operated independently of the OpenWatch package
(see [`INSTALLATION.md`](/docs/openwatch/installation)). A **PostgreSQL major-version upgrade**
(e.g. 15 to 16) is **never** performed by the OpenWatch package scriptlet — it is
a data-directory migration (`pg_upgrade` or dump/restore) that needs both server
versions and must be operator-supervised; doing it silently from a package
upgrade would risk the whole database. Plan it separately, with its own backup:
follow your distribution's procedure, stop `openwatch.service` first so no
connections are open, then start it again afterward and run the
[verification](#step-7--verify-the-upgrade) checks. (Minor PostgreSQL and
dependency updates are handled by `dnf`/`apt` via package dependencies — nothing
extra to do.)

## Troubleshooting

### Service fails to start after upgrade

```bash
sudo systemctl status openwatch
sudo journalctl -u openwatch -n 200 --no-pager
```

Common causes:

- Invalid or incomplete config — run
  `sudo -u openwatch openwatch check-config --config /etc/openwatch/openwatch.toml`.
- Missing database secret — confirm `/etc/openwatch/secrets.env` defines
  `OPENWATCH_DATABASE_DSN`.
- Missing signing/encryption key material — the server refuses to start without
  `[identity].jwt_private_key` and `[identity].credential_key_file`; the log line
  names the missing key.
- Schema not migrated — run Step 5.

### `migrate` fails

Re-run the command and read the error. The most common cause is the database
being unreachable or the DSN being wrong; verify with
`psql "$OPENWATCH_DATABASE_DSN" -c "SELECT 1;"`. Because migrations are
idempotent, a partial run can be retried after the underlying issue is fixed. If
the schema is in an unexpected state, restore the pre-upgrade backup.

### Health endpoint returns 503

A 503 from `/api/v1/health` means the service started but a dependency is
unhealthy — typically the database. Check `db_connected` in the response body
and confirm PostgreSQL is running and reachable.

## Post-upgrade checklist

- [ ] `/api/v1/health` returns `healthy` with `db_connected:true`.
- [ ] `/api/v1/version` reports the new version.
- [ ] `journalctl -u openwatch` shows a clean startup and no recurring errors.
- [ ] An administrator can sign in at `https://<host>:8443/`.
- [ ] A compliance scan completes end to end.
- [ ] The upgrade is recorded in your change log.
- [ ] The pre-upgrade backup is retained through the validation period.

---

# OpenWatch: Database migration guide
URL: https://www.hanalyx.com/docs/openwatch/database-migrations

**Last Updated:** 2026-06-22 · **Applies to:** OpenWatch 0.2.0-rc series (Go single-binary)

This guide covers how OpenWatch's PostgreSQL schema is versioned, how migrations
are applied in production, and how to add a new migration. OpenWatch is a single
Go binary (`/usr/bin/openwatch`) that serves the REST API and the embedded React
UI over HTTPS on port 8443. It uses PostgreSQL only — there is no MongoDB, Redis,
Celery, Alembic, or container runtime involved in migrations.

For end-to-end install and configuration, see
[`docs/guides/INSTALLATION.md`](/docs/openwatch/installation). This
document focuses specifically on the migration mechanism.

## How migrations work

Migrations are plain SQL files embedded into the `openwatch` binary at build
time. The applier is [`pressly/goose`](https://github.com/pressly/goose) running
in SQL-flavor mode.

| Aspect | Value |
|--------|-------|
| Database | PostgreSQL (UUID primary keys on most tables) |
| Migration tool | `goose` v3 (SQL flavor), embedded via `go:embed` |
| Migration directory | `internal/db/migrations/*.sql` |
| File naming | `NNNN_description.sql` (zero-padded ascending integer) |
| Version table | `goose_db_version` (created and managed by goose) |
| Applier code | `internal/db/migrations/runner.go` (`Apply`, `Status`) |
| CLI entry point | `openwatch migrate` (`cmd/openwatch/main.go`, `cmdMigrate`) |

Because the SQL files are compiled into the binary, the version of the schema a
binary expects always travels with that binary. There is no separate migration
package to install or path to configure at runtime.

Each migration file has an up section and a down section, delimited by goose
annotations:

```sql
-- +goose Up
CREATE TABLE example (...);

-- +goose Down
DROP TABLE IF EXISTS example;
```

The applier (`migrations.Apply`) only ever runs the `Up` direction
(`goose.UpContext`). The `Down` blocks exist for completeness and local
development; OpenWatch does not expose a rollback subcommand (see
[Rollback](#rollback)).

## Applying migrations in production

Run the `migrate` subcommand. It connects with the configured database DSN,
applies every pending `Up` migration, and prints the resulting version and the
list of embedded migration files.

```bash
sudo -u openwatch env $(cat /etc/openwatch/secrets.env | xargs) \
    openwatch migrate
```

The DSN comes from `OPENWATCH_DATABASE_DSN` in `/etc/openwatch/secrets.env` (or
`[database].dsn` in `/etc/openwatch/openwatch.toml`). The command times out after
60 seconds, applies migrations idempotently (goose skips versions already recorded
in `goose_db_version`), and reports output like:

```
applying migrations against postgres://openwatch:***@127.0.0.1:5432/openwatch ...
  current version: 46
  migration files: 46
    - 0001_initial.sql
    - 0002_audit_events_taxonomy.sql
    ...
migrations applied
```

Run `openwatch migrate` after every package upgrade and before starting (or
restarting) the service, so the schema matches the binary. The systemd unit
(`/usr/lib/systemd/system/openwatch.service`, `ExecStart=/usr/bin/openwatch serve`)
does not run migrations on boot — `serve` and `migrate` are separate subcommands.

A typical upgrade sequence:

```bash
sudo systemctl stop openwatch
sudo dnf upgrade openwatch          # or: sudo apt install --only-upgrade openwatch
sudo -u openwatch env $(cat /etc/openwatch/secrets.env | xargs) openwatch migrate
sudo systemctl start openwatch
```

## Checking the current schema version

The `migrate` subcommand prints the current version after applying. To inspect
the version table directly with `psql`:

```bash
psql "$OPENWATCH_DATABASE_DSN" -c \
  "SELECT version_id, is_applied, tstamp FROM goose_db_version ORDER BY id DESC LIMIT 5;"
```

The highest `version_id` with `is_applied = true` is the current schema version.
That number corresponds to the `NNNN` prefix of the last applied migration file.

## Adding a new migration

1. Create a new file in `internal/db/migrations/` named with the next ascending
   integer, for example `0023_add_scan_findings.sql`. Migration order is driven
   by the filename prefix, not by dates.

2. Write the `Up` and `Down` blocks using goose annotations:

   ```sql
   -- +goose Up
   CREATE TABLE scan_findings (
       id          UUID         PRIMARY KEY,
       host_id     UUID         NOT NULL REFERENCES hosts(id) ON DELETE RESTRICT,
       rule_id     TEXT         NOT NULL,
       status      TEXT         NOT NULL CHECK (status IN ('pass','fail','skipped','error')),
       created_at  TIMESTAMPTZ  NOT NULL DEFAULT now()
   );
   CREATE INDEX idx_scan_findings_host ON scan_findings (host_id);

   -- +goose Down
   DROP INDEX IF EXISTS idx_scan_findings_host;
   DROP TABLE IF EXISTS scan_findings;
   ```

3. Follow the conventions already in the tree:
   - Use `UUID` primary keys for new tables.
   - Add indexes for foreign keys and common query columns.
   - Make `Down` reverse `Up` exactly, dropping indexes before tables and using
     `IF EXISTS` guards.
   - Reference the owning behavioral spec in a comment when one exists (see
     existing files such as `0012_transaction_log.sql`).

4. Never edit a migration that has already shipped or been applied to a shared
   database. goose records each applied version; changing an applied file does
   not re-run it and leaves environments inconsistent. Add a new migration
   instead.

5. Verify locally:

   ```bash
   go build ./...
   go test ./internal/db/...
   openwatch migrate          # against a local dev PostgreSQL
   ```

   The `internal/db/` package includes tests that exercise the embedded
   migration set; run them before committing.

## Rollback

There is no `openwatch migrate down` subcommand. The applier only runs the `Up`
direction. The supported recovery path for a bad migration in production is
restore-from-backup, not an automated downgrade.

Plan accordingly:

- Back up the database before applying migrations on a production system (see
  [Backup before migrating](#backup-before-migrating)).
- For schema mistakes, prefer a new forward migration that corrects the prior
  one over any manual `DROP`.
- The `Down` blocks in each file are for local development and may be applied by
  hand with the `goose` CLI against a disposable database; they are not part of
  the production workflow.

> Roadmap / not yet implemented: a first-class rollback subcommand
> (`openwatch migrate down`) and a dry-run SQL preview are not present in the
> current binary. Do not assume they exist.

## Backup before migrating

Take a logical backup with `pg_dump` before applying migrations to any
environment you cannot afford to lose:

```bash
pg_dump "$OPENWATCH_DATABASE_DSN" \
  --format=custom \
  --file="openwatch_$(date -u +%Y%m%dT%H%M%SZ).dump"
```

Restore with `pg_restore` against a clean database if a migration must be
reverted:

```bash
pg_restore --clean --if-exists --dbname "$OPENWATCH_DATABASE_DSN" \
  openwatch_20260610T120000Z.dump
```

Run `pg_dump`/`pg_restore` from the host (or a PostgreSQL client package) — there
is no container to `exec` into.

## Troubleshooting

### `migrate` fails to connect

Symptom: `openwatch migrate: connect postgres://…: …`.

- Confirm PostgreSQL is reachable and the DSN is correct:

  ```bash
  psql "$OPENWATCH_DATABASE_DSN" -c "SELECT 1;"
  ```

- Confirm `OPENWATCH_DATABASE_DSN` is set in `/etc/openwatch/secrets.env` and
  uses the form `postgres://user:pass@host:port/db?sslmode=…`.
- Validate the resolved config without touching the database:

  ```bash
  sudo -u openwatch env $(cat /etc/openwatch/secrets.env | xargs) \
      openwatch check-config
  ```

### A migration fails partway

Symptom: `openwatch migrate: up: …` referencing a specific SQL error.

goose runs each migration in order and records a version only after it succeeds,
so a failure leaves the database at the last fully-applied version. To recover:

1. Read the error and inspect the current version:

   ```bash
   psql "$OPENWATCH_DATABASE_DSN" -c \
     "SELECT version_id, is_applied FROM goose_db_version ORDER BY id DESC LIMIT 5;"
   ```

2. Fix the offending migration file (only if it has never shipped) or write a
   corrective forward migration.

3. Re-run `openwatch migrate`. Already-applied versions are skipped.

If the schema was left in an inconsistent state by a partially-executed
statement, restore from the pre-migration backup.

### Service starts but behaves as if the schema is old

Confirm the binary version and the applied schema version line up:

```bash
openwatch --version
psql "$OPENWATCH_DATABASE_DSN" -c \
  "SELECT max(version_id) FROM goose_db_version WHERE is_applied;"
```

If the version is behind the binary, run `openwatch migrate` and restart:

```bash
sudo -u openwatch env $(cat /etc/openwatch/secrets.env | xargs) openwatch migrate
sudo systemctl restart openwatch
journalctl -u openwatch -n 50 --no-pager
```

## Reference

| Item | Location |
|------|----------|
| Migration files | `internal/db/migrations/*.sql` |
| Applier (`Apply`, `Status`, `List`) | `internal/db/migrations/runner.go`, `embed.go` |
| `migrate` subcommand | `cmd/openwatch/main.go` (`cmdMigrate`) |
| Config layering and DSN | `internal/config/`, `docs/guides/INSTALLATION.md` |
| systemd unit | `packaging/common/openwatch.service` |
| Install and upgrade flow | `docs/guides/INSTALLATION.md` |
| Compliance engine boundary | `docs/KENSA_OPENWATCH_BOUNDARY.md` |

OpenWatch's compliance engine, Kensa, runs SSH-based checks against native YAML
rules. There is no separate scan-content schema in this
database.

---

# OpenWatch: Linux Distribution Support Matrix
URL: https://www.hanalyx.com/docs/openwatch/linux-distribution-support

> **Scope.** OpenWatch targets Linux, but **not every Linux distribution is
> supported to the same degree** — and compliance *scanning* in particular is
> currently **RHEL-family only**, because that is what the bundled Kensa rule
> corpus covers. This page states, with evidence, which distributions work for
> (1) running the OpenWatch server and (2) being added as a managed/scanned
> host.

**Last verified:** 2026-06-16 against Kensa rule corpus **v0.4.3** (539 rules).

---

## TL;DR

- **Compliance scanning works on RHEL 8 / 9 / 10 and its binary-compatible
  rebuilds** (Rocky, AlmaLinux, CentOS Stream, Oracle Linux). **Every bundled
  rule declares `platforms: family: rhel`** — there are *no* Ubuntu, Debian,
  Fedora, or SUSE rules in the corpus.
- **Adding a non-RHEL host (e.g. Fedora, Ubuntu) is not blocked**, but the scan
  will **skip 100% of rules** because none apply to that platform. This is
  intentional: running RHEL hardening checks against a different distro would
  report *wrong* compliance, so Kensa skips rather than misreport.
- **Host discovery and Server Intelligence are OS-agnostic** (plain SSH +
  portable probes), so they nominally work on any SSH-reachable Linux —
  but they produce no compliance posture without applicable scan rules.

---

## 1. OpenWatch server (where the application runs)

OpenWatch ships as native packages and containers built for:

| Platform | Form | Notes |
|----------|------|-------|
| **RHEL 9 / CentOS Stream 9** | native **RPM** | Built/tested on CentOS Stream 9 (`packaging/rpm/`). RHEL 9, Rocky 9, AlmaLinux 9 are binary-compatible. |
| **Ubuntu 24.04 LTS** | native **DEB** | Built/tested on Ubuntu 24.04 (`packaging/deb/`). |
| **Container** | OCI images | Backend/worker on UBI 9, db/frontend on Alpine. FIPS via the OpenSSL 3.x FIPS provider. |

Other distributions may run the server from source (Go 1.26 + PostgreSQL 16),
but only the above are packaged, tested, and released.

> The server OS is **independent** of the managed-host OS — you can run the
> OpenWatch server on Ubuntu and scan RHEL hosts, or vice-versa.

---

## 2. Managed / scanned hosts

A host moves through three phases after you add it. Each has different OS
sensitivity:

| Phase | What it does | OS sensitivity |
|-------|--------------|----------------|
| **Discovery** | SSH in, read `/etc/os-release`, fingerprint OS/CPU/mem/disk | **OS-agnostic** — works on any SSH-reachable Linux |
| **Server Intelligence** | Collect packages/services/users/network/firewall | **OS-agnostic** — portable probes; `rpm -qa` *or* `dpkg -l`, `firewall-cmd`/`ufw`/`nft`/`iptables` |
| **Compliance scan (Kensa)** | Evaluate hardening rules, produce posture | **RHEL-family only** — the bundled corpus is 100% `family: rhel` |

### Support matrix

| Distribution | Discovery | Intelligence | Compliance scan | Overall |
|--------------|:---------:|:------------:|:---------------:|---------|
| **RHEL 8 / 9 / 10** | ✅ | ✅ | ✅ full | **Supported** |
| **Rocky Linux 8 / 9** | ✅ | ✅ | ✅ (matches `family: rhel` via `ID_LIKE`) | **Supported** |
| **AlmaLinux 8 / 9** | ✅ | ✅ | ✅ (matches `family: rhel` via `ID_LIKE`) | **Supported** |
| **CentOS Stream 9** | ✅ | ✅ | ✅ (matches `family: rhel` via `ID_LIKE`) | **Supported** |
| **Oracle Linux 8 / 9** | ✅ | ✅ | ✅ (matches `family: rhel` via `ID_LIKE`) | **Supported** |
| **Fedora** | ✅ | ✅ | ❌ **all rules skip** | **Not supported for scanning** |
| **Ubuntu 22.04 / 24.04** | ✅ | ✅ | ❌ **all rules skip** | **Inventory only** |
| **Debian 12** | ✅ | ✅ | ❌ **all rules skip** | **Inventory only** |
| **SUSE / openSUSE / SLES** | ✅ | ✅ | ❌ **all rules skip** | **Inventory only** |
| **Alpine / Arch / Gentoo / other** | ✅ (best-effort) | ⚠️ partial | ❌ **all rules skip** | **Unsupported** |

Legend: ✅ works · ⚠️ partial/unverified · ❌ no coverage.

> **"Inventory only"** means discovery + Server Intelligence populate the host
> (OS, packages, services, etc.) but there is **no compliance posture** — every
> scan reports 0 applicable rules. These distros are *recognized*, just not
> *scannable* with today's corpus.

---

## 3. Why a Fedora (or Ubuntu) host scans nothing

This is the behaviour you will see and it is **working as designed**, not a
crash:

1. **Discovery succeeds.** OpenWatch reads `/etc/os-release` and stores the
   `os_family`. The family is the lower-cased `ID` field
   (`internal/intelligence/discovery/helpers.go:31` `deriveOSFamily`):
   - Fedora → `os_family = "fedora"`
   - Ubuntu → `os_family = "ubuntu"`
   - The `ID_LIKE`→`rhel` rollup only runs when `ID` is **empty**, so a host
     that advertises its own `ID` keeps it. (RHEL clones still match because
     Kensa reads their `ID_LIKE`, which contains `rhel`.)
2. **Server Intelligence succeeds.** The collector branches on no OS family; it
   runs `rpm -qa … || dpkg -l` and portable probes, with partial-success
   semantics (a failed command leaves a field empty, it does not fail the
   cycle).
3. **The compliance scan skips everything.** Kensa SSHes to the host, reads
   `/etc/os-release`, and filters its corpus to rules whose `platforms` match
   the detected distro. **Every rule in v0.4.3 declares `platforms: family:
   rhel`** (version-pinned to `rhel8`/`rhel9`/`rhel10`). A Fedora or Ubuntu
   host matches none, so **all ~539 rules are skipped**.

Skipping is the correct outcome: applying RHEL 9 STIG/CIS checks to Fedora 40 or
Ubuntu 24.04 would evaluate the wrong files, services, and defaults and report
**false** compliance.

> **Did Server Intelligence actually fail on your host?** The collector is
> OS-portable, so a *distro* mismatch does not fail it. A genuine intelligence
> failure usually points at the **SSH/sudo/connectivity** for that specific host
> (e.g. the credential can't `sudo`, or the host is unreachable) rather than the
> distribution. Check the host's connectivity tile and the audit log for the
> actual error before assuming it is a Fedora limitation.

---

## 4. How OS family is determined

`deriveOSFamily(osID, osIDLike)` —
`internal/intelligence/discovery/helpers.go:31`:

| Precedence | Source | Result |
|-----------|--------|--------|
| 1 | `/etc/os-release` `ID` (lower-cased) | returned verbatim when non-empty (`rhel`, `ubuntu`, `rocky`, `fedora`, …) |
| 2 | first recognized token in `ID_LIKE` (only if `ID` empty) | rolled up to `rhel` / `debian` / `suse` / `alpine` / `arch` / `gentoo` |
| 3 | neither recognized | `"other"` |

The stored `os_family` drives the frontend OS label and the framework-lens
filter (`internal/server/host_compliance_lens_handler.go`, `osFamilyTokens`).
Note `fedora` is **not** in `osFamilyTokens`, so a Fedora host is also offered no
version-pinned framework lenses.

---

## 5. Adding support for another distribution

Compliance coverage is defined by the **Kensa rule corpus**, not by OpenWatch
application code. To make (say) Ubuntu or Fedora scannable, the
[Kensa](https://github.com/Hanalyx/kensa) project must ship rules that declare
those platforms in their `platforms:` block (e.g. `family: debian` /
`family: fedora`) with the appropriate framework mappings (CIS Ubuntu, etc.).

> **Status:** the **Kensa team is actively expanding distribution coverage.**
> Because applicability lives entirely in the rule corpus, broader distro
> support arrives by **bundling a newer `kensa-rules` package — with no
> OpenWatch code change.** Discovery and Server Intelligence already work on
> those hosts today; only the rules are missing. This matrix should be
> re-verified (the "Last verified" corpus version at the top) whenever the
> bundled Kensa version is bumped.

Until that corpus lands, treat non-RHEL hosts as **inventory-only**: useful for
visibility (packages, services, drift on the intelligence side) but without a
compliance score.

---

## Evidence

- OS-family derivation: `internal/intelligence/discovery/helpers.go:31`
- OS-release parsing: `internal/intelligence/probe/probe.go` (`ParseOSRelease`)
- Collector (OS-agnostic, `rpm || dpkg`): `internal/intelligence/collector/collector.go`
- Kensa invocation / rule load: `internal/kensa/scanfunc.go`, `internal/kensa/catalog.go`
- Framework-lens OS filter (`osFamilyTokens`): `internal/server/host_compliance_lens_handler.go`
- Rule corpus applicability (verified): Kensa v0.4.3 — **539/539 rules `platforms: family: rhel`**, pinned `rhel8`/`rhel9`/`rhel10`
- Framework mappings: `CLAUDE.md` (CIS RHEL 9 v2.0.0, STIG RHEL 9 V2R7)

---

# OpenWatch: Configuration and environment reference
URL: https://www.hanalyx.com/docs/openwatch/environment-reference

**Last Updated:** 2026-06-22 · **Applies to:** OpenWatch 0.2.0-rc series (Go single-binary)

This document is the field reference for how you configure the OpenWatch Go
binary: the TOML file, the environment-variable overrides, and the on-disk paths
that the service reads at boot.

OpenWatch is a single Go binary (`/usr/bin/openwatch`) that serves both the REST
API and the embedded React UI over HTTPS on port `8443`. It uses PostgreSQL only.
There is no container runtime, no Redis, no Celery, and no separate web tier to
configure. The compliance engine is Kensa (SSH-based, native
YAML rules).

For end-to-end install and first-run steps, see
[`docs/guides/INSTALLATION.md`](/docs/openwatch/installation). This page
documents only the configuration surface and does not repeat the install flow.

## Configuration layering

The binary resolves configuration from four layers. Higher layers win:

| Precedence | Layer | Source |
|------------|-------|--------|
| 1 (highest) | CLI flags | `--listen`, `--log-level`, `--config` |
| 2 | Environment variables | `OPENWATCH_<SECTION>_<KEY>` (see below) |
| 3 | TOML file | `/etc/openwatch/openwatch.toml` (override with `--config`) |
| 4 (lowest) | Built-in defaults | compiled into the binary |

The layering is defined in `internal/config/load.go` and `internal/config/config.go`.
Only the environment variables listed in `internal/config/load.go` (`envOverrides`)
are recognized. There is no reflection-based mapping, so an unrecognized
`OPENWATCH_*` variable has no effect.

Run `openwatch check-config` to print the resolved configuration (secrets redacted)
and validate it. Exit code `0` means valid, `1` means invalid.

## TOML configuration

The default file lives at `/etc/openwatch/openwatch.toml` (mode `0640`, owner
`root:openwatch`). The package ships it with `[server]`, `[database]`, and
`[logging]` populated. The `[identity]` section is optional in the file; when it
is absent the defaults below apply, and you can set the two key paths through the
TOML file or through the matching environment variables.

The full set of recognized keys, their sections, defaults, and the environment
variable that overrides each one:

| Section | Key | Default | Env override |
|---------|-----|---------|--------------|
| `server` | `listen` | `0.0.0.0:8443` | `OPENWATCH_SERVER_LISTEN` |
| `server` | `tls_cert` | `/etc/openwatch/tls/cert.pem` | `OPENWATCH_SERVER_TLS_CERT` |
| `server` | `tls_key` | `/etc/openwatch/tls/key.pem` | `OPENWATCH_SERVER_TLS_KEY` |
| `database` | `dsn` | `postgres://openwatch@localhost/openwatch?sslmode=disable` | `OPENWATCH_DATABASE_DSN` |
| `database` | `max_connections` | `25` | `OPENWATCH_DATABASE_MAX_CONNECTIONS` |
| `logging` | `level` | `info` | `OPENWATCH_LOGGING_LEVEL` |
| `logging` | `format` | `json` | `OPENWATCH_LOGGING_FORMAT` |
| `identity` | `jwt_private_key` | `/etc/openwatch/keys/jwt_private.pem` | `OPENWATCH_IDENTITY_JWT_PRIVATE_KEY` |
| `identity` | `credential_key_file` | `/etc/openwatch/keys/credential.key` | `OPENWATCH_IDENTITY_CREDENTIAL_KEY_FILE` |

These are the only configuration keys the binary reads. The values are validated
by `internal/config/validate.go`.

Example `/etc/openwatch/openwatch.toml`:

```toml
[server]
listen   = "0.0.0.0:8443"
tls_cert = "/etc/openwatch/tls/cert.pem"
tls_key  = "/etc/openwatch/tls/key.pem"

[database]
# Keep the password out of this file; set OPENWATCH_DATABASE_DSN in
# /etc/openwatch/secrets.env instead.
dsn             = "postgres://openwatch@localhost/openwatch?sslmode=disable"
max_connections = 25

[logging]
level  = "info"
format = "json"

[identity]
jwt_private_key     = "/etc/openwatch/keys/jwt_private.pem"
credential_key_file = "/etc/openwatch/keys/credential.key"
```

## Environment variables

### Configuration overrides

Each variable maps to exactly one TOML key (see the table above). The format is
`OPENWATCH_<SECTION>_<KEY>`, all uppercase. The recognized set is fixed:

| Variable | Overrides | Notes |
|----------|-----------|-------|
| `OPENWATCH_SERVER_LISTEN` | `[server].listen` | Must be `host:port`. |
| `OPENWATCH_SERVER_TLS_CERT` | `[server].tls_cert` | Path to the TLS certificate. |
| `OPENWATCH_SERVER_TLS_KEY` | `[server].tls_key` | Path to the TLS private key. |
| `OPENWATCH_DATABASE_DSN` | `[database].dsn` | Must parse as `postgres://` or `postgresql://`. |
| `OPENWATCH_DATABASE_MAX_CONNECTIONS` | `[database].max_connections` | Integer greater than `0`. |
| `OPENWATCH_LOGGING_LEVEL` | `[logging].level` | One of `debug`, `info`, `warn`, `error`. |
| `OPENWATCH_LOGGING_FORMAT` | `[logging].format` | One of `json`, `text`. |
| `OPENWATCH_IDENTITY_JWT_PRIVATE_KEY` | `[identity].jwt_private_key` | PEM RSA private key, mode `0600`. |
| `OPENWATCH_IDENTITY_CREDENTIAL_KEY_FILE` | `[identity].credential_key_file` | 32-byte AES-256 key, mode `0600`. |

The canonical place to set the database secret is `/etc/openwatch/secrets.env`,
which the systemd unit reads through `EnvironmentFile=-/etc/openwatch/secrets.env`.
Keeping the DSN there keeps the password out of the world-readable TOML file:

```bash
sudo tee /etc/openwatch/secrets.env >/dev/null <<'EOF'
OPENWATCH_DATABASE_DSN=postgres://openwatch:CHANGE_ME@localhost/openwatch?sslmode=require
EOF
sudo chown root:openwatch /etc/openwatch/secrets.env
sudo chmod 0640 /etc/openwatch/secrets.env
```

### Other environment variables read at runtime

| Variable | Default | Read by | Purpose |
|----------|---------|---------|---------|
| `OPENWATCH_LICENSE_FILE` | `/etc/openwatch/license.lic` | `serve`, `worker` | Path to the OpenWatch+ license file. A missing file is not fatal; the service runs at the free tier. |
| `OPENWATCH_POLICIES_DIR` | `/etc/openwatch/policies` | `serve` | Directory scanned when an admin triggers a policy reload through the API. |
| `OPENWATCH_DEV_MODE` | unset | `serve` | When set to `true`, accepts unsigned policy envelopes. Development only; never set in production. |

Standard PostgreSQL libpq environment variables (for example `PGSSLROOTCERT`) are
honored by the underlying driver when present, but OpenWatch itself only reads the
DSN. Prefer encoding connection options in the DSN query string
(`?sslmode=verify-full&...`) so the configuration stays in one place.

## On-disk paths

| Path | Owner / mode | Purpose |
|------|--------------|---------|
| `/usr/bin/openwatch` | `root`, `0755` | The single binary (API + UI + CLI). |
| `/etc/openwatch/openwatch.toml` | `root:openwatch`, `0640` | Main config file. |
| `/etc/openwatch/secrets.env` | `root:openwatch`, `0640` | `OPENWATCH_DATABASE_DSN` and other secrets; loaded by systemd. |
| `/etc/openwatch/tls/cert.pem` | readable by `openwatch` | TLS server certificate. |
| `/etc/openwatch/tls/key.pem` | `openwatch`, `0600` | TLS server private key. |
| `/etc/openwatch/keys/jwt_private.pem` | `openwatch`, `0600` | RSA key that signs access and refresh JWTs. |
| `/etc/openwatch/keys/credential.key` | `openwatch`, `0600` | AES-256 key encrypting MFA secrets and stored SSH credentials. |
| `/etc/openwatch/license.lic` | readable by `openwatch` | Optional OpenWatch+ license. |
| `/var/lib/openwatch` | `openwatch` | Service state directory (`ReadWritePaths` in the unit). |
| `/var/log/openwatch` | `openwatch` | Log directory; journald remains the primary log sink. |

The systemd unit (`packaging/common/openwatch.service`) runs the service as the
`openwatch` user with `ProtectSystem=strict` and writes only to
`/var/lib/openwatch` and `/var/log/openwatch`. Both `[server].tls_key`,
`[identity].jwt_private_key`, and `[identity].credential_key_file` must be present
and readable, or `openwatch serve` exits with an explicit error rather than falling
back to ephemeral keys.

## CLI subcommands

The binary's lifecycle is driven through these subcommands
(`cmd/openwatch/main.go`). All of them honor the same configuration layering.

| Subcommand | Purpose |
|------------|---------|
| `serve` | Run the HTTPS API + UI server. This is the default when no subcommand is given, which is what the systemd unit invokes. |
| `worker` | Run the scan-job claimer/dispatcher loop against the PostgreSQL-native queue. |
| `migrate` | Apply pending database migrations (`internal/db/migrations/`) and print the resulting version. |
| `create-admin` | Create the first admin user. Requires `--username` and `--email`; reads the password from `--password` or stdin. |
| `check-config` | Print the resolved, secret-redacted config and validate it. |

Global flags: `--config <path>`, `--listen <host:port>`, `--log-level <level>`,
`--version`, `-h`/`--help`.

Validate configuration before starting the service:

```bash
sudo -u openwatch env $(cat /etc/openwatch/secrets.env | xargs) \
    openwatch check-config --config /etc/openwatch/openwatch.toml
```

## Service control and verification

OpenWatch runs under systemd as `openwatch.service`:

```bash
sudo systemctl enable --now openwatch
sudo systemctl status openwatch
sudo journalctl -u openwatch -f
```

Logs are structured JSON on stdout/stderr, captured by journald. Boot, shutdown,
and per-request events carry a correlation ID. Health check:

```bash
curl -k https://localhost:8443/api/v1/health
```

The API is served under `/api/v1/`; `api/openapi.yaml` is the contract
source of truth. Role definitions live in
[`docs/engineering/rbac_registry.md`](../engineering/rbac_registry.md) and
`internal/auth/permissions.yaml`.

## Operational runbooks

These are operational procedures for the single binary on systemd with PostgreSQL.
All commands assume the default paths above.

### Service down

The service is not responding on `8443`.

1. Check the unit state and recent logs:

   ```bash
   sudo systemctl status openwatch
   sudo journalctl -u openwatch -n 200 --no-pager
   ```

2. If the service failed to start, validate the config and confirm the key files
   exist and are readable by the `openwatch` user:

   ```bash
   sudo -u openwatch env $(cat /etc/openwatch/secrets.env | xargs) openwatch check-config
   sudo ls -l /etc/openwatch/tls/ /etc/openwatch/keys/
   ```

   A missing or unreadable `tls_key`, `jwt_private_key`, or `credential_key_file`
   causes `serve` to exit immediately with an explicit error in the journal.

3. Confirm PostgreSQL is up and the DSN is reachable:

   ```bash
   sudo systemctl status postgresql
   psql "$OPENWATCH_DATABASE_DSN" -c 'SELECT 1;'
   ```

4. Restart and watch the logs:

   ```bash
   sudo systemctl restart openwatch
   sudo journalctl -u openwatch -f
   ```

### Service down after an upgrade

If the service fails immediately after a package upgrade, a migration may be
pending. Run it as the service user, then restart:

```bash
sudo -u openwatch env $(cat /etc/openwatch/secrets.env | xargs) openwatch migrate
sudo systemctl restart openwatch
```

### Disk full

A full disk most often manifests as failed writes to `/var/lib/openwatch`,
`/var/log/openwatch`, or the PostgreSQL data directory.

1. Find what filled up:

   ```bash
   df -h
   sudo du -xh /var/log/openwatch /var/lib/openwatch | sort -h | tail -20
   ```

2. journald is the primary log sink. If journald is the culprit, vacuum it:

   ```bash
   sudo journalctl --disk-usage
   sudo journalctl --vacuum-time=7d
   ```

3. If PostgreSQL's volume is full, free space there (archive or drop old data per
   your retention policy) before restarting the database and the service.

4. After freeing space, confirm recovery:

   ```bash
   sudo systemctl restart openwatch
   curl -k https://localhost:8443/api/v1/health
   ```

### High CPU

1. Identify the hot process:

   ```bash
   top -b -n1 | head -20
   sudo systemctl status openwatch
   ```

2. If the `openwatch` process is busy, check whether scan jobs are saturating the
   worker. Inspect the journal for scan and queue activity:

   ```bash
   sudo journalctl -u openwatch --since '15 min ago' | grep -i 'scan\|queue\|worker'
   ```

3. If PostgreSQL is the hot process, look for long-running or stuck queries:

   ```bash
   psql "$OPENWATCH_DATABASE_DSN" -c \
     "SELECT pid, state, now() - query_start AS runtime, left(query, 80) AS query
        FROM pg_stat_activity
       WHERE state <> 'idle' ORDER BY runtime DESC LIMIT 10;"
   ```

4. Reduce database connection pressure with `OPENWATCH_DATABASE_MAX_CONNECTIONS`
   (or `[database].max_connections`) if the pool is oversized for the host, then
   restart the service.

### Security incident

1. Contain. Stop the service to halt all API, UI, and scan activity:

   ```bash
   sudo systemctl stop openwatch
   ```

2. Preserve evidence. Export the journal and protect the audit trail before any
   remediation:

   ```bash
   sudo journalctl -u openwatch --since '24 hours ago' > /var/tmp/openwatch-incident.log
   ```

   Authentication and authorization events are emitted to the audit log and the
   journal. Review them for the affected window.

3. Rotate credentials. If key material may be exposed, rotate the database
   password (update `OPENWATCH_DATABASE_DSN` in `/etc/openwatch/secrets.env`), and
   rotate the JWT signing key and credential key only with a planned procedure —
   replacing `credential.key` makes previously encrypted SSH credentials and MFA
   secrets unreadable, so re-enrollment is required.

4. Verify file ownership and modes have not drifted:

   ```bash
   sudo ls -l /etc/openwatch /etc/openwatch/keys /etc/openwatch/tls
   ```

   `secrets.env` and the key files must be owner-only or `root:openwatch` `0640`/`0600`.

5. Patch and restart only after the cause is understood:

   ```bash
   sudo systemctl start openwatch
   sudo journalctl -u openwatch -f
   ```

## Not yet implemented

The following capabilities existed in the archived Python/Docker stack but are not
present in the current Go binary. Do not configure them; they have no effect.

| Capability | Status |
|------------|--------|
| Prometheus `/metrics` endpoint | Not implemented. Audit counters exist internally (`internal/audit/emit.go`) but are not exposed over HTTP. Use journald metrics and `pg_stat_*` views for observability. |
| Redis / Celery configuration | Removed. Background jobs use a PostgreSQL-native queue (`SKIP LOCKED`); there is nothing to configure. |
| MongoDB configuration | Removed. OpenWatch is PostgreSQL-only. |
| Container-runtime / docker-compose variables | Removed. The service is a native binary under systemd. |
| SMTP / LDAP environment variables | Not read by the binary today. Notification channels and SSO are configured through the API and database, not environment variables. |
| Separate CORS / `ALLOWED_ORIGINS` variable | Not a recognized config key. The UI is served from the same origin as the API by the single binary. |

---

# OpenWatch: API guide
URL: https://www.hanalyx.com/docs/openwatch/api-guide

**Last Updated:** 2026-06-22 · **Applies to:** OpenWatch 0.2.0-rc series (Go single-binary)

Most operators use the web UI for daily work — managing hosts, viewing fleet
health, reading compliance state, and triaging alerts. This guide is for
automation: scripting repetitive tasks, integrating with CI/CD, or building
tooling on top of OpenWatch.

OpenWatch is a single Go binary that serves both the REST API and the embedded
React UI over HTTPS on port `8443`. All API paths live under `/api/v1`. The
contract source of truth is `api/openapi.yaml` in the repository; the running
binary serves the same document, and `GET /api/v1/version` reports the build it
came from.

This guide reflects OpenWatch `0.2.0-rc.13`, a pre-release. The API surface is
still growing — endpoints that the legacy Python API exposed (scan execution,
remediation, exceptions, posture history, audit exports, the rule-reference
browser) are not yet part of `api/v1`. See [What is not yet in the
API](#what-is-not-yet-in-the-api) before you script against them.

When the OpenAPI document and this guide disagree, the OpenAPI document wins.

---

## Conventions

- Base URL is `https://<host>:8443`. The server is HTTPS-only. In a default
  install the certificate at `/etc/openwatch/tls/` is self-signed, so add
  `--cacert` (or, for a throwaway lab box only, `-k`) to your `curl` calls.
- Resource identifiers are UUIDs.
- Timestamps are ISO 8601 / RFC 3339 (for example `2026-06-10T14:30:00Z`).
- Mutating endpoints that exist to be retried safely take a required
  `Idempotency-Key` header (a unique string per logical operation). Replaying the
  same key with the same body returns the original result; replaying it with a
  different body returns `409`.
- An optional `X-Correlation-Id` header is propagated through logs and audit
  events. If you omit it, the server generates one and returns it in the
  response.

---

## Authentication

The API accepts two credential types. Both resolve to the same identity and
permission set:

- A `Bearer` access token in the `Authorization` header. This is the path for
  scripts and CI.
- The browser session cookie (`openwatch_session`), used by the web UI. Cookie
  rotation and the on-401 refresh flow are UI concerns and are not covered here.

Anonymous endpoints (`GET /api/v1/health`, `GET /api/v1/version`,
`POST /api/v1/auth/login`, `POST /api/v1/auth/refresh`) require no credential.
Everything else requires a valid identity.

### Log in

```bash
TOKEN=$(curl -s --cacert /etc/openwatch/tls/ca.crt \
  -X POST https://localhost:8443/api/v1/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"yourpassword"}' | jq -r '.access_token')
```

The request body is `{username, password}` with an optional `otp` (6 digits)
when the account has TOTP MFA enrolled. The response is:

```json
{
  "access_token": "…",
  "refresh_token": "…",
  "user": {"id": "…", "username": "admin", "email": "…", "role": "admin"}
}
```

All later examples assume `-H "Authorization: Bearer $TOKEN"`.

### Refresh, identity, and log out

| Method | Path | Purpose |
|--------|------|---------|
| `POST` | `/api/v1/auth/refresh` | Rotate the refresh token; returns a new access + refresh pair. Body: `{refresh_token}`. |
| `GET` | `/api/v1/auth/me` | Return the calling identity (`id`, `username`, `email`, `role`). |
| `GET` | `/api/v1/auth/me/permissions` | Return the caller's effective permission strings. |
| `POST` | `/api/v1/auth/logout` | Revoke the calling session (`204`). |
| `POST` | `/api/v1/auth/password:change` | Change the caller's password. Body: `{current_password, new_password}`. |
| `POST` | `/api/v1/auth/mfa:enroll` | Begin TOTP enrollment; returns a `provisioning_uri`. |
| `POST` | `/api/v1/auth/mfa:verify` | Confirm an enrolled secret. Body: `{otp}`. |

---

## Authorization

Authorization is permission-based, not role-based, at the endpoint level. Each
protected endpoint declares the permission it requires (visible as
`x-required-permission` in `api/openapi.yaml`, for example `host:read` or
`host:write`). Built-in roles bundle permission sets:

| Role | Intent |
|------|--------|
| `viewer` | Read-only access |
| `auditor` | Read plus audit/compliance review |
| `security_admin` | Security configuration |
| `admin` | Full system administration |

A caller missing the required permission receives `403`. The full permission and
role registry is the source of truth at
[`docs/engineering/rbac_registry.md`](../engineering/rbac_registry.md); it is
served at runtime via `GET /api/v1/auth/permissions:registry`.

---

## Hosts

| Method | Path | Permission | Purpose |
|--------|------|------------|---------|
| `GET` | `/api/v1/hosts` | `host:read` | List hosts. Query: `environment`, `tag`. |
| `POST` | `/api/v1/hosts` | `host:write` | Create a host. |
| `GET` | `/api/v1/hosts/{id}` | `host:read` | Host detail with liveness and compliance summary. Query: `framework`. |
| `PATCH` | `/api/v1/hosts/{id}` | `host:write` | Update mutable host fields. |
| `DELETE` | `/api/v1/hosts/{id}` | `host:delete` | Soft-delete a host (`204`; sets `deleted_at`). |
| `GET` | `/api/v1/hosts/{host_id}/monitoring/history` | `host:read` | Monitoring history. |
| `POST` | `/api/v1/hosts/{host_id}/maintenance` | `host:write` | Toggle maintenance mode. |
| `POST` | `/api/v1/hosts/{id}/connectivity:check` | `host:write` | Run a connectivity check (idempotent). |
| `GET` | `/api/v1/hosts/{id}/system-info` | `host:read` | Latest collected system intelligence. |
| `POST` | `/api/v1/hosts/{id}/discovery:run` | `host:write` | Run host discovery (idempotent). |
| `POST` | `/api/v1/hosts/{host_id}/credentials:resolve` | `host:read` | Resolve the effective credential for a host. |

### Create a host

```bash
curl -s --cacert /etc/openwatch/tls/ca.crt \
  -X POST https://localhost:8443/api/v1/hosts \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "hostname": "rhel9-web01.example.com",
    "ip_address": "10.0.1.50",
    "port": 22,
    "environment": "production",
    "tags": ["web", "rhel9"]
  }'
```

`hostname` and `ip_address` are required; `port`, `display_name`, `description`,
`environment`, `tags`, `group_id`, and `username` are optional. A successful
create returns `201`; a duplicate hostname in the same environment returns `409`.

---

## Credentials

SSH credentials are stored separately from hosts and scoped either to the whole
system (`scope: system`) or to one host (`scope: host`). Secret material is
encrypted at rest and never returned in responses.

| Method | Path | Purpose |
|--------|------|---------|
| `GET` | `/api/v1/credentials` | List credentials (secrets redacted). |
| `POST` | `/api/v1/credentials` | Create a credential. |
| `GET` | `/api/v1/credentials/{id}` | Get one credential. |
| `PATCH` | `/api/v1/credentials/{id}` | Update a credential. |
| `DELETE` | `/api/v1/credentials/{id}` | Delete a credential. |
| `POST` | `/api/v1/credentials/{id}:clone` | Clone to a new scope (secret inherited; no plaintext on the wire). |

A create body requires `scope`, `name`, `username`, and `auth_method` (one of
`ssh_key`, `password`, `both`). Provide `private_key` (and optional
`private_key_passphrase`) and/or `password` to match the chosen method.

---

## Fleet observability

These endpoints back the dashboard and require read access.

| Method | Path | Purpose |
|--------|------|---------|
| `GET` | `/api/v1/fleet/score` | Aggregate fleet compliance score. |
| `GET` | `/api/v1/fleet/liveness` | Fleet liveness breakdown. |
| `GET` | `/api/v1/fleet/top-failing-rules` | Rules failing across the most hosts. |
| `GET` | `/api/v1/fleet/top-failing-hosts` | Hosts with the most failing rules. |
| `GET` | `/api/v1/fleet/recent-changes` | Recent compliance state transitions. |
| `GET` | `/api/v1/fleet/connectivity/breakdown` | Connectivity status counts. |

---

## Alerts

| Method | Path | Purpose |
|--------|------|---------|
| `GET` | `/api/v1/alerts` | List alerts. |
| `GET` | `/api/v1/alerts/{id}` | Alert detail. |
| `POST` | `/api/v1/alerts/{id}:acknowledge` | Acknowledge an alert. |
| `POST` | `/api/v1/alerts/{id}:silence` | Silence an alert. |
| `POST` | `/api/v1/alerts/{id}:resolve` | Resolve an alert. |
| `POST` | `/api/v1/alerts/{id}:dismiss` | Dismiss an alert. |

---

## Intelligence and activity

| Method | Path | Purpose |
|--------|------|---------|
| `GET` | `/api/v1/intelligence/events` | Stream of intelligence-collection events. |
| `GET` | `/api/v1/intelligence/state/{host_id}` | Latest intelligence state for a host. |
| `GET` | `/api/v1/activity` | Unified recent-activity feed. |

---

## System configuration

Connectivity, intelligence-collection, and discovery behavior are configured
through the API. These are admin-level controls.

| Method | Path | Purpose |
|--------|------|---------|
| `GET` / `PUT` | `/api/v1/system/connectivity/config` | Connectivity polling config. |
| `GET` | `/api/v1/system/connectivity/status` | Connectivity worker status. |
| `GET` / `PUT` | `/api/v1/system/intelligence/config` | Intelligence-collection config. |
| `GET` / `PUT` | `/api/v1/system/discovery/config` | Discovery config. |
| `POST` | `/api/v1/system/discovery/sweep` | Trigger a discovery sweep (idempotent). |

---

## Users and roles

| Method | Path | Purpose |
|--------|------|---------|
| `GET` | `/api/v1/users` | List users. |
| `POST` | `/api/v1/users` | Create a user. Body: `{username, email, password}`. |
| `GET` | `/api/v1/users/{id}` | Get a user. |
| `PATCH` | `/api/v1/users/{id}` | Update a user. |
| `DELETE` | `/api/v1/users/{id}` | Delete a user. |
| `POST` | `/api/v1/users/{id}/roles:assign` | Assign a role. Body: `{role_id}`. |
| `POST` | `/api/v1/users/{id}/roles:unassign` | Remove a role. |
| `GET` | `/api/v1/roles` | List roles (built-in and custom). |
| `POST` | `/api/v1/roles:create` | Create a custom role. |

For first-admin bootstrap, prefer the CLI (`openwatch create-admin`) over the
API; see [Operations](#operations-the-cli-and-systemd).

---

## License

OpenWatch has a tiered license model (`free`, `openwatch_plus`, `enterprise`).
Premium-gated endpoints return `402` when the active tier lacks the feature.

| Method | Path | Purpose |
|--------|------|---------|
| `GET` | `/api/v1/license` | Current license tier, status, and features. |
| `POST` | `/api/v1/admin/license:verify` | Dry-run validate a license JWT without installing it. |

---

## Audit events

Every meaningful state change writes an audit event. The log is queryable and
cursor-paginated, newest first.

| Method | Path | Purpose |
|--------|------|---------|
| `GET` | `/api/v1/audit/events` | List audit events. |

Query parameters: `action`, `correlation_id`, `actor_type`, `since`, `until`
(both RFC 3339), `cursor`, and `limit` (1–200, default 50). Follow the `cursor`
field in each page to paginate.

---

## Health and version

These two endpoints are anonymous and are what monitoring should poll.

```bash
curl -s --cacert /etc/openwatch/tls/ca.crt https://localhost:8443/api/v1/health | jq
```

```json
{"status": "healthy", "db_connected": true, "version": "0.2.0-rc.13"}
```

`status` is `healthy` or `degraded`; the endpoint returns `503` when the service
cannot serve. `GET /api/v1/version` returns build metadata (`openwatch`, `kensa`,
`go`, `commit`, `build_time`).

---

## Error responses

Errors use a single envelope shape, not field-level Pydantic detail:

```json
{
  "error": {
    "code": "validation_failed",
    "fault": "client",
    "retryable": false,
    "human_message": "ip_address is required",
    "correlation_id": "…"
  }
}
```

`fault` is one of `client`, `server`, `policy`, or `external`. Status codes you
will encounter:

| Code | Meaning |
|------|---------|
| `400` | Bad request — invalid input or a violated business rule |
| `401` | Unauthorized — missing, expired, or invalid credential |
| `402` | Payment required — the license tier lacks this feature |
| `403` | Forbidden — the caller lacks the required permission |
| `404` | Not found |
| `405` | Method not allowed |
| `409` | Conflict — duplicate resource, or a reused `Idempotency-Key` with a different body |
| `502` | Bad gateway — an external dependency failed |
| `503` | Service unavailable — the service is degraded |

There is no API-layer request rate limiting in this release, and there is no
`422` validation status — validation failures return `400` with the envelope
above.

---

## Operations: the CLI and systemd

Automation that manages the deployment itself (rather than calling the API) uses
the `openwatch` binary and `systemd`, not Docker. The subcommands are:

| Command | Purpose |
|---------|---------|
| `openwatch serve` | Run the HTTPS API + UI server (the default subcommand; what `systemd` starts). |
| `openwatch worker` | Run the background job worker (PostgreSQL `SKIP LOCKED` queue). |
| `openwatch migrate` | Apply database migrations. |
| `openwatch create-admin` | Create the first admin user. |
| `openwatch check-config` | Validate `/etc/openwatch/openwatch.toml` and exit. |

Day-to-day lifecycle:

```bash
systemctl status openwatch
systemctl restart openwatch
journalctl -u openwatch -f
```

Configuration lives in `/etc/openwatch/openwatch.toml`, with environment
overrides of the form `OPENWATCH_<SECTION>_<KEY>` and the database DSN in
`/etc/openwatch/secrets.env` (`OPENWATCH_DATABASE_DSN`). For full install and
configuration steps, see
[`docs/guides/INSTALLATION.md`](/docs/openwatch/installation).

---

## What is not yet in the API

The compliance scanning workflow runs through Kensa and the background worker,
not yet through public REST endpoints. As of `0.2.0-rc.13`, `api/v1` does not
include:

- Scan execution or scan-result endpoints (`/api/v1/scans/…`).
- Remediation, compliance exceptions, posture history, or drift endpoints.
- Audit export / saved-query endpoints.
- A rule-reference browser endpoint.
- A Prometheus `/metrics` endpoint and a `/security-info` endpoint. Metrics are
  tracked as a roadmap item; use `GET /api/v1/health` for liveness today.

These are roadmap or worker-internal today. Do not script against them until
they appear in `api/openapi.yaml`. For how OpenWatch invokes Kensa, see
[`docs/KENSA_OPENWATCH_BOUNDARY.md`](../KENSA_OPENWATCH_BOUNDARY.md).

---

## What's next

- [Install guide](/docs/openwatch/installation) — install, configure, and run the service.
- [RBAC registry](../engineering/rbac_registry.md) — permission and role reference.
- [Kensa ↔ OpenWatch boundary](../KENSA_OPENWATCH_BOUNDARY.md) — how scanning works.
- `api/openapi.yaml` — the authoritative, always-current API contract.

---

# OpenWatch: Release notes
URL: https://www.hanalyx.com/docs/openwatch/release

Changelog for the OpenWatch Go rebuild, which lives at the repo root. The
legacy Python project was archived out of the repo on 2026-06-05.

Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
Versioning: [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

---

## [Unreleased]

---

## [0.2.0-rc.14] Eyrie — 2026-06-23

A maintenance candidate on top of rc.13: the Kensa engine moves to v0.6.0, a
known-hosts trust race is closed, hardened hosts that require
keyboard-interactive auth no longer report as falsely degraded, and the
operator guides + README were corrected to the Go reality.

### Changed
- **Kensa engine bumped to v0.6.0** ("atomicity engine" — remediation
  crash-recovery). The bundled rule corpus is now **538** (one rule removed
  upstream; was 539), verified by a live end-to-end scan against a RHEL host
  (completed, 538 rules, 0 errors). The three Kensa version pins (`go.mod`, the
  `internal/kensa` module constant, and the `kensa-executor` spec) move
  together. (#670)

### Security
- **Known-hosts `Put` fails closed on a concurrent first-use key conflict.**
  When a row already exists for a hostname with a different key — the first-use
  race where another connection recorded a key first — `Put` now returns a
  host-key mismatch instead of silently reporting success, so the
  trust-on-first-use callback can no longer accept an unverified (possibly
  MITM) host key on that connection. (#668)

### Fixed
- **Liveness privilege probe now routes through `internal/ssh`.** Hardened
  hosts that only offer keyboard-interactive (PAM) auth after a successful
  key/password handshake no longer show as falsely "degraded": the probe now
  shares the same ordered auth methods (key + password + keyboard-interactive)
  as the rest of the SSH layer, instead of a forked dialer that lacked
  keyboard-interactive. (#664)
- **Settings to Scanning**: removed the fake group-maintenance list; the
  section now points to the real Groups page. (#665)
- **Settings to Users**: renamed the "Invite member" action to "Add member" to
  match what it does. (#666)
- Operator-guide truthfulness + accuracy pass (`docs/guides/`): verified every
  documented `openwatch` CLI subcommand, REST endpoint, file path, env var, and
  systemd unit against the binary, `api/openapi.yaml`, and `packaging/`. Fixed
  the verified defects: the RBAC permission counts (`ops_lead` 30 to 32,
  `security_admin` 51 to 56), the scan-trigger endpoint (was the nonexistent
  `/api/v1/scans/kensa/`; the real on-demand trigger is
  `POST /api/v1/hosts/{id}/scans`, also corrected in the Quickstart, which had
  wrongly said no run-scan action exists), stale example versions
  (`0.2.0-rc.11` to `rc.13`), the migration-count example (22 to 46), the
  Kensa rule count (508 to 538), and password-policy claims (12-char to the
  real 8/15). Added "Last Updated" headers to every guide. (#667)

### Docs
- README rewritten to reflect the Go rebuild (accurate built-in roles, FIPS
  build path, per-IP auth rate limiting, 4h to 48h adaptive scan cadence) with
  a current host-management screenshot; redundancy trimmed. (#669)

---

## [0.2.0-rc.13] Eyrie — 2026-06-22

The Reports surface is now a complete compliance-artifact platform, spanning
the executive summary, an auditor/GRC bulk path, two new GRC read-model
kinds, and recurring email delivery.

### Added
- **Framework Attestation report kind** (auditor/GRC): a point-in-time,
  signed attestation that freezes the latest completed scan per in-scope
  host. Faces: a fleet **OSCAL 1.0.6 assessment-results (SAR)** with
  evidence referenced by content hash, a per-(host, rule) **CSV** evidence
  extract, a bounded one-page **PDF cover**, and the canonical **JSON**.
  A frozen, signed compliance rollup (compliance %, pass/fail, top-failing)
  is shown in-app and on the PDF cover.
- **Exception Register report kind** (Compliance/GRC): a point-in-time
  read-model of compliance waivers — counts by state (active, pending,
  expiring-soon) plus the register rows — with CSV + PDF + JSON faces.
- **Remediation Activity report kind** (Operations): a read-model of
  remediation requests over a look-back window (last 7/30/90 days), with
  an outcome summary and CSV + PDF + JSON faces.
- **Scheduled reports + email delivery**: a daily/weekly/monthly schedule
  generates a report and emails its rendered PDF (MIME attachment) through
  an email notification channel. Managed from a live **Scheduled** tab
  (create / pause-resume / delete). Endpoints under
  `/api/v1/reports/schedules`.
- **Asynchronous report rendering** + a new `report.ready` event on the
  event bus — the first producer of the in-app **notification bell**.
- **Ed25519 report signing** with an in-browser **offline Verify**, a
  fleet **framework catalog** endpoint, and a kind selector + scope/period
  pickers on the Library tab.
- Report kinds are admitted by `report_snapshots.kind` (migrations
  0043–0045); `report_schedules` lands in migration 0046.

### Audit
- Report generation and report-schedule create/delete/enable-disable now
  emit audit events (`report.generated`, `report.schedule.*`).

### Security / hardening
- The scheduled-report dispatcher claims due schedules with
  `FOR UPDATE SKIP LOCKED`, so concurrent dispatchers never double-send.
- Report-email subjects are CRLF-sanitized (header-injection defense).

### Docs
- Consolidated the operator/admin/end-user guides under `docs/guides/`
  (install guide, the operator runbooks, supported-distros), merged the two
  upgrade docs into one, and left `docs/engineering/` for internal design docs.
- Archived completed/Python-era docs out of the repo (legacy install guide,
  stage plans, the Q1 plan, the backend-functionality inventories) and fixed
  the stale Go API examples in the Hosts/Remediation guide (real `/api/v1`
  routes on `:8443`).
- Removed residual OpenSCAP/SCAP references from the Go-native docs (the
  engine is Kensa, SSH-based native YAML rules).

### Fixed
- Favicon: `index.html` referenced a missing `/favicon.svg` (404 on every
  page). Now ships the OpenWatch logo favicon set (`.ico` + PNG sizes +
  apple-touch-icon + web manifest), embedded in the single binary.

---

## [0.2.0-rc.12] Eyrie — 2026-06-20

The fleet activity stream and audit trail are now readable end to end: every
event renders a plain-language title instead of a raw dotted code, enum, or
resource UUID. Host detail gains live Activity and Audit tabs, Settings shows
readable audit rows, and the filtered audit trail can be exported to CSV or
JSON (NIST 800-53 AU-7). The Host Management page got its scan link, Group, and
Filters working with a server-persisted view preference, and a pre-release
security pass hardened the new export and fixed a cursor data-loss bug.

### Added

- Activity & audit readability: the unified feed and the audit list now render
  a server-built, human-readable title + summary for all five legs. The three
  legs that previously leaked machine codes (compliance/transaction,
  intelligence, audit) are humanized — a rule's catalog title instead of its
  id, "Package updated" instead of `system.package.updated`, "alice@example.com
  created a host" instead of `host.created` over a UUID. Unmapped codes degrade
  structurally (dots/underscores to spaces) so a newly-added code can never
  surface as a raw dotted enum. (#616, #617)
- Host detail: a live **Activity** tab (host-scoped unified feed) and a
  readable **Audit log** tab, with audit `message`/`resource` filters so you can
  pull one host's lifecycle trail. (#618, #619)
- Settings: a readable **Audit log** view with plain-language rows. (#622)
- Audit export: `GET /api/v1/audit/events/export` streams the filtered audit
  trail as a downloadable CSV (default) or JSON attachment, capped at 10000
  rows, `audit:read`-gated (NIST 800-53 AU-7). (#623)

### Changed

- Host Management: the host card's scan link now opens the latest scan, Group
  and Filters work, and the list/grid view toggle is persisted **per user**
  server-side instead of per browser. (#611)
- Scan detail: the header shows the host's hostname (falling back to its IP)
  instead of a raw host UUID. (#613)
- Automated, schedule-driven events are now attributed to **"The system"**
  instead of the misleading "Someone", which implied a logged-in operator
  clicked a button. (#620)

### Security

- Hardened the audit CSV export against spreadsheet formula injection
  (CWE-1236): a cell beginning with `=`, `+`, `-`, `@`, tab, or CR is prefixed
  with a single quote so it renders as literal text. A truncated export (at the
  10000-row cap) now sets an `X-OpenWatch-Export-Truncated` header and logs a
  warning, so a capped export is never mistaken for the complete trail. (#625)
- Fixed a cursor-pagination data-loss bug in the activity feed and audit list:
  the cursor encoded `occurred_at` alone, so rows sharing a boundary timestamp
  could be silently skipped on the next page (likely on the 5-leg UNION with
  batch inserts). Both now use a compound keyset cursor `(occurred_at, id)` with
  a row-value predicate. Bounded the attacker-controlled User-Agent and
  submitted-username strings recorded in audit detail (256-rune cap +
  control-char strip), neutralizing log forging. (#626)

---

## [0.2.0-rc.11] Eyrie — 2026-06-19

The bundled Kensa scan engine moves to v0.5.2, which corrects a class of false
compliance FAILs on TAB-delimited rules. Single-operator remediation no longer
deadlocks (free-core fixes auto-approve), the Remediation tab updates live and
serializes concurrent fixes, an expired session now redirects cleanly to login,
and the GA-readiness pass hardened CI and the release workflow.

### Added

- Remediation: free-core single-rule remediation is now **auto-approved** on
  request, so an operator can apply a fix without a separate approver. This
  removes the self-review deadlock for single-operator workspaces (you could
  request a fix but never approve your own request). The request lifecycle and
  the approve/reject flow with separation of duties are retained for the
  licensed bulk/automated remediation track (which requests with approval
  required). See `docs/engineering/remediation_governance_adr.md` ("A-keep").

### Changed

- Updated the bundled Kensa scan engine and rule corpus to v0.5.2. v0.5.2 fixes
  a `config_value` matching bug so a `" "` delimiter matches any whitespace
  (including TAB), correcting a class of false FAILs on TAB-delimited rules such
  as RHEL `login.defs` — affected hosts may see their compliance score improve.
  It also adds rule-engine correctness gates (check-method parameter contracts,
  value-domain validation, a comparator + delimiter engine, and a schema/engine
  parity gate). The corpus stays at 539 rules and the engine's frozen API
  surface is unchanged, so OpenWatch's library integration is unaffected
  (kensa v0.5.2).
- Documented remediation/exception governance: a remediation-approval ADR and a
  role matrix covering who can request versus approve a fix or exception, plus a
  self-review rule, and an RBAC spec that drift-locks the role/permission map.
- CI release safety: the release workflow now fails closed on a `v*` tag push
  when no GPG signing key is configured, rather than publishing unsigned
  packages. Manual `workflow_dispatch` trial builds stay permissive (warn +
  publish unsigned).
- CI frontend gate: a failing frontend Vitest suite now hard-fails the build.
  Results are still ingested by specter first (so coverage is reported), then a
  dedicated enforcement step aborts on a real test failure.
- `make lint` now warns when the locally installed `golangci-lint` does not
  match the version pinned in CI, so local runs reproduce CI.

### Fixed

- Auth: an anonymous request to a protected endpoint (no credentials, or a
  session cookie that expired in the browser and is no longer sent) now returns
  **401 `auth.required`** instead of 403. The SPA redirects to login on a 401,
  so an expired session surfaces as a clean re-login prompt rather than a
  dead-end "failed to load." An authenticated caller whose role lacks the
  permission still gets 403 `authz.permission_denied`.
- Remediation now updates live. The Remediation tab and the compliance score
  refresh automatically when a queued fix or rollback finishes, over the SSE
  event stream (new `remediation.completed` topic), instead of requiring a
  manual page refresh.
- Applying several fixes on the same host at once no longer fails the extra
  ones. Concurrent remediations on a host now serialize: a fix whose host is
  busy backs off and requeues (with a short delay, via a new delayed-visibility
  column on the job queue) until the host is free, instead of colliding on the
  per-host SSH guard and being marked failed.
- Documentation version drift: operator guides referenced `0.2.0-rc.5` while
  `packaging/version.env` was `0.2.0-rc.10`; all guides now match.
- SPA static-delivery tests are self-contained (in-memory fixture) instead of
  depending on a magic staged asset filename, so `go test ./internal/server/`
  passes against a real `vite build`, the Makefile stub, or no staged tree.
- Repository hygiene: a stray 34 MB root build artifact was removed and the
  root-level `openwatch` binary path is now gitignored; placeholder credentials
  were stripped from the prototype login page.

---

## [0.2.0-rc.10] Eyrie — 2026-06-17

Per-host SSH credentials become directly manageable from the UI, the bundled
Kensa engine moves to v0.5.0, and a packaging fix stops upgrades from
overwriting an operator's TLS certificate.

### Added

- SSH credentials can now be edited in place. The Settings credentials page
  updates a credential directly instead of deleting and recreating it, so
  changing a name, username, or authentication method no longer forces you to
  re-enter the key or password. Leave a secret field blank to keep the stored
  one (#595).
- Per-host SSH credential management from the host detail page. A host can be
  given its own credential, have that credential edited in place, or be reverted
  to the workspace default, all from the host Edit dialog and the Connectivity
  card's Edit credentials link (#595).
- A Reconnect action on the host Connectivity card runs OS discovery
  immediately, ahead of the scan queue, so you can confirm a host is reachable
  and its SSH credential works right after changing it (#595).

### Changed

- The host Connectivity card now shows the credential the host actually uses
  (its own override or the workspace default) instead of a fixed label (#595).
- Updated the bundled Kensa scan engine and rule corpus to v0.5.0. v0.5.0 adds
  native sudo-with-password support for hosts where passwordless sudo is
  disallowed (a common CIS/STIG control); the change is backward-compatible and
  OpenWatch's existing scan behavior is unchanged. The corpus stays at 539
  rules. The `kensa-rules` package version tracks the engine, so it becomes
  0.5.0 in the next build (#594).

### Fixed

- A package upgrade no longer overwrites your TLS certificate. The demo
  certificate previously shipped inside the package at the production path, so
  `dnf update` or `apt upgrade` silently replaced an operator-installed
  certificate with a fresh self-signed demo on every upgrade. The demo
  certificate is now generated at install time only when the TLS files are
  absent (the same generate-if-absent model already used for the server's
  identity keys), so a certificate you put in place survives upgrades untouched
  (#596). This also covers the one-time upgrade from an earlier build that did
  ship the demo certificate (rc.9 and before): your certificate is preserved
  rather than removed during that transition too (#598).

---

## [0.2.0-rc.9] Eyrie — 2026-06-17

Two fixes that came out of production hardening: password login now works on
PAM-hardened hosts, and compliance scans run across the fleet in parallel
instead of one host at a time.

### Changed

- Compliance scans now run several hosts at once instead of strictly one at a
  time. A new `scan_concurrency` setting (under `[server]`, default 4) controls
  how many hosts scan in parallel; different hosts run concurrently while two
  scans of the same host never overlap. Clearing a large fleet that used to take
  many hours of serial scanning now finishes far sooner (#592).

### Fixed

- Password authentication now works against hardened hosts that accept passwords
  only through PAM keyboard-interactive (servers with `PasswordAuthentication no`
  but `UsePAM yes`). Such a host advertised only `keyboard-interactive`, so a
  password credential previously failed the handshake with "no supported methods
  remain" even though the password was correct (SSH-key auth was unaffected, and
  development hosts that offer plain password auth were never affected) (#591).

---

## [0.2.0-rc.8] Eyrie — 2026-06-17

Settings became a working control panel, OpenWatch started learning how to
reach each host over SSH, package upgrades became a single safe command, and a
pre-release security review closed a batch of perimeter and access-control
gaps.

### Added

- Settings > Users now lets you invite people, manage their accounts, and
  assign roles from the UI instead of showing a placeholder (#552, #553).
- Settings > Notifications can now send compliance alerts to Slack, a generic
  webhook, or email over SMTP, and each channel can be edited after creation
  (#554, #555).
- Settings > Security is live end to end: scoped API tokens you can create and
  revoke, an authentication policy (password strength and session timeouts),
  and single sign-on through an OIDC identity provider (#556, #557, #558).
- Settings > Audit and About now browse the audit log in-app and show the live
  license and build details instead of static text (#552).
- Each host's actions menu now has Edit and Delete entries so you can correct
  or remove a host without leaving the list (#560).
- OpenWatch now learns how to reach each host: it records which SSH
  authentication method and sudo style actually worked and reuses them on the
  next discovery, intelligence, and liveness pass, so it stops retrying
  combinations that already failed (#566, #575, #576).

### Changed

- Upgrading is now one command. `dnf update` (or `apt upgrade`) applies any
  pending database migrations automatically, takes a full database backup
  first, and on a failed migration leaves the service stopped with clear
  recovery steps instead of running against a half-migrated schema. The
  PostgreSQL engine upgrade itself stays an operator-supervised step (#569).
- Web fonts now ship inside the application instead of loading from a font CDN,
  so the interface renders completely in air-gapped deployments (#561).
- Updated the frontend build and CI tooling (Vite, Vitest, lucide-react, zod,
  and several GitHub Actions) to current major versions (#571, #572, #573).
- The web UI now loads faster with no extra infrastructure: the embedded
  single-page app is gzip-compressed and its content-hashed assets are served
  with long-lived immutable caching, so the browser fetches each one only once
  (#582).
- Seven database migrations land this cycle (0030 through 0036: notification
  channels, API tokens, authentication policy, SSO, per-host connection profile,
  and the SSH known-hosts store); the one-command upgrade applies them
  automatically with a pre-upgrade backup (#552 through #558, #566, #584).

### Fixed

- A fresh install now boots on the first try: the Kensa rule corpus and the
  server identity keys are provisioned during installation rather than failing
  at first startup (#564).
- The SMTP notification channel edit form now pre-fills its current settings
  instead of opening blank (#561).
- Removed leftover demo and sample data that could appear on the dashboard,
  the activity feed, and the host lists (#562).
- An expired or invalid session now redirects you to the login page instead of
  leaving you on a page that silently fails to load (#583).

### Security

A pre-release security review (six parallel audit dimensions, every
high-severity finding re-verified by hand) closed eight findings (#584):

- State-changing requests made with a session cookie are now CSRF-protected
  with a double-submit token; a request without a matching token is rejected.
- The login and MFA-verify endpoints are now rate-limited per client address
  to slow online password and one-time-code guessing.
- Every response now carries security headers: HSTS, a content-security policy
  that forbids framing, no-sniff, and a strict referrer policy.
- SSH host keys learned on first connection are now stored in the database, so
  a restart no longer re-trusts every host and a changed host key is detected
  across restarts.
- New passwords are now screened against a built-in list of common and breached
  passwords, with an option to point at a full breach corpus; the check now
  runs in production instead of being silently skipped.
- Reading the audit-event API now requires the audit-read permission instead of
  being open to any caller.
- Creating an API token or assigning a role can no longer grant more access
  than the caller already holds.

---

## [0.2.0-rc.7] Eyrie — 2026-06-14

The navigable-application candidate: rc.6 made compliance scanning work, and
rc.7 makes it a product you move through. The full app shell came alive (a
public Radar homepage, a live fleet dashboard, an activity feed, a scans
overview, and Groups + Reports), historical scan evidence is now durable and
exportable as OSCAL, and the ~539-rule Kensa corpus is browsable. Still a
pre-release, pending the GA fleet-verification gate.

### Added

- **Durable per-scan compliance evidence + `/scans` surface** (#535). The
  write-on-change model (`host_rule_state` + `transactions`) overwrote a
  superseded scan's per-rule evidence, leaving historical proof unrecoverable.
  A content-addressed `scan_results` store (migration 0029, `internal/scanresult/`)
  now retains every rule's outcome and evidence per scan, deduped by content
  hash. New `scan:read`-gated API under `/api/v1/scans`: scan history by host,
  scan detail (per-rule verdicts with catalog title/category/description),
  per-rule evidence, and per-rule + whole-scan OSCAL 1.0.6 export reconstructed
  on demand via Kensa. A scan-detail page is reached from `/scans` (host
  history -> scan detail -> per-rule Formatted / Evidence / OSCAL drill-down,
  Evidence and OSCAL shown as raw JSON). Specs `system-scan-results-store`,
  `api-scans`, `frontend-scan-detail`.
- **Kensa rule-library browser** as the Rules tab on `/scans` (#536). The full
  ~539-rule corpus is browsable with search and severity / category / framework
  filters, framework-reference tags, and a remediation column (manual / atomic /
  staged, with reboot and service-restart hints), plus CSV export. Built on the
  Kensa v0.4.3 read model (`pkg/kensa.LoadRuleSummaries`). `scan:read`-gated.
  Specs `api-rules`, `frontend-rules-library`.
- **Per-rule evidence/OSCAL drill-down on the host Compliance tab** (#537). Each
  rule row on the host-detail Compliance tab expands into the same Formatted /
  Evidence / OSCAL panel as the scan-detail page. The host-compliance API stays
  evidence-free: the drill-down reaches the `scan:read`-gated `/scans` evidence
  endpoints for the host's latest scan, and the control shows only to callers
  holding `scan:read`. Spec `frontend-host-compliance-tab` v1.3.0.
- **Public Radar homepage + enhanced login** (#528). An unauthenticated landing
  page at `/`; the authenticated dashboard moved to `/dashboard`.
- **Fleet dashboard MVP** at `/dashboard`, wired to the live fleet endpoints
  (#529).
- **Activity feed MVP** at `/activity` (#530).
- **Scans overview MVP** at `/scans` — the home for scan history and the rule
  library (#531).
- **Groups + Reports MVP** (#533): Groups organizes the fleet by site and OS
  category; Reports is a reports library. Specs `api-groups`, `frontend-groups`,
  `api-reports`, `frontend-reports`.

### Changed

- Remediation is documented as shipping **beta in GA**; remaining scan-platform
  work split into its own tracking file and reconciled for the release
  (#525, #526).

### Fixed

- The topbar breadcrumb is now set on every navigation page instead of going
  stale after the first route (#534).
- Unrouted sidebar links are disabled rather than navigating to a 404 (#527).
- The frontend's synthesized role->permission baseline omitted `scan:read`,
  which would have redirected every user away from the new scan-detail route
  even though the backend grants `scan:read` to all built-in roles (#535).

---

## [0.2.0-rc.6] Eyrie — 2026-06-13

The compliance-scanning candidate: this RC turns OpenWatch from a scanner
shell into a working compliance platform. Kensa now scans real hosts on an
adaptive schedule, results are viewable through any framework lens, posture
trends accumulate, and operators can govern failing rules with approved
exceptions. Still a pre-release, pending the GA fleet-verification gate.

### Added

- **End-to-end compliance scanning.** On-demand scans (`POST
  /hosts/{id}/scans`, the Run scan button) run the full ~539-rule Kensa
  corpus against a host over an in-memory SSH transport (no private key on
  disk), persist results write-on-change to `transactions` + `host_rule_state`,
  and refresh the UI over the live event stream. (#515)
- **Lens model on the Compliance tab.** One scan, viewed through any
  framework: `GET /hosts/{id}/compliance` (+ `/frameworks`) projects the
  per-rule results through CIS / STIG / NIST / PCI / SRG at query time. The
  lens bar is OS-aware — a RHEL 8 host no longer offers RHEL 9/10 lenses,
  while OS-neutral frameworks always appear. (#515, #518)
- **Adaptive compliance scheduler.** Hosts auto-scan on a five-band,
  state-driven cadence (critical 4h … compliant 48h), operator-editable per
  band under Settings → Scanning & monitoring. Scan-queue depth, fleet
  per-state counts, and a 24-hour schedule strip are exposed. (#515)
- **Scan variables.** Operator overrides for the corpus's templated rule
  variables (Settings → Compliance policies), reloaded into the rule corpus
  on the next scan. (#517)
- **Posture trends.** A daily posture-snapshot rollup powers the 30-day
  per-host trend card and the fleet average-compliance delta on the hosts
  list. (#518)
- **Live host-detail hero tiles.** Auto-scan (next scan + cadence),
  Watchlist (active alerts + waived-rule count), and Connectivity now render
  real data instead of placeholders. (#518)
- **Compliance exception governance.** Operator-approved rule waivers with a
  request → approve/reject → revoke/expire lifecycle, separation of duties,
  and an audit trail. An exception never changes a rule's raw verdict (a
  waived failure stays failing); it is a read-time overlay marking accepted
  risk. Surfaced on the host detail (Waived/Pending badges + a request modal)
  and a fleet approver queue under Settings → Compliance policies.
  (#521, #522, #523)
- **New migrations** 0023–0026: `scan_runs` (scan logbook), compliance-state
  five-band CHECK + backfill, `posture_snapshots`, `compliance_exceptions`.

### Changed

- All tracked documentation reconciled with the Go codebase, including the
  Scanning & Compliance operator guide and the bannered legacy guides; the
  backlog rewritten for the Go rebuild. (#505, #507, #508, #520, #481)

### Fixed

- SSE event streams were silently killed at the HTTP server's 60-second
  write timeout; per-stream write deadlines are now cleared. (#515)
- The alerts lifecycle service was never wired into the serve path, so every
  `/api/v1/alerts*` endpoint returned 503 in production while passing tests.
  Wired, plus a generic guard that every server builder is registered in
  `main.go`. (#518, #519)
- Correlation-ID generation now reads the clock under the lock, restoring
  per-ID uniqueness under concurrency. (#503)
- Cleared the live CodeQL warnings; prototype mockups excluded from scans. (#514)

### CI / tooling

- `specter check --test` annotation-hygiene gate and a pre-push annotation
  hook; restored four dropped connectivity API specs; perf latency budgets
  made non-gating to stop CI flakes. (#512, #509, #510, #506)

### Dependencies

- npm production and development group bumps. (#489, #513)

---

## [0.2.0-rc.5] Eyrie — 2026-06-08

Package-refresh candidate: re-cuts the signed RPM/DEB from `main` so the
published artifacts include the version-endpoint and auth-redirect fixes
that landed after rc.4. Still a pre-release — not GA.

### Added

- `GET /api/v1/version` (anonymous) reporting the OpenWatch, Kensa, and Go
  versions plus commit/build-time, all sourced from build metadata rather
  than constants. Settings → About now renders these live instead of
  hardcoded strings (#500).

### Fixed

- Frontend: an expired/invalid session now always redirects to `/login`
  instead of leaving the user on a raw error envelope. A global, code-aware
  QueryCache/MutationCache handler covers every query and mutation;
  authorization (permission) errors are excluded so they never log a user
  out (#501).
- Release: `v*` tags with a pre-release suffix now publish as GitHub
  pre-releases (a bare `vX.Y.Z` is GA) (#499).

---

## [0.2.0-rc.4] Eyrie — 2026-06-08

The release-readiness candidate: OpenWatch Go is now a single, installable
product. `dnf install ./openwatch-*.rpm` / `apt install ./openwatch_*.deb`
lays down one binary that serves both the API and the built UI, with a
tag-driven, signed release pipeline behind it. This RC also folds in the
host-detail / OS-intelligence / discovery suite that accumulated since the
early rc.3 cut.

### Added

**Distribution & supply chain**

- Native multi-arch packages — RPM (CentOS Stream 9) and DEB (Ubuntu 24.04),
  each built for amd64 and arm64 via `CGO_ENABLED=0` cross-compile (#490).
- The React SPA is embedded into the binary via `go:embed` and served by the
  Go server (static assets + `index.html` fallback; `/api/` paths still 404),
  so one artifact is the whole product — air-gap clean, no separate web tier
  (#486).
- `release.yml` — on a `v*` tag, builds all four packages, generates a
  CycloneDX 1.5 SBOM per artifact (syft), writes `SHA256SUMS`, and publishes
  a GitHub Release (#491).
- Release signing — each RPM (`rpmsign`) and DEB (`dpkg-sig`) is GPG-signed,
  and `SHA256SUMS` gets both a detached GPG signature and a cosign sigstore
  signature; the Hanalyx public key ships as `KEYS`. Every signing layer is
  gated on its key secret and skipped gracefully when absent (#493, #494).
- `package-smoke.yml` — installs the built RPM on Rocky/Alma/Fedora/Oracle and
  the DEB on Ubuntu/Debian, then smoke-tests the binary (#492).
- `docs/runbooks/RELEASING.md` — the gated release process (docs freeze →
  RC → verification gate → GA), including signing-key setup (#492).
- Go module supply-chain spec + depguard allowlist + Dependabot for the
  module set (#416).

**Product features (since rc.3)**

- OS discovery — scheduler, first-contact policy, and fleet sweep that learns
  each host's distro over SSH and persists it to `hosts.os_family` (#467, #471).
- Server intelligence — packages/services/users/network/system collected over
  SSH and surfaced as a host-detail snapshot grid, with a settings page to
  tune collection (#455, #472).
- Host liveness — adaptive, per-state probe intervals and a fleet-health
  surface (#421, #434, #435).
- Alerts — router, persistence, and lifecycle (#424, #444, #445, #420).
- Fleet observability API — read-only fleet endpoints and `hosts/{id}`
  enrichment (liveness + compliance summary) (#427, #428).
- React 19 + MUI v7 + TanStack frontend for auth, hosts, host-detail,
  settings, and an activity feed, with five approved frontend specs at 100%
  AC coverage (#433, #436, #437, #468).
- 16 database migrations (`0007`–`0022`): credentials, compliance schedule,
  transaction log, host liveness, system config, multilayer monitoring,
  system info, intelligence, alerts. **`openwatch migrate` is required on
  upgrade.**

### Changed

- The admin CLI is retired: `openwatch` is the single binary and command
  (`serve`/`worker`/`migrate`/`create-admin`/`check-config`); lifecycle is
  managed by systemd, not a separate `owadm` (#487).
- Repository restructure — the Python backend/frontend were archived out of
  the repo and the Go tree was promoted from `app/` to the repo root (#482).
- Tooling — Prettier + a flat ESLint config for the frontend, with the lint
  pre-commit hook re-enabled; `.env` templates rewritten for the Go server
  (#483, #484, #485).
- Dependabot retargeted for the post-promotion layout (`gomod` at `/`, dead
  `backend`/`docker` ecosystems dropped) (#495).
- SSH connectivity — credential-password sudo fallback extended across
  liveness probes and discovery queries; auth offers both key and password
  methods when configured for both (#460, #469, #470).
- Frontend surfaces the backend's `human_message` instead of a generic HTTP
  error, and retries 401s transparently against the HttpOnly refresh cookie
  (#456, #466).

### Fixed

- Discovery persists the distro ID into `hosts.os_family` rather than the
  family rollup (#471, #022 migration).
- Firewall-rule probe records `0` (not `-1`) when the engine is present but
  inactive (#459).
- Activity feed no longer crashes on a `host_id`-filtered union and wires the
  service in `main` (#477, #478).

---

## [0.2.0-rc.3] Eyrie — 2026-05-25

API hygiene pass driven by manual testing of the rc.2 surface. Three
related cleanups that all surfaced from the same underlying issue —
inconsistent naming between API paths/fields and the role/permission
model behind them.

### Added

- `GET /api/v1/openapi.yaml` — serves the embedded OpenAPI 3 spec.
- `GET /docs/` — Swagger UI mounted from the binary (assets embedded
  via go:embed; no CDN dependency, air-gap clean).
- New spec `api-openapi-docs` with 4 ACs pinning the spec/UI endpoints,
  same-origin asset constraint, and byte-identical embed.
- Build-time copy: `make build` now syncs `api/openapi.yaml` into
  `internal/server/openapi_embed.yaml` (gitignored) before compiling.
- Migration 0010 — drops the `users.is_admin` column.

### Changed

**Path rename: resource CRUD moves off `/admin/`.**

The design doc (`docs/api_design_principles.md` §12.2) reserves the
`/admin/` namespace for system operations (`POST /admin/operations:*`),
not resource CRUD. Slice A inadvertently put resource endpoints under
`/admin/` which read as a role gate but isn't — `host:read` for example
is held by `viewer`. The rename collapses the disconnect:

| Before | After |
|---|---|
| `/api/v1/admin/users` | `/api/v1/users` |
| `/api/v1/admin/users/{id}` | `/api/v1/users/{id}` |
| `/api/v1/admin/users/{id}/roles:{assign,unassign}` | `/api/v1/users/{id}/roles:{assign,unassign}` |
| `/api/v1/admin/roles` | `/api/v1/roles` |
| `/api/v1/admin/roles:create` | `/api/v1/roles:create` |
| `/api/v1/admin/credentials*` | `/api/v1/credentials*` |
| `/api/v1/admin/hosts*` | `/api/v1/hosts*` |

Genuine operations stay where they belong:
- `/api/v1/admin/license:verify` (unchanged)
- `/api/v1/admin/policies:reload` (unchanged)

`operationId`s renamed in parallel (`postAdminUsers` → `postUsers`,
etc.) so the Swagger UI labels match.

**`users.is_admin` removed entirely.**

The column only ever drove password-policy selection but the API
exposed it as if it were a permission marker. Manual testing showed
the resulting drift case: unassigning the admin role left
`users.is_admin = true` because the column and `user_roles` had
independent lifecycles. The inverse case (assign admin role to a user
created with `is_admin: false`) was also possible and represented a
security gap (admin-tier user, default-tier password policy).

Replacement: password policy now derives from one source. At creation,
`CreateUser` takes an explicit `AdminPolicy` flag (the `create-admin`
CLI sets it true; the HTTP `POST /users` does not). On password
change, `UpdatePassword` looks up the user's primary role: admin role
→ AdminPolicy (15-char minimum), other → DefaultPolicy. No second
column to drift.

Wire response changes:
- `/auth/me` no longer carries `is_admin`. Admin status is implicit
  in `role == "admin"`.
- `/users/{id}` and `/users` response items no longer carry `is_admin`.
- `POST /users` request body no longer accepts `is_admin`.

### Fixed

- Test fixture `freshAPIServer` no longer sets the obsolete `is_admin`
  column when seeding role users.
- `seedAuthUser` test helper renamed parameter `isAdmin` → `adminPolicy`
  to reflect its actual effect.

### Lessons captured

Two API-design issues caught in two sessions of manual testing (the
`/admin/*` prefix overload and the `is_admin` drift) — both
semantic-conflation bugs that 100% per-spec coverage missed because
each individual behavior was tested in isolation. The Slice B spec
template will add a meta-AC pattern requiring that any wire field
naming a permission/role/state declare whether it's the SSOT or
documents how it stays in sync with the underlying data.

---

## [0.2.0-rc.2] Eyrie — 2026-05-25

Boot-wiring fixes for the admin surface. `rc.1` shipped a binary whose
JWT signing key and credential DEK were never loaded at boot, so every
`/auth/login` returned 500 and every credential / MFA action failed. The
tests passed because fixtures installed ephemeral keys directly; the
binary's `main.go` did not.

### Added

- `[identity]` config section with `jwt_private_key` and
  `credential_key_file` paths. Both are required for `openwatch serve` —
  no silent fallback to ephemeral keys.
- Env-var overrides: `OPENWATCH_IDENTITY_JWT_PRIVATE_KEY`,
  `OPENWATCH_IDENTITY_CREDENTIAL_KEY_FILE`.
- `openwatch create-admin --username --email --password` subcommand.
  Closes the chicken-and-egg in the bootstrap flow (`/admin/users`
  requires an existing admin).
- `release-admin-signoff` AC-14 + `TestRuntimeBoot_LoginEndToEnd` in
  `packaging/tests/runtime_boot_test.go`. Spawns the actual
  `dist/openwatch` binary against a real Postgres and exercises
  migrate → create-admin → serve → login → POST host. Catches the
  "tests pass but binary broken" class of bug that produced `rc.1`.

### Fixed

- `cmd/openwatch/main.go` now calls `identity.LoadJWTKey()` and
  `secretkey.LoadFromFile()` at boot. Missing or unreadable keys fail
  the server with an explicit error instead of allowing the binary to
  serve traffic that 500s on the first login.

### Security note

The `rc.1` regression was not a security issue (`/auth/login` 500-ed
rather than admitted attackers) but it was a release-blocking
correctness gap that 100% spec coverage missed. The new AC-14 binds
sign-off to the artifact, not just the unit tests.

---

## [0.2.0-rc.1] Eyrie — 2026-05-25 (yanked)

Tagged locally, never pushed. Superseded by 0.2.0-rc.2 — `cmd/openwatch/main.go`
did not load the JWT signing key or credential DEK at boot, so login
returned 500 against the actual binary. See 0.2.0-rc.2 entry for the
fix. Original deliverable details preserved below for traceability.

Release-candidate sign-off for real identity, user CRUD, host inventory,
credential store, the SSH dial layer, and the four admin HTTP surfaces
that knit them together.

### Added

**Specs (9 new, all 100% strict coverage):**
- `system-auth-identity` — Argon2id password hashing, NIST SP 800-63B
  policy, sessions, JWT (RS256), refresh-token rotation with reuse
  detection, TOTP MFA, production identity binder.
- `system-user-management` — users + user_roles + custom roles tables
  with the highest-privilege-wins resolver and `identity.Lookups`
  adapter.
- `system-credential-store` — credentials table (system + host scope),
  AES-256-GCM via the shared `internal/secretkey` DEK, host→system
  resolver, partial unique index for the "one system default" rule.
- `system-host-inventory` — hosts table with INET addresses, TEXT[]
  tags + GIN index, soft delete via `deleted_at`.
- `system-ssh-connectivity` — SSH dial (`golang.org/x/crypto/ssh`),
  known-hosts store, strict / trust-on-first-use modes, NIST SP 800-57
  key strength validation.
- `api-auth` — `/auth/login`, `/auth/me`, `/auth/logout`,
  `/auth/refresh`, `/auth/mfa:enroll`, `/auth/mfa:validate`,
  `/auth/password:change`.
- `api-users` — `/admin/users` (GET/POST), `/admin/users/{id}`
  (GET/DELETE), `/admin/users/{id}/roles:{assign,unassign}`,
  `/admin/roles:create`.
- `api-credentials` — `/admin/credentials` (GET/POST),
  `/admin/credentials/{id}` (GET/DELETE),
  `/admin/hosts/{host_id}/credentials:resolve`. Metadata-only at the
  wire; plaintext + ciphertext never cross the HTTP layer.
- `api-hosts` — `/admin/hosts` (GET/POST), `/admin/hosts/{id}`
  (GET/PATCH/DELETE) with environment + tag filters.

**RBAC additions:**
- New permissions: `credential:read`, `credential:write`,
  `credential:delete`.
- Assigned: `credential:read` to `ops_lead`; `credential:*` to
  `security_admin` and `admin` (via `*` wildcard).

**Audit additions:**
- New event codes: `credential.created`, `credential.deleted`,
  `host.created`, `host.updated`, `host.deleted` (host codes already
  existed; credential codes added in this release).

**End-to-end:**
- `TestAdminE2E_RealIdentity` exercises the full admin flow through a
  real session cookie: login → host create → system credential →
  host-scope credential → resolve (host wins) → soft-delete cred →
  resolve (system fallback) → soft-delete host → confirm audit
  pipeline emitted the expected rows.

**Sign-off:**
- `release-0.2.0-signoff` spec with 13 ACs; the 9 release specs are
  registered in `specter.yaml`.

### Security

- **Removed `X-Stub-Role` / `X-Stub-User-Id` header-based identity
  bypass** (previously inherited from the walking-skeleton phase).
  No exported symbol in `internal/auth`, no middleware mount in
  `server.go`. Identity is now bound exclusively by the production
  binder via session cookie or Bearer JWT. The previous binder was an
  authentication-bypass vector against unauthenticated callers; its
  removal is enforced by source-inspection tests
  (`system-rbac` AC-12 and `release-0.2.0-signoff` AC-13).
- Test fixture seeds one user per built-in role and mints a real
  session via `identity.IssueSession`; `asRole(t, ..., role, ...)`
  attaches the corresponding session cookie. No header-based identity
  short-circuit exists in the test path either.

### Fixed

- `TestResolve_HostScopeWins` in `internal/credential/credential_test.go`
  now seeds a host row before creating a host-scope credential — the
  deferred FK from migration 0008 had previously made the test order
  fragile.
- `release-package-build` AC-12 test now reads the Go-rebuild's own
  `packaging/version.env` first, falling back to the repo-root
  `VERSION`.

### Deferred (not in 0.2.0)

- `POST /hosts/{id}:connectivity-check` — moves to the next release
  with the scan executor.
- OIDC/SAML initiate endpoint that returns 402 — license feature
  `sso_saml` is in the registry; the endpoint lands with the SSO
  implementation work.
- PUT-style full updates on hosts and users — only PATCH ships in
  0.2.0.
- Bulk host import and cursor-paginated list — next release.

---

## [0.1.0-stage-0] — pre-2026-05-25

Walking-skeleton phase (pre-0.2.0). See
`docs/stage_0_walking_skeleton.md` and the `release-stage-0-signoff`
spec for the Definition of Done.

---

# Kensa: Install
URL: https://www.hanalyx.com/docs/kensa/install

## What you'll have when you're done

`kensa` and `kensa-rules` installed from signed packages, the verification
keys imported, and `kensa --version` printing `kensa 0.5.2`. From there,
[02-quickstart](/docs/kensa/quickstart) is the next step.

Target hosts (the machines you'll scan) need no kensa installation:
only OpenSSH and, for non-root remediation, sudo.

## Step 1: import the verification keys

Every release artifact is signed. The Hanalyx GNU Privacy Guard (GPG)
public key verifies the `.rpm` and `.deb` files; the Kensa cosign public
key verifies the
checksums file that anchors the whole set. Both keys live at
[`KEYS`](https://github.com/Hanalyx/kensa/blob/main/KEYS) in the repo
root.

```bash
# RHEL/Fedora/Rocky/Alma + Debian/Ubuntu: import the Hanalyx GPG key
sudo rpm --import https://raw.githubusercontent.com/Hanalyx/kensa/main/KEYS
# or for apt:
curl -fsSL https://raw.githubusercontent.com/Hanalyx/kensa/main/KEYS \
  | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/hanalyx.gpg

# Save the cosign block from KEYS as cosign.pub (everything between
# -----BEGIN PUBLIC KEY----- and -----END PUBLIC KEY-----).
```

With the Hanalyx GPG key imported, `dnf` and `apt` reject any unsigned
or wrong-key kensa package automatically, so you get the trust check for
free.

## Step 2: install

Pick one path.

### Connected RHEL/Fedora/Rocky/Alma

```bash
sudo dnf install kensa kensa-rules
```

`kensa` Recommends `kensa-rules`, so `dnf install kensa` alone pulls
both by default. Use `--setopt=install_weak_deps=False` to opt out
(you'll need `--rules-dir <path>` on every command).

### Connected Debian/Ubuntu

```bash
sudo apt install kensa kensa-rules
```

### Air-gapped

On a connected host, download the bundled tarball from
[the v0.5.2 release](https://github.com/Hanalyx/kensa/releases/tag/v0.5.2):

```
kensa_0.5.2_linux_<arch>_with-rules.tar.gz   # binaries + rules + LICENSE + KEYS
kensa_0.5.2_checksums.sha256                 # sha256 of every artifact
kensa_0.5.2_checksums.sha256.sig             # cosign signature of the checksums
```

Verify before transferring:

```bash
sha256sum -c kensa_0.5.2_checksums.sha256   # one OK line per artifact you downloaded
cosign verify-blob --key cosign.pub \
  --signature kensa_0.5.2_checksums.sha256.sig \
  kensa_0.5.2_checksums.sha256
```

Then copy the tarball to the air-gapped host and:

```bash
tar xzf kensa_0.5.2_linux_amd64_with-rules.tar.gz
sudo install -m 0755 kensa kensa-validate kensa-keygen /usr/local/bin/
sudo install -m 0755 kensa-systemd-helper /usr/libexec/
sudo mkdir -p /usr/share/kensa && sudo cp -r rules /usr/share/kensa/
```

Both the connected `.rpm`/`.deb` paths and this air-gap path install the
rules to `/usr/share/kensa/rules/`, which is where `kensa check` falls
back when `--rules-dir` is unset (see
[`rule-default-path-resolution`](../../specs/rule/default-path-resolution.spec.yaml)
spec).

## Step 3: generate a signing key

`kensa-keygen` writes a keypair to `~/.config/kensa/keys/`. The
private half is mode `0600`; the public half is what you distribute
to anyone running `kensa verify` against your evidence envelopes.

For a stable operator identity across runs, point `KENSA_SIGNING_KEY`
at the `.priv` file before running `kensa remediate`. Without that
env var, kensa generates an ephemeral key per process (fine for
trying things out, but your evidence envelopes won't share a stable
signer identity).

An evidence envelope is the signed record Kensa produces for each
committed transaction, capturing what changed and proving it wasn't
altered after the fact.

## Service handlers (optional)

You only need this step if your rules use `service_enabled`,
`service_disabled`, or `service_masked`.

The rpm/deb already ship `/etc/sudoers.d/kensa-systemd-helper` (the
`%kensa ALL=(root) NOPASSWD: /usr/libexec/kensa-systemd-helper` rule)
and create the `kensa` group **empty** at install time, so the grant is
inert until you opt a user in. The only remaining step is to add that
user to the group:

```bash
sudo usermod -aG kensa "$USER"   # log out / back in for it to take effect
```

(Installing from the air-gap tarball instead of the package? Create the
group and drop the sudoers file yourself: `sudo groupadd --system
kensa`, then write the one-line rule above to
`/etc/sudoers.d/kensa-systemd-helper` mode `0440` root-owned and run
`sudo visudo -c` to syntax-check.)

The canonical definition is
[`agent-systemd-helper`](../../specs/agent/systemd-helper.spec.yaml)
AC-09 / C-06 and
[`packaging-sudoers-helper`](../../specs/packaging/sudoers-helper.spec.yaml).
Without the helper, only the service handlers fail; everything else
(file permissions, sysctl, mount options, SELinux booleans, audit,
cron, packages, PAM) works as-is.

## Build from source

For contributors and for customising the build. Requires Go 1.26.4+ (the
version pinned in `go.mod`), GNU make, git:

```bash
git clone https://github.com/Hanalyx/kensa.git && cd kensa && make build
```

Five static binaries land in `bin/`: `kensa`, `kensa-fuzz` (the
destructive atomicity harness, not shipped in packages),
`kensa-validate`, `kensa-keygen`, `kensa-systemd-helper`. Install per
the air-gap path above (the `bin/` directory replaces the extracted
tarball). The rules corpus lives at `rules/` in the repo; copy it to
`/usr/share/kensa/rules/` or pass `--rules-dir ./rules` on every command.

## Verify

```bash
kensa --version
```

You're done when this prints `kensa 0.5.2 (kensa)`. If it doesn't,
the binary isn't on your `$PATH`; go back to **Step 2**.

## Next

[02-quickstart](/docs/kensa/quickstart) runs your first scan and your first
remediation.

---

# Kensa: Quickstart
URL: https://www.hanalyx.com/docs/kensa/quickstart

_Applies to: Kensa v0.6.0 — last updated 2026-06-22._

This chapter takes one host from "never scanned" to "remediated and rolled
back" in four commands: **detect** what the host can do, **check** its
compliance read-only, **remediate** the failures, then **roll back** if you
want the host returned to where it started. Every command here is real and
runs against the shipped binary.

You'll need `kensa` and `kensa-rules` installed (see
[01-install](/docs/kensa/install)) and SSH access to the target. The examples use
`-r rules/` to point at a local rules tree. If you installed `kensa-rules`,
drop `-r` and Kensa finds `/usr/share/kensa/rules` automatically.

## Before you start

The target host needs nothing installed, only OpenSSH reachable and, for
privileged checks, `sudo`. Kensa connects over your system SSH (honoring
your keys, agents, and `known_hosts`), so confirm a plain `ssh user@host`
works first.

Two flags recur on every command:

- `--sudo` wraps remote commands in `sudo`. Without it, Kensa runs as the
  login user and most hardening rules can't read or change privileged state.
- `--sudo-password` (or the `KENSA_SUDO_PASSWORD` env var) supplies a sudo
  password on hosts that don't allow `NOPASSWD`. Omit the value to be
  prompted on the TTY. With passwordless sudo you don't need it at all.

## Step 1: detect

Probe the host and print its capability set. This is read-only and mutates
nothing; it's the safe way to confirm Kensa can reach the host and to see
which subsystems (systemd, SELinux, apt, auditd, …) it found.

```bash
kensa detect -H 192.168.1.211 -u owadmin --sudo
```

Add `--sudo-password` if the host needs one:

```bash
kensa detect -H 192.168.1.211 -u owadmin --sudo --sudo-password
```

If detect can't connect, fix that before going further. Every other
command uses the same transport.

## Step 2: check (read-only)

Run the compliance checks. `check` never changes the host; it reports
`PASS` / `FAIL` / `ERROR` per rule, plus a `SKIP` for any rule that doesn't
apply to this host's platform, and ends with a tally. Rows stream as each
rule completes.

```bash
kensa check -H 192.168.1.211 -u owadmin --sudo -r rules/
```

Narrow the run with filters, by severity, framework, or category:

```bash
# Only critical and high-severity rules
kensa check -H 192.168.1.211 -u owadmin --sudo -r rules/ -s critical -s high

# Only rules mapping a CIS RHEL 9 control
kensa check -H 192.168.1.211 -u owadmin --sudo -r rules/ -f cis-rhel9
```

`check` is read-only and does **not** write to the transaction log by
default. Pass `--store` if you want the scan persisted as a session you can
query later with `kensa history`.

## Step 3: remediate (apply)

`remediate` applies the failing rules. Each rule runs as a four-phase atomic
transaction (capture, apply, validate, then commit or roll back), so a rule
whose change fails validation is reversed automatically before Kensa moves
to the next rule. A transaction is Kensa's unit of atomic change: one rule's
capture-apply-validate-commit-or-rollback cycle on one host. (The mental
model is [03-concepts](/docs/kensa/concepts).)

```bash
kensa remediate -H 192.168.1.211 -u owadmin --sudo -r rules/
```

The output adds a `FIXED` status to the check statuses. As with `check`, you
can scope the run, remediating only what you mean to change:

```bash
# Apply only critical PCI-tagged rules
kensa remediate -H 192.168.1.211 -u owadmin --sudo -r rules/ -s critical -t pci
```

Every committed transaction is written to the transaction log with a signed
evidence envelope, the tamper-evident record of what changed and the proof
it wasn't altered afterward. To preview a single rule's transaction without
touching the host, use `kensa plan` instead.

## Step 4: roll back

Remediation that *fails validation* rolls back on its own. Step 4 is the
**deliberate** rollback: returning a host to the state captured before a
remediation you've decided to undo.

First find the session you want to reverse:

```bash
kensa rollback --list
```

Inspect it (read-only) before committing to the reversal:

```bash
kensa rollback --info <SESSION_ID> --detail
```

Then execute the rollback for every committed transaction in that session.
This phase touches the host, so it needs the same target flags as
remediate:

```bash
kensa rollback --start <SESSION_ID> -H 192.168.1.211 -u owadmin --sudo
```

Rollback restores each transaction's captured pre-state. A mechanism Kensa
can't reverse (a `transactional: false` rule) was never captured and is
reported as skipped rather than silently "restored." See the reversal
table in [10-mechanisms](/docs/kensa/mechanisms).

## Where to go next

- [03-concepts](/docs/kensa/concepts): the four-phase transaction and why
  atomicity is the product.
- [10-mechanisms](/docs/kensa/mechanisms): every mechanism, where it runs, and
  what reversal you get.
- `kensa history`: query past transactions; `kensa diff` compares two
  stored sessions for drift.

---

# Kensa: Concepts
URL: https://www.hanalyx.com/docs/kensa/concepts

_Applies to: Kensa v0.6.0 — last updated 2026-06-22._

Kensa is a compliance engine, but its core is not the rules. It's the
*transaction*: the four-phase Kensa operation (capture, apply, validate,
commit or roll back). Every change Kensa makes to a host runs as a
transaction, and that atomicity is the product. The compliance rules are
the first application of it. This chapter is the mental model behind the
commands in [02-quickstart](/docs/kensa/quickstart); the per-mechanism specifics
live in [10-mechanisms](/docs/kensa/mechanisms).

## The four-phase transaction

When `remediate` applies a rule, the engine runs four phases in order:

1. **Capture.** Before touching anything, the engine records the host's
   exact prior state for everything the change is about to modify (file
   bytes and attributes, a service's enablement, a sysctl value, whatever
   the mechanism touches). This snapshot is written to durable storage
   *before* any mutation, so it survives a crash.
2. **Apply.** The change is made.
3. **Validate.** The engine re-checks the host to confirm the change landed
   and that no dependent validator broke.
4. **Commit or Rollback.** If validation passes, the transaction commits and
   a signed evidence record is written. If anything fails, the engine
   reverses every applied step from the captured pre-state, returning the
   host to exactly where it began.

There is no third outcome. A rule either lands completely or leaves the host
in the state it was in before the rule began: no "partially applied," no
"step 3 failed and steps 1–2 are stranded." That guarantee is the contract
in `docs/TRANSACTION_CONTRACT_V1.md`, and it's what distinguishes Kensa from
a remediation script.

## The per-rule transaction boundary

The transaction boundary is **one rule**. Each rule captures, applies,
validates, and commits or rolls back on its own before the engine moves to
the next rule. A failure in rule 7 rolls back rule 7; it doesn't unwind
rules 1 through 6, which already committed and are individually recorded.
This keeps the blast radius of any single failure to a single rule and makes
the transaction log a per-rule ledger you can query and selectively reverse.

## Agent mode vs shell fallback

Kensa changes a host through one of two paths, chosen automatically:

- **Agent mode** is the default on `remediate`. Kensa spawns a small agent on
  the target that drives kernel primitives directly: atomic file replace via
  `renameat2`, `/proc/sys` writes, `delete_module(2)`, systemd over D-Bus,
  audit over netlink. This is where the strongest atomicity guarantees live:
  a crash mid-apply leaves either the old bytes intact or the new bytes
  complete, never a torn file.
- **Shell fallback** runs the equivalent change over the plain SSH shell
  transport. It's used when agent bootstrap isn't viable, and it's selected
  per-mechanism when a kernel primitive or privilege isn't available. Both
  paths write byte-identical files and record an identical pre-state, so
  capture and rollback behave the same either way; the shell path's
  mid-apply crash semantics are best-effort rather than kernel-atomic.

Set `KENSA_NO_AGENT=1` to force the shell path everywhere, which helps when
the agent can't be bootstrapped. You lose the kernel-atomic guarantee on the
file mechanisms but keep the four-phase transaction and rollback.

`check` is read-only and doesn't need agent mode for its guarantees; the
agent matters where Kensa *writes*.

## Reversible vs non-reversible rules

A rule is `transactional: true` (the default and the majority of the corpus)
or `transactional: false`.

- **Transactional rules** are backed by a mechanism that can both capture the
  prior state and restore it. These are covered by the atomicity guarantee:
  apply-time failure rolls back automatically, and you can deliberately roll
  them back later with `kensa rollback`.
- **Non-transactional rules** use escape-hatch mechanisms (`command_exec`,
  `manual`, bootloader parameter changes) that Kensa can't manufacture an
  inverse for. Kensa runs them and records them in the transaction log for
  audit, but they are **not** under the rollback guarantee. A non-capturable
  step that ran successfully is not reversed on a later failure, and rollback
  reports it as skipped rather than pretending to restore it.

The reversal level of each shipped mechanism (Atomic, Reversible,
Best-effort, Staged, or None) is tabulated in
[10-mechanisms](/docs/kensa/mechanisms). Boot-parameter changes are a special
"Staged" case: Kensa never edits the saved boot default directly but stages
the change through a one-shot trial boot, so a host that fails to boot
reverts on its own.

## Platform gating

The shipped corpus is large and not every rule applies to every host. Before
running a rule, Kensa compares the rule's `platforms:` block against the
host's detected OS. A rule that doesn't apply to this host renders **`SKIP`**:
it is neither checked for a pass/fail nor, on `remediate`, ever applied.
This is deliberately lenient: a rule with no `platforms:` block runs
everywhere, and an undetectable host OS gates nothing.

Gating is the standalone-CLI safety net. It's why a `SKIP` is meaningful in
the output: it tells you the rule was *intentionally not applicable here*,
not that it errored. (In a fleet, an orchestrator typically pre-filters by
platform upstream; the in-engine gate protects the CLI user who has no such
upstream.)

## Signed evidence envelopes

Every transaction, committed or rolled back, produces a structured
evidence envelope: a signed record carrying the timestamp and duration,
host context, the pre-state snapshot, the change attempted, the validation
results, the commit-or-rollback decision, the post-state, and the
framework-control mappings. The envelope is signed with an Ed25519 key so an
auditor can verify a finding months later without access to the original
host; `kensa verify` checks that signature on an envelope file.

Evidence is stored alongside the change in the transaction log, not in a
separate silo that can drift out of sync, and it can be exported in Open
Security Controls Assessment Language (OSCAL) 1.0.6 for regulatory
submission. For a stable signer identity across runs,
point `KENSA_SIGNING_KEY` at your private key (see [01-install](/docs/kensa/install));
without it Kensa uses an ephemeral per-process key.

## Rollback completeness

The whole guarantee rests on one thing: **Capture must record every piece of
state that Apply touches.** If Apply changes something Capture didn't record,
rollback can't restore it; the reversal would be incomplete. So
capture-completeness is treated as a first-class property of every
capturable handler, not an afterthought.

This is enforced by a *footprint* pre-commit gate: in agent mode the kernel-IO
layer records each filesystem resource a handler actually touches, and the
engine refuses to commit (rolling back first) if a handler touched anything
it didn't capture (`observed ⊆ captured`). A pre-apply restorability probe also
refuses to mutate a captured resource that is immutable (`chattr +i`), because a
rollback that can't rewrite it is impossible. The gate is opt-in per handler and
covers the kernel-atomic (fsatomic-funnelled) filesystem writes on the agent
path; for the direct-SSH shell fallback and for non-opted handlers, the
mandatory human review every capture/rollback handler goes through
(`CONTRIBUTING.md`) remains the backstop. Either way, the operator-facing
promise is the same: a rule that can't be fully restored does not get to claim a
clean commit.

## Where this leads

- [02-quickstart](/docs/kensa/quickstart): these concepts as four commands.
- [10-mechanisms](/docs/kensa/mechanisms): every mechanism, where it runs, and its
  reversal level.
- `docs/TRANSACTION_CONTRACT_V1.md`: the external-facing atomicity
  commitment in full.

---

# Kensa: Scan and remediate
URL: https://www.hanalyx.com/docs/kensa/scan-and-remediate

_Applies to: Kensa v0.6.0 — last updated 2026-06-22._

Two commands do the work: `kensa check` reads a host and reports
compliance without touching it, and `kensa remediate` applies the
failing rules as atomic transactions. They share the same target,
rule-selection, and output flags, so most of this chapter applies to
both; the differences are called out where they matter.

- [`check`: read-only compliance](#check-read-only-compliance)
- [`remediate`: apply failing rules](#remediate-apply-failing-rules)
- [Choosing a target](#choosing-a-target)
- [Selecting rules](#selecting-rules)
- [Privilege: sudo](#privilege-sudo)
- [Live result rows](#live-result-rows-default-text-output)
- [Outcomes: pass, fail, skipped, error](#outcomes-pass-fail-skipped-error)
- [Platform gating](#platform-gating)
- [Output formats](#output-formats)
- [Agent mode](#agent-mode)

---

## `check`: read-only compliance

```bash
kensa check -H 192.168.1.211 -u owadmin --sudo -r ./rules
```

`check` evaluates each rule's check method against the host and prints a
verdict per rule. It changes nothing: no apply, no transaction, no
rollback. By default it does not even write to the SQLite store; pass
`--store` if you want the scan persisted as a session you can later
`diff` or feed to `rollback --list`.

`check` is the only scan command that accepts an **inventory** for
multi-host runs (see [Choosing a target](#choosing-a-target)).

## `remediate`: apply failing rules

```bash
kensa remediate -H 192.168.1.211 -u owadmin --sudo -r ./rules
```

`remediate` runs each rule that is failing as a four-phase atomic
transaction: **Capture → Apply → Validate → Commit**. If validation
fails, the engine rolls the host back to the captured pre-state in the
same run, so a rule either lands cleanly or leaves no trace. A rule that
already passes is reported `PASS` and is not re-applied. Every committed
transaction is written to the store, which is what makes it reversible
later with [`kensa rollback`](/docs/kensa/rollback-and-history).

`remediate` is single-host: `--host` is required and there is no
`--inventory`.

---

## Choosing a target

Both commands take the same SSH connection flags.

| Flag | Meaning |
|---|---|
| `-H, --host` | Target hostname. Required (for `check`, required unless `--inventory`). |
| `-u, --user` | SSH user. Defaults to the current local user. |
| `-k, --key` | SSH private-key path. |
| `-p, --password` | SSH password. Omit the value (`-p`) for a TTY prompt. |
| `-P, --port` | SSH port (default 22). |
| `--strict-host-keys` | Verify SSH host keys; reject an unknown host. |
| `--no-strict-host-keys` | Trust on first use. This is today's default. |
| `-C, --capability KEY=VALUE` | Override a detected capability. Repeatable; on duplicate keys the last value wins (for example, `-C apparmor=true -C selinux=false`). |

Kensa drives the system `ssh` client (with ControlMaster), so your
`~/.ssh/config`, agent keys, and jump hosts all work as they normally
do.

### Inventory (`check` only)

```bash
kensa check --inventory hosts.ini --sudo -r ./rules
kensa check --inventory hosts.ini -w 10 --sudo -r ./rules
```

| Flag | Meaning |
|---|---|
| `-i, --inventory` | Ansible-style `inventory.ini` for a multi-host check. |
| `-l, --limit` | Limit inventory hosts to a glob/group pattern (ansible `--limit` semantics). |
| `-w, --workers` | Concurrent SSH connections, 1–50 (1 = sequential, the default). |

Each host scans independently; stdout carries the concatenated per-host
result documents. `remediate` and `rollback` are single-host and have no
inventory mode.

---

## Selecting rules

Rules come from three additive sources, combined into one set before any
filter is applied:

| Flag | Meaning |
|---|---|
| `-r, --rules-dir DIR` | Scan a directory for `*.yml` rule files. |
| `--rule FILE` | Load one rule YAML file, strictly: a parse error fails the command. Repeatable; additive with `--rules-dir` and positional args. |
| *positional* `rule.yml …` | Load named rule files directly. |

With the `kensa-rules` package installed, the corpus lives at
`/usr/share/kensa/rules` and is picked up automatically when you pass no
rule source at all. (Resolution order: explicit `--rules-dir` wins;
positional rule files alone skip the directory walk; otherwise the
default path is used; otherwise you get a usage error naming all three
fixes.)

### Filters

Filters narrow the loaded set. They combine with AND across *kinds*
(a rule must satisfy every filter you give) and OR *within* the
repeatable ones.

| Flag | Repeatable | Meaning |
|---|---|---|
| `-s, --severity` | yes (OR) | `critical` \| `high` \| `medium` \| `low`. |
| `-t, --tag` | yes (OR) | A rule matches if its `tags:` array contains any of these. |
| `-c, --category` | no | Single category (for example, `-c access-control`). A later `-c` overrides an earlier one. |
| `-f, --framework` | no | Keep rules mapping any control under FRAMEWORK (for example, `-f cis-rhel9`). Hyphen and underscore are interchangeable: `-f cis-rhel9` == `-f cis_rhel9`. |
| `--control` | yes (OR) | `FRAMEWORK:CONTROL` (for example, `--control cis-rhel9:5.1.12`). The framework portion accepts a hyphen or an underscore. |

```bash
# Critical + high only:
kensa check -H 192.168.1.211 -s critical -s high -r ./rules

# One framework, one control:
kensa check -H 192.168.1.211 -f cis-rhel9 --control cis_rhel9:5.1.12 -r ./rules
```

### Rule variables

Some rules carry `{{ var }}` placeholders (for example a faillock
threshold). You supply values two ways:

| Flag | Meaning |
|---|---|
| `-x, --var KEY=VALUE` | Override one variable. Repeatable. Wins over `--config-dir`/`defaults.yml`. |
| `--config-dir DIR` | Directory holding `defaults.yml`, the variable-defaults source. |

> **Security note.** A `--var` VALUE is spliced literally into the rule
> YAML and may flow into shell commands the handlers run on the target.
> Pass only trusted input. (This is the documented `--var` trust limit in
> `docs/test_docs/security.md`.)

---

## Privilege: sudo

Most compliance checks read root-owned files, so you almost always
use `--sudo`. Kensa supports both passwordless and password sudo.

| Flag | Meaning |
|---|---|
| `--sudo` | Wrap remote commands in sudo. |
| `--sudo-password` | A sudo password for hosts where NOPASSWD is not configured. Omit the value (`--sudo-password`) for a TTY prompt, or set `KENSA_SUDO_PASSWORD`. Requires `--sudo`. |

- **Passwordless (default).** With `--sudo` alone, Kensa runs
  `sudo -n` and never prompts. If the host's sudoers policy requires a
  password, a connect-time probe fails fast: *configure NOPASSWD or
  supply a sudo password*.
- **With a password.** Supply it via `--sudo-password`, the prompt, or
  `KENSA_SUDO_PASSWORD`. The password is fed over the SSH session's
  **stdin**, never placed in argv and never recorded in evidence
  (`CheckEvidence` / Open Security Controls Assessment Language (OSCAL)).
  A wrong password is reported as *sudo password rejected*.

`SUDO_ASKPASS` / `sudo -A` is deliberately **not** supported: it needs an
askpass helper on the target, which the agentless model does not ship.

```bash
# Passwordless sudo:
kensa check -H 192.168.1.211 -u owadmin --sudo -r ./rules

# Password sudo from the environment (CI-friendly — no value in argv):
KENSA_SUDO_PASSWORD=… kensa remediate -H 192.168.1.211 -u owadmin --sudo -r ./rules

# Password sudo with an interactive prompt:
kensa check -H 192.168.1.211 -u owadmin --sudo --sudo-password -r ./rules
```

---

## Live result rows (default text output)

In the default text output, `check` and `remediate` stream their results
**live**: one aligned row per rule, printed **as each rule completes**,
in scan order. You watch a long scan advance instead of waiting for a
buffered report. There is no `--progress` flag and no separate progress
channel; the rows *are* the text result, on stdout.

```
───────────────────── Host: 192.168.1.211 ──────────────────────
  Platform: RHEL 9.6
  PASS   MED   cron-logging                 Ensure cron logging is enabled
  FAIL   LOW   journald-compress            Configure journald to compress logs  config_value: key "Compress" not found in /etc/systemd/journald.conf
  PASS   MED   rsyslog-installed            Ensure rsyslog is installed
  FAIL   MED   rsyslog-file-permissions     Ensure rsyslog log file creation mode is configured  config_value: key "$FileCreateMode" not found in /etc/rsyslog.conf

  6 passed, 8 failed  (of 14)
```

- A full-width `Host:` banner, then a `Platform:` line, then one indented row
  per rule. There is no column-header row.
- Columns are `STATUS  SEVERITY  RULE-ID  DESCRIPTION`, with a trailing detail
  appended on `FAIL` / `ERROR` / `SKIP` rows.
- `STATUS` is `PASS` / `FAIL` / `ERROR` / `SKIP`. `remediate` adds `FIXED`
  (remediated this run); `PASS` there means already compliant.
- `SEVERITY` renders as `CRIT` / `HIGH` / `MED` / `LOW`.
- `STATUS` and `SEVERITY` are colored **only when stdout is a terminal**;
  redirected or piped output is plain text with no escape sequences.
- The tally lists only the non-zero outcomes and ends with `(of N)`; it adds a
  `skipped` (and `error`) count when any rule is skipped or errors.
- `-v, --verbose` (text only) expands the compacted PASSED list.
- `-q, --quiet` suppresses the default output entirely; errors still go
  to stderr.

The live rows apply to the **default human output only** (`--format
table`/`text`, or no format flag, with no `-o FILE`). Choosing a machine
format or an `-o FILE` destination turns the stream off; machine output
is always buffered and structured, never interleaved with rows.

The exit code and every `-o` output are produced from the canonical
`ScanResult` / `RemediationResult`, not reconstructed from the rendered
rows. Read the result document for the record of what happened; the rows
are the same data rendered for a human as it arrives.

---

## Outcomes: pass, fail, skipped, error

Every rule resolves to exactly one of four outcomes, the canonical
compliance verdict carried on `ScanResult.Outcomes`:

| Outcome | Meaning |
|---|---|
| `pass` | The host already satisfies the rule. |
| `fail` | The host does not satisfy the rule. On `remediate`, this is what gets applied (and becomes `FIXED` on success). |
| `skipped` | The rule does not apply to this host (see [Platform gating](#platform-gating)). It was not evaluated and, on `remediate`, never applied. |
| `error` | The check could not be completed (a command failed, the host was unreachable mid-scan, a capability could not be probed). |

These outcomes are what an embedder (OpenWatch) consumes; the legacy
`Transactions` surface, where `committed`/`rolled_back` double as
compliant/non-compliant, is kept only for backward compatibility.

---

## Platform gating

A rule may declare a `platforms:` block (an OS family plus optional
`min_version`/`max_version`). Before evaluating, Kensa reads the host's
OS from `/etc/os-release` and compares:

- If the rule **does not apply**, it renders `SKIP` with a detail such as
  `not applicable: host RHEL 8.10, rule targets rhel >=9`, instead of a
  misleading pass or fail. On `remediate`, a skipped rule's remediation
  is **never applied** (the engine is provably not invoked for it).
- A rule with **no `platforms:` block** runs everywhere.
- A host whose OS **cannot be detected** is never gated (every rule
  runs), so a detection blip cannot silently skip a scan.

Platform gating is the standalone-CLI safety net. In a fleet, OpenWatch
pre-filters by platform upstream; this in-engine gate exists for CLI
users running without it.

> The shipped corpus currently targets RHEL. Running it unmodified
> against Ubuntu skips the RHEL-only rules (they render `SKIP`), so a
> stock Ubuntu scan reports few or no in-platform rules today.

---

## Output formats

Two flag families control output. `--format` is the legacy single-format
selector; `-o`/`--output` is the current, repeatable, file-or-stdout
destination selector. Prefer `-o`.

| Flag | Meaning |
|---|---|
| `--format` | `check`: `table`, `json`, `jsonl`. `remediate`: `table`, `json`. **Deprecated**; use `--output`. |
| `-o, --output FORMAT[:PATH]` | Output destination. Repeatable. `PATH` omitted (or `-`) means stdout. |
| `-q, --quiet` | Suppress default output. |
| `--oscal FILE` | (`remediate` only) **Deprecated** alias for `-o oscal:FILE`. |

`FORMAT:PATH` lets you fan out to several artifacts in one run:

```bash
# JSON to a file:
kensa check -H web01 --sudo -o json:result.json -r ./rules

# Two artifacts at once: JSONL on stdout + OSCAL to a file:
kensa check -H web01 --sudo -o jsonl -o oscal:assessment.json -r ./rules

# Remediation, JSON on stdout + OSCAL to a file:
kensa remediate -H web01 -u admin --sudo -o json -o oscal:/tmp/results.oscal.json -r ./rules
```

### Formats by command

| Format | `check` | `remediate` | Notes |
|---|---|---|---|
| `text`/`table` | yes | yes | Default; the live row stream. |
| `json` | yes | yes | The canonical result struct. |
| `jsonl` | yes | — | One JSON object per line (NDJSON); maps from `Outcomes`, with a first-class `skipped` count. |
| `csv` | yes | yes | Row-per-rule for spreadsheets. |
| `pdf` | yes | yes | Binary report (path required). |
| `evidence` | yes | yes | Kensa-native evidence document. |
| `oscal` | yes | yes | OSCAL 1.0.6 Assessment Results. |

### `-o evidence:` Kensa-native evidence

```bash
kensa check -H web01 --sudo -o evidence:scan-evidence.json -r ./rules
```

Reproducible per-check proof behind each verdict: session and host
context plus, per rule, every command run with its stdout/stderr, exit
code, expected vs actual, and a `Truncated` flag (64 KiB per-field cap).
On `check` this document is **unsigned**; the Ed25519 signature is
exclusive to the remediation evidence-envelope path.

### `-o oscal:` OSCAL 1.0.6

```bash
kensa check -H web01 --sudo -o oscal:assessment.json -r ./rules
```

A NIST OSCAL 1.0.6 Assessment Results document: one finding and one
observation per rule, framework refs as token-valid control-ids, the
verbatim command in `remarks`, raw stdout as base64 back-matter. The
output is conformance-gated against the vendored OSCAL 1.0.6 schema. The
scan-path OSCAL is **unsigned by design**; the remediation path
(`remediate -o oscal:`) anchors it in the signed evidence envelope.

> The evidence and OSCAL **schema files** are dev/CI assets and are not
> shipped in the rpm/deb until v1.0.0 (founder decision, 2026-06-13). The
> evidence and OSCAL **output** ships normally; only the schema files are
> unshipped.

---

## Agent mode

On `remediate`, Kensa runs the apply path through an on-host **agent** by
default. The agent drives kernel primitives directly (`/proc/sys` writes,
atomic file replacement, `delete_module(2)`, the systemd D-Bus helper,
audit netlink), which is what gives the file mechanisms their byte-exact
atomicity.

- `--sudo` spawns the agent as **root** (`sudo kensa agent`). The agent
  must run as root, or its direct `/proc`+`/etc` writes hit `EACCES`.
- Set `KENSA_NO_AGENT=1` to disable the agent and fall back to
  shell-best-effort. The fallback writes byte-identical files and records
  an identical pre-state, so capture and rollback are path-agnostic, but
  you lose the kernel-primitive atomicity on the agent-only mechanisms.

`check` is read-only and does not spawn the agent.

---

## Exit codes

`kensa` follows a small, stable convention (see `kensa --help`):

| Code | Meaning |
|---|---|
| `0` | Success (also `--help` / `--version`). |
| `1` | Runtime error (connect failure, host error). |
| `2` | Usage error (bad flag, unknown subcommand, missing required arg). |

A scan that completes but finds failing rules still exits `0`; the
failures are in the result, not the exit status. Read the result
document (or the rows) for the compliance verdict.

## Next

[05 · Rollback and history](/docs/kensa/rollback-and-history) covers undoing a
remediation, recovering from a crash, and querying the transaction log.

---

# Kensa: Rollback and history
URL: https://www.hanalyx.com/docs/kensa/rollback-and-history

_Applies to: Kensa v0.6.0 — last updated 2026-06-22._

Every `kensa remediate` writes what it did to a durable transaction log
(SQLite). That log is what makes a remediation reversible, what crash
recovery replays, and what you query to audit a host over time. This
chapter covers the four commands built on it:

- [`rollback`](#rollback-undo-a-remediation): undo committed transactions
- [`recover`](#recover-crash-recovery): repair an interrupted run
- [`history`](#history-query-the-transaction-log): query the log
- [`diff`](#diff-drift-between-two-sessions): compare two scans

It also explains [how a transaction reaches its terminal
status](#how-a-transaction-reaches-its-terminal-status).

---

## How a transaction reaches its terminal status

A `remediate` runs each rule as one **transaction**, the four-phase Kensa
operation (capture, apply, validate, then commit or roll back). The
terminal status the transaction lands in is what the log records:

| Status | How it gets there |
|---|---|
| `committed` | Every apply step succeeded and every validator passed. The host is in the target state; the signed evidence envelope (the Ed25519-signed record of what the transaction did) is persisted. This is the status `rollback` reverses. |
| `rolled_back` | Apply or validate failed, and the engine reversed every applied capturable step using the captured pre-state, in the **same run**. The host is back in its exact pre-change state. |
| `partially_applied` | A `transactional: false` rule ran at least one non-capturable step before failing. Those steps are not reversed; per-step `Stranded` flags say which. |
| `errored` | A phase could not complete within the deadline, or a terminal step (signing or persistence) failed. `HostUnchanged` distinguishes an abort that never mutated the host. |
| `rollback_failed` | Apply/validate failed and the engine tried to reverse, but the restoration could not be machine-verified (a rollback step failed or reported a partial restore). The host is in an unconfirmed state. |
| `recovered` | An interrupted transaction (the process died after pre-state was persisted but before any terminal record) was reversed out-of-band by [`kensa recover`](#recover-crash-recovery). |

The key distinction: **`rolled_back`** is the engine undoing a failure
*within the same remediate run*; **`kensa rollback`** is you reversing a
*committed* transaction later; **`recovered`** is `kensa recover`
cleaning up a *crash*.

A capturable mechanism writes its pre-state to durable storage **before**
any host change, which is what lets rollback and recovery restore the
host. A `transactional: false` rule captures nothing, so it is outside
the reversal guarantee (see [10 · Mechanisms](/docs/kensa/mechanisms)).

---

## `rollback`: undo a remediation

`kensa rollback` reverses already-**committed** transactions using their
captured pre-state. Pick exactly one mode.

### Read-only modes (no host needed)

```bash
kensa rollback --list
kensa rollback --info <session-id> --detail
```

| Flag | Meaning |
|---|---|
| `--list` | List rollback-able sessions (those with committed transactions). |
| `--info SESSION_ID` | Show a session's detail (its transactions and their statuses). |
| `--detail` | Modifier: add a per-step breakdown. Composes with `--list` and `--info` (not with `--start`/`--txn`). |

Sessions come from `kensa check --store` (the scan-persistence path); a bare
`kensa remediate` records individual transactions, not a session. So to roll
back a remediation you just ran on the CLI, find its transaction UUID with
`kensa history` and use `rollback --txn` (below). The session modes
(`--list` / `--info` / `--start`) apply once you have persisted sessions; find
their UUIDs with `kensa list sessions` (the `session_id` column).

### Executing a rollback (host required)

```bash
# Reverse every committed transaction in a session:
kensa rollback --start <session-id> -H 192.168.1.211 -u owadmin --sudo

# Legacy: reverse a single transaction by UUID:
kensa rollback --txn <txn-uuid> -H 192.168.1.211 -u owadmin --sudo
```

| Flag | Meaning |
|---|---|
| `--start SESSION_ID` | Execute rollback for **every** committed transaction in the session. Needs `--host`. |
| `-T, --txn TXN_UUID` | Legacy: roll back a single transaction by UUID. Needs `--host`. |

`--start` reverses a whole **session**; the binary's help labels `--txn` as
*legacy* because the session model is the intended primary path. On the
standalone CLI, though, `--txn` is what you use after a `kensa remediate`
(which records transactions, not sessions): take the UUID from `kensa history`
and roll it back. Both connect to the host, so they take the
same target flags as `check`/`remediate`: `-H/--host` (required here),
`-u/--user`, `-k/--key`, `-P/--port`, `--sudo`, `--sudo-password`
(and `KENSA_SUDO_PASSWORD`), `--strict-host-keys`/`--no-strict-host-keys`.

Output: `--format text` (default) or `json`; `-q/--quiet` suppresses it.

> What rollback restores depends on the mechanism. File mechanisms
> restore byte-for-byte; reversible mechanisms restore the captured state
> and verify it (a runtime aspect may need a restart or reboot, reported
> as a partial restore); best-effort mechanisms restore through the
> host's own tool. See the reversal-level table in
> [10 · Mechanisms](/docs/kensa/mechanisms).

---

## `recover`: crash recovery

If a `kensa` process is interrupted mid-transaction (killed, host lost,
power failure) **after** pre-state was persisted but **before** a
terminal record was written, the transaction is left open. `kensa
recover` compensates those open transactions from the durable
crash-recovery journal: each is rolled back from its captured pre-state
and recorded as `recovered`.

```bash
kensa recover -H 192.168.1.211 -u owadmin --sudo
```

| Flag | Meaning |
|---|---|
| `-H, --host` | Scope recovery to this host (also the SSH target). **Required.** |
| `-u, --user` | SSH user (default: current user). |
| `-P, --port` | SSH port (default 22). |
| `--key` | SSH private-key path. |
| `--sudo` | Wrap commands in sudo. |
| `--sudo-password` | Sudo password for non-NOPASSWD hosts. |
| `--strict-host-keys` | Verify SSH host keys; reject unknown. |
| `-q, --quiet` | Suppress default output. |
| `-D, --db` | SQLite transaction-log path (default `.kensa/results.db`). |

Run it **after a crash, when no live `kensa` is operating the host.**
`recover` reverses every captured mechanism (Atomic, Reversible, or
Best-effort); it cannot reverse a `transactional: false` step, because
none was captured.

### `recover.lock` fencing

`recover` takes an **exclusive** cross-process advisory lock (`flock`) at
`<db>.recover.lock` (`RecoverLockPath` is the store path plus
`.recover.lock`), so it can never race a second recovery run or a live
engine on the same store. A live engine takes the same lock **shared**;
the exclusive acquisition blocks while any engine holds it. If the lock
is already held, `recover` refuses to proceed rather than racing
(`ErrRecoverLocked`). The lock is released automatically if the process
dies (it is per-open-file-description).

The `-H/--host` scope is what keeps recovery surgical: only that host's
open transactions are compensated.

---

## `history`: query the transaction log

`kensa history` queries the log. With no filters it lists the most recent
transactions.

```bash
kensa history                                  # 50 most recent
kensa history -H 192.168.1.211 -S 24h          # one host, last 24h
kensa history -T <txn-uuid>                    # one transaction by UUID
kensa history -a by_host -S 7d                 # 7-day posture per host
kensa history --stats                          # summary counts, then exit
```

| Flag | Meaning |
|---|---|
| `-H, --host` | Filter by host ID. |
| `-R, --rule` | Filter by rule ID. |
| `-S, --since` | Since a duration (`24h`) or an RFC3339 time. |
| `-n, --limit` | Max rows (default 50). |
| `-T, --txn` | Fetch a single transaction by UUID. |
| `-a, --aggregate` | Aggregate key: `by_host`, `by_rule`, or `by_framework_control`. |
| `--stats` | Print summary stats (sessions, transactions, by status / severity / host) and exit. |
| `--format` | `table` (default), `json`, or `jsonl` (jsonl is transaction-list only). |
| `--prune N` | Delete sessions (and cascade) older than N days. **Destructive**; long-only. |
| `--force` | Skip the `--prune` confirmation prompt (required in non-interactive runs). |
| `-q, --quiet` | Suppress default output. |

```bash
# Stream the last 200 to a log aggregator as JSON Lines:
kensa history -n 200 --format jsonl | jq -c .

# Prune in CI/cron without a prompt:
kensa history --prune 30 --force
```

`--prune` cascades: deleting a session removes its transactions too. The
interactive form prompts for confirmation; add `--force` for cron and CI.

---

## `diff`: drift between two sessions

`kensa diff` compares two **stored** sessions and emits the per-rule
drift between them: status changes, rules added (in the second session
only), and rules removed (in the first only).

```bash
kensa list sessions                          # find the session IDs first
kensa diff <id1> <id2>                        # compact drift report
kensa diff <id1> <id2> --show-unchanged       # include unchanged rules
kensa diff <id1> <id2> --format json          # programmatic output
```

| Flag | Meaning |
|---|---|
| `--show-unchanged` | Also list rules whose status is identical between the two sessions. |
| `--format` | `text` (default) or `json`. |
| `-q, --quiet` | Suppress default output. |

The direction follows git's convention: `SESSION_ID_1` is the earlier
("before") snapshot and `SESSION_ID_2` is the later ("after") one;
reversing the arguments inverts the report. Comparing across hostnames is
allowed; a stderr note discloses the cross-host scope.

To populate sessions for `diff`, run `check --store` (a persisted scan)
or `remediate` (always persisted), then find the UUIDs with `kensa list
sessions`.

---

## Finding session IDs

Most of these commands key off a session UUID. List them with:

```bash
kensa list sessions                          # 20 most recent
kensa list sessions -H 192.168.1.211         # one hostname
kensa list sessions --format json -n 5       # last 5 as JSON
```

The `session_id` column is the UUID that `rollback --info` / `--start`
and both `diff` arguments expect.

## Next

[06 · Rule authoring](/docs/kensa/rule-authoring) covers writing your own
rules; [10 · Mechanisms](/docs/kensa/mechanisms) details the reversal level you
get from each mechanism.

---

# Kensa: Rule authoring
URL: https://www.hanalyx.com/docs/kensa/rule-authoring

_Applies to: Kensa v0.6.0 — last updated 2026-06-22._

A *rule* is a single, framework-independent statement of desired system state.
It carries its own check logic, its remediation, its framework cross-references,
and one or more capability-gated implementations. You write it once and it
applies across every supported OS version and framework. Rules are YAML, one
file per rule, under `rules/` organized by category.

Rules are *inputs to the transaction engine*. A rule declares *what* state it
wants and *which mechanism* produces it; the engine provides the *how* and the
atomicity *guarantee* (capture → apply → validate → commit-or-rollback). The
rule YAML never expresses capture, validation, or rollback; those are engine
concerns. The authoritative schema is
[`CANONICAL_RULE_SCHEMA_V1.md`](../foundation_docs/CANONICAL_RULE_SCHEMA_V1.md);
this chapter is the working subset.

## A complete rule

This is the canonical "disable SSH root login" rule. It shows every field you
reach for most of the time:

```yaml
id: ssh-disable-root-login          # unique, kebab-case, stable forever
title: Disable SSH root login        # imperative, max 100 chars
description: >                        # 2–4 sentences: what it enforces and why
  Direct root login over SSH is disabled so that administrators authenticate
  as themselves and escalate explicitly.
rationale: >                         # security justification
  Permitting root login removes individual accountability and exposes the most
  privileged account to remote password and key attacks.
severity: high                       # critical | high | medium | low
category: access-control             # must match a rules/ subdirectory
tags: [ssh, authentication, cis]     # free-form classification labels

references:                          # framework cross-references (all optional)
  cis:
    rhel9: { section: "5.2.7", level: "L1", type: "Automated" }
  stig:
    rhel9: { vuln_id: "V-257947", severity: "CAT II", cci: ["CCI-000770"] }
  nist_800_53: ["AC-6(2)", "IA-2(5)"]

platforms:                           # which OS families/versions this targets
  - family: rhel
    min_version: 8                   # inclusive; omit max_version for open-ended

implementations:                     # one or more check + remediation variants
  - when: sshd_config_d              # capability gate (optional)
    check:
      method: config_value
      path: "/etc/ssh/sshd_config.d"
      key: "PermitRootLogin"
      expected: "no"
      scan_pattern: "*.conf"
    remediation:
      mechanism: config_set_dropin
      dir: "/etc/ssh/sshd_config.d"
      file: "00-kensa-root-login.conf"
      key: "PermitRootLogin"
      value: "no"
      reload: "sshd"
  - default: true                    # exactly one implementation must be default
    check:
      method: config_value
      path: "/etc/ssh/sshd_config"
      key: "PermitRootLogin"
      expected: "no"
    remediation:
      mechanism: config_set
      path: "/etc/ssh/sshd_config"
      key: "PermitRootLogin"
      value: "no"
      reload: "sshd"
```

## Metadata and classification

`id`, `title`, `description`, `rationale`, and `severity` are required. The
`id` is stable for the life of the rule. Once assigned it never changes and is
never reused. `category` must match one of the directory names under `rules/`
(`access-control`, `audit`, `filesystem`, `kernel`, `logging`, `network`,
`services`, `system`), and `tags` is a free-form list for filtering
(`kensa check -t cis`, `-c access-control`).

## `transactional`

`transactional` is optional and defaults to `true`. Leave it at the default
when every step in every implementation uses a *capturable* mechanism; the
engine can then run the rule atomically and roll it back. You **must** set
`transactional: false` when any step uses a non-capturable mechanism
(`command_exec`, `manual`, `grub_parameter_set`, `grub_parameter_remove`); the
validator rejects a `transactional: true` rule that contains one. See
[Mechanisms reference](/docs/kensa/mechanisms) for which mechanisms are capturable.

## `references`: framework mappings

`references` maps the rule to external framework identifiers and is what
`--framework` and `--control` filter on. `cis` and `stig` are objects keyed by
`{os}{version}` (they carry version-specific section / vuln-id metadata). The
remaining frameworks (`nist_800_53`, `pci_dss_4`, `iso27001_2022`, `cmmc_l2`,
`hipaa`, `srg`) are flat lists of control IDs because those identifiers are
stable across OS versions.

## `platforms`: scope

Each entry has a required `family` and `min_version`, with optional
`max_version` (inclusive) and `derivatives` (defaults `true`). A rule with no
`platforms` block runs everywhere; a rule scoped to `rhel min_version: 9`
renders `SKIP` on RHEL 8 and is never remediated there (see
[Troubleshooting](/docs/kensa/troubleshooting) on out-of-platform skips).

## `implementations`: checks and remediations

Implementations are evaluated top to bottom; the first whose `when` capability
gate the host satisfies is selected, so order the specific variants before the
`default: true` fallback. **Exactly one** implementation must be `default: true`.

`when` may be a single capability, or `all:` / `any:` / `not:` over a list:

```yaml
when: sshd_config_d                      # single capability
when: { all: [authselect, pam_faillock] }
when: { any: [crypto_policy_modules, fips_mode] }
when: { not: systemd_resolved }
```

Each implementation has a `check` and a `remediation`:

- **`check.method`** is a read-only verb: `config_value`, `sysctl_value`,
  `package_state`, `file_exists`, `service_state`, `audit_rule_exists`,
  `mount_option`, `command` (escape hatch), and others. Each method declares its
  required fields; for example `config_value` needs `path`, `key`, and
  `expected`, and takes an optional `comparator` (`==`, `!=`, `<`, `<=`, `>`,
  `>=`; use `<=`/`>=` for thresholds like `PASS_MAX_DAYS <= 365`) and
  `delimiter`. Set `delimiter: " "` for whitespace-separated files such as
  `/etc/login.defs` (`KEY value`); the default delimiter is `=`. The full method
  table is schema §3.5.3.
- **`remediation.mechanism`** names the action that produces the desired state,
  plus that mechanism's fields. See the [Mechanisms reference](/docs/kensa/mechanisms)
  for the complete catalog, where each mechanism runs, and what reversal you get.

For ordered remediations use a `steps:` list instead of a single `mechanism`;
the engine captures pre-state for every step before any runs, and rolls back all
prior successful steps in reverse order if a later step fails.

## Variables and `{{ var }}` substitution

Site-specific values are templated with `{{ var }}` and substituted before the
rule is parsed. For example, a remote-logging rule writes
`value: "@@{{ rsyslog_remote_server }}"`. Supply values with `--var
KEY=VALUE` (repeatable) or from a `defaults.yml` in `--config-dir`; `--var`
wins over `defaults.yml`. A variable value is spliced literally into the rule
YAML and may flow into shell commands run by handlers, so pass only trusted
input.

## `depends_on` and relationships

`depends_on` lists rule IDs that must be satisfied first (for example, a
firewall-backend rule `depends_on: [service-enable-firewalld]`).
`conflicts_with` marks mutually exclusive rules and `supersedes` records rule
IDs this one replaces. All three are optional ID lists.

## Validate before you commit

Every rule must pass the validator before it enters the corpus. Run it over the
whole tree:

```bash
./bin/kensa-validate --rules-dir rules
```

A clean corpus reports `0 error(s)` (the sole expected warning is a stylistic
W005 on `selinux-policy-targeted.yml`). Any `FAIL` line names the file, the rule
ID, and the violated constraint, for example `exactly one implementation must
have default:true` if you forgot the fallback, or a `transactional: true` rule
that contains a non-capturable mechanism. Fix every error before opening a PR;
CI runs the same gate.

## Next

[07-integration](/docs/kensa/integration) covers consuming scan results downstream;
[08-troubleshooting](/docs/kensa/troubleshooting) covers what to do when a scan or
remediation does not behave as a rule expects.

---

# Kensa: Integration
URL: https://www.hanalyx.com/docs/kensa/integration

_Applies to: Kensa v0.6.0 — last updated 2026-06-22._

This chapter is for programs that **embed** Kensa (notably OpenWatch) rather
than run the CLI, consuming its `api`/`pkg/kensa` Go surfaces. The division of
labor: Kensa is to a single host what `git` is to a repository; OpenWatch (or
your own control plane) presents, schedules, and aggregates across a fleet, and
never re-implements what Kensa does for one host. Embedders import the **frozen**
`api/` contract (semver-stable; additive-only) plus the assembly layer
`pkg/kensa` (public, not frozen).

Programs importing the `api` Go package
(`github.com/Hanalyx/kensa/api`) should read a scan's compliance verdicts
from `ScanResult.Outcomes` (since v0.3.0): one `RuleOutcome` per rule
with a `ComplianceStatus` of `pass` / `fail` / `skipped` / `error`, the
rule's severity, a human-readable detail, and the rule's normalised
compliance-framework references (`FrameworkRefs`). Since v0.4.0 each
`RuleOutcome` also carries `Evidence []CheckEvidence`, one entry per
command the check ran, with the exact `Command`, captured
`Stdout`/`Stderr`, `ExitCode`, and `Expected` value: the reproducible
proof behind a verdict, so a consumer can show or re-verify the finding
without re-running the scan. `ScanResult` additionally exposes the
`Capabilities` and `Platform` the scan evaluated against, so the host
context a verdict was computed under is self-describing. The check-only
`ScanResult.Transactions` entries remain for backward compatibility, but
their `committed`/`rolled_back` statuses are a legacy encoding of
compliant/non-compliant; prefer `Outcomes` for an unambiguous verdict.
The doc comments in `api/scan.go` are the authoritative reference.

Loading the rule corpus from a consuming program is public surface too
(package `github.com/Hanalyx/kensa/pkg/kensa`, since v0.3.1):

- `kensa.LoadRules(dir, paths, vars)`—corpus → `[]*api.Rule` ready for
  `Scan`/`Remediate`. Uses the CLI's path-resolution policy (explicit
  dir → explicit files → the `kensa-rules` package's installed corpus at
  `/usr/share/kensa/rules`), and substitutes `{{ name }}` rule templates
  against kensa's embedded defaults merged with the caller's `vars`
  (caller wins). That `vars` map is where an orchestrator injects
  operator-configured values. Strict: a bad file or undefined variable
  fails the load with the file named; nothing is skipped silently.
- `kensa.BuiltInVars()`—the embedded variable defaults (name → value),
  for rendering an operator configuration UI. Review
  `rsyslog_remote_server`, `chrony_ntp_pool`, and `banner_text`: their
  defaults are organisation-specific placeholders.
- `kensa.RuleVariables(dir)`—template variable → rule IDs using it,
  for showing operators what an override affects.

Do not copy the rule files into a consuming repo and do not re-implement
the loader: the corpus ships as the signed `kensa-rules` package, and 23
of the 539 rules are `{{ var }}` templates that only parse through the
substitution chain above.

Constructing a scanner with your own transport is public surface as well
(since v0.3.2): embedders whose credential model the bundled on-disk-key
ssh factory cannot serve (for example, credentials decrypted in memory only)
supply their own `api.TransportFactory`:

- Scan-only (no engine, store, or signer constructed):
  `api.New(api.Config{Scanner: kensa.NewScanner(), TransportFactory: yours})`.
  The backend is stateless: one shared instance is safe for concurrent
  `Scan` calls. `Remediate` on this construction errors by design.
- Full service (remediate, history, transaction log, where a transaction is
  Kensa's four-phase change operation—capture, apply, validate, then commit
  or roll back):
  `kensa.DefaultWithTransportFactory(ctx, storePath, yours, engineOpts...)`.

Exporting a scan as a standards artifact is public surface too (package
`github.com/Hanalyx/kensa/pkg/kensa`, since v0.4.1). A scan's verdicts
and their embedded check evidence convert to an Open Security Controls
Assessment Language (OSCAL) 1.0.6 Assessment Results document with no
shelling out to the CLI:

- `kensa.ExportOSCALScan(result, hostname)` → `[]byte` of OSCAL 1.0.6 AR
  JSON (`kensa.WriteOSCALScan(w, result, hostname)` streams to an
  `io.Writer`). One finding + observation per rule, the `CheckEvidence`
  embedded as relevant-evidence, framework refs as control-ids. The scan
  document is **unsigned** by design; it is derived from the read-only
  `ScanResult`.
- `kensa.ExportOSCAL(envelope)` / `kensa.WriteOSCAL(w, envelope)`—the
  remediation counterpart, rendering a signed `api.EvidenceEnvelope`
  (the audit-truth-of-record a transaction produces) as OSCAL. This path
  is anchored on the envelope's Ed25519 signature.

The byte production lives in `internal/` and is conformance-gated against
the vendored NIST OSCAL 1.0.6 schema; these are the importable entry
points to it.

End-to-end, the whole consumer chain is public:
`kensa.LoadRules(…, operatorVars)` → construct (either form above) →
`Scan` → `ScanResult.Outcomes` → `kensa.ExportOSCALScan(…)`.

---

# Kensa: Troubleshooting
URL: https://www.hanalyx.com/docs/kensa/troubleshooting

_Applies to: Kensa v0.6.0 — last updated 2026-06-22._

Common failure modes, what they look like, and how to clear them. Each section
names the condition first so you can match it to what you are seeing, then the
remedy. When in doubt, run `kensa detect -H <host> --sudo` first; it reports
the host's detected OS and capability set, which explains most "why did this
rule skip / fail" questions.

## Sudo fails without a password

By default `--sudo` runs `sudo -n` (non-interactive). On a host whose sudoers
policy requires a password (itself a common Center for Internet Security (CIS)
and Security Technical Implementation Guide (STIG) control), that fails fast
at connect time rather than mid-scan. Supply the password with
`--sudo-password` (omit the value for a TTY prompt) or the
`KENSA_SUDO_PASSWORD` environment variable; the password is fed over the SSH
session's stdin, never placed in argv or recorded in evidence. If the supplied
password is wrong, the probe reports a rejected sudo password. The alternative
is to configure NOPASSWD sudo for the scan user. Note that `--sudo-password`
requires `--sudo`, and `SUDO_ASKPASS` / `sudo -A` is not supported (it would
need a helper on the target, which the agentless model does not ship).

## Agent bootstrap or GLIBC errors on remediate

`remediate` defaults to agent mode: it ships a small agent binary to the target
and runs the kernel-IO primitives there. If the agent fails to start (for
example a GLIBC-version mismatch on an older target than the build host, or a
spawn/`fork-exec` error), set `KENSA_NO_AGENT=1` to drop to the shell
best-effort path, which uses the host's own tools over the SSH transport. The
shell path is always available and produces byte-identical file writes; you lose
only the direct kernel-IO primitives. The agent must run as root, so pair
agent-mode remediation with `--sudo` (which spawns `sudo kensa agent`); without
root the agent's direct `/proc` and `/etc` writes fail with permission errors.
Service-handler (`service_enabled` / `_disabled` / `_masked`) remediation
additionally needs the systemd helper installed; see
[01-install](/docs/kensa/install) "Service handlers".

## A rule skips when you expected it to run

Two independent gates produce a `SKIP` row:

- **Out of platform.** Kensa compares each rule's `platforms:` block against the
  host's detected OS and skips out-of-platform rules. The shipped corpus targets
  RHEL, so scanning Ubuntu skips the RHEL-only rules; a full Ubuntu scan can
  return everything as `SKIP`. This is faithful behavior, not a bug; those rules
  have no in-platform implementation to run. Platform gating is lenient by
  design: a rule with no `platforms:` block runs everywhere, and a host whose OS
  Kensa cannot detect gates nothing.
- **Capability mismatch.** An implementation's `when:` gate references a
  capability the host lacks (for example no `sshd_config_d`, no `selinux`, or
  no `firewalld`) and no other implementation's gate matches. Run `kensa detect` to see
  the detected capability set, and use `-C KEY=VALUE` on `check` / `detect` to
  override a probe if it is wrong for your host (for example `-C selinux=true`).

## An audit rule only partially restores on rollback

On a host with an immutable audit configuration (`auditd` set to `enabled=2`,
the hardened CIS state), the live audit ruleset is locked until reboot.
`audit_rule_set` rollback restores the rule *file* and reconciles what it can,
but the in-kernel ruleset cannot change until the host reboots; Kensa reports
this as a partial restore. This is a limitation of immutable auditd itself
(`augenrules` has the same constraint), not of Kensa. Reboot to apply the
restored file, or set `enabled=1` if your policy permits a mutable audit config.

## "No rules found" or the wrong rules load

Rule-directory resolution follows a fixed order: an explicit `--rules-dir` wins;
positional `*.yml` paths (or `--rule FILE`) alone skip the directory walk; else
Kensa falls back to `/usr/share/kensa/rules` (where the `kensa-rules` package
installs); else it prints a usage error naming all three fix paths. If you
installed `kensa` without `kensa-rules` (for example with
`--setopt=install_weak_deps=False`), that fallback directory is empty; install
`kensa-rules` or pass `--rules-dir <path>` on every command. A from-source
checkout has the corpus at `rules/`; pass `-r rules`.

## Host-key prompts or "host key changed"

Kensa trusts the host key on first use (TOFU) by default
(`--no-strict-host-keys`), recording it for subsequent connections. If a host's
key legitimately changed (reinstall, new SSH host keys), remove the stale entry
from your `known_hosts` and reconnect to re-pin it. For a security-sensitive
run, pass `--strict-host-keys` to verify the key and reject an unknown or
changed one instead of trusting it. The transport uses system OpenSSH, so
`known_hosts`, jump hosts, and key agents behave exactly as your `ssh` config
defines them.

## Getting more detail

- `kensa detect -H <host> --sudo`—the detected OS and capability set behind
  platform/`when` skips.
- `kensa check ... -v`—expands the compacted PASS list in text output.
- `kensa check ... -o json` or `-o jsonl`—structured per-rule outcomes
  (`pass` / `fail` / `skipped` / `error`) for scripting.
- `kensa plan -H <host> <rule.yml>`—previews a rule's transaction (Kensa's
  four-phase change operation: capture, apply, validate, then commit or roll
  back) without executing it.
- `kensa history` and `kensa rollback <id>`—inspect and reverse a past
  transaction (see [05-rollback-and-history](/docs/kensa/rollback-and-history)).
- A non-zero exit code distinguishes a runtime error (`1`) from a usage error
  (`2`, bad flag / missing argument).

---

# Kensa: Command reference
URL: https://www.hanalyx.com/docs/kensa/reference

_Applies to: Kensa v0.6.0 — last updated 2026-06-22._

This chapter documents every `kensa` command and flag. It is the
exhaustive counterpart to the task-focused chapters: for *how* to scan
and remediate, see [04-scan-and-remediate](/docs/kensa/scan-and-remediate);
for rollback and the transaction log, see
[05-rollback-and-history](/docs/kensa/rollback-and-history); for the mechanism
catalog a rule's remediation can name, see
[10-mechanisms](/docs/kensa/mechanisms).

Every flag below is taken verbatim from `kensa <command> --help`. Run
that yourself any time for the authoritative form on your installed
version.

## Invocation

```
kensa [global flags] <command> [flags]
```

| Command | Purpose |
|---|---|
| `detect` | Probe a host and print its capability set |
| `check` | Run read-only compliance checks (no apply) |
| `remediate` | Apply failing rules to a host |
| `rollback` | Roll back a past transaction by ID |
| `recover` | Compensate transactions interrupted before a terminal status |
| `history` | Query the transaction log |
| `plan` | Preview a rule transaction without executing |
| `mechanisms` | List registered handler mechanisms |
| `coverage` | Alias for `mechanisms` today; reports framework control coverage with `--framework` |
| `list` | Introspection commands (`kensa list frameworks`, `kensa list sessions`) |
| `info` | Rule/control lookup—multi-criteria search over the corpus |
| `diff` | Compare two stored sessions and emit per-rule drift |
| `verify` | Validate the Ed25519 signature on an evidence-envelope JSON file |
| `migrate` | Apply pending schema migrations and backfill legacy sessions |
| `version` | Print version and exit |
| `agent` | Run kensa as a stdio agent on the target host (internal; see below) |

### Global flags

These apply to `kensa` itself, before the subcommand.

| Short | Long | Argument | Default | Meaning |
|---|---|---|---|---|
| `-h` | `--help` | | | Show help and exit |
| `-V` | `--version` | | | Print version and exit |
| `-D` | `--db` | `PATH` | `.kensa/results.db` | SQLite transaction-log path |

Run `kensa <command> --help` for a subcommand's own flags.

### Exit codes

| Code | Meaning |
|---|---|
| `0` | Success (also `--help` / `--version`) |
| `1` | Runtime error |
| `2` | Usage error (bad flag, unknown subcommand, missing required arg) |

`verify` overloads exit `1` to mean *signature INVALID* (tampered,
wrong key, or missing key); see [verify](#verify) for the full table.

### Environment variables

| Variable | Used by | Effect |
|---|---|---|
| `KENSA_SUDO_PASSWORD` | `detect`, `check`, `remediate`, `rollback`, `recover`, `plan` | Sudo password for non-NOPASSWD hosts, as an alternative to `--sudo-password`. Never placed in argv or recorded evidence. |
| `KENSA_NO_AGENT` | `remediate` | Set to `1` to disable agent mode and fall back to shell-best-effort for the kernel-atomic file handlers. |
| `KENSA_SIGNING_KEY` | `remediate` | Path to the Ed25519 `.priv` file for a stable signer identity; without it an ephemeral key is generated per process (see [01-install](/docs/kensa/install)). |
| `KENSA_CONFIG_DIR`, `XDG_CONFIG_HOME`, `HOME` | `verify` | Resolve the default trust directory for public keys (in that priority order). |

## Common option groups

Several commands share the same target, rule, and output option groups.
They are defined once here and referenced by each command.

### Target options

For commands that connect to a host over SSH.

| Short | Long | Argument | Default | Meaning |
|---|---|---|---|---|
| `-H` | `--host` | `string` | | Target hostname (required unless noted) |
| `-u` | `--user` | `string` | current user | SSH user |
| `-k` | `--key` | `string` | | SSH private key path |
| `-p` | `--password` | `string[="<prompt>"]` | | SSH password; omit the value for a TTY prompt. The literal `<prompt>` is reserved. |
| `-P` | `--port` | `int` | `22` | SSH port |
| | `--sudo` | | | Wrap commands in sudo |
| | `--sudo-password` | `string[="<prompt>"]` | | Sudo password for non-NOPASSWD hosts; omit the value for a TTY prompt, or set `KENSA_SUDO_PASSWORD`. Requires `--sudo`. |
| | `--strict-host-keys` | | | Verify SSH host keys; reject unknown (overrides `--no-strict-host-keys`) |
| | `--no-strict-host-keys` | | default today | Trust on first use (explicit form for a future config-file override) |
| `-C` | `--capability` | `stringArray` | | Override a detected capability `KEY=VALUE`; repeatable (for example `-C apparmor=true -C selinux=false`). On duplicate keys, last value wins. |

Sudo behaviour (passwordless vs. password, the stdin mechanism, and the
fail-fast probe) is covered in [04-scan-and-remediate](/docs/kensa/scan-and-remediate).
Not every command exposes every row: `rollback` and `recover` omit
`-p/--password`, and `recover` omits `-C/--capability`. Each command's
section lists exactly what it carries.

### Rule options

For commands that load and filter the rule corpus (`check`, `remediate`).

| Short | Long | Argument | Meaning |
|---|---|---|---|
| `-r` | `--rules-dir` | `string` | Directory to scan for `*.yml` rule files |
| | `--rule` | `stringArray` | Load a single rule YAML file (strict—parse errors fail the command); long-only, repeatable, additive with `--rules-dir` and positional `*.yml` args |
| `-s` | `--severity` | `stringArray` | Filter by severity, repeatable (`-s critical -s high`); choices: `critical\|high\|medium\|low` |
| `-t` | `--tag` | `stringArray` | Filter by tag, repeatable; matches rules whose `tags:` array contains any value |
| `-c` | `--category` | `string` | Filter by category (`-c access-control`); single value, NOT repeatable (later `-c` overrides earlier) |
| `-f` | `--framework` | `cis-rhel9` | Filter to rules mapping a control under FRAMEWORK; single value. Hyphen and underscore interchangeable (`-f cis-rhel9 == -f cis_rhel9`). |
| | `--control` | `stringArray` | Filter by `FRAMEWORK:CONTROL` (`--control cis-rhel9:5.1.12`); repeatable, OR across values; framework portion accepts hyphen or underscore |
| `-x` | `--var` | `stringArray` | Override a rule variable, `KEY=VALUE`; repeatable. Wins over `--config-dir`/`defaults.yml`. **VALUE is spliced literally into rule YAML and may reach shell commands. Pass only trusted input.** |
| | `--config-dir` | `string` | Directory holding `defaults.yml` (variable defaults source). Only `defaults.yml` is read today. |

Positional `rule.yml ...` arguments are additive with `--rules-dir` and
`--rule`. Default-path resolution when no rules are specified is covered
in [06-rule-authoring](/docs/kensa/rule-authoring) and the chapter on install
(`/usr/share/kensa/rules`).

### Output options

Most commands take a subset of these.

| Short | Long | Argument | Default | Meaning |
|---|---|---|---|---|
| | `--format` | `string` | `table` / `text` | Output format; the exact choices vary per command (see each section). Deprecated on the host commands in favour of `--output`. |
| `-o` | `--output` | `strings` | | Output destination `FORMAT[:PATH]`, repeatable (for example `-o json -o csv:results.csv`) |
| `-q` | `--quiet` | | | Suppress default output (errors still go to stderr) |
| `-v` | `--verbose` | | | (`check` only) Expand the compacted PASSED list; text format only |

---

## detect

Probe a host and print its capability set. Read-only; no mutations.

```
kensa detect [flags]
```

Carries the full [target options](#target-options) group (`-H` required).

**Output options:**

| Long | Argument | Default | Meaning |
|---|---|---|---|
| `--format` | `string` | `table` | Output format: `table` or `json` (deprecated; use `--output`) |
| `-o, --output` | `strings` | | Output destination `FORMAT[:PATH]`, repeatable |
| `-q, --quiet` | | | Suppress default output |

```bash
kensa detect -H 192.168.1.211 -u owadmin --sudo
kensa detect --host web-01 --user admin --format json
```

## check

Run read-only compliance checks against one host or an inventory. See
[04-scan-and-remediate](/docs/kensa/scan-and-remediate) for the workflow.

```
kensa check [flags] [rule.yml ...]
```

Carries the full [target options](#target-options) group (`-H` required
unless `--inventory` is given) plus the inventory flags below, the full
[rule options](#rule-options) group, and the output options.

**Inventory options (additional target options):**

| Short | Long | Argument | Default | Meaning |
|---|---|---|---|---|
| `-i` | `--inventory` | `string` | | Ansible-style `inventory.ini` for multi-host check |
| `-l` | `--limit` | `string` | | Limit inventory hosts to a glob/group pattern (ansible `--limit` semantics) |
| `-w` | `--workers` | `int` | `1` | Concurrent SSH connections in `--inventory` mode (1–50; 1 = sequential) |

**Output options:**

| Long | Argument | Default | Meaning |
|---|---|---|---|
| `--format` | `string` | `table` | `table`, `json`, or `jsonl` (deprecated; use `--output`) |
| `-o, --output` | `strings` | | Output destination `FORMAT[:PATH]`, repeatable |
| `-q, --quiet` | | | Suppress default output |
| `-v, --verbose` | | | Expand the compacted PASSED list (text format only) |

**Other options:**

| Long | Default | Meaning |
|---|---|---|
| `--store` | off | Persist the scan as a session + transactions record in the SQLite store (`check` is read-only by default) |

```bash
kensa check -H 192.168.1.211 -u owadmin --sudo -r /path/to/rules
kensa check --inventory hosts.ini -w 10 --sudo -r /path/to/rules
kensa check -H 192.168.1.211 -s critical -s high -r /path/to/rules
kensa check -H 192.168.1.211 -f cis-rhel9 --control cis_rhel9:5.1.12 -r /path/to/rules
kensa check -H web-01 -u admin --sudo -o jsonl rule1.yml rule2.yml
```

## remediate

Apply failing rules to a host. Each rule runs as a four-phase atomic
transaction; on validation failure the engine rolls back to the captured
pre-state. See [04-scan-and-remediate](/docs/kensa/scan-and-remediate).

```
kensa remediate [flags] [rule.yml ...]
```

Carries the full [target options](#target-options) group (`-H`
required), the full [rule options](#rule-options) group, and the output
options below.

**Output options:**

| Long | Argument | Default | Meaning |
|---|---|---|---|
| `--format` | `string` | `table` | `table` or `json` (deprecated; use `--output`) |
| `-o, --output` | `strings` | | Output destination `FORMAT[:PATH]`, repeatable |
| `--oscal` | `string` | | Write Open Security Controls Assessment Language (OSCAL) Assessment Results to this file (deprecated; use `--output oscal:PATH`) |
| `-q, --quiet` | | | Suppress default output |

Set `KENSA_NO_AGENT=1` to disable agent mode and fall back to
shell-best-effort for the kernel-atomic file handlers. Set
`KENSA_SIGNING_KEY` for a stable evidence-signer identity.

```bash
kensa remediate -H 192.168.1.211 -u owadmin --sudo -r /path/to/rules
kensa remediate -H 192.168.1.211 -s critical -t pci -r /path/to/rules
kensa remediate -H 192.168.1.211 -f cis-rhel9 --control cis_rhel9:5.1.12 -r /path/to/rules
kensa remediate -H web-01 -u admin --sudo -o json -o oscal:/tmp/results.oscal.json
```

## rollback

Roll back transactions using captured pre-state. Pick exactly one mode.
See [05-rollback-and-history](/docs/kensa/rollback-and-history).

```
kensa rollback [MODE] [flags]
```

**Mode (pick one):**

| Short | Long | Argument | Meaning |
|---|---|---|---|
| | `--list` | | List rollback-able sessions (read-only) |
| | `--info` | `SESSION_ID` | Show session detail (txns + statuses) |
| | `--start` | `SESSION_ID` | Execute rollback for every committed transaction in the session (needs `--host`) |
| `-T` | `--txn` | `TXN_UUID` | Legacy: single-transaction rollback (needs `--host`) |
| | `--detail` | | Modifier: per-step breakdown that composes with `--list` and `--info` (not `--start` or `--txn`) |

Find session UUIDs first with `kensa list sessions`.

**Target options** (required for `--start` and `--txn`): `-H, --host`
(required for those modes), `-u, --user`, `-k, --key`, `-P, --port`,
`--sudo`, `--sudo-password`, `--strict-host-keys`,
`--no-strict-host-keys`. This command does **not** carry `-p/--password`
or `-C/--capability`.

**Output options:**

| Long | Argument | Default | Meaning |
|---|---|---|---|
| `--format` | `string` | `text` | Output format: `text` or `json` |
| `-q, --quiet` | | | Suppress default output |

```bash
kensa rollback --list
kensa rollback --info 8c3a1e2b-... --detail
kensa rollback --start 8c3a1e2b-... -H 192.168.1.211 -u owadmin --sudo
kensa rollback --txn 9d4b... -H 192.168.1.211 -u owadmin --sudo   # legacy
```

## recover

Compensate transactions interrupted before they reached a terminal
status, using the durable crash-recovery journal. Each open transaction
is rolled back from its captured pre-state and recorded as recovered.
Holds an exclusive recover lock, so it never races a live kensa on the
same store. Run it after a crash, when no live kensa is operating the
host. See [05-rollback-and-history](/docs/kensa/rollback-and-history) and the
crash-recovery note in [10-mechanisms](/docs/kensa/mechanisms).

```
kensa recover [flags]
```

| Short | Long | Argument | Default | Meaning |
|---|---|---|---|---|
| `-H` | `--host` | `string` | | Scope recovery to this host (also the SSH target; required) |
| `-u` | `--user` | `string` | current user | SSH user |
| `-P` | `--port` | `int` | `22` | SSH port |
| | `--key` | `string` | | SSH private key path |
| | `--sudo` | | | Wrap commands in sudo |
| | `--sudo-password` | `string` | | Sudo password for non-NOPASSWD hosts |
| | `--strict-host-keys` | | | Verify SSH host keys; reject unknown |
| `-q` | `--quiet` | | | Suppress default output |
| `-D` | `--db` | `string` | `.kensa/results.db` | SQLite transaction-log path |

## history

Query the transaction log. Without filters, lists recent transactions.
See [05-rollback-and-history](/docs/kensa/rollback-and-history).

```
kensa history [flags]
```

| Short | Long | Argument | Default | Meaning |
|---|---|---|---|---|
| `-h` | `--help` | | | Show help and exit |
| `-H` | `--host` | `string` | | Filter by host ID |
| `-R` | `--rule` | `string` | | Filter by rule ID |
| `-S` | `--since` | `string` | | Filter since a duration (for example `24h`) or RFC3339 time |
| `-n` | `--limit` | `int` | `50` | Maximum rows to return |
| | `--format` | `string` | `table` | `table`, `json`, or `jsonl` (jsonl is transaction-list only) |
| `-T` | `--txn` | `string` | | Get a single transaction by UUID |
| `-a` | `--aggregate` | `string` | | Aggregate key: `by_host`, `by_rule`, `by_framework_control` |
| | `--stats` | | | Print summary stats (sessions, transactions, by status / severity / host) and exit |
| | `--prune` | `int` | | Delete sessions and cascade older than N days (destructive; long-only) |
| | `--force` | | | Skip the confirmation prompt for `--prune` (required in non-interactive runs) |
| `-q` | `--quiet` | | | Suppress default output |

```bash
kensa history                                  # 50 most recent
kensa history -n 200 --format jsonl | jq -c .  # streamable JSON Lines
kensa history -H 192.168.1.211 -S 24h          # one host, last 24h
kensa history -a by_host -S 7d                 # 7-day posture per host
kensa history --prune 30 --force               # non-interactive prune
```

## plan

Preview a rule transaction without executing it. Returns a structured
plan with captured pre-state, apply steps, validators, rollback plan,
and warnings.

```
kensa plan [flags] rule.yml
```

| Short | Long | Argument | Default | Meaning |
|---|---|---|---|---|
| `-h` | `--help` | | | Show help and exit |
| `-H` | `--host` | `string` | | Target hostname (required) |
| `-u` | `--user` | `string` | current user | SSH user |
| `-P` | `--port` | `int` | `22` | SSH port |
| `-k` | `--key` | `string` | | SSH private key path |
| `-p` | `--password` | `string[="<prompt>"]` | | SSH password; omit value for a TTY prompt |
| | `--strict-host-keys` | | | Verify SSH host keys; reject unknown |
| | `--no-strict-host-keys` | | default today | Trust on first use |
| | `--sudo` | | | Wrap commands in sudo |
| | `--sudo-password` | `string[="<prompt>"]` | | Sudo password for non-NOPASSWD hosts; requires `--sudo` |
| | `--format` | `string` | `text` | Output format: `text`, `markdown`, `json`, `plain` |
| `-q` | `--quiet` | | | Suppress default output |

```bash
kensa plan -H 192.168.1.211 -u owadmin --sudo --format markdown rule.yml
```

## mechanisms

List every handler mechanism registered with the kensa engine, marked
capturable (participates in atomic transactions) or non-capturable
(`transactional: false` escape hatch). See [10-mechanisms](/docs/kensa/mechanisms)
for the catalog and reversal levels.

```
kensa mechanisms [flags]
```

| Short | Long | Meaning |
|---|---|---|
| `-h` | `--help` | Show help and exit |

## coverage

An alias for `mechanisms` today; printing a deprecation warning on
stderr. With `--framework`, it reports which controls in the named
framework are referenced by rules in the loaded corpus (the rule IDs per
control). The report is **numerator-only** (controls with rules, not
the framework's full control set) because kensa does not bundle an
external control catalog yet.

```
kensa coverage [flags]
kensa coverage --framework FRAMEWORK --rules-dir DIR [flags]
```

**Without `--framework`** (alias mode): only `-h, --help`.

**With `--framework`** (coverage report):

| Short | Long | Argument | Default | Meaning |
|---|---|---|---|---|
| `-h` | `--help` | | | Show help and exit |
| `-f` | `--framework` | `cis-rhel9` | | Filter rules to those mapping a control under FRAMEWORK; single value, hyphen/underscore interchangeable |
| `-r` | `--rules-dir` | `string` | | Directory of rule YAMLs to scan (required) |
| | `--format` | `string` | `text` | Output format: `text` or `json` |
| | `--full` | | | In text output, show every rule ID per control (default: truncate to first 3) |
| `-q` | `--quiet` | | | Suppress default output |

```bash
kensa coverage --framework cis_rhel9 --rules-dir /path/to/rules
kensa coverage -f nist_800_53 -r /path/to/rules --format json
kensa coverage -f cis_rhel9 -r /path/to/rules --full
```

> Migrate scripts that rely on the mechanism listing to `kensa mechanisms`
> before upgrading: `kensa coverage` changes meaning in v0.2.0.

## list

Introspection commands for the rule corpus and the transaction store.

```
kensa list <subject> [flags]
```

| Subject | Purpose |
|---|---|
| `frameworks` | Per-framework control + rule counts (requires `--rules-dir DIR`) |
| `sessions` | List recent sessions in the transaction store (with IDs for `kensa diff`) |

### list frameworks

Lists every `framework_id` in the loaded corpus with the count of
distinct controls referenced and distinct rules referencing each. Counts
are DISTINCT, not entry-counts.

```
kensa list frameworks --rules-dir DIR [flags]
```

| Short | Long | Argument | Default | Meaning |
|---|---|---|---|---|
| `-h` | `--help` | | | Show help and exit |
| `-r` | `--rules-dir` | `string` | | Directory of rule YAMLs to scan (required) |
| | `--format` | `string` | `text` | Output format: `text` or `json` |
| `-q` | `--quiet` | | | Suppress default output |

### list sessions

Lists recent sessions in the transaction store. The `session_id` column
is the UUID needed by `kensa diff`.

```
kensa list sessions [flags]
```

| Short | Long | Argument | Default | Meaning |
|---|---|---|---|---|
| `-h` | `--help` | | | Show help and exit |
| `-H` | `--host` | `string` | | Filter by hostname |
| `-n` | `--limit` | `int` | `20` | Maximum sessions to show (0 = unlimited) |
| | `--format` | `string` | `text` | Output format: `text`, `json`, or `jsonl` |
| `-q` | `--quiet` | | | Suppress default output |

## info

Multi-criteria lookup over a loaded rule corpus. Pick one mode; filters
compose with a positional `QUERY` (case-insensitive substring search over
title + description). `--rules-dir` is required.

```
kensa info [MODE] --rules-dir DIR [filters] [QUERY]
```

**Mode (pick one):**

| Short | Long | Argument | Meaning |
|---|---|---|---|
| | `--rule` | `string` | Show details for a single rule by ID (long-only; `-r` is `--rules-dir`) |
| | `--control` | `string` | Show rules mapping `FRAMEWORK:ID` (for example `cis_rhel9:5.1.12`) |
| `-L` | `--list-controls` | `string` | List every control referenced under FRAMEWORK with rule counts |
| `-r` | `--rules-dir` | `string` | Directory of rule YAMLs to scan (required) |

**Filter options** (compose with QUERY; the framework shortcuts also
narrow within `--rule`/`--control`):

| Long | Argument | Meaning |
|---|---|---|
| `--cis` | | Filter to the Center for Internet Security (CIS) family (`cis_rhel8`, `cis_rhel9`, `cis_rhel10`) |
| `--stig` | | Filter to the Security Technical Implementation Guide (STIG) family (`stig_rhel8`, `stig_rhel9`, `stig_rhel10`) |
| `--nist` | | Filter to the NIST 800-53 family (`nist_800_53`; not RHEL-versioned, so does NOT compose with `--rhel`) |
| `--rhel` | `int` | Filter by RHEL version (8, 9, or 10); composes with `--cis`/`--stig` (not `--nist`) |

`--cis`, `--stig`, and `--nist` are mutually exclusive.

**Output options:**

| Long | Argument | Default | Meaning |
|---|---|---|---|
| `--format` | `string` | `text` | `text`, `json`, or `jsonl` (jsonl is QUERY mode only) |
| `--limit` | `int` | `100` | Cap text output rows (search + list-controls modes); 0 = unlimited |
| `-q, --quiet` | | | Suppress default output |

```bash
kensa info ssh --rules-dir /path/to/rules                # substring search
kensa info ssh --cis --rhel 9 --rules-dir /path/to/rules # SSH rules in CIS RHEL9
kensa info --rule sysctl-ip-forward-disabled --rules-dir /path/to/rules
kensa info --control cis_rhel9:5.1.12 --rules-dir /path/to/rules
kensa info --list-controls cis_rhel9 --rules-dir /path/to/rules
kensa info file --rules-dir /path/to/rules --limit 0     # no truncation
```

## diff

Compare two stored sessions and emit the per-rule drift report: status
changes, rules added (in `SESSION_ID_2` only), and rules removed (in
`SESSION_ID_1` only). The `from → to` direction follows git-diff
convention, so `SESSION_ID_1` is the earlier snapshot. Comparing across
hostnames is allowed (a stderr note discloses the cross-host scope). Find
session IDs with `kensa list sessions`.

```
kensa diff SESSION_ID_1 SESSION_ID_2 [flags]
```

| Short | Long | Argument | Default | Meaning |
|---|---|---|---|---|
| `-h` | `--help` | | | Show help and exit |
| | `--format` | `string` | `text` | Output format: `text` or `json` |
| | `--show-unchanged` | | | Include rules whose status is identical between the two sessions |
| `-q` | `--quiet` | | | Suppress default output |

```bash
kensa list sessions                       # find session IDs first
kensa diff <id1> <id2>                     # compact drift report
kensa diff <id1> <id2> --show-unchanged    # include unchanged rules
kensa diff <id1> <id2> --format json       # programmatic output
```

## verify

Verify the Ed25519 signature on a kensa evidence-envelope JSON file. The
public key is looked up by the envelope's `signing_key_id` in a trust
directory (a directory of `.pub` files produced by `kensa-keygen`).

```
kensa verify <evidence-file> [flags]
```

Default trust directory, in priority order:
`$KENSA_CONFIG_DIR/keys/`, then `$XDG_CONFIG_HOME/kensa/keys/`, then
`$HOME/.config/kensa/keys/`. Override with `--trust-dir`.

| Short | Long | Argument | Default | Meaning |
|---|---|---|---|---|
| `-h` | `--help` | | | Show help and exit |
| | `--trust-dir` | `string` | matches `kensa-keygen` output dir | Directory of `.pub` files to look up `signing_key_id` |
| | `--format` | `string` | `text` | Output format: `text` or `json` |
| `-q` | `--quiet` | | | Suppress default output |

**Exit codes (verify-specific):**

| Code | Meaning |
|---|---|
| `0` | Signature is valid (envelope is authentic) |
| `1` | Signature is **INVALID** (tampered, wrong key, missing key, unknown schema version) |
| `2` | Usage error (missing file, bad flag, malformed JSON) |

Critical failure modes that exit `1` (not `2`): signature mismatch
(tampered after signing), unknown key (`signing_key_id` has no `.pub` in
the trust dir), wrong key (signature doesn't match the found `.pub`), and
unknown schema version (envelope from a future kensa).

Success warnings that still exit `0`: `signed_by_rotated_key` (authentic
but signed by a rotated-out key) and `signing_key_id_mismatch` (the
signature is real but the matched key's id disagrees with the envelope's
`signing_key_id`; investigate).

```bash
kensa verify evidence.json
kensa verify evidence.json --trust-dir /etc/kensa/keys
kensa verify evidence.json --format json | jq -r .valid
```

## migrate

Apply pending schema migrations to the SQLite store and backfill
synthetic sessions for pre-Phase-4 transactions. Idempotent: a second
run finds no work and exits `0`. Pre-Phase-4 transactions (rows whose
`session_id` is NULL) are grouped one synthetic session per `host_id`,
with subcommand `legacy-backfill` so they're distinguishable from real
sessions in later `kensa history` / `kensa diff` runs.

```
kensa migrate [flags]
```

| Short | Long | Argument | Meaning |
|---|---|---|---|
| `-h` | `--help` | | Show help and exit |
| | `--db` | `string` | Override the default store path |
| `-q` | `--quiet` | | Suppress the migration summary (errors still go to stderr) |

```bash
kensa migrate
kensa migrate --db /var/lib/kensa/results.db
kensa migrate --quiet
```

## version

Print the kensa binary version. The top-level `--version` flag is the
canonical GNU/POSIX form; this subcommand is preserved for backward
compatibility and is planned for removal in v0.2.0.

```
kensa version
```

| Short | Long | Meaning |
|---|---|---|
| `-h` | `--help` | Show help and exit |

## agent

`kensa agent` runs kensa as a stdio agent on the *target* host. It is an
internal command; `remediate` spawns it on the target over SSH (as root
via `--sudo`) to drive the kernel-IO primitives directly; you do not
invoke it by hand. Run `kensa agent --help` on the target for its
protocol flags. See [10-mechanisms](/docs/kensa/mechanisms) for which handlers
use the agent path and `KENSA_NO_AGENT=1` to opt out.

## Next

That completes the operator guide. For the rule-authoring schema and
mechanism details referenced throughout, return to
[06-rule-authoring](/docs/kensa/rule-authoring) and
[10-mechanisms](/docs/kensa/mechanisms).

---

# Kensa: Mechanisms reference
URL: https://www.hanalyx.com/docs/kensa/mechanisms

**Document version:** 1.1 · **Last updated:** 2026-06-21

A *mechanism* is the named action a rule's remediation runs to change a host,
such as `sysctl_set`, `service_enabled`, or `file_content`. You set it in a
rule's `remediation` block. Each mechanism is backed by a handler in Kensa that
knows how to apply the change and, for most mechanisms, how to undo it.

Kensa runs every change as a *transaction*: capture the prior state, apply the
change, validate it, then commit or roll back. Two handler roles matter here:

- A *capture handler* records the host's exact prior state before the change.
- A *rollback handler* restores that prior state if the transaction fails.

A mechanism that has both is *capturable*—Kensa can reverse it. A mechanism
without them is *non-capturable*: Kensa applies and audits the change but can't
roll it back, so you mark its rule `transactional: false`.

This page lists every shipped mechanism, where it runs, and what reversal you
get. For the guarantee itself, see
[Rollback and history](/docs/kensa/rollback-and-history); for the rule fields, see
[Rule authoring](/docs/kensa/rule-authoring).

## Operating system support

Each mechanism depends on a tool or kernel feature that must be present on the
host. Kensa probes for it at runtime and skips a rule whose platform or
capability the host does not meet, so a mechanism never runs where its backing
tool is missing. The **Runs on** column below names where each mechanism's
backing tool exists:

- **Any Linux**—kernel or core-filesystem features present on every supported
  distribution.
- **RHEL 8–10**—the tool ships with Red Hat Enterprise Linux 8, 9, and 10
  (and compatible rebuilds and Fedora).
- **Debian, Ubuntu**—the tool is `apt`-family.
- A named subsystem (systemd, SELinux, GRUB, GNOME)—the host must run it.

Kensa is tested on RHEL 8, 9, and 10 and on Ubuntu. The shipped rule corpus
currently targets RHEL; running it against Ubuntu skips the RHEL-only rules.

## Reversal levels

| Level | What it means |
|---|---|
| **Atomic** | The change lands completely or not at all, and rollback restores the prior state byte-for-byte. |
| **Reversible** | Rollback restores the captured prior state and verifies it. A runtime aspect may need a restart or reboot; the limit is noted, and rollback reports a partial restore when it applies. |
| **Best-effort** | Rollback restores the prior state through the host's own tool (`dnf`, `apt`, `augenrules`). It depends on that tool and on the host's policy. |
| **Staged** | The change is pending until the next reboot. A failed boot reverts automatically. |
| **None** | Kensa records the change for audit but can't roll it back. The rule is `transactional: false`. |

## Files

| Mechanism | What it changes | Runs on | Reversal |
|---|---|---|---|
| `file_content` | A file's content and attributes | Any Linux | Atomic |
| `file_absent` | Removes a file | Any Linux | Atomic |
| `config_set` | A key's value in a config file | Any Linux | Atomic |
| `config_set_dropin` | A key in a drop-in config file | Any Linux | Atomic |
| `file_permissions` | A file's mode, owner, group, and SELinux context | Any Linux | Atomic |
| `config_append` | Appends a line to a config file | Any Linux | Best-effort |

## Services

| Mechanism | What it changes | Runs on | Reversal |
|---|---|---|---|
| `service_enabled` | Enables and starts a systemd unit | systemd (RHEL 8–10, Ubuntu) | Reversible |
| `service_disabled` | Disables and stops a systemd unit | systemd (RHEL 8–10, Ubuntu) | Reversible |
| `service_masked` | Masks and stops a systemd unit | systemd (RHEL 8–10, Ubuntu) | Reversible |

## Kernel runtime and persistence

| Mechanism | What it changes | Runs on | Reversal |
|---|---|---|---|
| `sysctl_set` | A kernel parameter (runtime and drop-in) | Any Linux | Reversible |
| `mount_option_set` | A mount option in `fstab`, then remounts | Any Linux | Reversible. Rollback restores the `fstab` line, remounts, and verifies the live options match; a mismatch is reported as a partial restore. |
| `selinux_boolean_set` | A persistent SELinux boolean | SELinux (RHEL 8–10) | Reversible |
| `kernel_module_disable` | Blacklists a kernel module | Any Linux | Reversible. If the module was loaded, rollback re-loads it and verifies; a module that can't re-load (in use, or boot-time only) is reported as a partial restore needing a reboot. |
| `audit_rule_set` | Loads an audit rule and persists it | RHEL 8–10; Ubuntu with `auditd` | Best-effort. Rollback restores the rule file and reconciles the live rules; on an immutable audit config the live ruleset is locked until reboot, which rollback reports as a partial restore. |

## Packages

| Mechanism | What it changes | Runs on | Reversal |
|---|---|---|---|
| `package_present` | Installs a package with `dnf` | RHEL 8–10 | Best-effort. Rollback removes the package, which removes its dependents. |
| `package_absent` | Removes a package with `dnf` | RHEL 8–10 | Best-effort. Rollback reinstalls; the version may differ from the prior one. |
| `apt_present` | Installs a package with `apt` | Debian, Ubuntu | Best-effort. Rollback removes the package. |
| `apt_absent` | Removes a package with `apt` | Debian, Ubuntu | Best-effort. Rollback reinstalls; the version may differ. |

## Authentication and policy

| Mechanism | What it changes | Runs on | Reversal |
|---|---|---|---|
| `pam_module_configure` | A PAM configuration file | RHEL 8–10, Ubuntu | Best-effort |
| `pam_module_arg` | An argument on a PAM module line | RHEL 8–10, Ubuntu | Best-effort |
| `authselect_feature_enable` | An `authselect` feature | RHEL 8–10 | Best-effort |
| `dconf_set` | A system `dconf` key, optionally locked | GNOME (RHEL 8–10, Ubuntu) | Best-effort |
| `crypto_policy_set` | The system-wide crypto policy | RHEL 8–10 | Reversible. Rollback restores and verifies the policy; services already running keep their startup crypto until restarted (for example `systemctl restart sshd`). |
| `cron_job` | A cron file | RHEL 8–10, Ubuntu | Best-effort |

## Boot parameters

| Mechanism | What it changes | Runs on | Reversal |
|---|---|---|---|
| `grub_parameter_set` | Adds a kernel boot parameter | GRUB (RHEL 8–10, Ubuntu) | Staged. A trial boot promotes the change; a failed boot reverts to the prior default. |
| `grub_parameter_remove` | Removes a kernel boot parameter | GRUB (RHEL 8–10, Ubuntu) | Staged |

Kensa never edits the saved boot default directly. It stages the change through
a one-shot trial boot, so a host that fails to boot reverts on its own. Kensa
does not reboot the host; the change is pending until you do.

## Non-reversible

| Mechanism | What it changes | Runs on | Reversal |
|---|---|---|---|
| `command_exec` | Runs an arbitrary command | Any Linux | None |
| `crypto_policy_subpolicy` | Applies a crypto subpolicy | RHEL 8–10 | None |
| `manual` | Marks a step for a human to perform | Any Linux | None |

These mechanisms run but Kensa can't undo them, so their rules are
`transactional: false`. Kensa records each one in the transaction log for audit,
but it isn't covered by the rollback guarantee. Use them only when no
reversible mechanism fits.

## Crash recovery

For a capturable mechanism, the prior state is written to durable storage
before any host change. If a Kensa process is interrupted mid-transaction, run
`kensa recover -H <host>` to roll the host back to the captured prior state.
Recovery reverses every mechanism in this page that is marked Atomic,
Reversible, or Best-effort; it can't reverse a `transactional: false` step,
because none was captured.

---

# Kensa: Release notes
URL: https://www.hanalyx.com/docs/kensa/release

All notable user-visible changes to kensa are documented here.
Format follows [Keep a Changelog](https://keepachangelog.com/) loosely:
sections grouped by category (Added / Changed / Deprecated / Removed
/ Fixed / Security) under each release heading. Pre-1.0 we ship
unreleased changes under `## Unreleased` and stamp them at tag time.

The CLI is governed by GNU/POSIX conventions. Long-form flags are
the canonical names; short forms are listed in `cmd/kensa/flags.go`.

## Unreleased

(no changes yet)

## v0.6.0 — 2026-06-23

MINOR — the frozen `api/` surface gains additive crash-recovery types
(verified no breaking change). This line ships the **atomicity engine**: a
remediation is now a verified transaction that is durably journaled, re-measured
after apply, reversed on crash, and — on the agent path — funnelled through
kernel-atomic primitives behind a pre-commit rollback-completeness gate.

### Added

- **Crash-recovery intent journal + `kensa recover`** (#126/#127). Each
  transaction's intent + captured pre-state is written to a durable SQLite
  journal (fsync, `synchronous=FULL`) BEFORE any host mutation; an interrupted
  transaction is reverse-replayed to its pre-state by `kensa recover` (exclusive
  `recover.lock`). No state is left *permanently* half-applied.
- **Mandatory post-apply VALIDATE re-check** (#122) + **verdict-computed
  rollback status** (#121): the engine re-reads the host after apply and labels
  `committed` only on machine-verified success, `rolled_back` only on verified
  restoration.
- **Post-state recapture into the signed evidence envelope** (#125) — the
  envelope carries a re-measured `PostStateBundle`.
- **Footprint pre-commit gate** (#133–#137): in agent mode the engine records
  every filesystem mutation an apply funnels and asserts `observed ⊆ captured`
  before commit, plus a pre-apply restorability probe that refuses to mutate a
  captured resource that is immutable (`chattr +i`).
- **Agent-mode kernel-IO** for runtime/file handlers (#107–#114, #130,
  #139/#140/#143/#144): `sysctl_set`, `mount_option_set`, `kernel_module_disable`,
  `audit_rule_set` (AUDIT netlink), `dconf_set`, the systemd service handlers,
  and `config_append` / `cron_job` / `pam_module_arg`. Each keeps a byte-identical
  shell fallback for the direct-SSH path.
- **`api/` (additive):** `JournalEntry`; `TransactionResult.RollbackResults`,
  `.HostUnchanged`, `.Decision`; `Transaction.Check`;
  `EvidenceEnvelope.RollbackResults`; `TransactionStatus` values `recovered`
  and `rollback_failed`.
- **Honest SKIP for GDM rules** (#138): a `dconf`/`gdm` capability gate so
  GNOME-display rules SKIP truthfully on servers without GDM; duplicate
  GDM-removed control reconciled (corpus 539 → 538).
- **Operator guide** completed: chapters 02–09 fully written and validated
  against the CLI, plus the mechanisms reference (10).

### Changed

- Runtime/file handlers are now **dual-path** (agent kernel-IO vs shell
  fallback); both write byte-identical files and record an identical pre-state,
  so capture/rollback are path-agnostic.
- `pam_module_arg` capture switched to **whole-file content** (byte-perfect
  rollback; legacy grep-snapshot pre-states still supported).
- The engine emits a per-phase `AUDIT_USER` record into auditd (best-effort,
  never fails a transaction).

### Fixed

- **Footprint Recorder now forwards `systemd.Transport` + `auditnl.AuditTransport`**
  (#141): a Phase-6 regression had silently routed the service and
  `audit_rule_set` handlers to their shell fallback on the agent path;
  compile-time assertions prevent recurrence.
- `pam_module_arg` / `cron_job` agent path now **preserves the existing file
  mode** instead of forcing 0644 (a non-0644 PAM include is no longer widened on
  apply or rollback).
- `kernelio` drop-in writes are create-or-replace (#115, live-caught); systemd
  helper invoked directly when the agent is already root (#116); service handlers
  fall back to shell when the helper cannot be invoked, not only when absent (#117).

### Security

- **Shell single-quote breakout** in the `sed` programs of `pam_module_arg` /
  `config_append` shell paths (pre-existing; SSH/agentless path only) fixed —
  the sed program is now `shellEscape`-wrapped.
- `mount_option_set` shell capture now `regexp.QuoteMeta`s the mount point
  (prevents wrong-fstab-line capture/rollback).
- Pre-apply restorability probe refuses to apply when a captured resource is
  immutable — a transaction whose rollback is impossible is refused, not
  committed unrollbackable.

> Known follow-ups tracked in `docs/roadmap/STATUS.md` (recover-vs-live-engine
> shared lock not yet wired; journal `cursor` write-ahead unwired).

## v0.5.2 — 2026-06-19

Closes a class of *silent* rule-engine drift: the rule schema, the
corpus, the check/remediation engine, and the validator could disagree
and a rule would still load and emit a confident-but-wrong compliance
verdict. This release converts each axis into a code-enforced gate that
fails CI on regression, fixes the one live-caught false verdict the new
gates exposed, and corrects a CLI-output mislabel. PATCH bump: all
changes are in `internal/`, `cmd/`, `specs/`, and `rules/` — the frozen
`api/` surface is untouched.

### Added

- **Check-method parameter contract (closed-world).** Every check
  method now declares the params it reads (`internal/check.CheckContracts`,
  the SSOT derived from the check implementations, not the prose schema);
  the rule validator rejects a check that declares a param the method
  never reads, so a rule's intent can no longer be silently dropped. A
  ratcheting allowlist tracks the remaining known-non-conforming corpus
  rules and may only shrink (#97).
- **Value-domain validation.** Param *values* are checked against their
  domain at rule load — comparator ∈ {`==`,`!=`,`<`,`<=`,`>`,`>=`},
  enum-valued params (apparmor/kernel_module/package state), and
  `config_set.separator` — rejecting out-of-domain values that previously
  loaded and misbehaved. Ratcheting allowlist (#98).
- **Comparator + first-class delimiter engine for value checks.**
  `config_value` and `sysctl_value` now accept an optional `comparator`
  (the six relational operators; numeric operands parsed as int64,
  non-numeric compares as non-compliant rather than erroring), and
  `config_value` accepts a first-class `delimiter` param. Both are now
  documented in `CANONICAL_RULE_SCHEMA_V1.md` §3.5.3 (regenerated from the
  contracts) with new §3.5.3.1 (comparator) and §3.5.3.2 (delimiter) (#100).
- **Full-spectrum behavior harness + schema/engine parity gate.** A
  fixture-driven harness exercises each check against real temp files for
  pass/fail/edge cases (#101); a parity test parses the check dispatch
  switch from source and asserts it matches `CheckContracts` in both
  directions, so the contract can never silently drift from the
  implementation again (#102).

### Fixed

- **`config_value` with `delimiter: " "` now matches whitespace, not only
  a literal space.** RHEL `login.defs` is TAB-delimited (`PASS_WARN_AGE⇥7`),
  so the space-only match silently returned "not found" and produced a
  false FAIL on the login.defs class of rules — exactly the wrong-verdict
  failure these gates exist to catch (found by the live-fleet review, not
  static analysis). Extraction is now whitespace-aware for the `" "`
  delimiter, with TAB regression coverage (#103).
- **`--format jsonl` no longer mislabels skipped rules as errors.** The
  jsonl scan writer read the legacy `Transactions` surface, where a
  platform-gated or not-applicable rule is recorded as `errored` for
  back-compat; it now maps from the canonical `ScanResult.Outcomes`
  (`pass`/`fail`/`skipped`/`error`). A first-class `skipped` count is added
  and skips no longer inflate `errors`. Other formats are unchanged;
  confirmed no-impact for OpenWatch, which consumes `Outcomes` via the
  library path (#104).

### Changed

- `output-writer` spec bumped to v0.2.0: new constraint C-07 documents the
  jsonl writer reading the canonical `Outcomes` surface, superseding C-03's
  byte-stability guarantee for the jsonl writer only (all other formats
  remain byte-identical) (#104).

## v0.5.1 — 2026-06-18

Fixes a packaging gap that blocked external consumers (e.g. OpenWatch)
from running remediation through the public `pkg/kensa` API (issue #94).
PATCH bump: `pkg/kensa` packaging only; the frozen `api/` surface is
untouched.

### Fixed

- **`pkg/kensa.Default*().Remediate` / `.Rollback` now work for external
  importers.** The apply-mechanism handlers (`file_permissions`,
  `config_set`, `service_enabled`, …) self-register via `init()` and were
  pulled into a binary only by blank imports of `internal/handlers/*` —
  which an external module cannot import. So a consumer building a service
  via `DefaultWithTransportFactory` hit `preflight: mechanism
  "file_permissions" is not registered` before any host change. The
  `Default*` constructors now register the standard handlers
  automatically; an external consumer needs only to upgrade. Scanning was
  never affected.

### Added

- **`pkg/kensa/handlers`** — a public, blank-importable bundle that
  registers the standard apply handlers, for consumers composing a Kensa
  via `api.New{…}` directly: `import _ "github.com/Hanalyx/kensa/pkg/kensa/handlers"`.

### Changed

- The kensa CLI now sources its handler set from the same
  `pkg/kensa/handlers` bundle instead of its own import list — a single
  source of truth, so the CLI and external-consumer handler sets cannot
  diverge. A CI completeness test fails if any `internal/handlers/*`
  package is missing from the bundle, so this class of gap cannot recur
  for a future handler.

## v0.5.0 — 2026-06-15

Sudo-with-password support across the scan/remediate lifecycle. Reverses
the earlier "`sudo -n` only, no password fallback" design so Kensa serves
organizations running **either** passwordless sudo **or** sudo-with-password
seamlessly — the latter common where "no NOPASSWD sudoers" is itself an
enforced CIS/STIG control. MINOR bump: additive `api/` field
(`HostConfig.SudoPassword`); the rest of the frozen `api/` surface is
untouched.

### Added

- **`api.HostConfig.SudoPassword`** — the password supplied to `sudo -S`
  for hosts whose sudoers policy is not NOPASSWD. Additive, backward-
  compatible: empty keeps the existing non-interactive `sudo -n` behavior.
  Held in memory only; fed to the target over the SSH session's stdin,
  never placed in argv and never captured into evidence/OSCAL.
- **`--sudo-password` flag** (and **`KENSA_SUDO_PASSWORD`** env fallback)
  on `detect`, `check`, `remediate`, `plan`, and `rollback`. Omitting the
  value prompts on the controlling TTY (like `--password`). Rejected with
  `--inventory` (the env var is the shared-fleet channel) and requires
  `--sudo`.
- Connect-time sudo probe now distinguishes "a password is required"
  (none supplied → configure NOPASSWD or pass a password) from "sudo
  password rejected" (supplied password wrong/expired), each with an
  actionable message.

### Changed

- The remote-command sudo wrap is now `sudo -S -p '' sh -c '…'` (password
  on stdin) when a sudo password is configured, and the unchanged
  `sudo -n sh -c '…'` otherwise. The `remediate` agent is spawned the same
  way; on a NOPASSWD host the password is dropped via a `sudo -n true`
  probe so it cannot corrupt the agent wire protocol. `SUDO_ASKPASS`/`-A`
  was deliberately not used — it requires an askpass helper on the target,
  incompatible with the agentless model.

### Security

- The sudo password is held in operator memory and transmitted only over
  the already-encrypted SSH session's stdin — never in argv (`/proc`-safe)
  and never in the recorded stdout/stderr that back `CheckEvidence`/OSCAL.
  `-p ''` keeps sudo's prompt off captured stderr. On a wrong password,
  sudo's auth-failure *text* (never the password) may appear in a check's
  stderr; the connect-time probe maps it to a clean error. See
  `docs/test_docs/security.md` limit #12.

## v0.4.3 — 2026-06-14

Public rule read model for catalog consumers — tranche 1 of the
OpenWatch read-model ask, scoped by the Kensa/OpenWatch ownership test
(publish the normalization Kensa owns; carry facts, not policy). PATCH
bump: the addition lives on `pkg/kensa`; the frozen `api/` surface is
untouched.

### Added

- **Public rule read model on `pkg/kensa`** — the normalized catalog
  projection an `api` consumer needs to render a rule browser without
  re-parsing the heterogeneous raw `references` map or loading the full
  `[]*api.Rule`:
  - `RuleFrameworkRefs(*api.Rule) []api.FrameworkRef` — the rule's
    framework references in the same normalized form the scanner puts on
    `ScanResult.Outcomes`, delegating to the existing
    `internal/mappings` normalization (no re-implementation, no drift
    from the canonical framework-id scheme).
  - `Framework` + `FrameworkFromID(id)` + `Frameworks(rules)` — a
    framework registry so consumers render labels/families consistently
    (`cis_rhel9` → `{Family:"cis", Version:"rhel9", Label:"CIS (RHEL 9)"}`)
    instead of hardcoding prefix strings; unknown frameworks degrade
    gracefully.
  - `RuleSummary` + `RuleToSummary` + `LoadRuleSummaries(dir, paths, vars)`
    — a lightweight catalog row (id/title/description/rationale/severity/
    category/tags/platforms/transactional + normalized framework refs +
    remediation summary), loaded via the existing `LoadRules` path.
  - `RemediationSummary` carries derivable **facts only**: `Available`,
    `Mechanisms`, `RestartsServices`, and `RebootBehavior`
    (`boot-param`/`none`). Per the Kensa/OpenWatch boundary it
    deliberately omits a remediation risk level (operator policy) and a
    blanket requires-reboot boolean (not derivable for change-specific
    cases). Spec `rule-read-model` (Tier 2). The frozen `api/` surface is
    untouched.

## v0.4.2 — 2026-06-14

Per-rule OSCAL export + an unmapped-rule conformance fix, prompted by the
OpenWatch team's per-rule-expansion question. PATCH bump: the addition
lives on `pkg/kensa` and the fix is in `internal/evidence`; the frozen
`api/` surface is untouched.

### Added

- **Per-rule OSCAL export on `pkg/kensa`** — `ExportOSCALOutcome` /
  `WriteOSCALOutcome` render a single `api.RuleOutcome` as its own valid
  one-finding OSCAL 1.0.6 AR document, preserving the parent scan's host
  context (HostID/Capabilities/Platform). The per-rule counterpart of
  `ExportOSCALScan`, for a UI that exports OSCAL from one expanded rule
  rather than the whole scan.

### Fixed

- **Unmapped rule produced invalid OSCAL.** A result with no
  framework-mapped control emitted an empty `include-controls`, which the
  OSCAL 1.0.6 schema rejects (`reviewed-controls` is required and a
  control-selection must select `include-all` or a non-empty
  `include-controls`). A whole-host scan never hit this (some rule is
  always mapped), but a single-rule document for an unmapped rule — the
  per-rule UI expansion — did. The exporter now falls back to OSCAL
  `include-all` when there are no control refs, on both the scan
  (`ExportOSCALScan`) and remediation (`ExportOSCAL`) paths.

## v0.4.1 — 2026-06-14

Public OSCAL export for `api` consumers. PATCH bump: the addition lives
on `pkg/kensa` (the public-but-not-frozen assembly layer); the frozen
`api/` surface is untouched. With v0.4.0's `RuleOutcome.Evidence`, this
makes the whole evidence-and-OSCAL feature reachable from outside the
CLI: `LoadRules → NewScanner → Scan → Outcomes → ExportOSCALScan` is now
entirely public for embedders (OpenWatch).

### Added

- **Public OSCAL export on `pkg/kensa`** — `ExportOSCALScan` /
  `WriteOSCALScan` (an `api.ScanResult` → OSCAL 1.0.6 Assessment
  Results) and `ExportOSCAL` / `WriteOSCAL` (a signed
  `api.EvidenceEnvelope` → OSCAL 1.0.6 AR). v0.4.0 shipped OSCAL export
  only through the CLI and `internal/evidence`, which an embedder cannot
  import; these thin wrappers lift it to the public-but-not-frozen
  assembly layer (where `LoadRules`/`NewScanner` live), completing the
  public chain `LoadRules → Scan → Outcomes → OSCAL` for consumers like
  OpenWatch. The frozen `api/` surface is untouched; byte production
  still lives in (and is conformance-gated by) `internal/evidence`. The
  scan-path export stays unsigned by design; the signature guarantee
  remains exclusive to the envelope path. Spec `oscal-public-export`
  (Tier 2).

## v0.4.0 — 2026-06-13

Native-evidence parity and OSCAL enrichment: a compliance **scan** now
produces reproducible, structured evidence — per-check command/output
proof — in two surfaces: a Kensa-native JSON document and a
standards-conformant OSCAL 1.0.6 Assessment Results document. MINOR
bump: the `api/` surface gains additive fields only; nothing is removed
or changed. Both schemas are vendored and validated in CI, and the
output was validated end-to-end against the live test fleet (RHEL
8.10/9.7, Ubuntu 24.04/26.04, full 539-rule corpus).

### Added

- **Structured per-check observation evidence on the scan path.**
  `api.RuleOutcome` gains `Evidence []api.CheckEvidence` — one
  `CheckEvidence` per command a rule's check executed, carrying the
  exact `Method`, `Command`, captured `Stdout`/`Stderr`, `ExitCode`, and
  the `Expected` value. This is the reproducible proof behind a verdict:
  an auditor can re-run the command and compare without re-running the
  scan. Captured via a recording transport that wraps the check
  transport, so none of the 24 check functions changed. Oversized output
  is truncated at a 64 KiB per-field cap and flagged (`Truncated`). (#74)
- **`-o evidence:PATH` on `kensa check`** — emits the Kensa-native
  evidence document (`schemas/kensa-evidence-v1.schema.json`): session +
  host context (hostname, detected platform, capabilities, effective
  variables) and one result per rule with its embedded `CheckEvidence`
  and framework refs, plus a pass/fail/skip summary. Full-file parity
  with the prior Python Kensa evidence shape. (#75)
- **`-o oscal:PATH` on `kensa check`** — renders the scan as an OSCAL
  1.0.6 Assessment Results document: one finding + observation per rule,
  the check evidence embedded as namespaced `relevant-evidence` props
  with the verbatim command in `remarks`, raw stdout carried as base64
  back-matter referenced by href, and framework refs as deduplicated
  control-id tokens. Unsigned by design (the signature guarantee remains
  exclusive to the remediation evidence-envelope path). (#77)
- **Host context on `api.ScanResult`** — `Capabilities CapabilitySet`
  and `Platform DetectedPlatform` (new `DetectedPlatform{Family,
  Version}` type), so a consumer records the exact capability/OS context
  a verdict was computed under without re-probing. (#75)
- **Vendored schemas + conformance gate** — the official NIST OSCAL
  1.0.6 Assessment Results schema and the `kensa-evidence-v1` schema are
  vendored, and a hard test gate validates every emitted document
  against OSCAL 1.0.6 (using a pure-Go validator that handles the
  schema's ECMA `\p{}` regex and draft-07 anchors). (#73, #76)

### Fixed

- **OSCAL 1.0.6 conformance gaps surfaced by the live test fleet** that
  the clean unit fixtures never exercised: a multi-line check command
  cannot live in an OSCAL prop value (single-line token), so it now goes
  to `relevant-evidence` remarks with prop values guarded; and NIST
  control-enhancement parentheses (`AU-5(2)`) are illegal in an OSCAL
  control-id token, now coerced to the dot-enhancement form (`AU-5.2`)
  on both the scan and remediation export paths. (#78)

## v0.3.2 — 2026-06-12

Public scanner construction with a caller-supplied transport. PATCH
bump: additions live on `pkg/kensa`; the frozen `api/` surface is
untouched. With v0.3.1's loader this completes the public consumer
chain: `LoadRules` → construct → `Scan` → `Outcomes`.

### Added

- **Public construction with a caller-supplied `TransportFactory`**
  (`pkg/kensa`) — for embedders whose credential model the bundled
  on-disk-key ssh factory cannot serve (e.g. an orchestrator holding
  SSH credentials in memory only):
  - `kensa.NewScanner()` — the standard `api.ScannerBackend`, for
    scan-only composition via `api.New(Config{Scanner, TransportFactory})`;
    no engine, store, or signer is constructed. Stateless and safe for
    concurrent `Scan` calls sharing one instance. `Remediate` on such a
    construction errors (engine not wired), by design.
  - `kensa.DefaultWithTransportFactory(ctx, storePath, tf, engineOpts...)`
    — `Default`'s full wiring with the transport swapped for the
    caller's factory (nil rejected at construction). `Default` and
    `DefaultWithEngineOptions` are unchanged.

## v0.3.1 — 2026-06-11

Public rule loader for programs that import the `api` package. PATCH
bump: additions live on `pkg/kensa` (the public-but-not-frozen assembly
layer); the frozen `api/` surface is untouched.

### Added

- **`pkg/kensa` public rule-loader surface** — external consumers (e.g.
  OpenWatch) can now load the shipped corpus without copying rule files
  or re-implementing the loader:
  - `kensa.LoadRules(dir, paths, vars)` — corpus → `[]*api.Rule` ready
    for `Scan`/`Remediate`. Follows the CLI's path-resolution policy
    (explicit dir → explicit files → the `kensa-rules` package's
    installed corpus at `/usr/share/kensa/rules`), and substitutes
    `{{ name }}` rule templates against kensa's embedded defaults merged
    with the caller's `vars` (caller wins) — the hook for
    operator-configured values. **Strict** by design: any unparseable
    file or undefined variable fails the load naming the file; nothing
    is skipped silently (deliberate divergence from the CLI's
    warn-and-skip directory walk).
  - `kensa.BuiltInVars()` — the embedded variable defaults (29 vars,
    STIG-strict), for rendering an operator configuration UI. The
    `rsyslog_remote_server`, `chrony_ntp_pool`, and `banner_text`
    defaults are organization-specific placeholders operators should
    always review.
  - `kensa.RuleVariables(dir)` — template variable → rule IDs using it,
    so operators can see what an override affects.

  Locked by spec `rule-public-loader`, including a production-corpus
  test: all 539 rules — the 23 `{{ var }}` templates included — load
  strictly with nil vars on built-in defaults alone.

### Documentation

- `docs/guide/04-scan-and-remediate.md` documents the v0.3.0 `SKIP`
  status (platform gating) for `check` and `remediate`.
- `docs/guide/07-integration.md` carries the api-consumer pointers:
  read verdicts from `ScanResult.Outcomes`; load rules via
  `kensa.LoadRules`; do not copy the rule files or re-implement the
  loader.

## v0.3.0 — 2026-06-11

Compliance-verdict API on `Scan`, platform gating for the standalone CLI,
and the param-contract fix that restores ~201 corpus rules whose handlers
read the wrong parameter names. MINOR bump: additive surface on the frozen
`api/` package (new types + a new `ScanResult` field; nothing existing
changed signature or semantics).

### Added

- **`api`: compliance-verdict surface on `Scan`** — `ScanResult` gains
  `Outcomes []RuleOutcome`, one per scanned rule in input order, each with
  a `ComplianceStatus` of `pass` / `fail` / `skipped` / `error`, the rule's
  severity, a human-readable detail, the error cause (iff `error`), and the
  rule's `FrameworkRefs` (CIS / NIST 800-53 / STIG, normalised from the
  rule's References block) so a consumer attributes verdicts to frameworks
  without re-joining the corpus. This is the canonical compliance result;
  the check-only `Transactions` entries (whose `committed` / `rolled_back`
  statuses double as compliant / non-compliant) are retained unchanged for
  backward compatibility. (#62, #63)
- **Platform gating for `check` and `remediate`** — kensa now detects the
  host OS (`/etc/os-release`) and compares each rule's `platforms:` block
  (family + `min_version` / `max_version`) against it. A rule that does not
  apply to the host — e.g. a `rhel >= 9` control on a RHEL 8 host — is
  reported `SKIP` (with a "not applicable: host RHEL 8.10, rule targets
  rhel >=9" detail) instead of a misleading pass/fail, and on `remediate`
  its remediation is **never applied**. Rules with no `platforms:` block
  run everywhere; an undetectable host OS gates nothing. The live row
  stream renders a dim `SKIP` status and a `N skipped` tally entry. (#64)
- **Param-contract gate** — `internal/mechanism` is now the single source
  of truth for each mechanism's parameter contract
  (CANONICAL_RULE_SCHEMA_V1.md §3.5.4). The rule validator rejects
  remediation params that violate the contract (`kensa-validate` + CI),
  and an integration test decodes every corpus rule's params through its
  real handler with a ratcheting divergence ledger — now empty. (#50)

### Fixed

- **Seven handlers read parameter names that contradicted the canonical
  schema and the shipped corpus**, so ~201 corpus rules failed at the
  Capture phase instead of remediating. Each handler now decodes the
  schema's names (pre-state data shapes unchanged, so existing rollback
  records still restore): `config_set` reads `path` (was `file`) (#51);
  `config_set_dropin` composes `dir`+`file` (#52); `kernel_module_disable`
  reads `name` (was `module`) (#53); `pam_module_configure` reads
  `type`/`args` (was `module_type`/`options`) (#54); `audit_rule_set`
  reads absolute `persist_file` (was bare `rule_file`) (#55);
  `mount_option_set` reads the `options` list (was singular `option`)
  (#56); `cron_job` makes `name` optional and supports an absolute `file`
  path (#57). Verified end-to-end on a live host: previously-failing rules
  across these mechanisms now apply and roll back byte-perfectly.

### Changed

- **Scanning a host whose OS no rule targets now reports `SKIP` per rule**
  (e.g. the RHEL corpus against a non-RHEL host renders all-SKIP) rather
  than a wall of misleading FAIL/ERROR rows. This is the honest result —
  those rules do not apply to that host.
- Dependency refresh: `godbus/dbus` v5.2.2, `golang.org/x/sys` v0.46,
  `golang.org/x/term` v0.44, `modernc.org/sqlite` v1.52 (pure-Go SQLite,
  portability CI unchanged). (#58)

### Internal

- CI lint stack modernised: golangci-lint v1.64.8 → **v2.12.2** (native
  Go 1.26 support) with `golangci-lint-action` v9 and a `GOTOOLCHAIN`
  pin that prevents the linter from being built against an older Go
  (the root cause of a repo-wide SA5011 false-positive storm); the full
  staticcheck `S*`/`ST*`/`QF*` tiers are now enforced. Specter pinned at
  v0.13.2 in CI with annotation-strictness sync. (#49, #59, #60, #61)

## v0.2.3 — 2026-06-08

Live result-row streaming for the default human output, plus engine and
agent/transport fixes surfaced while building it. No machine-format or
`api/` contract changes.

### Added

- **Live result-row streaming for `kensa check` and `kensa remediate`** —
  the default text output now renders one aligned row per rule **as each
  rule completes**, in scan order, directly on stdout:
  `STATUS  SEVERITY  RULE-ID  DESCRIPTION [detail]`, under a `── Host ──`
  banner, ending with a tally. `check` shows `PASS` / `FAIL` / `ERROR`;
  `remediate` shows `PASS` (already compliant) / `FIXED` / `FAIL` /
  `ERROR`. Status and severity are colored when stdout is a terminal.
  Machine formats (`--format json`, `-o FILE`) are unchanged — still
  buffered and structured, never interleaved with rows.

### Changed

- **`kensa check`'s default text output is now an in-order live row
  stream** rather than a grouped, buffered end-of-scan report. The result
  rows are the canonical text rendering and go to stdout; there is no
  separate progress channel. Machine/`-o` output is unchanged.
- **`--sudo` fails fast with an actionable error when the SSH user lacks
  passwordless sudo.** kensa runs sudo non-interactively (`sudo -n`) on
  every path and has no password fallback by design; a one-time probe at
  connect now reports *"configure passwordless sudo … or drop --sudo"*
  instead of letting every remote command fail cryptically. A non-password
  sudo failure (e.g. user not in sudoers) is surfaced verbatim.

### Fixed

- **Engine event-bus panic** — `InMemoryEventBus.Publish` could send on a
  closed channel when a subscription's context was canceled concurrently,
  panicking a live transaction. Delivery and channel-close are now mutually
  exclusive (Tier-1 spec `engine-event-bus` with a regression test).
- **Server login banner leaking into agent-mode output** — `remediate` /
  `rollback` (agent mode) re-authenticate over a fresh ssh session whose
  stderr is forwarded to the operator; a server consent/login banner (e.g.
  a USG banner) leaked there. The agent ssh now passes `-o LogLevel=ERROR`
  to suppress the banner while preserving real ssh errors, matching
  `kensa check`.

## v0.2.2 — 2026-06-05

Supply-chain and service-handler hardening on top of v0.2.1. The
headline operator-facing change is that the package now provisions the
systemd-helper sudo escalation path itself; the rest is supply-chain
trust posture (a govulncheck gate that immediately paid for itself by
forcing two toolchain CVE bumps) and a new capability probe.

### Added

- **Sudoers fragment + `kensa` group, shipped by the package** — the
  rpm and deb now install `/etc/sudoers.d/kensa-systemd-helper` (mode
  `0440`, root-owned, registered as a config file) granting
  `%kensa ALL=(root) NOPASSWD: /usr/libexec/kensa-systemd-helper`, and
  `postinst` creates the `kensa` group **empty**. The service handlers
  (`service_enabled` / `_disabled` / `_masked`) previously required a
  manual sudoers step; the operator's remaining action shrinks to
  `usermod -aG kensa <user>`. The empty group means a fresh install
  grants the escalation to nobody. Spec `packaging-sudoers-helper`.
- **`pam_tally2` capability probe** — detects the legacy account-lockout
  module present on older Debian/Ubuntu (≤18.04) and RHEL 7, where
  `pam_faillock` is absent, so rules can gate a fallback.
- **Supply-chain CI gates** — a `govulncheck` vulnerability scan, a
  `go mod tidy` drift check, top-level `GOFLAGS=-mod=readonly`, a
  `detect-secrets` baseline + pre-commit hook, and Dependabot for the
  `gomod` and `github-actions` ecosystems.

### Changed

- **Go toolchain pinned to 1.26.4** via the `go.mod` `go` directive;
  building from source now requires Go 1.26.4+.
- **CI actions moved to their Node 24 majors** — `actions/checkout@v5`,
  `actions/setup-go@v6`, `actions/setup-python@v6` — ahead of GitHub's
  Node 20 removal. `setup-go` now installs the exact toolchain from
  `go.mod` (`go-version-file`).
- **Install guide** rewritten for the now-automatic service-handler
  setup and the Go 1.26.4 build requirement.

### Security

- The shipped `%kensa` NOPASSWD rule is **inert on install**: the group
  is created empty, and the post-install guard checks `/etc/group`
  directly rather than via `getent`, so a same-named directory
  (LDAP/NIS/SSSD) group cannot silently inherit the grant. The residual
  limit — `sudo`'s own `%kensa` resolution still consults nsswitch — is
  documented as an explicit install-time precondition.
- **Eight standard-library CVEs cleared.** The new govulncheck gate
  surfaced six stdlib advisories (fixed by the 1.26.3 bump) and then
  `GO-2026-5039` (net/textproto) + `GO-2026-5037` (crypto/x509), fixed
  by 1.26.4.
- CI now **asserts the sudoers fragment ships at `0440` root:root** in
  both the rpm and the deb on every build (owner drift would silently
  disable the rule).

### Fixed

- The secret-scan CI job no longer false-fails on `detect-secrets`
  baseline metadata churn — it uses the non-mutating `detect-secrets-hook`
  entrypoint, the same code path as the pre-commit hook.
- Removed a stray committed gitlink (`.claude/worktrees/...`) that made
  every `git checkout` emit a submodule warning.

## v0.2.1 — 2026-05-28

Packaging-UX hardening on top of v0.2.0's first signed packages. No
binary-behaviour change — every difference here is in the package
metadata, install scripts, archive variants, and operator docs.

### Added

- **`KEYS` at repo root** — single canonical file holding both
  verification keys (Hanalyx LLC GPG public + Kensa cosign public) with
  inline `rpm --import` / `apt`-add / `cosign verify-blob` instructions
  as a fallback if the install guide is unreachable. Operators can
  point `rpm --import` straight at the raw GitHub URL.
- **`kensa_<v>_linux_<arch>_with-rules.tar.gz`** — second per-arch
  tarball variant carrying the full 539-rule corpus next to the
  binaries + LICENSE + KEYS. Single-download air-gap path called out
  in CLAUDE.md's packaging plan. Topic-dir layout preserved
  (`rules/<topic>/<rule>.yml`) so it matches the `kensa-rules`
  package's installed tree.
- **`packaging/postinst.sh`** — POSIX `/bin/sh` script wired into the
  `kensa` rpm and deb. Surfaces a warning when `/usr/share/kensa/rules`
  is empty after install with explicit next-step commands (install
  `kensa-rules` or pass `--rules-dir`). No network access, no signature
  re-verification — Fedora packaging guidelines forbid the former and
  the latter belongs to dnf/apt's existing trust chain.
- **`docs/guide/01-install.md` rewritten** for the packaged-release
  reality: signed-key import is Step 1, three install paths (dnf, apt,
  air-gap tarball), explicit verify with `cosign verify-blob` +
  `sha256sum -c` for the air-gap flow, and build-from-source kept as
  the contributors' path. Version references bumped 0.1.0 → 0.2.1.

### Changed

- **`kensa` rpm + deb now `Recommends: kensa-rules`** — `dnf install
  kensa` / `apt install kensa` alone pulls the corpus by default.
  Operator can opt out with `--setopt=install_weak_deps=False` (rpm)
  or `--no-install-recommends` (apt) if they bring their own corpus
  via `--rules-dir`.

### Internal

- Goreleaser bumped one notch on the second `archives:` entry —
  the `kensa-with-rules` ID adds `KEYS` to the bundled file list.
- The release-snapshot CI smoke job now produces 11 artifacts (was 9):
  the two `with-rules` tarballs added a binary-tarball variant per arch.

### Not changed

- The kensa CLI itself is byte-identical to v0.2.0 (same source, same
  build flags). The release is metadata + scripts + docs only.
- No `api/` change.
- All signing posture (GPG-signed rpm/deb + cosign-signed checksums)
  preserved from v0.2.0.

### Known follow-ups

- File the nfpm `gnu-dummy` subkey-decrypt bug upstream
  (`goreleaser/nfpm` `internal/sign/pgp.go:readSigningKey`). Once
  fixed, the GitHub `GPG_PRIVATE_KEY` secret can go back to a
  subkey-only export (smaller blast radius than the current full
  encrypted master+subkey).
- v0.3.x: ship the kensa-systemd-helper sudoers fragment in the rpm
  (`%files /etc/sudoers.d/kensa-systemd-helper`) so the manual step
  in `docs/guide/01-install.md` § "Service handlers" disappears.

## v0.2.0 — 2026-05-28

First "real" packaged release. Operators can now install kensa via
`dnf install kensa kensa-rules` (rpm) or
`apt install ./kensa_0.2.0_linux_amd64.deb ./kensa-rules_0.2.0_noarch.deb`
(deb) and run `kensa check <host>` with no flags — the default-path
fallback (this release) picks up the corpus from
`/usr/share/kensa/rules` automatically.

Also lands the full grub deadman guard (set + remove paths), the
`grub_parameter_set` and `grub_parameter_remove` handlers wired through
it, the first operator-guide chapter, and the supporting plumbing.

### Added

#### Packaging
- `LICENSE` at repo root — Business Source License 1.1 (→ Apache 2.0 on
  2029-01-01); same terms as the archived Python kensa. Required by
  rpm/deb metadata.
- `kensa` rpm + deb (amd64, arm64) — installs `/usr/bin/{kensa,
  kensa-validate, kensa-keygen}` + `/usr/libexec/kensa-systemd-helper`
  + `/usr/share/doc/kensa/{LICENSE,README,CHANGELOG}`. Signed with the
  Hanalyx GPG key.
- `kensa-rules` noarch rpm + deb — installs the 539-rule corpus to
  `/usr/share/kensa/rules`. Updates independent of the binary release.
  Signed with the Hanalyx GPG key.
- `kensa_<version>_linux_<arch>.tar.gz` — air-gapped install bundle for
  amd64 + arm64 (all 4 binaries + LICENSE + docs).
- `kensa_<version>_checksums.sha256` — sha256 over the full artifact
  set. cosign-signed.
- `.goreleaser.yaml` + tag-triggered `.github/workflows/release.yml` —
  hard-fails if any of GPG_PRIVATE_KEY, GPG_PASSPHRASE,
  COSIGN_PRIVATE_KEY, COSIGN_PASSWORD secrets is missing (no silent
  unsigned ship). Snapshot smoke job in `ci.yml` exercises the same
  pipeline on every PR.

#### Rules
- `rules/` — vendored the 539 SCAP-derived rules from the archived
  Python kensa (`/home/rracine/hanalyx/kensa.archive/rules`),
  byte-identical to source. Eight topic dirs (`access-control` 129,
  `audit` 101, `filesystem` 73, `kernel` 22, `logging` 14, `network`
  23, `services` 107, `system` 70). 2.2 MB. The archive is frozen;
  rule edits land via PR here going forward. `rules/README.md`
  documents layout, validate workflow, and provenance.
- `internal/rules.Resolve` — default-path resolution for `--rules-dir`.
  Explicit `--rules-dir` still wins; positional rule YAML paths alone
  skip the walk; when neither is given the CLI falls back to
  `/usr/share/kensa/rules` (where `kensa-rules` installs); when that
  path also doesn't exist the loader surfaces a usage error naming all
  three fix paths. Specced as `rule-default-path-resolution` v0.1.0;
  `cli-rule-flag` bumped to v0.2.0.

#### Grub deadman guard
- `internal/bootguard/` (PR #15) — Option-B one-shot trial entry +
  saved-default auto-fallback for grub parameter changes. RHEL/BLS
  via `grubby --copy-default`; Ubuntu legacy via `/etc/grub.d/11_kensa_bootguard`
  + `update-grub`. Confirm unit installed at arm time; healthy boot
  promotes onto the default, failed boot auto-reverts. Specs:
  `bootguard-{capture,arm-gate,allowlist,oneshot,confirm}`.
- `internal/handlers/grubparameterset` — replaces direct
  `GRUB_CMDLINE_LINUX` editing with `bootguard.ArmOneshot`. Refuses
  off-allowlist keys + non-armable hosts. PENDING until reboot.
- `internal/handlers/grubparameterremove` (PR #21) — same flow for
  REMOVAL via `bootguard.ArmOneshotRemove`. `bootguard-oneshot` bumped
  to v0.4.0 (C-07/C-08, AC-08..AC-11); `bootguard-confirm` v0.5.0.
- Verified end-to-end on real RHEL 9.7 (.213) and Ubuntu 24.04 (.249)
  with the destructive reboot matrix.

#### Docs
- `docs/guide/01-install.md` (PR #12) — first real operator-guide
  chapter.

### Changed
- `cli-rule-flag` C-04 + AC-04 — now acknowledge the default-path
  fallback layer and the three-fix-paths error wording.

### Internal
- `.golangci.yml` — extended the existing `internal/` godot exclusion
  to `cmd/` (same rationale: Go's identifier-first convention conflicts
  with `capital: true`).

## v0.1.1 — 2026-05-27

First tag carrying the ratified Kensa/OpenWatch boundary, so OpenWatch can
pin a kensa with the agreed `api/` surface. No `api/` contract change since
v0.1.0 — the public Go API is identical; this is a documentation +
internal-quality release.

### Changed

- `api/` event-stream godoc (`EventKind`) corrected to the ratified
  Kensa/OpenWatch boundary: OpenWatch owns liveness / heartbeat / drift;
  Kensa emits the shared event vocabulary (`transaction_started`,
  `committed`, `rolled_back`, `drift_detected`, `heartbeat_pulse`,
  `deadman_timer_armed` / `_fired`). Godoc only — the types are unchanged.

### Internal

- Spec-driven tests added for 10 previously-untested handlers.
- Code comments scrubbed of planning labels (e.g. roadmap phase / option
  names) per the new "Comments" section of `CONTRIBUTING.md`; a
  `make comment-lint` check guards new comments.

## v0.1.0 — 2026-05-14 (Sentinel)

First versioned release on the renamed repository (formerly
`Hanalyx/kensa-go`, now `Hanalyx/kensa` after the Python kensa was
archived). The 0.1.0 line is the development phase: the public
`api/` Go package is held to v1 semver for OpenWatch's consumption,
and the rest of the surface may change between MINOR versions with
one release of deprecation warning. See
[`VERSIONING_PLAN.md`](VERSIONING_PLAN.md) for the full release
contract.

### Added

- **VERSION file at the repo root** as the single source of truth for
  the version string. All five binaries (`kensa`, `kensa-fuzz`,
  `kensa-validate`, `kensa-keygen`, `kensa-systemd-helper`) read it
  via `-ldflags "-X main.version=$(cat VERSION)"` and report `0.1.0`
  from `--version` / `-V`.
- **`VERSIONING_PLAN.md`** documenting SemVer 2.0.0 discipline,
  codename Sentinel (guardianship theme), atomicity-contract changes
  as always-MAJOR, and the frozen `api/` v1 contract.
- **`docs/guide/`** operator manual skeleton: index plus nine chapter
  stubs (install, quickstart, concepts, scan-and-remediate,
  rollback-and-history, rule-authoring, integration, troubleshooting,
  reference). Content lands in subsequent releases.

### Changed

- **Module path**: `github.com/Hanalyx/kensa-go` →
  `github.com/Hanalyx/kensa`. GitHub keeps a URL redirect from the
  old path so existing `go get` continues to resolve, but consumers
  should migrate when convenient. Re-run `go mod tidy` after bumping.
- **`docs/man/` → top-level `man/`.** The manpage source
  (`gen-manpage.go`) is real Go code and `kensa.1` ships in the RPM;
  it doesn't belong under `docs/`. Makefile, specter.yaml, and CI
  paths updated.
- **`docs/*` is now gitignored except `docs/guide/`.** Internal
  working notes (vision, roadmap, foundation contracts, coordination,
  AI session logs, founder release sign-off) stay locally as
  untracked working material. The published documentation surface is
  the operator guide.
- **README rewritten** to operator-facing voice per the new
  documentation style guide: runnable example up top, guarantees
  stated as facts (no marketing language), portability and atomicity
  contract as compact tables, explicit pre-1.0 callouts so today's
  reader knows what works (`make build` + `--rules-dir <local-path>`)
  versus the documented v1.0 ship state.

### Security

- **Kernel-primitive deadman timer for control-channel-
  sensitive remediation (Phase 3 P-011/D-001..D-006).** The
  deadman timer — the safety net that rolls back a
  half-applied rule if the controller-target SSH connection
  drops mid-Apply — now uses kernel primitives instead of
  `at(1)`/`systemd-run` scheduled shell scripts when running
  in agent mode. The new architecture:
  - `timerfd(CLOCK_BOOTTIME)`: counts elapsed seconds INCLUDING
    system suspend. A laptop or VM suspended mid-Apply
    resumes with the correct deadline (the old `at(1)` path
    fired late or not at all because wall-clock-based
    scheduling missed the suspended interval).
  - `pidfd_open(getppid(), 0)`: race-free SSH-parent-death
    detection in <200ms. The old path had no equivalent —
    it relied on the scheduler firing on its own deadline,
    with second-granularity latency.
  - `signalfd` for SIGTERM (via `signal.Notify` + self-pipe
    forwarder, the Go-runtime-friendly equivalent).
  - `epoll_wait` single-thread event loop integrating all
    three.

  RHEL 8 kernels (<5.3) lack `pidfd_open`; the agent probes
  at startup and falls back to
  `prctl(PR_SET_PDEATHSIG, SIGKILL)` — the kernel SIGKILLs
  the agent on parent death (rollback doesn't fire under
  SIGKILL, accepted risk per Q3.a; the agent at least
  doesn't linger orphaned).

  **Direct-SSH mode retains the shell-based path** (opt-in via
  `KENSA_NO_AGENT=1`) for environments where agent bootstrap
  isn't viable. It does NOT gain suspend-resistance or
  clock-jump-immunity — those properties require the
  in-process agent-side primitives.

- **Kernel-atomic file operations under agent mode (Phase 2,
  fix/phase-2-rework drop + P-011 default flip).** For the
  file-touching capturable handlers — `file_content`,
  `file_absent`, `config_set`, `config_set_dropin` — kensa now
  delivers literal kernel-primitive atomicity when remediation
  runs in agent mode (the default; opt-out via
  `KENSA_NO_AGENT=1`). The primitives:
  - `AtomicWrite` (new files): `O_TMPFILE` + `linkat` via
    `/proc/self/fd/<N>` — partially-written bytes are never
    visible as a half-named file in the directory.
  - `AtomicReplace` (existing files): `renameat2(RENAME_EXCHANGE)`
    for symmetric old↔new swap, with `renameat` rename-into-
    place fallback for filesystems that don't support the
    flag (kernel <3.15, NFS, vfat, some overlayfs).
  - `AtomicRemove`: `unlinkat` + parent-dir `fsync`.
  - Every primitive issues a parent-directory `fsync` barrier so
    the directory entry persists across crashes.

  **Direct-SSH mode is preserved as an explicit opt-out.**
  Operators who set `KENSA_NO_AGENT=1` get the shell-pipeline
  best-effort semantics for these mechanisms — intended for
  environments where agent bootstrap is not viable (noexec
  /tmp, locked-down SSH user, etc.). The `kensa remediate` CLI
  prints a one-line stderr disclosure on every run
  ("agent mode (default) — kernel-atomic primitives" or
  "direct-SSH mode (KENSA_NO_AGENT=1) — shell-pipeline
  best-effort; unset KENSA_NO_AGENT for kernel-atomic") so
  audit reviewers see the basis without reading the source.

- **Typed `ErrParentDirMissing`.** Returned by all three
  primitives when the parent directory of the target doesn't
  exist (or an intermediate component is missing). Replaces
  the previous generic "open intermediate" wrap. Operators
  hitting this error get a clear pointer to the missing
  component name.

- **Symlink-traversal refusal.** The fsatomic primitives walk
  the target path component-by-component with `O_NOFOLLOW` and
  refuse to operate if any component (including the base) is a
  symlink. An attacker who plants
  `/etc/sudoers.d/99-foo → /etc/passwd` cannot use kensa to
  rewrite the symlink target; the operation fails with a
  typed `ErrSymlinkInPath` error and the original target is
  unmodified. Rules that legitimately want to target a symlink
  must pass the resolved path; today's corpus has none.

- **Path-traversal defense.** Every file-touching handler
  rejects rule-supplied paths that are relative or contain
  `..` segments after `filepath.Clean`. Closes the previously-
  undefended direct-SSH shell path where
  `path: "../../etc/shadow"` would have been honored.

- **`renameat2` probe is per-filesystem, not global.** The
  RENAME_EXCHANGE support probe caches by `st_dev` so a
  heterogeneous mount layout (ext4 + NFS + xfs, typical on
  federal hosts) cannot poison the cache. First observation
  of an unsupported filesystem emits a one-time stderr
  warning to the operator.

### Fixed

- **Mode preservation across re-Apply (P0-A from the
  post-merge correctness review).** When a rule omits the
  `mode` parameter and the target file exists, `file_content`
  and `config_set_dropin` now preserve the file's current
  mode bits instead of silently widening to `0o644`. Matches
  `printf > file` shell semantics. A file previously
  tightened to `0o600` (e.g., one containing secrets) is no
  longer widened on re-Apply.

- **setuid/setgid/sticky bit preservation in `config_set`.**
  The Apply and Rollback paths previously used
  `info.Mode().Perm()` which silently dropped the special
  bits. `config_set` now uses `fsatomic.FileModeBits` which
  preserves all 12 Unix mode bits. `sed -i` preserves all 12;
  kensa now matches.

- **`config_set` regex char-class divergence on CRLF files.**
  The Go `[[:space:]]` class matches `\t\n\v\f\r ` but `sed
  -E` with `LC_ALL=C` matches only `\t ` — divergence on
  CRLF-line-ending files. The regex now spells the class as
  `[\t ]`, byte-equivalent to sed.

- **Rollback no longer silently defaults missing captured
  mode to `0o644`.** A `file_content` or `file_absent`
  rollback with an empty captured `mode` (indicating a
  Capture bug) now fails loudly rather than widening
  permissions on the restored file. Operator is instructed
  to re-run capture.

### Changed

- **`api.AtomicTransport` moved to
  `internal/agent/fsatomic.Transport`.** The capability
  interface previously lived in `api/` but atomicity is an
  agent-side concern that external `api.Transport`
  implementers (OpenWatch) are not expected to provide.
  Moving it to `internal/agent/fsatomic` prevents the
  `api/` surface from growing to 6+ sibling capability
  interfaces by Phase 7. Public consumers should not be
  affected; OpenWatch does not type-assert this interface.
  See `internal/agent/fsatomic/transport.go` for the new
  location.

### Breaking changes

- **Agent-mode is now the default for `kensa remediate` (P-011).**
  The `KENSA_USE_AGENT=1` env-var (opt-in) is replaced by
  `KENSA_NO_AGENT=1` (opt-out). Operators scripting against the
  old sense must invert the check OR drop the env-var entirely
  to get the new default behavior. Rationale (Q1.c ratified
  2026-05-12): kensa is pre-production; the kernel-atomic
  path is the safer default. Direct-SSH stays available for
  hosts where agent bootstrap is not viable. No deprecation
  period — kensa is pre-1.0.

- **CLI Phase 3 short-letter table reconciliation (C-024).** Four
  short letters are reassigned to align with `Python kensa` parity.
  Operators scripting kensa with the old short forms must migrate
  before upgrade. Long forms are unchanged; only the short letters move.

  | Flag | Old short | New short | Migration |
  |---|---|---|---|
  | `--port` | `-p` | `-P` | use `--port` long form, or `-P` |
  | `--sudo` | `-s` | (none) | use `--sudo` long form |
  | `--txn` | `-t` | `-T` | use `--txn` long form, or `-T` |
  | `--format` | `-f` | (none) | `--format` already deprecated; use `-o`/`--output` |

  The freed short letters (`-p`, `-s`, `-t`, `-f`) are reserved for
  Phase 3 deliverables: `--password` (C-026), `--severity` (C-030),
  `--tag` (C-031), `--framework` (C-033). Operators using the old
  short forms after upgrade get pflag's "unknown shorthand" error
  (exit code 2). No deprecation period — kensa is pre-1.0 and
  the migration doc explicitly notes Python kensa has no production
  users to migrate.

### Added

- `-o, --output FORMAT[:PATH]` (repeatable) on `kensa detect`,
  `kensa check`, and `kensa remediate` (C-019). Operators can
  dispatch one in-memory result to multiple destinations
  concurrently:
  ```
  kensa check -H prod-01 -u admin --sudo \
      -o json -o csv:results.csv -o pdf:report.pdf
  ```
  Supported formats: `text`, `json`, `jsonl`, `csv`, `pdf`,
  `evidence`, `oscal`, `markdown`. Not every format applies to
  every payload type — `oscal`/`evidence` are remediation-only;
  `pdf`/`text`/`csv` apply to scan and remediation;
  `jsonl` is scan-only. Specifying an unsupported format
  returns exit code 2 (usage error) with a clear message.

- `-q, --quiet` flag on body-emitting subcommands (C-018).
  Suppresses default human-readable output to stdout. Errors
  and warnings still emit on stderr; exit codes (0/1/2) are
  unchanged. Pairs naturally with `-o FILE` for CI scripts:
  `kensa check --quiet -o json:results.json`.

- CSV serializer (C-013) registered for scan, remediation, and
  history result types. RFC 4180-compliant escaping via
  `encoding/csv`.

- PDF serializer (C-015) using `maroto v2` (MIT, pure Go, no
  cgo). Registered for scan and remediation; produces operator
  triage / handoff PDFs (NOT audit-grade evidence — that's the
  signed envelope output from `--output evidence:PATH`).

- OSCAL Assessment Results serializer wired through `-o`
  (C-016). Equivalent to the legacy `--oscal` flag (which is
  now deprecated).

- Evidence-envelope JSON serializer wired through `-o` (C-017).
  Each transaction's signed `*api.EvidenceEnvelope` emits as an
  independently-verifiable JSON document. Pre-M7, signatures
  are empty bytes; the wire shape is final.

- **Session model + session-aware subcommands (CLI Phase 4,
  C-039 .. C-050).** SQLite migration 2 introduces a `sessions`
  table that groups transactions from one CLI invocation.
  Twelve new operator-facing surfaces:

  - `kensa migrate` (C-040) — applies pending schema
    migrations; backfills synthetic sessions for pre-Phase-4
    transactions. Idempotent.
  - `kensa check --store` (C-041) — persists a check run as a
    session + transactions in the SQLite log. Default off
    (check is read-only by convention).
  - `kensa history --stats` (C-042) — aggregate counts by
    session / transaction / status / severity / host. Works
    with `--host` and `--since` filters.
  - `kensa history --prune DAYS` (C-043) — destructive
    cleanup. Deletes sessions older than N days plus the
    cascade (transactions, steps, pre_states, framework_refs,
    rollback_events). Requires `--force` for non-interactive
    runs; otherwise prompts on TTY.
  - `kensa mechanisms` (C-044) — canonical name for the
    handler-mechanism listing. `kensa coverage` is a
    deprecated alias (see Changed section below).
  - `kensa coverage --framework FRAMEWORK --rules-dir DIR`
    (C-045) — framework control coverage report. Numerator
    only (controls referenced); denominator (controls in
    framework) is a future deliverable.
  - `kensa list frameworks` (C-046) — list framework_ids in
    the loaded corpus with control + rule counts.
  - `kensa list sessions` (C-048) — surface session UUIDs
    from the transaction store for use with `kensa diff` and
    `kensa rollback --start`.
  - `kensa info` (C-047) — multi-criteria rule/control
    lookup. Four modes: `--rule R`, `--control FRAMEWORK:ID`,
    `--list-controls/-L FRAMEWORK`, positional `QUERY`. All
    pairwise mutually exclusive. Filters: `--cis` / `--stig`
    / `--nist` / `--rhel`.
  - `kensa diff SESSION1 SESSION2` (C-048) — per-rule drift
    report between two stored sessions. SESSION1 is "before";
    SESSION2 is "after" (git-diff convention). `--show-
    unchanged` includes the unchanged section in text output.
  - `kensa rollback --list / --info SESSION_ID / --start
    SESSION_ID / --detail` (C-049) — session-aware rollback.
    `--list` and `--info` are read-only; `--start` executes
    bulk rollback with a hostname guard. Legacy `--txn UUID`
    form preserved for surgical single-txn rollback. Only
    sessions created by `kensa remediate` are rollback-able
    (check sessions have no captured pre-state to revert).

- **Phase 5a operator surfaces (C-051 .. C-056).**
  - `--format jsonl` on `kensa history` (C-051), `kensa list
    sessions` (C-052), and `kensa info QUERY` (C-052). One
    compact JSON object per line — natural for piping into
    Splunk / ELK / Loki. Document-shaped modes (history's
    --aggregate / --stats / --txn; info's --rule / --control
    / --list-controls) reject `--format jsonl` with usage
    error pointing at `--format json`.
  - `kensa agent --stdio` placeholder (C-054) — reserves the
    subcommand name in v1.0. Exits 1 with "planned for v1.1
    with the kernel-primitive migration (Track L Phase 1)";
    `kensa agent --help` discloses the planned wire-protocol
    direction so consumers can write integration code today.
  - `kensa(1)` Unix manpage (C-055). Hand-written wrapper
    + generated flag body. `make manpage` produces
    `dist/kensa.1`; `make manpage-check` is the drift gate
    for CI. Source-of-truth committed at `man/kensa.1`.

### Changed

- `kensa history` paginated output trailer ("N of M transactions
  shown") now goes to stderr when `--format` is anything other
  than `text` (e.g., `json`, `csv`) so the trailer doesn't
  corrupt row-oriented formats. The `text` format still emits
  the trailer on stdout for human readability.

- `kensa coverage` → renamed to `kensa mechanisms` (C-044). The
  canonical name for the handler-mechanism listing is now
  `kensa mechanisms`. The `coverage` name remains a working
  alias today and emits a stderr warning explaining the
  upcoming v0.2 repurpose. **The `coverage` name is NOT being
  removed — it is being repurposed in v0.2 to report framework
  control coverage** (a different feature). Migrate scripts to
  `mechanisms` to preserve current output across the upgrade.
  Suppress the warning in pre-migrated CI with
  `KENSA_NO_REPURPOSE_WARNINGS=1` (a SEPARATE knob from the
  `KENSA_NO_DEPRECATION_WARNINGS` flag-rename switch — the
  semantic-flip warning is categorically louder).

### Deprecated

- `--format` / `-f` on `detect`, `check`, and `remediate`
  (C-020). Emits a one-line stderr warning when used. Use
  `--output` / `-o` instead. **Will be removed in v0.2.**

- `--oscal` on `remediate` (C-020). Emits a one-line stderr
  warning when used. Use `--output oscal:PATH` instead.
  **Will be removed in v0.2.**

  Migration:
  ```
  # before
  kensa check ... --format json
  kensa remediate ... --format json --oscal /tmp/asmt.json

  # after
  kensa check ... -o json
  kensa remediate ... -o json -o oscal:/tmp/asmt.json
  ```

  The deprecation warnings always emit on stderr regardless of
  `--quiet` so operators can't accidentally silence the
  migration signal.

  Operators who have planned the migration but can't migrate
  immediately can silence the warnings with the env var
  `KENSA_NO_DEPRECATION_WARNINGS=1` (exact match on "1"). Use
  `2>/dev/null` would silence real errors too; the env var is
  the targeted opt-out. NOT a substitute for migrating — the
  flags will still be removed in v0.2 regardless of whether
  the warning was visible.

### Fixed

- `kensa history --format csv` no longer corrupts the CSV file
  with a trailing pagination summary (the summary now goes to
  stderr instead of being mixed into stdout).

- `kensa remediate --oscal /path` no longer creates a 0-byte
  file when no transaction produced an envelope; the file is
  not created and a stderr line explains why.

- Long SCAP-style rule IDs in PDF reports (`xccdf_org.ssgproject.
  content_rule_…` 75-95 chars) wrap with character-level breaks
  instead of overflowing into adjacent table columns.

---

# Specter: Specter QuickStart
URL: https://www.hanalyx.com/docs/specter/quickstart

Get from zero to a working spec pipeline in under 5 minutes.

---

## 1. Install

**Fastest path — VS Code extension:** search `Specter SDD` in the Extensions panel, install, then run **Specter: Add CLI to Shell PATH** from the command palette once. The extension auto-downloads the CLI binary for your OS and architecture.

**CLI-only, macOS / Linux:**
```bash
OS=$(uname -s | tr '[:upper:]' '[:lower:]')
ARCH=$(uname -m); case "$ARCH" in x86_64) ARCH=amd64 ;; aarch64) ARCH=arm64 ;; esac
VERSION=$(curl -sL https://api.github.com/repos/Hanalyx/specter/releases/latest | grep '"tag_name"' | head -n1 | cut -d'"' -f4 | sed 's/^v//')
curl -LO "https://github.com/Hanalyx/specter/releases/download/v${VERSION}/specter_${VERSION}_${OS}_${ARCH}.tar.gz"
tar xzf "specter_${VERSION}_${OS}_${ARCH}.tar.gz" && sudo mv specter /usr/local/bin/
specter --version
```

**CLI-only, Windows (PowerShell):**
```powershell
$v = (Invoke-RestMethod https://api.github.com/repos/Hanalyx/specter/releases/latest).tag_name -replace '^v',''
Invoke-WebRequest -Uri "https://github.com/Hanalyx/specter/releases/download/v${v}/specter_${v}_windows_amd64.zip" -OutFile specter.zip
Expand-Archive specter.zip -DestinationPath "$env:USERPROFILE\.specter\bin"
[Environment]::SetEnvironmentVariable("Path", "$env:Path;$env:USERPROFILE\.specter\bin", "User")
```

For `.deb`, `.rpm`, and other install methods see the [Specter README](../README.md#install).

---

## 2. Bootstrap specs from your code

Point Specter at your source directory — it generates draft specs automatically:

```bash
specter reverse src/        # TypeScript / JavaScript
specter reverse app/        # Python / Django / FastAPI
specter reverse ./          # Go
```

This creates a `specs/` directory with one `.spec.yaml` per file group.

> **Already have specs?** Specter uses its own schema (see [Schema Reference](/docs/specter/spec-schema-reference)).
> Run `specter reverse --dry-run --json src/` to preview what it generates,
> then decide whether to migrate your existing specs or start fresh.

---

## 3. Initialize the workspace

```bash
specter init
```

Creates `specter.yaml` — the manifest that tells Specter (and the VS Code extension) where your specs and tests live.

---

## 4. Review one spec

Open any generated spec in VS Code. It will look like this:

```yaml
spec:
  id: user-create
  version: "1.0.0"
  status: draft
  tier: 2
  context:
    system: User service
  objective:
    summary: "Create a new user account"
  constraints:
    - id: C-01
      description: "POST /users accepts email and password"
  acceptance_criteria:
    - id: AC-01
      description: ""        # ← empty, needs your intent
      gap: true              # ← human review needed
```

The `gap: true` flag lives on individual acceptance criteria that the reverse compiler extracted structurally but couldn't infer intent for. Until those gaps are filled, the AC counts as uncovered (`specter sync` will fail on it), which is what forces the triage.

**Pass this prompt to your AI assistant:**

```
Here is a draft spec generated by Specter's reverse compiler:

[paste the spec]

Please:
1. Fill in all empty acceptance_criteria descriptions with clear, testable behavior
2. Add any constraints that are obviously missing based on the context
3. Remove the `gap: true` flag once all gaps are filled
4. Keep all existing IDs (C-01, AC-01, etc.) unchanged
5. Use MUST/MUST NOT language for constraints per RFC 2119
```

---

## 5. Annotate one test

Add two comment lines to an existing test:

```typescript
// @spec user-create
// @ac AC-01
test('creates user and returns 201', async () => { ... });
```

```python
# @spec user-create
# @ac AC-01
def test_create_user_returns_201(): ...
```

```go
// @spec user-create
// @ac AC-01
func TestCreateUser_Returns201(t *testing.T) { ... }
```

Source comments are enough for `specter coverage`. For `specter coverage --strict`, also expose `user-create/AC-01` in the test title or print `@spec` and `@ac` during the test before running `specter ingest`.

---

## 6. Check your coverage

```bash
specter coverage
```

```
Spec ID        Tier  ACs  Covered  Coverage  Status
user-create    T2    3    1        33%       FAIL   ← below T2's 80% threshold
  uncovered: AC-02, AC-03
```

A tier-2 spec needs 80% coverage to pass; tier-1 needs 100%; tier-3 needs 50%. Each `@ac` annotation you add moves the percentage up. Once `user-create` hits 80%+, status flips to `PASS` and `specter sync` can succeed.

---

## 7. Run the full pipeline

```bash
specter sync
```

```
PASS parse:    1 spec parsed
PASS resolve:  no dependency issues
PASS check:    0 errors
PASS coverage: thresholds met

All checks passed.
```

Add this to CI and you're protected.

---

## What's next?

- **[Getting Started](/docs/specter/getting-started)** — full walkthrough from zero specs to 100% coverage, with AI prompts for every step and VS Code workspace guide
- **[CLI Reference](/docs/specter/cli-reference)** — every command and flag
- **[AI Prompts](/docs/specter/ai-prompts)** — ready-to-use prompts for the full SDD loop
- **[FAQ](/docs/specter/faq)** — "Do I need to migrate my existing specs?"

---

# Specter: Getting Started with Specter
URL: https://www.hanalyx.com/docs/specter/getting-started

This guide takes you from **zero specs to full coverage** — step by step, with every command, every AI prompt, and a complete VS Code workspace walkthrough.

If you just want to see it work in 5 minutes, start with the [QuickStart](/docs/specter/quickstart) instead.

---

## Before You Begin

### What Specter is

Specter is a **type system for specs**. It validates, links, and checks `.spec.yaml` files the same way `tsc` validates `.ts` files. The core idea: your specification is the source of truth — not the code, not the tests, and not the AI output. Specter enforces that.

### What Specter is not

Specter is **not** a general-purpose spec format reader. If you already use specs — Gherkin/Cucumber `.feature` files, OpenAPI `.yaml`, Notion docs, Confluence pages — Specter does not read those formats. It uses its own structured schema (see [Schema Reference](/docs/specter/spec-schema-reference)).

**If you have existing specs in another format**, you have two options:

| Option | When to use |
|--------|-------------|
| **Run `specter reverse`** on your source code, then discard your old specs | Your existing specs are high-level or informal — the code is the real source of truth |
| **Migrate your existing specs** into Specter's schema manually or with AI | Your existing specs are detailed and authoritative — you want to preserve them |

For migration, use this AI prompt:

```
I have specifications in [Gherkin / OpenAPI / plain text / other format].
I want to migrate them to Specter's .spec.yaml format.

Specter's schema requires these top-level fields:
  spec.id — lowercase letters, digits, hyphens only, must start with a letter
            (e.g. "user-create", "payment-process"). No uppercase, no underscores.
  spec.version — quoted semver string: "1.0.0" (must be quoted or YAML parsing fails)
  spec.status — one of: draft | review | approved | deprecated | removed
  spec.tier — integer only, no quotes: 1 (Security/Money), 2 (Core Business), 3 (Utility)
  spec.context.system — string (name of the system or service)
  spec.objective.summary — string inside an objective object, not a bare string
  spec.constraints — array of objects. Required fields per item: id (format: C-01, C-02...),
                     description. Optional fields: type (technical | security | performance |
                     accessibility | business), enforcement (error | warning | info).
                     No other fields are allowed.
  spec.acceptance_criteria — array of objects. Required fields per item: id (format: AC-01,
                     AC-02...), description. Optional fields: references_constraints (array
                     of C-XX strings), inputs, expected_output, priority (critical | high |
                     medium | low). No other fields are allowed.

Here is my existing spec:
[paste your spec]

Please convert it to Specter's .spec.yaml format. Keep all the intent,
rewrite constraints to use MUST/MUST NOT/SHOULD language (RFC 2119),
and break acceptance criteria into individually testable AC-XX items.
```

---

## Phase 1 — Install

### CLI

Release assets follow the goreleaser convention `specter_<version>_<os>_<arch>.<ext>` — lowercase `linux`/`darwin`/`windows`, Go's `amd64`/`arm64` (not `uname`'s `x86_64`/`aarch64`). Each snippet below translates and picks the latest version automatically.

**macOS / Linux (tar.gz):**
```bash
OS=$(uname -s | tr '[:upper:]' '[:lower:]')
ARCH=$(uname -m); case "$ARCH" in x86_64) ARCH=amd64 ;; aarch64) ARCH=arm64 ;; esac
VERSION=$(curl -sL https://api.github.com/repos/Hanalyx/specter/releases/latest | grep '"tag_name"' | head -n1 | cut -d'"' -f4 | sed 's/^v//')
curl -LO "https://github.com/Hanalyx/specter/releases/download/v${VERSION}/specter_${VERSION}_${OS}_${ARCH}.tar.gz"
tar xzf "specter_${VERSION}_${OS}_${ARCH}.tar.gz" && sudo mv specter /usr/local/bin/
specter --version
```

**DEB package (Ubuntu/Debian):**
```bash
ARCH=$(dpkg --print-architecture)
VERSION=$(curl -sL https://api.github.com/repos/Hanalyx/specter/releases/latest | grep '"tag_name"' | head -n1 | cut -d'"' -f4 | sed 's/^v//')
curl -LO "https://github.com/Hanalyx/specter/releases/download/v${VERSION}/specter_${VERSION}_linux_${ARCH}.deb"
sudo dpkg -i "specter_${VERSION}_linux_${ARCH}.deb"
```

**RPM package (Fedora/RHEL):**
```bash
ARCH=$(uname -m); case "$ARCH" in x86_64) ARCH=amd64 ;; aarch64) ARCH=arm64 ;; esac
VERSION=$(curl -sL https://api.github.com/repos/Hanalyx/specter/releases/latest | grep '"tag_name"' | head -n1 | cut -d'"' -f4 | sed 's/^v//')
curl -LO "https://github.com/Hanalyx/specter/releases/download/v${VERSION}/specter_${VERSION}_linux_${ARCH}.rpm"
sudo rpm -i "specter_${VERSION}_linux_${ARCH}.rpm"
```

**Windows (PowerShell):**
```powershell
$v = (Invoke-RestMethod https://api.github.com/repos/Hanalyx/specter/releases/latest).tag_name -replace '^v',''
Invoke-WebRequest -Uri "https://github.com/Hanalyx/specter/releases/download/v${v}/specter_${v}_windows_amd64.zip" -OutFile specter.zip
Expand-Archive specter.zip -DestinationPath "$env:USERPROFILE\.specter\bin"
[Environment]::SetEnvironmentVariable("Path", "$env:Path;$env:USERPROFILE\.specter\bin", "User")
```

**Build from source (Go 1.22+):**
```bash
git clone https://github.com/Hanalyx/specter.git
cd specter/specter && make build
sudo mv bin/specter /usr/local/bin/
```

Verify (restart terminal first on Windows so `PATH` picks up):
```bash
specter --version
specter --help
```

### VS Code Extension

1. Open VS Code
2. Press `Ctrl+Shift+X` (Extensions panel)
3. Search `Specter SDD`
4. Click **Install**

The extension activates automatically once `specter.yaml` exists in your workspace (Phase 2, Step 3).

---

## Phase 2 — Bootstrap Your Specs

### Step 1 — Run the reverse compiler

Point Specter at your source directory. It analyzes your code and generates draft specs automatically:

```bash
specter reverse src/        # TypeScript / JavaScript / Next.js
specter reverse app/        # Python / Django / FastAPI
specter reverse ./          # Go
specter reverse packages/   # monorepo — point at the package root
```

What gets created:
```
specs/
  user-create.spec.yaml
  payment-process.spec.yaml
  auth-jwt.spec.yaml
  ...
```

Each spec reflects the structure Specter found in your code: routes, models, validation rules, and constraints. Some acceptance criteria may have `gap: true` at this stage. That is expected.

> **`gap: true` means:** Specter extracted the structure but could not infer the *intent*. A human (or AI) needs to complete it before it becomes authoritative.

### Step 2 — Initialize the workspace manifest

```bash
specter init
```

Creates `specter.yaml`:
```yaml
schema_version: 1
system:
  name: my-project
  tier: 2
domains:
  default:
    tier: 2
    description: Add spec IDs here as you create them. See the spec-manifest spec for the schema.
settings:
  specs_dir: specs
  coverage:
    tier1: 100
    tier2: 80
    tier3: 50
  exclude:
    - node_modules
    - dist
    - .git
    - vendor
```

This file is required for:
- The VS Code extension to activate
- `specter sync` to know where to look
- Discovery settings and coverage thresholds

Commit this file. It belongs in source control.

### Step 3 — Check the VS Code extension activated

Open VS Code in your project folder. Look at the activity bar (left sidebar) for the **Sp** icon. Click it — you should see the **Specter: Coverage** panel listing your specs with their current coverage percentages.

If you see: `Specter: no specter.yaml found in this workspace` — run `specter init` first (Step 2).

---

## Phase 3 — Close the Gaps

This is the most important phase. You are turning AI-extracted drafts into authoritative specifications.

### Step 1 — Run `specter check` to see the landscape

```bash
specter check
```

This reports structural issues: orphaned constraints (no AC references them), broken dependencies, missing required fields. Fix every `error` before moving on. `warn` items can be addressed incrementally.

### Step 2 — Review each spec with AI

Open a generated spec. It looks like this:

```yaml
spec:
  id: user-create
  version: "1.0.0"
  status: draft
  tier: 2
  context:
    system: User service
    description: "Handles user account creation"
  objective:
    summary: "Create a new user account"
  constraints:
    - id: C-01
      description: "POST /users accepts email and password"
  acceptance_criteria:
    - id: AC-01
      description: ""
      gap: true
    - id: AC-02
      description: ""
      gap: true
```

**AI prompt — fill the gaps:**

```
Here is a draft spec generated by Specter's reverse compiler for my [language/framework] codebase.

[paste the spec]

Please complete this spec:
1. Fill each empty acceptance_criteria description with a specific, testable behavior
   (e.g., "Valid email + password creates user and returns 201 with JWT token")
2. Add any obviously missing constraints based on the context
3. Add `references_constraints` arrays to each AC linking back to the C-XX IDs it validates
4. Set `status: draft` (leave it — we'll promote later)
5. Remove `gap: true` once all gaps are filled
6. Keep all existing IDs (C-01, AC-01, etc.) unchanged — do not renumber
7. Use MUST/MUST NOT/SHOULD language in constraint descriptions (RFC 2119)

Return only the completed YAML.
```

**AI prompt — add missing constraints:**

```
Review this Specter spec for [feature name]. Based on the context and objective,
what constraints are likely missing?

[paste the spec]

For each gap you identify, add a new constraint with:
- Sequential ID (next after the last C-XX)
- A MUST/MUST NOT description
- type: technical | security | performance | business
- enforcement: error | warning

Also add a corresponding AC that references it.
Return the updated YAML only.
```

### Step 3 — Validate each spec

After AI edits, validate before moving on:

```bash
specter parse specs/user-create.spec.yaml
```

```
PASS specs/user-create.spec.yaml — user-create@1.0.0
```

If it fails, Specter tells you exactly what's wrong:

```
FAIL specs/user-create.spec.yaml
  error [pattern] spec/constraints/0/id: must match "^C-\d{2,}$"
```

Common fixes:
- IDs must be `C-01` not `c1`, `C1`, or `constraint-1`
- Version must be quoted: `"1.0.0"` not `1.0.0`
- `tier` must be an integer: `2` not `"2"`
- `status` must be one of: `draft`, `review`, `approved`, `deprecated`, `removed`

### Step 4 — Repeat for all specs

```bash
specter parse    # validates all specs at once
specter check    # checks structural relationships
```

When both commands exit cleanly with no errors, Phase 3 is complete.

---

## Phase 4 — Write Tests Against the Specs

Specter reads test annotations from two places. Both matter.

1. **Source comments**: `// @spec <spec-id>` and `// @ac AC-NN` above the test function. Read by `specter coverage`.
2. **Runner-visible annotation**: the `<spec-id>/AC-NN` pair in the test title, or a `console.log('// @spec ...')` inside the test body. Read by `specter ingest` into `.specter-results.json`. Required by `specter coverage --strict`.

Source comments alone: `coverage` counts it, `--strict` demotes it. Write both forms so both commands work.

Full rules, per-language examples, parameterized tests, migration recipe, and troubleshooting: see [`TEST_ANNOTATION_REFERENCE.md`](/docs/specter/test-annotation-reference).

### Step 1 — See what's uncovered

```bash
specter coverage
```

```
Spec Coverage Report

Spec ID        Tier   ACs   Covered   Coverage   Status
------------------------------------------------------------
user-create    T2     4     0         0%         PASS
payment        T1     6     0         0%         FAIL  ← tier 1 needs 100%
auth-jwt       T2     5     0         0%         PASS
```

### Step 2 — Get AI to write annotated tests

Use `specter explain` to get a ready-to-copy annotation example for any AC:

```bash
specter explain user-create
```

Then pass the spec to your AI assistant:

**AI prompt — write tests for a spec:**

```
Here is a Specter spec:

[paste the spec]

Write tests in [TypeScript/Jest | Python/pytest | Go testing].

Rules:
1. One test per AC.
2. The test title carries [spec-id/AC-NN]:
   TypeScript:  it('[spec-id/AC-NN] brief description', () => { ... })
   Python:      use a runtime print/log form; pytest function names do not
                contain the `/` or `:` separator that ingest requires
   Go:          t.Run("spec-id/AC-NN brief description", ...)
   AC-NN is zero-padded: AC-01, not AC-1.
3. Above each test, add:
   // @spec [spec-id]
   // @ac [AC-NN]
4. Use [your test framework/mocks].
5. Tests are runnable. Use realistic inputs from the spec's `inputs` fields.

Return only the test code.
```

**TypeScript/Jest:**
```typescript
// @spec user-create
// @ac AC-01
test('[user-create/AC-01] valid email and password creates user and returns 201 with JWT', async () => {
  const res = await request(app).post('/users').send({
    email: 'alice@example.com',
    password: 'correct-horse-battery',
  });
  expect(res.status).toBe(201);
  expect(res.body).toHaveProperty('token');
});

// @spec user-create
// @ac AC-02
test('[user-create/AC-02] invalid email format returns 400', async () => {
  const res = await request(app).post('/users').send({
    email: 'not-an-email',
    password: 'correct-horse-battery',
  });
  expect(res.status).toBe(400);
  expect(res.body.error).toContain('email');
});
```

**Python/pytest** — Python function names cannot contain `/` or `:`. Use runtime output so `specter ingest` can read the pair from JUnit `<system-out>`.
```python
# @spec user-create
# @ac AC-01
def test_valid_registration_returns_201(client):
    print('// @spec user-create')
    print('// @ac AC-01')
    response = client.post('/users', json={
        'email': 'alice@example.com',
        'password': 'correct-horse-battery'
    })
    assert response.status_code == 201
    assert 'token' in response.json()

# @spec user-create
# @ac AC-02
def test_invalid_email_returns_400(client):
    print('// @spec user-create')
    print('// @ac AC-02')
    response = client.post('/users', json={
        'email': 'not-an-email',
        'password': 'correct-horse-battery'
    })
    assert response.status_code == 400
```

Run pytest with JUnit logging enabled:

```bash
pytest --junitxml=test-results.xml -o junit_logging=all -o junit_log_passing_tests=True
```

**Go** — use `t.Run` so each AC has its own runner-visible subtest title. `specter ingest` reads subtest names from `go test -json` output.
```go
// @spec user-create
// @ac AC-01
// @ac AC-02
func TestCreateUser(t *testing.T) {
    t.Run("user-create/AC-01 valid credentials returns 201 with JWT", func(t *testing.T) {
        body := `{"email":"alice@example.com","password":"correct-horse-battery"}`
        rec := httptest.NewRecorder()
        req := httptest.NewRequest(http.MethodPost, "/users", strings.NewReader(body))
        handler.ServeHTTP(rec, req)
        assert.Equal(t, http.StatusCreated, rec.Code)
        assert.Contains(t, rec.Body.String(), "token")
    })

    t.Run("user-create/AC-02 invalid email returns 400", func(t *testing.T) {
        body := `{"email":"not-an-email","password":"correct-horse-battery"}`
        rec := httptest.NewRecorder()
        req := httptest.NewRequest(http.MethodPost, "/users", strings.NewReader(body))
        handler.ServeHTTP(rec, req)
        assert.Equal(t, http.StatusBadRequest, rec.Code)
    })
}
```

### Step 3 — Check coverage after each test file

```bash
specter coverage
```

```
user-create    T2    4 ACs    2 covered    50%    PASS
```

Repeat until all tier 1 specs hit 100% and tier 2 specs hit 80%.

---

## Phase 5 — VS Code Workspace Walkthrough

With `specter.yaml` in place and specs annotated, the VS Code extension gives you real-time feedback as you write code.

### Coverage panel

Click the **Sp** icon in the activity bar. The **Specter: Coverage** panel shows every spec with its current coverage percentage. Red means below threshold. Click a spec to open it.

### Inline diagnostics

The extension underlines `@ac` annotations in test files when the referenced AC does not exist in any spec. This catches typos and stale references immediately — before CI.

### Run Sync from VS Code

Open the Command Palette (`Ctrl+Shift+P`), type **Specter: Run Sync**. This runs the full `specter sync` pipeline and reports results in the Output panel without leaving VS Code.

### Drift detection

When a spec changes (you edit a constraint or AC description), the extension highlights test files that reference that spec. This is the **intent drift** warning — your tests may no longer match the updated specification.

---

## Phase 6 — Lock It Into CI

Once `specter sync` passes locally, add it to your CI pipeline. This is the gate that prevents specs and tests from drifting apart on every PR.

**GitHub Actions (composite action, pinned version — preferred):**
```yaml
- uses: hanalyx/specter-sync-action@v1
  with:
    version: 0.9.2
```

**Or inline download (if you can't use the composite action):**
```yaml
- name: Install specter
  shell: bash
  run: |
    OS=$(echo "${{ runner.os }}" | tr '[:upper:]' '[:lower:]')
    case "${{ runner.arch }}" in X64) ARCH=amd64 ;; ARM64) ARCH=arm64 ;; esac
    VERSION=0.9.2   # pin a version; don't rely on "latest" in CI
    curl -LO "https://github.com/Hanalyx/specter/releases/download/v${VERSION}/specter_${VERSION}_${OS}_${ARCH}.tar.gz"
    tar xzf "specter_${VERSION}_${OS}_${ARCH}.tar.gz" && sudo mv specter /usr/local/bin/
- name: Specter sync
  run: specter sync
```

---

## Phase 7 — Promote Specs to Approved

When a spec is fully covered and reviewed by your team, promote it:

```yaml
spec:
  id: user-create
  status: approved    # ← was draft
```

`approved` means the team has accepted the spec as authoritative. Specter checks `draft` and `review` specs by default too. For release gates, set `settings.warn_on_draft: true` and `settings.strict: true` so drafts and warnings block the pipeline.

**AI prompt — review a spec before promotion:**

```
Review this Specter spec before we promote it from draft to approved:

[paste the spec]

Check for:
1. All constraints use RFC 2119 language (MUST/MUST NOT/SHOULD/MAY)
2. Every constraint is referenced by at least one AC
3. Every AC has a specific, testable description (not vague)
4. The objective scope clearly states what is excluded
5. Tier assignment is appropriate (1=Security/Money, 2=Core Business, 3=Utility)

Flag any issues. If it looks good, say so and I'll promote it.
```

---

## Troubleshooting

| Problem | Cause | Fix |
|---------|-------|-----|
| `Specter: no specter.yaml found` | Manifest missing | Run `specter init` |
| `error [required] spec/id` | Missing required field | Add the field; see [Schema Reference](/docs/specter/spec-schema-reference) |
| `error [pattern] spec/constraints/0/id` | Wrong ID format | Must be `C-01`, `C-02`, etc. |
| AC shows 0% after annotating tests | Annotation not found | Check `@spec` ID matches `spec.id` exactly; check `settings.tests_glob` or pass `--tests <glob>` |
| `specter reverse` generates too many specs | Large codebase | Use `--exclude` flag or add patterns to `specter.yaml` |
| Coverage drops after refactor | Tests deleted | Re-annotate new tests; run `specter coverage` to find the gap |

---

## Reference

- **[QuickStart](/docs/specter/quickstart)** — 5-minute path if you just want to see it work
- **[Spec Schema Reference](/docs/specter/spec-schema-reference)** — every field, type, and validation rule
- **[CLI Reference](/docs/specter/cli-reference)** — all commands and flags
- **[AI Prompts](/docs/specter/ai-prompts)** — ready-to-use prompts for the full SDD loop
- **[Specter's own specs](../specs/)** — production specs from Specter itself
- **[FAQ](/docs/specter/faq)** — common questions about SDD and Specter

---

# Specter: Specter CLI Reference
URL: https://www.hanalyx.com/docs/specter/cli-reference

Specter is a spec compiler toolchain — "a type system for specs." It validates, links, and type-checks `.spec.yaml` files the way `tsc` validates `.ts` files.

---

## Installation

Install the VS Code extension for the smoothest path — it auto-downloads the CLI and sets PATH. For CLI-only installs (tar.gz, `.deb`, `.rpm`, Windows zip, or build from source), see the [Install section in the Specter README](../README.md#install). Asset naming pattern: `specter_<version>_<os>_<arch>.<ext>` with lowercase `linux`/`darwin`/`windows` and `amd64`/`arm64`.

---

## Global Options

```
specter --version             Print the Specter version
specter --help                Show top-level help
specter <command> --help      Show help for a specific command
```

---

## Commands

### `specter parse`

Parse and validate `.spec.yaml` files against the canonical JSON Schema.

**Synopsis:**

```
specter parse [files...] [--json]
```

**Arguments:**

| Argument | Description |
|----------|-------------|
| `files...` | One or more `.spec.yaml` file paths. If omitted, discovers all `*.spec.yaml` files recursively from the current directory (or `specs_dir` from `specter.yaml`), skipping `testdata/` and configured excludes. |

**Options:**

| Option | Description |
|--------|-------------|
| `--json` | Output results as JSON instead of human-readable text. |

**Examples:**

```
$ specter parse
PASS specs/auth.spec.yaml — spec-auth@1.0.0
PASS specs/payments.spec.yaml — spec-payments@2.1.0

$ specter parse specs/auth.spec.yaml --json
{
  "file": "specs/auth.spec.yaml",
  "ok": true,
  "value": { "id": "spec-auth", "version": "1.0.0", ... }
}

$ specter parse broken.spec.yaml
FAIL broken.spec.yaml
  error [required] spec.id: must have required property 'id'
  error [pattern] spec.constraints[0].id: must match pattern "^C-\d{2,}$"
```

**Exit codes:** `0` = all files valid. `1` = one or more errors, or no files found.

---

### `specter resolve`

Build and validate the spec dependency graph. Constructs a directed acyclic graph from `depends_on` declarations and detects structural graph issues.

**Synopsis:**

```
specter resolve [--json] [--dot] [--mermaid]
```

**Options:**

| Option | Description |
|--------|-------------|
| `--json` | Output the graph and diagnostics as JSON. |
| `--dot` | Output the dependency graph in DOT format (for Graphviz). |
| `--mermaid` | Output the dependency graph in Mermaid format (renders natively in GitHub PRs). |

**Diagnostics:**

| Diagnostic | Description |
|------------|-------------|
| `circular_dependency` | Two or more specs form a cycle. Reports the full cycle path. |
| `dangling_reference` | A `depends_on.spec_id` does not match any discovered spec. Suggests similar IDs and a fix path. |
| `version_mismatch` | A `depends_on.version_range` is not satisfied by the target spec's actual version. |
| `duplicate_id` | Two spec files declare the same `id`. |

**Example:**

```
$ specter resolve
Spec Graph: 4 specs, 4 dependencies

Resolution order:
  spec-parse@1.0.0
  spec-resolve@1.0.0 -> spec-parse
  spec-check@1.0.0 -> spec-parse, spec-resolve
  spec-coverage@1.0.0 -> spec-parse

No dependency issues found.

$ specter resolve --mermaid
graph BT
    spec-parse["spec-parse@1.0.0"]
    spec-resolve["spec-resolve@1.0.0"]
    spec-resolve -->|"^1.0.0"| spec-parse
```

**Exit codes:** `0` = no issues. `1` = one or more errors.

#### `specter resolve dependents <spec-id>`

Reverse traversal of the dependency graph: returns all specs whose `depends_on` includes the given spec id (direct dependents only). Bare `specter resolve` keeps its build-and-validate behavior; this sub-subcommand switches to query mode.

Exit code 0 even when no dependents exist (an empty set is a valid result); non-zero only when the spec id does not exist in the resolved graph.

**Synopsis:**

```
specter resolve dependents <spec-id> [--json]
```

**Options:**

| Option | Description |
|--------|-------------|
| `--json` | Output as JSON: `{"spec_id": "<id>", "dependents": ["<id1>", ...]}` |

**Example:**

```
$ specter resolve dependents spec-parse
spec-check
spec-coverage
spec-doctor
spec-explain
spec-manifest
spec-resolve
spec-reverse
spec-sync
spec-vscode

$ specter resolve dependents leaf-spec     # no dependents → empty, exit 0
$ echo $?
0

$ specter resolve dependents unknown-spec  # not in graph → error, exit 1
error: spec "unknown-spec" not found in graph
```

Future operations (`dependencies` for forward traversal, `cycles` for cycle enumeration, etc.) follow the same `specter resolve <op>` pattern.

---

### `specter check`

Run structural type-checking rules across the spec dependency graph. Detects semantic consistency issues between connected specs.

**Synopsis:**

```
specter check [--json] [--tier <n>] [--strict] [--test]
```

**Options:**

| Option | Description |
|--------|-------------|
| `--json` | Output diagnostics as JSON. |
| `--tier <n>` | Override the tier enforcement level for all specs (1, 2, or 3). |
| `--strict` | Treat warnings as errors. Also configurable via `settings.strict` in `specter.yaml`. |
| `--test`, `-t` | Cross-reference test-file `@spec` / `@ac` annotations against parsed specs. |

**Diagnostics:**

| Diagnostic | Severity by tier | Description |
|------------|-----------------|-------------|
| `orphan_constraint` | T1=error, T2=warning, T3=info | A constraint is not referenced by any acceptance criterion. Individual constraints may override severity via `constraint.enforcement`. |
| `structural_conflict` | error (override via `constraint.enforcement`) | An upstream constraint requires something that a downstream AC handles as absent. |
| `tier_conflict` | warning | A higher-tier spec depends on a lower-tier spec (e.g., Tier 1 depends on Tier 3). |
| `unknown_spec_ref` | error (under `--test`) | A test annotates `@spec <id>` but no spec with that ID was parsed. Emitted only under `--test`. |
| `unknown_ac_ref` | error (under `--test`) | A test annotates `@ac AC-NN` but the spec has no AC with that ID. Emitted only under `--test`. |
| `unreachable_annotation` | by `settings.strictness`: annotation→suppressed, threshold→warning, zero-tolerance→error | Source-comment `@ac` whose enclosing test produces no runner-visible `<spec-id>/AC-NN` token (Convention A) and no runtime print (Convention B). Such annotations would silently demote under `coverage --strict`. Per-file off-switch: `// @reachable manual` (`# @reachable manual` for Python). Added in v0.13.0. |
| `unreachable_annotation_unknown` | warning (regardless of strictness) | The reachability scanner could not recognize the test shape (custom helper, non-Go/TS/Python language, dynamically-generated tests). Soft form of the above; never fails a gate. Same off-switch suppresses it. Added in v0.13.0. |

When a constraint has a `type` (e.g. `security`, `performance`), it appears in parentheses after the constraint ID so diagnostics can be grouped by category.

**Example:**

```
$ specter check
warn [orphan_constraint] spec-auth C-04 (security): C-04 is not referenced by any AC
error [tier_conflict] spec-payments: Tier 1 spec depends on Tier 3 spec-util

1 error(s), 1 warning(s), 0 info

$ specter check --strict
# Warnings are now treated as errors — exits 1
```

**Exit codes:** `0` = no errors (warnings allowed unless `--strict`). `1` = one or more errors.

---

### `specter coverage`

Generate a spec-to-test traceability matrix. Scans test files for `@spec` and `@ac` annotations and maps them to spec acceptance criteria. Enforces tier-based coverage thresholds.

**Synopsis:**

```
specter coverage [--json] [--failing] [--strict] [--scope <domain>] [--tests <glob>] [--strictness <level>] [--quiet]
```

**Options:**

| Option | Default | Description |
|--------|---------|-------------|
| `--json` | — | Output the coverage report as JSON. |
| `--failing` | — | Show only specs below 100% coverage in the table. Summary header still reflects the full report. When all specs are at 100%, emits a single-line confirmation instead of an empty table. Added in v0.9.2. |
| `--strict` | — | Require `.specter-results.json` and treat any annotated AC whose status is not `passed` as uncovered, across **all tiers**. Missing file is a hard failure; empty file emits a warning and proceeds. Pairs with `specter ingest`. Added in v0.10. |
| `--scope <domain>` | — | Narrow `--strict`'s demand to ACs of specs in the named `specter.yaml` domain. Specs outside the domain fall back to v0.9 boolean-passed logic. Enables staged adoption. Requires `--strict`; unknown domain fails fast. Added in v0.10. |
| `--strictness <level>` | manifest setting | Override `settings.strictness`. Values: `annotation`, `threshold`, `zero-tolerance`. `threshold` and `zero-tolerance` route through the same strict path as `--strict` — `.specter-results.json` is required (the error names the active mode and offers both remedies) and non-passed annotated ACs demote across all tiers. Because the manifest default is `threshold`, plain `specter coverage` behaves strictly unless `settings.strictness: annotation` is set. `annotation` keeps structural annotation counting. `--strict` enables the same strict path and is equivalent to `--strictness threshold` under the default manifest strictness; it does not override a manifest-set level (a `zero-tolerance` manifest keeps its zero-tolerance gates under `--strict`, and `--strict` with `strictness: annotation` is an error). |
| `--tests <glob>` | auto-discover | Glob pattern for test files. Default discovers `*.test.ts`, `*.test.js`, `*.test.py`, `*_test.go`, `*_test.py`. |
| `--quiet` | — | Suppress per-AC source-only hints under `--strict`. JSON output still includes `diagnostic_hints`. |

**Annotation format:**

Specter reads annotations from two places.

1. **Source comments** above the test function: `// @spec <id>` and `// @ac AC-NN`. `specter coverage --strictness annotation` counts these structurally.
2. **Test title or runtime log** carrying `<spec-id>/AC-NN`. `specter ingest` reads this. The strict path (`--strict`, or the default `threshold`/`zero-tolerance` strictness) requires it.

Source comments alone: `--strictness annotation` counts it; the strict path — including plain `specter coverage` under the manifest default `threshold` — demotes it. Write both forms.

For the full rules (regex contract, zero-padding, one-AC-per-test, per-runner examples, parameterized tests, Python limitation, migration recipe, troubleshooting), see [`TEST_ANNOTATION_REFERENCE.md`](/docs/specter/test-annotation-reference).

```typescript
// @spec user-registration
// @ac AC-01
test('[user-registration/AC-01] valid registration returns 201', () => { ... });
```

```python
# @spec user-registration
# @ac AC-01
def test_user_registration_AC_01_valid_returns_201():
    ...
```

```go
// @spec user-registration
// @ac AC-01
func TestUserRegistration(t *testing.T) {
    t.Run("user-registration/AC-01 valid returns 201", func(t *testing.T) {
        // ...
    })
}
```

**Rules for runner-visible annotations:**

- Spec id is kebab-case, lowercase: `[a-z][a-z0-9-]*[a-z0-9]`.
- AC id is zero-padded: `AC-01`, not `AC-1`. Must match the spec's AC id exactly.
- Separator between spec id and AC id is `/` or `:`.
- One test (or subtest) covers one `(spec-id, AC-NN)` pair. Do not put two ACs in one test.

**Alternate form — runtime log.** When you can't rename titles (shared naming, snapshot tests, external contracts), emit the pair from inside the test body:

```typescript
test('rejects zero amount', () => {
  console.log('// @spec payment-charge');
  console.log('// @ac AC-03');
  // assertions
});
```

```go
func TestCharge_ZeroAmount(t *testing.T) {
    t.Log("// @spec payment-charge")
    t.Log("// @ac AC-03")
    // assertions
}
```

Pick one form per file. Do not mix title-based and runtime-log forms in the same file.

**Coverage thresholds by tier:**

| Tier | Required Coverage |
|------|-------------------|
| 1 (Security / Money) | 100% |
| 2 (Core Business Logic) | 80% |
| 3 (Utility / Internal) | 50% |

**Example (table output, v0.9.2+):**

```
$ specter coverage

Spec Coverage Report — 2 specs · 83% avg coverage
  Tier 1: 0/1 passing (0%)
  Tier 2: 1/1 passing (100%)

Spec ID                                   Tier   ACs      Covered   Coverage   Status
----------------------------------------------------------------------------------
spec-auth                                 T1     6        4         67%        FAIL
  uncovered: AC-01, AC-03
spec-payments                             T2     5        5         100%       PASS

2 specs: 1 passing, 1 failing
```

**Table output shape (since v0.9.2):**

- A **summary header** precedes the table: total-specs count, arithmetic-mean coverage, and per-tier breakdown (`Tier K: X/Y passing (Z%)`). Tiers with zero specs in the workspace are omitted.
- Entries are **sorted worst-first**: failing (below threshold) → partial (below 100% but passing threshold) → 100% covered. Within each bucket, tier descending (T1 before T2 before T3) so higher-risk specs surface first.
- Spec IDs longer than 40 characters are **truncated** in the table with a trailing ellipsis (`…`). This keeps column alignment on workspaces with long path-derived IDs. The `--json` output is unaffected — it emits the full spec_id.

**Example (`--failing`, v0.9.2+):**

```
$ specter coverage --failing

Spec Coverage Report — 2 specs · 83% avg coverage
  Tier 1: 0/1 passing (0%)
  Tier 2: 1/1 passing (100%)

Spec ID                                   Tier   ACs      Covered   Coverage   Status
----------------------------------------------------------------------------------
spec-auth                                 T1     6        4         67%        FAIL
  uncovered: AC-01, AC-03

2 specs: 1 passing, 1 failing
```

When every spec is at 100%, `--failing` emits a single-line confirmation in place of the empty table:

```
$ specter coverage --failing

Spec Coverage Report — 2 specs · 100% avg coverage
  Tier 1: 1/1 passing (100%)
  Tier 2: 1/1 passing (100%)

All 2 specs at 100% coverage.
```

**Example (`--json`):**

Since v0.9.0, `--json` **always emits a CoverageReport JSON document to stdout**, including when one or more spec files fail to parse. The process exit code signals pass/fail; the presence of JSON does not. This is a breaking change from earlier versions which emitted no JSON on parse failure.

```json
{
  "entries": [
    {
      "spec_id": "spec-auth",
      "tier": 1,
      "total_acs": 6,
      "covered_acs": ["AC-01", "AC-02", "AC-03", "AC-04"],
      "uncovered_acs": ["AC-05", "AC-06"],
      "coverage_pct": 66.7,
      "threshold": 100,
      "passes_threshold": false,
      "test_files": ["tests/auth/login.test.ts"],
      "spec_file": "specs/spec-auth.spec.yaml"
    }
  ],
  "summary": {
    "total_specs": 1,
    "fully_covered": 0,
    "partially_covered": 1,
    "uncovered": 0,
    "passing": 0,
    "failing": 1
  },
  "spec_candidates_count": 1
}
```

When specs fail to parse, the report carries a `parse_errors` array and a grouped `parse_error_patterns` summary:

```json
{
  "entries": [],
  "summary": { "total_specs": 0, "passing": 0, "failing": 0, ... },
  "parse_errors": [
    {
      "file": "specs/broken.spec.yaml",
      "path": "spec.objective",
      "type": "required",
      "message": "Missing required field 'objective'",
      "line": 12,
      "column": 3
    }
  ],
  "parse_error_patterns": [
    {
      "type": "required",
      "path": "spec.objective",
      "count": 20,
      "example_file": "specs/auth.spec.yaml",
      "files": ["specs/auth.spec.yaml", "specs/payments.spec.yaml", ...]
    }
  ],
  "spec_candidates_count": 22
}
```

**Report field reference:**

| Top-level field | Since | Description |
|---|---|---|
| `entries[]` | v1.0 | One per parseable spec. Always present; may be empty. |
| `summary` | v1.0 | Roll-up counts. |
| `parse_errors` | v0.9.0 | Per-file schema violations. Absent or `null` when every spec parsed cleanly. |
| `parse_error_patterns` | v0.9.0 | `parse_errors` grouped by `(type, path)` sorted by count desc. Useful for naming schema drift ("20 specs missing `objective`"). |
| `spec_candidates_count` | v0.9.0 | Number of `.spec.yaml` files discovered on disk before parsing. Distinguishes "no specs exist" (count 0) from "specs exist but failed to parse" (count > 0, entries empty). |

| Entry field | Since | Description |
|---|---|---|
| `spec_file` | v0.9.0 | Path to the source `.spec.yaml` for this entry. Lets downstream consumers open the file. |

**Exit codes:**
- `0` — all specs parsed AND all meet their coverage thresholds
- `1` — one or more specs failed to parse, OR one or more specs are below threshold, OR `.specter-results.json` is missing under a strict mode (`--strict`, or effective strictness `threshold`/`zero-tolerance`)
- `2` — zero-tolerance strictness: an annotated AC has a results-file status other than `passed`
- `3` — zero-tolerance strictness: an AC carries `approval_gate: true` with an unset `approval_date`

**Consuming the JSON programmatically:**

Since v0.9.0 emits JSON in every state, the pattern for scripting is:

```bash
specter coverage --json > /tmp/cov.json
rc=$?
if [ "$(jq '.parse_errors | length' /tmp/cov.json)" -gt 0 ]; then
  echo "Parse errors — fix spec files first"
  exit 2
elif [ $rc -ne 0 ]; then
  echo "Coverage below threshold"
  exit 1
fi
```

---

### `specter sync`

Run the full validation pipeline: parse → resolve → check → coverage. Exits non-zero on any failure. This is the CI gate command.

**Synopsis:**

```
specter sync [--json] [--tests <glob>] [--only <phase>] [--strict] [--strictness <level>]
```

**Options:**

| Option | Description |
|--------|-------------|
| `--json` | Output the pipeline result as JSON. |
| `--tests <glob>` | Glob pattern for test files. |
| `--only <phase>` | Run only one phase: `parse`, `resolve`, `check`, or `coverage`. Prerequisites run without halting on failure. |
| `--strict` | Treat warnings as errors. Alias for `--strictness zero-tolerance` when `--strictness` is not set. |
| `--strictness <level>` | Override `settings.strictness` for the coverage phase. Values: `annotation`, `threshold`, `zero-tolerance`. Matches `coverage --strictness` semantics exactly — sync's coverage phase delegates to the strict path so demotions match. When both `--strict` and `--strictness` are passed, `--strictness` wins. |

**Example:**

```
$ specter sync

Specter Sync

  PASS parse: 5 spec(s) parsed — no schema violations
  PASS resolve: 5 specs, 8 dependencies — no cycles or broken refs
  PASS check: 0 errors, 0 orphan constraints
  PASS coverage: 5 spec(s) meet coverage thresholds

All checks passed.
```

**CI integration (GitHub Actions):**

```yaml
- name: Validate specs
  run: specter sync
```

**Exit codes:** `0` = all phases pass. `1` = any phase fails. Under an effective `zero-tolerance` strictness (the `--strictness` flag, the `--strict` alias, or the manifest setting), the coverage phase refines the failure exit to match `coverage`: `2` = an annotated AC did not pass, `3` = `approval_gate: true` with unset `approval_date`.

---

### `specter reverse`

Extract draft `.spec.yaml` files from existing source code. Analyzes source and test files using language-specific adapters to extract constraints from validation schemas and acceptance criteria from test assertions.

**Synopsis:**

```
specter reverse [path] [--adapter <lang>] [--output <dir>] [--group-by <strategy>]
                [--dry-run] [--overwrite] [--exclude <pattern>] [--json]
```

**Arguments:**

| Argument | Description |
|----------|-------------|
| `path` | Directory to analyze (default: `.`). |

**Options:**

| Option | Default | Description |
|--------|---------|-------------|
| `--adapter <lang>` | auto | Language adapter: `typescript`, `python`, `go`. Auto-detects from file extensions if omitted. |
| `--output <dir>` / `-o` | `specs` | Output directory for generated `.spec.yaml` files. |
| `--group-by <strategy>` | `file` | Grouping strategy: `file` (one spec per source file) or `directory` (one spec per directory). |
| `--dry-run` | false | Preview generated YAML to stdout without writing files. |
| `--overwrite` | false | Overwrite existing spec files. Default skips files that already exist. |
| `--exclude <pattern>` | — | Exclude paths matching pattern. Can be repeated. |
| `--json` | false | Output results as JSON. |

**Example:**

```
$ specter reverse src/auth --output specs/auth
GENERATED specs/auth/auth-service.spec.yaml — auth-service@1.0.0 (3 constraints, 5 ACs)
  warning: AC-03 description is a gap — review and complete manually

Summary: 1 spec(s) generated, 3 constraint(s), 4 assertion(s), 1 gap(s)

DRAFT: 1 AC(s) require manual review — specter reverse can extract structure but not intent.
       Review each gap and fill in description, inputs, and expected_output.

$ specter reverse --dry-run  # Preview without writing
```

**Supported languages:** TypeScript, Python, Go. Extracts constraints from Zod/Yup schemas (TypeScript), Pydantic models (Python), and validation logic (Go).

**Exit codes:** `0` = one or more specs generated. `1` = no specs generated.

---

### `specter init`

Initialize a `specter.yaml` project manifest, or scaffold a draft `.spec.yaml` from a template.

**Synopsis:**

```
specter init [--name <name>] [--force] [--template <type>]
specter init --refresh [--dry-run]
specter init --install-hook
specter init --ai <tool>
```

**Options:**

| Option | Description |
|--------|-------------|
| `--name <name>` | System name for the manifest. Defaults to the current directory name. |
| `--force` | Overwrite an existing `specter.yaml`. Mutually exclusive with `--refresh`. |
| `--template <type>` | Create a draft `.spec.yaml` from a template instead of a manifest. Types: `api-endpoint`, `service`, `auth`, `data-model`. |
| `--refresh` | Update only `domains.default.specs` in an existing `specter.yaml`. Preserves every other field — `settings`, `registry`, tier overrides, custom domains. Added in v0.9.2. |
| `--dry-run` | Used with `--refresh`: print the proposed diff to stdout without writing the file. Added in v0.9.2. |
| `--install-hook` | Install a git pre-push hook that blocks implementation-only pushes with no `@spec` / `@ac` annotation delta. |
| `--ai <tool>` | Write an AI assistant instruction file. Values: `claude`, `codex`, `cursor`, `copilot`, `gemini`. |

**Behaviour (v0.9.0+):**

`specter init` scans the workspace's `specs/` directory and populates the manifest's default domain based on what it finds.

- **Greenfield workspace (no spec files):** emits a manifest with an empty `domains.default` entry whose description invites you to add spec IDs as you author them.
- **Workspace with parseable specs:** reads each one, extracts its `spec.id`, and populates `domains.default.specs: [...]`.
- **Workspace with specs that fail to parse:** still writes the manifest (with an explanatory placeholder default domain) and prints a warning that includes a **Pattern analysis** block naming the shape of the failure — if every discovered spec hit the same error, init calls out schema version drift and points at `specter doctor` for deeper diagnosis.

**Important (v0.9.0+):** init always emits a `domains:` section, even in the greenfield case. Previous versions omitted `domains:` entirely when no spec IDs were discovered, which caused later `specter sync` runs to silently skip every spec the user added afterward — a silent-exclusion footgun now eliminated.

**Example (greenfield):**

```
$ specter init
Created specter.yaml with 0 spec(s) in system "my-project"
```

**Example (existing parseable specs):**

```
$ specter init
Created specter.yaml with 14 spec(s) in system "specter"
```

**Example (existing specs with schema drift):**

```
$ specter init
Created specter.yaml with 0 spec(s) in system "my-project"

Warning: 22 spec file(s) were discovered but could not be parsed:
  Every failing spec hit the same error: [additionalProperties] at "spec".
  This is the signature of schema version drift — the specs may
  have been written against an older Specter schema. Run `specter
  doctor` for a full report, then fix the specs and re-run
  `specter init --force` to populate the manifest.

The manifest was still written with an empty default domain as a
placeholder. Add your spec IDs under `domains.default.specs` once
the parse errors are resolved.
```

**Refresh mode (v0.9.2+):**

`specter init --refresh` is the non-destructive counterpart to `--force`. It reads the existing `specter.yaml`, rescans `settings.specs_dir` (or default `specs/`), and updates **only** `domains.default.specs` with the current on-disk spec set. Every other field is preserved — `settings`, `registry`, system metadata, and any custom domains declared under `domains.<name>` (anything that isn't `default`).

Specs claimed by a custom domain (listed under a non-default `domains.<name>.specs`) stay in that domain and are **not** migrated into `default`. A spec belongs to exactly one domain.

Specs that were previously listed in `domains.default.specs` but are no longer discoverable on disk (deleted, renamed, or now failing to parse) are removed from the list. The summary line reports the change counts.

**Example (`--refresh`):**

```
$ specter init --refresh
updated specter.yaml: +1 added, -0 removed
```

**Example (`--refresh --dry-run`):**

Prints the proposed diff without writing. The file on disk is byte-identical before and after. Useful for review before committing.

```
$ specter init --refresh --dry-run
Dry run — no changes will be written.

Proposed changes to domains.default.specs:
  + spec-payments
  - spec-legacy-auth

Run `specter init --refresh` (without --dry-run) to apply.
```

**Flag conflict:** `--refresh` and `--force` are mutually exclusive. `--force` rewrites the entire manifest; `--refresh` is surgical. Combining them exits non-zero with a clear error.

**Example (template):**

```
$ specter init --template api-endpoint
Created api-endpoint.spec.yaml (template: api-endpoint)
Edit the file to replace placeholder values, then run: specter sync
```

---

### `specter doctor`

Run pre-flight project health checks before the full pipeline. Reports `PASS`, `WARN`, or `FAIL` for each check.

**Synopsis:**

```
specter doctor [--fix] [--dry-run] [--yes]
```

**Options:**

| Option | Description |
|--------|-------------|
| `--fix` | Apply known-safe schema-drift rewrites in place. Currently beta-gated. |
| `--dry-run` | With `--fix`, print what would change without writing files. Skips the beta prompt. |
| `--yes`, `-y` | Skip the `--fix` beta confirmation prompt for non-interactive use. |

**Checks performed:**

| Check | PASS | WARN | FAIL |
|-------|------|------|------|
| `manifest` | `specter.yaml` found | No `specter.yaml` (optional) | — |
| `spec-files` | ≥1 `.spec.yaml` found | — | No spec files found |
| `parse` | All specs parse cleanly | — | Parse errors in ≥1 spec |
| `annotations` | `@spec`/`@ac` annotations found in tests | No annotations found | — |
| `coverage` | All specs meet tier thresholds | — | ≥1 spec below threshold |

**Example (happy path):**

```
$ specter doctor

specter doctor

  manifest     [PASS]  specter.yaml found at specter.yaml
  spec-files   [PASS]  5 spec file(s) discovered
  parse        [PASS]  All specs parse cleanly
  annotations  [WARN]  No @spec/@ac annotations found in test files
  coverage     [WARN]  No specs to check coverage for

Result: OK — project is ready for `specter sync`
```

**Pattern analysis on parse failure (v0.9.0+):**

When the parse check fails, `specter doctor` prints a **Pattern analysis** block that groups errors by `(type, path)`. If every discovered spec hit the same pattern, doctor names it explicitly as the signature of schema version drift — a common shape for projects whose specs predate the current schema.

```
$ specter doctor

  manifest     [PASS]  specter.yaml found at specter.yaml
  spec-files   [PASS]  22 spec file(s) discovered
    specs/auth.spec.yaml: Unknown field 'trust_level'. Remove it or check for a typo in the field name.
    specs/payments.spec.yaml: Unknown field 'trust_level'. Remove it or check for a typo in the field name.
    ...
  parse        [FAIL]  22 spec file(s) have parse errors (see above)

  Pattern analysis:
    Every 22 discovered spec hit the same failure: [additionalProperties] at "spec".
    This pattern is the signature of schema version drift —
    your specs may have been written against an older Specter
    schema. Check the spec-parse changelog and migrate each file.

  annotations  [PASS]  8 annotation(s) found across 45 test file(s)
  coverage     [WARN]  Skipping coverage check — specs have parse errors

Result: FAIL — fix the issues above before running `specter sync`
```

When errors are heterogeneous (multiple distinct failure shapes), doctor lists the top patterns with counts instead of claiming drift:

```
  Pattern analysis:
    [required] at "spec.objective" — 12 occurrence(s) across 12 file(s)
    [enum] at "spec.status" — 3 occurrence(s) across 3 file(s)
    [additionalProperties] at "spec" — 2 occurrence(s) across 2 file(s)
```

**Exit codes:** `0` = all checks PASS or WARN. `1` = any check FAIL.

---

### `specter explain`

Show coverage status and annotation examples for a spec's acceptance criteria.

**Synopsis:**

```
specter explain <spec-id>[:<ac-id>]
```

**Arguments:**

| Argument | Description |
|----------|-------------|
| `<spec-id>` | The spec ID to explain. Lists all ACs with COVERED/UNCOVERED status. |
| `<spec-id>:<ac-id>` | Show full details and annotation example for one AC. |

**Example:**

```
$ specter explain user-registration

specter explain user-registration

  Status   AC        Description
  ------------------------------------------------------------
  COVERED  AC-01     Valid email and password creates user and...
  UNCOVERED AC-02    Invalid email format returns 400...
  UNCOVERED AC-03    Weak password returns 400...

  Scanned 12 test file(s)
  Run `specter explain user-registration:<ac-id>` for annotation examples

$ specter explain user-registration:AC-02

specter explain user-registration:AC-02

  Spec:   user-registration (v1.0.0, tier 1)
  AC-02:  Invalid email format returns 400 with field-level error
  Status: UNCOVERED

  To cover this AC, add annotations in your test file:

  TypeScript / JavaScript:
    // @spec user-registration
    // @ac AC-02
    it('AC-02: Invalid email format returns 400 with field-level error', () => {
      // test implementation
    });
```

---

### `specter watch`

Re-run the full sync pipeline whenever spec or test files change. Uses `fsnotify` with a 150ms debounce to coalesce rapid saves.

**Synopsis:**

```
specter watch
```

Runs once immediately on startup, then re-runs on every `.spec.yaml` or test file change. Press `Ctrl+C` to stop.

**Example:**

```
$ specter watch

specter watch

  Watching: specs, test files
  Press Ctrl+C to stop

[14:32:01] PASS  5 spec(s)  33/33 ACs covered  (5 passing, 0 failing)
[14:32:15] FAIL  parse
[14:32:22] PASS  5 spec(s)  33/33 ACs covered  (5 passing, 0 failing)
```

---

### `specter diff`

Polymorphic diff verb — the single command for diffing any Specter artifact. Dispatches on an optional first `<kind>` argument; defaults to the `spec` kind for backward compat with v1.x.

**Synopsis:**

```
specter diff <path>[@<ref>] <path>[@<ref>]              # spec kind (implicit; backward compat)
specter diff spec <path>[@<ref>] <path>[@<ref>]         # spec kind (explicit)
specter diff coverage <baseline.json> <current.json>    # coverage kind
```

**Kinds:**

| Kind | Purpose |
|------|---------|
| `spec` | Semantic diff between two spec versions (v1.x behavior; default when no kind argument is present). Classifies as `breaking`, `additive`, `patch`, or `unchanged`. |
| `coverage` | Per-spec AC delta between two `coverage --json` snapshots. Useful for tracking coverage drift across CI runs. |

Future cycles add more kinds (e.g., `ingest`, `check`) under the same `specter diff <kind>` grammar. New diffable artifacts MUST NOT introduce a per-subcommand `--diff` flag — they land as kinds here.

**spec kind — change classes:**

| Class | Meaning |
|-------|---------|
| `breaking` | ACs or constraints removed, or descriptions changed in a way that narrows the contract. Requires a MAJOR version bump. |
| `additive` | New ACs or constraints added. Requires a MINOR version bump. |
| `patch` | Wording-only changes that don't alter meaning. PATCH version bump. |
| `unchanged` | No changes detected. |

**Example — spec kind:**

```
$ specter diff specs/auth.spec.yaml@HEAD~3 specs/auth.spec.yaml

spec spec-auth 1.0.0 → 1.1.0 [additive]

  +AC-05: Returns 401 when token is expired
  ~C-02: MUST require 8-character passwords → MUST require 12-character passwords

$ specter diff specs/auth.spec.yaml specs/auth.spec.yaml
spec spec-auth 1.1.0 → 1.1.0: no changes
```

**Example — coverage kind:**

```
$ specter coverage --json > baseline.json   # later, after changes:
$ specter coverage --json > current.json
$ specter diff coverage baseline.json current.json

+spec spec-new-feature
+spec-auth/AC-05
-spec-auth/AC-03
~spec-auth coverage_pct: 80.0 → 90.0 (passes_threshold: true → true)
```

Exit code is always 0 for both kinds — diff is a diagnostic surface, not a gate.

---

### `specter ingest`

Convert CI-native test output (JUnit XML, `go test -json`) into `.specter-results.json`, the file `specter coverage --strict` reads to demote annotated-but-failing ACs. Added in v0.10.

**Synopsis:**

```
specter ingest [--junit <path>] [--go-test <path>] [--output <path>] [--verbose]
```

At least one of `--junit` or `--go-test` is required. Both flags accept glob patterns and may be repeated. Multiple sources can be combined in one invocation; results are merged by the worst-status-wins rule.

**Options:**

| Option | Default | Description |
|--------|---------|-------------|
| `--junit <path>` | — | JUnit XML file (vitest, jest, pytest, playwright). Accepts glob patterns; may be repeated. |
| `--go-test <path>` | — | Newline-delimited JSON from `go test -json`. Accepts glob patterns; may be repeated. |
| `--output <path>` | `.specter-results.json` | Where to write the merged results. |
| `--verbose` | — | Emit one stderr line per dropped testcase (testcases without a recognizable `(spec_id, ac_id)` annotation). Off by default; the summary line is always emitted. |

**Diagnostics:** every run writes to stderr a summary line:

```
Scanned N test cases; extracted M (spec_id, ac_id) pairs; dropped K with no runner-visible annotation.
```

If `M` is 0 despite `N` being non-zero, your tests carry annotations only in source comments — those are invisible to `ingest` by design. See the explainer's Conventions A (test title) and B (runtime `t.Log`) for migrating.

**Annotation extraction:**

Each test needs a discoverable `(spec_id, ac_id)` pair or it's dropped silently. Sources in order of preference:

1. **Test name** — `spec-id/AC-NN` or `spec-id:AC-NN` embedded in the test case name.
2. **Classname** — same pattern, parsed from the JUnit `classname` attribute.
3. **Test body** — `// @spec <id>` and `// @ac <AC-id>` comments surfaced via `system-out` (JUnit) or `output`-action lines (go test -json).

**Status mapping:**

| Source | Maps to |
|--------|---------|
| JUnit `<testcase>` with no children | `passed` |
| JUnit `<failure>` child | `failed` |
| JUnit `<skipped>` child | `skipped` |
| JUnit `<error>` child | `errored` |
| go test `{"Action":"pass"}` | `passed` |
| go test `{"Action":"fail"}` | `failed` |
| go test `{"Action":"skip"}` | `skipped` |

**Worst-status rule:** when the same `(spec_id, ac_id)` is hit by multiple tests, the emitted entry uses the worst observed status: `errored > failed > skipped > passed`. One failing test is sufficient to demote an AC.

**Example (CI):**

```yaml
# run tests, emit JUnit
- run: pytest --junitxml=test-results/pytest.xml
- run: vitest run --reporter=junit > test-results/vitest.xml

# ingest, then gate
- run: specter ingest --junit 'test-results/*.xml' --output .specter-results.json
- run: specter coverage --strict
```

**Example (local):**

```bash
$ go test -json ./... > /tmp/go-test.json
$ specter ingest --go-test /tmp/go-test.json
Wrote 34 result entries to .specter-results.json

$ specter coverage --strict
Spec Coverage Report — 15 specs · 99% avg coverage
  Tier 1: 4/4 passing (100%)
  Tier 2: 9/9 passing (100%)
  Tier 3: 2/2 passing (100%)
...
```

Pairs with `specter coverage --strict`. Without `ingest`, `--strict` fails with `--strict requires .specter-results.json — run 'specter ingest' first`.

---

## The `specter.yaml` Manifest

An optional `specter.yaml` file at the project root configures discovery, thresholds, and settings. Specter searches upward from the current directory to find it.

```yaml
schema_version: 1
system:
  name: my-project
  tier: 2

domains:
  default:
    tier: 2
    description: Default domain for my-project specs
    specs:
      - user-create

settings:
  specs_dir: specs
  strict: false
  warn_on_draft: false
  strictness: threshold
  tests_glob:
    - "**/*_test.go"
    - "**/*.test.ts"
  coverage:
    tier1: 100
    tier2: 80
    tier3: 50
  exclude:
    - node_modules
    - dist
    - .git
    - vendor
```

---

## `.specterignore` File

An optional `.specterignore` file in the project root controls which paths are skipped during spec discovery. Follows `.gitignore` conventions.

```
# Ignore test fixtures
testdata/

# Ignore generated specs
specs/generated/
```

---

## Exit Codes

| Code | Meaning |
|------|---------|
| `0` | All checks passed. |
| `1` | One or more errors, or no spec files found. |

---

# Specter: Spec Schema Reference
URL: https://www.hanalyx.com/docs/specter/spec-schema-reference

> Canonical reference for `.spec.yaml` files validated by Specter.
> Schema source: `internal/parser/spec-schema.json` (v1.0.0)

---

## Table of Contents

- [Document Structure](#document-structure)
- [Required Fields](#required-fields)
- [Optional Fields](#optional-fields)
- [Context Object](#context-object)
- [Objective Object](#objective-object)
- [Constraint Object](#constraint-object)
- [Constraint Validation Object](#constraint-validation-object)
- [Acceptance Criterion Object](#acceptance-criterion-object)
- [Error Case Object](#error-case-object)
- [Dependency Reference Object](#dependency-reference-object)
- [Environment Object](#environment-object)
- [Changelog Entry Object](#changelog-entry-object)
- [Changelog Change Object](#changelog-change-object)
- [Generated From Object](#generated-from-object)
- [Naming Conventions](#naming-conventions)
- [Versioning Rules](#versioning-rules)
- [Tier Definitions](#tier-definitions)
- [Status Lifecycle](#status-lifecycle)
- [Worked Examples](#worked-examples)

---

## Document Structure

Every `.spec.yaml` file is a YAML document with a single top-level key: `spec`. All fields live inside this wrapper. No additional top-level keys are permitted.

```yaml
spec:
  id: my-feature
  version: "1.0.0"
  status: draft
  tier: 2
  context: { ... }
  objective: { ... }
  constraints: [ ... ]
  acceptance_criteria: [ ... ]
  # optional fields follow
```

The `spec:` wrapper exists so the schema can be extended in the future (e.g., adding a top-level `meta:` or `tooling:` key) without breaking existing files.

---

## Required Fields

These fields must be present inside `spec:`. Omitting any of them is a validation error.

| Field | Type | Format / Validation | Description |
|---|---|---|---|
| `id` | `string` | Regex: `^[a-z][a-z0-9-]*$` (kebab-case, starts with letter) | Unique identifier for this spec. Used in `depends_on` references and test annotations. |
| `version` | `string` | Semver: `MAJOR.MINOR.PATCH` with optional pre-release tag. Regex: `^(0\|[1-9]\d*)\.(0\|[1-9]\d*)\.(0\|[1-9]\d*)(-[a-zA-Z0-9.]+)?$` | Semantic version of the spec. Must be quoted in YAML to avoid float interpretation. |
| `status` | `string` | Enum: `draft`, `review`, `approved`, `deprecated`, `removed` | Lifecycle status. Specter parses and checks all discovered specs. Use `settings.warn_on_draft` and `settings.strict` when draft specs should block release gates. |
| `tier` | `integer` | Enum: `1`, `2`, `3` | Risk tier. Determines enforcement strictness and coverage thresholds. |
| `context` | `object` | See [Context Object](#context-object) | What system this spec belongs to and why it exists. |
| `objective` | `object` | See [Objective Object](#objective-object) | What this spec aims to achieve, including scope boundaries. |
| `constraints` | `array` | Minimum 1 item. Items: [Constraint](#constraint-object) | Inviolable rules. Each is a hard boundary on the solution space. |
| `acceptance_criteria` | `array` | Minimum 1 item. Items: [Acceptance Criterion](#acceptance-criterion-object) | Testable conditions that define "done". Each AC maps to at least one test. |

### Field Examples

```yaml
spec:
  id: payment-create-intent         # kebab-case, starts with letter
  version: "1.0.0"                  # always quote — YAML parses 1.0 as float
  status: approved                  # no quotes needed for enum values
  tier: 1                           # integer, not string
```

---

## Optional Fields

These fields may be omitted entirely. When absent, Specter does not supply defaults at the document level (except where noted in sub-objects).

| Field | Type | Description |
|---|---|---|
| `title` | `string` | Human-readable display name. When omitted, consumers should fall back to `id`. Added in v0.7.0. |
| `coverage_threshold` | `integer` | Per-spec coverage threshold override, 0-100. Overrides the tier default for this spec only. |
| `depends_on` | `array` of [Dependency Reference](#dependency-reference-object) | Other specs this depends on. Creates edges in the dependency graph. |
| `environment` | [Environment Object](#environment-object) | Required environment variables and deployment targets. |
| `tags` | `array` of `string` | Free-form tags for categorization and filtering. |
| `changelog` | `array` of [Changelog Entry](#changelog-entry-object) | Version history. Most recent entry first. |
| `generated_from` | [Generated From Object](#generated-from-object) | Provenance tracking. Present only on reverse-compiled specs. |

---

## Context Object

Describes **what system** this spec belongs to and **why it exists**.

| Field | Required | Type | Description |
|---|---|---|---|
| `system` | **Yes** | `string` | What system or service does this spec belong to? |
| `feature` | No | `string` | What feature area within the system? |
| `description` | No | `string` | Plain-language overview of why this spec exists. |
| `dependencies` | No | `array` of `string` | External dependencies (libraries, services, APIs). |
| `existing_patterns` | No | `string` | Relevant coding patterns, conventions, or architectural decisions. |
| `related_specs` | No | `array` of `string` | Other spec files that interact with this one. |
| `assumptions` | No | `array` of `string` | Things taken as given. If wrong, the spec needs revision. |

**Note:** All fields in the context object are declared above. Unknown fields are rejected at parse time — `specter parse` returns an error naming the offending key. If you need to capture additional metadata, use the spec-level `tags` array for categorical data, or `context.description` for free-form narrative. Extension by adding ad-hoc keys to `context` is not supported; propose a new schema field instead. (Changed in v0.7.0 — earlier versions silently dropped unknown context keys via `additionalProperties: true`, causing data loss for AI-assisted migrations.)

```yaml
context:
  system: Specter toolchain
  feature: YAML-to-AST parser
  description: >
    The foundational tool in the Specter toolchain. Parses .spec.yaml files
    and produces typed SpecAST objects.
  dependencies:
    - "yaml (eemeli/yaml 2.x)"
    - "ajv (8.x)"
  assumptions:
    - "Input files are UTF-8 encoded YAML"
```

---

## Objective Object

Describes **what this spec aims to achieve**. Uses the Delta Principle: describe the *change*, not the *state*.

| Field | Required | Type | Description |
|---|---|---|---|
| `summary` | **Yes** | `string` | 1-3 sentence description of what this spec achieves. |
| `scope` | No | `object` | Explicit scope boundaries. |
| `scope.includes` | No | `array` of `string` | What is in scope. |
| `scope.excludes` | No | `array` of `string` | What is out of scope. Prevents AI scope creep. |

`additionalProperties` is `false` on both the objective and scope objects.

```yaml
objective:
  summary: >
    Parse .spec.yaml files into validated, typed SpecAST objects.
    Reject malformed specs with actionable error messages.
  scope:
    includes:
      - "YAML parsing with syntax error handling"
      - "JSON Schema validation"
    excludes:
      - "Dependency resolution (that is spec-resolve)"
```

---

## Constraint Object

A constraint is an **inviolable rule** -- a hard boundary on the solution space. Every spec must have at least one.

| Field | Required | Type | Format / Validation | Description |
|---|---|---|---|---|
| `id` | **Yes** | `string` | Regex: `^C-\d{2,}$` (e.g., `C-01`, `C-02`, `C-10`) | Unique constraint ID within this spec. |
| `description` | **Yes** | `string` | Should use RFC 2119 language (MUST, MUST NOT, SHOULD, MAY) | Human-readable constraint statement. |
| `type` | No | `string` | Enum: `technical`, `security`, `performance`, `accessibility`, `business` | Category of constraint. Surfaces in `specter check` diagnostics so issues can be grouped by category. |
| `enforcement` | No | `string` | Enum: `error`, `warning`, `info` | Overrides the tier-based default severity when Specter emits a diagnostic about this constraint (orphan, structural conflict). Omit to use the tier default (T1=error, T2=warning, T3=info for orphans). |
| `validation` | No | [Constraint Validation](#constraint-validation-object) | Machine-readable validation rule | Enables deterministic checking by spec-check. |

```yaml
constraints:
  - id: C-01
    description: "MUST validate against the canonical JSON Schema"
    type: technical
    enforcement: error

  - id: C-02
    description: "SHOULD support YAML anchors and aliases"
    type: technical
    enforcement: warning
    validation:
      field: yaml_features
      rule: enum
      value: ["anchors", "aliases"]
```

---

## Constraint Validation Object

An optional machine-readable validation rule attached to a constraint. Enables deterministic checking by spec-check without requiring AI interpretation.

| Field | Required | Type | Description |
|---|---|---|---|
| `field` | **Yes** | `string` | The field this validation applies to. |
| `rule` | **Yes** | `string` — enum: `type`, `min`, `max`, `pattern`, `enum`, `required`, `format`, `custom` | The type of validation rule. |
| `value` | **Yes** | `string` or `number` or `boolean` or `string[]` | The value for the rule. Actual type depends on the rule type. |

```yaml
validation:
  field: password
  rule: min
  value: 8
```

---

## Acceptance Criterion Object

An acceptance criterion (AC) is a **testable condition** that defines "done." Each AC should map to at least one test. The `references_constraints` field links ACs to the constraints they validate -- this is used for orphan constraint detection.

| Field | Required | Type | Format / Validation | Description |
|---|---|---|---|---|
| `id` | **Yes** | `string` | Regex: `^AC-\d{2,}$` (e.g., `AC-01`, `AC-02`, `AC-10`) | Unique AC ID within this spec. |
| `description` | **Yes** | `string` | -- | Human-readable description of the expected behavior. |
| `inputs` | No | `object` | Free-form (additionalProperties: true) | Input values or conditions that trigger this behavior. |
| `expected_output` | No | `object` | Free-form (additionalProperties: true) | What should happen when the inputs are provided. |
| `error_cases` | No | `array` of [Error Case](#error-case-object) | -- | Error conditions and their expected handling. |
| `references_constraints` | No | `array` of `string` | Each item: regex `^C-\d{2,}$` | Which constraints this AC validates. Used for orphan detection. |
| `gap` | No | `boolean` | Default: `false` | `true` if this AC was identified by gap analysis (no existing test covers it). |
| `priority` | No | `string` | Enum: `critical`, `high`, `medium`, `low` | Implementation priority. |
| `notes` | No | `string` | -- | Free-form narrative about edge cases, rationale, or non-obvious implementation details. Rendered in the VS Code `@ac` hover and `specter explain` output. Added in v0.7.0. |
| `approval_gate` | No | `boolean` | Default: `false` | Marks this AC as requiring explicit human approval before it can be considered done. In threshold mode this is metadata. Under `strictness: zero-tolerance`, an AC with `approval_gate: true` and no `approval_date` is demoted and the command exits with the approval-gate failure code. Added in v0.7.0. |
| `approval_date` | No | `string` | ISO-8601 date: `YYYY-MM-DD` | Date a human verified this AC. Meaningful only in conjunction with `approval_gate: true`. Added in v0.7.0. |

```yaml
acceptance_criteria:
  - id: AC-01
    description: "Valid spec file is parsed into a SpecAST"
    inputs:
      file: "fixtures/valid/simple.spec.yaml"
    expected_output:
      type: "SpecAST"
      fields_present: ["id", "version", "status", "tier"]
    references_constraints: ["C-01", "C-04"]
    priority: critical

  - id: AC-02
    description: "Missing required field returns ParseError"
    inputs:
      file: "fixtures/invalid/missing-id.spec.yaml"
    expected_output:
      type: "ParseError"
      error_path: "spec.id"
    error_cases:
      - condition: "id field is absent"
        expected_behavior: "Return error with field path spec.id"
    references_constraints: ["C-01", "C-02"]
    priority: critical
```

---

## Error Case Object

Describes an error condition and its expected handling. Used inside acceptance criteria.

| Field | Required | Type | Description |
|---|---|---|---|
| `condition` | **Yes** | `string` | The error condition. |
| `expected_behavior` | **Yes** | `string` | How the system should handle this condition. |

---

## Dependency Reference Object

Declares a dependency on another spec. Creates an edge in the dependency graph built by spec-resolve.

| Field | Required | Type | Format / Validation | Description |
|---|---|---|---|---|
| `spec_id` | **Yes** | `string` | Regex: `^[a-z][a-z0-9-]*$` (same as spec `id`) | The `id` of the spec being depended on. |
| `version_range` | No | `string` | Semver range (e.g., `^1.0.0`, `>=2.0.0 <3.0.0`, `~1.2.0`) | If omitted, any version is accepted. |
| `relationship` | No | `string` | Enum: `requires`, `extends`, `conflicts_with`. Default: `requires` | Nature of the dependency. |

### Relationship Types

| Relationship | Meaning |
|---|---|
| `requires` | This spec cannot function without the dependency. The dependency must be present and its constraints satisfied. |
| `extends` | This spec builds on top of the dependency, adding to its capabilities. |
| `conflicts_with` | This spec is incompatible with the referenced spec. Both cannot be active simultaneously. |

```yaml
depends_on:
  - spec_id: spec-parse
    version_range: "^1.0.0"
    relationship: requires

  - spec_id: spec-resolve
    version_range: "^1.0.0"
    relationship: requires
```

---

## Environment Object

Declares runtime environment requirements for the implementation.

| Field | Required | Type | Description |
|---|---|---|---|
| `required_vars` | No | `array` of `string` | Environment variables this implementation requires. |
| `deployment_targets` | No | `array` of `string` | Where this deploys (e.g., `production`, `staging`, `edge`). |

```yaml
environment:
  required_vars:
    - STRIPE_SECRET_KEY
    - DATABASE_URL
  deployment_targets:
    - production
    - staging
```

---

## Changelog Entry Object

Records a version change. Entries are ordered most recent first.

| Field | Required | Type | Format / Validation | Description |
|---|---|---|---|---|
| `version` | **Yes** | `string` | Semver (same regex as spec `version`) | The version this entry describes. |
| `date` | **Yes** | `string` | ISO 8601 date (`YYYY-MM-DD`) | When this version was created. |
| `author` | No | `string` | -- | Who authored this change. |
| `type` | No | `string` | Enum: `initial`, `major`, `minor`, `patch` | Classification of the change. |
| `description` | **Yes** | `string` | -- | Summary of what changed. |
| `changes` | No | `array` of [Changelog Change](#changelog-change-object) | -- | Itemized list of individual changes. |

```yaml
changelog:
  - version: "1.0.0"
    date: "2026-03-28"
    author: "specter-team"
    type: initial
    description: "Initial spec for the parser"
    changes:
      - type: addition
        section: constraints
        detail: "Added C-01 through C-08"
```

---

## Changelog Change Object

An individual change within a changelog entry.

| Field | Required | Type | Format / Validation | Description |
|---|---|---|---|---|
| `type` | **Yes** | `string` | Enum: `addition`, `removal`, `modification`, `deprecation` | What kind of change. |
| `section` | No | `string` | -- | Which section of the spec was affected. |
| `detail` | **Yes** | `string` | -- | Description of the specific change. |

---

## Generated From Object

Provenance tracking for reverse-compiled specs -- specs that were extracted from existing code rather than written first. This field should only be present on reverse-compiled specs.

| Field | Required | Type | Format / Validation | Description |
|---|---|---|---|---|
| `source_file` | No | `string` | -- | Path to the source code this spec was reverse-compiled from. |
| `test_files` | No | `array` of `string` | -- | Paths to associated test files. |
| `extraction_date` | No | `string` | ISO 8601 date (`YYYY-MM-DD`) | When the reverse compilation occurred. |

```yaml
generated_from:
  source_file: "src/core/parser/parse.ts"
  test_files:
    - "tests/core/parser/parse.test.ts"
  extraction_date: "2026-03-28"
```

---

## Naming Conventions

| Element | Convention | Pattern | Examples |
|---|---|---|---|
| Spec ID | kebab-case, starts with a letter | `^[a-z][a-z0-9-]*$` | `user-registration`, `payment-create-intent`, `auth-jwt-validation` |
| Constraint ID | `C-` prefix + zero-padded number (2+ digits) | `^C-\d{2,}$` | `C-01`, `C-02`, `C-10` |
| Acceptance Criterion ID | `AC-` prefix + zero-padded number (2+ digits) | `^AC-\d{2,}$` | `AC-01`, `AC-02`, `AC-10` |
| Spec filename | `{spec-id}.spec.yaml` | -- | `user-registration.spec.yaml` |
| Version | Quoted semver | `MAJOR.MINOR.PATCH[-prerelease]` | `"1.0.0"`, `"2.1.0"`, `"1.0.0-draft"` |

**Important:** Always quote `version` values in YAML. Unquoted `1.0` is parsed as the float `1.0`, not the string `"1.0"`.

---

## Versioning Rules

Spec versions follow semantic versioning. The type of change determines which component to bump.

### MAJOR (breaking)

Increment MAJOR when the change could break downstream specs or implementations.

- Removing a constraint
- Removing an acceptance criterion
- Changing a constraint from `SHOULD` to `MUST NOT` (inverting intent)
- Removing a field from `context` that dependents rely on
- Changing the `id` of the spec
- Narrowing scope in a way that invalidates existing implementations

### MINOR (additive)

Increment MINOR when adding new capabilities that do not break existing behavior.

- Adding a new constraint
- Adding a new acceptance criterion
- Adding optional fields to context or objective
- Expanding scope (new `includes` items)
- Adding a new `depends_on` entry
- Changing enforcement from `error` to `warning` (relaxing)

### PATCH (clarification)

Increment PATCH for non-functional changes.

- Fixing typos in descriptions
- Improving clarity of constraint wording without changing meaning
- Adding or updating `changelog` entries
- Adding or modifying `tags`
- Updating `assumptions` or `description` text

---

## Tier Definitions

Tiers classify specs by risk level. The tier affects enforcement strictness, coverage thresholds, and diagnostic severity.

| Tier | Name | Description | Coverage Threshold | Orphan Severity |
|---|---|---|---|---|
| **1** | Security / Money | Specs governing authentication, authorization, payment processing, PII handling, cryptographic operations. Failures cause data breaches or financial loss. | 100% | `error` |
| **2** | Core Business Logic | Specs governing domain logic, workflow orchestration, data transformations. Failures cause incorrect behavior visible to users. | 80% | `warning` |
| **3** | Utility / Internal | Specs governing internal tooling, logging, formatting, dev-only features. Failures are inconvenient but not critical. | 50% | `info` |

### Choosing a Tier

- If a bug in this feature could lose money or leak data: **Tier 1**
- If a bug would be visible to end users and affect core functionality: **Tier 2**
- If a bug would only affect internal workflows or developer experience: **Tier 3**

When in doubt, choose the higher (stricter) tier.

---

## Status Lifecycle

Specs move through a linear lifecycle. Specter checks every discovered, parseable spec by default. Lifecycle status is still useful for review discipline; combine `settings.warn_on_draft: true` with `settings.strict: true` when draft specs should fail release gates.

```
draft --> review --> approved --> deprecated --> removed
```

| Status | Meaning | Default pipeline behavior |
|---|---|---|
| `draft` | Work in progress. May be incomplete or rapidly changing. | Checked. Warns only when `warn_on_draft` is enabled. |
| `review` | Complete and submitted for team review. Should not change without discussion. | Checked. |
| `approved` | Accepted as the source of truth. Implementation must conform. | Checked. |
| `deprecated` | Superseded or scheduled for removal. Existing implementations still valid but should migrate. | Checked unless excluded by project configuration. |
| `removed` | No longer active. Kept in version history only. | Checked unless excluded by project configuration. |

---

## Worked Examples

### Minimal Spec

The smallest valid spec. Contains only required fields with the minimum required sub-fields.

```yaml
spec:
  id: test-minimal
  version: "1.0.0"
  status: draft
  tier: 3

  context:
    system: Test system

  objective:
    summary: A minimal spec with only required fields and no optional fields.

  constraints:
    - id: C-01
      description: "MUST work with minimal fields"

  acceptance_criteria:
    - id: AC-01
      description: "Parser succeeds with only required fields present"
```

### Full Spec

A production spec using most available fields.

```yaml
spec:
  id: spec-parse
  version: "1.0.0"
  status: approved
  tier: 1

  context:
    system: Specter toolchain
    feature: YAML-to-AST parser
    description: >
      The foundational tool in the Specter toolchain. Parses .spec.yaml files,
      validates them against the canonical JSON Schema, and produces typed SpecAST
      objects. Every other tool depends on spec-parse producing correct, validated output.
    dependencies:
      - "yaml (eemeli/yaml 2.x)"
      - "ajv (8.x)"
    assumptions:
      - "Input files are UTF-8 encoded YAML"
      - "The canonical JSON Schema is the source of truth for spec structure"

  objective:
    summary: >
      Parse .spec.yaml files into validated, typed SpecAST objects.
      Reject malformed specs with actionable error messages that include
      line numbers and field paths.
    scope:
      includes:
        - "YAML parsing with syntax error handling"
        - "JSON Schema validation against the canonical spec-schema.json"
        - "Typed AST construction from validated YAML"
        - "Error reporting with line numbers and JSON field paths"
      excludes:
        - "Dependency resolution (that is spec-resolve)"
        - "Semantic validation across specs (that is spec-check)"
        - "File discovery and glob patterns (that is the registry)"

  constraints:
    - id: C-01
      description: "MUST validate against the canonical JSON Schema (spec-schema.json)"
      type: technical
      enforcement: error

    - id: C-02
      description: "MUST report errors with the YAML line number and JSON path"
      type: technical
      enforcement: error

    - id: C-03
      description: "SHOULD support YAML anchors and aliases"
      type: technical
      enforcement: warning

  acceptance_criteria:
    - id: AC-01
      description: "Valid spec file is parsed into a SpecAST with all required fields"
      inputs:
        file: "fixtures/valid/simple.spec.yaml"
      expected_output:
        type: "SpecAST"
        fields_present: ["id", "version", "status", "tier", "context", "objective", "constraints", "acceptance_criteria"]
      references_constraints: ["C-01"]
      priority: critical

    - id: AC-02
      description: "Spec missing required field returns ParseError with field path"
      inputs:
        file: "fixtures/invalid/missing-id.spec.yaml"
      expected_output:
        type: "ParseError"
        error_path: "spec.id"
      error_cases:
        - condition: "id field is absent"
          expected_behavior: "Return error with field path spec.id and error_type required"
      references_constraints: ["C-01", "C-02"]
      priority: critical

    - id: AC-03
      description: "Malformed YAML returns ParseError with line number"
      inputs:
        file: "fixtures/invalid/bad-yaml.spec.yaml"
      expected_output:
        type: "ParseError"
        has_line_number: true
      references_constraints: ["C-02"]
      priority: critical

  tags:
    - parser
    - core
    - foundational

  environment:
    required_vars: []
    deployment_targets:
      - production
      - staging

  changelog:
    - version: "1.0.0"
      date: "2026-03-28"
      author: "specter-team"
      type: initial
      description: "Initial spec for the Specter YAML parser"
      changes:
        - type: addition
          section: constraints
          detail: "Added C-01 through C-03"
        - type: addition
          section: acceptance_criteria
          detail: "Added AC-01 through AC-03"
```

### Spec with Dependencies

A spec that depends on other specs, creating edges in the dependency graph.

```yaml
spec:
  id: spec-check
  version: "1.0.0"
  status: approved
  tier: 1

  context:
    system: Specter toolchain
    feature: Spec type checker
    description: >
      Validates semantic consistency across specs in the dependency graph.
    related_specs:
      - "spec-parse.spec.yaml"
      - "spec-resolve.spec.yaml"

  objective:
    summary: >
      Perform structural type-checking across the spec dependency graph.
      Detect orphan constraints and structural conflicts between connected specs.
    scope:
      includes:
        - "Orphan constraint detection"
        - "Structural conflict detection between dependent specs"
      excludes:
        - "Semantic conflict detection (AI-assisted, future phase)"

  depends_on:
    - spec_id: spec-parse
      version_range: "^1.0.0"
      relationship: requires
    - spec_id: spec-resolve
      version_range: "^1.0.0"
      relationship: requires

  constraints:
    - id: C-01
      description: "MUST detect all orphan constraints (constraints with no AC reference)"
      type: technical
      enforcement: error

  acceptance_criteria:
    - id: AC-01
      description: "Constraint not referenced by any AC produces OrphanConstraint diagnostic"
      inputs:
        spec: "spec with C-01 (referenced), C-02 (not referenced)"
      expected_output:
        type: "Diagnostic"
        kind: "orphan_constraint"
        constraint_id: "C-02"
      references_constraints: ["C-01"]
      priority: critical

  changelog:
    - version: "1.0.0"
      date: "2026-03-28"
      author: "specter-team"
      type: initial
      description: "Initial spec for the Specter type checker"
```

---

# Specter: Test Annotation Reference
URL: https://www.hanalyx.com/docs/specter/test-annotation-reference

How to annotate tests so `specter coverage` counts them and `specter coverage --strict` verifies them.

This is the counterpart to `SPEC_SCHEMA_REFERENCE.md`. The spec reference defines the schema for `.spec.yaml` files. This reference defines the schema for test annotations.

---

## What Specter reads

Specter reads annotations from two places. They serve different purposes and both exist for a reason.

| Channel | Source | Read by | Purpose |
|---|---|---|---|
| 1 | `// @spec <id>` and `// @ac AC-NN` comments above the test function | `specter coverage` | Counts which ACs have annotated tests. |
| 2 | `<spec-id>/AC-NN` in the test's runner-visible output (test title or runtime log) | `specter ingest` | Records pass/fail per AC in `.specter-results.json`. Required by `specter coverage --strict`. |

**Write both.** A test with only channel 1 is counted but not verified. Under `--strict`, counted-but-not-verified equals uncovered, and the AC demotes.

---

## The rules

1. **Source comment format.** Above every test function:
   ```
   // @spec <spec-id>
   // @ac AC-NN
   ```
   One `// @spec` per test. One `// @ac` per AC the test covers. Languages other than C-family use their own comment character: `#` for Python, `--` for SQL, etc.

2. **Runner-visible format.** The `(spec-id, AC-NN)` pair appears in one of:
   - The test title: `<spec-id>/AC-NN` or `<spec-id>:AC-NN` somewhere in the name.
   - The test body, printed at runtime: `// @spec <spec-id>` and `// @ac AC-NN` on separate lines.

3. **Spec id format.** Lowercase kebab-case. Matches the regex `[a-z][a-z0-9-]*[a-z0-9]`. Starts with a letter. Ends with a letter or digit. No underscores, no uppercase, no leading or trailing dash.

4. **AC id format.** Zero-padded two-digit minimum: `AC-01`, `AC-02`, `AC-12`, `AC-100`. **`AC-1` does not match `AC-01`.** The regex accepts `AC-\d+` so single-digit forms extract as `AC-1` — but the coverage gate compares by string equality against the spec, and the spec uses `AC-01`.

5. **One AC per test.** Each test function or subtest covers exactly one `(spec-id, AC-NN)` pair. A JUnit `<testcase>` entry or a `go test -json` test event carries one title, so `specter ingest` assigns one pair per entry. A multi-AC test loses ACs under `--strict`.

6. **One convention per file.** Ingest accepts both forms. Mixing them in one file is legal but error-prone during migration. Pick one form per file.

7. **The extraction regex** (from `specter/internal/ingest/annotations.go`):
   ```
   ([a-z][a-z0-9-]*[a-z0-9])[/:](AC-\d+)
   ```
   The separator between spec id and AC id is `/` or `:`. Nothing else. `_`, `-`, `.`, and whitespace do not work.

---

## By runner and language

### Go (`go test -json`)

Use `t.Run` so each AC has its own subtest and its own runner-visible entry.

```go
// @spec user-create
// @ac AC-01
// @ac AC-02
func TestCreateUser(t *testing.T) {
    t.Run("user-create/AC-01 valid credentials returns 201", func(t *testing.T) {
        // assertions
    })
    t.Run("user-create/AC-02 invalid email returns 400", func(t *testing.T) {
        // assertions
    })
}
```

`go test -json` emits events with `Test: "TestCreateUser/user-create/AC-01 valid credentials returns 201"`. The regex matches `user-create/AC-01`.

### TypeScript / Jest / Vitest (JUnit reporter)

Put the pair in each `it` or `test` title.

```typescript
// @spec user-create
// @ac AC-01
test('[user-create/AC-01] valid email and password creates user and returns 201 with JWT', () => {
    // assertions
});

// @spec user-create
// @ac AC-02
test('[user-create/AC-02] invalid email format returns 400', () => {
    // assertions
});
```

Run with JUnit output:
- Jest: `jest --reporters=jest-junit`
- Vitest: `vitest run --reporter=junit --outputFile=test-results.xml`

JUnit `<testcase name="...">` carries the full title. The regex matches the pair inside the brackets.

### Python / pytest (known limitation)

Python function names cannot contain `/` or `:`. Convention A (title-based) does not work for pytest by default. **Use Convention B (runtime log) for Python.**

```python
# @spec user-create
# @ac AC-01
def test_valid_registration_returns_201(client):
    print('// @spec user-create')
    print('// @ac AC-01')
    response = client.post('/users', json={...})
    assert response.status_code == 201
```

Run pytest with JUnit output:
```
pytest --junitxml=test-results.xml -o junit_logging=all -o junit_log_passing_tests=True
```

`-o junit_logging=all` captures `print()` output into `<system-out>` for every test case. `specter ingest` reads `<system-out>` and matches the body regex `//\s*@spec\s+([a-z][a-z0-9-]*[a-z0-9])` and `//\s*@ac\s+(AC-\d+)`.

**Why not function names.** `def test_user_create_AC_01_valid_returns_201` emits the JUnit title `test_user_create_AC_01_valid_returns_201`. The regex requires `/` or `:` between `user-create` and `AC-01`. `_` does not match. Use Convention B (runtime `print('// @spec ...')`) for Python.

### Rust / `cargo test`

No first-party ingest flavor today. Emit Convention B to stdout and parse manually.

### Runner-log form — Convention B

Works in every language. Use it when you cannot rename test titles (shared naming contract, snapshot tests, external expectations, Python function names).

```typescript
test('rejects zero amount', () => {
    console.log('// @spec payment-charge');
    console.log('// @ac AC-03');
    // assertions
});
```
```go
func TestCharge_ZeroAmount(t *testing.T) {
    t.Log("// @spec payment-charge")
    t.Log("// @ac AC-03")
    // assertions
}
```
```python
def test_rejects_zero_amount(client):
    print('// @spec payment-charge')
    print('// @ac AC-03')
    # assertions
```

---

## Parameterized tests

A parameterized test produces one JUnit entry per case. Each case carries its own title, so each case needs its own `(spec-id, AC-NN)`.

### Vitest `test.each`

```typescript
// @spec payment-charge
// @ac AC-04
test.each([
    { amount: 0,  ac: 'AC-04', desc: 'rejects zero' },
    { amount: -1, ac: 'AC-04', desc: 'rejects negative' },
])('[payment-charge/$ac] $desc', ({ amount }) => {
    // assertions
});
```

Each case emits a title like `[payment-charge/AC-04] rejects zero`. The regex matches.

### pytest `@pytest.mark.parametrize`

Use Convention B inside the test body — titles are parameter-suffixed function names, which again don't contain `/` or `:`.

```python
# @spec payment-charge
# @ac AC-04
@pytest.mark.parametrize('amount', [0, -1])
def test_rejects_invalid_amount(amount):
    print('// @spec payment-charge')
    print('// @ac AC-04')
    # assertions
```

### Go table tests

```go
// @spec payment-charge
// @ac AC-04
func TestReject(t *testing.T) {
    cases := []struct{ name string; amount int }{
        {"payment-charge/AC-04 zero",     0},
        {"payment-charge/AC-04 negative", -1},
    }
    for _, c := range cases {
        t.Run(c.name, func(t *testing.T) {
            // assertions
        })
    }
}
```

Both subtests emit the same `(spec-id, AC-NN)`. `specter ingest` merges by worst-status (errored > failed > skipped > passed), so one failing case demotes the AC.

---

## Migrating from v0.9-style source-only

v0.9 and earlier taught source-only annotations: `// @spec` / `// @ac` above the function, no runner-visible form. Those tests work with `specter coverage` (annotation counting) but demote under `--strict` (no results entry).

### Migration recipe

1. **Add `--reporter=junit`** (or `go test -json`) to the CI test command. Reporter output is additive; keep the existing reporters.
2. **Rename test titles** file by file. Inside each file, add `<spec-id>/AC-NN` to every test title. Keep the source comments.
3. **Wire ingest + strict**:
   ```
   specter ingest --junit 'test-results/*.xml'
   specter coverage --strict
   ```
4. **Use `--scope <domain>` for staged rollout.** Enforce `--strict` on one domain at a time. Specs outside the scoped domain keep v0.9 annotation-counting behavior. See `CLI_REFERENCE.md` → `specter coverage` → `--scope`.

### File-atomic discipline

Migrate whole files at once. A half-migrated file (some tests renamed, some not) under `--strict` will demote the unrenamed tests even though the renamed ones pass. `--tests <glob>` scopes by path; it cannot scope by test-title form within a file.

---

## Common mistakes

**`AC-1` instead of `AC-01`.** The regex accepts single-digit; the coverage gate compares against the spec, which uses `AC-01`. Zero-pad always.

**`_` between spec id and AC id.** Python users hit this. `_` is not in the regex. Use Convention B.

**Underscore in spec id.** Spec ids are kebab-case. `user_create/AC-01` does not match; `user-create/AC-01` does.

**Uppercase in spec id.** `User-create/AC-01` does not match. The spec id matches `[a-z][a-z0-9-]*[a-z0-9]`.

**Two ACs in one test.** Under `--strict`, `specter ingest` assigns one pair per runner entry. `test('[spec-foo/AC-01 AC-02] two things', ...)` captures only `spec-foo/AC-01`. Split into two tests.

**Source-only annotations in a migrated file.** `specter coverage` will count them; `specter ingest` will drop them from the results; `specter coverage --strict` will demote them. Check `ingest`'s summary line (`Scanned N; extracted M; dropped K`) — K should be zero for fully-migrated files.

**Mixed Convention A and B in one file.** Ingest handles both, but the mix is a migration smell. Pick one.

**Reporter not wired.** `ingest --junit` against a file that doesn't exist is a hard error. `--junit 'test-results/*.xml'` against an empty glob produces zero entries; `--strict` then demotes everything and emits the empty-results warning.

---

## Suppressing `unreachable_annotation` per-file

Added in v0.13.0.

`specter check --test` runs a language-aware reachability scanner that
catches source-comment `@ac` declarations whose enclosing test produces
no runner-visible token. The diagnostic name is `unreachable_annotation`
(and `unreachable_annotation_unknown` when the test shape is custom).

**Suppress for an entire file** by placing the marker anywhere in the
file (top of file is conventional):

| Language family | Marker line |
|---|---|
| Go, TypeScript, JavaScript, Rust | `// @reachable manual` |
| Python, shell, YAML | `# @reachable manual` |

The marker is file-level scope. One declaration anywhere in the file
opts every `@ac` in that file out of BOTH `unreachable_annotation` and
`unreachable_annotation_unknown`, regardless of `settings.strictness`.

**When to use the marker:**

- The test runner is custom (not Go's `testing`, Jest/Vitest, or
  pytest) and the scanner can't recognize the test shape.
- Tests are dynamically generated and the `@ac` lives on the generator,
  not on a recognizable test function.
- The operator has manually verified that the test does cover the
  annotated ACs through some out-of-band channel.

**Don't use it as a noise-suppressor.** If the diagnostic fires because
a real Go subtest title is missing the `<spec-id>/AC-NN` token, fix the
subtest title — the marker disables a real signal. The same applies to
TypeScript `describe`/`it` titles and pytest `print()` lines.

**Strictness routing** (when the marker is absent):

- `settings.strictness: annotation` — diagnostics suppressed (exit 0).
- `settings.strictness: threshold` (default) — `warning` (exit 0).
- `settings.strictness: zero-tolerance` — `error` (exit non-zero).

`unreachable_annotation_unknown` is always a `warning` regardless of
strictness — the scanner can't tell whether the test really covers the
AC, only that no language-aware parser recognized the shape.

Example (Go test with a custom helper that hides the subtest from
go/ast's recognition):

```go
// @reachable manual
package foo

import "testing"

// @spec my-spec
// @ac AC-01
func TestThing(t *testing.T) {
    runWithCustomHelper(t, "my-spec/AC-01")
}
```

Without the marker, `check --test` would emit
`unreachable_annotation_unknown` because `go/ast` can't unwrap
`runWithCustomHelper` to see whether the test title carries
`my-spec/AC-01`. The marker asserts the operator has verified it does.

---

## Troubleshooting

**Symptom**: `specter ingest` reports `Scanned N; extracted 0; dropped N`.
**Cause**: Test titles don't match the regex. Usually missing `/` or `:`, or missing the spec id entirely.
**Check**: `specter ingest --junit <path> --verbose` lists every dropped test name. Scan for the pattern you expected.

**Symptom**: `specter coverage --strict` demotes every annotated AC.
**Cause**: `.specter-results.json` has zero entries, or no entry matches the annotated `(spec-id, AC-NN)`.
**Check**: The empty-results warning fires before the demotion report. Read `.specter-results.json`; it should have one entry per AC your tests cover.

**Symptom**: The AC number in the test title is `AC-1`, the spec has `AC-01`, and `--strict` demotes.
**Cause**: String-equality mismatch. Zero-pad the test title.

**Symptom**: pytest tests don't produce annotation entries.
**Cause**: pytest isn't capturing `print()` output in the JUnit XML by default.
**Fix**: `pytest --junitxml=out.xml -o junit_logging=all -o junit_log_passing_tests=True`.

**Symptom**: `specter check --test` emits `unreachable_annotation` for a test that you believe does cover the AC.
**Cause**: The test title or test body doesn't carry the `<spec-id>/AC-NN` token in a form the scanner recognizes (Convention A in the subtest title, or Convention B as a runtime print). The scanner reports the source `@ac` is unreachable because `specter ingest` would not extract a matching pair from this test's runner output.
**Fix**: Add the spec-id/AC-NN token to the subtest title (Convention A), or print `// @spec <id>` / `// @ac AC-NN` from the test body (Convention B). If the test legitimately covers the AC through a channel the scanner can't see, use `// @reachable manual` (or `# @reachable manual` for Python) at the top of the file. Documented in "Suppressing `unreachable_annotation` per-file" above.

**Symptom**: `specter check --test` emits `unreachable_annotation_unknown` rather than `unreachable_annotation`.
**Cause**: The test file is in a language the language-aware reachability scanner doesn't have a parser for (anything other than Go, TypeScript / Jest / Vitest, or Python), OR the test shape (custom helpers, table-driven tests with non-literal names, dynamically-generated tests) is structurally unrecognized.
**Fix**: `_unknown` is always a `warning` regardless of strictness mode — it does not fail any gate. If you've manually verified the test does cover the AC, add `// @reachable manual` at the top of the file to silence both `_unknown` and the `unreachable_annotation` diagnostic for that file.

---

## See also

- `CLI_REFERENCE.md` → `specter coverage` (the `--strict`, `--scope`, `--tests` flags)
- `CLI_REFERENCE.md` → `specter ingest` (JUnit and `go test -json` flavors, `--verbose`)
- `docs/explainer/v0.10-ci-gated-coverage.md` (design rationale for the two-channel split)

---

# Specter: Triage Discipline
URL: https://www.hanalyx.com/docs/specter/triage-discipline

How the Specter project decides which feature requests, schema changes, and design proposals get implemented vs. deferred vs. rejected.

## Two filters before scope

Before a feature request earns a release window, it passes two filters.

### Universality test

A feature must benefit *most* projects using Specter — or share the pain across multiple unrelated projects — to earn implementation. If a proposed feature only helps the project that asked for it, reject it.

- "JWTMS needs X" or "OpenWatch wants Y" alone is not sufficient. The pattern has to generalize.
- For pain that is real but only manifests in one project, prefer external tooling (a translator, an adapter, project-side scripts) or `specter migrate --from=<dialect>` over baking it into the core.
- For genuinely hard calls (clearly useful to one project, plausibly useful to others), surface explicitly rather than deciding alone. Either confirm a second project's use case or close until one surfaces.

This is a triage filter, not the only filter. Items that pass still have to clear mission focus and scope discipline.

### Schema conservatism

The spec schema is pre-1.0 (not locked), but schema changes are *not cheap*. Every delta forces downstream cost: docs rewritten, existing specs migrated, AI instruction templates updated, JSON consumers verified, dogfood specs touched. Pre-1.0 is permission to iterate, not encouragement to.

- Before recommending a schema change, look for non-schema alternatives: doctor canonicalization, `specter migrate --from=<dialect>` translators, tooling-side workarounds.
- "Match the existing X shape" or "fix the asymmetry" is *not* sufficient justification on its own. A documented user-friction case is required — and even then, ask whether the friction can be absorbed by tooling rather than the canonical schema.
- When a schema change is genuinely warranted, enumerate the full blast radius (parser, schema, type model, JSON output, doctor migration, editor surfaces, docs, dogfood specs, AI instruction templates) before proposing scope.
- Default framing for a schema-change idea: *"needs design discussion"* rather than *"vX.Y candidate"*. Don't pre-commit a release window before the design call happens.

## Specter Schema Request Brief (SSRB)

Each schema-change request — opened as a GitHub issue, raised internally, or surfaced during code review — gets a written brief documenting the decision and its reasoning. Briefs live at `docs/ssrb/SSRB-NNN.md`. The number matches the GitHub issue when one exists; otherwise sequential.

### When to write an SSRB

- Any field addition, removal, or shape change to the canonical spec schema
- Any change to the `acceptance_criteria` shape, `manifest` schema, or top-level metadata
- Cross-cutting design questions about the schema (e.g., "should ACs have a lifecycle?")
- Anything where future requesters will likely re-tread the same ground

### When NOT to write an SSRB

- Pure bug fixes that don't change schema
- Code refactors that don't surface to users
- Documentation improvements
- Tooling changes that don't alter the schema (CLI flags, output formats, etc.)

### Process

1. Copy `docs/ssrb/TEMPLATE.md` to `docs/ssrb/SSRB-NNN.md`.
2. Fill in §1 (Request) and §2 (Origin) immediately, before forming an opinion.
3. Apply the universality test (§3) and the cost analysis (§4).
4. Survey existing coverage (§5) and alternatives (§6).
5. Write §7 (Decision) only after the prior sections are complete. The reasoning must reference §3, §4, §5 explicitly.
6. Add §8 (Reconsideration triggers) — concrete criteria for revisiting.
7. Add a one-line entry to `docs/ssrb/INDEX.md`.
8. If the request originated as a GitHub issue, post a link to the SSRB on the issue thread when closing.

### Reusing prior reasoning

When a new request arrives that resembles a prior SSRB, link the new one to the existing brief. Either close the new request as "see SSRB-NNN" or write a fresh SSRB only if the new request differs in scope from the original. The catalog at `docs/ssrb/INDEX.md` is the entry point for "has this been asked before?"

### Why this exists

The v0.11.0 review cycle produced four schema-change requests (GH #97/#98/#99/#100), each requiring ~500 words of reasoning. Without a durable home, that reasoning would live only in scattered GitHub comments. SSRBs preserve the analysis, force consistent rigor on each new request, and give future requesters reference points so the project doesn't re-decide settled questions.

---

# Specter: AI Prompts for Specter
URL: https://www.hanalyx.com/docs/specter/ai-prompts

Specter's schema is detailed by design — but you are not meant to write specs by hand. The intended workflow is a collaboration: **you provide intent, your AI coding assistant translates it into a full spec and tests, you review and approve.**

These prompts are ready to paste into Claude, Cursor, Copilot, or any AI coding assistant. Use them in order for a new feature, or individually when you need a specific step.

---

## 1. Intent → Spec

The starting point for every feature. You describe what you want; the AI produces the full `.spec.yaml`.

```
I want to build [module name]. Here is my intent:

[2-3 sentences describing what it does]

Key constraints I care about:
- [constraint 1]
- [constraint 2]

Non-obvious decisions / trade-offs:
- [anything the AI shouldn't guess about]

Generate a complete `.spec.yaml` for this using Specter's schema.
- id: [kebab-case name]
- status: draft
- tier: [1 = security/money, 2 = business logic, 3 = utility]
- Use MUST / MUST NOT language for constraints
- Generate ACs that cover each constraint, including error cases
- Do not invent requirements I haven't mentioned
```

---

## 2. Review a Generated Spec

Run this before approving any spec. The spec is the approval gate — problems caught here cost nothing. Problems caught after implementation are expensive.

```
Review this Specter spec before I approve it.

Check for:
- Are all constraints actually testable?
- Are there obvious missing edge cases in the ACs?
- Does the objective accurately match the constraints?
- Is the tier appropriate for this module's criticality?
- Are any constraints redundant or contradictory?
- Would anything here surprise a developer implementing it?

Be direct — flag problems, don't just validate.

[paste spec]
```

---

## 3. Spec → Tests

Run this after the spec is reviewed and approved. Tests come from the ACs directly. No guessing, no scope creep.

Specter reads test annotations from two places:

- **Source comments**: `// @spec <id>` and `// @ac AC-NN` above the test. Read by `specter coverage`.
- **Test title or runtime log**: the `<spec-id>/AC-NN` pair visible in the test runner output. Read by `specter ingest`. Required by `specter coverage --strict`.

Source comments alone: `coverage` counts it, `--strict` demotes it. Write both forms.

Full rules, per-runner examples, parameterized tests, and troubleshooting: see [`TEST_ANNOTATION_REFERENCE.md`](/docs/specter/test-annotation-reference).

```
Using this Specter spec as the contract, write [Go/Python/TypeScript] tests
for every acceptance criterion.

Rules:
- One test function per AC. No multi-AC tests — each test runner entry
  maps to one (spec, AC) pair under --strict.
- The test title carries [spec-id/AC-NN]:
    TypeScript:  it('[spec-id/AC-NN] brief description', () => { ... })
    Python:      def test_spec_id_AC_NN_brief_description(...): ...
    Go:          t.Run("spec-id/AC-NN brief description", func(t *testing.T) { ... })
  AC-NN is zero-padded: AC-01, not AC-1. Spec-id and AC-NN match the spec.
- Add source comments above the test function:
    // @spec [spec-id]
    // @ac [AC-NN]
- Tests are executable. No pseudocode, no TODOs.
- Cover happy path and error cases from the spec.
- Do not test anything outside the spec.

[paste spec]
```

**Alternate form — runtime log.** When you can't rename titles (shared naming, snapshot tests, external contracts), emit the pair at runtime from inside the test body:

```typescript
test('rejects zero amount', () => {
  console.log('// @spec payment-charge');
  console.log('// @ac AC-03');
  // assertions
});
```
```go
func TestCharge_ZeroAmount(t *testing.T) {
    t.Log("// @spec payment-charge")
    t.Log("// @ac AC-03")
    // assertions
}
```

Pick one form per file. Do not mix title-based and runtime-log forms in the same file.

---

## 4. Spec → Implementation

The AI implements against the spec as a contract. The tests already exist and define what done looks like.

```
Implement [module name] using this Specter spec as the contract.

Rules:
- Every C-NN constraint must be satisfied — do not skip or soften any
- Do not add behavior not described in the spec
- If the spec is ambiguous on something, ask before assuming
- The tests already exist — your implementation must make them pass

[paste spec]
```

---

## 5. Clean Up a Reverse-Generated Spec

After running `specter reverse` on an existing codebase, use this to turn raw extracted drafts into meaningful specs worth approving.

```
I ran `specter reverse` on my codebase. These are draft specs extracted from
existing code. Improve them:

- Replace `gap: true` ACs with real descriptions based on the constraints
- Improve the objective summary to describe intent, not just structure
- Make constraint descriptions precise using MUST / MUST NOT language
- Suggest the correct tier (1/2/3) based on what the module does
- Remove placeholder text like "auto-generated placeholder"
- Keep status: draft — I will promote to approved after review

[paste specs]
```

---

## 6. Full Loop (new feature end-to-end)

For when you want to run the complete workflow in a single session. The pause after step 1 is non-negotiable — never let the AI write tests or code against an unreviewed spec.

```
I want to build [feature]. My intent:

[brief description]
[key constraints]
[non-obvious decisions]

Do the following in order:
1. Write a complete `.spec.yaml` (status: draft).
2. Write [language] tests for every AC. One test per AC. Test title carries
   `[spec-id/AC-NN]`. Add `// @spec` and `// @ac` comments above the test.
   See section 3 for the exact form.
3. Implement the feature so the tests pass.

After step 1, pause and show me the spec. I will review it before you proceed
to step 2. Do not write tests or code until I approve the spec.
```

---

## The Order Matters

These prompts follow the SDD loop in sequence:

```
Intent → Spec → [Review] → Tests → Implementation → specter sync
```

Skipping the review step defeats the purpose. The spec is where your intent is captured — if the AI misunderstood something, that is the cheapest place to catch it.

Run `specter sync` after the implementation step. If it passes, the feature is done. If it fails, the spec tells you exactly what is missing.

---

# Specter: Specter FAQ
URL: https://www.hanalyx.com/docs/specter/faq

Frequently asked questions about Specter and Spec-Driven Development.

---

## What is SDD?

Spec-Driven Development (SDD) is a methodology where structured specification files are the Single Source of Truth (SSOT) for every feature in a system. The spec defines the "what and why" -- context, objective, constraints, and acceptance criteria -- before any code is written. When the spec and the code disagree, the spec is right and the code is wrong. SDD is designed for AI-assisted development, where specifications constrain the solution space and provide a verifiable contract for generated code. For the full methodology, see the [Mastering SDD course material](../../sddbook/INDEX.md).

---

## What is a micro-spec?

A micro-spec is the fundamental unit of specification in SDD. Every micro-spec has three pillars:

- **Context** -- What system, feature, and dependencies does this spec describe? What assumptions are being made?
- **Objective** -- What should this component do? What is in scope and what is explicitly excluded?
- **Constraints** -- What rules must be followed? Each constraint has an ID (e.g., `C-01`), a description, and an enforcement level (`error`, `warning`, or `info`).

A micro-spec also includes **acceptance criteria** -- testable conditions that prove the constraints are satisfied. Each AC references the constraints it validates, creating a traceable link from requirement to verification.

---

## What kinds of specs can I write with Specter?

Specter is content-agnostic. Anything that fits the spec shape -- context, objective, constraints, acceptance criteria -- is a valid spec. The pipeline (parse -> resolve -> check -> coverage -> sync) does not inspect what category of contract you are describing.

Examples that all live in `.spec.yaml`:

- **Behavioral contracts** -- runtime behavior of an endpoint, a background job, a state machine.
- **Data invariants** -- "every payment has a tenant_id," "PII fields are encrypted at rest."
- **Security and policy** -- authorization rules, audit logging, compliance constraints.
- **Schema and structural contracts** -- the shape of a configuration file, a webhook payload, a registry entry format.
- **Architecture rules** -- "service A must not call service B directly."

The structure stays the same in every case: constraints define the rules, ACs define how to verify them, tests provide mechanical proof. Specter does not need to know whether the constraint is a runtime invariant or a policy -- only that it has an ID, a description, and an AC that references it.

A practical implication: when your contract has *load-bearing data* (an exemptions list, a permission catalog, a feature-flag table), keep the data in a sidecar file and let the spec describe the format the entries must satisfy. The spec is the type definition; the data is the catalog. Specter validates the type definition; your test suite or lint rule validates the catalog against it.

---

## Why YAML?

Specter uses YAML as the spec format for several reasons:

- **Human-readable.** YAML is easy to read and write without tooling. Developers can author specs in any text editor.
- **AI-readable.** Large language models parse YAML reliably. It can be included directly in prompts and context windows.
- **Diffable.** YAML produces clean diffs in version control, making spec changes easy to review in pull requests.
- **Existing ecosystem.** YAML has mature parsers in every language, JSON Schema validation support, and broad IDE support with syntax highlighting and autocompletion.
- **Structured but flexible.** YAML supports the nested, typed structure that specs require (objects, arrays, enums) without the syntactic noise of JSON or the ambiguity of Markdown.

---

## How is Specter different from OpenAPI / Swagger?

OpenAPI describes **API surfaces** -- endpoints, request/response schemas, status codes. It answers "what shape does the data have?"

Specter describes **component contracts** -- context, intent, constraints, and acceptance criteria for any component in a system, not just APIs. The contract can be runtime behavior, data invariants, security policy, schema constraints, or any other rule a component must satisfy. It answers "what must this component do or guarantee, why, and how do we verify it?"

Key differences:

- OpenAPI is scoped to HTTP APIs. Specter specs cover any feature: background jobs, state machines, authentication flows, data pipelines.
- OpenAPI does not express constraints like "MUST NOT exceed 200ms response time" or "MUST retry 3 times before failing." Specter constraints are first-class.
- OpenAPI does not model dependencies between specifications. Specter builds a dependency graph (`depends_on`) and detects circular dependencies, version mismatches, and structural conflicts across specs.
- Specter specs include acceptance criteria with explicit traceability to constraints. OpenAPI has no equivalent.

The two are complementary. An API spec in Specter might reference an OpenAPI schema for the data format while adding constraints and acceptance criteria on top.

---

## How is Specter different from Cucumber / BDD?

Cucumber and BDD tools are **test execution frameworks**. They run scenarios written in Gherkin against a live system to verify behavior.

Specter is a **pre-implementation validation tool**. It analyzes specs statically -- before any code or tests exist -- to catch structural problems: orphan constraints, circular dependencies, missing acceptance criteria, breaking changes between versions.

Key differences:

- Cucumber requires a running implementation. Specter works on specs alone.
- Cucumber validates "does the code match the scenario?" Specter validates "is the spec internally consistent and compatible with its dependencies?"
- Cucumber scenarios are prose. Specter specs are structured YAML with typed fields, constraint IDs, and explicit dependency declarations.
- Specter's coverage tool measures whether tests *exist* for each acceptance criterion. Cucumber measures whether tests *pass*.

A mature SDD workflow uses both: Specter validates specs before implementation, and test frameworks (including BDD tools) validate the implementation after.

---

## How do I adopt Specter on an existing codebase?

Start small and work outward:

1. **Generate draft specs from existing code.** Run `specter reverse src/ --output specs/` to extract draft specs from TypeScript, Python, or Go source files. Specter analyzes validation schemas, test assertions, and function signatures to produce `.spec.yaml` drafts.
2. **Review and complete the drafts.** The reverse compiler extracts structure but not intent. Review each generated spec, complete any ACs marked as gaps, and add missing constraints.
3. **Annotate existing tests.** Add `@spec` and `@ac` annotations to tests that cover the specified behavior. Run `specter coverage` to see where gaps remain.
4. **Integrate into CI.** Add `specter sync` to your CI pipeline. It exits 0 only when all discovered specs meet their configured thresholds.
5. **Expand incrementally.** Add specs for new features as they are built. Over time, spec coverage grows organically.

Use `specter doctor` to check project health at any time. It runs all pre-flight checks and tells you exactly what needs attention before running the full pipeline.

---

## What are tiers?

Tiers represent the risk level of a spec and determine how strictly Specter enforces rules.

| Tier | Label | Description | Example |
|------|-------|-------------|---------|
| 1 | Critical | Core business logic, payment flows, authentication, data integrity. Failures are costly or dangerous. | `payment-create-intent`, `user-auth` |
| 2 | Standard | Important features with moderate risk. The default tier for most specs. | `notification-send`, `report-generate` |
| 3 | Advisory | Low-risk features, internal tools, experimental work. | `admin-dashboard-layout`, `dev-metrics` |

Tier affects enforcement throughout the toolchain:

- **`specter check`**: Orphan constraints are errors in Tier 1, warnings in Tier 2, and info in Tier 3.
- **`specter coverage`**: Tier 1 requires 100% AC coverage by tests. Tier 2 requires 80%. Tier 3 requires 50%.

Set the tier in the spec file:

```yaml
spec:
  id: payment-create-intent
  version: "1.0.0"
  status: approved
  tier: 1
```

---

## What are constraint IDs and AC IDs?

Every constraint and acceptance criterion in a spec has a unique identifier:

- **Constraint IDs** follow the format `C-01`, `C-02`, `C-03`, etc.
- **AC IDs** follow the format `AC-01`, `AC-02`, `AC-03`, etc.

These IDs serve several purposes:

- **Traceability.** Each AC declares which constraints it validates via `references_constraints`. This creates a verifiable link from requirement to test.
- **Orphan detection.** `specter check` flags any constraint that is not referenced by at least one AC. If a constraint exists but nothing tests it, that is a gap.
- **Test annotation.** Test files reference AC IDs with `// @ac AC-01` annotations so that `specter coverage` can map tests back to specific acceptance criteria.
- **Communication.** In code reviews and discussions, "C-03 is not covered" is more precise than "that one constraint about retries."

Example:

```yaml
constraints:
  - id: C-01
    description: "MUST validate email format before submission"
    type: technical
    enforcement: error

acceptance_criteria:
  - id: AC-01
    description: "Valid email is accepted"
    references_constraints: ["C-01"]
    priority: critical

  - id: AC-02
    description: "Invalid email returns a validation error"
    references_constraints: ["C-01"]
    priority: critical
```

Specter enforces the ID format during parsing. `c1` or `ac-1` will be rejected; the correct formats are `C-01` and `AC-01`.

---

## How do I integrate Specter with CI?

Use `specter sync` as your CI gate. It runs the full pipeline (parse → resolve → check → coverage) in sequence and exits non-zero on any failure:

```yaml
# GitHub Actions
- name: Validate specs
  run: specter sync
```

`specter sync` exits `0` only when:
- All spec files parse without schema errors
- The dependency graph has no cycles or broken references
- No check-phase errors; warnings fail only under `--strict` or `settings.strict: true`
- All Tier 1 specs have 100% AC coverage, all Tier 2 specs have 80%, all Tier 3 specs have 50%

The exit code is `0` on success and `1` on failure, so it integrates with any CI system that checks exit codes. You can also run individual pipeline stages (`specter parse`, `specter resolve`, `specter check`, `specter coverage`) for more granular control.

---

## What languages does Specter support?

The `.spec.yaml` format is **language-agnostic**. Specs describe component contracts -- runtime behavior, data invariants, schemas, policies -- not implementation details. You can write specs for systems built in any language.

The toolchain is built in Go and distributed as a single binary with zero runtime dependencies.

For test coverage (`specter coverage`), annotation scanning supports:

- `//` comments (JavaScript, TypeScript, Go, Rust, Java, C#, etc.)
- `#` comments (Python, Ruby, Shell, YAML, etc.)

The reverse compiler (`specter reverse`) supports TypeScript, Python, and Go. It auto-detects the language from file extensions, or you can specify it with `--adapter typescript|python|go`.

---

## Can I use Specter with Claude Code / Cursor / Copilot?

Yes. Specter specs are designed to be consumed by AI tools as part of their context.

**Claude Code:** Add spec file paths to your `CLAUDE.md` so Claude reads them before writing code. Specter's own `CLAUDE.md` demonstrates this pattern -- it instructs Claude to read the relevant spec before writing or modifying any source file.

**Cursor:** Reference specs in your `.cursorrules` file. For example:

```
When working on the payment module, read specs/payment-create-intent.spec.yaml first.
All constraints in the spec are mandatory requirements.
```

**GitHub Copilot:** Include the spec content in your prompt or open the spec file in an adjacent tab so it is available as context.

The key principle: specs are plain YAML files that fit within AI context windows. Any AI tool that can read files can use them. The structured format (context, objective, constraints, acceptance criteria) gives the AI precisely the information it needs to generate correct implementations and tests.

---

# Specter: Release notes
URL: https://www.hanalyx.com/docs/specter/release

All notable changes to Specter will be documented in this file.

## [0.3.1] - 2026-04-04

### Fixed

- Route-path ID now takes priority over generic-filename logic for Next.js App Router (`onboarding-route` → `onboarding`)
- Zod `.min(N, "message")` / `.max(N, "message")` / `.email("msg")` with custom error messages now extracted correctly
- CLI discovers `prisma/schema.prisma` from parent directories when scanning a subdirectory
- Inline Python comment false positives eliminated (`# isort:skip`, `# noqa` on import lines)

### Added

- `--overwrite` flag for `specter reverse` — existing spec files are skipped by default, preserving manual edits
- `--exclude` flag for `specter reverse` — scope scans with path exclusions
- 3-tier pre-release CI workflow validating against 12 open-source repos (chi, fiber, validator, FastAPI, pydantic, django, cal.com, create-t3-app, payload, tRPC, TanStack, refine)

## [0.3.0] - 2026-04-03

### Added

- `specter.yaml` project manifest — defines system metadata, domain grouping, coverage thresholds, and spec registry
- `specter init` command — scaffolds specter.yaml from existing specs with `--name` and `--force` flags
- Domain grouping — group specs by business area (payments, auth, content) with tier inheritance
- Tier cascade — spec tier → domain tier → system tier → default (2)
- Configurable coverage thresholds per tier via manifest `settings.coverage`
- Auto-maintained spec registry rebuilt on every sync run
- Domain-level coverage aggregation
- `spec-manifest.spec.yaml` — the manifest's own spec (10 constraints, 14 ACs)
- 104 tests (up from 85), 7 specs (up from 6)

## [0.2.4] - 2026-04-03

### Fixed

- Route-path ID now takes priority over generic-filename logic for Next.js App Router files (`onboarding-route` → `onboarding`, `slug-route` → `blog-slug`)
- Zod `.min(N, "message")` and `.max(N, "message")` with custom error messages now extracted correctly (previously the closing paren regex failed when extra args were present)
- Zod `.email("message")` and `.url("message")` with custom messages now extracted correctly
- CLI discovers `prisma/schema.prisma` from parent directories when scanning a subdirectory like `src/`

### Added

- `--exclude` flag for `specter reverse` — exclude paths from scanning (e.g., `--exclude src/components --exclude "*.test.*"`)

## [0.2.3] - 2026-04-03

### Fixed

- Strip inline Python comments before constraint extraction — eliminates false positives from `# isort:skip`, `# noqa`, etc. on import lines (Django: 67 → 31 constraints, 36 false positives removed)

## [0.2.2] - 2026-04-03

### Fixed

- P0: Map unknown `validation.rule` values to `"custom"` — Specter no longer rejects its own output for Go struct tags (`gte`, `lte`, `oneof`), Python Field kwargs (`min_length`, `max_length`), or Prisma attrs (`unique`)
- P1: Add `.spec.tsx`, `.spec.jsx`, `.spec.js` to TypeScript test file detection — previously 713 test assertions silently lost in refine
- P1: Fix test description truncation on embedded quotes — `it("'visible' value")` no longer stops at the first `'`
- P1: Filter Python comment directives (`# isort`, `# noqa`, `# type:`, `# pragma`) from constraint extraction
- P2: Incorporate parent directory into spec ID for generic filenames (`index.ts` → `auth-index`, `main.go` → `rest-main`, `route.ts` → `users-route`)

### Added

- `normalizeValidationRule()` in core engine with alias mapping (gte→min, lte→max, oneof→enum, etc.)
- Generic filename detection list for spec ID collision prevention
- Improvement roadmap document (`docs/IMPROVEMENT_ROADMAP.md`)
- Updated CLAUDE.md with mission statement, design principle, and bug priority framework

## [0.2.1] - 2026-04-03

### Fixed

- Fix null `validation.value` crash when Zod patterns have no extractable literal (`.email()`, `.url()`, `.optional()`, `.refine()`, `z.boolean()`, `z.array()`)
- Fix all Next.js App Router files generating the same spec ID `route` — now derives ID from route path (e.g., `/api/webhooks/stripe` → `webhooks-stripe`)
- Include source file path in `validation_failed` diagnostics for easier debugging
- Find `package.json`/`go.mod`/`pyproject.toml` in parent directories for system name inference

### Added

- TypeScript adapter: extract constraints from TypeScript enums, union types, and `as const` arrays
- TypeScript adapter: extract constraints from Prisma schema files (`.prisma`) — field types, `@unique`, `@db.VarChar(N)`, required/optional
- TypeScript adapter: extract role and status constraints from code patterns (e.g., `session.user.role === "ADMIN"`)
- TypeScript adapter: recognize `.url()`, `z.boolean()`, `z.array()` Zod patterns
- Zod enum now extracts values (e.g., `z.enum(["a", "b"])` → `Value: "a", "b"`)
- 85 tests (up from 77)

## [0.2.0] - 2026-04-03

### Added

- `specter reverse` command — reverse compiler extracts draft .spec.yaml from existing code
- Plugin adapter architecture with 3 built-in adapters: TypeScript, Python, Go
- TypeScript adapter: Zod schemas, Jest/Vitest tests, Next.js/Express routes
- Python adapter: Pydantic models, pytest tests, FastAPI/Django routes
- Go adapter: struct validate tags, table-driven tests, net/http/gin/chi routes
- Gap detection: flags constraints without test coverage
- Auto-detection of language adapter from file extensions
- spec-reverse.spec.yaml (10 constraints, 14 ACs)
- 77 total tests (up from 37)

## [0.1.0-alpha.2] - 2026-04-02

### Changed

- Migrated from TypeScript/Node.js to Go (single static binary, zero runtime dependencies)
- All 5 MVP tools rewritten: parse, resolve, check, coverage, sync
- Cross-platform binary distribution via goreleaser (linux, darwin, windows)
- DEB package available for Debian/Ubuntu

### Added

- GitHub Actions release workflow (tag-triggered)
- Community files: issue templates, PR template, CODEOWNERS
- CHANGELOG.md

## [0.1.0-alpha.1] - 2026-03-28

### Added

- Initial TypeScript implementation of Specter
- MVP tools: parse, resolve, check, coverage, sync
- Canonical spec schema (JSON Schema draft 2020-12)
- Dogfooding: Specter validates its own specs