Circuit Breaker 1.0.0 Service Objectives¶
Status: Approved 2026-08-28 under ADR-0002 (operations, architecture, product, release).
Every target below is an approved candidate value. RC-05 needs the SRV-03/REL-21 metrics,
which nothing emits yet; RC-06 needs ACC-09 through ACC-15 and REL-21 through REL-26. Until
those run, these numbers are the project's stated intent, not a measured promise.
Requirements: RC-05, RC-06
Companion slice: specs/1.0.0/slices/rc-2-compatibility-service-objectives.md
These objectives define the measurable 1.0.0 service contract. Targets are candidates until SRV-03, ACC-14, ACC-15, and REL-21 through REL-26 produce release-candidate evidence.
Health-state contract¶
| State | User meaning | Safe operations | Unsafe operations | Current implementation status |
|---|---|---|---|---|
| Startup | Process is booting, migrations/config/vault initialization may still be running. | None unless explicitly documented by the installer. | Writes, background jobs, agent dispatch, backup/restore. | Implemented. GET/HEAD /api/v1/startupz returns 503 with started: false until initialization completes, and 200 thereafter — including while stopping, so a slow migration is not mistaken for a dead process. |
| Live | Process can answer a local HTTP request. | Restart watchdog and process supervision. | Treating this as user readiness. | Implemented. GET/HEAD /api/v1/livez touches no dependency and takes no lock; it answers 200 whenever the event loop is running, including through a database or Redis outage. This is the only probe a restart decision should turn on. |
| Ready | Database schema is usable, required dependencies are reachable or safely optional, and writes can be served. | Normal UI/API, CLI admin, workers, agent dispatch. | None under the approved support matrix. | Implemented as a signal only. GET/HEAD /api/v1/readyz returns 200 with ready: true when server state is ready and every dependency probe (Postgres schema check, Redis) answers ok, and 503 otherwise. It reports readiness; it does not enforce it — see the "Not ready" row. |
| Degraded | Core inventory is available, but optional capability is unavailable or stale. | Read inventory, edit inventory when DB is healthy, view known degradation. | Dispatching work through the degraded capability unless the operation has an explicit fallback. | Implemented. app.core.health.HealthState carries all five states, and readyz reports the derived one in health alongside a degraded list naming the optional dependencies that are down. Redis is the optional dependency today: losing it costs shared rate limits, the telemetry cache and pub/sub, so it is degraded — reads and inventory edits stay safe and writes stay open. The lifecycle state field keeps its previous starting \| ready \| stopping meaning. |
| Not ready | Required dependency, schema, vault, or migration state prevents safe writes. | Health/diagnostics and documented recovery commands. | Writes, destructive admin actions, background dispatch, agent work. | Implemented. WriteAdmissionMiddleware refuses every mutating request under /api/ while a required dependency cannot answer (PostgreSQL, or a schema that does not match this build) or while the process is starting or draining — 503 with error_code SERVICE_NOT_READY or SERVER_DRAINING, a health field, and Retry-After: 5. Reads, WebSocket sessions and the four health endpoints are deliberately not guarded. readyz reports writes_permitted so the signal and the enforcement cannot disagree. The dependency verdict is cached for CB_HEALTH_CACHE_TTL_S (default 2 s), which bounds how long a write can be admitted against a database that has just gone away. |
| Stopping | Process is shutting down. | Drain in-flight requests where possible. | New writes and new dispatch. | Implemented. readyz returns 503 with state: "stopping" so a load balancer stops sending new requests before the process goes away, while livez and startupz stay 200 so the supervisor does not kill the drain. |
Readiness must not be defined as “the readiness endpoint is green.” It means the named user operations are safe. All five rows now meet that bar: Degraded is a distinct state the server reports, and Not ready is enforced rather than merely advertised. What the table still describes as candidate is the measured behaviour under load — see the SLO section below.
The legacy GET/HEAD /api/v1/health keeps its previous response shape for the frontend poll and
the installer's readiness wait. Container HEALTHCHECK directives no longer poll it — they poll
/api/v1/livez, and tests/build/test_restart_probes.py fails if one reverts to /health. New consumers should use /api/v1/livez,
/api/v1/readyz and /api/v1/startupz.
RC-05's acceptance also requires that "SRV-03 and REL-21 tests produce the named metrics." The
process-lifetime series that measure availability, latency, health-state occupancy, write-admission
rejections and background-job outcomes are emitted today — circuitbreaker_http_requests_total,
circuitbreaker_http_request_duration_seconds, circuitbreaker_health_state,
circuitbreaker_write_admission_rejections_total and circuitbreaker_background_job_runs_total,
documented in Metrics § Service-objective series.
What is still missing is the other half: no REL-21 through REL-26 load, soak or fleet benchmark has been run against a release candidate, so no objective below has a measured value. A target with an instrument but no measurement is still a candidate target, which is why the SLOs, the scale ceilings and the RPO/RTO values on this page remain unevidenced in the requirement ledger. The queue-depth and lag indicators the "Background processing backlog" objective names are also not yet emitted.
Candidate SLOs¶
| Objective | Candidate target | Window | Measurement source | Evidence owner |
|---|---|---|---|---|
| API availability for authenticated UI/API requests | 99.5% successful non-5xx responses inside supported deployment boundary | 30 days | Reverse-proxy/backend request logs or future HTTP request metrics | REL-21 owner |
| API latency for common inventory reads | p95 under 500 ms at the approved medium dataset | 24-hour load test | REL load-test report | REL-21 owner |
| UI startup | p95 initial usable UI under 3 seconds on supported browsers at medium dataset | Browser performance test | ACC-4, REL-24 evidence | ACC-4 owner |
| Monitoring execution freshness | 99% of enabled checks complete within 2 scheduled intervals when their vantage is ready | 24-hour and 7-day soak | Monitoring run records and scheduler metrics | REL-25 owner |
| Notification delivery enqueue | 99% of notification events enqueue within 60 seconds when NATS/worker are ready | 24-hour soak | Notification worker logs/metrics | REL-25 owner |
| Agent presence freshness | 99% of connected agents update presence within the approved freshness window | 24-hour and 7-day soak | Agent presence records and websocket events | AGT-7 owner |
| Backup completion | Scheduled backup succeeds at least once per 24 hours when configured and storage is healthy | 7-day soak | Backup records, logs, restored artifact evidence | ACC-14 owner |
| Background processing backlog | Queues remain below approved bounds and drain after dependency recovery | 24-hour and failure-injection tests | NATS/worker metrics and logs | REL-22 owner |
Targets that lack instrumentation remain candidate targets, not release promises. Missing metrics become implementation work under SRV-07 or REL slices.
Recovery and retention objectives¶
| Objective | Candidate value | Evidence required |
|---|---|---|
| RPO | 24 hours for scheduled backups; manual backup recommended before upgrades or destructive actions. | ACC-14 backup-under-load and restore evidence. |
| RTO | 4 hours for documented restore to a clean supported host at the approved medium dataset. | ACC-14 and ACC-15 restore procedure evidence. |
| Local backup retention default | 7 snapshots. | Admin DB settings/defaults plus prune tests. |
| Remote/S3 backup retention default | 30 snapshots when remote backup is configured. | Admin DB settings/defaults plus prune tests. |
| Audit-log retention | Retain indefinitely by default unless a later retention feature is approved. | REL-2 bounds and storage-growth evidence. |
| Upload/icon/branding retention | Retain until user deletion or restore replacement. | ACC-14 restore reconciliation evidence. |
| Metrics retention | Prometheus scrape retention is external; Circuit Breaker does not promise internal metrics history in 1.0.0. | Docs and metrics contract review. |
| Telemetry/check history retention | Candidate default pending REL-2 and REL-25 evidence. | Soak and bounded-growth evidence. |
Candidate scale ceilings¶
| Profile | Candidate maximum | Evidence required |
|---|---|---|
| Small | 100 inventory objects, 10 agents, 50 monitors, 1 active user. | ACC-3 and REL-21 smoke/load evidence. |
| Medium | 1,000 inventory objects, 50 agents, 500 monitors, 5 concurrent users. | REL-21 through REL-24 load evidence. |
| Fleet | Deferred for 1.0.0 support. | REL-26 plus AGT fleet evidence before promotion. |
If a deployment exceeds the approved profile, support is best-effort until REL evidence expands the ceiling. The application should fail with bounded queues, backpressure, and actionable errors rather than unbounded resource growth.
Evidence traceability¶
| Contract area | Owning acceptance evidence |
|---|---|
| Health-state behavior | SRV-03 |
| Upgrade compatibility | ACC-12 |
| Failed upgrade and migration recovery | ACC-13 |
| Backup RPO/RTO | ACC-14, ACC-15 |
| Load and latency | REL-21 through REL-24 |
| Soak, retention, and leak/growth budgets | REL-25 |
| Fleet scale | REL-26, AGT-7 |