Architecture & data flow · self-hosted infrastructure platform

Circuit Breaker

A homelab and small-datacenter control plane: it discovers what you run, draws it as a live topology, polls it for health, and keeps a tamper-evident record of every change. Every data path in the system exists to feed one screen — the topology map in §9, and the whole machine assembled in §12.

73.8kPython · backend
89.0kJS/JSX · frontend
12.7kGo · edge agent
45.4kLines of test
435Routed endpoints
109DB migrations
11Map layout engines
10Entity types graphed

§1The shape of the repo

Three deployable units in one monorepo, plus the packaging that ships them

Everything lives under apps/. The backend owns all domain logic and is the only thing that touches the database; the frontend is a pure API consumer; the Go agent is an optional edge process that runs on machines you want first-party telemetry from.

PathWhat it isStack
apps/backend API, workers, integrations, migrations. 57 router modules, 82 service modules, 80+ ORM models. Python 3.12 · FastAPI · SQLAlchemy 2 · Alembic · Pydantic v2
apps/frontend Single-page app. 29 pages, 164 components, 27 custom hooks — half of which are stream consumers. React · Vite · React Flow · sigma + graphology · d3-force · Plotly · i18next
apps/agent Edge collector: host facts, network scoping, remote probes. Ships its own updater and spool. Go 1.25 · Noise IK (flynn/noise) · gorilla/websocket · miekg/dns
docker/ · deploy/ Single-image runtime: supervisord unit files, nginx config, Postgres bootstrap, migration and OOBE hooks. Dockerfile.mono · supervisord · nginx · pgbouncer
specs/ · docs/ Release-gate evidence, ADRs, security suppression ledger, and the user-facing MkDocs site. Markdown · JSON policy files checked in CI

The one rule that shapes the backend

Routers decode and authorize; services do the work. An API module under app/api/ resolves the caller, validates a Pydantic schema, and calls into app/services/. No SQL and no domain branching live in the route layer, which is what makes the same logic reusable from the six background workers — they import services directly and never call the HTTP API.

§2One image, one process tree

Postgres, NATS, Redis, the API, six workers and nginx under a single supervisor

The deployment target is somebody's homelab, so the whole stack ships as one container with a read-only root filesystem and a single writable /data volume. Supervisord runs the process tree; the API binds only to loopback and nginx is the only thing listening on a public port. Postgres, NATS and Redis are embedded by default and each can be pointed at an external instance through an env var instead.

CONTAINER · READ-ONLY ROOTFS · CAP_DROP ALL · NO-NEW-PRIVILEGES Browser React SPA cb-agent hosts Go · Noise over WSS HTTPS nginx :8080 / :8443 FastAPI · uvicorn 2 workers, loopback only 6 workers discovery notification telemetry monitor-scheduler monitor-poll probe-dispatch NATS + JetStream publish consume SQLALCHEMY SESSIONS PostgreSQL 15 TimescaleDB · pgbouncer Redis cache · rate limits · presence supervisord — restarts any unit in place; no orchestrator required /data — pgdata · nats store · redis · uploads · vault key · worker heartbeats every embedded service is swappable for an external one via env var Managed hardware iDRAC · iLO · UPS · SNMP Proxmox · Uptime Kuma LAN under discovery nmap · mDNS · ARP · DHCP
Fig 1 Nothing but nginx is reachable from the network. The API and the six workers never call each other directly — they meet on the NATS bus and in the database, which is why any worker can be restarted, scaled out, or run against an external Postgres without the API noticing.

§3What happens to one request

Seven middleware layers, then authorization, then a service, then an audit entry

Every call from the SPA carries a HttpOnly session cookie and, on writes, a double-submit CSRF token. Starlette wraps middleware in reverse registration order, so the last one added in main.py is the first one executed — the diagram below is in execution order, which is the order that actually matters when you are debugging a 403.

Browser — axios, withCredentials Cookie: session · Header: X-CSRF-Token nginx — TLS, buffering off for streams MIDDLEWARE EXECUTION ORDER 1 · TenantMiddleware resolve tenant scope 2 · TenantRateLimit token bucket 3 · SecurityHeaders CSP · HSTS · frame-deny 4 · Logging redacted at the filter 5 · LegacyToken deprecated API tokens 6 · CSRF double-submit on writes 7 · CORS same-origin unless configured Route match — 435 endpoints every one covered by a checked-in auth policy Dependencies — require_auth / require_role role hierarchy + scopes, session revocation check Service module all domain logic lives here PostgreSQL one transaction per request Redis — counters viewer · editor · admin · demo + granular read/write scopes audit_log — SHA-256 chain actor · IP · diff · prev_hash NATS publish → live clients
Fig 2 The interesting part is layer 8. All 435 routes are enumerated into endpoint_inventory.json and diffed against endpoint_policy.json in CI: adding a route without an auth dependency fails the build unless it is explicitly listed as an allowed exception. Authorization is a tested property of the app, not a convention.

§4Telemetry: poll, queue, batch, cache

Device polling is decoupled from database writes by a durable JetStream queue

The telemetry worker polls every enabled device on its own interval — Redfish for iDRAC and iLO, SNMP for UPS units and switches, the Proxmox API for hypervisors. Credentials are pulled from the Fernet vault at poll time and never held in plaintext at rest. The naive version of this writes a row per poll; instead each sample is published to a JetStream subject and a separate pull consumer batches up to 50 rows per commit.

Devices Redfish · SNMP IPMI · Proxmox POLL telemetry_collector per-device interval client pool reuse Fernet vault decrypt credentials PUB TELEMETRY stream telemetry.ingest.{hw_id} durable across restarts PULL ingest worker batches ≤ 50 / commit acks after write TimescaleDB hypertables, retained Redis cache 60 s TTL, latest state WS /api/v1/telemetry/stream health rings update in place to every open map FALLBACK · NATS UNAVAILABLE → COLLECTOR WRITES DIRECT Decoupling buys three things: poll latency never blocks on the DB, an ingest-worker restart loses nothing, and a DB hiccup degrades to a queue depth instead of dropped samples.
Fig 3 The dashed fallback is the point of the design. When NATS is unavailable the collector writes straight through rather than failing — the queue is an optimization the system can lose without losing monitoring.

§5Real-time fan-out

One bus, six transports, one hook per stream on the client

Anything that changes state publishes a message; nothing polls the database to find out what happened. Subjects are namespaced <domain>.<entity>.<event> and declared as constants in a single module, so the set of things the system can announce is greppable in one file. Each subject family surfaces on exactly one transport.

PRODUCERS SUBJECTS TRANSPORT CLIENT Publishers 6 worker processes API route handlers agent link session integration sync notifications.* · alert.* discovery.scan.* · device.found telemetry.update · proxmox.* alert.monitor.down.{id} topology.node.* · cable.* agents.event SSE /events/stream WS /discovery/stream WS /telemetry/stream WS /monitors/stream WS /topology/stream WS /agents/stream sseClient → mitt emitter useDiscoveryStream useTelemetryStream useMonitorStream useTopologyStream useAgentLive Every stream re-checks its session every 15 s — revoking a session kills an open stream instead of waiting for the client to disconnect. SSE degrades to a 2 s DB poll if NATS is down.
Fig 4 Notifications use SSE because they are one-directional and need to survive proxies; the rest use WebSockets because the client also sends (viewport hints, subscription filters). Connections are capped globally and per-IP so a stuck browser tab cannot exhaust the server.

§6The monitoring engine

One clock, two work queues, deliberately separated

Uptime checks run on a single scheduler guarded by a Postgres advisory lock — many replicas may be running, exactly one enqueues. Each tick atomically claims due items and advances their next_due_at in the same statement, so all scheduling state lives in the database and a restart resumes cleanly with nothing wedged.

The design decision worth pointing at is the second queue. Checks executed by a remote agent go to their own JetStream work queue, never onto the server's. An agent that stops draining backs up its own queue and nothing else.

monitor_items next_due_at, interval scheduler PG advisory lock 1 s tick · ≤200/tick ≤50 per vantage MONITOR_POLL queue mon.poll.item monitor_poll worker executes from server HTTP · TCP · ICMP · DNS targets on your LAN MONITOR_PROBE queue mon.probe.remote probe_dispatch routes to a vantage cb-agent executes checks from inside the net SEPARATE STREAMS — A STALLED AGENT CANNOT DELAY A SERVER CHECK monitor_events · uptime_events state transition detected here, not at poll time notification_worker email · webhook · UI alert A check only becomes an event when its status changes — flapping is absorbed by the transition test, so a degraded target does not turn into a thousand notifications.
Fig 5 Fair sharing lives in the scheduler: no single vantage may take more than 50 slots of a 200-item tick, so one agent with a thousand assigned checks cannot starve the rest.

§7The edge agent link

Noise-encrypted, spool-backed, and written to assume the network lies

The Go agent connects outbound over WSS and authenticates with a Noise IK handshake — there is no session cookie and no bearer token on this path; the handshake is the authentication, and the agent's static key is its identity from enrollment onward. Both endpoints bypass session middleware entirely, which is why they get their own nginx locations and their own tests.

Most of the interesting engineering is in failure handling. A severed network is often not a closed socket: a firewall DROP, a container detach, or a stale NAT entry produces a black hole where writes keep succeeding into nothing. Both sides therefore run a 60 s read deadline against a 20 s heartbeat — a silent peer is a disconnect, and a disconnect is what hands the outage to the disk spool.

cb-agent Go, on your host /api/v1/agents/link FastAPI WebSocket WSS dial · pinned TLS · 10 s handshake budget Noise IK — static key from enrollment hello — host facts, arch, capabilities offered hello.ack + capabilities.set — server decides what is granted data frames — host samples, probe results, discovery findings ping every 20 s · both sides read-deadline at 60 s OUTAGE — NO FIN, NO RST, JUST SILENCE spool to disk · 64 MiB ring samples survive a reboot read deadline trips → reconnect with backoff paced catch-up — ≤40 frames/s, ≤2.5 MiB/s from the spool head transport.rekey · signed update offer when a new build ships
Fig 6 The pacing budget is the reason a day-long outage does not become an incident: 2,880 queued samples clear in about 72 seconds instead of arriving as one burst that knocks the ingest path over.

§8Discovery never writes to your inventory

Eight probe sources land in a staging table; a human promotes them

Scans fan out across whatever the deployment can reach — port sweeps, mDNS, the kernel ARP cache, DHCP leases, SNMP and LLDP neighbor tables, the Docker socket, the Proxmox API, OPNsense. Results are fingerprinted against a vendor catalog and a device knowledge base, then written to scan_results, which nothing else reads from.

Promoting a result into real hardware, service or network rows requires an explicit merge. This is a product decision with an architectural consequence: the discovery pipeline is append-only and can be re-run, dropped, or replaced without any risk to the inventory a user has curated.

Probe sources nmap · masscan mDNS / zeroconf ARP cache DHCP leases SNMP · LLDP Docker socket Proxmox · OPNsense discovery worker progress → NATS → UI fingerprint OUI · catalog · device KB scan_results staging only, inert Review a person clicks merge live inventory hardware · service dismissed kept as history Reconcilers run the same merge logic on a schedule for authoritative sources — a Proxmox cluster's own node list, an agent's own network facts — but even those write through the staging table, so every change to inventory has the same shape and the same audit trail.
Fig 7 Everything upstream of the review gate is disposable. That is what makes it safe to run aggressive scans against a production network you care about.

§10Security posture

The controls that shaped the architecture rather than sitting on top of it

This is infrastructure tooling holding IPMI credentials and hypervisor API keys for someone's entire estate. Several of the design choices above exist because of that, not despite it.

Route policy gate

All 435 routes are enumerated into a checked-in inventory and diffed against an explicit policy file in CI. A new endpoint without an auth dependency fails the build.

Fernet secrets vault

Every credential is encrypted at rest with a key that lives outside the database. The vault starts uninitialized and refuses to generate an ephemeral key at import time — a wrong key is a loud failure, never silent data loss.

Hash-chained audit log

Each entry stores SHA-256(payload + prev_hash) under a write lock, so concurrent appends cannot fork the chain. Verification walks the chain and reports the first break.

Mid-stream revalidation

Long-lived SSE and WebSocket connections re-check session validity every 15 seconds. Revoking a session ends the stream instead of waiting for the client to reconnect.

Least-privilege container

Read-only root filesystem, cap_drop: ALL with seven capabilities added back for named reasons, no-new-privileges, and supervisor-dropped UIDs per service.

SSRF and egress control

Integration targets pass a URL validator and a network ACL before any request leaves the process; outbound traffic can be pinned to a proxy, and an air-gap mode disables it entirely.

The project is pre-1.0 and says so in its own README: not fully audited, LAN-only until the release gates pass. The gates themselves — the suppression ledger, the endpoint policy, the evidence files under specs/ — are checked into the repo and run in CI.

§11Five things worth saying out loud

The decisions a reviewer will ask about, and the answers

  1. Why one container instead of a compose stack? The users are homelabbers, and every additional service is a support burden. Bundling Postgres, NATS and Redis under supervisord means the install is one command and one volume — and because each is reached through a URL from config, any of them can be pointed at an external instance when someone outgrows the default. The convenience is the default, not the ceiling.
  2. Why a message bus in a single-node app? Not for scale — for decoupling and durability. It is what lets the collector stop caring whether the database is slow, lets a worker restart mid-upgrade without losing samples, and lets six independent processes fan work out without any of them knowing the others exist. Every path that uses it also has a degraded mode that works without it.
  3. Why does the agent exist at all when SNMP already works? Vantage point. An agent sees what an out-of-band controller cannot — process-level host facts, the kernel neighbor table, and checks executed from inside a network segment the server cannot route to. It is optional, and the system is fully functional without it.
  4. What's the hardest bug you fixed here? The silent-link failure in the agent. Writes to a black-holed socket keep succeeding, so the agent believed a dead link was healthy and an entire outage's samples went into the void instead of the spool. The fix — symmetric read deadlines on both ends, tied to a heartbeat interval both sides agree on — is three lines and a long comment explaining why they are load-bearing.
  5. What would you change? main.py is 2,100 lines of router registration and lifespan wiring; that wants to be a declarative router manifest. MapPage.jsx is 3,000 lines even after ten hooks were extracted from it — the remaining orchestration should become a reducer. The frontend is JSX with JSDoc types rather than TypeScript, which the stream-heavy hooks would benefit from most. And the 80-model single ORM module should be a package split by bounded context — there is a proposal for it already in the tree.

§12The whole system, connected

Every piece from §2 through §9 on one canvas, following one loop

Read it left to right and the story is simple: the estate is polled and scanned by workers, everything they learn is announced on a bus, the API turns that into inventory and pushes live changes down open streams, and the map draws it. Read the bottom bar and the loop closes — what the user does on the map re-enters the system as new work.

AGENTS DIAL OUT — NOISE IK OVER WSS, NO SESSION, NO INBOUND PORT ESTATE 01 · COLLECT 02 · DISTRIBUTE 03 · SERVE 04 · RENDER Managed hardware Redfish · SNMP · IPMI Hypervisors & APIs Proxmox · OPNsense · Kuma LAN segments scan and check targets Agent hosts Go binary, your machines Your inbox & chat SMTP · webhooks telemetry_collector per-device interval integration sync cluster + VM inventory discovery worker scan → stage → review monitor scheduler → poll · → probe dispatch notification worker routes alerts to sinks NATS JetStream CB_EVENTS TELEMETRY MONITOR_POLL MONITOR_PROBE durable queues, at-least-once, every consumer has a degraded mode sub pub NATS → WS bridge in the API process FastAPI 435 routes, policy-gated 7 middleware layers RBAC + audit on write SSE + 5 WebSocket streams capped globally and per-IP, revalidated every 15 s nginx TLS · static SPA · per-stream proxy locations React SPA — 29 pages, 164 components TOPOLOGY MAP ENGINE 11 layout engines · React Flow + Sigma/WebGL semantic graph and visual layer fetched apart telemetry, monitor and discovery overlays fold onto nodes without touching geometry 10 saved maps over one shared inventory Discovery review · Monitors · Agents · IPAM Racks · Storage · Logs · Audit · Settings EVERY PAGE READS THE SAME API THE MAP DOES PERSIST PostgreSQL inventory · 80+ tables TimescaleDB telemetry hypertables audit_log SHA-256 hash chain Redis cache · limits · presence Fernet vault credentials at rest 05 · THE LOOP CLOSES A scan finds a host → you merge it → a node appears on the map → you right-click and add a monitor → the scheduler enqueues it → a poll worker or an agent runs the check → the event alerts you and repaints that same node. Every user action re-enters at 01 · COLLECT.
Fig 9 Three things are true everywhere on this canvas: nothing but nginx is reachable from the network, no two processes call each other directly, and no path writes to your inventory without either a human decision or an audit row. The map is not a view bolted onto the platform — it is what the platform is for.