Fleet runtime observability

Fleet runtime observability answers one operator question fast: is BMO’s fleet-regulation loop healthy, and if not, why? Is the controller up, is the observer armed, is the loop stuck (sticky-degraded) — with the same answer whether you ask from the CLI, TUI, HTTP, or an agent. Reach for it when you run fleet regulation and need a quick runtime read without opening logs, or when you’re triaging a stuck loop after a failed cycle.

It is the runtime-observability surface for the fleet-regulation loop (legacy alias: Fleet metabolism), rendering one posture envelope across every surface. Three nouns recur throughout this page:

• Feature — Fleet runtime observability (the operator-facing capability)
• Read model — the posture snapshot (fleetmetabolism.PostureSnapshot)
• Envelope — the rendered posture (RenderPosture), identical on every surface modulo captured_at

Maturity: Advanced integration surface. The fleet-regulation observer is off by default and starts in observe_only mode. Auto-apply is explicitly armed, not default-on. Use bmo metabolism inspect when you need design-time decision genealogy instead of a fast runtime read. Generate a validator-backed starter config with bmo config suggest-fleet-regulation instead of hand-authoring the first recommend_rules stanza.

Observability family

Fleet runtime observability is the fleet-scoped member of BMO’s runtime observability genus. Its session-scoped sibling is Run observability — recent run summary, cue ledger, and trace lens for the current session family. The two families share the observability pattern (one read model, many surfaces, hub-smoke parity) but not scope or read model.

When to use it

• You need a single read-only posture for FleetController and FleetRegulationObserver without opening logs.
• You are triaging sticky-degraded state or gate reasons after a failed cycle.
• You want the same posture JSON from TUI, CLI, HTTP, native agent tool, or MCP — for operator scripts or in-process agents.
• You are pairing metabolism.fired → metabolism.action events in logs and want the ring and histogram to corroborate.

Use bmo metabolism inspect when the question is ADM manifest genealogy, not live fleet state. Use /metabolism (the inspector dialog) when you want an interactive cycle/recommendation viewer — it complements but does not replace the inline posture report.

Runtime vs. design-time metabolism

Question	Surface	Scope
”What is the live fleet-regulation loop doing right now?”	bmo config show-fleet-metabolism, /fleet-metabolism, metabolism_posture, GET /v1/metabolism/posture	Runtime posture snapshot (this feature)
“What does the ADM spine say about decision genealogy?”	bmo metabolism inspect	Design-time / decision metabolism (ADM manifest)

The two families are intentionally disjoint. Maintainer detail on code-path separation lives in the Fleet runtime observability topic (legacy path: fleet-metabolism).

The posture snapshot

A single posture snapshot aggregates:

• Source and freshness — every posture says whether it came from a live runtime or a detached config-backed projection, and whether the current surface is actually attached to the runtime that produced it.
• FleetController state — present/absent, mode (disabled, observe_only, active, paused), pause reason if any, leader flag, gate reason. Sourced from App.FleetControllerManager().FleetControllerStatus.
• FleetRegulationObserver state — enabled-in-config flag, configured mode (disabled, observe_only, recommend_only, auto_apply), API error-rate count, synthetic-monitor count and critical count, last observed timestamp.
• Sticky-degraded flag — lifecycle-shaped health: StickyDegraded plus DegradedSince and a closed-set Reasons vocabulary (see below). The flag persists across at least one quiescence interval after the last fault; the next observed success emits metabolism.action degraded_recovered and clears the flag.
• Action × outcome histogram — bounded count matrix over the closed MetabolismAction × Outcome enums.
• Recent-event ring — fixed-capacity (32) FIFO of metadata-only RecentMetabolismEvent records, hashed identifiers only.

flowchart TB
    subgraph inputs [Signal ingress]
        healthSignals["POST /v1/health-signals"]
    end
    subgraph controller [FleetController]
        fleetCtrl["Mode, cycles, actuations"]
    end
    subgraph observer [FleetRegulationObserver]
        fleetObs["Rules, aggregates, recommendations"]
    end
    subgraph snapshot [PostureSnapshot]
        sticky["Sticky-degraded flag"]
        histogram["Action x outcome histogram"]
        ring["Recent event ring cap 32"]
    end
    healthSignals --> fleetObs
    fleetObs --> fleetCtrl
    fleetCtrl --> snapshot
    fleetObs --> snapshot

Operator surfaces

Surface	Scope	Purpose
bmo config show-fleet-metabolism (--format=text|json, --source=live|config)	fleet runtime	Headless posture envelope; prefers a loopback-safe live read and falls back to an explicit detached config-backed projection
bmo config suggest-fleet-regulation	fleet runtime	Print a validated starter stanza for observe_only, recommend_only, or auto_apply without editing disk
/fleet-metabolism (aliases /fleet_metabolism, /metabolism-posture, /metabolism_posture)	fleet runtime	Inline posture text in the chat transcript
/metabolism	fleet runtime	Interactive Metabolism inspector dialog (cycle/recommendation viewer)
Metabolism sidebar chip	fleet runtime	Live status badge from PostureSnapshot.State
GET /v1/metabolism/posture	fleet runtime	JSON posture envelope over HTTP (requireAuth)

The CLI posture surface is filed under config show-* for discoverability alongside Run observability (bmo config show-run-observability). Both families expose runtime truth through a config show-* entry point. For fleet metabolism, detached reads stay explicit instead of collapsing into a fake live failure: use --source=config when you want the config-backed view even if a local runtime is available.

Agent and integration surfaces

Surface	Scope	Purpose
metabolism_posture	fleet runtime	Native agent read-only posture JSON
list_recent_metabolism_events	fleet runtime	Native agent recent-ring read
bmo_metabolism_posture	fleet runtime	MCP parity twin for cross-process agents

Naming map

The product noun (Fleet runtime observability) differs from stable surface identifiers (legacy alias Fleet metabolism). Use this table when searching across layers:

Layer	Name
Product noun (this page)	Fleet runtime observability
Legacy alias	Fleet metabolism — logs metabolism.*, package fleetmetabolism, parity row fleet_metabolism
Feature page slug / TUI slash	fleet-metabolism
HTTP route group	metabolism (/v1/metabolism/posture, control routes)
Agent / MCP tools	metabolism_posture, bmo_metabolism_posture

The surface-parity matrix row fleet_metabolism (integration_surface in docs/agent-native/surface-parity-matrix.yaml) is the maintainer index for all aliases above.

Cross-surface parity

The same posture snapshot flows through CLI, slash, sidebar, agent tool, MCP, and HTTP. A Pattern-4 hub composition smoke (TestFleetMetabolismHubSmoke_AllSurfacesEmitTheSameJSON) asserts JSON equality modulo captured_at, preventing formatter-divergence across surfaces.

Every live surface reads the app-owned builder fleetmetabolism.BuildSnapshot. That shared seam keeps HTTP, TUI, native tools, and MCP aligned without routing through HTTP as an internal dependency.

Posture vocabulary

What the loop did (action × outcome)

MetabolismAction is bounded to: observe, recommend, auto_apply, pause, unpause, mode_change, cycle, degraded_recovered.

Outcome is bounded to: ok, failed, timeout, canceled, skipped.

Why it is degraded (reason × error category)

Reasons is a closed PostureReason enum:

Reason	Meaning
controller_unreachable	FleetController manager is nil or returned a non-success status; runtime cannot read live mode/pause state
config_absent	[options.fleet_regulation] is missing or the regulation section did not load; observer cannot fire rules
policy_denied	A configured rule predicate failed the runtime policy gate (e.g. mode=observe_only with an auto_apply request)
persistence_unhealthy	Controller or observer state-store write failed; sticky-degraded persists until a successful op clears it
auto_apply_skipped_pending	An auto-apply rule matched but a prior actuation is still pending evaluation; the patch was deliberately skipped

ErrorCategory is bounded to: rule_predicate_failed, patch_apply_failed, patch_rejected, rate_limited, controller_missing, status_unavailable, context_done, config_absent, policy_denied, pending (plus the empty sentinel for ok outcomes).

Enable / pause workflow

Generate a validated starter with:

bmo config suggest-fleet-regulation --mode observe_only
bmo config suggest-fleet-regulation --mode recommend_only
bmo config suggest-fleet-regulation --mode auto_apply --auto-apply-confirmed --shared-runtime --partition-id tenant-1

Or author directly in bmo.toml:

[options.fleet_regulation]
enabled = true
mode = "observe_only"           # or "recommend_only", "auto_apply"

  [[options.fleet_regulation.recommend_rules]]
  id = "high-error-rate-single-topology"
    [options.fleet_regulation.recommend_rules.when]
    api_error_rate_at_least = 0.15
    [options.fleet_regulation.recommend_rules.recommend]
    gene = "topology"
    value = "single"

[options.fleet_controller]
mode = "observe_only"           # disabled, observe_only, active, paused
interval_seconds = 30

Mutate the live posture through the existing runtime control routes: POST /v1/metabolism/mode, POST /v1/metabolism/pause, POST /v1/metabolism/unpause, and the agent-tool equivalents (metabolism_set_mode, metabolism_pause, metabolism_unpause). This iteration adds no new mutation routes (Scope Boundaries).

Fail-closed contract

When the loop cannot safely act, it never silently reports ok. It sets sticky-degraded with a named Reason, records the skip on the histogram and recent ring, and auto-recovers on the next successful cycle via metabolism.action degraded_recovered.

The matrix below is the proof: these arms never fire auto_apply_* and never advance the histogram into a misleading ok outcome.

stateDiagram-v2
    [*] --> healthy: nominal observe cycle
    healthy --> configAbsent: regulation section missing
    healthy --> controllerUnreachable: manager nil or status fail
    healthy --> policyDenied: auto_apply blocked by mode
    healthy --> persistenceUnhealthy: state-store write fail
    healthy --> pendingActuation: prior actuation still pending
    configAbsent --> stickyDegraded: Reasons += config_absent
    controllerUnreachable --> stickyDegraded: Reasons += controller_unreachable
    policyDenied --> stickyDegraded: Reasons += policy_denied
    persistenceUnhealthy --> stickyDegraded: Reasons += persistence_unhealthy
    pendingActuation --> stickyDegraded: Reasons += auto_apply_skipped_pending
    stickyDegraded --> healthy: degraded_recovered on next success
    state configAbsent:::blocked
    state controllerUnreachable:::blocked
    state policyDenied:::blocked
    state persistenceUnhealthy:::blocked
    state pendingActuation:::blocked
    classDef blocked stroke-dasharray:4 2

Arm	Behavior
[options.fleet_regulation] missing	observer is a no-op; emits metabolism.action observe outcome=skipped error_category=config_absent; posture Reasons includes config_absent
FleetController manager nil / unavailable	controller posture reports present=false; posture Reasons includes controller_unreachable; sticky-degraded set
Mode is disabled or observe_only, request is auto_apply	observer emits metabolism.action auto_apply outcome=skipped error_category=policy_denied; no patch applied; posture Reasons includes policy_denied
Persistence write fails	sticky-degraded set with Reasons += persistence_unhealthy; clears on next success via metabolism.action degraded_recovered
Rule predicate matched but prior actuation pending	observer emits metabolism.action auto_apply outcome=skipped error_category=pending; posture Reasons includes auto_apply_skipped_pending

Every gating decision lands on the recent ring (cap 32) with hashed session_id_hash, rule_id_hash, and gene_hash. Raw session, request, or rule identifiers never reach the log layer or the posture surfaces.

Tracing recipe

Walk a fleet-metabolism cycle end-to-end without leaving the operator seat:

sequenceDiagram
    participant Operator
    participant TUI as TUI /fleet-metabolism
    participant CLI as bmo config show-fleet-metabolism
    participant Logs as bmo logs --tail
    participant Ring as Recent event ring

    Operator->>TUI: confirm controller and observer posture
    Operator->>CLI: same RenderPosture envelope headless
    Operator->>Logs: jq filter metabolism.* events
    Logs->>Ring: bounded FIFO metadata only
    Operator->>Logs: pair metabolism.fired with metabolism.action
    Note over Ring: session_id_hash correlates lifecycle pairs

1. /fleet-metabolism (TUI) or bmo config show-fleet-metabolism — confirm controller/observer state and sticky-degraded flag.
2. Tail recent events:

   bmo logs --tail 1000 | jq -c 'select(.msg|startswith("metabolism."))'

3. Filter the failure arms only:

   bmo logs --tail 1000 \
     | jq -c 'select(.msg=="metabolism.action" and (.outcome=="failed" or .outcome=="timeout"))'

4. Pair metabolism.fired → metabolism.action by session_id_hash for the full action lifecycle:

   bmo logs --tail 2000 \
     | jq -c 'select(.msg|startswith("metabolism."))' \
     | jq -s 'group_by(.session_id_hash) | map({session_id_hash: .[0].session_id_hash, events: map({msg, action, outcome, error_category, rule_id_hash, gene_hash, latency_ms})})'

5. If Reasons includes persistence_unhealthy, the next successful observe/cycle will emit metabolism.action action=degraded_recovered outcome=ok and clear the sticky flag automatically.

The full recipe lives in agent tracing recipes and is fixture-validated in the regression suite.

Current posture

Fleet runtime observability ships with explicit maintainer validation around hub composition, fail-closed behavior, and cross-surface parity:

• Hub composition is covered by smoke tests (TestFleetMetabolismHubSmoke_*) in CI across the surfaces above.
• Fail-closed matrix green across all five arms.
• Surface-parity matrix fleet_metabolism row remains parity with no divergent renderers.

The operator-facing surface documented here is the observable/read-only loop plus explicitly armed actuation paths.

Related

• Run observability — sibling runtime-observability family (session-scoped).
• Run ledger — paired posture/ring pattern donor.
• Agent mesh — mesh.fired/.action pattern donor.
• Fleet runtime observability parity matrix (topic) — maintainer-facing v0/v1/v2 signal flow and ADM split callout.
• Tracing recipes — metabolism.* recipe block.
• Tools reference