External Benchmarks

BMO can participate in third-party coding-agent repair benchmarks as a black-box patch-producing CLI.

Benchmark-specific task materialization, prediction files, official harness execution, grading normalization, and comparison reports should live in an external project space. BMO’s product contract is the generic agent process: bmo run, provider/model selection, tool policy, logs, and bounded run evidence.

External harnesses should record model, budget, task set, allowed tools, BMO version, harness version, patch artifacts, stdout/stderr, and grading provenance. If a harness uses workspace-intelligence or stigmergic comparison lanes, it must inject only bounded non-oracular trace summaries and must not feed gold patches or official grading verdicts back into live attempts.

For the benchmark-swe and benchmark-repair run profiles, a backtick Verify with or Verification: task declaration must match an explicit --completion-validation-command exactly after surrounding whitespace is trimmed. BMO refuses conflicting commands before agent work starts. Render both from the same active-checkout manifest; prompt text does not authorize host execution of a verifier. Distinct commands in those task forms are also refused when no explicit CLI command is supplied.

Use these reports for comparison context only until a reviewed external benchmark baseline is promoted. Deterministic product confidence remains owned by BMO’s in-repo eval readiness gate.

For maintainer details, see the quality topic for external benchmarks.