I like markets and I like looking for edges, so I wanted tooling that could find them faster than I could by hand. The plan was straightforward: build a pipeline, let an LLM write the connectors, and gate everything so nothing sketchy ever runs against real money.

What I learned is the thing I keep learning at work. The tooling commoditizes and the domain knowledge doesn’t. A pipeline that can generate connectors is worth almost nothing if you don’t already know which markets are worth connecting to. Shocking, I know.

So what this actually is now is honest infrastructure wrapped around a question I hadn’t answered yet.

The gate chain is real and fully implemented. LLM-proposed connector code has to clear static analysis, run sandboxed under resource limits, produce byte-identical output across two runs, and integrate against the live engine. Shadow deployment is the terminal state; nothing has ever been promoted.

What has never happened is an LLM-written connector clearing the chain end to end. The single artifact that ever persisted came from a hardcoded fake agent, because I built the verifier before I had anything worth verifying.

That’s not much of a pitch. It’s an accurate one, and the gates work.

The trust pipeline

The system can extend itself: an LLM drafts a new venue connector — raw exchange payload in, normalized quote out. That’s a genuinely useful lever and an obvious hazard, so the design position is that trust never comes from the model. Generated code faces a gate chain (factory/gates.py, factory/pipeline.py):

  • G0 — static analysis. AST walk. Import allowlist, forbidden-call list (eval, exec, open, dunder access), fail closed on anything unparseable.
  • G1 — schema conformance. The candidate runs in an isolated subprocess against sample inputs; output must satisfy the typed contract exactly.
  • G2 — replay determinism. Same inputs, run twice; outputs must be byte-identical. Kills hidden state and clock reads.
  • G3 — integration. The candidate’s output is built into real quote models and pushed through a real detector — it has to compose with the system, not just pass its own audition.
  • G4 — adversarial review. A second, independent LLM pass grades conformance, soundness, and lookahead bias, and it runs last, only on code the deterministic gates already cleared — a probabilistic reviewer can reject clean code but can never rescue broken code.

Pass everything and the artifact lands in shadow: hot-loaded into the live engine each cycle (factory/loader.py), run against real inputs, output tagged and excluded from headline results. Fail anything and it’s rejected. Either way the artifact and its full gate trace persist immutably, so every generated line in the system can answer “who let you in.”

Quarantine is deliberately one-way right now — safe failure mode over premature trust. The specifics live in the limitations section below.

The validation lifecycle

This is the part I’d defend hardest in a design review (validation/lifecycle.py).

Every signal type declares its threshold before data collection — the floor is part of the hypothesis, not a fit to the results. When a detector fires above its floor, one row lands in an append-only signal log with the price seen at that moment. The timestamp never slides, the side is fixed at first sighting, and graded rows are immutable.

Grading happens later, against realized ground truth — settlement results, realized rates, the actual priced path — under deliberately conservative conventions: a bar that spans both stop and target counts as a stop; a gap through a level exits at the open; if ground truth never arrives, the row is void, because a signal you can’t grade is a signal you shouldn’t log.

Then the scorecard decides. Per setup type: win rate with a Wilson interval, expectancy in R with a 95% confidence interval, and a verdict — PASS only when the entire interval sits above zero at n ≥ 100, RETIRE when it sits entirely below, INSUFFICIENT_N as the honest default in between. “Surfaced as an edge” is a property of the scorecard, not a manual decision — a setup that stops earning its PASS stops being shown, automatically. Several setups I liked have been retired by their own scorecards. That’s the system working.

Underneath it, regression evals run in CI: committed fixtures replay through the real detectors and the real fill simulator, six scorecard metrics are asserted against committed baselines with per-metric tolerances, and drift fails the build (evals/runner.py). The paper fills themselves are priced at detection time plus latency, minus fees and slippage — the metric that matters is not “edges found” but “edge remaining after honesty.”

Humans gate anything that acts

Nothing in the system acts on its own. Opportunities that qualify for action become tickets with a TTL; I approve or dismiss from the dashboard; approval re-checks expiry and hands off to an executor that is paper by default and a fail-closed stub for live (engine/tickets.py). When the LLM proposes cross-venue market pairings, the proposals queue for review and approval never auto-writes the mapping. The consequence boundary — reads run free, acts get a human — turned out to matter more than any layering scheme.

What survived building it twice

The first version of this project was a .NET agent host: RabbitMQ transport, long-running domain agents, markdown skill files parsed and inlined into prompts at startup, an Opus-backed meta-orchestrator routing their outputs. It worked, taught me a lot, and is archived — superseded by this system in mid-2026.

I used to describe both systems as implementations of one five-layer architecture. Rebuilding forced me to be more honest: the layers didn’t survive the rewrite. The message bus is gone — a single-process cycle scheduler does the job at this scale. The skill-file loader is gone — domain knowledge moved into deterministic detectors and pre-registered thresholds, where it can be graded instead of trusted. The agent framing is gone — what runs is a scheduled pipeline, and calling it that is more accurate than calling it an orchestra.

What survived are boundaries, and they show up in both codebases I still run:

  • Deterministic core. No LLM call sits in a detection path — the harness states it as policy in code (detectors/ — “no LLM calls here: pure, fast, testable, replayable”), and the MLB platform dispatches grading through pure decision functions keyed on canonical stats. The scheduler does run a few LLM-backed analysis jobs on slow wall-clock gates — a nightly memo, a classifier, a pairing proposer — each wrapped so a model outage can’t break a cycle, and none able to create a signal or a mapping without human review. The boundary is drawn at consequence, not at the process edge.
  • Probabilistic and uncontrolled input quarantined behind contracts. Generated code lives behind the gate chain and shadow state here; in the MLB platform, unknown stat labels land in an unmapped-signals inbox for human review instead of a best-guess default branch.
  • Humans gate actions. Tickets and proposal review here; the review inbox there.
  • Evals as the admission price. CI-enforced regression evals and scorecard verdicts here; a grader-comparison harness and regression tests pinned to specific past bugs there.

A pattern that only survives as those four sentences is a smaller claim than a five-layer architecture. It’s also the true one, which makes it the only one worth publishing.

Known limitations

Stated plainly, because a trust pipeline that oversells itself is self-refuting:

  • G4 runs only from the CLI. The deployed server path exercises the deterministic gates G0–G3; the adversarial LLM review fires only when the factory is invoked by hand with a model backend configured. The chain the running service enforces is shorter than the chain the design describes.
  • Eval coverage is narrow. The committed fixtures exercise three of the eight live detectors end-to-end. They are regression rails, not a validation suite.
  • Single process, by design. Cycle cadence is seconds; the work is I/O-bound polling and cheap pure computation. There is no HA story and none planned at this scale.

What I deliberately don’t publish

The domain content — which venues, which thresholds, which setups currently hold a PASS — stays private, for the obvious reason: a published edge is a closed edge. The about page frames the betting work in deliberately abstract terms, and that’s the level this page stays at. What’s worth describing is the machinery that keeps the system honest, because that part generalizes to any domain where a model proposes and reality grades.