e2e: pin the cursor model, and scale a per-test prompt timeout

Commit

Stefan Haubold3d ago

Two follow-ups from the prompt-timeout review.

Cursor pinned no model, so it ran on auto -- a real listed id, and Cursor's default -- which re-routes per run. Turn duration therefore moved for reasons unrelated to the change under test, which is what made the original 188s measurement unreadable. E2E_CURSOR_MODEL now pins it, applied through cursorArgs() so the real argv and the artifact's displayCmd cannot drift, and covering StartSession as well as RunPrompt.

Checking "every other paid runner has a knob" showed it was not true. E2E_CLAUDE_MODEL had been documented far longer than anything read it, so setting it got you haiku and no warning; copilot-cli pinned a default but took no override, and its StartSession hardcoded the model where antigravity's and opencode's read theirs. Both are wired up. Droid is deliberately left alone: its default is a BYOK custom model that Bootstrap() declares in ~/.factory/settings.json, so a knob moving only the flag would name a model the settings do not declare.

The workflows pin gpt-5.4-nano-medium, measured at $0.0047 per 3-commit prompt against composer-2.5's $0.0169 (n=3 each, all completing). composer uses the FEWEST tokens of the candidates and is still 3.6x the dearest, because it bills cache reads at $0.20/M against nano's $0.02/M and an agentic loop re-sends the conversation every turn -- so do not rank these by input price. Nano is the weakest tier offered and only one prompt shape was probed, so the e2e.yml comment says to bump to gpt-5-mini (same cost, one tier up) on a capability-shaped failure rather than debug the CLI. There is no in-code default: agent --list-models is account-scoped, and a stale id exits 1 naming every valid one rather than falling back to auto.

Second, a per-test WithPromptTimeout is now scaled by the runner's TimeoutMultiplier, the same factor runForAgents already applies to the scenario timeout. The five call sites were authored when they bound half the matrix; they now bind all ten, and a duration written in a test file is authored for the work rather than for whichever runner draws the subtest. Left unscaled it bound hardest on exactly the agents the multiplier exists to give room to: TestRapidSequentialCommits asks three commits of antigravity in 120s while granting it a 10m scenario budget for being 2.5x slow.

It only ever widens, the one asymmetry with the scenario timeout. vogon and roger-roger at 0.5x are local and deterministic, so a tighter per-prompt threshold there could only fire on a loaded machine. A runner's own default and E2E_TIMEOUT stay verbatim: the first is already agent-specific and would be double-counted, the second is typed by a human for one particular run.

TestEveryRunPromptResolvesThroughPromptTimeout now also pins that each runner passes its own receiver as the scaler. That argument decides how far a budget is widened, so a runner handing over a literal or another agent's multiplier is the same silent drift the guard already existed to catch -- the run still passes, on the wrong budget. Both branches were negative- tested.

The reference docs also said nine runners resolve through promptTimeout and left antigravity out of the 60s default list; both were stale after Gemini CLI was removed and antigravity added, and are corrected here alongside the paragraphs this change adds.

Not addressed, and flagged for a maintainer: multi_session_test.go asks 3m per prompt under a 3m scenario budget with two prompts, so the number cannot fire as written for a 1.0x runner; and Claude.StartSession passes no --model at all, running interactive tests on the account default rather than the haiku its own RunPrompt pins.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com Entire-Checkpoint: 01M6DQG12X9NGA8W8PZAV2KQNR