Codex CLI calibration rules

The work

Three small JavaScript repositories: an event ledger, a build cache, and an access policy engine. Each starts with broken code and two visible tests. Codex can read files, edit code, and run local tests.

The first work order fixes a bug. The second adds a requirement. Each order starts a fresh Codex session. Source files and added tests carry over; conversation history does not. The agent does not receive evaluation results between orders.

The score

Each order has 32 checks: eight fixed boundary cases, plus generated cases. The second order includes eight regression cases. Each check earns one point. A project passes only when all 64 checks pass. The full run has 192 checks. No model judges the code.

Python computes expected results. A separate JavaScript reference implementation checks the test design. Reference scores are test fixtures, never model results. The exporter reruns saved code before it publishes a score.

The limits

ChatGPT subscription only, through Codex CLI. The runner selects the lowest supported effort from the local catalog. Six sessions share 600 seconds of agent time, with at most 120 seconds each. Time includes CLI startup, generation, and tool use. Local grading is outside this budget.

The runner does not retry sessions, change models, or use an API key. Codex may retry transport requests internally. Error events, timeouts, and missing answers stop the run and leave its total unscored. Code errors fail checks. Changes to protected files fail all checks for that order.

Each check gets a fresh QuickJS context: 32 MiB memory, 1 MiB stack, and 500 ms CPU time. Source is limited to 24 JavaScript files and 128 KiB. No host files or network are exposed to the grader.

Compare like with like

The seed fixes generated inputs. Levels increase input sizes; they do not change the number of checks. Compare only matching suite hashes, seeds, levels, budgets, and runtimes. Use several preselected seeds and keep every attempt. Report time and token use separately from correctness.

Build cases use one quarter of the operation count, with at least eight requests. Policy cases add sentinel records. Fixed boundary cases have their own sizes. Exact inputs are saved with each run.

What it does not prove

This tests repairs and requirement changes in small, synthetic repositories. It does not test large application design, user interfaces, deployment, or long-term agent memory. One run cannot rank models.

Generators, prompts, and references are public. Evaluation inputs are kept outside the task directory, but the local CLI sandbox is not a secure hidden-test boundary. Workspace write limits restrict writes; they do not fully isolate reads from the host. These results are not contamination-proof.

Version 2.0 fixes the contract, generators, scoring, and budgets. A change to those rules requires a new version. Future versions can add harder projects while old results remain reproducible. No fixed test suite can promise to stay difficult for ten years.