Counterexample Lab / Attempt 3
GPT-5.5
Oct 1, 2026 · low reasoning · Codex CLI · completed
5 / 8faults exposed
12 / 12events used
3 / 3valid traces
20.7 secwall-clock subject time
Which faults did it find?
Exposure means at least one fully valid trace made this faulty ledger disagree with the reference.
- 01Lost request historyExposed
- 02Forgotten rejectionExposed
- 03Unchecked payloadExposed
- 04Revalidated retryExposed
- 05Partial transferNot exposed
- 06Missing revisionExposed
- 07Stale writeNot exposed
- 08Overwritten historyNot exposed
The submitted test
Trace 1
Every expected output matches the reference ledger.
| # | Event | Model’s expected output |
|---|---|---|
| 01 | {
"op": "transfer",
"id": "r1",
"from": "a",
"to": "b",
"amount": 3,
"expected_revision": 0
} | {
"status": "ok"
} |
| 02 | {
"op": "transfer",
"id": "r1",
"from": "a",
"to": "b",
"amount": 3,
"expected_revision": 0
} | {
"status": "ok"
} |
| 03 | {
"op": "transfer",
"id": "r1",
"from": "a",
"to": "b",
"amount": 4,
"expected_revision": 0
} | {
"status": "conflict"
} |
| 04 | {
"op": "inspect"
} | {
"balances": {
"a": 7,
"b": 3,
"c": 0
},
"revisions": {
"a": 1,
"b": 1,
"c": 0
}
} |
The submitted test
Trace 2
Every expected output matches the reference ledger.
| # | Event | Model’s expected output |
|---|---|---|
| 01 | {
"op": "transfer",
"id": "r2",
"from": "b",
"to": "a",
"amount": 1,
"expected_revision": 0
} | {
"status": "insufficient_funds"
} |
| 02 | {
"op": "transfer",
"id": "r3",
"from": "a",
"to": "b",
"amount": 5,
"expected_revision": 0
} | {
"status": "ok"
} |
| 03 | {
"op": "transfer",
"id": "r2",
"from": "b",
"to": "a",
"amount": 1,
"expected_revision": 0
} | {
"status": "insufficient_funds"
} |
| 04 | {
"op": "inspect"
} | {
"balances": {
"a": 5,
"b": 5,
"c": 0
},
"revisions": {
"a": 1,
"b": 1,
"c": 0
}
} |
The submitted test
Trace 3
Every expected output matches the reference ledger.
| # | Event | Model’s expected output |
|---|---|---|
| 01 | {
"op": "transfer",
"id": "r5",
"from": "a",
"to": "c",
"amount": 4,
"expected_revision": 0
} | {
"status": "ok"
} |
| 02 | {
"op": "restart"
} | {
"status": "restarted"
} |
| 03 | {
"op": "transfer",
"id": "r5",
"from": "a",
"to": "c",
"amount": 4,
"expected_revision": 0
} | {
"status": "ok"
} |
| 04 | {
"op": "inspect"
} | {
"balances": {
"a": 6,
"b": 0,
"c": 4
},
"revisions": {
"a": 1,
"b": 0,
"c": 1
}
} |
The original response
Preserved verbatim from the subject. The source artifact also includes the prompt and complete scoring result.
Open raw model response
{"traces":[{"events":[{"op":"transfer","id":"r1","from":"a","to":"b","amount":3,"expected_revision":0},{"op":"transfer","id":"r1","from":"a","to":"b","amount":3,"expected_revision":0},{"op":"transfer","id":"r1","from":"a","to":"b","amount":4,"expected_revision":0},{"op":"inspect"}],"expected":[{"status":"ok"},{"status":"ok"},{"status":"conflict"},{"balances":{"a":7,"b":3,"c":0},"revisions":{"a":1,"b":1,"c":0}}]},{"events":[{"op":"transfer","id":"r2","from":"b","to":"a","amount":1,"expected_revision":0},{"op":"transfer","id":"r3","from":"a","to":"b","amount":5,"expected_revision":0},{"op":"transfer","id":"r2","from":"b","to":"a","amount":1,"expected_revision":0},{"op":"inspect"}],"expected":[{"status":"insufficient_funds"},{"status":"ok"},{"status":"insufficient_funds"},{"balances":{"a":5,"b":5,"c":0},"revisions":{"a":1,"b":1,"c":0}}]},{"events":[{"op":"transfer","id":"r5","from":"a","to":"c","amount":4,"expected_revision":0},{"op":"restart"},{"op":"transfer","id":"r5","from":"a","to":"c","amount":4,"expected_revision":0},{"op":"inspect"}],"expected":[{"status":"ok"},{"status":"restarted"},{"status":"ok"},{"balances":{"a":6,"b":0,"c":4},"revisions":{"a":1,"b":0,"c":1}}]}]}Suite SHA-256: a56a4aadf5b20a319c7a57309608477d9e523da3d99c37a69bb6353f6b16f0c1