Counterexample Lab / Attempt 1
GPT-5.6 Luna
Oct 1, 2026 · low reasoning · Codex CLI · completed
7 / 8faults exposed
12 / 12events used
4 / 4valid traces
29.9 secwall-clock subject time
Which faults did it find?
Exposure means at least one fully valid trace made this faulty ledger disagree with the reference.
- 01Lost request historyExposed
- 02Forgotten rejectionExposed
- 03Unchecked payloadExposed
- 04Revalidated retryExposed
- 05Partial transferNot exposed
- 06Missing revisionExposed
- 07Stale writeExposed
- 08Overwritten historyExposed
The submitted test
Trace 1
Every expected output matches the reference ledger.
| # | Event | Model’s expected output |
|---|---|---|
| 01 | {
"op": "transfer",
"id": "r1",
"from": "a",
"to": "b",
"amount": 3,
"expected_revision": 0
} | {
"status": "ok"
} |
| 02 | {
"op": "inspect"
} | {
"balances": {
"a": 7,
"b": 3,
"c": 0
},
"revisions": {
"a": 1,
"b": 1,
"c": 0
}
} |
| 03 | {
"op": "transfer",
"id": "r1",
"from": "a",
"to": "b",
"amount": 4,
"expected_revision": 0
} | {
"status": "conflict"
} |
| 04 | {
"op": "restart"
} | {
"status": "restarted"
} |
| 05 | {
"op": "transfer",
"id": "r1",
"from": "a",
"to": "b",
"amount": 3,
"expected_revision": 0
} | {
"status": "ok"
} |
| 06 | {
"op": "inspect"
} | {
"balances": {
"a": 7,
"b": 3,
"c": 0
},
"revisions": {
"a": 1,
"b": 1,
"c": 0
}
} |
The submitted test
Trace 2
Every expected output matches the reference ledger.
| # | Event | Model’s expected output |
|---|---|---|
| 01 | {
"op": "transfer",
"id": "r2",
"from": "a",
"to": "b",
"amount": 1,
"expected_revision": 1
} | {
"status": "stale_revision"
} |
| 02 | {
"op": "transfer",
"id": "r3",
"from": "a",
"to": "b",
"amount": 1,
"expected_revision": 0
} | {
"status": "ok"
} |
| 03 | {
"op": "transfer",
"id": "r2",
"from": "a",
"to": "b",
"amount": 1,
"expected_revision": 1
} | {
"status": "stale_revision"
} |
| 04 | {
"op": "inspect"
} | {
"balances": {
"a": 9,
"b": 1,
"c": 0
},
"revisions": {
"a": 1,
"b": 1,
"c": 0
}
} |
The submitted test
Trace 3
Every expected output matches the reference ledger.
| # | Event | Model’s expected output |
|---|---|---|
| 01 | {
"op": "transfer",
"id": "r4",
"from": "missing",
"to": "missing",
"amount": 1,
"expected_revision": 0
} | {
"status": "unknown_account"
} |
The submitted test
Trace 4
Every expected output matches the reference ledger.
| # | Event | Model’s expected output |
|---|---|---|
| 01 | {
"op": "transfer",
"id": "r5",
"from": "a",
"to": "a",
"amount": 1,
"expected_revision": 0
} | {
"status": "same_account"
} |
The original response
Preserved verbatim from the subject. The source artifact also includes the prompt and complete scoring result.
Open raw model response
{"traces":[{"events":[{"op":"transfer","id":"r1","from":"a","to":"b","amount":3,"expected_revision":0},{"op":"inspect"},{"op":"transfer","id":"r1","from":"a","to":"b","amount":4,"expected_revision":0},{"op":"restart"},{"op":"transfer","id":"r1","from":"a","to":"b","amount":3,"expected_revision":0},{"op":"inspect"}],"expected":[{"status":"ok"},{"balances":{"a":7,"b":3,"c":0},"revisions":{"a":1,"b":1,"c":0}},{"status":"conflict"},{"status":"restarted"},{"status":"ok"},{"balances":{"a":7,"b":3,"c":0},"revisions":{"a":1,"b":1,"c":0}}]},{"events":[{"op":"transfer","id":"r2","from":"a","to":"b","amount":1,"expected_revision":1},{"op":"transfer","id":"r3","from":"a","to":"b","amount":1,"expected_revision":0},{"op":"transfer","id":"r2","from":"a","to":"b","amount":1,"expected_revision":1},{"op":"inspect"}],"expected":[{"status":"stale_revision"},{"status":"ok"},{"status":"stale_revision"},{"balances":{"a":9,"b":1,"c":0},"revisions":{"a":1,"b":1,"c":0}}]},{"events":[{"op":"transfer","id":"r4","from":"missing","to":"missing","amount":1,"expected_revision":0}],"expected":[{"status":"unknown_account"}]},{"events":[{"op":"transfer","id":"r5","from":"a","to":"a","amount":1,"expected_revision":0}],"expected":[{"status":"same_account"}]}]}Suite SHA-256: a56a4aadf5b20a319c7a57309608477d9e523da3d99c37a69bb6353f6b16f0c1