Skip to content
mager-bench1.3
← All attempts
Counterexample Lab / Attempt 1

GPT-5.5

Oct 1, 2026 · low reasoning · Codex CLI · completed

6 / 8faults exposed
12 / 12events used
3 / 3valid traces
21.0 secwall-clock subject time

Which faults did it find?

Exposure means at least one fully valid trace made this faulty ledger disagree with the reference.

  • 01Lost request history
    Exposed
  • 02Forgotten rejection
    Exposed
  • 03Unchecked payload
    Exposed
  • 04Revalidated retry
    Exposed
  • 05Partial transfer
    Not exposed
  • 06Missing revision
    Exposed
  • 07Stale write
    Exposed
  • 08Overwritten history
    Not exposed
The submitted test

Trace 1

Every expected output matches the reference ledger.

5 events
#EventModel’s expected output
01
{
  "op": "transfer",
  "id": "r1",
  "from": "a",
  "to": "b",
  "amount": 3,
  "expected_revision": 0
}
{
  "status": "ok"
}
02
{
  "op": "restart"
}
{
  "status": "restarted"
}
03
{
  "op": "transfer",
  "id": "r1",
  "from": "a",
  "to": "b",
  "amount": 3,
  "expected_revision": 0
}
{
  "status": "ok"
}
04
{
  "op": "transfer",
  "id": "r1",
  "from": "a",
  "to": "b",
  "amount": 4,
  "expected_revision": 0
}
{
  "status": "conflict"
}
05
{
  "op": "inspect"
}
{
  "balances": {
    "a": 7,
    "b": 3,
    "c": 0
  },
  "revisions": {
    "a": 1,
    "b": 1,
    "c": 0
  }
}
The submitted test

Trace 2

Every expected output matches the reference ledger.

4 events
#EventModel’s expected output
01
{
  "op": "transfer",
  "id": "s1",
  "from": "a",
  "to": "b",
  "amount": 1,
  "expected_revision": 1
}
{
  "status": "stale_revision"
}
02
{
  "op": "transfer",
  "id": "s2",
  "from": "a",
  "to": "c",
  "amount": 2,
  "expected_revision": 0
}
{
  "status": "ok"
}
03
{
  "op": "transfer",
  "id": "s1",
  "from": "a",
  "to": "b",
  "amount": 1,
  "expected_revision": 1
}
{
  "status": "stale_revision"
}
04
{
  "op": "inspect"
}
{
  "balances": {
    "a": 8,
    "b": 0,
    "c": 2
  },
  "revisions": {
    "a": 1,
    "b": 0,
    "c": 1
  }
}
The submitted test

Trace 3

Every expected output matches the reference ledger.

3 events
#EventModel’s expected output
01
{
  "op": "transfer",
  "id": "t1",
  "from": "a",
  "to": "b",
  "amount": 10,
  "expected_revision": 0
}
{
  "status": "ok"
}
02
{
  "op": "transfer",
  "id": "t2",
  "from": "b",
  "to": "c",
  "amount": 10,
  "expected_revision": 1
}
{
  "status": "ok"
}
03
{
  "op": "inspect"
}
{
  "balances": {
    "a": 0,
    "b": 0,
    "c": 10
  },
  "revisions": {
    "a": 1,
    "b": 2,
    "c": 1
  }
}

The original response

Preserved verbatim from the subject. The source artifact also includes the prompt and complete scoring result.

Source artifact
Open raw model response
{"traces":[{"events":[{"op":"transfer","id":"r1","from":"a","to":"b","amount":3,"expected_revision":0},{"op":"restart"},{"op":"transfer","id":"r1","from":"a","to":"b","amount":3,"expected_revision":0},{"op":"transfer","id":"r1","from":"a","to":"b","amount":4,"expected_revision":0},{"op":"inspect"}],"expected":[{"status":"ok"},{"status":"restarted"},{"status":"ok"},{"status":"conflict"},{"balances":{"a":7,"b":3,"c":0},"revisions":{"a":1,"b":1,"c":0}}]},{"events":[{"op":"transfer","id":"s1","from":"a","to":"b","amount":1,"expected_revision":1},{"op":"transfer","id":"s2","from":"a","to":"c","amount":2,"expected_revision":0},{"op":"transfer","id":"s1","from":"a","to":"b","amount":1,"expected_revision":1},{"op":"inspect"}],"expected":[{"status":"stale_revision"},{"status":"ok"},{"status":"stale_revision"},{"balances":{"a":8,"b":0,"c":2},"revisions":{"a":1,"b":0,"c":1}}]},{"events":[{"op":"transfer","id":"t1","from":"a","to":"b","amount":10,"expected_revision":0},{"op":"transfer","id":"t2","from":"b","to":"c","amount":10,"expected_revision":1},{"op":"inspect"}],"expected":[{"status":"ok"},{"status":"ok"},{"balances":{"a":0,"b":0,"c":10},"revisions":{"a":1,"b":2,"c":1}}]}]}

Suite SHA-256: a56a4aadf5b20a319c7a57309608477d9e523da3d99c37a69bb6353f6b16f0c1

Read the scoring methodCompare the other attempts