mager-bench
← all challenges

binary-search

Binary search implementation with full docs

the prompt

$ cat binary-search.prompt
Implement `binary_search(arr: list[int], target: int) -> int` in Python. It should return the index of target in a sorted list, or -1 if not found. Write a proper docstring with Args, Returns, and Raises sections. Add inline comments explaining the algorithm logic. Include 3 test cases as examples in the docstring.

rubric

correctness

Is the binary search algorithm correct? Does it handle edge cases (empty list, target not found, duplicates)?

quality

Is the loop/recursion clean? Correct handling of integer overflow for mid calculation?

documentation

Does it have Args/Returns/Raises docstring? Are inline comments meaningful (not just restating the code)? Are the 3 examples correct?

results

Claude Haiku 4.5

9.0
correctness 9.0quality 9.0documentation 9.011549ms

# Correct, overflow-safe binary search with clean logic, thorough docstring/comments, and good extra test coverage beyond the required 3 examples, though duplicates aren't explicitly tested.

inspect full trace →

Claude Sonnet 4.6

8.7
correctness 9.0quality 8.0documentation 9.030328ms

# Correct O(log n) binary search with overflow-safe mid calculation and solid edge-case handling, though the added O(n) sortedness check slightly undermines efficiency and the response ends with a truncated 'sample output' section; documentation is thorough with accurate docstring examples and meaningful comments.

inspect full trace →

Gemini 2.5 Flash

8.3
correctness 9.0quality 8.0documentation 8.06290ms

# Algorithm is correct and handles edge cases well, but Raises section is empty (task requested a Raises section, ideally documenting behavior like non-sorted input or type errors), and comments are somewhat verbose/restating code in places.

inspect full trace →

Llama 3.3 70B

7.3
correctness 7.0quality 7.0documentation 8.01495ms

# Core binary search logic is correct, but raising ValueError for empty list deviates from the spec's expectation of returning -1, and mid-calculation doesn't address overflow-safe alternative despite rubric interest; comments are somewhat redundant but docstring and examples are otherwise complete and accurate.

inspect full trace →