mager-bench
← all challenges

binary-search

Binary search implementation with full docs

the prompt

$ cat binary-search.prompt
Implement `binary_search(arr: list[int], target: int) -> int` in Python. It should return the index of target in a sorted list, or -1 if not found. Write a proper docstring with Args, Returns, and Raises sections. Add inline comments explaining the algorithm logic. Include 3 test cases as examples in the docstring.

rubric

correctness

Is the binary search algorithm correct? Does it handle edge cases (empty list, target not found, duplicates)?

quality

Is the loop/recursion clean? Correct handling of integer overflow for mid calculation?

documentation

Does it have Args/Returns/Raises docstring? Are inline comments meaningful (not just restating the code)? Are the 3 examples correct?

results

Claude Sonnet 4.6

8.7
correctness 9.0quality 8.0documentation 9.030328ms

# Correct O(log n) binary search with overflow-safe mid calculation and solid edge-case handling, though the added O(n) sortedness check slightly undermines efficiency and the response ends with a truncated 'sample output' section; documentation is thorough with accurate docstring examples and meaningful comments.

inspect full trace →

GPT-OSS 120B

8.3
correctness 9.0quality 8.0documentation 8.03700ms

# Correct and clean implementation with proper mid calculation (no overflow issue in Python), but docstring lacks a duplicate-handling example as implied by rubric and adds unrequested TypeError validation not reflected in a raising example.

inspect full trace →

Claude Haiku 4.5

8.3
correctness 9.0quality 7.0documentation 9.09962ms

# Algorithm is correct with proper mid calculation and edge case handling, but the added sortedness validation (O(n log n) sorted() check) is an unrequested, questionable design choice that adds overhead and complexity not asked for in the task.

inspect full trace →

Gemini 2.5 Flash

8.3
correctness 9.0quality 8.0documentation 8.06290ms

# Algorithm is correct and handles edge cases well, but Raises section is empty (task requested a Raises section, ideally documenting behavior like non-sorted input or type errors), and comments are somewhat verbose/restating code in places.

inspect full trace →

Llama 3.3 70B

7.3
correctness 7.0quality 7.0documentation 8.01495ms

# Core binary search logic is correct, but raising ValueError for empty list deviates from the spec's expectation of returning -1, and mid-calculation doesn't address overflow-safe alternative despite rubric interest; comments are somewhat redundant but docstring and examples are otherwise complete and accurate.

inspect full trace →