Use your open-source and code-review experience to judge repository-level software engineering tasks. You check whether each task, its reference patch and its tests hold up, and you write clear feedback against a rubric.
What you will do
- Review repository-level benchmark tasks for quality, correctness and reproducibility.
- Audit reference patches, test harnesses and Docker isolation, and check grading integrity.
- Spot answer leakage and reward hacking.
- Write clear, rubric-based feedback on each task.
What you need
- 3+ years of professional software engineering.
- Real open-source contribution or maintainer experience, such as merged PRs or committer and maintainer roles.
- Fluency in Python and at least one of Java, Go, TypeScript or C++.
Nice to have: familiarity with SWE-Bench (Verified) or similar repository benchmarks, maintainer history on major Python open-source projects such as Django, Flask, scikit-learn, sympy or pytest, and prior code-review or task-grading experience.
Apply (opens in a new tab)