Put your code review instincts to work on real pull requests. You turn completed PRs from substantial repositories into hard engineering tasks, build the tests and grading that check them, and run coding agents against them to find where they fall short. Part-time and remote, for engineers who like digging into unfamiliar codebases.
What you will do
- Find repositories with enough real work to support challenging tasks, and pick suitable completed pull requests.
- Study each problem, its reference solution, and the repository's architecture, tests and conventions.
- Write task prompts and build evaluations from automated tests, shell commands and grading prompts.
- Check each evaluation against the reference solution and against deliberately broken versions, to catch missing checks, wrong grades and criteria that limit valid solutions.
- Run the evaluations many times across different models and harnesses, and work out whether a failure comes from the agent's solution, the environment or the grading.
- Refine the evaluations through repeated testing and review, document your findings and grading decisions, and act on feedback.
What you need
- Strong programming fundamentals and hands-on experience in large, complex codebases. Tasks cover TypeScript/JavaScript, Python, Java, Kotlin, Go, Ruby, PHP, C++ and Rust.
- The ability to read unfamiliar code, investigate subtle behavior, and judge whether two implementations solve the same problem correctly.
- Experience writing meaningful tests, including edge cases and regression coverage.
- Attention to detail and patience for repeated investigation and refinement.
- Practical experience with AI coding tools, and the judgment to verify their output and catch mistakes.
- Confidence with Git, test runners and CI tooling.
- Clear, fluent written and spoken communication. You own a task on your own and ask questions when requirements are unclear.
Experience from open-source, private or enterprise repositories all counts. Knowing a given language helps, but learning a repository and making sound engineering calls matters more than its popularity or your public contribution history.
How pay works
The listed rate is $100-120 an hour.
How you get started
You get onboarding to the evaluation framework and ongoing review feedback. After onboarding, you pick tasks, build evaluations and iterate on them without step-by-step direction.
Apply (opens in a new tab)