Bring your data science and analytic methodology experience to a remote contract on LLMs inside real analytic pipelines. You map scaled-analysis scenarios, write realistic analytic cases including failure modes, and grade responses on reasoning, source handling and practical use. No prior AI experience is needed: your domain knowledge is what counts.
What you will do
- Map scaled-analysis and pipeline-integration scenarios to an established taxonomy.
- Write detailed analytic cases based on real deployment patterns, such as automated triage, enrichment and analysis at scale, plus typical real-world failure modes.
- Grade responses on source handling, reasoning, understanding of limitations and practical use.
- Flag cases where a model refuses too readily and would block legitimate analytic work or break a pipeline.
- Document hard edge cases and methodology concerns, with practical recommendations.
- Give clear written and verbal feedback.
What helps
- Experience integrating LLMs, such as government GPT variants, into complex data or analytic pipelines.
- Proven analysis at scale, with an understanding of automation failure modes and the right evaluation metrics.
- Teaching, mentoring or technical leadership in data science or a related analytic field.
- Clear technical writing and strong written English.
- Exposure to designing evaluation frameworks or benchmarks.
- Familiarity with defense, intelligence or government analytic work is highly valued.
- Ability to work well in distributed, international teams.
How pay works
This job is paid hourly, at the rate listed above.
Apply (opens in a new tab)