The Data Scientists' primary responsibility is to design, build, and maintain benchmarks for model testing and agentic workflow testing using InspectAI. They define evaluation datasets, test scenarios, metrics, scoring approaches, and repeatable evaluation workflows that measure model and agent performance in a consistent, auditable way. They also clean, transform, validate, and analyze supporting data; generate metrics and visualizations; and produce findings that support model evaluation, reporting, and decision-making. They partner with Researchers to ground benchmark design in sound methodology, and with Developers to operationalize evaluation workflows into stable, repeatable tooling. Data Scientists should also be able to document dataset lineage, scoring rationale, benchmark assumptions, and evaluation outputs so results are reproducible, traceable, and usable in constrained or airgapped environments.
Technologies / Skills Needed:
- InspectAI
- Python
- MongoDB
- Jupyter notebooks
- Data visualization tools
- Statistical analysis
- Data cleaning and validation
- Cloud or containerized environments
- Git-based version control
- Collaboration tools such as Jira or Confluence
- Experience operating in airgapped or constrained environments
** Position Requires an Active TS/SCI with a Full-Scope Polygraph Clearance**
What You’ll Love About Synergist:
- We have an actual “award-winning” culture
- We recognize that our employees are our most valuable asset, and we do everything in our power to give you the best experience every day
- We’re big believers in transparency and you’ll always have a seat at the table for big decisions and can even attend key business meetings if you want
- We offer a best-in-class benefits package that we can customize for your unique needs. Offerings include healthcare, dental, and vision coverage for you and your dependents, flexible Paid Time Off (PTO) accrual, 10% 401k employer contribution, annual education allowance, and many perks from a robust referral program, weekly catered lunches, monthly happy hours, access to company suite at Cap One Arena, financial advisory, and so much more
- We have a deep catalog (and growing!) of potential project opportunities for you to choose from, and when you’re ready for new challenges, we’ll work to make that happen
We Want to Hear From You If:
- You have a BS in Computer Science or similar technical field (4 years of relevant experience may be substituted in lieu of a BS degree)
- You have experience with AI Evaluation & Frameworks such as InspectAI, LLM Benchmarking, Agentic Workflow Evaluation (tool-use, multi-step task performance)
- You have experience with Statistical Analysis, Data Cleaning & Validation, Data Visualization Tools, Jupyter NotebooksData & Environment: MongoDB, Cloud/Containerized Environments, Airgapped & Constrained Operations
- You're familiar with Git Version Control, Experimental Design, Dataset Lineage/Traceability, Jira, and Confluence
Here’s What You Can Expect Out Of This Role:
- AI Model Benchmarks — Design, implement, and maintain benchmark suites that measure model performance against defined evaluation criteria, including dataset curation, scoring, and results reporting.
- Agentic AI Benchmarks — Develop benchmarks and test scenarios that evaluate agentic workflows and multi-step task performance, including tool-use, task completion, and reliability metrics.
- Evaluation Dataset Development — Curate, validate, and version evaluation datasets and test scenarios, including any adversarial or edge-case sets needed to stress agent behavior.
- Metrics, Analysis, and Reporting — Produce scoring frameworks, visualizations, and written analyses that translate raw evaluation results into actionable findings for stakeholders.
Job Type: Full-time
Pay: $100,000.00 - $280,000.00 per year
Benefits:
- 401(k)
- Dental insurance
- Employee assistance program
- Flexible schedule
- Flexible spending account
- Health insurance
- Health savings account
- Life insurance
- Paid time off
- Parental leave
- Professional development assistance
- Referral program
- Tuition reimbursement
- Vision insurance
Security clearance:
Work Location: In person
