Top 100 Quality rater Jobs in California

Assessment Designer & Learning Analyst
Mercor —San Francisco, CA
- $120,000 - $160,000 a year
- Full-time
- Writing skills
- Mid-level
- Office
9d
Senior HERS Rater / Commissioning Agent
VCA Green —Orange County, CA
- $80,000 - $95,000 a year
- Full-time
- Background check
- Clean driving record
- Construction site
- Paid holidays
- Health insurance
Quick Apply
Medical Director -- Part Time Contracted
Ventura County Health Care Plan —Oxnard, CA
- $156,000 - $195,000 a year
- Part-time
- Medicine
- BC/BE
- 3 years
Quick Apply
Appeals and Grievances Specialist II
L.A. Care Health Plan —Los Angeles, CA
- $60,778 - $91,166 a year
- Full-time
- Hybrid work
- Writing skills
- Mid-level
- Office
- Wellness program
- Health insurance
Clinical Data Review Coordinator
UCLA Health —Santa Monica, CA
- $95,400 - $208,300 a year
- Nursing
- RN License
- Mid-level
Clinical Quality Reviewer - BCBA
Onos Health —San Francisco, CA
- $110,000 - $140,000 a year
- Full-time
- Hybrid work
- Mid-level
- Office
- Analysis skills
- Commuter assistance
- Paid parental leave
WORKERS' COMPENSATION CONSULTANT
Department of Industrial Relations —Riverside County, CA
- $6,646 - $8,324 a month
- 8 hour shift
- Hybrid work
- Microsoft Excel
- Microsoft Outlook
- Writing skills
- Paid holidays
- Health insurance
6d
Office Specialist (Full-time & Part-time)
City of Ontario, CA —Ontario, CA
- $4,969.47 - $6,037.20 a month
- Temporary
- Pass drug screening
- No experience needed
- Office
- Retirement plan
6d
Homeless Services Manager
City of Ontario, CA —Ontario, CA
- $12,486.94 - $15,178.80 a month
- Online Application
- Spanish
- Clean driving record
- No experience needed
MSO VBC SPECIALIST I
NORTH EAST MEDICAL SERVICES —Burlingame, CA
- $36.92 - $41.85 an hour
- Full-time
- Microsoft Outlook
- Bachelor of Science
- Mid-level
- Health insurance
- Dental insurance
Technical Training and Development Manager
Precision Castparts Corp. —Santa Ana, CA
- $142,000 - $160,000 a year
- Full-time
- Data analysis skills
- Analysis skills
- Mentoring
- Paid holidays
- Health insurance
PI/Peer Review Coordinator - Performance Improvement - Full time 10 hours Days (Non-Union, Exempt)
USC —Arcadia, CA
- $48.85 - $73.27 an hour
- Full-time
- Nursing
- RN License
- Hospital
ML Evaluation Specialist, Human Data
Apple —Cupertino, CA
- $144,600 - $263,800 a year
- Master's degree
- Bachelor's degree
- Organizational skills
- Employee stock purchase plan
- Health insurance
6d
Custodian
City of Simi Valley —Simi Valley, CA
- $20.75 - $26.47 an hour
- Overtime
- Customer service
- Pass drug screening
- Driver's License
- Paid holidays
- Health insurance
1d
RN Care Coordination
St Mary Medical Center —Long Beach, CA
- $58.70 - $73.30 an hour
- PRN
- Acute care
- Nursing
- RN License
2d
PI/Peer Review Coordinator - Performance Improvement - Full time 10 hours Days (Non-Union, Exempt)
Keck Medical Center of USC —Arcadia, CA
- $48.85 - $73.27 an hour
- Full-time
- Background check
- Nursing
- RN License
Clinical Rater | Psychiatric Assessment | Diagnostician | Part-time PRN
Apex Clinical Research —San Diego, CA
- $80 - $150 an hour
- Part-time
- Nursing
- Medicine
- Psychiatry
Quick Apply
8d
Human Resource Administrator
Stanford University —Stanford, CA
- $32.21 - $38.64 an hour
- Full-time
- Hybrid work
- Mid-level
- Office
- University
1d
Care Coordinator RN
Marian Regional Medical Center —Santa Maria, CA
- $59.31 - $88.50 an hour
- PRN
- Acute care
- Nursing
- RN License
12d
PI/Peer Review Coordinator - Performance Improvement - Full time 10 hours Days (Non-Union, Exempt)
USC Arcadia Hospital —Arcadia, CA
- $48.85 - $73.27 an hour
- Full-time
- Background check
- Nursing
- RN License

I want to receive the latest job alert for quality rater in california

By signing in to your account, you agree to SimplyHired's Terms of Service and consent to our Cookie and Privacy Policy.

Explore jobs in more locations

quality rater jobs near california

Assessment Designer & Learning Analyst

Mercor -
San Francisco, CA

Apply Now

Job Details

Full-time
$120,000 - $160,000 a year
9 days ago

Qualifications

Data preprocessing
Quantitative applied research
Master's degree
Regression analysis
Computer skills

Full Job Description

About Mercor

Mercor's mission is to organize human intelligence to power the AI economy. We partner with leading AI labs and enterprises to provide the human intelligence essential to AI development. Our vast talent network trains frontier AI models in the same way teachers teach students: by sharing knowledge, experience, and context that can't be captured in code alone. Today, more than 30,000 experts in our network collectively earn over $3 million a day.

Mercor is creating a new category of work where expertise powers AI advancement. Achieving this requires an ambitious, fast-paced and deeply committed team. You’ll work alongside researchers, operators, and AI companies at the forefront of shaping the systems that are redefining society. Mercor is a profitable Series C company valued at $10 billion. We work in-person five days a week in our San Francisco, NYC, or London offices.

We're looking for an Assessment Designer & Learning Analyst who can build rigorous measurement systems and use data to understand what actually drives expert performance.

This is not an instructional design role. You won't be building courses or writing training materials. You will be designing the assessments and certification frameworks that measure whether our talent experts and internal teams are genuinely skilled — and then doing the analytical work to understand what those assessments reveal, what predicts expert effectiveness, and how our programs should evolve based on evidence. You will be working closely with the Learning & Development team to understand the relationship between materials and assessments, and making recommendations to the team based on your analysis.

If you've come from an ed school background, taught in a high-accountability environment, and completed quantitative projects or theses, and are energized by the measurement and data side of education — this role is for you.

What You'll Do

Assessment Design

Design and continuously improve assessments and certification frameworks that validly and reliably measure expert readiness for specific project types
Build assessments and measurements of skills that are consistent, interpretable, and actually predictive of on-the-job performance — not just checklists.
Develop item banks, scoring guides, and inter-rater reliability protocols for evaluating complex human judgment tasks.
Run validity studies: do our assessments measure what we think they measure?

Learning Analytics & Impact Analysis

Analyze the relationship between instructional materials, assessments, and expert performance — identifying what's working and what isn't and make recommendations accordingly.
Analyze assessment data at the item level — difficulty, discrimination, reliability — and iterate based on findings.
Investigate the relationship between assessment performance and real-world expert effectiveness: who performs well on our assessments, and does that predict quality outcomes?
Build reports and dashboards that surface actionable insights to program and operations teams.
Design and analyze quasi-experimental, quantitative and qualitative (mixed methods) studies to understand what interventions actually move the needle on expert quality.

Ongoing Measurement & Improvement

Track certification and assessment outcomes over time and flag when programs need revision
Partner with learning designers and project teams to translate your findings into program improvements
Bring a continuous improvement mindset — ship, measure, learn, iterate

What We're Looking For

Education

Master's degree in Learning Sciences, Educational Psychology, Educational Measurement, Psychometrics, or a closely related field — required
Coursework in quantitative research methods, psychometrics, and educational statistics — required
Familiarity with classical test theory (CTT) and ideally item response theory (IRT)

Quantitative Skills — Required This role requires genuine comfort with numbers. We're looking for someone who can do the following and show their work:

Item-level analysis: difficulty index, discrimination index, inter-rater reliability (Cohen's kappa, Krippendorff's alpha, ICC)
Assess and report on assessment validity and reliability — and know what to do when results look off
Analyze relationships between variables: correlation, regression, and basic predictive modeling
Work fluently in Excel or Google Sheets for data cleaning and summaries
Use Python, STATA or R for deeper analysis (basic proficiency expected; we'll grow this with you)
Translate quantitative findings into plain-language recommendations for non-technical stakeholders

We will ask you to demonstrate this. Finalists will complete a short take-home exercise involving a real assessment dataset — you'll analyze item performance, identify problems, and recommend improvements.

Experience

1–2 years of experience in assessment design, educational research, learning analytics, or a related role
Teaching or similar experience in a high-accountability environment (Teach For America, urban education, or similar) is a strong plus; people who've lived with assessment data in the classroom understand it differently
Experience designing assessments with a clear theory of what you're measuring — not just writing questions
A portfolio or work samples showing both assessment design and quantitative analysis — we want to see how you think

Skills

Deep understanding of measurement: validity, reliability, and what makes an assessment actually good
Ability to move between data and meaning — you can run the analysis and explain what it means for the program
Strong writing — you can communicate complex findings clearly to non-technical audiences
Systems thinker — you see how individual assessments connect to broader operational quality and expert performance
Comfortable with ambiguity and rapid iteration — this is a fast-moving environment and you'll need to ship and improve continuously

Nice to Have

Experience with item response theory (IRT) or latent variable modeling
Familiarity with data annotation, labeling, or AI evaluation workflows
Experience in tech, AI/ML, or data operations environments
Background in competency-based or mastery learning frameworks
Experience building and analyzing assessments

Why This Role

The quality of AI systems depends on the quality of the humans who train them. Your job is to measure that quality rigorously, understand what drives it, and help Mercor build smarter systems for developing expert performance. It's a rare opportunity to apply serious measurement science at a company operating at the frontier of AI development — where the stakes for getting it right are unusually high.

Compensation Range: $120K - $160K

Apply Now

Refine Your Search

quality rater jobs in california

Assessment Designer & Learning Analyst

Senior HERS Rater / Commissioning Agent

Medical Director -- Part Time Contracted

Appeals and Grievances Specialist II

Clinical Data Review Coordinator

Clinical Quality Reviewer - BCBA

WORKERS' COMPENSATION CONSULTANT

Office Specialist (Full-time & Part-time)

Homeless Services Manager

MSO VBC SPECIALIST I

Technical Training and Development Manager

PI/Peer Review Coordinator - Performance Improvement - Full time 10 hours Days (Non-Union, Exempt)

ML Evaluation Specialist, Human Data

Custodian

RN Care Coordination

PI/Peer Review Coordinator - Performance Improvement - Full time 10 hours Days (Non-Union, Exempt)

Clinical Rater | Psychiatric Assessment | Diagnostician | Part-time PRN

Human Resource Administrator

Care Coordinator RN

PI/Peer Review Coordinator - Performance Improvement - Full time 10 hours Days (Non-Union, Exempt)

I want to receive the latest job alert for quality rater in california

Related Searches

Explore jobs in more locations

About Mercor

What You'll Do

What We're Looking For

Nice to Have

Why This Role

Job Seeker Tools

Employer Tools

Browse

Stay Connected