Top 48 Junior python developer Jobs in Massachusetts

Senior QA Engineer
Third Way Health, Inc. —Cambridge, MA
- Full-time
- 5 years
- Data analysis skills
- System design
Quick Apply
Senior Software Engineer, Device Control
SimpliSafe —Boston, MA
- $117,300 - $156,400 a year
- Hybrid work
- Azure
- System design
- Office
Quick Apply

More jobs to discover

Find similar jobs that you might be interested in.

Building Automation Systems (BAS) Technician
Confidential —Salisbury, MA
Execute scheduled preventive maintenance tasks for all BMS field devices, HVAC equipment, and mechanical systems. 3–5 years of BAS/BMS experience.
$100,000 - $130,000 a year
Quick Apply
Maintenance Mechanic - 3rd Shift
Cedars Mediterranean Foods —Haverhill, MA
Maintain, diagnose, and repair electrical, hydraulic, steam, water, waste and pneumatic systems/components found in food manufacturing equipment related to all…
$20 - $40 an hour
Quick Apply

Show more similar jobs

Explore jobs in cities nearby

Check out jobs in cities near your current location, Columbus

Junior Embedded Systems Developer
Battelle —Columbus, OH
Bachelor’s degree in Computer Engineering, Electrical Engineering, Computer Science, or related technical field; or equivalent experience.

I want to receive the latest job alert for junior python developer in massachusetts

By signing in to your account, you agree to SimplyHired's Terms of Service and consent to our Cookie and Privacy Policy.

Explore jobs in more locations

junior python developer jobs near massachusetts

Senior Software Engineer jobs in Boston, MA

Senior Quality Assurance Engineer jobs in Logan Circle, DC

Quality Assurance jobs in Philadelphia, PA

Senior Quality Assurance Engineer jobs in Albany, NY

Senior Quality Assurance Engineer jobs in Milford, MA

Remote Rn Quality Assurance jobs in New York State

Senior Quality Assurance Engineer jobs in Cary, NC

Senior Software Engineer jobs in East Palo Alto, CA

Part Time Senior jobs in Rhode Island

Senior Software Engineer jobs in Poland, IN

Software Engineer jobs in Mountain View, CA

Senior Software Engineer jobs in Texas

Senior Software Engineer jobs in Georgetown, TX

Senior QA Engineer

Third Way Health, Inc.
Cambridge, MA

Quick Apply

Job Details

Full-time

Qualifications

AI models
AI chatbots
Software engineering
Quality control product testing
IT system monitoring
System design
Quality assurance experience within IT
System design for system development
Collaboration with product development teams
Production validation processes
APIs
Production monitoring
System development
Test validation method
Model evaluation
Python
MLOps
Cross-functional communication
System performance monitoring

Full Job Description

Who we are:

Third Way Health (www.thirdway.health) helps medical practices and healthcare organizations across the United States to improve the patient experience while reducing the administrative burden on healthcare staff. We enable healthcare organizations to enhance the experience of their patients by providing them with a leading technology platform and world class services. What unites us is our passion to support physicians and help patients from all backgrounds to have a better healthcare experience.

About the position

We’re seeking a Senior QA / Eval Engineer to own and evolve the quality and evaluation infrastructure behind our AI-powered patient engagement platform. Our eval system is multi-factorial (deterministic, LLM-based, human expert), running against every voice interaction we handle. You’ll be responsible for the verification and evaluation infrastructure that determine whether each interaction meets multi-faceted quality criteria.

This is a high-impact individual contributor role. You won’t just write test cases — you’ll shape how we define “quality” across our automated and manual workflows and build the tooling that makes that definition measurable and actionable.

Responsibilities

Own and extend our multi-layered eval pipeline and verification portfolio: deterministic quality checks on tool calls, risk-factor heuristics, and LLM-graded transcript evaluation.
Advance our capabilities to evaluate end-to-end system performance (across orchestrated agents, RAG-supported responses, multi-party voice conversations) with modular and auditable verification that is independent of any single model provider.
Drive improvements to our observability stack to surface eval metrics, detect regressions, and enable data-driven quality decisions across the team.
Build real-time monitoring and verification loops that catch issues in production interactions as they happen, intervening with context and feeding back for system refinement.
Partner with ML engineers, product managers, and operations leads to translate real-world failure modes into automated checks, closing the loop between production incidents and eval coverage.
Build and maintain adversarial and edge-case test suites — including prompt injection resistance, guardrail robustness, and graceful degradation under ambiguous patient inputs.
Champion “shift-left” quality practices: embed eval criteria into prompt engineering workflows, define acceptance criteria for new agent behaviors, and make quality a first-class concern in the development cycle.
Contribute to the design of our QA pipeline orchestration (background processing, Slack notifications, risk assessment persistence) to improve throughput, reliability, and developer experience.

Required skills and qualifications

5+ years of software engineering or test engineering experience, with 3+ years focused on quality infrastructure for AI/ML or data-intensive systems.
Strong proficiency in Python, particularly for building test frameworks, eval pipelines, and API-level integration tests (e.g., pytest, FastAPI TestClient, Pydantic).
Demonstrated experience designing evaluation or verification systems for LLM-based applications — with a clear understanding that the model is a generation layer, not the quality layer. Comfort with both deterministic and model-graded assessment approaches, and a point of view on when each is appropriate.
Familiarity with the architectural tradeoffs of relying on LLM outputs in production — including variance across model versions, prompt sensitivity, and the need for external verification infrastructure that remains stable as underlying models change.
Experience building extensible, rule-based validation systems (check registries, plugin architectures, or similar patterns) that scale across a growing surface area of features.
Solid understanding of voice AI or conversational AI systems, including tool-calling patterns, transcript analysis, and interaction-level quality metrics.
Hands-on experience with observability and metrics instrumentation in production environments.
Excellent communication skills, with the ability to collaborate effectively across engineering, product, and non-technical stakeholders.
Strong interest in healthcare innovation and building AI systems that meaningfully improve health outcomes.

Desired skills and qualifications

Experience building QA or eval systems in healthcare or regulated environments, with familiarity with standards such as HIPAA, GDPR, or FDA guidance.
Proven experience leading complex technical initiatives and mentoring junior engineers.
Experience building or operating systems where quality guarantees live in the verification infrastructure rather than in any single model.
Familiarity with risk-scoring systems, anomaly detection, or production safety nets for autonomous AI agents.
Experience with AI safety testing, including adversarial evaluation, jailbreak testing, and bias detection in LLM outputs.
Hands-on experience with CI/CD pipelines for eval automation (CircleCI, GitHub Actions, or equivalent) and infrastructure-as-code deployment patterns.
Experience with voice UI testing tools and platforms, with a focus on evaluating speech generation and response quality.
Knowledge of accessibility testing and inclusive design principles.

At this time, we are unable to provide employment visa sponsorship. Candidates must have authorization to work in the United States without requiring sponsorship now or in the future.

Quick Apply

Refine Your Search

junior python developer jobs in massachusetts

Senior QA Engineer

Senior Software Engineer, Device Control

More jobs to discover

Explore jobs in cities nearby

I want to receive the latest job alert for junior python developer in massachusetts

Related Searches

Explore jobs in more locations

Job Seeker Tools

Employer Tools

Browse

Stay Connected