Job description
Job Summary
InnoVet Health is a growing small business providing Health IT professional services to the Department of Veterans Affairs (VA). We are seeking an experienced Quality Assurance and User Acceptance Testing Specialist to validate AI and machine learning solutions as they move from controlled testing into operational pilots across VA.
This role goes beyond traditional software testing. AI systems do not produce a single expected output, so validation depends on acceptance thresholds, curated evaluation datasets, and testing for behavior rather than exact results. You will design and execute that testing, coordinate User Acceptance Testing with clinical and operational stakeholders, and determine whether AI outputs are accurate, interpretable, and actually usable inside real VA workflows. We are looking for strong QA and UAT fundamentals plus the analytical curiosity to work in a space where pass and fail are matters of degree. Direct AI testing experience is welcome but not required.
This is a full‑time, 100% remote, salaried position and does not require relocation. InnoVet Health offers competitive pay and benefits, as well as opportunities for professional growth.
ResponsibilitiesTest Strategy & Evaluation Design
- Develop test plans, scenarios, cases, and scripts for AI and machine learning solutions, including acceptance thresholds and tolerances where system outputs are probabilistic rather than deterministic.
- Build and maintain curated evaluation datasets and reference sets used to validate model behavior consistently across releases.
- Define regression suites of high‑consequence cases that must not degrade between model versions, recognizing that aggregate accuracy can improve while specific cases or subgroups get worse.
- Partner with the AI Architect and data science team to translate evaluation strategy into executable test coverage.
Test Execution & Defect Management
- Execute functional, regression, smoke, integration, and end‑to‑end testing across AI‑enabled applications and the data pipelines that feed them.
- Test non‑deterministic behavior using repeated‑run and semantic equivalence approaches rather than exact‑match comparison.
- Conduct adversarial and robustness testing of LLM‑based features, including groundedness and hallucination checks, prompt injection attempts, refusal behavior, and unintended disclosure of sensitive information in model output.
- Execute subgroup performance testing to confirm results hold across relevant demographic and clinical strata.
- Document, prioritize, and track defects through resolution using Jira or equivalent, applying severity classification that accounts for clinical and operational risk rather than functional impact alone.
- Verify defect fixes and perform regression testing to ensure no unintended impacts.
User Acceptance Testing & Workflow Validation
- Coordinate and facilitate User Acceptance Testing with VA clinical and operational stakeholders, including scheduling, test execution, issue tracking, and stakeholder communications.
- Validate that AI outputs are interpretable and actionable in real workflows, including how results are presented, whether confidence and limitations are clear, and whether users can override or escalate.
- Support shadow‑mode and silent‑running validation, comparing system output against current practice before users are exposed.
- Surface workflow burden and adoption barriers identified during UAT, and feed findings back to product and architecture teams.
Test Data, Evidence & Compliance
- Define and manage the test data approach within PHI constraints, including synthetic data, de‑identified extracts, and validation that test environments handle sensitive data appropriately.
- Maintain testing documentation, test evidence, traceability matrices, and execution reports suitable for audit, security authorization, and model documentation.
- Perform Section 508 accessibility testing of user‑facing components.
Release & Post‑Deployment Validation
- Validate enhancements, model updates, defect fixes, and configuration changes prior to production deployment.
- Verify that monitoring, drift detection, and alerting behave as designed, and validate rollback procedures.
- Support post‑deployment validation, production verification, and multiple implementation waves across concurrent pilots.
Agile Delivery & Stakeholder Communication
- Participate in Agile ceremonies including sprint planning, backlog refinement, daily standups, sprint reviews, and retrospectives.
- Identify risks, issues, and lessons learned, and provide regular status updates to project leadership and VA stakeholders.
QualificationsRequired
- Bachelor’s degree in Computer Science, Information Systems, Business, Engineering, or another technical discipline, or equivalent experience.
- Minimum of 5 years of experience in software quality assurance and user acceptance testing.
- Experience developing test plans, test cases, test scripts, and test execution documentation.
- Experience coordinating and facilitating User Acceptance Testing with business stakeholders.
- Strong understanding of Agile software development methodologies.
- Experience with defect lifecycle management and issue tracking.
- Experience using Jira, Azure DevOps, Rally, ALM, or similar application lifecycle management tools.
- Working proficiency in SQL and comfort inspecting datasets directly to verify system behavior.
- Analytical rigor sufficient to reason about tolerances, thresholds, and acceptance criteria rather than binary pass or fail outcomes.
- Ability to interpret business requirements and translate them into effective test scenarios.
- Excellent analytical, organizational, problem‑solving, communication, and documentation skills.
- Proficient with Microsoft Office Suite, including Excel, Word, PowerPoint, and Teams.
Preferred
- Experience testing AI, machine learning, or LLM‑based systems, including evaluation datasets, model regression testing, or adversarial testing.
- Experience supporting the Department of Veterans Affairs or other federal healthcare organizations.
- Experience testing cloud‑based applications in Azure Government, AWS GovCloud, or VAEC.
- Familiarity with Python or notebook‑based validation workflows.
- Familiarity with Databricks, Spark, or similar analytics platforms.
- Experience with test data management under PHI or other sensitive data constraints.
- Familiarity with interface validation and data verification.
- Experience working within SAFe or Agile Scrum environments.
- ISTQB, CSTE, or other QA certification.
- Healthcare IT, government, or enterprise software implementation experience.
Additional Requirements
- U.S. Citizenship required.
- Ability to obtain and maintain VA suitability, a federal PIV badge, and any required federal security clearances.
- No visa sponsorship is available.
- No 1099, Corp‑to‑Corp, international outsourcing, or staffing agencies.
Job Type: Full-time
Pay: From $105,000.00 per year
Benefits:
- 401(k)
- 401(k) matching
- Dental insurance
- Health insurance
- Paid time off
- Vision insurance
Application Question(s):
- This position works directly with US government contracts and under Order 11935, it requires either U.S. Citizenship or valid Green Card. Please answer 2 if you are a US citizen, 1 if you have a valid
- Please include your LinkedIn URL
- How many years of experience do you have with testing AI, machine learning or LLM based system - including evaluation datasets, model regression testing or adversarial testing?
Experience:
- federal consulting: 3 years (Preferred)
- developing test plans and test cases: 3 years (Preferred)
- agile software development methodologies: 3 years (Preferred)
- Software quality assurance: 5 years (Required)
- User acceptance testing: 5 years (Required)
Work Location: Remote
