Key Responsibilities:
AI Evaluation Framework Development:
Develop mechanisms to assess:
Response accuracy
Completeness
Relevance
Clarity
Tone
Groundedness
Policy compliance
Employee experience quality
Build configurable evaluation scorecards and quality thresholds.
Enable evaluation across multiple use cases, channels, employee groups, and support domains.
Maintain versioned evaluation datasets, benchmarks, and test scenarios.
Salesforce Application Engineering:
Design, configure, and develop Salesforce solutions supporting AI quality assurance and evaluation.
Build custom objects, Apex services, Lightning Web Components, flows, validation rules, and automation.
Develop reusable services for capturing AI prompts, responses, source references, confidence scores, and evaluation outcomes.
Extend Salesforce Service Cloud and related employee support capabilities.
Implement secure, scalable, and maintainable solutions aligned with Salesforce engineering standards.
Support sandbox, development, testing, staging, and production environments.
Response Quality Measurement:
Create automated evaluation pipelines for AI-generated and human-assisted responses.
Compare responses against approved knowledge sources, expected answers, and business policies.
Develop scoring logic for factual accuracy, relevance, completeness, and actionability.
Capture evaluator feedback and convert it into structured quality metrics.
Build workflows for sampling and reviewing high-risk or low-confidence interactions.
Enable trend analysis by use case, model version, region, policy area, and interaction type.
Hallucination Detection and Grounding Validation:
Develop mechanisms to identify unsupported, fabricated, or inconsistent AI responses.
Validate whether responses are grounded in approved enterprise knowledge sources.
Capture and assess citations, source references, retrieval results, and confidence indicators.
Flag responses that contain unverifiable claims or contradict enterprise policy.
Route suspected hallucinations for human review and remediation.
Partner with AI and data science teams to improve retrieval, prompting, and model behavior.
Regression Testing Infrastructure:
Build automated regression suites for AI-enabled Salesforce capabilities.
Maintain benchmark prompts, expected responses, edge cases, and negative test scenarios.
Compare response quality across model, prompt, knowledge base, workflow, and application releases.
Detect quality degradation before production deployment.
Integrate AI evaluation tests into CI/CD and release-management pipelines.
Establish release gates based on defined quality and compliance thresholds.
Policy and Compliance Validation:
Translate employee support policies, procedures, and regulatory requirements into executable evaluation rules.
Build automated checks for prohibited content, sensitive data handling, required disclosures, and escalation requirements.
Ensure AI responses comply with applicable HR, privacy, security, legal, and corporate policies.
Maintain audit trails for evaluations, overrides, approvals, and corrective actions.
Support compliance reviews, audits, and evidence collection.
Implement access controls and data-retention standards for evaluation data.
Human Evaluation Workflows:
Design reviewer interfaces and queues for human evaluation within Salesforce.
Enable quality analysts and subject-matter experts to score, annotate, and classify responses.
Support blind reviews, consensus scoring, adjudication, and reviewer calibration.
Create task-routing logic based on risk, business domain, language, and evaluator expertise.
Capture structured reviewer feedback for model and process improvement.
Monitor reviewer agreement and evaluation consistency.
Quality Monitoring and Analytics:
Track metrics such as:
Response accuracy rate
Hallucination rate
Policy compliance rate
Regression failure rate
Human escalation rate
Reviewer agreement
Employee satisfaction
Resolution effectiveness
Provide drill-down capabilities by model, release, use case, interaction type, and policy category.
Develop alerts for quality threshold breaches and emerging failure patterns.
Provide stakeholders with actionable insights and remediation recommendations.
Integration and Data Engineering:
Integrate Salesforce with AI platforms, large language models, knowledge systems, data warehouses, and analytics tools.
Develop secure REST, event-driven, batch, and middleware-based integrations.
Ingest conversation logs, model Clientdata, retrieval context, evaluation scores, and reviewer feedback.
Ensure data quality, lineage, traceability, and reconciliation.
Optimize data models and processing pipelines for high-volume evaluation workloads.
Protect employee and enterprise data through appropriate security and privacy controls.
Testing, Deployment and Production Support:
Develop unit, integration, regression, performance, and security tests.
Support user acceptance testing, release readiness, deployment, and hypercare.
Troubleshoot Salesforce, integration, data, workflow, and evaluation-processing issues.
Lead root-cause analysis for production incidents and quality failures.
Implement preventive controls and technical improvements.
Maintain operational documentation, runbooks, design specifications, and support procedures.
Cross-Functional Collaboration:
Partner with AI Engineering, Product Management, Data Science, Employee Experience, HR, Compliance, Legal, Security, and Quality teams.
Translate business quality expectations into technical requirements and evaluation criteria.
Participate in architecture reviews, sprint planning, backlog refinement, and release governance.
Communicate risks, quality trends, technical decisions, and remediation plans.
Provide technical leadership and knowledge transfer to engineering and support teams.
Required Qualifications:
Bachelor's degree in Computer Science, Engineering, Information Systems, or a related discipline.
5+ years of enterprise application development experience.
3+ years of hands-on Salesforce development experience.
Strong experience with:
Apex
Lightning Web Components
Salesforce Flow
SOQL
REST APIs
Salesforce security and data models
Experience building applications on Salesforce Service Cloud or Employee Service platforms.
Experience designing automated testing or quality assurance frameworks.
Strong knowledge of software engineering, integration, and CI/CD practices.
Experience working with AI, machine learning, conversational AI, or large language model applications.
Strong analytical, troubleshooting, and communication skills
Preferred Qualifications:
Experience building evaluation infrastructure for generative AI or conversational AI systems.
Knowledge of AI evaluation concepts such as groundedness, relevance, factuality, hallucination detection, and model regression.
Experience with Salesforce Einstein, Agentforce, Data Cloud, or comparable AI-enabled Salesforce capabilities.
Experience integrating Salesforce with enterprise knowledge bases and AI platforms.
Familiarity with Python, SQL, data pipelines, analytics platforms, and automated evaluation libraries.
Knowledge of prompt management, retrieval-augmented generation, model observability, or LLM operations.
Experience supporting HR, employee
The pay range that the employer in good faith reasonably expects to pay for this position is $41.61/hour - $65.01/hour. Our benefits include medical, dental, vision and retirement benefits. Applications will be accepted on an ongoing basis.
Tundra Technical Solutions is among North America’s leading providers of Staffing and Consulting Services. Our success and our clients’ success are built on a foundation of service excellence. We are an equal opportunity employer, and we do not discriminate on the basis of race, religion, color, national origin, sex, sexual orientation, age, veteran status, disability, genetic information, or other applicable legally protected characteristic. Qualified applicants with arrest or conviction records will be considered for employment in accordance with applicable law, including the Los Angeles County Fair Chance Ordinance for Employers and the California Fair Chance Act. Unincorporated LA County workers: we reasonably believe that criminal history may have a direct, adverse and negative relationship with the following job duties, potentially resulting in the withdrawal of a conditional offer of employment: client provided property, including hardware (both of which may include data) entrusted to you from theft, loss or damage; return all portable client computer hardware in your possession (including the data contained therein) upon completion of the assignment, and; maintain the confidentiality of client proprietary, confidential, or non-public information. In addition, job duties require access to secure and protected client information technology systems and related data security obligations.
