Principal Agentic AI Operations Engineer

Claritev is seeking a Principal Agentic AI Operations Engineer to provide technical leadership for the operationalization, evaluation, governance, and reliable production performance of agentic AI systems powering the next generation of healthcare solutions.

In this role, you will define what high-quality, trustworthy AI looks like in complex healthcare workflows and build the evaluation frameworks, observability capabilities, operational tooling, and deployment standards that ensure agents perform safely, efficiently, and consistently at scale.

Working closely with Product, Engineering, AI Science, Security, and business leaders, you will establish the operational foundation for Claritev's agentic AI platform while influencing architecture, mentoring teams, and driving measurable improvements in reliability, quality, governance, and cost efficiency.

What You'll Do

• Lead the end-to-end operational lifecycle of production agentic AI systems, including deployment, rollout strategies, monitoring, incident response, governance, and continuous improvement.

• Design and implement evaluation frameworks, benchmarking platforms, golden datasets, simulation environments, regression testing, and online evaluation methodologies.

• Define, monitor, and optimize agent quality metrics including task completion, tool-call accuracy, grounding quality, hallucination rates, latency, safety, and operational costs.

• Build observability and distributed tracing solutions across LLM interactions, retrieval workflows, tool invocations, memory operations, and orchestration layers.

• Establish LLMOps and AgentOps best practices, including CI/CD pipelines, prompt and model versioning, deployment gates, drift detection, and incident management.

• Operate and optimize AI, agentic, and RAG workloads across cloud platforms, vector databases, enterprise data systems, and containerized environments.

• Implement guardrails, auditability, human-in-the-loop controls, and policy enforcement frameworks for responsible AI operations.

• Manage AI infrastructure utilization, throughput, capacity planning, model routing, caching strategies, and cost governance.

• Ensure compliance with privacy, HIPAA, PHI/PII protection, security, access control, and enterprise governance requirements.

• Develop reusable operational frameworks, runbooks, dashboards, and evaluation harnesses that enable efficient enterprise-wide adoption of agentic AI.

• Provide technical leadership and architectural guidance across complex cross-functional initiatives.

• Mentor engineers and data scientists while fostering a culture of operational excellence, measurement-driven decision making, and continuous innovation.

What You Will Bring

Required Qualifications

• Bachelor's degree in Computer Science, Engineering, Data Science, a quantitative discipline, or a related field required.

• 10+ years of experience in software engineering, ML engineering, platform engineering, SRE, or a related technical discipline.

• 5+ years operating production AI or machine learning systems in MLOps, LLMOps, or ML platform environments.

• 3+ years of experience with Generative AI, LLMs, RAG architectures, and agentic AI systems in production.

• Experience building evaluation and benchmarking capabilities for LLM and agentic systems with measurable quality outcomes.

• Proven success leading complex technical initiatives from concept through production deployment and business impact.

• Expert-level Python skills and experience designing scalable services, APIs, and distributed systems.

• Deep expertise in AI evaluation methodologies including evaluation harnesses, golden datasets, LLM-as-judge frameworks, statistical testing, and regression detection.

• Experience with observability and evaluation platforms such as LangSmith, Langfuse, Arize Phoenix, Ragas, DeepEval, Braintrust, OpenAI Evals, or equivalent tools.

• Experience with agentic AI frameworks including LangGraph, LangChain, AutoGen, CrewAI, and related orchestration patterns.

• Experience with cloud environments, large-scale data platforms, vector databases, embeddings, and RAG systems.

• Experience implementing MLOps and LLMOps practices including CI/CD, deployment strategies, experimentation frameworks, monitoring, and troubleshooting.

• Experience with Kubernetes, containerization, Terraform, and infrastructure-as-code practices.

• Working knowledge of deep-learning frameworks such as PyTorch or TensorFlow.

• Strong leadership, communication, organizational, and problem-solving skills.

Preferred Qualifications

• Master's degree or PhD preferred.

• Experience with Oracle Cloud Infrastructure (OCI), including Generative AI, Data Science, Database, and Observability services.

• Healthcare, insurance, claims, payment integrity, healthcare technology, or other regulated-industry experience.

• Experience operating AI systems that process PHI or PII within HIPAA-regulated environments.

• Experience with AI red-teaming, adversarial testing, safety evaluation, and responsible AI governance.

• Experience driving process automation and enterprise workflow integration.

• Site Reliability Engineering experience including SLOs, SLIs, error budgets, and reliability practices for AI systems.

Compensation

The salary range for this position is $165,000 - $185,000. Actual compensation is based on experience, skills, education, work location, and internal equity. This position may also be eligible for incentive compensation, health insurance, 401(k), and bonus opportunities.