Senior AI Engineer

Microsoft is a company where passionate innovators come to collaborate, envision what canbeand take their careersfurther. This is a world ofmorepossibilities, more innovation, more openness, and the skyisthelimitthinkingina cloud-enabled world.

Microsoft's Azure Data engineering team is leading the transformation of analytics in the world of data with products like databases, data integration, big data analytics, messaging & real-timeanalytics,and business intelligence.Theproductsour portfolioincludeMicrosoft Fabric, Azure SQL DB, Azure Cosmos DB, Azure PostgreSQL, Azure Data Factory, Azure Synapse Analytics, Azure Service Bus, Azure Event Grid, and Power BI. Our mission is tobuildthedata platform for the age of AI, powering a new class of data-firstapplicationsand driving a data culture.

Within Azure Data, the messaging and real-time analytics team provides comprehensive solutions and a robust platform that enables users to ingest high granularity signals (real-time & observability) and complex data, converting those into a competitive advantage in real-time for both end users and modern applications.

The Real-Time Intelligence team is hiring a Senior AI Engineer to build and scale the evals platform that underpins our real-time, agentic solutions on live operational data. You will focus on creating human and LLM-as-judge pipelines, and telemetry that measure how well our autonomous agents, anomaly detection, and decisioning systems perform in the real world. In this role,you'llown end-to-end implementation of eval pipelines and tooling, integrate them into CI/CD and production workflows, and partner closely with product and platform teams so that every AI feature is measurable,debuggable, and continuously improving. Rigorous evaluation is the most critical ingredient in building reliable agents, thisis a highly leveraged, key role that will directly shape how we design, ship, andoperatereal-time AI agentic products.

We do not just value differences or different perspectives. We seek them out and invite them in so we can tap into the collective power of everyone in the company. As a result, our customers arebetter served

- Build and evolve the Real-Time Intelligence evaluations platform: implement offline and online eval pipelines, including golden datasets, human review workflows, and LLM-as-judge / auto-raters for agents, anomaly detectors, anddecisioningsystems.
- Instrument agentic solutions for observability by wiring up telemetry, tracing, structured logging, and dashboards so quality, safety, latency, and cost are easy tomonitorand debug.
- Integrate evals into the development lifecycle by connecting pipelines to CI/CD,canaryand A/B experiments, and phased rollouts, making it simple for partner teams to run and interpret evaluations.
- Collaborate and mentor across product, research, and engineering teams, sharing best practices on eval design, LLM-as-judge usage, and Responsible AI, and providing code reviews and guidance that raise the bar for the AI features.

Embody ourcultureandvalues

- Master's Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR Bachelor's Degree in Computer Science or related technical field AND 8+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience.
- Proven ability to define architectures, make trade-offs, and lead multi-team projects from design through launch.
- Background in evaluation of ML/LLM systems: designing test sets, rubrics,humanand auto-rater evaluation pipelines.
- Excellent communication skills and the ability to explain complex trade-offs to both technical and non-technical stakeholders.