Senior ML Engineer — LLM observability

Datadog is hiring a Senior Senior ML Engineer — LLM observability on the AI monitoring team (Hybrid · London, United Kingdom). This is a hands-on role in LLM & generative AI—not a generic “AI enthusiast” post.

You will own LLM features, retrieval, agents, and evaluation end to end: problem framing, offline evaluation, safe rollout, and iteration from production signals. Success is measured with clear metrics (quality, latency, cost, or business KPIs)—not slide decks alone.

Datadog helps thousands of teams monitor cloud and AI workloads. London engineers extend observability to LLM traces, eval scores, and cost attribution.

Senior ML Engineer — LLM observability: build product surfaces for prompt versioning, drift detection, and red-team signal aggregation.

Ideal if you have shipped agent or RAG systems and wished you had better production visibility on day one.

What you will do

• Design metrics and traces for LLM chains, tool calls, and retrieval hops.
• Prototype eval integrations (LLM-as-judge, golden sets) inside Datadog workflows.
• Work with enterprise customers on secure VPC deployments and ACL-aware logging.
• Contribute to open standards for AI observability where possible.

Stack & environment

• Python, OpenAI-compatible APIs, LangChain, SQL analytics.
• Kubernetes, Docker, high-cardinality time-series backends.
• Hybrid London Clerkenwell hub.

What we look for

• Production LLM or MLOps experience with measurable reliability wins.
• Strong debugging skills across distributed systems.
• Right to work in the UK.