Contract LLM Safety & Eval Engineer (90 days, renewable)

Helmline AI Safety is hiring a Senior Contract LLM Safety & Eval Engineer (90 days, renewable) on the Trust & safety team (Remote · Canada). This is a hands-on role in LLM & generative AI—not a generic “AI enthusiast” post.

You will own LLM features, retrieval, agents, and evaluation end to end: problem framing, offline evaluation, safe rollout, and iteration from production signals. Success is measured with clear metrics (quality, latency, cost, or business KPIs)—not slide decks alone.

Helmline — Build red-team suites, human preference pipelines, and regression gates for frontier-class chat products.

Remote; overlap with PT / ET.

Helmline AI Safety needs someone who can move LLM features, retrieval, agents, and evaluation from experiment to production—with tests, monitoring, and stakeholder-ready communication.

The Trust & safety group partners closely with product and domain experts; you will translate ambiguous requirements into measurable ML or LLM outcomes.

What you will do

• Design and ship LLM features, retrieval, agents, and evaluation with offline evals, dashboards, and rollback plans.
• Improve datasets, labeling, and metrics that correlate with user or business outcomes.
• Review model changes for safety, bias, latency, and infrastructure cost before wide release.
• Document decisions so on-call, support, and compliance teams can act without guesswork.
• Collaborate async with written specs, demos, and recorded design reviews.

What we look for

• Demonstrated production use of core skills such as Python, LangChain, OpenAI API, ChatGPT—with repos, writeups, or metrics we can verify.
• Comfort operating the listed toolchain (Automation, Notion) in real environments—not tutorials only.
• Ownership of at least one feature or model from design through post-launch review—including failure modes you handled.
• Clear communication on trade-offs (quality, cost, latency, safety) with product and non-technical stakeholders.
• Eligible to work in Canada (or remote policy as stated in the listing).

Benefits & ways of working

• Paid breaks between contract renewals
• async docs culture.