1 per engagement

Agentic AI

LLM Evaluation Engineer

Seniority band: 5–9 years1 per engagement

Builds the measurement layer most teams skip: golden sets, regression suites and human-review loops that make model quality a number instead of an opinion.

Discuss this roleAll roles

What this role owns

  • Golden datasets built from real user traffic, not synthetic guesses
  • Regression suites wired into CI so no model or prompt change ships blind
  • Human-review workflows and inter-rater agreement
  • Red-team exercises against anything customer-facing

Tooling and capability

Agentic AI

Eval harnessesLLM-as-judgeRed-teamingHallucination detectionFine-tuning

Software

PythonPyTorchpytestPolarsDuckDB

Platform

Weights & BiasesMLflowGCPDocker

Other roles

Senior Agentic AI Engineer8–12 yearsAI Solutions Architect10+ yearsAI Automation Analyst3–6 years