The production AI checklist

Evals

If you cannot measure quality, you cannot change anything safely.

A fixed test set of real questions with expected answers, kept in version control
Evals run automatically on every prompt, model or retrieval change
A release is blocked when quality drops below an agreed score
LLM-as-judge scores are checked against human labels before you trust them
Retrieval is scored on its own: did it find the right document?

When a user says it gave a wrong answer, you need to see exactly why.

Every request is traced end to end: prompt, retrieved context, tool calls, output
Latency is tracked per step, including time to first token
Token usage and cost are recorded per request and per user or team
Errors, timeouts and refusals have alerts, not just logs
Personal data is masked before it reaches traces and logs

Providers go down and models misbehave. The product should not.

AI features that work in a demo can cost a fortune at real traffic.

An LLM that can call tools can be talked into doing the wrong thing.

Most bad answers are bad retrieval, not a bad model.

Treat prompts, models and indexes like code, because they change behaviour like code.

GPUs are the most expensive thing you run. Most teams waste a large share of what they pay for.

GPU metrics exported from every node, with dashboards for utilisation, memory and request queue depth
Alerts for GPU out-of-memory errors, thermal throttling and pods that cannot be scheduled
Autoscaling driven by queue depth or latency, not GPU utilisation alone
Cooldown periods set so replicas do not scale up and down every few minutes
A maximum replica count, so a traffic spike cannot run up an unlimited bill
GPU nodes tainted so only GPU workloads land on them
GPU sharing (time-slicing or partitioning) where workloads are small
Spot capacity for work that can be interrupted, with the interruption handling actually tested
Scale to zero during known quiet hours, and budget alerts in the cloud console
Checkpoints for long training jobs, so a lost node does not lose the run