CI/CD for AI

LLM apps don't fit traditional CI. They have non-deterministic outputs, models that change under you, and prompts that need versioning.

Test pyramid for LLM apps

• Unit tests — pure functions: parsers, retrievers (without LLM), tool implementations.
• Mocked LLM tests — exercise control flow with stubbed LLM responses.
• Eval-based tests — run prompts against eval set; pass if metrics ≥ threshold.
• Integration tests — full pipeline against a small set with real LLM, recorded outputs (or live).
• Smoke tests — production deploy gates: 20 quick eval queries that must pass.

Most teams over-rely on integration tests because mocking is annoying. Invest in mocks. Recorded LLM responses (vcr.py, MSW) make CI deterministic.

Prompt versioning

• Treat prompts as code. Store in repo, not in DB.
• Frameworks: LangSmith Hub, Promptfoo, custom YAML files.
• Each prompt gets: version, model, params, eval scores, change log.
• A/B between versions in production via feature flags.

Model upgrade strategy

• Pin model versions (gpt-4o-2024-11-20, not gpt-4o).
• When a new version drops:
  1. Run full eval set on new vs old.
  2. Check cost / latency deltas.
  3. Canary deploy to 5% of traffic.
  4. Compare online metrics 24-72h.
  5. Roll forward or back.

Deployment patterns

• Feature flags for prompts/models/configs. Atomic rollback.
• Canary for new models / major prompt changes.
• Shadow mode for high-stakes — run new path in parallel, log diff, don't serve to user.
• Blue/green for whole-stack changes.

What goes in CI

• Lint prompts (token count, schema validity).
• Run offline eval on changed prompts; fail if regression > threshold.
• Run integration tests on critical paths.
• Snapshot tool schemas; alert on breaking changes.

Failure modes

• Silent prompt drift. Someone tweaks a prompt; eval doesn't catch it because it's not in CI; quality degrades over weeks.
• Model deprecation surprise. Provider deprecates a model with 90-day notice; you didn't pin → outage.
• Eval set rot. Set never updated; production drift hides quality issues.
• No rollback path. Prompts in DB, not git → can't roll back atomically with code.

A simple, durable workflow

1. Prompts in git (prompts/ dir).
2. Eval set in git, versioned.
3. CI runs eval on every prompt change.
4. Deploys are atomic: code + prompts + eval results.
5. Production carries a feature-flag layer for emergency rollback.