Applied GenAI Engineering

Most first LLM code fails in production for reasons that have nothing to do with AI.

• 1.1 Mental model. Tokens, windows, sampling, what the model can know. Cost and latency as functions of tokens.
• 1.2 Prompts that survive users. Structure, delimiters, few-shot, empty and adversarial input.
• 1.3 Structured output. JSON schemas, constrained generation, validation, repair loops.
• 1.4 Flaky dependency. Timeouts, backoff, rate limits, streaming, degradation, provider fallbacks.
• 1.5 Instrumentation. Prompts, responses, tokens, latency and cost per request from day one.

Ship #1: deployed LLM-backed API with schema-validated output, retries, streaming, and a live cost/latency dashboard.

Deliberately second, not last. Everything after this sprint gets measured.

• 2.1 Defining success. Testable specification, acceptance criteria, good enough to ship.
• 2.2 Golden dataset. Real inputs, labelling, edge cases, test data out of prompts.
• 2.3 Automated evaluation. Deterministic checks first, then a judge you calibrate.
• 2.4 Regression in CI. A prompt change that breaks 8% of cases fails the build.
• 2.5 Reading results honestly. Sample size, variance, why 3% on 20 examples is not an improvement.

Ship #2: eval harness on real data, gating deploys in CI, with a baseline report on Ship #1.

The unglamorous 60% is document processing. That is where most RAG projects actually fail.

• 3.1 Document processing. Multi-column PDFs, scans, tables, HTML cruft, near-duplicates.
• 3.2 Chunking with intent. Fixed, recursive, semantic, structure-aware; metadata; parent-child. Evaluated, not guessed.
• 3.3 Search. Embeddings, vector stores, BM25, hybrid fusion — including when keywords win.
• 3.4 Reranking. Cross-encoders, top-k, latency versus quality.
• 3.5 Trustworthy answers. Grounding, citations, abstention, conflicting sources.
• 3.6 Diagnosing failures. Parse, chunk, retrieve, rank, assemble, or generate — with evidence.

Ship #3: deployed RAG over a genuinely messy corpus, with citations, abstention, and retrieval metrics in the Sprint 2 harness.

Build the loop yourself before a framework hides the bug.

• 4.1 Function calling. Tool schema, validation, idempotency, recoverable errors.
• 4.2 The agent loop. Plan, act, observe, repeat — built by you first.
• 4.3 Frameworks in perspective. Where complexity earns its keep, and where it obscures the bug.
• 4.4 State and memory. History, summarisation losses, session state, what belongs in a database.
• 4.5 Bounded autonomy. Step limits, cost ceilings, timeouts, approval, kill switch.
• 4.6 When not to use an agent. Most “agentic” tasks are a fixed pipeline.

Ship #4: deployed agent on a real multi-step task with tools, guardrails, budgets and full run tracing.

Fine-tuning is a decision to justify, not a trophy model.

• 5.1 Decision framework. Prompting vs retrieval vs fine-tuning on your own failing cases.
• 5.2 Training data. Construction, formatting, deduplication, splits. Quality dominates method.
• 5.3 LoRA / QLoRA. Small open model, focused task, hyperparameters, overfitting.
• 5.4 Proving it worked. Sprint 2 baseline, regression checks, adapter behind an API.
• 5.5 Small models as a cost strategy. Routing, distillation, measured savings.

Ship #5: fine-tuned model serving behind an endpoint, with an evidence-based report — including whether it was worth it.

Cut Ship #3 cost about in half without measurable quality loss. Then operate it.

• 6.1 Serving. Containers, concurrency, streaming, pooling, health checks, load tests.
• 6.2 Cost engineering. Caching, trimming, batching, routing, per-tenant budgets.
• 6.3 Observability. Tracing, token and cost attribution, drift alerts after a provider update.
• 6.4 Safety. Direct and indirect prompt injection, PII, output filters, least-privilege tools, what never to log.
• 6.5 Operating it. Versioned prompts, staged rollout, rollback, runbooks, incident walkthrough.

Ship #6: hardened deployment with published latency and cost SLOs, tracing, alerting, and a runbook.

A real problem — ideally from your workplace — from statement to deployed system. The 20-minute defence is questions on trade-offs, failure modes and cost, not the demo.

• Written problem definition with success criteria agreed up front.
• Architecture decision record: what you chose and what you rejected.
• Retrieval and/or agents as the problem warrants.
• Evaluation against a stated baseline; deployed; cost per task; runbook; known limitations.