MLOps and infrastructure

Topics covered

• Docker
• Kubernetes
• CI/CD
• Cloud
• Model serving
• Monitoring
• Logging
• Observability
• Security
• Cost optimisation

Lessons in this unit

1. Serving models in production 85 min
   From a model that works in your notebook to a service that handles real traffic - validation, micro-batching, timeouts, retries with backoff, idempotency and health checks.
2. Observability, percentiles and drift 80 min
   Knowing what's happening to your model in production before a customer tells you - metrics, latency percentiles, sliding windows, SLOs and data drift detection with PSI and KS.
3. Reproducibility, tracking and CI/CD for models 65 min
   Making sure any model in production can be rebuilt, explained and replaced safely - hashes of data and configuration, experiment records, a model registry and automatic promotion rules.
4. Cost, caching, secrets and privacy 80 min
   Knowing how much each request costs before the bill arrives, making it cheaper with caching without breaking anything, and protecting keys and personal data the way a serious team would.
5. Paged KV cache — virtual memory for serving LLMs 80 min
   Why reserving the KV cache contiguously wastes most of the memory, how PagedAttention splits it into blocks with page tables, shares prefixes with copy-on-write and decides whom to evict when it runs out.
6. Quantizing for serving — what speeds up, what fits and what breaks 75 min
   4-bit weights, W8A8, FP8 and 8-bit KV cache seen from production — when they lower latency, when they raise capacity, how SmoothQuant tames activation outliers and how to decide with numbers.
7. Progressive delivery — shadow, canary and rollback 65 min
   How to take a new model to production without betting everything on all or nothing — shadow traffic, canaries with growing real traffic, when a difference in errors is real and how to automate the rollback.

Prerequisites

Before this unit it helps to have done:

• Distributed AI (Course 5 · Unit 3)