Part I. ML Foundations

0/10

001 The mathematical objects: tensors, shapes, broadcasting 002 The forward pass: a neural network as a pure function 003 The backward pass: autograd, gradients, the chain rule made mechanical 004 Loss functions and optimization, in just enough depth 005 Tokens, vocabularies, and the tokenizer is the bug 006 Attention from first principles 007 The transformer end to end 008 The decoding loop: autoregressive generation, sampling, controllability 009 Embeddings and rerankers: what they are, why they're separate, why they're cheap 010 The model card: reading model lineage adversarially

Part II. Training, Fine-Tuning, Alignment

0/10

011 Pretraining at scale: data, compute, curriculum 012 Distributed training: DDP, FSDP, ZeRO, tensor/pipeline/sequence parallel 013 Mixed precision training: fp16, bf16, fp8, loss scaling 014 Tokenizer training: BPE and SentencePiece from scratch 015 Fine-tuning: full, LoRA, QLoRA, adapters, the PEFT family 016 SFT and instruction tuning: turning a base model into an assistant 017 RLHF, DPO, KTO, Constitutional AI 018 Distillation, pruning, and training-time compression 019 Synthetic data and self-improvement loops 020 Evaluation: the hardest unsolved problem in ML

Part III. Inference Internals & Production Serving

0/36

021 Prefill vs decode: the two-phase nature of LLM inference 022 The KV cache: the single most important optimization in LLM inference 023 Batching: static, dynamic, continuous (Orca) 024 PagedAttention and vLLM as a virtual-memory system for KV cache 025 FlashAttention and the GPU memory hierarchy 026 Quantization: INT8, INT4, FP8, AWQ, GPTQ, SmoothQuant 027 Speculative decoding: Medusa, EAGLE, MTP 028 Tensor, pipeline, expert, and sequence parallelism for inference 029 Prefix caching, prompt caching, radix attention 030 Cost modeling for inference: tokens, GPUs, dollars 031 Latency budgets, tail latency, and the p99 problem 032 Multimodal: vision-language, audio, the tokenizer trick 033 The attention compression family: MHA, MQA, GQA, MLA 034 Mixture of Experts: routing, balancing, and the inference cost story 035 Long context: RoPE, YaRN, position interpolation, ring attention, sparse attention 036 Disaggregated prefill/decode: production reality with workload-dependent payoff 037 KV cache compression and offload: LMCache, RDMA, NVMe tiering 038 Hardware-aware kernel design: CUDA, CUTLASS, Triton, TVM 039 CUDA programming for ML engineers 040 Model optimization and compilation 041 State-space models: Mamba, Mamba-2, hybrid architectures 042 Test-time compute and reasoning models: o1, R1, MCTS-decoding 043 Structured generation: guided decoding, JSON mode, regex constraints, FSM masking 044 The serving framework landscape: vLLM, SGLang, TensorRT-LLM, TGI, llama.cpp, MLC, Triton 045 Inference servers and orchestration: KServe, BentoML, Seldon, Ray Serve, Triton Inference Server 046 Composing an inference platform: the umbrella pattern 047 KServe InferenceService anatomy: runtime, predictor, transformer, autoscaling 048 vLLM in production: every flag that matters 049 TEI for embeddings and rerankers in production 050 AI gateways: Envoy AI Gateway and the OpenAI-compatible front door 051 Autoscaling GPU inference with KEDA 052 The model cold-start problem and pre-cached weights 053 KV cache sharing across replicas: LMCache and friends 054 Warmup, readiness probes, and the model-isn't-ready-yet problem 055 Benchmarking inference: methodology, tools, gotchas 056 Content safety as inference: guardrails architecture

Part IV. Information Retrieval & RAG

0/9

057 Information retrieval primer: TF-IDF, BM25, why they still matter 058 Dense retrieval: embeddings, contrastive learning, MTEB 059 Vector index internals: HNSW, IVF, ScaNN, FAISS 060 Hybrid search and fusion 061 Chunking strategies 062 Reranking with cross-encoders 063 Query rewriting, HyDE, multi-query, query decomposition 064 RAG evaluation: Ragas, LLM-as-judge, golden sets 065 Designing a RAG system end to end

Part V. Agents, Tool Use, Workflow Orchestration

0/7

066 Tool calling and function calling: the wire protocols 067 The agent loop: ReAct, plan-and-execute, reflection 068 Multi-agent patterns and when they're worth it 069 MCP: the Model Context Protocol in depth 070 Workflow vs agent: when to use durable execution vs an LLM loop 071 Production agent failure modes 072 Designing an agent orchestration layer

Part VI. Distributed Systems & Request Lifecycle

0/12

073 The unified gateway pattern: one API in front of many backends 074 AuthN vs AuthZ: the distinction interviewers test on 075 Identity propagation across service boundaries 076 Rate limiting algorithms: token bucket, leaky bucket, sliding window, GCRA 077 Backpressure, flow control, and queue theory 078 Idempotency keys and exactly-once semantics 079 Sync, async, SSE, WebSocket, batch: five execution modes 080 Workflow orchestration: Temporal vs Airflow vs Step Functions vs Cadence 081 Inter-service trust patterns 082 The operations service pattern: why job lifecycle is its own SoR 083 Metering and billing pipelines 084 Telemetry firehoses: Kafka as the backbone

Part VII. The Data Plane

0/7

085 Object storage primer: the S3 model 086 Document stores: MongoDB and DynamoDB compared 087 Time-series databases: TimescaleDB, Prometheus TSDB, InfluxDB 088 Streaming data: Kafka, Strimzi, the broader log model 089 Caching patterns: Redis, cache-aside, write-through, write-behind 090 The lakehouse story: Parquet, Iceberg, Delta, Hudi 091 Feature stores: Feast, Tecton, the offline/online split

Part VIII. Observability, Reliability, Incidents

0/9

092 The four golden signals, RED, USE, LETS 093 Metrics: Prometheus internals 094 Logs: Loki, structured logging, sampling, retention economics 095 Distributed tracing: OpenTelemetry, Jaeger, Tempo 096 Continuous profiling: Pyroscope, Parca 097 SLI, SLO, SLA, error budgets 098 Canary patterns: traffic-shifted, statistical, ML model canaries 099 Incident management: postmortems, blameless culture 100 ORR: what production-readiness actually means

Part IX. Build, Deploy, Operate

0/13

101 Build systems for monorepos: Bazel, Pants, Buck, Nx 102 Container fundamentals: namespaces, cgroups, OCI spec 103 Dependency injection patterns: Wire, manual, runtime DI 104 API contract design: OpenAPI vs gRPC vs GraphQL vs Connect 105 Python tooling: uv, ruff, black 106 The OCI image lifecycle: registries, digest pinning, the digest-update pattern 107 GitOps philosophy: ArgoCD, Flux, the App-of-Apps pattern 108 Helm vs Kustomize vs CDK8s 109 Multi-cluster, multi-region, multi-cell architecture 110 IaC: Terraform, Pulumi, CDK 111 Secrets management: 1Password, Vault, External Secrets Operator 112 Edge ingress: Cloudflare Tunnels, Ingress controllers, service meshes 113 CI as a system: path filters, per-service builds, coverage gates

Part X. ML System Design Interview Playbook

0/17

114 The interview framework: clarify, estimate, design, drill, ops 115 Capacity planning math: the back-of-envelope kit 116 Design a chatbot for 1M users 117 Design a RAG system over 10TB of documents 118 Design a content moderation pipeline 119 Design a real-time recommendation system 120 Design a multi-tenant model serving platform 121 Design a multi-tenant fine-tuning service 122 The vocabulary that interviewers respect 123 Common mistakes and how to recover mid-interview 124 The coding interview: twenty ML systems algorithms 125 Design: the upstream platforms 126 Design: the production infrastructure 127 Design: the frontier scenarios 128 Behavioral interviews and the levels ladder 129 The day of the interview 130 Company-specific prep and mock interview transcripts

Appendices

A Glossary B Reading list C Math review for the rusty D Hardware reference E Interview question bank

Search the book

Notes Flashcards Playground Tools Formulas Comparisons Cheat sheets PDF

My highlights & notes

All your highlights and notes across the book. Stored locally in your browser.

No highlights yet

Select text in any chapter and click "Highlight" to save it here.