Part II: Training, Fine-Tuning, Alignment

Pre-Chinchilla models like GPT-3 were trained on far fewer tokens than Chinchilla-optimal. The law shows that doubling compute should split roughly evenly between more parameters and more data, not go entirely to bigger models.

When the same text appears thousands of times, the model strongly memorizes it verbatim, hurting generalization and creating extraction risks. Deduplication also prevents benchmark contamination if test-set text appears in the crawl.

With a 100k vocabulary, token IDs don't fit in 1 byte (max 255) but do fit in 2 bytes (uint16, max 65535 — or use int32 at 4 bytes for safety). In practice most implementations use int32/uint32 giving 60 TB, but 2-byte uint16 is common for storage-optimized pipelines.

A frontier pretraining run costs $50M–$500M+ in compute alone, requires custom distributed infrastructure, and the resulting model rarely outperforms open-weights alternatives for most downstream tasks. Fine-tuning open weights is orders of magnitude cheaper and often produces better task-specific results.

With Stage 3, each GPU only holds a shard of the parameters. Before computing any layer, all ranks must all-gather that layer's parameters, then discard them after use. This all-gather happens once per layer per forward pass and again per backward pass — significant communication but the only way to fit models whose parameters alone exceed single-GPU memory.

In Megatron-style column/row tensor parallelism, each GPU computes a partial result of the matrix multiply. An all-reduce sums these partial results across all tensor-parallel ranks before the output is passed to the next operation. (Reduce-scatter + all-gather is equivalent but used in sequence parallel variants.)

In a simple pipeline with F stages, up to F-1 stages are idle (bubbled) while the first micro-batch moves through. Micro-batching and interleaved schedules reduce but don't eliminate this overhead. The bubble fraction is (F-1)/(m + F-1) for m micro-batches.

AdamW stores two fp32 (4-byte) moments per parameter: 2 × 4 × 70×10^9 = 560 GB of optimizer state total. ZeRO Stage 1 shards this evenly across all ranks, so each of the 64 GPUs holds 560/64 ≈ 8.75 GB — still significant, but far less than the 560 GB total.

fp16 has only a 5-bit exponent, limiting its dynamic range. Gradients that are small (common late in training) underflow to zero. bf16's 8-bit exponent matches fp32's dynamic range, so very small or very large gradients are representable without loss scaling.

fp16's smallest normal number is ~6×10^-5. Late-stage gradients are often much smaller. Loss scaling multiplies the loss by a large constant before backward, scaling all gradients up so they don't underflow; the optimizer then divides them back before the parameter update.

When a gradient update is much smaller than the current weight magnitude, adding them in bf16 can round the update away entirely. The fp32 master copy preserves these small deltas accurately. After each optimizer step the fp32 master is downcast to bf16 for use in the forward/backward passes.

fp8 (e4m3) can represent values only in a narrow range. Without per-tensor or per-channel scale factors, most activations and gradients will either overflow to NaN or underflow to zero. Calibration measures the actual value distributions and chooses scale factors that keep values in fp8's representable range.

BPE always merges the most frequent adjacent pair. This greedy frequency-first approach is the core algorithm and has no notion of morphology or entropy.

Each additional merge reduces average sequence length, but the new merged token is rarer in the corpus. Rarer tokens have fewer learning examples, making each token harder to embed well.

Every token embedding in the pretrained model is keyed to a specific integer ID. Retraining the tokenizer changes which string maps to which ID, invalidating all learned embeddings.

Languages like Chinese and Japanese have no word-boundary spaces. SentencePiece processes the raw character stream with space as a symbol, so the same algorithm applies across all writing systems without a language-specific pre-tokenizer.

LoRA decomposes the weight update dW into A (d x r) times B (r x k). Rank r is the bottleneck dimension; smaller r means fewer trainable parameters and stronger regularization.

QLoRA keeps the base model in NF4 (roughly 4 bits/parameter) on GPU, which cuts the base model's memory footprint by ~4x. Adapter parameters and optimizer state remain in higher precision.

Separate adapters add an extra matrix multiply at every forward pass. Merging (W_merged = W + AB) bakes the delta directly into the weight, eliminating adapter overhead with no quality change.

2000 examples is tiny relative to the number of parameters. A high learning rate on such a small dataset commonly drives the model to overfit the training data while erasing pretrained knowledge, a classic catastrophic forgetting failure.

The goal of SFT is to teach the model to produce good responses. Backpropagating through prompt tokens would train the model to predict user inputs, which is not the desired behavior and dilutes the training signal.

Catastrophic forgetting happens when gradient updates for the new task overwrite weights that encode general skills. Including diverse general instruction examples in the training mix preserves those capabilities.

SFT is imitation learning. A model trained on noisy or low-quality responses learns to produce noisy, low-quality outputs. A small set of excellent demonstrations reliably outperforms a large set of mediocre ones.

The chat template is baked into the token sequence the model was trained on. Different special tokens and delimiters mean the model cannot reliably parse which tokens are system, user, and assistant turns.

Reward hacking: unconstrained PPO discovers outputs the reward model scores highly but that are nonsensical or repetitive. The KL penalty keeps the policy close to the SFT reference, bounding how far it can deviate.

DPO derives a closed-form expression for the optimal reward in terms of the policy and the reference, then collapses the pipeline into a single loss on preference pairs without needing a separate reward model.

PPO-based RLHF needs the active policy (being updated), a frozen copy of the SFT reference (for KL), the reward model (for scoring rollouts), and a value model (critic for advantage estimation).

DPO requires matched (prompt, chosen, rejected) triples. Real-world feedback systems often collect per-response thumbs up or down without creating matched pairs. KTO is designed for this unpaired binary signal.

Soft targets expose inter-class similarity in the teacher's probability distribution (e.g., a 'cat' image scoring 0.01 for 'dog' is informative). Hard one-hot labels discard all of this structural information.

Dividing logits by T before softmax stretches the distribution. At T=1 the distribution is often near one-hot; at T>1 smaller logits produce larger probabilities, giving the student richer gradient signal from the similarity structure.

Irregular sparse weight matrices cannot use standard dense matrix multiply hardware efficiently. Without hardware-native sparse support, the theoretical compute savings do not translate to wall-clock speedup, whereas INT8 and FP8 have native Tensor Core paths.

Identifying the winning ticket requires full training (or iterative magnitude pruning) of the original network. You can only prune after training, eliminating any training-time compute savings.

Self-Instruct uses a model to bootstrap diversity from a small seed set, generating new instructions and responses that are then filtered and used for fine-tuning. This breaks the dependence on large human annotation.

Rejection sampling is useful when you can define 'best' cheaply. For math and code, exact correctness is verifiable. For open-ended tasks, a reward model provides the ranking signal.

Each model generation produces outputs that reflect its own biases and blind spots. When the next generation trains on those outputs, it learns the biases more strongly, and the cycle compounds across generations.

The self-critique loop is only as good as the model's ability to judge its own outputs. A weak or misaligned model produces approvals for its own flawed responses, making the synthetic preference data meaningless or harmful.

If a model has seen benchmark questions during training, it can recall rather than reason. The resulting score reflects memorization, not capability, making comparisons across models unreliable.

LLM judges show position bias (preferring whichever response comes first), verbosity bias (preferring longer answers), and self-enhancement bias (preferring outputs similar to their own training distribution).

Static benchmarks have fixed public questions that can leak into training data. Arena questions are unpublished user queries submitted live, so there is no fixed test set to contaminate.

No single eval tells the full story. A focused fine-tune commonly trades some general capability for task performance. The right interpretation requires comparing the tradeoff against the actual user requirement, not declaring one eval the winner.