Advanced Fine-tuning & MlOps

Each week follows the same high-level pattern:
• A core theme and set of learning objectives.
• Required textbook readings (as in below Table).
• Two deep-read research papers, discussed in detail in class.
• A lab or project component (Ray, Kubeflow, MLflow, W&B, Arize).
• An Oliver Twist list: extra papers, talks, and essays.

Focus
Scaling laws, foundation models, and the shift from training-from-scratch to fine-tuning.

Key Takeaways

• Hyperscalers pretrain; practitioners adapt (fine-tuning as the norm)

• Chinchilla shows compute-optimal scaling, not just bigger models

• Transfer learning dominates modern ML practice

Core Readings

• Bommasani et al. — Foundation Models

• Hoffmann et al. — Chinchilla

Lab

• Zero/few-shot evaluation of open LLMs (LLaMA/Mistral)

• Batched inference with Ray

• Metrics via MLflow or W&B

Explore More (Optional)
Kaplan (Scaling Laws) · Wei (Emergence) · LLM Alignment (RLHF/DPO) · Karpathy talks

• Next-token and masked-token prediction act as surrogate supervision, shaping semantic manifolds

• BERT demonstrates representation learning; GPT-3 demonstrates scale → few-shot behavior

• Embeddings become reusable substrates for downstream tasks

• Linear probes expose latent knowledge without task-specific training

• Emergent abilities and in-context learning arise without explicit supervision

• Fine-tuning moves models along a learned parameter manifold, not from scratch

• Fisher information and curvature explain which directions change behavior vs. destroy knowledge

• Catastrophic forgetting arises from moving along “stiff” directions (EWC as constraint)

• Preference-based fine-tuning (RLHF-style) reshapes behavior without full retraining

• Sharpness and loss landscape geometry predict robustness and generalization

• Full fine-tuning maximizes adaptation but is costly and risks forgetting

• Partial fine-tuning (last-N layers) trades flexibility for stability and efficiency

• PEFT methods constrain learning to low-dimensional subspaces

• LoRA reframes adaptation as low-rank updates to pretrained weights

• Memory, compute, and training time—not accuracy alone—drive method choice

• Instruction tuning converts generic LMs into task-following assistants

• FLAN shows scale + diverse instructions → strong zero-shot generalization

• Self-Instruct bootstraps alignment using model-generated instructions

• LoRA enables instruction tuning at low cost and risk

• Evaluation shifts from perplexity to behavioral correctness and usefulness

• Transfer learning generalizes beyond language to vision and multimodal domains

• ResNets show how depth and residuals enable scalable visual representations

• CLIP aligns vision and language via contrastive pretraining

• Multimodal embeddings become shared interfaces across tasks and modalities

• Fine-tuning adapts perception models to domain shift with minimal data

• One-off experiments fail without reproducibility and orchestration

• Pipelines formalize data, training, and evaluation as first-class artifacts

• Ray scales preprocessing and training across distributed workloads

• Kubeflow operationalizes ML workflows from experiment to production

• Model registries (MLflow) turn models into versioned, auditable assets

• Reinforcement learning frames learning as policy improvement under feedback

• Policy gradients connect rewards to parameter updates

• PPO stabilizes learning via clipped, KL-regularized updates

• PPO’s simplicity and robustness make it the workhorse of RLHF

• Understanding PPO clarifies how preference signals reshape LLM behavior

• RLHF formalizes alignment as learning from human preferences

• Stage 1: Supervised fine-tuning creates a reasonable base policy

• Stage 2: Reward models learn to score outputs from preference data

• Stage 3: PPO optimizes the policy under a KL constraint

• Monitoring reward, KL, and entropy reveals alignment–capability tradeoffs

• In RLHF, the reward model—not the policy—is the true objective

• Preference learning translates human judgments into a trainable signal

• Mis-specified rewards lead to overoptimization and reward hacking

• Calibration and evaluation of reward models are as critical as policy tuning

• Offline preference optimization exposes alignment gaps before deployment

• Preference optimization without explicit reinforcement learning

• Direct Preference Optimization (DPO) treats the LM as an implicit reward model

• Constitutional AI (RLAIF) uses AI-generated feedback for scalable alignment

• KL control, training stability, and sample quality are key metrics

• Hybrid human+AI feedback improves efficiency and reduces annotation burden

• Multi-agent RL extends single-agent learning to interacting policies

• Agents specialize into roles: drafter, critic, planner, executor

• Feedback loops between agents enable self-improvement and lightweight alignment

• Observability, credit assignment, and evaluation become system-level challenges

• MARL principles applied to multi-agent LLMs (Voyager, AutoGen, AutoGPT)

• Production LLMs require observability for safety, performance, and alignment

• Log prompts, responses, embeddings, and feedback to detect drift

• Dashboards and metrics enable monitoring of model behavior over time

• Red-teaming and guardrails reduce harms and mitigate failures

• Multi-objective alignment and failure analysis inform iterative improvements