LLM Workshop with RAG, Fine-Tuning and MLOps

Focus

Key Takeaways

• Cognitive & Attention Foundations: Sensory noise isolation, finite resource distribution, sequence-to-sequence latent capture, and the Attention Is All You Need paradigm.
• Probability, Surprise & Temperature: Information-theoretic surprise, Negative Log-Likelihood (NLL), softmax normalization mechanics, and temperature as a sampling viscosity throttle.
• The Core Attention Mechanism: Query, Key, and Value vector formulation, the scaled dot-product attention equation, and dictionary lookup dynamics.
• Mechanistic Interpretability & LarQL: Probing polysemanticity with Sparse Autoencoders (SAEs), LarQL 3-tier token lifecycle (Early/Middle/Late layers), and the residual stream highway with QKV projection blocks

Lab

• Contrastive Loss
• LarQL: Querying open-weight model internals as relational databases

Focus

Key Takeaways

• Text Embeddings & Pooling Strategies: Deep bidirectional transformers (BERT), pre-training vs. fine-tuning transfer learning, and distilling sequence context via CLS and mean pooling.
• Vision Transformers (ViT): Patch-based linear projection (“An Image is Worth 16×16 Words”), visual tokens as vocabulary, locality vs. global attention, and inductive bias tradeoffs in parameter efficiency.
• Multimodal Latent Spaces & Cross-Modal Alignment: Semantic and hierarchical chunking, the paradigm shift of Late Chunking, and contextual chunking.

• Embedding Basics
• Vision language understanding(CLIP, BLIP, BLIP-2)

Focus

Key Takeaways

• Chunking & Failure Modes:

  Context rot, knapsack problem, the centroid delusion in large chunks, and endophora / grammatical anchor severance.
• State-of-the-Art Chunking & Visual Modalities: Contextual chunking, late chunking.

• Derivative Artifacts:

  Pre-computed proposition factoids, passage rewrites, anticipatory QA pairs, and abstractive summarization to align document registers with search intent.
• Hierarchical Tree Retrieval (RAPTOR): Gaussian clustering of semantic embeddings, recursive summarization trees, and “zoom lens” multi-altitude navigation.

Lab

Focus

Key Takeaways

• Graph Science & Complex Networks:

  Small world phenomena, power laws (Barabási–Albert), scale-invariance / fractals, and ontological triples.

• Graph Theory & Community Detection:

   Adjacency, degree, and Laplacian matrices, modularity maximization, and Louvain / Leiden partitioning algorithms.

• Relational vs. Spatial Semantics:

  Continuous spatial proximity vs. topological connectivity, perfect data provenance, and the Sidecar architecture pattern

• MemGraphRAG & Personalized PageRank:

  3-layer topological layout (class-instance indexing), Personalized PageRank semantic energy diffusion, and Banach fixed-point optimization.

Lab

• GraphRAG
• LightRAG

Focus

Key Takeaways

• The Evolution of Code and Understanding: From natural language to binary, and the core process of understand-and-generate.
• The Enduring Power of SQL: Relational algebra origins, fundamental SQL operations, the NoSQL misconception, and the return to SQL.
• The Challenge of Enterprise SQL Generation: Enterprise schema reality, DDL pathologies, the insanity of naive RAG, and read-only safety protocols.

• Advanced Techniques for Robust SQL Systems:

  Functional partitioning via graph theory, the Shadow DDL and aliasing, automated Exploratory Data Analysis (EDA), the Common Table Expression (CTE) strategy, and the recursive improvement loop.

Lab

• Text2SQL from First Principles

Focus

Key Takeaways

• The Visual Page as a Semantic Unit: Overcoming spatial document layout destruction caused by linear text parsers on tables, forms, and multi-column figures.
• DeepSeek OCR Architecture & Tokenization: End-to-end vision-language optical character recognition, dual-resolution visual encoding, and compressing high-resolution page features.
• Text RAG vs. Visual RAG Tradeoffs: Comparing classical OCR-plus-chunking pipelines against direct visual embedding indexing and token-efficient page representations.
• Multimodal Document Understanding: Cross-modal grounding for structured forms, diagrams, scientific literature, and video frame chunking.

Lab

• Visual document parsing (DeepSeek OCR)
• Text RAG vs. Visual RAG

Focus

Key Takeaways

• Prompt Robustness & Epistemic Uncertainty:

  Human ambiguity vs. machine predictability, and the escalation ladder.
• Structured Prompting Frameworks: Zero-shot and few-shot pattern setting, the CO-STAR framework, metaprompting, and Chain of Thought reasoning vs. non-reasoning models.
• The AI Escalation Ladder & Model Routing: Prompt Engineering → RAG → SFT: Capability Tiers, Epistemic Capacity Saturation, Model-Routing Economics, and the Anti-Pattern
• DSPY: Declarative signatures, parameterized modules, the optimizer as a natural language compiler, and the PyTorch paradigm in prompting.
• Optimization Algorithms: COPRO coordinate ascent, MIPRO Bayesian joint-space optimization, GEPA genetic-Pareto adaptation, TextGrad backpropagation in natural language, and evolutionary frameworks (APE, Prompt Breeder, EvoPrompt).
• Production Stability & Economics: The four variables of silent prompt degradation, model sunsetting and prompt migration, semantic core vs. linguistic variation, and token economics.

Lab

• Prompt Engineering
• Prompt Optimization: DSPy Basics, DSPy Optimizers

Focus

Key Takeaways

• Core Mental Models & Two-Gate Architecture: The Wizard of Oz inversion, Gate 1 (Request security) vs. Gate 2 (Response conscience), and defense-in-depth.
• Request-Side Threat Vectors: Prompt injection, PoisonedRAG, canary tokens, indirect injection, and rate limiting / denial-of-wallet protections.
• Response-Side Grounding & Verification: Structural claim decomposition, bipartite graph modeling, RAGAS faithfulness triad, NLI execution, and the Three Doors of Refusal.
• Enterprise Authorization & Entitlements: The RAG security trilemma, pre-retrieval ACL/RBAC/ABAC/ReBAC filtering, and surfaces of information leakage.

Lab

• Input Guardrails
• Response Grounding

Focus

Key Takeaways

• Foundations of Metrology & Dataset Design: Goodhart’s Law in AI evaluation, failure of public benchmarks, and SME-curated measurement yardsticks.
• Retrieval Metrics Suite: Unranked metrics (Precision, Recall, F1), rank-aware metrics (MRR, MAP), and graded relevance (NDCG with exponential discounting).

• Statistical Rigor & Significance Testing:

  Cluster-robust standard errors, paired statistical testing, and establishing statistical power.
• Calibration & Search Diversity: Incomplete judgments, expected calibration, and Maximal Marginal Relevance (MMR) for polar cap aperture control.

Lab

• Retrieval metrics

Focus

Key Takeaways

• Sparse, Dense & Multi-Vector Models: BM25, SPLADE contextual sparse expansion, ColBERT token-level late interaction ($MaxSim$), and Matryoshka Representation Learning (MRL).
• Contrastive Learning & Loss Dynamics: InfoNCE mechanics, temperature ($tau$) viscosity throttle, Decoupled Contrastive Loss, and the Long Context Mirage.

• Query Transformation & Semantic Caching:

  Query pathologies/verbs, HyDE (crossing register gaps), and semantic cache threshold mechanics.

Lab

• Matryoshka Representation Learning
• Retrieval Funnel
• Query Transformation Pipeline
• Semantic Cache

Focus

Key Takeaways

• Error Modes & Sins of Generation: Fluent substitution, intrinsic hallucination, parametric leakage, FActScore atomic decomposition, and abstinence-aware scoring rules.
• Generative Evaluation Frameworks: RAGAS Triad (Context Relevance, Faithfulness, Answer Relevance), citation verification with ALCE, and the Four RGB foundational abilities.
• LLM-as-a-Judge Calibration & Synthetic Data: Inter-annotator agreement (Cohen’s Kappa), ensemble jury panels, and Prediction-Powered Inference (PPI / ARES).
• Production Observability & Diagnostics: Pass@K metrics, silent decay monitoring, shadow/canary traffic routing, and cost-per-correct-answer optimization.

Lab

• RAGAS Evals
• Ablation Study on Retrieval Funnel

Focus

Key Takeaways

• Historical Precedents & Epistemological Challenges: The Mundaneum precedent, spatiotemporal extents (“currently” problem), and provenance vs. empirical validation trust gradients.
• Open Knowledge Format (OKF): Minimalist Markdown bundle specification, front matter vs. body separation, progressive disclosure (`index.md`), and moving intelligence to authoring time.
• Cognitive Architecture & 4 Memory Tiers: Working memory, episodic session state, long-term semantic memory, and procedural tool playbooks.
• System Governance & Self-Annealing Knowledge: Git-backed knowledge bases, machine patrols vs. human adjudication, preventing corpus rot, and defenses against hallucination.

Lab

• OKF bundle authoring and governance
• Memory frameworks (to decide)

Focus

Key Takeaways

• Operational Landscape & Scaling Laws: Inference vs. adaptation decision gates, neural scaling laws, Chinchilla compute constraints.
• Geometry Engineering & Contrastive Loss: Anisotropy-to-isotropy transformation, Triplet loss, InfoNCE batch repulsion, hard negative mining, and Sentence-BERT bi-encoders.
• Economics & Hardware Constraints: GPU memory economics, and selective layer fine-tuning.
• Parameter-Efficient Fine-Tuning (PEFT): Mathematics of LoRA low-rank decomposition, rank allocation, quantization, and soft prompting / prefix tuning.

Lab

Focus

Key Takeaways

• Foundations of Reinforcement Learning: Markov Decision Processes (MDP), state-action value functions, policy gradients (REINFORCE), and the credit assignment problem.
• The InstructGPT Alignment Pipeline: 3-Step Methodology: SFT → Reward Modeling → PPO, KL-Divergence Tethering, and Reward-Hacking Defenses
• Direct Preference Optimization (DPO): Critic-free implicit reward derivation and pairwise preference loss without auxiliary reward models.
• Group Relative Policy Optimization (GRPO) & Reasoning: Baselines from group variance, DeepSeek R1 paradigm, and Reinforcement Learning with Verifiable Rewards (RLVR).

   Lab

Focus

Key Takeaways

• MLOps vs. DevOps Lifecycle: Galileo’s Square-Cube law in infrastructure, immutable data versioning, tournament of libraries, and model registries.
• Data & Model Versioning: Managing dataset immutability, large-file tracking, and artifact lineage with DVC, LakeFS, and Git LFS.
• Experiment Tracking & Observability: Metric visualization, run comparison, and LLM tracing using MLflow, TensorBoard, Weights & Biases (W&B), and Arize Phoenix.

   Lab

Focus

Key Takeaways

• Physics of Distributed Computing & Parallelism: Latency hierarchy, Data Parallelism (DDP, FSDP, DeepSpeed ZeRO), Tensor Parallelism, Pipeline Parallelism, and the CAP theorem.
• The Ray Distributed Framework: Hardware shift from Spark to Ray, Ray Tasks (stateless) vs. Ray Actors (stateful), Global Control Store, and Apache Arrow shared memory.

• High-Throughput Serving & Scalability:

  Ray Core, Ray Train, Ray Tune, Ray Serve with vLLM, continuous batching, and speculative decoding.

Lab

• Ray core and Ray data