# Wiki — M.E.F LLM Studio Source: https://www.mef-llm-studio.com/en/wiki/ Look up concepts, explore connections and suggest improvements. Wiki SHA-256 a6faabca957c80d6bc83223d25f6cda0ba14e3901435e609cab7dca71bc273f1 ## LLM Fundamentals What a large language model learns, generates, and does not guarantee. **What does it mean?:** A large language model estimates probabilities for linguistic continuations from learned parameters. It produces plausible sequences but has, on its own, neither current source authority nor a proof of truth for individual statements. **Why does it matter?:** Without this basic picture, fluency, knowledge, reasoning, and verification are easily confused with one another. **Important limit:** An output that sounds probable is neither automatically true nor traceable to a specific source. - [Transformers](https://www.mef-llm-studio.com/en/wiki/#transformers) - [Tokenization](https://www.mef-llm-studio.com/en/wiki/#tokenization) - [LLM Inference](https://www.mef-llm-studio.com/en/wiki/#llm-inference) - [Hallucination Detection](https://www.mef-llm-studio.com/en/wiki/#hallucination-detection) ## Transformers An architecture that processes sequences through attention instead of recurrent state chains. **What does it mean?:** Transformers consist of repeated attention and feed-forward blocks. They model relationships between positions in an input sequence and can be trained with a high degree of parallelism. **Why does it matter?:** They form the technical foundation of most modern language models and of many multimodal models. **Important limit:** The architecture says nothing about whether a result is correct, current, or authorized. - [Attention Mechanism](https://www.mef-llm-studio.com/en/wiki/#attention) - [Multi-Head Attention](https://www.mef-llm-studio.com/en/wiki/#multi-head-attention) - [Transformer Block](https://www.mef-llm-studio.com/en/wiki/#transformer-block) - [Residual Stream](https://www.mef-llm-studio.com/en/wiki/#residual-stream) - [Positional Encoding](https://www.mef-llm-studio.com/en/wiki/#positional-encoding) - [LLM Fundamentals](https://www.mef-llm-studio.com/en/wiki/#llm-fundamentals) ## Attention Mechanism Each position mixes the positions it may read, with weights that add up to 100%. **What does it mean?:** For a query position, attention scores every key it may read, turns the scores into weights with softmax and adds up the values with those weights. Each query head computes its own weights; in GPT-2-style models each head also has its own key and value projections, while with grouped-query attention (Llama, Qwen2) several query heads share one key/value head. **Why does it matter?:** This lets models capture long-range references and context-dependent relationships. **Important limit:** Attention weights are not a reliable explanation and not a proof of causal use. - [Transformers](https://www.mef-llm-studio.com/en/wiki/#transformers) - [Multi-Head Attention](https://www.mef-llm-studio.com/en/wiki/#multi-head-attention) - [Attention Head](https://www.mef-llm-studio.com/en/wiki/#attention-head) - [Causal Mask](https://www.mef-llm-studio.com/en/wiki/#causal-mask) - [Softmax](https://www.mef-llm-studio.com/en/wiki/#softmax) - [Embeddings](https://www.mef-llm-studio.com/en/wiki/#embeddings) - [Observability](https://www.mef-llm-studio.com/en/wiki/#observability) ## Multi-Head Attention Several attention projections view the same sequence in parallel. **What does it mean?:** Multi-head attention splits the representation into several learned attention heads. Each head can capture different relationships or projections; the results are then merged. **Why does it matter?:** The parallel views increase the expressiveness of a transformer block. **Important limit:** A head has no firmly guaranteed semantic role unless that role has been measured separately. - [Attention Mechanism](https://www.mef-llm-studio.com/en/wiki/#attention) - [Attention Head](https://www.mef-llm-studio.com/en/wiki/#attention-head) - [Transformer Block](https://www.mef-llm-studio.com/en/wiki/#transformer-block) - [Transformers](https://www.mef-llm-studio.com/en/wiki/#transformers) - [LLM Evaluation](https://www.mef-llm-studio.com/en/wiki/#llm-evaluation) ## Tokenization The breaking of text into discrete model units. **What does it mean?:** Tokenization translates text into IDs a model can process. Depending on the method, tokens correspond to words, word parts, characters, or bytes. **Why does it matter?:** It affects sequence length, multilingualism, number representation, and the model's view of spelling variants. **Important limit:** Token boundaries are technical boundaries and not automatically linguistic or ontological ones. - [Embeddings](https://www.mef-llm-studio.com/en/wiki/#embeddings) - [Context Windows](https://www.mef-llm-studio.com/en/wiki/#context-windows) - [LLM Fundamentals](https://www.mef-llm-studio.com/en/wiki/#llm-fundamentals) ## Embeddings Learned vectors for discrete or structured objects. **What does it mean?:** Embeddings map tokens, sentences, documents, or other objects to dense numeric vectors. Similar vectors can indicate statistically similar usage. **Why does it matter?:** They make discrete inputs processable for neural models and power semantic search methods. **Important limit:** Vector proximity is not identity, not truth, and not an authority relationship. - [Token Embedding](https://www.mef-llm-studio.com/en/wiki/#token-embedding) - [Semantic Search](https://www.mef-llm-studio.com/en/wiki/#semantic-search) - [Vector Databases](https://www.mef-llm-studio.com/en/wiki/#vector-databases) - [Attention Mechanism](https://www.mef-llm-studio.com/en/wiki/#attention) ## Positional Encoding Information about where an element sits in a sequence. **What does it mean?:** Because attention alone knows no order, models add position information: GPT-2-style models add a learned position vector to each token embedding, rotary encoding (RoPE, used by Llama and Qwen2) rotates queries and keys by position inside attention, and relative schemes bias the attention scores. This lets the model distinguish order and distance. **Why does it matter?:** The choice affects length generalization and the learnability of relational order. **Important limit:** A position signal can create unwanted shortcuts when position and target correlate during training. - [Transformers](https://www.mef-llm-studio.com/en/wiki/#transformers) - [Attention Mechanism](https://www.mef-llm-studio.com/en/wiki/#attention) - [Context Windows](https://www.mef-llm-studio.com/en/wiki/#context-windows) ## Prompt Engineering Systematic design of the instruction and the provided context. **What does it mean?:** Prompt engineering structures task, examples, output format, boundaries, and context so that a model responds more reliably in a desired role. **Why does it matter?:** It is a fast lever for behavior, especially without training. **Important limit:** A good prompt is neither a security mechanism nor a guarantee of factual or tool fidelity. - [Few-Shot Learning](https://www.mef-llm-studio.com/en/wiki/#few-shot-learning) - [Structured Outputs](https://www.mef-llm-studio.com/en/wiki/#structured-outputs) - [Prompt Evaluation](https://www.mef-llm-studio.com/en/wiki/#prompt-evaluation) - [Guardrails](https://www.mef-llm-studio.com/en/wiki/#guardrails) ## Few-Shot Learning Task steering through a few examples in the context. **What does it mean?:** In few-shot learning the model receives a few input-output examples inside the prompt and infers the desired pattern for new cases from them. **Why does it matter?:** It can convey format and task schema without a weight update. **Important limit:** Examples consume context and can anchor unwanted patterns or errors as a template. - [Prompt Engineering](https://www.mef-llm-studio.com/en/wiki/#prompt-engineering) - [Context Windows](https://www.mef-llm-studio.com/en/wiki/#context-windows) - [Fine-Tuning](https://www.mef-llm-studio.com/en/wiki/#fine-tuning) ## Chain-of-Thought Intermediate steps a model produces before or alongside its answer. **What does it mean?:** Chain-of-thought refers to textual intermediate reasoning that can break complex tasks into smaller steps. Visible reasoning can be helpful, but is itself model text. **Why does it matter?:** The technique can improve handling of multi-step tasks and make failure points visible. **Important limit:** A plausible chain of thought is no proof that exactly those steps were internally causal or factually correct. - [Structured Outputs](https://www.mef-llm-studio.com/en/wiki/#structured-outputs) - [LLM Evaluation](https://www.mef-llm-studio.com/en/wiki/#llm-evaluation) - [Observability](https://www.mef-llm-studio.com/en/wiki/#observability) ## RAG Retrieval-augmented generation combines search with text generation. **What does it mean?:** RAG retrieves relevant content at runtime and gives it to a generator as context. This means current or private facts do not have to reside entirely in model weights. **Why does it matter?:** It improves updatability and can enable source attribution. **Important limit:** Found does not mean admitted; context does not mean evidence; citing does not mean the statement follows from the source. - [Semantic Search](https://www.mef-llm-studio.com/en/wiki/#semantic-search) - [Reranking](https://www.mef-llm-studio.com/en/wiki/#reranking) - [Chunking Strategies](https://www.mef-llm-studio.com/en/wiki/#chunking) - [Vector Databases](https://www.mef-llm-studio.com/en/wiki/#vector-databases) ## Vector Databases Storage and indexes for similarity search over vectors. **What does it mean?:** A vector database stores embeddings together with IDs and metadata and finds entries by a distance or similarity measure. **Why does it matter?:** It makes semantic search over large collections practical. **Important limit:** The index decides proximity, not validity, currency, rights, or contradiction. - [Embeddings](https://www.mef-llm-studio.com/en/wiki/#embeddings) - [Semantic Search](https://www.mef-llm-studio.com/en/wiki/#semantic-search) - [RAG](https://www.mef-llm-studio.com/en/wiki/#rag) - [Hybrid Search](https://www.mef-llm-studio.com/en/wiki/#hybrid-search) ## Semantic Search Search by proximity of meaning rather than only identical character strings. **What does it mean?:** Semantic search encodes query and content into representations and ranks candidates by modeled proximity of meaning. **Why does it matter?:** It finds paraphrased or conceptually related content that a pure keyword search misses. **Important limit:** Semantic proximity can confidently deliver wrong candidates and needs open-set and authority boundaries. - [Embeddings](https://www.mef-llm-studio.com/en/wiki/#embeddings) - [Vector Databases](https://www.mef-llm-studio.com/en/wiki/#vector-databases) - [Hybrid Search](https://www.mef-llm-studio.com/en/wiki/#hybrid-search) - [Reranking](https://www.mef-llm-studio.com/en/wiki/#reranking) ## Hybrid Search A combination of lexical and semantic search. **What does it mean?:** Hybrid search combines exact or term-based hits with embedding-based similarity and merges both result lists through rules or ranking. **Why does it matter?:** It covers both precise names and IDs as well as linguistic variants. **Important limit:** The fusion needs a defined calibration; two scores from different spaces are not automatically comparable. - [Semantic Search](https://www.mef-llm-studio.com/en/wiki/#semantic-search) - [Reranking](https://www.mef-llm-studio.com/en/wiki/#reranking) - [Vector Databases](https://www.mef-llm-studio.com/en/wiki/#vector-databases) ## Reranking A second, more precise sorting step for candidates that were already found. **What does it mean?:** A reranker re-scores a small candidate set with more context or a stronger model. It optimizes the ordering, not the original coverage. **Why does it matter?:** It can place relevant evidence earlier and limit the active payload. **Important limit:** A reranker cannot bring back a missing candidate and must not confuse ranking with admission. - [Semantic Search](https://www.mef-llm-studio.com/en/wiki/#semantic-search) - [Hybrid Search](https://www.mef-llm-studio.com/en/wiki/#hybrid-search) - [RAG](https://www.mef-llm-studio.com/en/wiki/#rag) - [Model Routing](https://www.mef-llm-studio.com/en/wiki/#model-routing) ## Chunking Strategies How sources are split into retrievable sections. **What does it mean?:** Chunking divides documents into retrieval units by length, sentences, sections, structure, or semantic boundaries. **Why does it matter?:** The split determines whether enough context is included without blurring the search with large blocks. **Important limit:** A chunk must not lose its source spans, version, and neighborhood; otherwise later provenance is only approximate. - [RAG](https://www.mef-llm-studio.com/en/wiki/#rag) - [Context Windows](https://www.mef-llm-studio.com/en/wiki/#context-windows) - [Semantic Search](https://www.mef-llm-studio.com/en/wiki/#semantic-search) ## Context Windows The maximum active input sequence of a model call. **What does it mean?:** The context window includes system instruction, user input, examples, retrieved content, and prior output that a model can take into account in one call. **Why does it matter?:** It limits active information, cost, and runtime and influences which content is processed together. **Important limit:** More context guarantees no better use; irrelevant or contradictory content can degrade performance. - [Context](https://www.mef-llm-studio.com/en/wiki/#context) - [Chunking Strategies](https://www.mef-llm-studio.com/en/wiki/#chunking) - [RAG](https://www.mef-llm-studio.com/en/wiki/#rag) - [KV Cache](https://www.mef-llm-studio.com/en/wiki/#kv-cache) - [AI Cost Optimization](https://www.mef-llm-studio.com/en/wiki/#cost-optimization) ## Function Calling Model output in a predefined function schema. **What does it mean?:** Function calling lets a model propose a function name and structured arguments instead of only producing free text. **Why does it matter?:** It creates a machine-readable seam between language and program code. **Important limit:** Schema-conformant arguments are not automatically sensible, safe, or authorized. - [Tool Calling](https://www.mef-llm-studio.com/en/wiki/#tool-calling) - [Structured Outputs](https://www.mef-llm-studio.com/en/wiki/#structured-outputs) - [MCP](https://www.mef-llm-studio.com/en/wiki/#mcp) ## Tool Calling A model proposes the use of an external capability. **What does it mean?:** Tool calling extends function calling with concrete external actions such as search, database access, code execution, or business operations. **Why does it matter?:** This lets a language model obtain information or affect real systems. **Important limit:** The model proposal must never alone be the execution authorization; errors, partial results, and side effects need their own contracts. - [Function Calling](https://www.mef-llm-studio.com/en/wiki/#function-calling) - [AI Agents](https://www.mef-llm-studio.com/en/wiki/#ai-agents) - [MCP](https://www.mef-llm-studio.com/en/wiki/#mcp) - [Guardrails](https://www.mef-llm-studio.com/en/wiki/#guardrails) ## AI Agents Systems that pursue goals across several model and tool steps. **What does it mean?:** An AI agent combines model decisions, state, tools, and control logic to work on a task over multiple steps. **Why does it matter?:** Agents can solve dynamic tasks where the next step only emerges from the previous result. **Important limit:** More autonomy enlarges error chains and side effects; a loop is not governance. - [Agentic Workflows](https://www.mef-llm-studio.com/en/wiki/#agentic-workflows) - [Tool Calling](https://www.mef-llm-studio.com/en/wiki/#tool-calling) - [Memory Systems](https://www.mef-llm-studio.com/en/wiki/#memory-systems) - [Guardrails](https://www.mef-llm-studio.com/en/wiki/#guardrails) ## Agentic Workflows Explicit flow structure for agentic steps and decisions. **What does it mean?:** Agentic workflows arrange model calls, tools, checks, retries, and handoffs into a controlled process. **Why does it matter?:** A workflow makes autonomy boundable and allows specialized components instead of an unclear endless loop. **Important limit:** A graphical flow proves no correct authority or error handling; every edge needs semantics. - [AI Agents](https://www.mef-llm-studio.com/en/wiki/#ai-agents) - [Multi-Agent Systems](https://www.mef-llm-studio.com/en/wiki/#multi-agent-systems) - [Observability](https://www.mef-llm-studio.com/en/wiki/#observability) - [Production GenAI Systems](https://www.mef-llm-studio.com/en/wiki/#production-genai) ## Memory Systems Persistent or runtime-adjacent state outside the model context. **What does it mean?:** Memory systems store conversations, observations, facts, working state, or summaries beyond individual model calls. **Why does it matter?:** They enable continuity and reduce the need to put everything into the prompt again. **Important limit:** Stored does not mean true; memories need provenance, version, validity, and deletion or revocation rules. - [RAG](https://www.mef-llm-studio.com/en/wiki/#rag) - [AI Agents](https://www.mef-llm-studio.com/en/wiki/#ai-agents) - [Vector Databases](https://www.mef-llm-studio.com/en/wiki/#vector-databases) - [Observability](https://www.mef-llm-studio.com/en/wiki/#observability) ## Multi-Agent Systems Several separate agent roles work on the same task. **What does it mean?:** Multi-agent systems distribute planning, research, critique, or execution across several model- or rule-based actors with defined message paths. **Why does it matter?:** Specialization and cross-checking can structure complex work. **Important limit:** Agreement among several models is not independent truth; shared training imprint can multiply the same error. - [AI Agents](https://www.mef-llm-studio.com/en/wiki/#ai-agents) - [Agentic Workflows](https://www.mef-llm-studio.com/en/wiki/#agentic-workflows) - [LLM Evaluation](https://www.mef-llm-studio.com/en/wiki/#llm-evaluation) ## MCP An open protocol for tools, resources, and prompts between AI hosts and servers. **What does it mean?:** The Model Context Protocol standardizes communication between host, clients, and specialized servers. Servers can offer clearly declared resources, tools, and prompts; capabilities are negotiated explicitly. **Why does it matter?:** It reduces proprietary point-to-point integrations and makes specialized services combinable. **Important limit:** MCP standardizes the transport and the capability surface, not truth, tool authorization, or evidence quality. - [Tool Calling](https://www.mef-llm-studio.com/en/wiki/#tool-calling) - [Function Calling](https://www.mef-llm-studio.com/en/wiki/#function-calling) - [AI Agents](https://www.mef-llm-studio.com/en/wiki/#ai-agents) ## Structured Outputs Model outputs that follow a fixed machine-readable schema. **What does it mean?:** Structured outputs constrain the output to fields, types, and allowed values, usually via JSON Schema, grammar, or constrained decoding. **Why does it matter?:** They make outputs validatable and reduce parsing ambiguity. **Important limit:** Structurally valid is not semantically correct; a perfectly valid object can carry wrong values. - [Function Calling](https://www.mef-llm-studio.com/en/wiki/#function-calling) - [Guardrails](https://www.mef-llm-studio.com/en/wiki/#guardrails) - [LLM Evaluation](https://www.mef-llm-studio.com/en/wiki/#llm-evaluation) ## Optimizer The rule that turns gradients into parameter updates during training. **What does it mean?:** An optimizer uses gradients and a learning rate to update model parameters. Methods differ in how they combine past gradients, scale individual updates, and apply weight decay. **Why does it matter?:** It affects training stability, memory use, and how quickly a model reaches a useful loss. **Important limit:** An optimizer does not define the training objective or guarantee generalization. This app currently uses AdamW rather than offering an optimizer selector. - [SGD](https://www.mef-llm-studio.com/en/wiki/#sgd) - [RMSprop](https://www.mef-llm-studio.com/en/wiki/#rmsprop) - [Adam](https://www.mef-llm-studio.com/en/wiki/#adam) - [AdamW](https://www.mef-llm-studio.com/en/wiki/#adamw) - [Fine-Tuning](https://www.mef-llm-studio.com/en/wiki/#fine-tuning) ## SGD Stochastic gradient descent updates weights from sampled-batch gradients. **What does it mean?:** Plain SGD subtracts the learning-rate-scaled gradient from each parameter. Momentum is an optional variant that carries part of earlier updates forward. **Why does it matter?:** Its simple update makes optimizer behavior and the effect of learning rate easy to inspect. **Important limit:** Plain SGD has little persistent state, but can need careful learning-rate tuning. It is a comparison concept, not a selectable optimizer in this app. - [Optimizer](https://www.mef-llm-studio.com/en/wiki/#optimizer) - [RMSprop](https://www.mef-llm-studio.com/en/wiki/#rmsprop) - [Adam](https://www.mef-llm-studio.com/en/wiki/#adam) - [AdamW](https://www.mef-llm-studio.com/en/wiki/#adamw) ## RMSprop An adaptive update scaled by a running average of squared gradients. **What does it mean?:** RMSprop tracks recent squared gradients per parameter and divides the gradient by their root-scale estimate, with a small stabilizing term. **Why does it matter?:** It illustrates why different parameters can receive different effective step sizes. **Important limit:** Adaptive steps do not guarantee better validation loss or prevent overfitting. RMSprop is not selectable in this app. - [Optimizer](https://www.mef-llm-studio.com/en/wiki/#optimizer) - [SGD](https://www.mef-llm-studio.com/en/wiki/#sgd) - [Adam](https://www.mef-llm-studio.com/en/wiki/#adam) - [AdamW](https://www.mef-llm-studio.com/en/wiki/#adamw) ## Adam Adaptive Moment Estimation combines moving averages of gradients and their squares. **What does it mean?:** Adam keeps a first-moment estimate of gradients and a second-moment estimate of squared gradients for each parameter, then uses them to scale updates. **Why does it matter?:** It explains the extra optimizer-state memory shown when planning a run. **Important limit:** Adam uses additional state and does not itself ensure generalization. The app uses AdamW, not a selectable Adam option. - [Optimizer](https://www.mef-llm-studio.com/en/wiki/#optimizer) - [RMSprop](https://www.mef-llm-studio.com/en/wiki/#rmsprop) - [AdamW](https://www.mef-llm-studio.com/en/wiki/#adamw) ## AdamW Adam-style updates with weight decay separated from the gradient step. **What does it mean?:** AdamW retains Adam's first and second moment estimates but applies weight decay separately from the adaptive gradient update. This app uses AdamW for pre-training and fine-tuning. **Why does it matter?:** The separation makes the role of weight decay clearer, while the two moment buffers remain part of the memory plan. **Important limit:** The app does not offer an optimizer choice; AdamW and weight decay still require validation rather than guaranteeing better outputs. - [Optimizer](https://www.mef-llm-studio.com/en/wiki/#optimizer) - [Adam](https://www.mef-llm-studio.com/en/wiki/#adam) - [Fine-Tuning](https://www.mef-llm-studio.com/en/wiki/#fine-tuning) - [SFT](https://www.mef-llm-studio.com/en/wiki/#sft) ## Fine-Tuning Further training of a pretrained model for a task or domain. **What does it mean?:** In fine-tuning, model parameters are updated with a smaller, targeted dataset. This can specialize behavior, format, or domain performance. **Why does it matter?:** It shifts recurring patterns out of the prompt and into the weights. **Important limit:** Fine-tuning guarantees neither generalization nor current facts and can shift existing capabilities. - [SFT](https://www.mef-llm-studio.com/en/wiki/#sft) - [LoRA & QLoRA](https://www.mef-llm-studio.com/en/wiki/#lora-qlora) - [PEFT](https://www.mef-llm-studio.com/en/wiki/#peft) - [DPO](https://www.mef-llm-studio.com/en/wiki/#dpo) - [RLHF](https://www.mef-llm-studio.com/en/wiki/#rlhf) ## SFT Supervised fine-tuning on demonstrated assistant responses. **What does it mean?:** SFT continues training a pretrained model on example conversations. In this app the full conversation supplies context, but only assistant response tokens contribute to the training loss. **Why does it matter?:** It teaches a base model to answer in the demonstrated format instead of merely continuing raw text. **Important limit:** SFT learns from the examples it receives; it does not prove factual accuracy, preference alignment, or safety. This app updates model weights rather than training LoRA adapters. - [Fine-Tuning](https://www.mef-llm-studio.com/en/wiki/#fine-tuning) - [DPO](https://www.mef-llm-studio.com/en/wiki/#dpo) - [PEFT](https://www.mef-llm-studio.com/en/wiki/#peft) - [AdamW](https://www.mef-llm-studio.com/en/wiki/#adamw) ## LoRA & QLoRA Low-parameter adaptation via low-rank add-on matrices. **What does it mean?:** LoRA freezes the base matrix and trains small low-rank adapters. QLoRA combines this principle with a quantized base model to reduce memory use further. **Why does it matter?:** This makes it possible to adapt large models with far fewer trainable parameters. **Important limit:** Fewer trainable parameters do not automatically mean less data need, better quality, or easier evaluation. LoRA and QLoRA are not offered in this app, which trains all weights of its own GPT checkpoints. - [PEFT](https://www.mef-llm-studio.com/en/wiki/#peft) - [Fine-Tuning](https://www.mef-llm-studio.com/en/wiki/#fine-tuning) - [Quantization](https://www.mef-llm-studio.com/en/wiki/#quantization) ## PEFT A family of parameter-efficient fine-tuning methods. **What does it mean?:** Parameter-efficient fine-tuning changes only a small part of additional or selected parameters, such as adapters, prefixes, or low-rank matrices. **Why does it matter?:** It lowers memory and compute cost for many specialized variants. **Important limit:** PEFT is a training method, not a guarantee of isolation between roles or of safe adapter combination. This app offers no PEFT method; pretraining, SFT, DPO and ORPO update all weights of app-trained GPT checkpoints. - [LoRA & QLoRA](https://www.mef-llm-studio.com/en/wiki/#lora-qlora) - [Fine-Tuning](https://www.mef-llm-studio.com/en/wiki/#fine-tuning) - [Model Compression](https://www.mef-llm-studio.com/en/wiki/#model-compression) ## RLHF Optimizing a model with feedback learned from human preferences. **What does it mean?:** Reinforcement Learning from Human Feedback uses human comparisons, a preference or reward model, and subsequent optimization to align responses with desired behavior. **Why does it matter?:** It can improve helpfulness, style, and safety behavior. **Important limit:** Preference is not truth; reward hacking and unclear reviewer standards remain possible. RLHF with a reward model is not implemented in this app; its DPO and ORPO modes learn from offline preference pairs instead. - [DPO](https://www.mef-llm-studio.com/en/wiki/#dpo) - [Fine-Tuning](https://www.mef-llm-studio.com/en/wiki/#fine-tuning) - [AI Safety](https://www.mef-llm-studio.com/en/wiki/#ai-safety) - [LLM Evaluation](https://www.mef-llm-studio.com/en/wiki/#llm-evaluation) ## DPO Direct optimization toward preferred over rejected responses. **What does it mean?:** Direct Preference Optimization trains a model directly from response pairs, without running a separate classic RL loop with its own reward model. **Why does it matter?:** It simplifies preference alignment compared to many RLHF pipelines. **Important limit:** The method inherits the quality, bias, and ontology of the preference data and proves no factual fidelity. - [SFT](https://www.mef-llm-studio.com/en/wiki/#sft) - [RLHF](https://www.mef-llm-studio.com/en/wiki/#rlhf) - [Fine-Tuning](https://www.mef-llm-studio.com/en/wiki/#fine-tuning) - [Prompt Evaluation](https://www.mef-llm-studio.com/en/wiki/#prompt-evaluation) ## Knowledge Distillation A smaller model learns from the outputs or representations of a larger teacher. **What does it mean?:** Distillation transfers a teacher model's behavior to a smaller student, for example via soft targets, logits, intermediate states, or generated training examples. **Why does it matter?:** It can produce compact specialists that run locally and cheaply. **Important limit:** The student also inherits the teacher's errors and blind spots; teacher agreement is no independent gold standard. Distillation is not a training mode in this app. - [Model Compression](https://www.mef-llm-studio.com/en/wiki/#model-compression) - [Fine-Tuning](https://www.mef-llm-studio.com/en/wiki/#fine-tuning) - [LLM Evaluation](https://www.mef-llm-studio.com/en/wiki/#llm-evaluation) ## Quantization Representing weights or activations with fewer bits. **What does it mean?:** Quantization reduces numeric precision, for example from FP32 to FP16, INT8, or lower formats. It can happen after training or during quantization-aware training. **Why does it matter?:** It lowers memory use and can speed up inference. **Important limit:** Rounding and scaling errors can create rare functional failures that an aggregate hides. - [Model Compression](https://www.mef-llm-studio.com/en/wiki/#model-compression) - [LLM Inference](https://www.mef-llm-studio.com/en/wiki/#llm-inference) - [LoRA & QLoRA](https://www.mef-llm-studio.com/en/wiki/#lora-qlora) ## Model Compression Methods to reduce model size or compute cost. **What does it mean?:** Model compression covers quantization, pruning, distillation, factorization, and other methods that make a model smaller or faster. **Why does it matter?:** It enables local execution and lower operating resources. **Important limit:** Compression is only a success if the relevant quality and safety axes are preserved. - [Quantization](https://www.mef-llm-studio.com/en/wiki/#quantization) - [Knowledge Distillation](https://www.mef-llm-studio.com/en/wiki/#knowledge-distillation) - [AI Cost Optimization](https://www.mef-llm-studio.com/en/wiki/#cost-optimization) ## LLM Inference Running a trained model on new inputs. **What does it mean?:** Inference covers preprocessing, forward passes, decoding, cache management, and output preparation of an already trained model. **Why does it matter?:** This is where real latency, memory use, parallelism, and runtime behavior arise. **Important limit:** A correctly loaded checkpoint and deterministic decoding prove no factual correctness. - [Forward Pass](https://www.mef-llm-studio.com/en/wiki/#forward-pass) - [Sampling](https://www.mef-llm-studio.com/en/wiki/#sampling) - [KV Cache](https://www.mef-llm-studio.com/en/wiki/#kv-cache) - [Speculative Decoding](https://www.mef-llm-studio.com/en/wiki/#speculative-decoding) - [LLM Serving](https://www.mef-llm-studio.com/en/wiki/#llm-serving) - [Quantization](https://www.mef-llm-studio.com/en/wiki/#quantization) ## KV Cache A cache for attention keys and values of already processed tokens. **What does it mean?:** In autoregressive decoding, the KV cache stores computed key and value states so that earlier tokens are not fully reprocessed for each new token. **Why does it matter?:** It speeds up generation greatly but can occupy a lot of memory. **Important limit:** The cache stores keys and values, not attention weights or outputs: each new query still computes fresh weights over all cached keys. With grouped-query attention it holds one entry per key/value head, not per query head. Decoding with a cache and recomputing the whole context are the same calculation on paper but not bit for bit; in bfloat16 the logits can differ noticeably. - [LLM Inference](https://www.mef-llm-studio.com/en/wiki/#llm-inference) - [Context Windows](https://www.mef-llm-studio.com/en/wiki/#context-windows) - [vLLM](https://www.mef-llm-studio.com/en/wiki/#vllm) ## Speculative Decoding A fast draft is verified in bulk by a target model. **What does it mean?:** In speculative decoding, a smaller or cheaper method produces several token proposals that the target model accepts or rejects in fewer steps. **Why does it matter?:** It can lower latency without deliberately changing the target distribution. **Important limit:** The method speeds up decoding but solves no evidence, hallucination, or authority question. - [LLM Inference](https://www.mef-llm-studio.com/en/wiki/#llm-inference) - [LLM Serving](https://www.mef-llm-studio.com/en/wiki/#llm-serving) - [vLLM](https://www.mef-llm-studio.com/en/wiki/#vllm) ## LLM Serving Providing models as robust, plannable runtime services. **What does it mean?:** LLM serving covers model loading, request queue, batching, parallelism, timeouts, streaming, scaling, and error isolation. **Why does it matter?:** It decides whether a model is reproducible and operable under real load. **Important limit:** High throughput is no proof of quality or governance; serving must not bypass domain gates. - [vLLM](https://www.mef-llm-studio.com/en/wiki/#vllm) - [LLM Inference](https://www.mef-llm-studio.com/en/wiki/#llm-inference) - [Observability](https://www.mef-llm-studio.com/en/wiki/#observability) - [Production GenAI Systems](https://www.mef-llm-studio.com/en/wiki/#production-genai) ## vLLM A specialized open-source serving stack for high LLM load. **What does it mean?:** vLLM is an inference and serving engine with memory management for the KV cache, continuous batching, and further optimized execution paths. **Why does it matter?:** It can provide high throughput for compatible models and hardware. **Important limit:** vLLM is infrastructure: it defines neither evidence lineage nor model role, tool authority, or claim validation. - [LLM Serving](https://www.mef-llm-studio.com/en/wiki/#llm-serving) - [KV Cache](https://www.mef-llm-studio.com/en/wiki/#kv-cache) - [Speculative Decoding](https://www.mef-llm-studio.com/en/wiki/#speculative-decoding) ## Model Routing A request is assigned to a suitable model or path. **What does it mean?:** Model routing selects between several models and processing paths based on task, domain, risk, cost, or availability. **Why does it matter?:** It allows specialization and prevents every request from using the most expensive monolith. **Important limit:** A wrong router can produce internally plausible results in the wrong scope; unknown must be its own exit. - [AI Agents](https://www.mef-llm-studio.com/en/wiki/#ai-agents) - [Reranking](https://www.mef-llm-studio.com/en/wiki/#reranking) - [Production GenAI Systems](https://www.mef-llm-studio.com/en/wiki/#production-genai) ## Guardrails Technical boundaries for inputs, outputs, and actions. **What does it mean?:** Guardrails check or limit formats, content, values, permissions, and actions before, during, or after a model call. **Why does it matter?:** They reduce clearly named error classes and can enforce fail-closed behavior. **Important limit:** A downstream filter cannot retroactively invent missing evidence and must not become the hidden decider. - [AI Safety](https://www.mef-llm-studio.com/en/wiki/#ai-safety) - [Structured Outputs](https://www.mef-llm-studio.com/en/wiki/#structured-outputs) - [Hallucination Detection](https://www.mef-llm-studio.com/en/wiki/#hallucination-detection) - [Tool Calling](https://www.mef-llm-studio.com/en/wiki/#tool-calling) ## AI Safety Recognizing, limiting, and demonstrating risks of AI systems. **What does it mean?:** AI safety covers technical and organizational methods against misbehavior, misuse, loss of control, unwanted side effects, and unsustainable decisions. **Why does it matter?:** Capability without limited consequences is not a sufficient goal in real systems. **Important limit:** Safety is not a single metric and not a prompt; it depends on system context, rights, and error costs. - [Guardrails](https://www.mef-llm-studio.com/en/wiki/#guardrails) - [Observability](https://www.mef-llm-studio.com/en/wiki/#observability) - [LLM Evaluation](https://www.mef-llm-studio.com/en/wiki/#llm-evaluation) - [Production GenAI Systems](https://www.mef-llm-studio.com/en/wiki/#production-genai) ## Hallucination Detection Checking whether a generated statement is unsupported or contradictory. **What does it mean?:** Hallucination detection compares outputs with sources, rules, other models, or consistency signals to flag unsupported statements. **Why does it matter?:** It can make errors visible before delivery. **Important limit:** A detector is itself error-prone; after-the-fact detection replaces no architecture that binds claims to evidence. - [RAG](https://www.mef-llm-studio.com/en/wiki/#rag) - [Guardrails](https://www.mef-llm-studio.com/en/wiki/#guardrails) - [LLM Evaluation](https://www.mef-llm-studio.com/en/wiki/#llm-evaluation) - [Structured Outputs](https://www.mef-llm-studio.com/en/wiki/#structured-outputs) ## LLM Evaluation Systematic measurement of model behavior on defined surfaces. **What does it mean?:** LLM evaluation combines datasets, metrics, baselines, human judgments, counterfactuals, and robustness tests for clearly delimited questions. **Why does it matter?:** Only a bound evaluation separates progress from impression, prompt luck, or test fitting. **Important limit:** An aggregate metric can hide functional failures, seed tails, coverage, and abstention. - [Prompt Evaluation](https://www.mef-llm-studio.com/en/wiki/#prompt-evaluation) - [Observability](https://www.mef-llm-studio.com/en/wiki/#observability) - [AI Safety](https://www.mef-llm-studio.com/en/wiki/#ai-safety) ## Prompt Evaluation Comparing prompt variants on a fixed task and sample. **What does it mean?:** Prompt evaluation measures how instructions, examples, and format requirements affect defined outputs. **Why does it matter?:** It prevents individual attractive examples from counting as a general improvement. **Important limit:** Prompt quality is only one component; provider, model, data, and decoder changes must be separated. - [Prompt Engineering](https://www.mef-llm-studio.com/en/wiki/#prompt-engineering) - [LLM Evaluation](https://www.mef-llm-studio.com/en/wiki/#llm-evaluation) - [Few-Shot Learning](https://www.mef-llm-studio.com/en/wiki/#few-shot-learning) ## Observability Understanding from runtime data what a system did and where it failed. **What does it mean?:** Observability connects structured logs, traces, metrics, and artifact versions across components and requests. **Why does it matter?:** It makes errors, cost, latency, and drift investigable without guessing the run after the fact. **Important limit:** A log shows an event, but not automatically its domain justification or causal meaning. - [Production GenAI Systems](https://www.mef-llm-studio.com/en/wiki/#production-genai) - [LLM Evaluation](https://www.mef-llm-studio.com/en/wiki/#llm-evaluation) - [AI Safety](https://www.mef-llm-studio.com/en/wiki/#ai-safety) ## AI Cost Optimization Quality-bound reduction of compute, memory, and operating cost. **What does it mean?:** Cost optimization uses routing, smaller models, batching, caching, quantization, limited payloads, and better utilization. **Why does it matter?:** It decides whether a system is economically scalable. **Important limit:** A smaller or cheaper output is no success if evidence coverage, tail safety, or task quality drops. - [Model Routing](https://www.mef-llm-studio.com/en/wiki/#model-routing) - [Quantization](https://www.mef-llm-studio.com/en/wiki/#quantization) - [Model Compression](https://www.mef-llm-studio.com/en/wiki/#model-compression) - [LLM Serving](https://www.mef-llm-studio.com/en/wiki/#llm-serving) ## Multimodal AI Joint processing of several modalities such as text, image, and audio. **What does it mean?:** Multimodal systems connect representations and tasks across text, image, audio, video, or sensor data. **Why does it matter?:** Many real decisions do not rest on text alone. **Important limit:** Each modality has its own measurement errors, timelines, and provenance; joint encoding does not remove these boundaries. - [Vision-Language Models](https://www.mef-llm-studio.com/en/wiki/#vision-language-models) - [Production GenAI Systems](https://www.mef-llm-studio.com/en/wiki/#production-genai) - [Embeddings](https://www.mef-llm-studio.com/en/wiki/#embeddings) ## Vision-Language Models Models for joint image and language tasks. **What does it mean?:** Vision-language models encode visual inputs and language in a shared or coupled model space, for example for captioning, visual questions, or document analysis. **Why does it matter?:** They open up screenshots, diagrams, scans, and real scenes for language workflows. **Important limit:** An image description is a perception proposal; it is not automatically a provable source record. - [Multimodal AI](https://www.mef-llm-studio.com/en/wiki/#multimodal-ai) - [Structured Outputs](https://www.mef-llm-studio.com/en/wiki/#structured-outputs) - [Hallucination Detection](https://www.mef-llm-studio.com/en/wiki/#hallucination-detection) - [Visual Encoder](https://www.mef-llm-studio.com/en/wiki/#visual-encoder) - [Encoder-Free Multimodal Model](https://www.mef-llm-studio.com/en/wiki/#encoder-free-multimodal) ## AI Coding Agents Agents that read, change, and test code and operate development tools. **What does it mean?:** Coding agents connect language models with repository search, editors, shell, build, tests, and version control. **Why does it matter?:** They can work on multi-step software tasks over real artifacts. **Important limit:** Code generation is not code correctness; rights, scope, tests, review, and reversible changes remain necessary. - [AI Agents](https://www.mef-llm-studio.com/en/wiki/#ai-agents) - [Tool Calling](https://www.mef-llm-studio.com/en/wiki/#tool-calling) - [MCP](https://www.mef-llm-studio.com/en/wiki/#mcp) - [Production GenAI Systems](https://www.mef-llm-studio.com/en/wiki/#production-genai) ## Production GenAI Systems The whole operable service around models, data, tools, and governance. **What does it mean?:** A production GenAI system includes not only a model but data and model versions, APIs, identities, queues, monitoring, tests, rollback, cost, safety boundaries, and human processes. **Why does it matter?:** Most real errors arise at interfaces and operating states, not only in the forward pass. **Important limit:** A demo with good answers is not yet a production-ready system. - [Observability](https://www.mef-llm-studio.com/en/wiki/#observability) - [LLM Serving](https://www.mef-llm-studio.com/en/wiki/#llm-serving) - [AI Safety](https://www.mef-llm-studio.com/en/wiki/#ai-safety) - [Agentic Workflows](https://www.mef-llm-studio.com/en/wiki/#agentic-workflows) ## JSON A text format for structured values such as objects and arrays. **What does it mean?:** JSON represents objects, arrays, strings, numbers, booleans, and null with a small, language-independent syntax. **Why does it matter?:** Model examples, configuration, and tool responses often use JSON because software can parse its structure. **Important limit:** Valid JSON syntax does not prove that a record has the fields or meaning required by a training task. - [JSONL](https://www.mef-llm-studio.com/en/wiki/#jsonl) - [Structured Outputs](https://www.mef-llm-studio.com/en/wiki/#structured-outputs) - [Dataset](https://www.mef-llm-studio.com/en/wiki/#dataset) ## JSONL One complete JSON value per line, commonly one example per line. **What does it mean?:** A JSON Lines file is UTF-8 text in which each non-empty line is independently valid JSON. A .jsonl extension is customary. **Why does it matter?:** Examples can be read and validated one record at a time without loading one large JSON array. **Important limit:** A syntactically valid line can still be the wrong kind of training example; schema and content checks remain necessary. - [JSON](https://www.mef-llm-studio.com/en/wiki/#json) - [Dataset](https://www.mef-llm-studio.com/en/wiki/#dataset) - [SFT](https://www.mef-llm-studio.com/en/wiki/#sft) - [DPO](https://www.mef-llm-studio.com/en/wiki/#dpo) ## Checkpoint A saved model state from a particular point in a run. **What does it mean?:** A checkpoint stores model parameters and identifying metadata for a training step. In this app a resumable checkpoint also holds the optimizer moments, the step, the data position and the random state (the learning rate is recomputed from the step); weights-only 'best' checkpoints can be used but not resumed. **Why does it matter?:** It lets you inspect or compare earlier model states instead of treating the last step as the only result. **Important limit:** A checkpoint is not by itself proof of model quality and is not necessarily a portable inference package. - [Training](https://www.mef-llm-studio.com/en/wiki/#training) - [Dataset](https://www.mef-llm-studio.com/en/wiki/#dataset) - [Validation Loss](https://www.mef-llm-studio.com/en/wiki/#validation-loss) - [LLM Inference](https://www.mef-llm-studio.com/en/wiki/#llm-inference) ## Dataset A collection of examples prepared for training or evaluation. **What does it mean?:** A dataset selects and organizes examples, their expected targets, and often separate training and validation portions. **Why does it matter?:** Its source, coverage, and split determine what a model can learn and what a reported loss actually measures. **Important limit:** A large or well-formatted dataset is not automatically representative, clean, or free of overlap between splits. - [JSONL](https://www.mef-llm-studio.com/en/wiki/#jsonl) - [Tokenization](https://www.mef-llm-studio.com/en/wiki/#tokenization) - [Training](https://www.mef-llm-studio.com/en/wiki/#training) - [Validation Loss](https://www.mef-llm-studio.com/en/wiki/#validation-loss) ## GPT-2 An OpenAI language model and a widely reused byte-level BPE tokenizer. **What does it mean?:** GPT-2 is a Transformer language model introduced in 2019. Its released tokenizer maps UTF-8 bytes to symbols and applies learned byte-pair merges. **Why does it matter?:** Its published vocabulary gives this app a fixed comparison point beside tokenizers trained on your own text. **Important limit:** Selecting the GPT-2 tokenizer does not turn a different model into GPT-2 or transfer GPT-2's learned weights. - [Byte-Level](https://www.mef-llm-studio.com/en/wiki/#byte-level) - [Tokenization](https://www.mef-llm-studio.com/en/wiki/#tokenization) - [Transformers](https://www.mef-llm-studio.com/en/wiki/#transformers) ## Training Repeatedly adjusting model parameters from examples and a loss signal. **What does it mean?:** In each training step the model predicts, a loss measures error, gradients are computed, and an optimizer updates parameters. **Why does it matter?:** Watching steps and loss connects model behavior to the data and objective that shaped it. **Important limit:** A falling training loss does not guarantee improvement on unseen examples or useful answers. - [Dataset](https://www.mef-llm-studio.com/en/wiki/#dataset) - [Loss](https://www.mef-llm-studio.com/en/wiki/#loss) - [Validation Loss](https://www.mef-llm-studio.com/en/wiki/#validation-loss) - [Optimizer](https://www.mef-llm-studio.com/en/wiki/#optimizer) - [Checkpoint](https://www.mef-llm-studio.com/en/wiki/#checkpoint) ## Byte-Level A tokenizer representation built from bytes rather than a fixed list of characters. **What does it mean?:** Byte-level tokenization begins with the bytes of encoded text; byte-level BPE can then merge frequent byte sequences into larger tokens. **Why does it matter?:** A byte base can represent text outside a small character vocabulary without an unknown-character token. **Important limit:** Byte-level BPE does not mean every final token is exactly one byte or one visible character. - [GPT-2](https://www.mef-llm-studio.com/en/wiki/#gpt-2) - [Tokenization](https://www.mef-llm-studio.com/en/wiki/#tokenization) ## Loss A number that measures error under a chosen training objective. **What does it mean?:** A loss function compares predictions with targets. During training, its gradients guide parameter updates; next-token models commonly use cross-entropy loss. **Why does it matter?:** The loss curve shows whether the model is fitting the task it was given. **Important limit:** Lower training loss is not a direct measure of truth, usefulness, or performance on new data. - [Training](https://www.mef-llm-studio.com/en/wiki/#training) - [Validation Loss](https://www.mef-llm-studio.com/en/wiki/#validation-loss) - [Optimizer](https://www.mef-llm-studio.com/en/wiki/#optimizer) ## Validation Loss Loss measured on held-out examples without updating model weights. **What does it mean?:** Validation loss applies the training objective to a separate validation split while leaving parameters unchanged. It helps compare checkpoints and detect a gap from training loss. **Why does it matter?:** It gives a limited check of generalization beyond the examples used for updates. **Important limit:** Repeated choices based on the same validation split can overfit that split; a low value is not a guarantee of good answers. Values are only comparable between runs that share a tokenizer and a validation split. - [Loss](https://www.mef-llm-studio.com/en/wiki/#loss) - [Training](https://www.mef-llm-studio.com/en/wiki/#training) - [Dataset](https://www.mef-llm-studio.com/en/wiki/#dataset) - [Checkpoint](https://www.mef-llm-studio.com/en/wiki/#checkpoint) ## Forward Pass One run of the model from the input tokens to scores for the next token. **What does it mean?:** In a forward pass, the token IDs of the context go through the embedding and then through every transformer block in order. For generation, the output head reads the last position and scores every vocabulary token; each new token needs another pass. With a KV cache, later passes run only the newest token and read the earlier keys and values from the cache. **Why does it matter?:** It is the unit of work behind every predicted token. The Brain Explorer draws one pass as token lanes running through the blocks. **Important limit:** A pass computes scores but does not choose a token; sampling does that afterwards. No weights change during inference. - [LLM Inference](https://www.mef-llm-studio.com/en/wiki/#llm-inference) - [Transformer Block](https://www.mef-llm-studio.com/en/wiki/#transformer-block) - [Output Head](https://www.mef-llm-studio.com/en/wiki/#output-head) - [Sampling](https://www.mef-llm-studio.com/en/wiki/#sampling) ## Transformer Block One repeated layer of a transformer: attention, then an MLP, each added to the residual stream. **What does it mean?:** A block normalizes its input, lets its attention heads read from other positions and adds their result to the residual stream. It then normalizes again, applies the MLP at every position and adds that result too. A model stacks many blocks with the same structure but their own weights and runs them one after another. **Why does it matter?:** Blocks give the model its depth: each one can refine the state that every later block reads. **Important limit:** A block is not a head: the heads of multi-head attention work side by side inside one block, while blocks follow one another. The number of blocks alone says nothing about quality. - [Transformers](https://www.mef-llm-studio.com/en/wiki/#transformers) - [Attention Head](https://www.mef-llm-studio.com/en/wiki/#attention-head) - [Multi-Head Attention](https://www.mef-llm-studio.com/en/wiki/#multi-head-attention) - [MLP (Feed-Forward)](https://www.mef-llm-studio.com/en/wiki/#mlp) - [Residual Stream](https://www.mef-llm-studio.com/en/wiki/#residual-stream) - [LayerNorm](https://www.mef-llm-studio.com/en/wiki/#layer-norm) ## Attention Head One of several attention computations that run side by side inside a transformer block. **What does it mean?:** Each head has its own query projection onto a slice of the vector width (head_dim dimensions). In classic multi-head attention (GPT-2, your own checkpoints) every head also has its own key and value projections. With grouped-query attention (Llama, Qwen2) there are fewer key/value heads than query heads: several query heads share one key and value. Each query head still computes its own attention weights, so heads can weigh the same positions differently; their outputs are joined and projected back into the residual stream. **Why does it matter?:** Heads let one block follow several relationships at once. The Brain Explorer shows the weights of one recorded head; for an external recording, the Head Finder compares all heads of a recorded round. **Important limit:** Heads are not blocks: all heads of a block work in parallel, blocks work in sequence. One head's weights show only part of what the whole attention layer adds. - [Multi-Head Attention](https://www.mef-llm-studio.com/en/wiki/#multi-head-attention) - [Attention Mechanism](https://www.mef-llm-studio.com/en/wiki/#attention) - [Transformer Block](https://www.mef-llm-studio.com/en/wiki/#transformer-block) - [Causal Mask](https://www.mef-llm-studio.com/en/wiki/#causal-mask) ## Causal Mask The rule that a position may attend only to itself and to earlier positions. **What does it mean?:** Before the softmax, the attention scores for later positions are set to minus infinity, so their weights become exactly zero. In an attention matrix, this leaves the area above the diagonal empty. **Why does it matter?:** It lets a model learn next-token prediction at all positions at once without seeing the tokens it has to predict. **Important limit:** A masked cell is not a learned weight of zero; that position cannot be reached at all. The mask says nothing about which earlier positions matter. - [Attention Mechanism](https://www.mef-llm-studio.com/en/wiki/#attention) - [Attention Head](https://www.mef-llm-studio.com/en/wiki/#attention-head) - [Softmax](https://www.mef-llm-studio.com/en/wiki/#softmax) - [Positional Encoding](https://www.mef-llm-studio.com/en/wiki/#positional-encoding) ## MLP (Feed-Forward) The part of a transformer block that processes each position on its own. **What does it mean?:** The MLP widens a position's vector, applies a non-linear function and projects it back to the original width; the result is added to the residual stream. In the GPT-2-style models trained in this app, it widens by a factor of four and uses GELU. Llama and Qwen2 use a gated MLP (SwiGLU): two parallel up-projections, one passed through SiLU, are multiplied elementwise and then projected down; the widened size is intermediate_size. **Why does it matter?:** Attention moves information between positions; the MLP transforms it within each position and holds a large share of the parameters. **Important limit:** The MLP does not read from other positions. Within one block, every position uses the same MLP weights. - [Transformer Block](https://www.mef-llm-studio.com/en/wiki/#transformer-block) - [Residual Stream](https://www.mef-llm-studio.com/en/wiki/#residual-stream) - [Attention Mechanism](https://www.mef-llm-studio.com/en/wiki/#attention) - [Transformers](https://www.mef-llm-studio.com/en/wiki/#transformers) ## Residual Stream The running vector of each position that every block reads from and adds to. **What does it mean?:** In GPT-2-style models, each position starts with its token embedding plus its position embedding. Attention and the MLP compute contributions that are added to this vector instead of replacing it (residual connections), so the stream carries information through all blocks to the final normalization (LayerNorm, or RMSNorm in Llama/Qwen2). **Why does it matter?:** The additions make deep models trainable. They are why the Brain Explorer draws a grey line with plus nodes through all blocks. **Important limit:** The stream is a vector with many dimensions; the few components a visualization shows are only a slice, and a single number rarely has a fixed meaning. - [Transformer Block](https://www.mef-llm-studio.com/en/wiki/#transformer-block) - [MLP (Feed-Forward)](https://www.mef-llm-studio.com/en/wiki/#mlp) - [Attention Mechanism](https://www.mef-llm-studio.com/en/wiki/#attention) - [LayerNorm](https://www.mef-llm-studio.com/en/wiki/#layer-norm) - [Embeddings](https://www.mef-llm-studio.com/en/wiki/#embeddings) ## LayerNorm Rescales one position's vector to a standard mean and spread before the next computation. **What does it mean?:** Layer normalization subtracts the mean of a vector's components, divides by their standard deviation and then applies a learned scale and shift. Pre-norm blocks apply it before attention and before the MLP; a final LayerNorm prepares the last state for the output head. Llama and Qwen2 use RMSNorm in the same places, which only divides by the root mean square and applies a learned scale, with no mean subtraction and no shift. **Why does it matter?:** Stable value ranges make training reliable, even as the residual stream grows from block to block. **Important limit:** LayerNorm works within the vector of a single position and does not mix positions. Because of the learned scale and shift, its output is not always exactly mean 0 and variance 1. - [Transformer Block](https://www.mef-llm-studio.com/en/wiki/#transformer-block) - [Residual Stream](https://www.mef-llm-studio.com/en/wiki/#residual-stream) - [Output Head](https://www.mef-llm-studio.com/en/wiki/#output-head) ## Output Head The final projection that turns the last state into one score per vocabulary token. **What does it mean?:** After the final normalization (LayerNorm, or RMSNorm in Llama/Qwen2), the output head multiplies a position's state by a matrix with one row per vocabulary token; during generation, only the last position is read. The result is the logits. Many GPT-style models tie this matrix to the token embedding; this app offers that as a model option. **Why does it matter?:** This is where the model's internal vector becomes a prediction about concrete tokens. **Important limit:** Despite its name, the output head is not an attention head. It scores tokens but does not choose one; sampling does that. - [Logits](https://www.mef-llm-studio.com/en/wiki/#logits) - [Softmax](https://www.mef-llm-studio.com/en/wiki/#softmax) - [LayerNorm](https://www.mef-llm-studio.com/en/wiki/#layer-norm) - [Embeddings](https://www.mef-llm-studio.com/en/wiki/#embeddings) - [Attention Head](https://www.mef-llm-studio.com/en/wiki/#attention-head) ## Logits Raw, unnormalized scores for every possible next token. **What does it mean?:** The output head produces one real number per vocabulary token. Higher means more likely, but the values can be negative and do not add up to a fixed total; softmax turns them into probabilities. **Why does it matter?:** Temperature and filters act on the logits, so they are the starting point of every sampling decision. **Important limit:** A logit is not a probability, and differences between logits matter more than their absolute values. - [Output Head](https://www.mef-llm-studio.com/en/wiki/#output-head) - [Softmax](https://www.mef-llm-studio.com/en/wiki/#softmax) - [Temperature](https://www.mef-llm-studio.com/en/wiki/#temperature) - [Sampling](https://www.mef-llm-studio.com/en/wiki/#sampling) ## Softmax Turns a list of scores into probabilities that are positive and sum to one. **What does it mean?:** Softmax applies the exponential function to each score and divides by the sum of all exponentials. Larger scores get larger shares, and adding the same constant to every score changes nothing. Transformers use it for attention weights and for next-token probabilities. **Why does it matter?:** It makes scores comparable as shares, which is what probability bars and attention arcs show. **Important limit:** Softmax probabilities describe the model's distribution, not the chance that a statement is true. - [Logits](https://www.mef-llm-studio.com/en/wiki/#logits) - [Attention Mechanism](https://www.mef-llm-studio.com/en/wiki/#attention) - [Temperature](https://www.mef-llm-studio.com/en/wiki/#temperature) - [Causal Mask](https://www.mef-llm-studio.com/en/wiki/#causal-mask) ## Sampling Choosing the next token from the model's probability distribution. **What does it mean?:** After temperature and optional filters such as top-k and top-p, the next token is drawn at random in proportion to the remaining probabilities. Greedy decoding instead always takes the most likely token, which corresponds to temperature 0. **Why does it matter?:** The same model and prompt can produce different continuations; the sampling settings control how varied and how risky they are. **Important limit:** Sampling changes neither what the model has learned nor its scores. A drawn token is not automatically the best or a correct one. - [Temperature](https://www.mef-llm-studio.com/en/wiki/#temperature) - [Top-k](https://www.mef-llm-studio.com/en/wiki/#top-k) - [Top-p](https://www.mef-llm-studio.com/en/wiki/#top-p) - [Seed](https://www.mef-llm-studio.com/en/wiki/#seed) - [Softmax](https://www.mef-llm-studio.com/en/wiki/#softmax) - [Logits](https://www.mef-llm-studio.com/en/wiki/#logits) ## Temperature Divides the logits before sampling: below 1 the distribution gets peakier, above 1 flatter. **What does it mean?:** The logits are divided by the temperature before the softmax. Below 1 the distribution gets peakier and likely tokens become even more likely; above 1 it gets flatter and unlikely tokens gain probability; temperature 0 always takes the most likely token. **Why does it matter?:** It is the main control between predictable and varied text. **Important limit:** Temperature changes the selection distribution, not the model's scores or its knowledge. - [Sampling](https://www.mef-llm-studio.com/en/wiki/#sampling) - [Logits](https://www.mef-llm-studio.com/en/wiki/#logits) - [Softmax](https://www.mef-llm-studio.com/en/wiki/#softmax) - [Top-k](https://www.mef-llm-studio.com/en/wiki/#top-k) - [Top-p](https://www.mef-llm-studio.com/en/wiki/#top-p) ## Top-k A sampling filter that keeps only the k most likely tokens. **What does it mean?:** All tokens outside the k highest scores are removed before drawing, and the remaining probabilities are normalized again. In this app, k = 0 switches the filter off; it is applied before top-p. **Why does it matter?:** It cuts off the long tail of unlikely tokens that can derail a continuation. **Important limit:** A fixed k ignores how the probability is spread: sometimes k tokens are too many, sometimes too few. - [Top-p](https://www.mef-llm-studio.com/en/wiki/#top-p) - [Sampling](https://www.mef-llm-studio.com/en/wiki/#sampling) - [Temperature](https://www.mef-llm-studio.com/en/wiki/#temperature) ## Top-p A sampling filter that keeps the smallest set of tokens that together reach probability p. **What does it mean?:** Tokens are sorted by probability and kept until their cumulative probability reaches p; the rest are removed before drawing. The most likely token always stays, and p = 1 keeps every token. **Why does it matter?:** Unlike top-k, it adapts to the distribution: few candidates when the model is confident, more when it is unsure. **Important limit:** Top-p limits the probability mass that is drawn from, not the quality of the continuation. - [Top-k](https://www.mef-llm-studio.com/en/wiki/#top-k) - [Sampling](https://www.mef-llm-studio.com/en/wiki/#sampling) - [Temperature](https://www.mef-llm-studio.com/en/wiki/#temperature) - [Softmax](https://www.mef-llm-studio.com/en/wiki/#softmax) ## Seed The start value of a pseudo-random number generator. **What does it mean?:** The same seed with the same code, model, settings and hardware repeats the same random choices, for example the tokens drawn during sampling or the order of examples in training. **Why does it matter?:** It makes experiments repeatable and comparable. **Important limit:** A seed does not make a result correct, and equal seeds with different inputs or settings lead to different draws. Some GPU operations are not fully deterministic even with a fixed seed. - [Sampling](https://www.mef-llm-studio.com/en/wiki/#sampling) - [Temperature](https://www.mef-llm-studio.com/en/wiki/#temperature) - [Training](https://www.mef-llm-studio.com/en/wiki/#training) ## Context The tokens a model actually processes in one pass: the prompt plus everything generated so far. **What does it mean?:** During generation the context grows by one token per round: first the input tokens, then the tokens chosen in earlier rounds. It can grow up to the context window; beyond that the app's own models keep only the most recent block_size tokens, and models with a KV cache compute only the new token and reuse the cached keys and values for the rest. **Why does it matter?:** The next token depends only on the context: if one token in it changes, every later lane can change. **Important limit:** The context is not the context window. A longer context is not automatically used better, and text beyond the context window never reaches the model. - [Context Windows](https://www.mef-llm-studio.com/en/wiki/#context-windows) - [Tokenization](https://www.mef-llm-studio.com/en/wiki/#tokenization) - [Forward Pass](https://www.mef-llm-studio.com/en/wiki/#forward-pass) - [KV Cache](https://www.mef-llm-studio.com/en/wiki/#kv-cache) ## Token Embedding The learned vector a language model looks up for each token ID before the first block. **What does it mean?:** The embedding table holds one row of n_embd numbers for every vocabulary token. Each token ID in the context selects its row; in GPT-2-style models the vector of its position is added, and the sum enters the first block. **Why does it matter?:** It turns discrete IDs into numbers the blocks can compute with; tokens used in similar ways can end up with similar vectors. **Important limit:** The vector is learned and has no fixed dictionary meaning; on its own it knows nothing about the token's position or neighbours. - [Embeddings](https://www.mef-llm-studio.com/en/wiki/#embeddings) - [Tokenization](https://www.mef-llm-studio.com/en/wiki/#tokenization) - [Positional Encoding](https://www.mef-llm-studio.com/en/wiki/#positional-encoding) - [Residual Stream](https://www.mef-llm-studio.com/en/wiki/#residual-stream) ## Diffusion Model A generator that learns to reverse noise corruption through repeated updates. **What does it mean?:** The Vision learning model predicts added noise from a noisy image, a diffusion timestep and a caption. Generation starts from noise and uses the learned predictions in a sequence of scheduler updates. **Why does it matter?:** Explains why image generation has intermediate states across the whole image. **Important limit:** A denoising update changes the sample, while a training update changes the weights. - [Noise Prediction](https://www.mef-llm-studio.com/en/wiki/#noise-prediction) - [Noise Schedule](https://www.mef-llm-studio.com/en/wiki/#noise-schedule) - [Latent Diffusion](https://www.mef-llm-studio.com/en/wiki/#latent-diffusion) ## Diffusion Timestep An index identifying the noise level supplied to the denoising model. **What does it mean?:** The small Vision model receives a timestep alongside the noisy image. During training each example can use a different timestep; generation walks through its diffusion schedule. **Why does it matter?:** Separates noise level from the number of optimizer updates. **Important limit:** A timestep is not elapsed wall-clock time or the training-step counter. - [Noise Schedule](https://www.mef-llm-studio.com/en/wiki/#noise-schedule) - [Diffusion Model](https://www.mef-llm-studio.com/en/wiki/#diffusion-model) ## Noise Schedule The sequence of noise strengths used by a diffusion process. **What does it mean?:** The Vision learning model stores noise coefficients for its diffusion timesteps. Training combines an image with noise using those coefficients; generation uses the corresponding reverse update. **Why does it matter?:** Connects a timestep to the amount of corruption in the input. **Important limit:** A noise schedule differs from a learning-rate schedule, which controls weight updates. - [Diffusion Timestep](https://www.mef-llm-studio.com/en/wiki/#diffusion-timestep) - [Noise Prediction](https://www.mef-llm-studio.com/en/wiki/#noise-prediction) ## Image Caption A text description paired with an image for conditioning a model. **What does it mean?:** Vision pairs every training image with a caption of its visible content. The small model encodes UTF-8 bytes and averages learned byte embeddings to condition noise prediction. **Why does it matter?:** Consistent descriptions help connect visual variation to text. **Important limit:** This small byte encoder differs from the pretrained starter's text encoder. - [Dataset](https://www.mef-llm-studio.com/en/wiki/#dataset) - [Tokenization](https://www.mef-llm-studio.com/en/wiki/#tokenization) - [Diffusion Model](https://www.mef-llm-studio.com/en/wiki/#diffusion-model) ## Noise Prediction Estimating the noise component in a corrupted image or latent state. **What does it mean?:** In the small Vision trainer, random noise is added to an image. The model predicts that noise from the corrupted image, caption and timestep; MSE compares the prediction to the known noise. **Why does it matter?:** The training target is available automatically because the code creates it. **Important limit:** The prediction is noise, not a finished image or a semantic object label. - [Diffusion Model](https://www.mef-llm-studio.com/en/wiki/#diffusion-model) - [Mean Squared Error](https://www.mef-llm-studio.com/en/wiki/#mse) - [Diffusion Timestep](https://www.mef-llm-studio.com/en/wiki/#diffusion-timestep) ## Mean Squared Error The average squared difference between predicted and target values. **What does it mean?:** MSE squares each prediction error and averages the results. Vision applies it to predicted and added noise; batch training loss and held-out validation loss use this objective. **Why does it matter?:** Provides a scalar objective from many numerical prediction errors. **Important limit:** Its scale depends on the target representation; it is not a perceptual image-quality score. - [Loss](https://www.mef-llm-studio.com/en/wiki/#loss) - [Noise Prediction](https://www.mef-llm-studio.com/en/wiki/#noise-prediction) - [Validation Loss](https://www.mef-llm-studio.com/en/wiki/#validation-loss) ## Latent Diffusion Diffusion performed in a compressed image representation rather than directly in RGB pixels. **What does it mean?:** The pretrained Vision starter updates a latent state. Its VAE decoder turns that state into image previews. The app's own small trainer instead updates pixel-space states. **Why does it matter?:** Explains why a visible preview differs from the state the denoiser processes. **Important limit:** Latent components are not RGB pixels or individually named objects. - [Diffusion Model](https://www.mef-llm-studio.com/en/wiki/#diffusion-model) - [Variational Autoencoder](https://www.mef-llm-studio.com/en/wiki/#vae) - [Noise Prediction](https://www.mef-llm-studio.com/en/wiki/#noise-prediction) ## Variational Autoencoder An encoder-decoder model that learns a probabilistic compressed representation. **What does it mean?:** A VAE encodes images into a latent distribution and decodes latent representations back into images. Vision uses the starter model's decoder to display recorded latent states. **Why does it matter?:** Connects compressed diffusion states to the pictures users can inspect. **Important limit:** Decoding can lose or change details; the tiny pixel-space trainer has no VAE. - [Latent Diffusion](https://www.mef-llm-studio.com/en/wiki/#latent-diffusion) - [Embeddings](https://www.mef-llm-studio.com/en/wiki/#embeddings) ## U-Net A downsampling and upsampling network with skip connections between matching resolutions. **What does it mean?:** U-Net-style denoisers combine broader features from reduced-resolution layers with finer features passed through skip connections. Vision's small denoiser uses a compact down-and-up convolutional path with a skip connection. **Why does it matter?:** Explains how image features at different spatial scales can be combined. **Important limit:** A U-Net need not include attention, and its stages are not Transformer blocks. - [Diffusion Model](https://www.mef-llm-studio.com/en/wiki/#diffusion-model) - [Noise Prediction](https://www.mef-llm-studio.com/en/wiki/#noise-prediction) ## Classifier-Free Guidance Combining conditioned and unconditioned predictions to adjust prompt influence. **What does it mean?:** The pretrained Vision path uses a guidance scale to combine denoiser predictions with and without text conditioning. The small pixel-space model has no classifier-free guidance control. **Why does it matter?:** Connects the guidance setting to text conditioning during generation. **Important limit:** Higher guidance does not necessarily improve quality; it is not sampling temperature. - [Latent Diffusion](https://www.mef-llm-studio.com/en/wiki/#latent-diffusion) - [Image Caption](https://www.mef-llm-studio.com/en/wiki/#image-caption) - [Noise Prediction](https://www.mef-llm-studio.com/en/wiki/#noise-prediction) ## Cross-Attention Attention in which image positions read the text encoder's token vectors. **What does it mean?:** In the Stable Diffusion U-Net, cross-attention layers let every spatial position of the latent image weigh the prompt tokens: queries come from image features, keys and values from the text encoder's output. The Vision Explorer shows the weights of one recorded cross-attention layer per denoising step. **Why does it matter?:** Shows where in the image the prompt can steer the prediction during denoising. **Important limit:** The weights show what one layer reads from the prompt, not why something is drawn; other layers and heads also contribute. - [Attention Mechanism](https://www.mef-llm-studio.com/en/wiki/#attention) - [U-Net](https://www.mef-llm-studio.com/en/wiki/#u-net) - [Text Encoder (CLIP)](https://www.mef-llm-studio.com/en/wiki/#text-encoder) - [Classifier-Free Guidance](https://www.mef-llm-studio.com/en/wiki/#classifier-free-guidance) ## Scheduler (Sampler) The rule that turns each step's predicted noise into the next, less noisy state. **What does it mean?:** A diffusion scheduler sets the sequence of timesteps and, at every step, combines the current noisy state with the model's noise prediction into the next state. The pretrained Vision models use PNDM; each recorded step shows the state before, the predicted noise and the change. **Why does it matter?:** Explains why the same model can give different images with another scheduler or step count. **Important limit:** A scheduler has no trained weights of its own and does not decide what is drawn. - [Noise Schedule](https://www.mef-llm-studio.com/en/wiki/#noise-schedule) - [Diffusion Timestep](https://www.mef-llm-studio.com/en/wiki/#diffusion-timestep) - [Noise Prediction](https://www.mef-llm-studio.com/en/wiki/#noise-prediction) - [Diffusion Model](https://www.mef-llm-studio.com/en/wiki/#diffusion-model) ## Text Encoder (CLIP) The network that turns the prompt's tokens into vectors the image denoiser reads. **What does it mean?:** Stable Diffusion splits the prompt into at most 77 tokens, pads the rest and runs them through a CLIP text encoder. Its output vectors are the keys and values of the U-Net's cross-attention at every denoising step. **Why does it matter?:** Connects the typed prompt to what the denoiser can read while it removes noise. **Important limit:** The text encoder stays unchanged during image generation and does not produce text. - [Tokenization](https://www.mef-llm-studio.com/en/wiki/#tokenization) - [Cross-Attention](https://www.mef-llm-studio.com/en/wiki/#cross-attention) - [Latent Diffusion](https://www.mef-llm-studio.com/en/wiki/#latent-diffusion) - [Classifier-Free Guidance](https://www.mef-llm-studio.com/en/wiki/#classifier-free-guidance) ## Visual Encoder A pretrained image network that turns a picture into tokens a language model can read. **What does it mean?:** In encoder-based vision-language models, a vision transformer such as CLIP or SigLIP cuts the image into patches, lets them attend to each other in both directions and passes the result through an adapter and a projector into the language model as visual tokens. It brings a visual prior learned from large collections of image-text pairs. **Why does it matter?:** The language model can use images from the first training step instead of learning to see from raw pixels. **Important limit:** This is the image side of models that understand images. In Stable Diffusion, the CLIP text encoder reads the prompt, not an image. - [Vision-Language Models](https://www.mef-llm-studio.com/en/wiki/#vision-language-models) - [Image Patch Token](https://www.mef-llm-studio.com/en/wiki/#image-patch-token) - [Encoder-Free Multimodal Model](https://www.mef-llm-studio.com/en/wiki/#encoder-free-multimodal) - [Text Encoder (CLIP)](https://www.mef-llm-studio.com/en/wiki/#text-encoder) - [Attention Mechanism](https://www.mef-llm-studio.com/en/wiki/#attention) ## Encoder-Free Multimodal Model A language model that reads image patches directly, without a separate pretrained image network. **What does it mean?:** Encoder-free models project raw image patches straight into the language model, which has to learn visual features itself. In a controlled comparison (Chen et al., 2026) they reached a higher multimodal loss than encoder-based models at the measured sizes, but their loss fell faster with more compute; their early layers took over the work a visual encoder would do. **Why does it matter?:** Shows a different design: one network for image and text instead of an image encoder plus a language model. **Important limit:** The predicted catch-up at about 10^22 training FLOPs is an extrapolation of fitted scaling laws, far beyond anything this app trains. - [Visual Encoder](https://www.mef-llm-studio.com/en/wiki/#visual-encoder) - [Image Patch Token](https://www.mef-llm-studio.com/en/wiki/#image-patch-token) - [Scaling Laws](https://www.mef-llm-studio.com/en/wiki/#scaling-laws) - [Vision-Language Models](https://www.mef-llm-studio.com/en/wiki/#vision-language-models) - [Transformer Block](https://www.mef-llm-studio.com/en/wiki/#transformer-block) ## Image Patch Token A small square of an image turned into one vector, so a transformer can treat the image as a sequence. **What does it mean?:** The image is cut into fixed-size patches, for example 16×16 pixels; each patch is flattened and projected to the model width, giving one token per patch. Within one image these tokens are often allowed to attend to each other in both directions, unlike text tokens under a causal mask. **Why does it matter?:** Lets images use the same token and attention machinery as text. **Important limit:** A patch token holds pixels, not a word; it only means what the network learns from it. - [Tokenization](https://www.mef-llm-studio.com/en/wiki/#tokenization) - [Token Embedding](https://www.mef-llm-studio.com/en/wiki/#token-embedding) - [Causal Mask](https://www.mef-llm-studio.com/en/wiki/#causal-mask) - [Visual Encoder](https://www.mef-llm-studio.com/en/wiki/#visual-encoder) - [Encoder-Free Multimodal Model](https://www.mef-llm-studio.com/en/wiki/#encoder-free-multimodal) ## Scaling Laws Fitted rules for how loss falls as model size, training data and compute grow. **What does it mean?:** Scaling laws fit validation loss as a power law of training compute, model size and number of tokens, using many smaller runs. The Chinchilla method trains models of different sizes at the same compute budget (IsoFLOP profiles) to find the compute-optimal split between parameters and tokens. **Why does it matter?:** Lets researchers predict large runs from small ones and compare designs at equal compute. **Important limit:** A fitted law describes the measured range; anything beyond it is an extrapolation and can be wrong. - [Training](https://www.mef-llm-studio.com/en/wiki/#training) - [Loss](https://www.mef-llm-studio.com/en/wiki/#loss) - [Validation Loss](https://www.mef-llm-studio.com/en/wiki/#validation-loss) - [Encoder-Free Multimodal Model](https://www.mef-llm-studio.com/en/wiki/#encoder-free-multimodal) ## RGB444 colour tokens A palette of 4,096 colours: one token ID per pixel colour. **What does it mean?:** RGB444 uses four bits for each red, green and blue channel. With channel values 0–15, ID = red × 256 + green × 16 + blue. Display channels multiply those values by 17. The shapes learner predicts these colour IDs per image row. **Why does it matter?:** Makes image tokenization directly reversible and inspectable without a learned image codebook. **Important limit:** A colour token contains neither location nor object identity. A larger palette does not shorten the image sequence. - [Tokenization](https://www.mef-llm-studio.com/en/wiki/#tokenization) - [Image Caption](https://www.mef-llm-studio.com/en/wiki/#image-caption) ## Teacher forcing Training predicts the next target using the correct preceding data rather than its own generated output. **What does it mean?:** In the shapes row model, each target row is predicted from the caption, position, recurrent state and correct preceding image row. Generation instead feeds back the predicted preceding row. The training loss is cross-entropy over 4,096 colour classes per pixel. **Why does it matter?:** Explains why a falling training loss can coexist with mistakes accumulating during generation. **Important limit:** It does not guarantee correct free-running images. Pixels within the same generated row are conditionally independent. - [RGB444 colour tokens](https://www.mef-llm-studio.com/en/wiki/#rgb444) - [Training](https://www.mef-llm-studio.com/en/wiki/#training) - [Loss](https://www.mef-llm-studio.com/en/wiki/#loss) ## Learning Rate How far the optimizer moves the weights in one step. **What does it mean?:** The learning rate scales every change the optimizer makes to the weights. On the Training page it rises during a short warmup and then falls along a cosine curve to a tenth of its peak; on the Shapes page it falls along a cosine curve to a tenth or stays constant. **Why does it matter?:** Too high a rate makes the loss jump or diverge; too low a rate wastes steps. **Important limit:** There is no universally right value. It interacts with batch size, optimizer and model size, and a good value for one run is only a starting point for another. - [Optimizer](https://www.mef-llm-studio.com/en/wiki/#optimizer) - [Step](https://www.mef-llm-studio.com/en/wiki/#training-step) - [Batch Size](https://www.mef-llm-studio.com/en/wiki/#batch-size) - [AdamW](https://www.mef-llm-studio.com/en/wiki/#adamw) - [Training](https://www.mef-llm-studio.com/en/wiki/#training) ## Batch Size How many examples one step averages before the weights change. **What does it mean?:** A batch is the group of examples whose loss is averaged into one gradient and one update. On the Training page the batch of a step is built from micro batches: the micro batch is what fits on the card at once, and gradient accumulation adds micro batches up to the planned size of the step. On the Shapes page the batch size is the number of pictures per step. **Why does it matter?:** Larger batches give a steadier gradient per step but cost memory and time; together with the steps they set how much data a run sees. **Important limit:** A larger batch is not automatically better, and a different batch size usually needs a different learning rate. The micro batch changes speed and memory, not the size of a step. - [Step](https://www.mef-llm-studio.com/en/wiki/#training-step) - [Epoch](https://www.mef-llm-studio.com/en/wiki/#epoch) - [Learning Rate](https://www.mef-llm-studio.com/en/wiki/#learning-rate) - [Training](https://www.mef-llm-studio.com/en/wiki/#training) ## Step One update of the weights: a batch in, a loss, a gradient, one optimizer change. **What does it mean?:** A training step runs a batch through the model, computes the loss, propagates the gradient back and lets the optimizer change the weights once. The step counter of a run counts these updates; checkpoints, evaluations and the learning-rate schedule are placed by step. **Why does it matter?:** Steps are the clock of a training run: run length, save interval and loss curves are measured in them. **Important limit:** A step is not an epoch and not a generation step. How much one step learns depends on batch size and learning rate. The denoising steps of image generation are a different counter. - [Batch Size](https://www.mef-llm-studio.com/en/wiki/#batch-size) - [Epoch](https://www.mef-llm-studio.com/en/wiki/#epoch) - [Learning Rate](https://www.mef-llm-studio.com/en/wiki/#learning-rate) - [Checkpoint](https://www.mef-llm-studio.com/en/wiki/#checkpoint) - [Training](https://www.mef-llm-studio.com/en/wiki/#training) ## Epoch One full pass of the training data through the model. **What does it mean?:** An epoch is complete when every training example has been used once. With a fixed batch size, epochs ≈ steps × batch size ÷ number of training examples. The Shapes page shows this estimate next to its steps; the Training page counts steps and tokens instead. **Why does it matter?:** Epochs tell how often the model sees the same data again, which is what drives memorisation on small datasets. **Important limit:** The app trains by steps, not by epochs, so an epoch count is an estimate. More epochs on a small set can lower the training loss while validation gets worse. - [Step](https://www.mef-llm-studio.com/en/wiki/#training-step) - [Batch Size](https://www.mef-llm-studio.com/en/wiki/#batch-size) - [Dataset](https://www.mef-llm-studio.com/en/wiki/#dataset) - [Validation Loss](https://www.mef-llm-studio.com/en/wiki/#validation-loss) ## Gate A learned number between 0 and 1 that mixes two sources into one result. **What does it mean?:** In the Shapes model every pixel computes a gate g between 0 and 1. The probability of a colour is g × the colour the description names plus (1 − g) × the model's own colour prediction. The Explorer shows g as the share taken from the description; if the description names no colour, the gate is closed (0). **Why does it matter?:** It shows whether a pixel's colour was taken from the text or came from what the model learned, for example the colour of the row above. **Important limit:** A gate is a computed weight, not a decision or an intention of the model. A gate near 100 % says where the colour came from, not that the pixel lies in the right place. - [RGB444 colour tokens](https://www.mef-llm-studio.com/en/wiki/#rgb444) - [Teacher forcing](https://www.mef-llm-studio.com/en/wiki/#teacher-forcing) - [Attention Mechanism](https://www.mef-llm-studio.com/en/wiki/#attention) - [Softmax](https://www.mef-llm-studio.com/en/wiki/#softmax) ## Python A programming language used to build many AI tools. **What does it mean?:** Python runs programs. LLM Studio uses Python on its computing side to prepare data and call libraries such as PyTorch. The Windows interface is a separate part of the app. **Why does it matter?:** This explains why setup prepares a Python environment and packages as well as the application itself. **Important limit:** Python by itself is not a trained model. You do not need to write Python to use the learning paths. - [PyTorch](https://www.mef-llm-studio.com/en/wiki/#pytorch) - [NumPy](https://www.mef-llm-studio.com/en/wiki/#numpy) ## PyTorch A software toolkit for computing with neural networks. **What does it mean?:** PyTorch is a library that can be used from Python. It computes with tensors, provides neural-network building blocks and automatically calculates derivatives used to update weights during training. LLM Studio uses it on its computing side. **Why does it matter?:** The model is the architecture plus its learned weights; PyTorch is the tool that performs the calculations. **Important limit:** PyTorch is not a ready-made chatbot or a specific model. CPU or GPU use depends on installation, hardware and the selected workflow. - [Python](https://www.mef-llm-studio.com/en/wiki/#python) - [Tensor](https://www.mef-llm-studio.com/en/wiki/#tensor) - [CPU](https://www.mef-llm-studio.com/en/wiki/#cpu) - [GPU](https://www.mef-llm-studio.com/en/wiki/#gpu) - [CUDA](https://www.mef-llm-studio.com/en/wiki/#cuda) - [Optimizer](https://www.mef-llm-studio.com/en/wiki/#optimizer) ## Tensor An array of numbers with a defined shape. **What does it mean?:** In PyTorch, a tensor may hold a word vector, image colours or layer weights. Its shape gives the size of each axis: 3 × 64 × 64 can mean three colour channels of a 64 × 64 image. Its data type determines how each number is stored. **Why does it matter?:** Tensors connect the colour grids and vector strips in the explorers to actual numbers inside the model. **Important limit:** Shape alone does not tell you what the axes mean. Three axes can represent very different things at different calculation steps. - [PyTorch](https://www.mef-llm-studio.com/en/wiki/#pytorch) - [NumPy](https://www.mef-llm-studio.com/en/wiki/#numpy) - [Embeddings](https://www.mef-llm-studio.com/en/wiki/#embeddings) - [Safetensors](https://www.mef-llm-studio.com/en/wiki/#safetensors) ## CPU The computer’s general-purpose main processor. **What does it mean?:** CPU stands for Central Processing Unit. It runs programs, organises data and computes tensor operations. In CPU computation, most model data is held in system memory, called RAM. **Why does it matter?:** CPU and GPU are two possible places to compute. This helps explain why a task can take longer without a graphics card. **Important limit:** PyTorch CPU support does not mean every app feature or an arbitrarily large model is enabled on CPU. Runtime and available RAM limit what is practical. - [GPU](https://www.mef-llm-studio.com/en/wiki/#gpu) - [PyTorch](https://www.mef-llm-studio.com/en/wiki/#pytorch) ## GPU A processor designed for many parallel calculations. **What does it mean?:** GPU stands for Graphics Processing Unit. Many neural-network operations can be split into numerous parallel calculations. A suitable GPU can accelerate them; model data and intermediate results need room in its video memory, called VRAM. **Why does it matter?:** This explains why both GPU support and free VRAM matter during training. **Important limit:** A GPU does not accelerate every task. Hardware, algorithms and software must be compatible; system RAM and VRAM are different memory pools. - [CPU](https://www.mef-llm-studio.com/en/wiki/#cpu) - [CUDA](https://www.mef-llm-studio.com/en/wiki/#cuda) - [PyTorch](https://www.mef-llm-studio.com/en/wiki/#pytorch) ## CUDA NVIDIA’s software platform for computing on supported NVIDIA GPUs. **What does it mean?:** CUDA provides programming interfaces and tools for general-purpose computation on NVIDIA GPUs. With a compatible PyTorch setup, tensor operations run on the GPU rather than the CPU. A compatible NVIDIA driver is required. **Why does it matter?:** The term connects the graphics card, its driver and the installed PyTorch build. **Important limit:** CUDA is not a graphics card or a model. A CPU installation does not need it; an arbitrary GPU or a separately installed toolkit version does not guarantee compatibility. - [GPU](https://www.mef-llm-studio.com/en/wiki/#gpu) - [PyTorch](https://www.mef-llm-studio.com/en/wiki/#pytorch) - [CPU](https://www.mef-llm-studio.com/en/wiki/#cpu) ## NumPy A Python library for numerical arrays and data processing. **What does it mean?:** NumPy provides multidimensional arrays and numerical operations. This app uses it on the computing side, including data preparation. PyTorch tensors resemble these arrays but can also compute on supported GPUs and track derivatives for training. **Why does it matter?:** This separates data-processing tools from model libraries: both work with numbers but have different jobs. **Important limit:** A NumPy array is not a trained neural network. NumPy and PyTorch are not interchangeable in every operation or memory layout. - [Python](https://www.mef-llm-studio.com/en/wiki/#python) - [Tensor](https://www.mef-llm-studio.com/en/wiki/#tensor) - [PyTorch](https://www.mef-llm-studio.com/en/wiki/#pytorch) ## Hugging Face Hub A platform for model files, datasets and their documentation. **What does it mean?:** Hugging Face runs the Hub and develops libraries such as Transformers and Diffusers. A model repository may contain weights, configuration, a tokenizer and a Model Card describing the model. Downloaded models can run locally with suitable software. **Why does it matter?:** This separates the download source, the software library and the model itself. **Important limit:** A Hub listing does not guarantee quality or app compatibility. Check the model description and its terms of use; loading locally is different from calling a hosted API. - [Transformers (Hugging Face)](https://www.mef-llm-studio.com/en/wiki/#transformers-library) - [Diffusers](https://www.mef-llm-studio.com/en/wiki/#diffusers) - [Safetensors](https://www.mef-llm-studio.com/en/wiki/#safetensors) ## Transformers (Hugging Face) A library for loading and running supported model architectures. **What does it mean?:** Hugging Face Transformers provides model architecture implementations and tools for tokenizers, inputs and outputs. This app uses the library for supported external models. The trained weights are separate files. **Why does it matter?:** Transformers can refer either to the general architecture or to this particular software library. **Important limit:** The library is not itself Qwen or another checkpoint. Library support does not automatically mean every architecture and visualisation is supported in LLM Studio. - [Transformers](https://www.mef-llm-studio.com/en/wiki/#transformers) - [PyTorch](https://www.mef-llm-studio.com/en/wiki/#pytorch) - [Hugging Face Hub](https://www.mef-llm-studio.com/en/wiki/#hugging-face-hub) - [Tokenization](https://www.mef-llm-studio.com/en/wiki/#tokenization) ## Diffusers A library of building blocks for diffusion models. **What does it mean?:** Hugging Face Diffusers provides model components, schedulers and pipelines for diffusion methods. The diffusion path on the Vision page uses such components. A pipeline can connect a text encoder, noise predictor, scheduler and image decoder. **Why does it matter?:** This separates the library, the image model and the individual steps shown in the Vision Explorer. **Important limit:** Diffusers is not one particular trained image model. The Shapes learning mode with colour tokens uses a different computation path, not a Diffusers pipeline. - [PyTorch](https://www.mef-llm-studio.com/en/wiki/#pytorch) - [Hugging Face Hub](https://www.mef-llm-studio.com/en/wiki/#hugging-face-hub) - [Diffusion Model](https://www.mef-llm-studio.com/en/wiki/#diffusion-model) ## Safetensors A file format for storing tensors, often model weights. **What does it mean?:** Safetensors stores named tensors with their shapes, data types and values. Unlike Python Pickle, reading this format does not require executing embedded Python objects. Using the weights still needs a matching architecture, configuration and, depending on the model, a tokenizer. **Why does it matter?:** This explains why a downloaded weights file is not enough by itself and why checkpoints must match the model type. **Important limit:** The format does not establish origin, quality or usage rights. The rest of the model package and its program code also need review. - [Tensor](https://www.mef-llm-studio.com/en/wiki/#tensor) - [PyTorch](https://www.mef-llm-studio.com/en/wiki/#pytorch) - [Hugging Face Hub](https://www.mef-llm-studio.com/en/wiki/#hugging-face-hub) Contact: support@mef-llm-studio.com