Prompting and Reasoning

Robert Minneker

2026-02-17

Sources

Content derived from: J&M Ch. 7 (Large Language Models)

Today’s Lecture:

• In-context learning: zero-shot and few-shot prompting
• Chain-of-thought and structured reasoning strategies
• Structured prompting for controlled outputs
• Agentic workflows: planning, tool use, and failure modes

• OpenAI Prompt Engineering Guide — practical best practices for crafting effective prompts
• Simon Willison’s blog on prompt injection — essential reading on the security risks of LLM-powered applications

Learning Objectives

By the end of this lecture, you will be able to:

1. Explain how in-context learning enables task adaptation without parameter updates
2. Design effective zero-shot and few-shot prompts following principled design guidelines
3. Apply chain-of-thought prompting and compare advanced reasoning strategies
4. Use structured prompting to enforce schema-constrained outputs
5. Describe agentic LLM architectures and identify their failure modes

• This is the first LLM applications lecture — prompting as inference-time control, the cheapest and fastest way to steer model behavior
• Assumes familiarity with transformer architecture and the pretraining objective

Roadmap for our Final Push

Building

TODAY

Inference-Time Control

Prompting, decoding, self-consistency

Training-Time Control

SFT/PEFT, continued pretraining, pref. tuning

System-Time Augmentation

RAG, tools, agents

Understanding & Governing

Understanding

Interpretability, causal methods

Measuring

Evaluation, contamination, protocols

Governing & Shipping

Safety engineering, deployment constraints

• Course roadmap showing six complementary dimensions of LLM engineering
• Row 1 covers building: inference-time → training-time → system-time control
• Row 2 covers understanding & governance: interpretability → evaluation → safety/deployment
• Today we focus on inference-time control — the cheapest, fastest knob: prompting and reasoning strategies

Running Example: Medical Triage Assistant

Throughout this lecture, we’ll develop prompts for a concrete scenario: a medical triage assistant that helps patients describe symptoms and routes them to appropriate care.

Scenario:

Setting: Hospital front-desk chatbot for a busy emergency department
Goal: Classify patient urgency (emergency / urgent / routine), suggest department
Constraints: Must never diagnose, always recommend professional evaluation, handle anxiety sensitively
Key challenge: Get this right with prompting alone — no finetuning budget

• As we cover each prompting strategy, we’ll ask: how would you prompt the triage assistant?

• This running example grounds abstract prompting concepts in a high-stakes, real-world application
• Students should consider: what happens when the prompt fails? What are the safety implications?

Part 1: In-Context Learning

State Change: Learning Without Training

Finetuning adapts model weights. In-context learning adapts model behavior through the prompt alone, keeping all parameters frozen.

In-context learning (ICL) enables task adaptation by conditioning on natural language prompts

The model processes a prompt \(p\) and input \(x\) to produce output \(y\) with frozen weights \(\theta\):

\[ y = \mathrm{LLM}_\theta(p, x) \]

Finetuning

Updates parameters θ using labeled data

Expensive, persistent

In-Context Learning

Steers frozen model via prompt context

Cheap, ephemeral

• The prompt acts as an inductive bias, tapping into latent knowledge from pretraining

• In-context learning is distinct from finetuning: weights are never updated
• The model leverages patterns learned during pretraining to follow prompt instructions
• This is sometimes called “meta-learning” — learning to learn from context

Zero-shot prompting provides only a task description, while few-shot adds demonstrations

Prompt Type	What the Model Sees	Example
Zero-shot	Task description only	Translate to French: 'cat'
Few-shot	k input-output demonstrations	cat → chat, dog → chien, house → ?

• Few-shot prompting provides \(k\) exemplars so the model infers the pattern from demonstrations
• The model is not updating parameters — it performs statistical pattern completion over the context

• Common misconception: the model “learns” from few-shot examples like a human — it actually does pattern matching over the context window
• Few-shot works well when examples are representative and diverse
• Zero-shot works best for tasks closely aligned with pretraining data

Prompt wording, formatting, and example order significantly affect model predictions

The output distribution is sensitive to prompt structure:

\[ P(y \mid x, p_1) \neq P(y \mid x, p_2) \quad \text{if} \quad p_1 \neq p_2 \]

Wording

"Summarize" vs. "In one sentence, summarize the main argument for a graduate audience"

Formatting

Whitespace, punctuation, and template consistency alter tokenization

Example Order

The ordering of few-shot exemplars introduces recency bias

Prompt sensitivity is real: Zhao et al. (2021) showed that simply reordering few-shot examples can swing GPT-3 accuracy from near-chance to near-SOTA on the same benchmark — a 30+ percentage point difference from permutation alone.

• Diverse, balanced exemplars calibrate outputs and reduce unwanted bias

• Small prompt changes can cause dramatically different outputs — this is not intuitive to students
• Prompt design is both an art and an emerging science
• Best practice: test multiple prompt variants and measure variance

Small prompt changes produce dramatically different triage decisions

Bad Prompt

"Classify this patient's urgency: chest pain after exercise"

→ "Routine — likely muscle soreness from exercise"

Improved Prompt

"You are a medical triage assistant. Classify urgency as EMERGENCY, URGENT, or ROUTINE. Err on the side of caution. Always recommend professional evaluation.\n\nPatient: chest pain after exercise"

→ "URGENT — Chest pain requires professional evaluation regardless of suspected cause. Recommend immediate medical assessment."

• This is not hypothetical — prompt sensitivity in medical applications is a real patient safety concern
• The improved prompt adds: role, output format, safety bias (“err on the side of caution”), and a disclaimer
• The same clinical scenario produces a dangerous vs. safe response based on prompt design alone

Think-Pair-Share: Choosing a Prompting Strategy

For our triage assistant, would you use zero-shot or few-shot prompting? Consider:

• Latency: How fast must the response be? (few-shot uses more tokens)
• Accuracy: How much does task performance improve with examples?
• Label space: Is {EMERGENCY, URGENT, ROUTINE} clear enough without examples?
• Calibration: Do you need the model to express uncertainty?

Discuss with a neighbor for 2 minutes, then share your recommendation.

Part 2: Chain-of-Thought and Structured Reasoning

State Change: Beyond Direct Answers

Standard prompting asks for a final answer. Chain-of-thought prompting elicits intermediate reasoning steps, dramatically improving performance on complex tasks.

Chain-of-thought prompting decomposes complex tasks into explicit intermediate steps

CoT biases the model toward generating a reasoning trace rather than a single direct answer:

Example: Arithmetic Word Problem

Prompt:

"If Alice has 3 apples and buys 2 more, how many does she have? Let's think step by step."

→

LLM Output:

"Alice starts with 3. She buys 2 more. 3 + 2 = 5. Alice has 5 apples."

Standard vs. chain-of-thought prompting from Wei et al. (2022)

• The model breaks complex problems into tractable subproblems, avoiding shallow heuristics
• CoT is most effective for tasks requiring multi-step logic: arithmetic, symbolic reasoning, multi-hop QA

• Intuition: by requesting stepwise logic, we bias the model toward systematic decomposition
• CoT is less helpful for simple pattern-matching tasks that don’t require reasoning
• The key insight: intermediate tokens serve as a “scratchpad” for computation

Zero-shot CoT adds a trigger phrase while few-shot CoT provides worked examples

Zero-shot CoT

Append "Let's think step by step." to the prompt

Leverages the model's learned prior for multi-step explanations

Few-shot CoT

Include exemplars with full reasoning traces

Provides direct inductive bias via demonstration

• On GSM8K, GPT-3 accuracy rises from below 20% (direct) to over 50% with zero-shot CoT
• Even a single trigger phrase dramatically changes model behavior due to learned priors

• A common misconception: CoT always helps — it’s actually less effective for simple tasks
• Few-shot CoT is generally stronger but requires crafting worked examples
• The choice depends on task complexity and availability of example reasoning traces

Adding “Let’s think step by step” produces dramatic accuracy gains across reasoning tasks (Kojima et al., 2022)

• A single phrase appended to the prompt consistently doubles accuracy on math reasoning benchmarks
• The effect is most pronounced on tasks requiring multi-step arithmetic

• Kojima et al. (2022) showed this simple trigger phrase activates latent reasoning capabilities
• The gains are task-dependent — CoT helps most where step-by-step decomposition is natural
• This result popularized the idea that prompting strategies can unlock emergent abilities

Chain-of-thought helps reasoning tasks but not simple pattern matching

CoT Helps

✓ Multi-step arithmetic

✓ Multi-hop reasoning

✓ Logical deduction

✓ Complex triage decisions

CoT Doesn't Help (or Hurts)

✗ Simple factual recall

✗ Single-label classification

✗ Pattern matching / extraction

✗ Tasks where the answer is obvious

• For our triage assistant: CoT helps when symptoms are ambiguous (“chest pain + shortness of breath + recent travel” requires reasoning about multiple possibilities)

• CoT adds latency and cost — only use it when the task genuinely requires multi-step reasoning
• Wei et al. (2022) showed CoT gains are most pronounced for larger models (>100B params)

GPT-4 and the Bar Exam

GPT-4 scored in the 90th percentile on the Bar Exam (vs. GPT-3.5’s 10th percentile). Takeaway: Benchmarks can overstate reasoning ability — multiple-choice format may not reflect true open-ended legal reasoning.

• OpenAI’s GPT-4 (2023) scored in the 90th percentile on the Uniform Bar Exam
• Critics noted: (1) multiple-choice format may overstate reasoning, (2) training data likely contained exam prep materials, (3) exam scores measure recognition, not open-ended generation
• The full GPT-4 Technical Report (arXiv:2303.08774) contains performance across dozens of standardized exams

Self-consistency, tree-of-thought, and least-to-most extend basic chain-of-thought

Self-Consistency

Sample N chains, take majority vote

ŷ = mode{f(Cᵢ)}

Tree-of-Thought

Branch-and-explore reasoning paths with backtracking

BFS over reasoning states

Least-to-Most

Decompose into ordered sub-questions, solve bottom-up

Hierarchical decomposition

• Self-consistency reduces stochasticity by aggregating over multiple reasoning chains
• These methods trade compute for accuracy — more reasoning paths yield more reliable answers

• Self-consistency is simple but powerful: sample multiple CoT traces, majority-vote the final answer
• Tree-of-thought is more expensive but allows exploring and comparing solution branches
• Least-to-most is ideal for problems with natural hierarchical structure

Tree-of-Thought explores multiple reasoning branches and backtracks from dead ends

• Unlike linear CoT, ToT can backtrack from unpromising paths and explore alternatives
• The LLM serves as both the generator (proposing steps) and the evaluator (scoring them)

• Tree-of-Thought is especially effective for tasks with multiple valid solution paths (e.g., creative writing, math proofs)
• The evaluation function can be the LLM itself (“Is this step leading to the correct answer?”)
• Cost trade-off: ToT requires many more API calls than standard CoT

Choosing a reasoning strategy involves compute-accuracy tradeoffs

Strategy	Compute Cost	Robustness	Best For
Basic CoT	1×	Low	Simple multi-step problems
Self-Consistency	N×	High	When you need reliable answers (triage!)
Tree-of-Thought	N×M×	Highest	Creative/open-ended exploration
Least-to-Most	K×	Medium	Hierarchical decomposition

• For our triage assistant, self-consistency is the right choice: sample 5 triage classifications, take the majority — a wrong classification has real consequences

• N = number of samples for self-consistency (typically 5-20)
• M = branching factor for ToT (typically 3-5 branches per level)
• K = number of sub-questions for least-to-most

Concept Check

Why does self-consistency improve over a single chain-of-thought? Under what conditions might it fail? Think about the relationship between answer diversity and aggregation quality.

Discussion (2 min)

Consider a multi-hop science question: “What element has the highest electronegativity, and what compounds does it commonly form?” Would you use basic CoT, self-consistency, or least-to-most prompting? Why?

Hallmark Discoveries in Prompting and Reasoning

In-Context Learning as a Scale-Era Behavior

Large language models can perform new tasks from a few demonstrations in the prompt — no gradient updates required. This emergent capability fundamentally changed how practitioners interact with models.

Chain-of-Thought as Eliciting Latent Computation Traces

Wei et al. (2022) showed that prompting models to "show their work" unlocks reasoning capabilities already present in the weights — intermediate tokens serve as a scratchpad that transforms single-step prediction into multi-step computation.

Zero-Shot CoT Triggers as Steering Priors

Kojima et al. (2022) demonstrated that a single phrase — "Let's think step by step" — activates latent reasoning without any demonstrations, revealing that models encode reasoning priors that can be unlocked through simple prompt design.

• These three discoveries established prompting as a first-class paradigm for controlling LLM behavior
• In-context learning appeared to “emerge” with scale — smaller models show much weaker ICL capabilities
• CoT and zero-shot CoT are now standard tools in every practitioner’s toolkit

Part 3: Structured Prompting Techniques

State Change: Controlling Output Format

Reasoning strategies improve answer quality. Structured prompting goes further by constraining the format of model outputs, enabling integration with downstream systems.

Structured prompting constrains LLM outputs to machine-readable formats like JSON or XML

By specifying output schemas, we reduce output entropy and enable robust downstream parsing:

\[ f_{\text{LLM}}(x; P) \in \mathcal{S} \]

where \(\mathcal{S}\) is the set of valid outputs (e.g., all well-formed JSON objects).

Example: Information Extraction

Prompt:

"Extract fields and reply in JSON: { 'name': ..., 'date': ..., 'location': ... }"

→

Output:

{"name": "Alice", "date": "2026-02-17", "location": "Seattle"}

• Schema-constrained prompts support automation: API calls, data validation, pipeline integration

• Structured prompting is essential for production NLP systems that consume LLM outputs programmatically
• Vague prompts lead to inconsistent formats — explicitness is key
• Modern APIs (e.g., OpenAI’s JSON mode) enforce structural constraints at the decoding level

Prompt templates and role instructions reinforce consistent output structure

Role Prompting

"You are an information extraction system. Output results in valid JSON only."

Sets behavioral context

Template Prompting

"Return your answer as: <result><answer>...</answer></result>"

Defines exact output shape

• System instructions (role prompts) establish context and reinforce schema adherence
• Clear templates improve reliability across repeated calls and diverse inputs

• Role prompting and system instructions are now standard in production deployments
• The clearer and more explicit the schema, the higher the reliability
• Challenges remain for complex or deeply nested structures

The generate-validate-retry pattern ensures reliable structured outputs

1. Generate

LLM outputs JSON

→

2. Validate

Schema check

→

3a. Accept ✓

Valid output

↓ if invalid

3b. Retry

Feed error back to LLM

→

Re-generate

With error message

Triage output schema:

{"urgency": "EMERGENCY|URGENT|ROUTINE",
 "reasoning": "string",
 "recommended_department": "string",
 "disclaimer": "Always seek professional medical advice"}

• This pattern is standard in production LLM systems
• Modern APIs (OpenAI’s JSON mode, Anthropic’s tool use) enforce schemas at the decoding level
• For our triage assistant, schema validation ensures every response includes the disclaimer

Retrieval-augmented prompting injects external context to ground LLM responses in facts

Query q

User question

→

Retrieve R(q)

Top-k passages

→

Augmented Prompt

concat(R(q), q)

→

LLM Answer

Grounded response

\[ \text{Prompt}(q) = \text{concat}\left(\mathcal{R}(q),\ q\right) \]

• RAG mitigates knowledge cutoff by dynamically providing up-to-date or domain-specific information
• Key prompting pattern for RAG: inject evidence, instruct to cite, handle context limits
• We’ll cover RAG architecture in depth in a later lecture — today, focus on how the prompt changes

• For our triage assistant, RAG could inject relevant medical guidelines into the prompt
• Full RAG pipeline (chunking, embedding, retrieval, reranking) covered in the next lecture
• Today’s key insight: the prompt must instruct the model to USE the retrieved context, not ignore it

Concept Check

A biomedical QA system retrieves 20 PubMed abstracts for each query, but the LLM’s context window is 4,096 tokens. What strategies could you use to handle this? What are the trade-offs?

Part 4: Agentic Workflows

State Change: From Answering to Acting

Prompting strategies so far produce text. Agentic workflows enable LLMs to reason, plan, and invoke external tools to complete tasks in the real world.

The ReAct framework interleaves reasoning traces with tool-calling actions

LLMs alternate between generating intermediate thoughts and invoking external tools:

\[ \pi: (x, h) \rightarrow (r, a) \]

where \(r\) is a reasoning step, \(a\) is an action (API/tool call), and \(h\) is the interaction history.

ReAct Loop Example

Reason

"To answer, I need the current weather in Paris."

Act

weather_api("Paris") → {"temp": "12°C", "condition": "cloudy"}

Answer

"It is currently 12°C and cloudy in Paris."

ReAct framework comparison from Yao et al. (2023): reasoning-only vs. action-only vs. ReAct

• Tool calls are generated as output tokens, extending LLM capacity beyond language modeling

• ReAct (Yao et al., 2023) formalizes the reason-act loop that underlies modern AI agents
• Tool use enables capabilities impossible with text alone: computation, data retrieval, code execution
• The LLM serves as both the planner and the controller of external systems

ReAct grounds each reasoning step in observable actions and environmental feedback

• ReAct agents outperform pure reasoning (CoT) and pure acting (tool-only) approaches by combining both
• The observation step grounds the model in real data, reducing hallucination

• ReAct was a foundational paper for modern AI agents
• The key insight: reasoning without action leads to hallucination; action without reasoning leads to incoherent tool use
• Modern agent frameworks implement ReAct as a core pattern

Agents combine planning, tool use, and reflection — depth in Feb 24

• Planning: Decompose goals into sub-tasks with explicit success criteria
• Tool use: Invoke APIs, run code, query databases — covered in depth Feb 24
• Observe-act loop: Agents iterate on environment feedback (ReAct pattern)

For our triage assistant, a simple agent loop might:

1. classify urgency
2. look up relevant medical guidelines via RAG
3. verify the classification is consistent with guidelines

• Autonomous coding agents (e.g., SWE-Agent) can resolve real GitHub issues, but high failure rates (~88%) highlight compounding error problems
• Production coding assistants take a more constrained approach than fully autonomous agents
• Full treatment of tool catalogs, function calling patterns, and agent architectures in the RAG & Tool Use lecture (Feb 24)

Agentic pipelines face compounding errors, hallucination, and prompt injection attacks

For a pipeline of \(n\) steps where each step has error probability \(p_i\):

\[ P_{\text{failure}} = 1 - \prod_{i=1}^n (1 - p_i) \]

Compounding Errors

Early mistakes cascade through all downstream steps

Hallucination

Plausible but incorrect facts or invalid reasoning steps

Prompt Injection

Adversarial inputs override system instructions and safety guardrails

• Reliability decreases exponentially with pipeline depth, even when individual step error rates are low

• Example: a retrieval error in step 1 misleads all downstream synthesis
• Prompt injection can fully subvert agentic control — not just minor formatting issues
• Adversarial inputs like “Ignore previous instructions” can override system goals
• Prompt injection defense patterns (input sanitization, output validation, monitoring) covered in depth in Mar 12 (Ethics, Safety & Deployment)
• Simon Willison documented indirect prompt injection in Bing Chat (2023) — hidden instructions in webpages could hijack chatbot behavior
• The “Ignore Previous Instructions” attack is a fundamental unsolved problem: LLMs cannot reliably distinguish system instructions from user-provided content

Concept Check

If each step in a 5-step agentic pipeline has a 10% error rate, what is the overall failure probability? What does this imply about designing agentic systems?

Summary: Prompting and Reasoning – Key Takeaways

1. In-context learning enables task adaptation through prompts alone, with no parameter updates — the prompt serves as an inductive bias over frozen weights

2. Prompt design matters: wording, formatting, and example order significantly affect model behavior; principled prompt engineering is essential

3. Chain-of-thought prompting elicits step-by-step reasoning, yielding large accuracy gains on arithmetic, logic, and multi-hop tasks

4. Advanced strategies (self-consistency, tree-of-thought, least-to-most) trade additional compute for more reliable reasoning

5. Structured prompting constrains outputs to machine-readable formats, enabling integration with production systems

6. Agentic workflows extend LLMs from text generators to planners and actors, but face compounding errors, hallucination, and prompt injection risks

3 prompting heuristics you can apply tomorrow:

1. Be explicit: Specify role, format, constraints, and edge case handling in every prompt
2. Test variants: Run 3+ prompt phrasings and measure variance before committing
3. Match compute to stakes: Use self-consistency for high-stakes decisions, basic CoT for everyday tasks

• These techniques represent the state of the art in getting the most from LLMs without changing parameters
• Next lecture: Finetuning & Adaptation — when prompting alone isn’t enough, we turn to training-time methods

Coming Up Next

Finetuning & Adaptation (Feb 19)

When prompting hits its limits — insufficient domain knowledge, inconsistent behavior, or the need for persistent style changes — we turn to training-time adaptation.

• Full finetuning vs. parameter-efficient methods (LoRA, adapters)
• Instruction tuning and RLHF/DPO alignment
• Choosing between prompting, finetuning, and RAG