ReAct Agent with LangChain: from the Loop to the Harness

Authored by Dr. Tiziana Ligorio for AI Agents - CSCI 395.32 taught at Hunter College of The City University of New York¶

In this tutorial we build a tool-using ReAct agent. The agent alternates between reasoning (deciding what to do next) and acting (calling a tool), and it reads each tool result before deciding again.

We first look under the hood at a minimal ReAct loop written by hand, so that nothing is hidden. Then we build the agent with a framework, the way practitioners usually do. We will use LangChain’s create_agent, and see how much of the loop the framework handles for us. Finally, we evaluate the agent and improve it.

By the end of this tutorial you will be able to:

1. Explain what a ReAct loop does, step by step, and what a minimal loop leaves out.

2. Build a tool-using agent with LangChain and identify the harness around the model: the loop, the tools, the context, and the guardrails.

3. Connect tools through MCP, both from a server you write and from an existing one.

4. Evaluate the agent on a small evaluation test set (end-to-end), read its traces to find where it fails, and improve one component at a time (component-level).

This tutorial follows the ReAct and Evals lectures.

What we will build: the Riverview Course Helper¶

Riverview College is a fictional school (it does not describe Hunter College or CUNY). We will build a small Course Helper that answers student questions about the course catalog and the Fall 2026 academic calendar, such as:

• “What do I need to take before CSCI 310?”

• “How many weeks are there between the first day of classes and the withdrawal deadline?”

• “If I get an A in CSCI 210 and a B+ in MATH 220, what is my GPA for those two courses?”

The data is small and fixed on purpose. Because every question has a known correct answer, we can check whether the agent got it right, which is exactly what we need for the evaluation part of this tutorial.

Sections

1. Installs and imports

2. Setup your API key

3. The Riverview data and tools

4. Under the hood: a ReAct loop by hand

5. Build the agent with LangChain

6. Tools through MCP

7. Evaluate the agent

8. Iterate: change one thing, measure again

9. Optional: tracing and evals with LangSmith

10. Reflect and your turn

1. Installs and Imports¶

%%capture hides the pip install output to keep the notebook clean. We also install uv, which provides the uvx command used to launch an MCP server in Section 6. If you are running locally, uv sync installs the same packages from pyproject.toml (see SETUP.md).

2. Setup your API Key¶

Step 1 — Get an OpenRouter API key¶

OpenRouter provides access to models from different vendors through one API. This is the only key the core tutorial needs.

1. Go to https://openrouter.ai

2. Sign in (or create an account if you don’t have one)

3. Once logged in, navigate to https://openrouter.ai/settings/keys

4. Click Create Key

5. Give the key a name, e.g. colab-react-agent

6. Copy the key immediately (you won’t be able to see it again)

Important: Treat this key like a password. Do not share it, paste it into notebooks, or commit it to GitHub.

Step 2 — Store your API key¶

If running in Google Colab¶

1. On the left sidebar, click 🔑 Secrets.

2. Add a new secret named OPENROUTER_API_KEY with your key as the value.

3. Toggle the switch to give the notebook access to the secret.

If running locally¶

Create a .env file in the project folder containing:

OPENROUTER_API_KEY=your_openrouter_key_here

Make sure .env is listed in .gitignore.

Load the API Key¶

In Colab:¶

Uncomment and run the cell below if you’re using Google Colab.

Locally:¶

Run the cell below if you’re running locally with a .env file.

Choose the model¶

We use openai/gpt-4.1-mini through OpenRouter: a small, inexpensive model that is reliable at tool calling.

We set the temperature to 0. Temperature controls how much randomness the model uses when choosing its next words. At 0 the model almost always makes the same choices for the same input. That matters twice in this tutorial: when debugging, we want to reproduce a behavior, and when evaluating, we want a score to reflect our changes rather than chance. (Even at temperature 0, outputs can occasionally vary slightly, which is one reason evaluations use several test cases rather than one.)

3. The Riverview Data and Tools¶

A tool is just a function the agent can run. Before any agent exists, we write the tools as plain Python functions and test them directly. This is good practice: if a tool is wrong, no amount of prompting will make the agent right.

Our Course Helper will have four tools:

Later, in Section 6, the agent will also get today’s date from an MCP server.

First, the data. In a real system these would be calls to the college’s catalog and calendar services. Here they are small Python dictionaries, so every answer is known in advance. This is a good way to start prototyping: to evaluate a system on a small, well known set of data to inspect behavior before connecting to the full database.

Tool 1: get_course_info¶

Students type course codes in many ways (csci310, CSCI 310, Csci 310), so the tool normalizes the code first.

Notice what happens when the course does not exist: the tool returns an error message instead of crashing. The model reads tool results, so a clear message such as “No course CSCI 999 in the catalog” lets it recover, for example by telling the student it could not find the course.

We also simulate something every real service does from time to time: a temporary outage. The first lookup of STAT 205 raises a timeout; the next one succeeds. We will use this later to see how the harness can retry a failed tool call.

Tool 2: get_academic_date¶

The tool accepts only the event names in the calendar. Restricting the input to a fixed list of values makes the tool easier for the model to use correctly, and we will state that list in the tool’s description.

Tool 3: days_between¶

Language models are unreliable at date arithmetic (months have different lengths, and the model has to count). A tiny tool does it exactly.

Tool 4: calculator¶

A calculator is the classic first tool (recall how bad our agent was at counting in our first tutorial), and it is a good example of tool safety. The tempting shortcut is Python’s eval(), but eval() runs any Python code. When the input comes from a model, that means agent-generated code could open files, import modules, or run system commands. That would be setting your agent up for easy manipulation from prompt injection.

Instead, we parse the expression into a syntax tree with Python’s ast module and allow only numbers, arithmetic operators, parentheses, and round(). Anything else is rejected. We also cap exponents so a request like 9**9**9 cannot freeze the notebook.

How ast works¶

When Python runs code, it first parses the text into an abstract syntax tree (AST): a tree of objects, called nodes, that records what the code means and in what order to compute it. Each node is an object of a specific type, so a program can inspect it without running anything. The ast module lets us do that parsing step ourselves.

ast.parse("2 + 3 * 4", mode="eval") produces this tree (mode="eval" means “expect a single expression”):

Expression
└── body: BinOp            (a binary operation)
    ├── left:  Constant 2
    ├── op:    Add
    └── right: BinOp
        ├── left:  Constant 3
        ├── op:    Mult
        └── right: Constant 4

The tree already reflects operator precedence: 3 * 4 sits deeper, so it is computed before the addition.

The calculator uses four node types:

_evaluate walks the tree recursively. A Constant returns its number. A BinOp first evaluates its left and right children, then looks up its operator in _BINARY_OPS, a dictionary that maps each allowed node type (such as ast.Add) to the matching Python function (operator.add). UnaryOp and round() follow the same pattern.

Any node type not listed, such as a variable name, an attribute like os.system, or a call to anything except round, reaches the final raise ValueError. The check never runs those nodes, so input like __import__("os") is rejected before it can do anything.

Test the tools directly¶

Always test tools on their own before giving them to an agent, including the cases that should fail.

Pedagogical note: two kinds of tool failure¶

Our tools fail in two different ways, on purpose:

• Expected problems (an unknown course, a misspelled event, a bad date format) return an error message. These are part of normal use. The model reads the message and can correct itself or explain the problem to the student.

• Unexpected, temporary problems (a service timeout) raise an exception. The model cannot fix a network outage by thinking harder. Whether to retry, how many times, and what to do if retries run out are decisions for the harness, the code around the model. We will see this in Section 5.

4. Under the Hood: a ReAct Loop by Hand¶

Agent frameworks run the ReAct loop for us. Before we use one, we write the smallest loop that works, so we know exactly what the framework is doing on our behalf.

Native tool calling¶

Modern models support native tool calling. Along with the conversation, we send the model a description of each tool (a JSON schema). The model then replies in one of two ways:

• with text: it has the final answer, or

• with one or more tool calls: a tool name plus arguments in JSON, built from the schema.

The model never runs anything itself. Our code runs the requested function and sends the result back to the model as a new message. Then we ask the model again. That cycle is the ReAct loop: Think → Act → Observe, repeated until the model answers.

What native tool calling replaced. Early ReAct agents asked the model to write plain text such as Thought: ... / Action: {"action": "calculator", "action_input": "5+2"} and then pulled the action out with regular expressions. That worked, but it was fragile: one missing quote or a slightly different format broke the agent. Native tool calling makes the request structured, so we no longer parse free text.

Step 1 — A client for the model¶

We call the model through the OpenAI SDK, pointed at OpenRouter’s compatible API.

Step 2 — Describe the tools by hand¶

For this minimal loop we give the model three tools. Each description is a contract:

• for the model, it is documentation: the description says when to use the tool, and the parameters say how to call it;

• for our code, it is a schema: we use the same names to find the Python function to run.

Notice the enum for the calendar events: listing the valid values makes a wrong argument much less likely.

Step 3 — The loop¶

The system prompt tells the model how to behave (its policy). Asking for one short sentence before each tool call gives us the Thought part of the trace to read.

Then the loop itself. Read it carefully: it is the whole ReAct pattern.

Step 4 — Run it¶

This question needs several steps: two calendar lookups, a date difference, and a division.

Look at the trace above. Each Action is a structured tool call the model built from a schema, each Observation is the result our code sent back, and the model decided after every observation what to do next. That is ReAct.

(The model may request two tools in the same turn, for example both calendar lookups at once. The loop runs them one after the other.)

What this loop does not handle¶

Our loop works for a friendly question. A real agent needs much more around the model. Here is what is missing:

Everything in that table, together with the loop itself, is part of the harness: the software around the model that makes it an agent. Writing and maintaining all of it by hand is real work. This is what agent frameworks wrap up for us, so in the next section we build the same agent with LangChain and see where each piece went.

Running a model locally instead. The same loop works with a local model served by Ollama, which offers the same OpenAI-compatible interface. Install Ollama, pull a model that supports tool calling, and change only the client and model name:

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
MODEL = "qwen3"   # any locally pulled model that supports tool calling

Smaller local models follow tool schemas less reliably, which makes this a good experiment in how much the model matters.

5. Build the Agent with LangChain¶

We now build the same Course Helper with LangChain’s create_agent. The model is the same and the tool functions are the same. What changes is who writes the harness: most of the pieces we listed at the end of Section 4 are now provided by the framework.

Here are the LangChain docs for your reference.

Step 1 — Turn the functions into tools¶

In Section 4 we wrote a JSON schema by hand for every tool and kept it in sync with the Python function. The @tool decorator generates the schema for us:

• the function name becomes the tool name,

• the type hints become the parameters (a Literal becomes a list of allowed values, like our enum),

• the docstring becomes the description.

That makes the docstring part of the program: it is exactly what the model reads to decide when to use the tool. We wrap the functions from Section 3 rather than rewriting them.

Let’s look at the schema LangChain generated for the calendar tool and compare it with the one we wrote by hand in Section 4.

The two say the same thing: a name, a description taken from the docstring, and an event parameter restricted to the six calendar events. The difference is maintenance. With @tool, changing the function’s signature or docstring changes the schema automatically.

Descriptions say when, parameters say how. If a docstring is vague, the model may skip a tool it needs or call the wrong one. Writing tool descriptions is part of designing an agent, and we will come back to it when we evaluate.

Step 2 — The model¶

LangChain’s ChatOpenAI wrapper talks to any OpenAI-compatible API, so we point it at OpenRouter with the same model and temperature as before.

Step 3 — The system prompt is the agent’s policy¶

The system prompt states the agent’s policy: what it is for, when to rely on tools, what to do when information is missing, and the facts it needs that no tool provides (here, the grade-point scale).

Step 4 — Guardrails as middleware¶

LangChain lets us add pieces of the harness as middleware: code that runs around every model call or tool call. We add three guardrails from Lecture 7:

These are three of LangChain’s prebuilt middleware. Others include model retry and fallback models, summarizing long conversations, human approval before a tool runs, a to-do list for planning, and more. See the full list in Prebuilt middleware.

Then create_agent assembles the loop: the model, the tools, the policy, and the guardrails.

That is the whole agent. The ReAct loop, the tool calls, passing observations back to the model, and stopping when the model answers are all handled by create_agent.

Step 5 — A readable trace¶

agent.invoke(...) returns the full conversation: the question, every model turn, every tool call, and every tool result. That is a lot to read, so we write a small helper that turns it into the Thought → Action → Observation trace from the lecture.

We return the trace as data (a list of steps) and print it separately. Keeping the trace as data is what will let us evaluate many runs in Section 7.

Step 6 — Run the agent¶

A question that needs the catalog and the calculator: the agent must look up both courses’ credits, then compute a credit-weighted average.

Check the arithmetic yourself: CSCI 210 has 4 credits and MATH 220 has 3, so the GPA is (4 × 4.0 + 3 × 3.3) / 7 = 25.9 / 7 ≈ 3.70. Notice that the agent had to find the credits with a tool; they were not in the question.

Step 7 — When a tool fails¶

Remember the simulated outage: the first lookup of STAT 205 raises a timeout. First, an agent without the retry guardrail:

The exception went straight through the agent, exactly like our hand-written loop. Now the same question with the agent that has ToolRetryMiddleware:

This time the agent answered. The trace shows a single successful get_course_info call: the failed first attempt and the retry happened inside the harness, before the model ever saw a result. That is convenient, but it also means the trace we build from the messages does not show everything that happened. Observability tools such as LangSmith (optional Section 9) record those hidden steps too.

Where did the harness go?¶

Every piece of the hand-written loop from Section 4 still exists. It just moved into the framework:

Agent = Model + Harness. We did not change the model at all between Section 4 and Section 5. Everything we added (schemas, limits, retries, a trace) is harness.
When you improve an agent, you will most often be changing the harness, and the way to know whether a change helped is to evaluate it.

6. Tools through MCP¶

So far every tool was a Python function inside this notebook. That works for one agent, but every other agent (or app) that wants the same tools would need its own copy.

The Model Context Protocol (MCP) is an open standard for exposing tools to any compatible agent:

• an MCP server is a small program that offers capabilities through the protocol;

• an MCP client (here, our agent) connects to one or more servers and uses what they offer.

A server can offer three kinds of things:

In this section we do both: we write a tiny MCP server for Riverview, and we use an existing, official MCP server for the current date and time.

Step 1 — Write a tiny MCP server for Riverview¶

We use FastMCP, part of the official MCP Python SDK. Our server offers one of each kind:

• a tool: get_academic_date, the same calendar lookup as before;

• a resource: the whole Fall 2026 calendar as a Markdown table;

• a prompt: a template for drafting an email to the Academic Advising Center.

An MCP server is a separate program, so it cannot see this notebook’s variables. That is why the calendar data appears again inside the server file. %%writefile riverview_mcp_server.py makes Jupyter save the cell’s text as a file next to the notebook, without running any of it.

The decorators do the same job @tool did in LangChain: the function name, type hints, and docstring become what a client sees. The difference is that any MCP client can now use this server, not just our agent. For example, you could connect the same file to Claude Desktop or an IDE assistant without changing a line.

Step 2 — Connect to two MCP servers¶

We connect to two servers at once:

1. our Riverview server, started with the notebook’s own Python;

2. the official time server, mcp-server-time, maintained by the MCP project. uvx downloads and launches it automatically. It gives the agent something no model knows on its own: today’s date.

Both are local servers: they run as subprocesses and talk to the client over stdin/stdout ("transport": "stdio"). Remote servers work the same way but use a URL instead of a command.

Connecting to MCP servers is input/output work, so the client’s methods are asynchronous and we call them with await. Jupyter allows await directly in a cell; in a regular Python script it would have to be inside an async def function.

The client turned every MCP tool into a regular LangChain tool, so the agent can use them like the ones we wrote with @tool.

Tools are only one of the three kinds. Here is the resource our server offers. The harness (the application around the model), not the model, decides to load it, for example to put the whole calendar into the agent’s context.

And the prompt template, filled in with arguments. A user (or an app’s menu) would pick this template; it returns ready-to-send messages.

Step 3 — An agent with MCP tools¶

We give the agent the calendar tool from our MCP server (instead of the local one) plus the time server’s tools, keep the other local tools, and add one rule to the policy about today’s date.

MCP tools are asynchronous, so we run the agent with await agent.ainvoke(...) instead of agent.invoke(...).

Three tools from three different places worked together: the time server supplied today’s date, our Riverview server supplied the deadline, and the local days_between tool did the arithmetic. The model did not need to know where each tool came from.

Because this answer depends on today’s date, it changes every day you run the notebook. Keep that in mind for the next section: when an answer depends on time, a fixed expected answer will not work.

Other ways to give an agent tools¶

Lecture 7 listed five ways to give an agent tools. We built custom tools and used MCP. Briefly, the other three:

• Framework integrations. LangChain and similar frameworks ship ready-made tools for popular services (search engines, Wikipedia, databases). They save writing a wrapper, but you depend on someone else’s description and error handling (see LangChain’s list of tool integrations).

• Skills. A folder of instructions (plus optional scripts and examples) that the agent reads only when a task needs it, like a short manual. The agent sees only each skill’s name and one-line description up front; when a request matches a description, it opens the skill’s full instructions, and any scripts or examples, with its file tools. Skills add know-how without adding more tools to every request.

• Generic HTTP tools. One tool that can call any URL, for example the official mcp-server-fetch server, which downloads any web page. It is flexible but it is an open door: a fetched page can contain instructions that try to hijack the agent (prompt injection), and a tool that can send data anywhere can leak private information. If you use one, fence it: allow only specific domains, default to read-only requests, and require human approval for anything that changes data.

Agents can also write and run their own code as a tool. That is powerful and needs a sandbox.

Keep the tool set small. Every tool’s description is sent to the model on every call. More tools mean more tokens, more cost, and more chances to pick the wrong one.

7. Evaluate the Agent¶

An agent that works on the three questions we happened to try is not the same as an agent that works. In the Evals lecture we saw how to turn “it seems fine” into evidence:

• End-to-end checks against a known correct answer, when one exists (objective);

• an LLM judge with a rubric for qualities that need judgment (subjective);

• error analysis: reading the traces of the failures to find where they went wrong.

We will do all three on a small test set. Because our tools use fixed data, most checks are objective and free: plain Python, no extra model calls.

Step 1 — A small sample test set¶

Each test case is a question plus what a correct run looks like: the expected answer, how to check it, and which tools the agent should use.

We use four cases. The agent handles two of them well. The other two expose problems. A real evaluation test set should try to cover all types of cases, instpect all behaviors and expose problems.

Four cases is far too few to trust a percentage, but it is enough to learn the method. Real test sets grow over time: every failure you find in use becomes a new test case.

A note on the last two cases:

• spring_deadline: the calendar covers only Fall 2026, so the correct answer is that the Spring 2027 date is not available. The case lists phrases that count as saying so, and a forbidden pattern: the Fall 2026 date must not be given as the Spring 2027 date. Mentioning the Fall date to explain what is available is fine.

• days_left_no_clock: the answer depends on today’s date, and unlike the MCP agent in Section 6, this agent has no clock. The correct behavior is to say it does not know today’s date (or ask for it), and the forbidden pattern catches a guessed number of days.

If you give an agent a clock, a fixed expected value would be wrong tomorrow. Compute it in code when the test runs instead, for example (date(2026, 11, 6) - date.today()).days.

Each case also has a tool-call budget (max_tool_calls): the fewest tool calls a correct run needs. A run can get the right answer and still waste calls; the budget check catches that.

Step 2 — Run every case and keep the traces¶

run_case runs the agent on one question and saves everything we need to grade it: the final answer, the full trace, the tools used, and any error. If the run crashes, we record the error instead of stopping the whole evaluation.

Step 3 — Three checks that cost nothing¶

Each check answers one question about a run:

1. Answer: is the final answer correct? Numbers are compared with a small tolerance; for the abstain cases, the agent must say the information is unavailable and must not state the value it cannot know.

2. Tools: did the agent use the tools it needed, and only those? This checks the trajectory, not just the answer: an agent that gets the GPA right by doing arithmetic “in its head” passes the answer check but fails this one.

3. Budget: did it stay within the case’s tool-call budget (max_tool_calls)? Extra calls cost time and money even when the answer is right.

Now run the whole test set on the agent from Section 5. This makes a few model calls per case, so it takes a minute.

Read the table row by row before looking at the totals. A failure in the tools column with a pass in the answer column is worth noticing: the answer was right, but not for a reason you can trust.

Step 4 — An LLM judge for a subjective quality¶

Some qualities have no exact expected answer. Here we ask a question our string checks cannot: is every fact in the answer supported by what the agent actually saw? A fluent answer can still slip in an invented detail.

We use the model as a judge with a short rubric and ask for a structured verdict. Two cautions from the lecture:

• the judge is itself a model and can be wrong, so we spot-check it against human labels;

• using the same model as both agent and judge risks it “agreeing with itself”; in practice teams often use a different (or stronger) model as the judge.

Spot-check the judge¶

Pick three cases, read their traces yourself (use print_trace(record["steps"])), and record your verdict in MY_LABELS (True = grounded, False = not grounded). Then compare. If you and the judge disagree often, fix the rubric before trusting the judge on more cases.

Step 5 — Error analysis: where did it go wrong?¶

A failing check tells you that something went wrong. The trace tells you where. For each failing case we label the first failure stage, using the same labels as the final project:

(A RAG agent would also have a retrieval stage.) The helper below suggests a stage from the checks, but the suggestion is only a starting point: read each trace and correct the label if needed.

The tally tells you which part of the harness to work on next. Many routing failures point at tool descriptions or the system prompt; tool failures point at error handling or tool design; generation failures point at the instructions for the final answer.

8. Iterate: Change One Thing, Measure Again¶

Improving an agent is an experiment: form a hypothesis, change one component of the harness, re-run the whole test set, and compare. Changing several things at once makes it impossible to know which change helped. Re-running every case catches regressions, where a fix for one case breaks another.

Our error analysis gives us a target. In spring_deadline, the agent looked up the withdrawal deadline, received 2026-11-06, and reported it as the Spring 2027 deadline. The observation was correct; the agent misapplied it. Look at what the tool returned: a date and a note, with nothing saying which term the date belongs to.

Hypothesis: if the calendar tool’s result states its term, the agent will notice that Fall 2026 is not Spring 2027 and say the Spring date is not available.

We change only the tool’s output, adding "term": "Fall 2026". The model, the system prompt, the other tools, and the test cases stay the same.

Read the new trace for the case we targeted:

Did the evidence support the hypothesis? Compare the two columns, the changed cases, and the new trace.

Notice where the fix happened. The failure showed up in the final answer (a generation failure: the observation was right, the answer was wrong), but the cheapest fix was in the tool. Tools that return self-describing results (the term, the units, the source) make the model’s job easier. Error analysis tells you where a failure appears; deciding where to fix it is a design judgment.

Notice also what did not change: days_left_no_clock still makes more tool calls than it needs. One change targets one problem. That case needs a different change, for example giving the agent a clock as in Section 6, and its own experiment.

Check your checks, too. A check may fail too. Notice that an answer that mentions the Fall date while explaining that the Spring date is unavailable could fail while being correct if the check misses the correct meaning of the answer. Whenever a result flips (or fails to flip) unexpectedly, read the trace before trusting the number. That is the case for spring_deadline here: the new trace shows a correct answer (it says the Spring 2027 date is not available and does not pass off the Fall date as the Spring one), yet the table still marks the answer as FAIL, because the wording matches none of the case’s abstain_phrases. The agent is fixed; now the check needs fixing. Try it: add phrases such as "not for", "not listed", and "not provided" to the case’s abstain_phrases in Section 7, re-run Sections 7 and 8, and compare again. Then ask yourself whether a list of phrases can ever cover every correct wording, and when an LLM judge (Section 7, Step 4) would be the better check.

Keep three cautions in mind:

• Four cases is a small sample. One case flipping can be noise; run again, or add cases, before drawing a strong conclusion.

• One change at a time. If you change the tool and the system prompt, you cannot tell which one mattered.

• Do not tune to the test set. If you keep editing until these four cases pass, you may just be memorizing them. Add new cases that represent real use.

This loop, build → evaluate → iterate, is how agents improve in practice, and it is the core of harness engineering: the model stayed the same throughout this notebook, and everything we measured and changed was the harness around it.

9. Optional: Tracing and Evals with LangSmith¶

Everything in Sections 7 and 8 ran in plain Python, which is the best way to learn what an evaluation does. Teams that run many experiments usually add an observability platform that records every run and stores test sets and results. LangSmith is LangChain’s. It records each model call, tool call, latency, token count, and retry, including the steps our message-based trace cannot see (like the hidden STAT 205 retry).

This section is optional and needs a free LangSmith account and a second API key. LangSmith tracing is not reliable in Google Colab, so run this section locally.

Get a LangSmith API key: sign in at https://smith.langchain.com, open Settings → API Keys, click + API Key, and copy it. Add it to your .env file as LANGSMITH_API_KEY=... (never paste keys into code cells), then re-run the “Load the API Key” cells in Section 2.

Inside a tracing_context block, every agent run is recorded automatically; we do not change the agent. (We switch tracing on in code rather than with the LANGSMITH_TRACING environment variable, because LangSmith reads that variable once and remembers it; setting it after the agent has already run in earlier sections has no effect.) Run the outage question again, then open the riverview-course-helper project in LangSmith and look for the failed tool attempt and the retry.

LangSmith can also run an evaluation. We upload our test set as a dataset and reuse the same three checks from Section 7 as LangSmith evaluators. Each evaluator receives the agent’s outputs and the example’s reference_outputs and returns a score. The results appear in LangSmith as an experiment you can compare with later runs.

The checks are the same functions you wrote; LangSmith adds storage, a dashboard, and side-by-side comparison of experiments. The framework-provided versions of these ideas also exist: for example, the agentevals package offers ready-made trajectory evaluators (did the agent call the expected tools?) and LLM-as-judge evaluators.

10. Reflect and Your Turn¶

What this tutorial demonstrated¶

1. The ReAct loop is simple; the harness is not. Our hand-written loop was about 25 lines. Making it dependable required schemas, limits, retries, error handling, and a trace.

2. Frameworks package the harness. create_agent, @tool, and middleware gave us the same loop with far less code, but the pieces did not disappear; they moved into the framework, where we configure them.

3. MCP makes tools portable. A tool behind an MCP server can be used by any compatible agent, and an agent can combine tools from many servers.

4. Evaluation turns “it seems to work” into evidence. Free objective checks on answers and trajectories, a judge for subjective qualities, and error analysis of the traces tell us what to change next.

5. Agent = Model + Harness. The model never changed. Everything we built, measured, and changed was the harness.

Reflect¶

• In Section 7, which check caught the most interesting problem: the answer check or the tools check? Why can an answer be right for the wrong reason?

• Pick one failing case from your run. Was the first failure stage routing, tool, or generation, and what evidence in the trace supports your label?

• Did the experiment in Section 8 support the hypothesis? How confident are you, given four cases?

Your turn¶

• Add a tool and a test case. For example, a get_course_sections tool with meeting times, plus at least two test cases (one that needs the tool, one that should not use it).

• Fix days_left_no_clock: give the agent a clock (a tool for today’s date), compute the expected answer when the test runs, and measure the change.

• Improve a tool description that caused a routing failure, then re-run the whole test set to check for regressions.

• Change the judge: use a different model as the judge and compare its verdicts with yours.

• For your project: consider a human-in-the-loop guardrail that pauses for approval before any action that changes data (LangChain offers HumanInTheLoopMiddleware).

DELETE ---- Remaining housekeeping, whenever you’re ready: ---- DELETE¶

Commit: the notebook, README.md, SETUP.md, pyproject.toml and images/agent_model_harness.png. Leave out the _previous_semester copy. riverview_mcp_server.py: the notebook creates it when run. Commit it, or add it to .gitignore. Retire the Hugging Face tutorial: update the course book’s table of contents and submodule, the hub README, and the tutorial order in Agents_course/CLAUDE.md.