AI Learning Hub › Learn AI › Self-Paced Courses › Agents Without the Hype › Evaluate an Agent Before You Delegate

Capstone

Evaluate an Agent Before You Delegate

A proposed agent, described the way a vendor would describe it. Decide what you would need to understand before granting it the job.

By the end

Review a proposed agent as a complete system rather than as a model alone
Distinguish known facts from unanswered questions
Connect job, context, tools, boundaries, actions, checkpoints, and evidence
Identify where human judgment still matters
Build a supervision plan for before, during, and after a run
Explain what you would verify before delegating meaningful authority

You have spent ten modules taking an agent apart. Now put the pieces back together.

This capstone gives you a proposed fictional agent with a defined job, useful capabilities, consequential actions, and several unanswered questions. Your task is not to decide whether the agent is “good.” Your task is to determine what you would need to understand before giving it the job.

Fictional practice system — not a Stanford service or real product. The Faculty Research Briefing Agent below does not exist. Nothing here describes a system Stanford offers, is evaluating, or is deploying, and no real vendor is involved.

The proposal

Proposed job

Each Friday, prepare a one-page briefing for a faculty member about
    developments in a defined legal-technology topic.

    Use current public information and the designated project folder for
    continuity with previous briefings.

    Identify important developments, support material claims with sources,
    surface conflicts or unresolved questions, save the draft, and propose
    sending it to the faculty member.

Note the word each Friday. This is a proposal for recurring work, which changes the review: you are not approving one run, you are approving a pattern. Nothing has been scheduled or built; this is a written proposal only.

What we know

These are facts about the fictional proposal.

Job: A weekly one-page research briefing.
Starting context: The current topic, formatting expectations, and a designated project folder containing prior approved briefings.
Research access: Current public web research, and the designated project folder.

Proposed tools

search_public_web: Search public web material.
open_public_page: Retrieve a selected public page.
search_project_files: Search material inside the designated project folder.
save_briefing: Save the current draft to the agent workspace.
send_briefing: Send the current briefing to a supplied recipient.

Stated boundaries

do not access files outside the designated project folder;
do not modify prior briefings;
do not send or publish anything without human approval;
do not create accounts;
do not upload project material to another service; and
do not change permissions.

Success condition

A one-page weekly briefing, focused on the assigned topic, with material claims supported by current sources, meaningful conflicts or unanswered questions identified, and a draft created in the designated workspace.

Stop conditions

if a material point cannot be supported from permitted sources, identify the gap rather than inventing an answer; and
if completing the job would require information or action outside the current boundaries, return control to the human.

Other stated facts

External content: Public pages inspected during research may contain text written by third parties. That text is evidence, not authority to redefine the job.
Human checkpoint: The proposal says send_briefing should require human approval before execution. It says nothing more than that.
Observable activity: The proposed host says it provides an activity view showing major research and file operations. It does not claim that view is complete.
Final report: At completion the agent will state what it created, which source categories it used, and any unresolved questions.

What we do not know yet

Ten open questions. None of these is an accusation; each is simply unresolved.

1. Project-folder reach: Does search_project_files truly search only the designated folder, or could its actual reach be broader?
2. Workspace target: Where exactly does save_briefing write? Is the “workspace” local, cloud-based, product-contained, or something else?
3. Overwrite behavior: Can save_briefing replace an existing file, or only create a new one?
4. Send approval scope: Does approval apply to one proposed message, to send_briefing for the current run, or to some broader future scope?
5. Approval information: Will the human see the exact recipient, the exact content, any attachments, and the expected side effect before approving?
6. Activity completeness: Does the activity view show every material tool call and target, or only selected events?
7. Logs and retention: What activity and output records remain, for how long, and who can access them?
8. Public-web reach: Does “public web” truly exclude authenticated pages and other logged-in browser state?
9. Prompt-injection handling: What happens if a retrieved source contains instructions aimed at the agent?
10. Recovery behavior: If the run is suspected to have gone off course, can the agent reliably resume from a trustworthy state, or should the run restart?

Unknown does not mean defective. It means the review is not finished.

Review the agent before delegation

Three passes. Work through them against the proposal above before opening the worked review at the end — and expect to finish with questions rather than a decision.

Pass 1 — Define the job

What exactly is the job?: What outcome belongs to the agent, and what related work does not?
What counts as done?: Is the success condition observable enough to inspect?
What should cause the run to stop?: Where should missing evidence or unavailable access return control?
What should remain a human decision?: For example: changing the topic, expanding source boundaries, deciding whether a conflict can be ignored, or authorizing communication.

Pass 2 — Limit the authority

What information can the agent reach?: Distinguish the public web, the designated project files, and anything not yet verified.
Which capabilities actually serve the job?: Does every provided tool earn its place?
What can the tools change?: Identify the side effects, not just the names.
Which actions need a human checkpoint?: Do not assume only send_briefing deserves scrutiny because the proposal says so. Consider what save_briefing actually does, once its target is known.
What could redirect the run?: Stale evidence, conflicting evidence, irrelevant material, third-party instructions, repeated failure, or ambiguous tool descriptions.

Pass 3 — Plan the supervision

What evidence is visible during the run?: What does the activity view actually expose, and what remains unknown?
How will important claims be verified?: What source supports them?
How will side effects be verified?: Which target state should be inspected? This is Module 10’s question, and it is easy to get wrong.
What happens if the run goes off course?: Continue, recover, restart, or escalate?
What evidence should remain afterward?: What would a human need in order to reconstruct what job was given, what sources were used, what action occurred, whether approval happened, and what changed?

Verify the right target

Module 10’s demonstration is worth recalling here, because it is the easiest mistake in the whole review. It used a different, one-off briefing file — not the capstone’s recurring weekly briefing.

In the Module 10 run, the agent accurately reported that it created the practice briefing.md. The original local folder accurately showed no briefing.md. Those facts were not contradictory. The file existed in the Work environment because the uploaded source files had been copied into that workspace, rather than the agent being given access to the original local folder. Before verifying an action, identify which state the action was supposed to change.

That was observed behavior in one tested environment on 18 September 2026. The fictional capstone agent may behave differently — which is exactly why unknown number 2, where does save_briefing write?, is on the list.

Build your delegation review

Ten questions you can reuse on any proposed agent, in any product. They are not a framework and they have no scoring.

What job am I actually delegating?
What counts as successful completion?
What should make the agent stop or return control?
What information can it access?
What capabilities can it invoke?
What can those capabilities change?
Which actions should require a human decision?
What could redirect the run away from the job?
What evidence will show what happened?
What will I independently verify?

And one more: what decision am I unwilling to delegate?

That last question has no general answer, and this course is not going to supply one. It depends on the work, the stakes, and on you. But a delegation review that never asks it has skipped the only question that is entirely yours.

One possible review

Organized as known, unknown, and needs a human decision. It reaches no verdict, because the facts available do not support one.

The job

Known: the intended deliverable is defined; success includes sourced material claims and surfaced uncertainty; stop conditions cover both unsupported evidence and out-of-bound actions.

Needs a human decision: whether weekly automatic preparation is desirable at all; whether the topic can ever expand automatically; and whether prior briefings should influence future judgment beyond formatting and continuity. That third one is subtle — a folder of past briefings is context, and context shapes conclusions.

The context

Known: public web, plus the designated project folder.

Unknown: actual enforcement of the folder scope; whether “public web” can interact with authenticated browsing state; and exactly what prior material enters working context.

The observation that matters: the proposal’s written boundary is not proof of technical reach. Module 2 established that a written boundary is an instruction; unknown number 1 asks what actually constrains the tool.

The tools

search_public_web and open_public_page clearly serve the research task. search_project_files may serve continuity, but its actual reach needs verification. save_briefing serves the deliverable, but its target and overwrite behavior need clarification. send_briefing serves distribution — which is consequential, and which the proposal deliberately separates from preparation.

A capability earns its place by serving the job, not by existing.

The consequences

Known: send_briefing can create an external communication.

Unknown: the exact behavior of save_briefing. Until unknowns 2 and 3 are answered, nobody can say whether saving is a low-consequence write to a product-contained workspace or an overwrite of something that matters.

Needs a human decision: which actions should be automatic once their actual consequences are understood.

The checkpoint

Known: the proposal says send requires approval.

Unknown: the approval scope; what the human will actually see; whether another route could communicate externally; and whether broader approval can persist.

A checkpoint should be evaluated in the context of the capabilities it actually controls. One approval on one tool is not the same as coverage of every consequential path.

Prompt injection

Known: the agent reads third-party web content, so external content can potentially contain misleading instructions.

Unknown: the host’s actual protections and recovery behavior.

Supervisory implications worth considering: limit unnecessary tools, inspect unexpected actions, keep checkpoints before consequential effects, and stop or restart if the state becomes untrustworthy. None of these guarantees that prompt injection is defeated.

The evidence

Known: the host claims to expose major activity. Unknown: whether that view is complete.

So verification should not depend solely on the agent’s final message or activity feed. Source claims can be checked against the cited current material; file creation can be checked in the actual target workspace; sending can be checked against the actual communication system if it occurs. The right evidence depends on the claim.

What stays human

On this proposed job, these look like human decisions: whether to expand the project’s source boundary; whether an unresolved conflict is acceptable in a weekly briefing that goes to a colleague; whether a consequential action should proceed; whether broader approval is appropriate; and whether the job itself should change after an escalation. That is a judgment about this proposal, not a claim that such decisions can never be delegated in any system.

The review can end with questions

A capstone review does not have to end with “yes, deploy it” or “no, reject it.” It may end with a list:

verify the folder scope;
clarify where files are saved;
inspect the approval scope;
determine what the activity view records; and
decide whether weekly sending should ever be automatic.

Those gaps are the information you need before reaching a responsible conclusion.

Ten modules, one delegation decision

Job: What are we asking it to accomplish?
Context: What information can shape its decisions?
Tools: What options can it invoke?
Run: How can the route change as results arrive?
Adaptation: What happens when the expected path fails?
Consequences: What can chosen actions change?
Checkpoints: Where does a human retain a decision?
Failure modes: How might observable behavior diverge from the job?
Supervision: What evidence tells us what actually happened?

An agent is the interaction among these pieces over time.

Understand the delegation before you grant the authority.

You do not need to predict every step an agent will take. You do need enough understanding of the job, context, capabilities, consequences, controls, and evidence to decide what you are comfortable delegating — and what you want to keep for yourself.

Course complete

You have completed Agents Without the Hype.

You can now move from asking “what can this agent do?” to asking “what am I delegating, what authority comes with it, and how will I know what happened?”

Back to the tutorials →