Designing Your AI Strategy

Choosing deliberately where AI adds value, when not to automate, and how to stay current without chasing every product.

About 15 minutes

By the end of this lesson you should be able to

Weigh a task against criteria including consequence, reversibility, review burden, and value
Classify candidate tasks as keep human or conventional software, AI + review, or automate with controls, and justify each choice
Name the evidence, stop conditions, ownership, and re-evaluation that an AI-assisted process needs

A strategy is a set of decisions about work

An AI strategy is not a list of products. For an individual, a course, or an office, it is a set of reasoned decisions about which parts of the work AI should do, which parts it should help with, and which parts should stay with people, together with how those decisions will be checked and revisited. Products change every few months. A good decision process does not.

This lesson closes the course, so it draws on all of it: the task and its information, the system around the model, the oversight, the evaluation, and the ways things fail.

Thirteen questions to ask of a task

Group them into four plain questions.

Is it worth doing?

Problem worth solving: is there a real cost now, in time, errors, or delay?
Frequency: is the task done often enough to repay the effort of setting it up?
Expected value: after review time is counted, what is actually saved or improved?
Human alternative: would a template, a checklist, ordinary software, or one more person do as well?

What does it depend on?

Evidence and sources: does it need authoritative, current sources, and can the system reach them?
Information sensitivity: what kind of information is involved, and which tools are approved for it?
Available institutional tools: is there a Stanford-provided service that fits?
Integration and permissions: what would it need to connect to, and act on, under whose identity?

What happens if it goes wrong?

Consequences of error: who is affected, and how badly?
Reversibility: can a mistake be undone, and how quickly would anyone notice it?
Expertise needed for review: can the reviewer actually tell a good output from a bad one?

What will it cost to keep right?

Evaluation burden: can you build a test set and define critical failures (Lesson 4)?
Maintenance burden: who will retest when the model, sources, tools, or rules change?

Three possible decisions

A practical decision aid used by this course, not an exhaustive governance framework. Many processes mix them, with different decisions for different steps.

Keep human / conventional software: Generative AI does not materially help, or should not make the judgment. The work may remain with a person, or a mechanical step may be handled by ordinary deterministic software applying an established rule. Examples: a sensitive conversation; a judgment about someone’s opportunities; work that course rules or learning goals require the student to perform themselves; sorting a list or applying a fixed rule in a registration system.
AI + review: AI assists with a bounded part of the task, such as a first draft, a summary, or a classification, while a person reviews it and makes the decision. Most of the uses in this course belong here.
Automate with controls: A repeatable, evaluated process carries out defined steps without a person approving each one, with monitoring, approval for consequential actions, a safe way to stop, and a named owner. It suits frequent, well-defined, low-consequence, reversible steps.

Moving a task from the second decision to the third should require evidence: an evaluation with no critical failures, a period of review showing few corrections, and a plan for when it breaks. NIST frames the same point as determining “whether the AI system achieves its intended purposes and stated objectives and whether its development or deployment should proceed” (AI RMF 1.0, MANAGE 1.1). (NIST AIRC: AI RMF Core (opens in a new tab); AI RMF 1.0 is being revised.)

How the course fits together

Selected course ideas and the strategic question each answers
Idea	Where	Question it answers
PAUSE	Beginner 6	Should I use AI for this task at all, and how will I check it?
Task design	Intermediate 1	Is the task defined well enough to hand over?
Context	Intermediate 2	Does the system have what it needs, and only that?
Verification	Intermediate 4	How will outputs be checked against sources?
Workflow design	Intermediate 6	Can the steps be written down, reviewed, and repeated?
Systems and tools	Advanced 1–2	What can it see, decide, and change?
Oversight	Advanced 3	Who decides, with what information and authority?
Evaluation	Advanced 4	What evidence shows it improves the task?
Failure handling	Advanced 5	What happens when it goes wrong?

Staying current without chasing every product

Judge a new product by the same questions. A new feature changes the answer only if it changes a criterion: sources, permissions, cost, or evaluation results.
Keep a small test set for each process you rely on, and rerun it when the model, tool, source, or rules change.
Revisit decisions on a schedule, for example each term, and whenever a critical failure occurs.
Watch the governing sources, not the announcements: Stanford’s approved services, your course or office rules, and the provider’s documentation for what you actually use.

Capstone: a public workshop program

A fictional scenario. There is no hidden answer key; the quality of the reasoning is what counts.

A law school unit runs a recurring series of free public workshops. Each has about sixty seats, and demand often exceeds them. The unit wants to improve the program and is considering AI for these tasks:

Summarize public feedback forms after each workshop.
Identify recurring themes across a term’s feedback.
Draft the public announcement for the next workshop.
Sort registrations by date received.
Answer routine public questions (time, place, parking).
Change a registration when someone asks.
Decide who receives the limited seats when a workshop is oversubscribed.
Email participants reminders and follow-ups.

Classify each task as keep human / conventional software, AI + review, or automate with controls, with a one-sentence justification.
What evidence would you collect before expanding automation?
What would make you stop?
Who owns the final process?
What needs periodic re-evaluation?

One well-reasoned response, and where others may differ

Classification.
- Summarize feedback: AI + review. Frequent and useful; a staff member compares the summary with the forms, since a summary can drop the one comment that matters.
- Identify themes: AI + review. Helpful across many forms; a person checks that each theme is supported by actual comments before anyone acts on it.
- Draft announcement: AI + review. Public and reputational; every date, room, and name is checked against the event record.
- Sort registrations: keep human / conventional software. Ordinary software sorts by date without generative AI; check the range and that no entries are lost.
- Routine questions: automate with controls, if an evaluated assistant answers only from the approved listing, declines anything else, and hands off to staff. Otherwise AI + review, with staff sending replies.
- Change registrations: keep human, or automate only through the registration system itself. It changes a record about a person; an AI assistant should at most route the request.
- Allocate limited seats: keep generative AI out of the allocation decision. It decides who receives an opportunity. Apply whatever approved allocation rule governs the program, for example a deterministic first-come rule or a lottery, through ordinary registration software or human administration. The fictional exercise does not establish which allocation rule SLS should use.
- Email participants: keep human / conventional software for fixed reminders sent by the registration system from approved templates; AI + review for any AI-drafted message, which a person approves before it is sent.
Evidence: an evaluation with a test set including missing, stale, and out-of-scope cases and no critical failures; a period of AI + review in which staff record how often they correct outputs; and feedback from participants.
Stop conditions: any wrong date, time, or room sent to the public; any action on a registration nobody approved; a rise in corrections; complaints; or a change in the source or tool that has not been retested.
Owner: a named program manager in the unit, not the tool or its vendor, who approves changes and answers for results.
Re-evaluate: each term, and after any change to the model, tools, listing, templates, or rules; also the classification itself, since a task that looked low-risk can turn out not to be.

Others might reasonably keep routine questions as AI + review until an evaluation exists, or decide that feedback forms are few enough to read without AI at all. What is hard to defend is letting a generative AI system judge who receives seats or change registrations on its own, or expanding automation without evidence.

Finishing the course

You have moved from what AI tools are, through how they work and how to use them well, to how to supervise systems built around them. Three ideas have run through every level:

AI is a tool. Choose it for a task, as you would any other tool.
Human-sounding fluency is not sentience or authority. A confident answer still needs its sources.
Human responsibility does not disappear because the workflow becomes more sophisticated. Someone still owns the decision.

For hands-on practice with agents, the Hub’s Agents Without the Hype goes deeper, and it is optional. For help with a real task, the Law Library is at library@law.stanford.edu.

Resources

The PAUSE Rule — the library’s five-step check before using AI for a task.
Intermediate Lesson 6: Building Repeatable Workflows — the workflow template this lesson assumes.
NIST: AI Risk Management Framework (opens in a new tab) — a voluntary framework for governing, mapping, measuring, and managing AI risk; version 1.0 is being revised.
Stanford UIT: Approved AI Services (opens in a new tab) — which services are approved for which information.
Agents Without the Hype: capstone — optional practice evaluating an agent before delegating to it.