From Demo to Decision: Designing a 30-Day AI Agent Pilot
An AI agent can draft an answer, inspect a file, invoke a tool and complete a sequence that feels almost magical. A successful demonstration does not prove that the system deserves investment, and it certainly does not prove production readiness. A useful pilot must answer a bounded business question: does a defined workflow become faster, more accurate, safer and economically better when an agent participates? Reaching that answer requires a pilot designed as an evidence system. The team defines the process, baseline, authority boundaries, failure scenarios, quality and cost measures, then states in advance what will cause it to continue, change or stop. Thirty days is enough for meaningful learning when the problem is narrow and model capability is not confused with product readiness.
1. Define the decision before selecting a model
The pilot should not prove that AI agents are important. It should enable a concrete decision: whether to automate a particular process, at what scope, under which controls and with what budget. The first document is therefore not a prompt but a decision brief. It describes the current process, the people who perform it, the required result, its failure points and the cost of error. It then names real alternatives: extend the pilot, keep the agent as an assistant, redesign the workflow or stop. A useful decision has an owner and a date. Without that frame, every improvement can be called success and every fault can be dismissed as temporary. The pilot becomes a technology performance instead of a management instrument. Choose a question that a month can answer—for example, whether an agent can prepare a sourced analysis draft for human approval—not whether an entire department can be replaced.
2. Choose a workflow with value, boundaries and available evidence
The first use case should matter enough to justify learning while remaining limited enough for risk to be controlled. A suitable process has lawfully available inputs, an output that can be evaluated, an operator who understands the work, and actions that can be paused or reversed. Document research, enquiry triage, proposal preparation, reconciliation review or a draft report can work when a person still owns the decision. Irreversible payment, permission changes, medical decisions or legal commitments are poor starting points without a much deeper control layer. Study the process as it exists, not its idealised description. If input arrives through email, PDFs and a manual spreadsheet, the pilot must meet that complexity. Artificially cleaned data can prove that a model works in laboratory conditions while concealing that the product cannot fit the real operation.
3. A baseline turns enthusiasm into comparison
Before the agent runs, measure how people complete the task today. Total completion time, active work, system handoffs, correction rate, material errors, waiting time, cost and perceived quality form the baseline. Separate averages from distributions: a process may look fast most of the time but stall on complex cases. Sample difficult cases as well as convenient work. Then define a meaningful improvement. Saving one minute in an hour-long process may not justify integration, while removing several manual transfers may create value even if runtime is unchanged. Measures need outcomes and guardrails: speed beside accuracy, automation beside escalations, model cost beside human review time. The primary metric is end-to-end task success, not the number of model calls or the fluency of the text displayed on screen.
4. Separate the model, harness, tools and state
An agent is a system, not a single model. OpenAI distinguishes among a managed agent runtime, the Agents SDK where an application owns tools and storage, and direct work through the Responses API. That choice changes where work runs, how state is managed, how approvals and traces operate, and who carries operating responsibility. A pilot should draw four layers: the model that interprets and decides; the harness that manages the loop; tools that read or change systems; and state preserved across steps. Each layer needs an owner and a contract. A tool needs a clear schema, bounded input, verifiable output and predictable failure behavior. State needs retention and deletion rules. The harness must know when to stop, request approval or retry. Separation lets a team change models without rewriting the business process, reduce permissions without editing a prompt, and diagnose whether a failure came from reasoning, data, a tool or orchestration.
5. A tool is authority, not merely a function
The Model Context Protocol specification dated July 28, 2026 defines a standard way to connect LLM applications with data sources and tools. It also stresses the security and trust implications of arbitrary data access and execution. In product terms, every tool call is a request for authority. Reading inventory, sending a message, updating a CRM record and cancelling an order do not carry the same risk. Build a permissions matrix: which tools exist, who may expose them, what scope each run receives, which parameters are accepted and what requires approval. Default to read-only and least privilege. Keep credentials in a protected execution layer rather than placing them in model context. A sensitive action should have a readable preview, explicit approval and an idempotency key that prevents duplication. MCP recommends a human in the loop with the ability to deny tool invocations; the pilot should measure how many approvals were requested, denied and supported by enough context for a real decision.
6. Build the evaluation set before the first iteration
Manually reviewing a few conversations is not enough for a nondeterministic system. Create a small but representative dataset containing ordinary tasks, edge cases, missing input, conflicting instructions and prompt-injection attempts. Give every case an expected result and separate criteria: whether the right tool was selected, parameters were correct, sources supported the conclusion, escalation occurred when necessary and the final action was permitted. OpenAI’s guidance for evaluating agent workflows recommends starting with traces while behavior is still being debugged, then moving into repeatable datasets and eval runs once the team knows what good looks like. That distinction matters. An evaluator should score the path, not only the final answer. An agent can reach the correct result through a prohibited tool or a dangerous guess. Re-run the same set after every material prompt, model, tool or routing change to detect regression instead of relying on selective memory of the latest demo.
7. Tracing is a product and operating layer
When a workflow has several steps, the final response is insufficient for investigation. A trace should connect input, the agent’s decision, tool call, result, guardrail, handoff, time and cost. OpenAI documents tracing as an end-to-end record of model calls, tools, handoffs and guardrails, and the same traces can support grading. Decide in advance which information is recorded and which sensitive fields are redacted. The objective is not to collect everything but to reconstruct why the system acted. Use a small failure taxonomy: wrong tool, bad parameter, missing source, hallucination, insufficient permission, loop, timeout, incorrect human approval or unavailable destination. Each class gets an owner and response. Operating measures include time per task, number of steps, cost, retry rate, escalation rate and human review time. Without traces, a team may tune the prompt when the actual problem is API latency or stale source data.
8. Design sandboxing, state and recovery
Useful agents inspect files, execute code and modify artifacts. OpenAI’s April 15, 2026 Agents SDK update highlighted controlled sandbox execution and separation between harness and compute, including keeping credentials away from model-generated code and supporting durable work through snapshots and rehydration. A small pilot should adopt the principle: an ephemeral execution environment with a bounded filesystem, restricted network, runtime limit, resource quota and secrets outside code access. Business state stays outside compute so a run can be stopped and resumed without losing the authoritative record. Deliberately test a timeout, lost container, failed tool and partial response. Recovery must prevent duplicate actions, identify what has already completed and show a person where continuation begins. An agent that cannot fail safely is not an autonomous product. It is a complex script with a wider failure surface.
9. Measure risk as part of value
The NIST AI Risk Management Framework uses four functions—Govern, Map, Measure and Manage—as a continuing approach to risk. A pilot can translate them into practical work. Govern means owners, policy, documentation and a stop mechanism. Map means users, context, data, affected parties and foreseeable misuse. Measure means evaluations, security tests, accuracy, bias, privacy and cost. Manage means control choices, exceptions, remediation and monitoring. NIST states that AI RMF 1.0 is being revised, so it should not become a frozen checklist. Its value is an organisational language connecting product, engineering, security, legal, operations and users. Define a harm budget beside the value budget for each case: the maximum plausible damage, the number of records or transactions exposed and the time required to detect and contain it. Expansion then depends not only on success rate but on the organisation’s ability to control failure.
10. A four-week operating plan
In week one, map the workflow, measure the baseline, select 30–50 historical tasks and set decision gates. In week two, build a narrow happy path with read-only tools, structured output and complete tracing; run it outside systems that can change production. In week three, add edge cases, guardrails, approvals and one low-impact write tool. Repeat evaluations, threat scenarios and recovery tests. In week four, run in shadow mode beside the human process or with a tightly limited population, collect total cost and prepare a decision memo. Hold a short weekly review with an operator, engineer, risk owner and business decision-maker. A polished interface is unnecessary if it is not part of the question; a simple shell can be enough when the measured experience represents real work. Approvals and logging cannot be postponed, however, because they are part of the product being tested rather than production decoration.
11. Go, iterate or stop
Go is justified when the agent meets quality and safety thresholds on a representative set, improves a business measure, has acceptable total cost and can be operated through exceptions. Iterate fits when value is visible but a defined weakness needs a change—perhaps a tool is too broad, human review remains high or one document class performs poorly. Stop is a professional outcome when there is no improvement over baseline, supervision eliminates the saving, safe authority cannot be granted or the underlying process is too unstable to automate. The decision should contain evidence: a metrics table, representative traces, open failures, cost at expected scale and a transition plan. If the project continues, assign an owner, service objectives, monitoring, versioning, periodic review and a manual fallback. If it stops, retain the dataset and lessons; they may become useful when the process or technology changes.
12. The real deliverable is a decision system
The ecosystem is moving quickly. Better tools, MCP, sandboxing and observability make agents more capable, while the same progress increases the need for product discipline. An agent does not create value because it performs more steps. It creates value when it improves an outcome the organisation can measure while respecting boundaries people can understand. Dor Arad’s product perspective follows the logic of any serious engineering pilot: begin with a decision, expose the system to real constraints, measure separate layers and rehearse failure before expansion. AI adds behavioral uncertainty, which makes datasets, traces and approvals architectural components. A 30-day pilot does not promise production in a month. It promises something more useful: enough evidence for leadership to know what to build, what to constrain and what should not be built at all.