Before You Buy an AI Agent, Ask These Three Questions
The demo will work. Vendor demos always work, because they run on data the vendor controls and a workflow the vendor picked. Your business is neither of those things. What I watch owners do is buy the agent first and decide what it was supposed to improve second, which is backwards. It is why a pilot that looked brilliant in March gets switched off in July with nobody willing to say out loud that it failed.
The conversation you run before you sign is the project. Three questions decide whether you bought a tool or a twelve-month distraction. What number says it is working, what does it need to touch, and who switches it off when it goes wrong?
You can find those three questions on a dozen listicles by lunchtime. None of them show the framework being executed inside an actual small business, with the workflow step named, the success number attached and the off switch agreed before anyone flips it on. I have helped more than 300 businesses in medical, ecommerce, local services, fintech, education and real estate, and scoping and running AI automations is a standing part of that work. The procedure below runs from whether an agent fits the job at all to the day you keep it or switch it off. It holds for any agentic AI purchase in a small business.
The bill for a bad agent purchase arrives long after the demo
The license fee is the small number. The expensive part is the operations person you assign to babysit the rollout and the four months of status meetings about why the outputs still need checking. Meanwhile whatever was actually costing you money stays untouched while everyone talks about the agent.
Gartner said in a June 2025 press release, reported by Robert J. Szczerba in Forbes in July 2026, that more than 40% of agentic AI projects will be canceled by the end of 2027. Read the stated reasons before you read the number: escalating costs, unclear business value, inadequate risk controls. Not one of those is a model problem. All three are decisions somebody either made badly or never made at all, in a room, before the contract was signed.
Szczerba put the diagnosis in one line.
The ones that fail rarely die because the models were too dumb to do the work. They die because companies turn agents loose without a success metric, without access to the right data, and without a plan for what happens when the thing goes sideways. The coming cancellation wave is a management problem wearing a technology costume.
Robert J. Szczerba, Forbes, July 7, 2026
Three failures, three fixes, and every one of them is free before you pay anyone. A success metric, the right access and a plan for going sideways are answers a business owner gives. If the owner cannot give them, no vendor on earth can supply them afterward.
Does the job need an AI agent at all?
A regular AI tool answers. You ask, it responds, and you decide what happens next. An AI agent is given a goal and takes the steps to reach it on its own, using your other software along the way. "Write me an email" is a job for a tool. "Find everyone who hasn't paid, draft the right reminder for each, and send them" is a job for an agent, including the parts that can go wrong without you watching.
That autonomy is the whole pitch and the whole risk. A chatbot that writes a bad sentence wastes ten seconds. An agent that emails the wrong reminder to the wrong client, or pays the wrong invoice, creates a real mess in the real world.
A job suits an agent when the task is repetitive and well defined, a mistake is cheap and reversible, and you can check the work quickly. Sorting incoming emails, drafting replies for your approval, pulling data from one system into a report and flagging a number when it crosses a line all pass that test. Most of them stop short of taking an irreversible action on their own.
Check what you already pay for before you buy a platform. Several mainstream tools now bundle basic automation, scheduling assistants and inbox triage into plans you probably already have. Testing one of those features under supervision keeps your risk and your spend low while you learn what the technology is good at inside your business. The cheapest agent to test is the one already inside a tool you trust.
Gartner counted thousands of vendors and found about 130 building the real thing
The Gartner research behind that 40% forecast, covered in the same Forbes writeup, also looked at who is selling agents. Of the thousands of vendors claiming agentic capability, Gartner judged only around 130 to be building something that deserved the label. Everyone else was repackaging. Agent washing is the industry term for putting the word agent on a chatbot, a scripted automation or a wrapper around someone else's model.
You do not need to know what is under the hood to catch this. You need to know what a real agent does differently in practice. Every difference shows up in a question you can ask in plain English on a sales call.
| What you are checking | Something that earns the word agent | A chatbot in a costume |
|---|---|---|
| How work starts | Runs a multi-step job from a trigger in your business, then reports what it did. | Waits for a human to type a prompt, one step at a time. |
| Contact with your systems | Named read and write permissions in the exact systems the job touches, granted by you. | Demoed on sample data; integration described as coming soon or handled in onboarding. |
| Definition of working | One business number the vendor will be judged on, agreed in writing before go-live. | Usage stats: messages handled, tasks completed, seats active. |
| Behavior when it is wrong | Stops, logs, escalates to a named person, and can be switched off by your staff in minutes. | Keeps going, and you find out from a customer. |
| Where it breaks in production | Vendor names their own past production failures unprompted: a missing invoice field, a duplicated customer record, a permission wall the demo never hit. | Vendor insists it just works; you discover the edge cases yourself after go-live. |
The last row separates vendors with real deployments behind them from vendors with a sales deck. Szczerba's examples of how demos die in production are mundane. A field that is empty in your data and full in theirs. A customer who exists twice because your front desk spells the name two ways. A permission wall the sandbox never hit. Ask a vendor which of those killed their last deployment and watch what happens to the room.
Three questions, one sentence each, or keep your money
I do not ask about model architecture, and neither should you. The vetting works because these three questions are unanswerable by anyone who has not thought about your business, and answerable in one sentence by anyone who has.
Question one flushes out the vendors selling activity. A good answer sounds like a sentence with a noun and a direction in it, something like quotes sent within one business day, up from six in ten to nine in ten. A bad answer is a dashboard. Then ask who inside your business agreed that this is the number, because a target nobody owns is a target nobody defends. If the answer includes the phrase time saved, ask whose time, doing what instead, and how you would see it in the P&L. Self-reported hours are the easiest number in business to feel and the hardest to bank. I have written up how to prove an AI time saving is real, and the short version is that a saving nobody redeployed is not a saving.
Question two is where most small business deployments actually die, and the reason is unglamorous. The data the agent needs lives in three places, one of which is a spreadsheet on someone's desktop. Ask what systems it reads, what it writes into, and who has to approve that access. Then ask the follow-up that vendors hate, which is what the agent does when the field it expects is empty. Access is also a risk question. Opening your systems to an autonomous tool is a different exposure from opening them to a person who can be fired. Agent-ready and agent-safe are not the same thing, and the gap between them is where the quiet incidents happen.
Question three is the one owners skip because it feels pessimistic on a sales call. Skip it and you have bought a system that fails silently. Silence is expensive because the cost accumulates in customer trust, which shows up in your numbers a quarter later with no obvious cause.
The first deployment should be one step, one number, one off switch
The listicles stop at the questions. What you do with the answers decides the outcome. When I scope an automation for a client, the sequence never changes, and most of the elapsed time goes to waiting for data.
Take a physio clinic with two therapists and one person on the front desk. It is an illustration I made up, and the clinic is not a client of mine. The circled step is the rebooking follow-up after a discharge visit, one step with a clear start and a clear end. Patient communication as a whole is too big to circle. The number is the share of discharged patients who book their next appointment within seven days. It comes from the practice software for the two weeks before anything changes, and the clinic manager is the name attached to it. Access is read on the appointment calendar and write into a drafts queue only. It gets no access to payments and cannot send anything without approval for the first two weeks. The rollback is the manager revoking one integration key, triggered by any clinically inaccurate message or any message going to a patient flagged as discharged from care.
Everything about that scope is boring, and boring is the point. If the number moves, you have a result with a baseline behind it and a manager who will defend it. If it does not move, you have lost two weeks and one key.
I scope this narrowly because I have seen what happens when the constraint moves. When I filled a London ADHD clinic's calendar for three straight months, their bottleneck stopped being demand and became capacity. They had to hire more specialists and outsource the overflow. If you automate a step that was never the constraint, you have bought a faster version of something that was already fine. You will feel busy while the number that matters stays flat.
What to measure after go-live
MIT's NANDA research, reported by Fortune on August 18, 2025, found that about 95% of enterprise generative AI pilots produced little to no measurable impact on profit and loss. Only around 5% saw rapid revenue acceleration. That study rests on 150 leader interviews, a survey of 350 employees and 300 public deployments. Treat it as a serious signal. Assume by default that your pilot belongs in the 95% until you can point at a number that moved.
Aditya Challapally, the lead author, described what the winners did differently.
Some large companies' pilots and younger startups are really excelling with generative AI. It's because they pick one pain point, execute well, and partner smartly with companies who use their tools.
Aditya Challapally, MIT NANDA lead author, quoted in Fortune, August 18, 2025
One pain point is step one of the scoping sequence, reached independently by researchers looking at hundreds of deployments. It is also the habit behind the small businesses actually winning with AI, who picked one boring, repeatable task and learned to manage the machine doing it. A 2026 Writer.com adoption survey points the same way. It found 97% of executives had deployed agents in the past year while only 29% saw significant return. It is one vendor's survey of its own market, and I read it as directional.
Your measurement plan needs three dates and one number. Day 30, does the single agreed number show any movement against the pre-agent baseline. Day 60, is the defect log shrinking or is your team still correcting the same category of mistake. Day 90, keep, rescope or kill, decided against criteria you wrote down before go-live, with sunk cost kept out of the room.
The traps are metrics that measure the agent and never touch the business. Tasks completed tells you the thing ran. Messages handled tells you volume, which goes up when quality goes down. Self-reported hours saved is the worst of them, because everyone reports a saving and nobody can find it in a bank statement. Adoption rate measures whether your staff logged in. A clinic manager would defend none of these in a meeting, and that is the test. If nobody would argue about a number, it is decoration.
The price tier can mislead you too. Plenty of owners buy the premium "fully autonomous" tier, then sensibly review everything it does anyway. That is paying agent prices for chatbot value. If you are checking every action, you do not need the agent tier yet. Buy it once the defect log has earned the trust to let the agent run.
Then run the full tally before you call it a win: savings, minus the subscription, minus your supervision time, minus the cleanup when it gets something wrong. Subscription fees run whether or not the agent earns them. A tool that handles forty tasks well and botches the one high-stakes task each month can still be a net loss once you price the cleanup. Vendors quote the best case. Your books should record the average one.
Frequently Asked Questions
How do I tell if an AI agent vendor is selling a real agent?
A chatbot answers and leaves the action to you. A real agent takes the steps toward a goal on its own inside your systems. Ask what it reads, what it writes into, and what it does when a field it expects is empty. Gartner reviewed thousands of vendors claiming agentic capability and judged only around 130 to be building something that deserved the label. A vendor with real deployments will describe a specific production failure without being pushed, usually something small like a duplicated customer record or a permission wall. A vendor who cannot name one has only ever run demos.
What should the first AI agent in a small business actually do?
One step in one workflow, with a clear start, a clear end and a number already being tracked. The follow-up message after a visit is the right size, and so is drafting the weekly quote chase for a person to approve. Customer service or marketing as a whole is too big. MIT's NANDA research found the pilots that succeeded picked a single pain point and executed it well. Pick the step that costs you money when it gets skipped, and confirm it is your actual bottleneck before automating it.
How long should I run an AI agent before I decide it failed?
Ninety days, with checkpoints at 30 and 60, and the kill criteria written before go-live. Spend the first two weeks with the agent drafting and a human approving every output, and log each correction. That log tells you whether the errors are shrinking or repeating. At day 30 you are looking for any movement in your one agreed number against the baseline you measured before switching it on. If day 90 arrives with no movement and the same defects, switch it off and keep the learning. Gartner puts most cancellations down to escalating costs and unclear value, and dragging a dead pilot into year two is how the cost part happens.
Every vendor selling you an agent asks you to trust their technology. The three questions turn that around and ask whether your own business is clear enough to automate, and most are not yet. Run the same questions on your next hire, your next agency and your next software purchase. A step nobody can measure and nobody owns was already broken before anyone mentioned AI. If you want me to look over a scope before you sign, book a call and bring the vendor's proposal with you.
