- Published
- Updated
- 11 min read

TL;DR: AI that works runs on a schedule with real data, logs what it did, hands unclear cases to a person and moves a number you already track. AI that sounds impressive needs someone at the keyboard. Research from MIT, S&P Global, Gartner and RAND shows most pilots stall for workflow reasons, not model reasons.
Updated October 2026: an earlier version said Gartner expects half or more of generative AI projects to die after proof of concept; Gartner's figure is at least 30%, and two unsourced claims were removed.
Most of the AI you get pitched is a demo. AI that works looks different, and it is usually more boring.
Someone opens a chat window, types a clean prompt, and the model produces something that looks sharp in a meeting. People nod. Budget gets set aside. Six months later nothing has shipped that moves revenue, cost or error rate.
That gap is the whole story. AI that works is not about how clever the model sounds on stage. It is about whether a workflow runs without you babysitting it, and whether you can point to a number that changed.
What do the numbers say about AI that works?
The research lines up. Most AI projects stall, and the reasons are mostly about the business around the model.
MIT: about 5% of pilots move revenue fast
MIT's NANDA initiative studied generative AI in companies through 150 leader interviews, a survey of 350 employees and an analysis of 300 public deployments. Fortune's summary: "about 5% of AI pilot programs achieve rapid revenue acceleration; the vast majority stall, delivering little to no measurable impact on P&L." (Fortune (opens in a new tab))
The researchers did not blame the models. They pointed to a "learning gap" for both tools and organizations, and to "flawed enterprise integration." In plain words, the tools never attached to how the business already runs, so they never moved from interesting to useful.
S&P Global: more companies are killing AI projects
S&P Global Market Intelligence surveyed more than 1,000 people in North America and Europe in early 2025. The share of companies abandoning most of their AI initiatives "jumped to 42%, up from 17% last year," and "The average organization scrapped 46% of AI proof-of-concepts before they reached production." Cost, data privacy and security risks topped the list of obstacles. (CIO Dive (opens in a new tab))
That reads like bad news. I read it as buyers getting stricter about AI that works. A demo that never earned a production job is getting cut, and that is healthy.
Gartner: at least 30%, then over 40% for agents
In July 2024 Gartner said "At least 30% of generative AI (GenAI) projects will be abandoned after proof of concept by the end of 2025, due to poor data quality, inadequate risk controls, escalating costs or unclear business value." (THE Journal, quoting Gartner (opens in a new tab)) The old version of this note rounded that up to "half or more." It is 30%.
For agents, Gartner went further in June 2025: "More than 40% of agentic AI projects will be canceled by the end of 2027." (MarTech (opens in a new tab)) The reasons Gartner gave were cost, unclear value and weak risk controls. None of them is "the model is not smart enough." Every one of them is something AI that works has to plan for.
RAND: the root causes are familiar
RAND interviewed 65 experienced data scientists and engineers about why AI projects fail. It notes that "By some estimates, more than 80 percent of AI projects fail," twice the rate of IT projects without AI. (RAND (opens in a new tab)) That is an estimate RAND cites, not one it measured, so treat it as a range rather than a fact.
The five root causes are the useful part. Leaders misunderstand or miscommunicate the problem. The data is not good enough. The team chases "the latest and greatest technology" instead of the user's problem. There is not enough infrastructure to manage data and deploy models. And some problems are simply too hard for current AI. If you have lived through a few pilots, every one of those sounds familiar.
Stanford: impressive is not the same as reliable
Stanford's 2026 AI Index calls it "the jagged frontier of AI." A model earned a gold medal at the International Mathematical Olympiad, yet "the top model reads analog clocks correctly just 50.1% of the time." Agents went "from 12% to ~66% task success on OSWorld," a test of real computer tasks, "though they still fail roughly 1 in 3 attempts." (Stanford HAI AI Index 2026 (opens in a new tab))
One in three is fine for a draft a person reviews. It is not fine for a task that runs unattended and touches a customer. That is why, for AI that works, the design around the model matters more than the model.
What does demo-ware look like next to AI that works?
Demo-ware needs a human. Someone has to click run, clean the sample data, or rephrase the prompt when the answer goes sideways. Success is a reaction in the room. The data is curated, the path is happy-path only, and there are no retries, alerts or logs you would trust at 2 a.m.
AI that works looks almost dull by comparison:
- A trigger fires. A form submission, a new invoice, a schedule. Nobody has to remember to start it.
- A queue picks up the work. Jobs wait their turn instead of disappearing when a service is slow.
- Failures are handled. The system retries, falls back, or sends the case to a person with a clear reason.
- A person sees the exceptions. Anything the model is unsure about goes to someone who can decide.
- A number moves. Hours saved, tickets closed, leads answered, errors down.
Where the budget sits tells you a lot too. MIT found that "More than half of generative AI budgets are devoted to sales and marketing tools," yet the biggest return showed up "in back-office automation." (Fortune (opens in a new tab)) The screenshots live in sales. AI that works tends to live in operations.
Chat is a demo interface. A trigger, a queue, logs, a human path and a metric make AI that works.
When does an impressive answer become a liability?
Air Canada's chatbot is still the clearest case. A customer asked about bereavement fares after a death in the family. The chatbot told him he could buy full-price tickets and apply for the reduced rate "within 90 days of the date your ticket was issued." The airline's actual policy did not allow that after travel. (CBC News (opens in a new tab))
Air Canada argued the chatbot was "a separate legal entity that is responsible for its own actions." The tribunal member called that "a remarkable submission" and ordered the airline to pay $812. (CBC News (opens in a new tab)) The amount was small. The lesson was not: whatever your tools say to a customer, your business said it.
That is not an argument against every customer-facing assistant. It is a reminder that a free-text system with no grounding in your real policy is a stage prop the moment real money depends on its answer. If you would not trust it unattended, it is not AI that works yet.
Why does so much AI implementation fail?
Technology-first builds fail for predictable reasons. Someone starts with "we need an agent" instead of "this process costs us twelve hours a week and breaks every month." The data is a mess. Nobody owns monitoring. The "AI project" is really a few if-then rules dressed up for a board deck.
Good AI implementation starts from the other end: the workflow, the people who run it today, and the number that would prove it improved.
Buy or partner before you build a platform
MIT's research found that "Purchasing AI tools from specialized vendors and building partnerships succeed about 67% of the time, while internal builds succeed only one-third as often." (Fortune (opens in a new tab)) The study covers larger companies, but the lesson travels. A small business, a creator or an agency should not invent a private AI platform before one boring workflow runs reliably.
The data and plumbing are most of the work
RAND's advice to leaders is blunt: stay "focused on the problem to be solved, not the technology used to solve it," and invest up front in "infrastructure to support data governance and model deployment." (RAND (opens in a new tab)) The prompt is the small part of AI that works. Clean inputs, access rules, retries and logs are the big part.
How do you test for AI that works? The vacation test
Here is the test I use. If you leave for a week, does the system still run, and does it produce a number you can check when you get back?
If the answer depends on you opening a chat tab every morning, you have a hobby. If it depends on a demo environment that never sees production data, you have theater. If it runs on a schedule, logs what it did, fails loudly when something breaks, sends the unclear cases to a person and moves a metric you care about, you have AI that works.
What passes the test at different sizes?
- A small business: invoice reminders that go out on schedule, onboarding packets assembled from a form, a weekly report that builds itself.
- A creator: research and first drafts prepared on a schedule, with every piece reviewed before it is published.
- An agency or IT provider: ticket triage with a person on every escalation, lead routing, scheduled client reports.
One production loop with a metric beats five pilots with nice slides. That is what AI that works looks like at any size.
What does AI that works look like in practice?
The market still sells autonomous coworkers. What holds up for most teams is narrower: a dependable pipeline with a model step where judgment is needed. Workflow tools that already understand triggers, retries, schedules and approvals are adding AI as one step inside a process, not as a free-roaming employee.
AI that works usually shares a few traits:
- One workflow, not a portfolio of experiments. Pick the one that hurts most and has a clear measure.
- Production data from day one. A polished sample set hides the problems you will hit next month.
- A trigger nobody has to remember. A schedule or an event, not a person's habit.
- Checks on every output. AI quality gates check accuracy, format and tone before anything ships, and send failures to a person.
- Logs you can audit and alerts you trust. If you cannot see what it did, you cannot trust it.
- A metric that matters without the word AI. If the slide still makes sense with "AI" deleted, you picked the right number.
If a checklist and a timer would solve it, do not force an agent onto it. RAND's interviewees made the same point: some problems are too hard for AI, and others do not need it. Save model calls for the messy steps that actually need language, classification or drafting, then wrap them in the same guardrails you would put around any unattended job.
Rising abandonment is ugly if you sell hype. If you sell outcomes, it is a sign the market is maturing. Buyers now ask for production criteria, and only AI that works survives that question.
Where should you start with AI that works this month?
- List three repetitive tasks your team does every week. Write down how long each takes and how often it goes wrong.
- Pick the one with a number you already track, like response time or invoices past due.
- Write the rules first. If the rules cover most cases, automate the rules. Use a model only where the rules run out.
- Decide who handles exceptions and how they will be told.
- Run it on real data for 30 days and compare the number to where you started.
Amelia, my wife and Kief Studio's co-founder and CEO, wrote about AI agents that run at 3am versus ones that demo well (opens in a new tab) and about why AI pilots stall before production (opens in a new tab). Both pair well with this note.
Frequently asked questions
What is the difference between an AI demo and AI that works?
A demo needs a person to start it, clean its input and judge its output. A working system runs on a trigger with real data, retries when something fails, logs what it did, sends unclear cases to a person and moves a metric.
How many AI pilots actually succeed?
Few move revenue quickly. MIT's NANDA research found about 5% of generative AI pilots achieve rapid revenue acceleration, and S&P Global found the average company scrapped 46% of AI proofs of concept before production.
Why do most AI projects fail?
Usually for business reasons: the wrong problem, poor data, chasing the newest technology, no infrastructure to run it, or a task current AI cannot handle. Those are RAND's five root causes, and none of them is fixed by a better prompt.
Should a small business build its own AI platform?
Rarely. MIT found bought tools and partnerships succeed about twice as often as internal builds. Get one workflow running reliably first, with a person handling exceptions, before you think about anything bigger.
Is a company responsible for what its chatbot says?
In the Air Canada case, yes. A British Columbia tribunal rejected the argument that the chatbot was a separate entity and ordered the airline to pay. Ground any customer-facing assistant in your real policies and review what it says.
What is a good first AI project?
A repetitive task with clear rules, real data and a number you already track, such as invoice reminders or intake forms. Automate the rules, use a model only where judgment is needed, and keep a person on the exceptions.
Where we fit
Kief Studio is in Shrewsbury, and this is what our AI automation and implementation work is for: taking an AI idea out of the demo and into daily work, where input is messy, services fail, and a person checks the cases the model is unsure about. We pick the workflow, define what good output looks like, build it with retries and quality gates, keep a human path, and run it for you on LTFI, so nothing is installed on your side.
When the work repeats every week, our content operations automation runs the repeatable steps and sends exceptions to a person. We do not build demo-only chatbots or projects meant to cut your people. Your people drive the work.
First conversation is free. No commitment.




