Over the last eighteen months I have sat through a lot of AI pilots, ours and other people’s. Here is the uncomfortable pattern: most of them worked. Almost none of them shipped.
That gap is the only part of this conversation I find interesting anymore. Getting an LLM to do something impressive is no longer a technical achievement. Getting it to do that thing ten thousand times, inside a real workflow, on data nobody cleaned up first, while somebody’s business depends on the answer: that is still hard. And it fails for reasons that have almost nothing to do with the model.
What AI actually changed, and what it did not
I have been building software for thirteen years. For most of that time, a project carried two expensive risks.
The first was technical. Could we build it, and how long before we knew? Did we pick the right technology? Did we set up the architecture so it works on day one and still holds up when the client grows?
The second was product. Did we build the right thing for the client? Did we misread the requirements? Did we manage expectations well? Was there scope creep, or scope we underestimated, or scope we overestimated so badly that it created its own confusion?
Building software properly takes more than code. It takes practices, a team that works well together, real experience with hard problems, and an architecture you can keep extending. Digesting requirements, putting together product presentations, building prototypes so people have something concrete to react to: all of that took time. Even an experienced product owner or designer still needs to hold the big picture, the vision, the details of each user journey, and a sense of what a genuinely good product feels like. That is where we add value for a client.
LLMs collapsed a real part of that. Not all of it, but enough to matter. At Slash we now turn a requirements document into technical scoping, structured epics, user stories and a risk log in an afternoon, using our own LLM-based scoping and quotation skills. That used to mean days of workshops and hand-written specs. The same goes for prototype screens and workshop presentations.
But that is not the whole story, because we are still in the loop. Our product, design, software and AI people layer their expertise on top of what the model produces, refining the scope, sharpening the prototype, shaping the code. We have a responsibility to understand what the LLM gave us, push back on it, and stay on top of the output.
So what did not change is the thing that was always the real work: closing the distance between “the client has a problem” and “the team understands exactly what to build and why.” Mechanical scoping got cheap, so the effort moved somewhere else. It moved into judgment calls, reading between the lines, and catching what the requirements did not say.
At the end of the day, we are building for humans to use.
Time to prove out code has dropped hard. Technical unknowns are cheaper to retire than they have ever been. But whether it is traditional development or an AI-based product or workflow, a pilot can still fail. It does not fail on capability. It fails because nobody was strict about which problem was worth solving, or about what “working” would have to mean before anyone would trust it on a Monday morning.
Why a good demo is weak evidence
A demo proves the model can do the task once, on data you picked, with you in the room to steer it. Production asks a different question: can this run on its own, on inputs nobody screened, and can you tell when it is wrong?
So before we pilot anything, for a client or for ourselves, we run the idea through four gates. This is not a maturity model. These are four questions that have each killed real work of ours.
The Four Gates
1. Repetition: does this job happen often enough, and does automating it make the work better?
How often it happens, times how long it takes each time, is the starting point of the business case. A task somebody does forty times a week is a candidate. A task done twice a quarter, beautifully automated, is a hobby.
But saving time is only half of it. The other half is where that time goes. When we started building in-house agents and workflows for our own day-to-day work, we did not start with “what can AI do?” We started by listing the repetitive pain points that kept pulling our people away from the work that needs them most.
Quotation and scoping was at the top of that list. We do it constantly, and most of the effort used to go into reading, digesting and logging details from long requirement documents. So we built AI into that pipeline. It now absorbs the product and project requirements, gets us to clarity faster, and turns all that reading into something we can actually use: a presentation, a prototype, an architecture diagram. The time we save goes back into the part that matters, which is helping clients shape and build the right product.
2. Judgment: where does the human give direction, and can the AI ask for it?
Some tasks carry judgment that lives in one person’s head and has never been written down anywhere. AI can help there, but it cannot own it. The useful question is not “can the model decide this?” It is “what does the human review step look like, and does the workflow still save time once that step is in it?” If reviewing the output costs as much as doing the task, you have moved the work, not removed it.
AI agents can now take on a lot of work and finish it. But we still need to be in the loop, giving direction and deciding what should happen next. This matters most during a pilot, when we often do not yet have enough data or clear direction. At that stage we cannot simply let the AI make its own judgment calls.
So we have been putting real effort into closing that human-in-the-loop gap in one of our own products. The idea is simple. When the AI gets stuck, it should be able to say exactly what it needs from you to continue. And the person in the loop should be able to answer quickly, in a way that turns into useful feedback for the work, the product and the workflow. A good checkpoint is a short, clear question, not a long review of everything the AI did.
3. Ground truth: can you tell right from wrong at scale?
This is the gate that kills the most ideas, and it is the one most people skip. If you cannot check the output automatically against something you trust, you cannot ship it, watch it, or improve it.
Our document AI work passes this gate. We pull quotations, purchase orders and invoices into structured data, and extraction has a right answer: the total either matches the document or it does not. Better still, these documents check each other. When we test, we use full sets where the quotation, the purchase order and the invoice for the same deal should agree on items, quantities and totals. If the three do not line up, either the documents have a real problem or our extraction does, and both are worth knowing. Open-ended summaries of the same documents look far better in a demo and are far harder to gate, because nobody can say what the correct answer is.
We apply the same thinking to our own quotation tooling. We split the output into the part a machine can check and the part a person has to judge. The machine part is strict: every story has a size, and every total recomputes from one source file, so any number in a proposal can be traced back. The judgment part, whether a size is realistic for this client and this team, goes to our leads, who check it against their own delivery experience. Knowing which half is which is what lets us trust the output.
Vague success criteria are not a small problem here. They are the reason a pilot can run for three months and end with everyone agreeing it “looked promising.”
4. Ownership: who owns this after we leave?
An AI feature is not a deliverable. It is a system with a running cost. Prompts drift, models get retired, new edge cases show up, the workflow changes. If nobody on the client side owns it, it quietly gets worse and takes trust with it. We ask this in the first conversation now, because the honest answer sometimes changes the shape of the whole project.
We hold ourselves to the same rule. Every internal skill, agent and MCP server we run has someone who looks after it, and looking after it includes deciding when part of it should go. Our quotation tooling used to produce a full Excel workbook. Over time it became clear that the spreadsheet was one more thing to keep in sync, and it was not where decisions were being made. So we retired it, made a single CSV file the source of truth, and moved the live views onto our quotation platform. Nobody would have made that call if nobody owned the tool. Ownership is not only keeping something running. It is also being willing to switch parts of it off.
The scorecard
We score each gate 1 to 5 and add them up. It is deliberately blunt. Its job is to force the conversation, not to produce a number anyone should worship.
| Gate | The question | Kill signal |
| Repetition | How often, how long each time, and does it make the work better? | Low volume, or time saved that does not improve the outcome |
| Judgment | Where does the human give direction? Can the AI ask for what it needs? | Review costs as much as doing the task |
| Ground truth | What can a machine check, and what must a person judge? | No correct answer defined for either part |
| Ownership | Who maintains it, improves it, and decides when to retire it? | No named owner, on the client side or ours |
Under 12. Do not pilot. Fix the problem definition first, or say no.
12 to 15. Pilot, narrowly, with exit criteria written before the first line of code.
16 and above. Skip the toy phase. Build it as production from day one, because that is where it is going.
What a pilot is actually for
A pilot is not a demonstration. It is an experiment where you agree on the result you need before you start. If you cannot write the exit criteria up front, that is already the finding.
We commit to three numbers: a quality bar (accuracy against ground truth, measured on data we did not hand-pick), a throughput bar (how much it handles without a human stepping in), and a cost ceiling per unit of work. Then a decision date. Hit the bars and we scale. Miss them and we kill it, having spent weeks instead of quarters to learn something true.
That discipline comes from higher-stakes work. On the WFP PRISM platform, a bug in a data pipeline means aid gets routed on wrong information. On financial systems audited under the Monetary Authority of Singapore framework, it means a regulator’s trust. Those environments teach you that “we will catch it in QA” is not a plan. You surface the unknowns early and take the risk out of them, even when early is slower. AI systems fail quietly and unpredictably, so they deserve at least that much care.
The part nobody budgets for
Once a use case clears the gates, most of the work left is not AI work. It is plumbing: tool definitions the model cannot misuse, managing context and memory, MCP servers that expose your real systems with real authentication, human checkpoints, monitoring, and a test suite that tells you when a model update quietly broke something. This is the layer we have invested in most, including building our own agent runtime on the Agent SDK, because it is the difference between a pilot and a product.
We learned this the hard way, not in theory. On one project years ago, we chose an architecture that worked beautifully for the client’s first set of requirements, but it could not stretch to handle the changes and shifts that came later. The lesson was not “pick better tech.” It was that failures like that never have one clean cause. Product clarity, alignment on the long-term roadmap, and resourcing were quietly making it worse long before the technical ceiling showed up. The technical decision was just the easiest thing to blame.
That is the same reason we do not scale AI use cases on enthusiasm. Enthusiasm is very high right now, and it is the least reliable signal in the room.
Where we have landed
Our own rollout followed this exact shape. We did not switch everything on at once. We onboarded the team slowly, one product at a time, and each one had to earn its place through the same gates.
- Scoping and proposal building. This came first, because the volume was obvious. It started as a script for a single proposal. Today it is a full flow: structured scoping output, a platform where the team reviews and adjusts it, and skills that turn it into presentations and designs.
- Agentic coding. Next, we moved our developers onto LLM-based agentic coding tools such as Claude Code. It started with one or two teams, against clear criteria, before it went company-wide. The tool was the easy part. The real work was the practices around it: how we give the agent instructions, how we review what it produces, and when a human has to step in.
- Daily briefs. Then came Should Do. It connects to the tools each person already uses and gives them a daily brief: suggestions on what to focus on, a heads-up on what needs attention, and a way to dive deeper into the work in front of them.
- Personal agent workflows. Should Do is also becoming our agent runtime platform, where anyone on the team can set up small workflows of their own. A bot that takes notes. One that logs change requests coming out of product discussions. One that reads incident emails and proposes the actions to take.
None of these scaled because they were exciting. They scaled because the volume was there, we could tell when the output was wrong, and someone owned them.
This is also where SlashFlow came from. After running these workflows on ourselves, and learning where they break, we turned that experience into a managed workflow automation service for our clients. We take a client’s repetitive work through the same four gates, build the workflows that pass, and keep running them after launch. That last part is deliberate. Ownership is the gate most pilots fail, so with SlashFlow we take it on ourselves.
If you are sitting on a pilot that impressed everyone and has not shipped, the question worth asking is probably not about the model. It is whether anyone ever defined what “working” would have to mean, and who would own it once it did.
Slash is a software boutique helping startups and businesses build their platforms. If you are evaluating where AI genuinely fits in your product or operations, we are happy to run your use cases through the gates with you.
