Designing Systems in Dialogue With AI
AI-assisted coding works best as design dialogue, not delegation fantasy. Teams that win use agents to expose assumpt...
AI Won't Run Your Company by Itself
9 min read
The Fantasy Is Cheap. The Cleanup Is Expensive.
18.05.2026, By Stephan Schwab
A surprising number of executives still talk about AI as if it were a diligent new employee who never sleeps, never argues, and can quietly run software development, operations, or half the office if given the right prompt. That fantasy is attractive for the same reason crash diets are attractive. It promises a shortcut around discipline. It also fails for the same reason: reality still exists.
Adoption Is Real. Magical Autonomy Isn’t.
"Most companies are using AI. Far fewer are getting the magical payoff they were promised."
Let’s start with the part that is true.
AI is already inside plenty of real businesses. Stanford HAI’s 2025 AI Index Report says 78% of organizations reported using AI in 2024, up from 55% the year before. That is not fringe behavior. The tools are here. Budgets are moving. Staff are experimenting whether leadership understands the mechanics or not.
But the same market is full of executives talking like AI will soon develop software on its own, run support on autopilot, process internal decisions, and maybe replace the annoying human middle of the company altogether.
That is where the thinking gets sloppy.
PwC’s 29th Global CEO Survey is useful here because it cuts through conference-stage swagger. It is based on responses from 4,454 chief executives across 95 countries and territories, which makes it a better signal than another US tech-panel performance. Yes, 30% of CEOs reported increased revenue from AI over the previous 12 months. Fine. But 56% reported neither revenue gains nor lower costs, and only 12% reported both. That is not a picture of effortless transformation. That is a picture of broad experimentation, uneven payoff, and a lot of executives buying tools faster than they are building operating discipline around them.
In plain English: AI adoption is real. AI magic is not.
Why Leaders Keep Believing the Fairy Tale
"The dream is not really about AI. It is about escaping the messiness of human coordination."
Non-technical leadership often sees the worst part of organizational life very clearly: delays, meetings, politics, rework, missed handoffs, and software teams explaining yet again why a feature that looked simple was not simple. That pattern is not uniquely American. You can find it in a Mittelstand firm in Germany, a bank in Panama, a retailer in Colombia, a public-sector office in Spain, or a fast-growing company in Singapore. The nouns change. The dysfunction does not.
Then AI shows up with slick demos.
Type a prompt. Get a workflow.
Type a prompt. Get code.
Type a prompt. Get a report.
After ten minutes of that, it is tempting to conclude that the expensive, stubborn, opinionated humans were the bottleneck all along.
They were not.
The bottleneck was unmanaged complexity. Humans were just the only ones carrying it.
Software development is not typing. Office operations are not document generation. Delivery is not a pile of tasks waiting for obedient execution. The hard part is deciding what matters, spotting contradictions, resolving ambiguity, handling exceptions, assigning accountability, and absorbing reality when it refuses to match the plan.
Those are judgment problems.
AI can support judgment. It does not own it.
Why Autonomous AI Breaks on Contact With Real Work
"Autonomy is safest where success is clear, feedback is fast, and failure is cheap. Executive work rarely looks like that."
There is a reason the strongest current AI agent stories tend to come from narrow, instrumented environments.
Anthropic’s Building effective agents makes the point with unusual honesty: start with the simplest solution possible and consider not building agents at all. Their argument is not anti-agent. It is anti-fantasy. Workflows are more predictable for well-defined tasks. Agents make sense when you genuinely need flexible, model-driven decision-making and can tolerate higher cost and the risk of compounding errors.
That last phrase matters: compounding errors.
This is exactly what executives underestimate.
If a junior employee makes one wrong assumption in a meeting, somebody else usually catches it. If an AI agent makes one wrong assumption at the start of a long autonomous chain, it can spend the next twenty steps building beautiful nonsense with perfect confidence.
OpenAI’s Why language models hallucinate states the uncomfortable part plainly: hallucinations remain a fundamental challenge for large language models. Newer models reduce them. They do not eliminate them. The issue is not just occasional factual error. The issue is that the model can produce plausible falsehoods exactly in the tone busy executives most like to trust: crisp, structured, and calm.
Now put that behavior inside software development, finance operations, procurement, legal review, HR, multilingual customer support, or executive reporting spread across jurisdictions.
The immediate problem is not that the AI is stupid. The immediate problem is that it is fluent.
Fluency creates false trust.
Software Development Is the Wrong Place for Magical Thinking
"A passing demo is not a running system. A generated feature is not a maintained product."
This is where a lot of executive AI strategy goes off the rails.
Leaders watch an agent produce a screen, an API endpoint, a migration, or a bundle of tests. They see visible output and assume the invisible work is now optional.
It isn’t.
Someone still has to decide:
- what the system is actually supposed to do
- how failure is detected and recovered
- what happens under concurrency and partial outage
- how security boundaries are enforced
- what gets logged, measured, and alerted
- how future changes will be made without tearing the whole thing apart
That is why Vibe Coding Isn’t Software Development is not just a complaint about bad practice. It is a warning about category error. People confuse code generation with product development. They are not the same thing.
The same applies to agentic development. An agent can be extremely useful in a codebase when the work is bounded and the feedback loop is hard. If tests fail, the code is wrong. If deployment checks fail, the change is not ready. If observability shows regressions, you stop.
That is why Tests Beat Instructions for AI Coding Agents matters. Verifiable environments make AI much more useful because the system can be corrected by reality instead of by optimism.
Executives hoping AI will “just build the software” usually skip that entire discipline layer. Then they act surprised when a fast generator produces fast debt.
Offices Don’t Run on Text Generation Either
"An office is not a backlog of documents. It is a web of commitments, incentives, exceptions, and consequences."
The office version of this fantasy is just as flawed.
Yes, AI can draft memos, summarize meetings, classify tickets, prepare first-pass analyses, route routine requests, and answer standard questions faster than many humans. Good. Use that.
But running an office is not the same as producing office-shaped text.
An office runs on things that AI still does not own well:
- priorities that conflict with one another
- exceptions that break the process map
- reputational risk
- legal and contractual exposure
- political context inside the company
- tacit knowledge people never bothered to document
- accountability when something goes wrong
The ugly truth is that many leaders do not actually want autonomy. They want deniability with lower payroll.
That is a terrible operating model.
If an AI system rejects the wrong candidate, approves the wrong payment, classifies the wrong customer complaint, summarizes a risk report incorrectly, or sends leadership in the wrong direction, the organization does not get to shrug and blame the machine. The responsibility snaps straight back to the humans who delegated without control. That remains true whether the formal title on the door says CEO, managing director, general manager, founder, or country head.
Where AI Actually Earns Its Keep
"AI is strongest as an amplifier inside a controlled system, not as a magical substitute for one."
Used well, AI is already valuable.
It accelerates first drafts. It helps compare options. It reduces tedious lookup work. It surfaces patterns in large piles of text. It assists with exploratory code changes. It shortens the path from question to candidate answer. In software development, it can be excellent at boilerplate, refactors, test generation, repository exploration, and legacy code explanation. In office work, it can help with triage, summaries, templates, knowledge retrieval, and repetitive coordination.
That is not a small thing. It is real leverage.
But leverage is not self-government.
The strongest implementations tend to share a few boring characteristics:
- the task is narrow enough to define success clearly
- a human owner remains accountable for the outcome
- the system has checkpoints, logs, and rollback paths
- errors are detectable before they spread
- domain experts remain close to the loop
Boring is good. Boring scales. Boring survives contact with auditors, customers, production incidents, month-end close, and quarter-end pressure.
Magic does not.
What Leaders Should Do Instead
"Buy bounded autonomy. Demand proof. Keep humans where judgment actually lives."
If you lead a company, the practical move is not to ask whether AI can run on its own. The practical move is to decide where autonomy is cheap, where it is dangerous, and where it is foolish.
And yes, this applies whether you lead a listed company, a family-owned manufacturer, a regional bank, a public institution, or a software firm operating across several countries. The governance details differ. The need for judgment does not.
Start here:
- Put AI in workflows with clear verification.
If the result can be tested, checked, reconciled, or reviewed against an objective standard, good. Those are strong candidates.
- Keep human owners on business-critical decisions.
If the work changes risk exposure, customer trust, legal position, cash movement, system reliability, or organizational direction, a named human needs to own the call.
- Demand evidence, not demos.
Ask how error is detected. Ask how rollback works. Ask who reviews exceptions. Ask what metrics prove the workflow is better, not just faster.
- Do not cut the domain experts first.
The people who understand the process, the customer, the codebase, or the regulatory edge cases are the ones who make AI safer and more profitable. Remove them too early and you keep the tool while deleting the judgment.
- Treat AI as an operating model change, not a software purchase.
If roles, controls, metrics, escalation paths, and accountability do not change, then you did not transform the organization. You bought a faster way to produce unverified output.
The Serious Question
The serious question is not whether AI can act autonomously.
Of course it can, in bounded contexts.
The serious question is whether your company has the discipline to decide where that autonomy belongs.
That is leadership work. The machine does not do it for you.
Executives who expect AI to run software development or the office by itself are not being bold. They are trying to skip the part where management becomes specific.
The tools are useful. The shortcuts are fake. Build around that reality and AI becomes an amplifier. Ignore it and AI becomes a very fast way to manufacture confusion.
Sources
- Stanford HAI, 2025 AI Index Report
- PwC, 29th Global CEO Survey
- Anthropic, Building effective agents
- OpenAI, Why language models hallucinate
Contact
Let's talk about your real situation. Want to accelerate delivery, remove technical blockers, or validate whether an idea deserves more investment? I listen to your context and give 1-2 practical recommendations. No pitch, no obligation. Confidential and direct.
Need help? Practical advice, no pitch.
Let's Work Together🎭 This Week's Episodes
Telenovelas show what we can't say in client meetings. The drama is heightened, but the patterns are real.
The First Pairing Session
Stefan proposes pairing and, worse, mob programming. The team expects chaos. Daniel fears losing the last lever he still pulls: approval. Mariana e...
Technical Consultancy
Embedded Delivery Partner
Embedded into your team as an active contributor, reducing delivery friction and helping important work move cleanly.
Technical Advisor
Peer-style technical assessments before far-reaching decisions; reduce architectural and product risk early.
Product & Delivery
Ship working software to real users earlier. Measure impact and adapt based on evidence rather than assumption.
Custom Software Development
High quality, maintainable software. Short-term augmentation that leaves lasting capability in your own team.
Recent Articles
- AI Won't Run Your Company by Itself (18.05.2026)
- Quality Was the Point of Agile Coaching (15.05.2026)
- What Real AI for Normal People Looks Like (11.05.2026)
- A Chatbot Is Not an AI Strategy (04.05.2026)
- Designing Systems in Dialogue With AI (01.05.2026)