Building an AI app: what a demo doesn't tell you
An AI feature in an app is easy to demo and surprisingly hard to keep running. The questions that decide whether it survives year two are rarely about the model — and nearly always about cost per user, behaviour without a network, and what you do with people's data.
Why does a demo prove nothing?
Because a demo has one user, one network, and one carefully chosen example.
The questions that decide whether the app survives year two never appear in it. What happens when the network drops? When the model is slow? When the answer is wrong? And when there are ten thousand users each costing money every month?
Those are design questions, not implementation details. They belong at the start, not once the app is already in the store.
What does one user cost per month?
This is the difference between an AI app and an ordinary one. In an ordinary app an extra user is effectively free. In an AI app every user costs money each month in model usage.
Work it out at realistic volumes before you start. An app that works technically but loses money per user is still a problem — you just find it at the moment you grow, which is the worst possible timing.
Caching, smaller models where they suffice, and sensible limits decide whether your business model holds at ten thousand users. Those choices live in the architecture, not in a later tuning pass.
What happens when it goes wrong?
Network gone. Model slow. Answer unusable. In all three the app still has to do something sensible.
On Loop Sleep, the sleep companion we built for Loop Earplugs, that got very concrete. Switching to streaming text-to-speech meant audio started playing while the rest was still being generated. Just before sleep, a ritual has to feel right immediately — a loading screen takes exactly that away.
Choices like that never show up in a demo. They do decide whether someone opens the app a second time.
Privacy is a store requirement, not a preference
What data the app sends, where it goes, and what the user gets to say about it: in the app stores there is no way around this.
So decide early where the model runs and what data it sees. That partly determines which models are usable. Discover it during review and your launch slips.
Where does AI genuinely add something?
In practice, in three places. Outside them, rarely.
- Onboarding that feels like a conversationInstead of an eight-screen form. Loop Sleep opens by asking what kind of night it is, and builds the whole ritual from the answer.
- Content generated per userRather than pulled from a fixed library. That solves the content fatigue audio apps usually hit within a week.
- Recommendations from real behaviourAs Trai does with triathlon plans built from the Strava data an athlete already generates.
And where it does not
On features a simple rule or a good search box would handle just as well. There you are adding cost, latency, and uncertainty, and nothing else.
The sober starting point: begin with the feature someone would actually open the app for, and only then decide whether AI is the best way to deliver it. Loop Sleep V1.0 shipped in four weeks on a ruthlessly scoped list — everything that missed the date was explicitly deferred, not dropped. Apps that start with everything at once rarely reach a second release.
Start with that one feature
Once you know what someone would open your app for, we can plan a first version that reaches real users quickly. The rest follows from what they do.