On-Device AI or the Cloud? Sorting It Out After WWDC 2024

Two weeks ago at WWDC, Apple put a question front and center that most of us building AI features answer by default: where does the model actually run? For a lot of apps the answer has been somebody else’s GPU over HTTPS. Apple’s answer is on the device when it can, on Apple’s own servers when it can’t, and it published a surprising amount of detail about both. None of it has shipped; it’s all announced for a beta this fall. It’s still a good moment to think through the same split for your own app.

What Apple announced

The June 10 newsroom announcement describes Apple Intelligence as a personal intelligence system built into iOS 18, iPadOS 18 and macOS Sequoia. The features include Writing Tools that rewrite, proofread and summarize text in third-party apps too, email and notification summaries, image generation, and a Siri that can act inside apps. Apple says many of the models run entirely on device, and that more complex requests go to larger server models through something it calls Private Cloud Compute.

The hardware list is the first practical detail worth noticing. Apple said Apple Intelligence will be available on iPhone 15 Pro, iPhone 15 Pro Max, and iPad and Mac with M1 and later, with Siri and the device language set to U.S. English, in beta this fall, with more features and languages over the following year. That leaves out the regular iPhone 15. Apple’s announcement doesn’t explain the cutoff, but the lesson for developers is plain enough: on-device AI comes with a hardware floor, and plenty of your users will be below it.

Apple also said ChatGPT access is coming to Siri and Writing Tools later this year, with Siri asking the user before any question is sent. That’s a third destination, and for Siri it waits on the user saying yes.

The model on the phone

Apple’s Machine Learning Research team posted an overview of the foundation models behind all this. Two matter here: an on-device language model of roughly 3 billion parameters, and a larger server model that runs on Apple silicon servers through Private Cloud Compute.

The on-device model is squeezed hard. Apple describes low-bit palletization with a mix of 2-bit and 4-bit weights, which it calls critical to meeting memory, power and performance requirements, plus shared input and output embedding tables to save memory. Rather than a separate model per feature, it uses small LoRA adapters layered on top of the base model. For the 3B model a rank 16 adapter typically takes tens of megabytes, and adapters are loaded, cached in memory and swapped as the task changes. On iPhone 15 Pro, Apple reports about 0.6 milliseconds of time to first token per prompt token, and generation at 30 tokens per second.

Do the arithmetic. A 1,000 token prompt means roughly 0.6 seconds before anything appears, and a 300 token reply takes about ten more seconds. Fine for summarizing a notification, less fine for a long document while someone watches a spinner.

On quality, Apple says human graders preferred its 3B model over larger open models including Phi-3-mini, Mistral-7B and Gemma-7B on its set of real-world prompts, and that the server model compares favorably to DBRX-Instruct, Mixtral-8x22B and GPT-3.5. Those are Apple’s own evaluations. The more telling point is structural: Apple still built a server model and a whole server platform. Even with this much tuning, a phone-sized model doesn’t do everything.

The server side, and why it’s different

Apple’s Security Research post on Private Cloud Compute opens with a frank point about cloud AI: to run a large model on your request, the server needs unencrypted access to that request, which rules out end to end encryption. So the design is about constraining what can happen to the data while it’s in the clear.

Three of its requirements stand out. Stateless computation means personal data in a request is used only to fulfill that request, isn’t available to Apple staff even during processing, and isn’t kept after the response, including in logs or for debugging. The node’s data volume keys are randomized on every reboot and never persisted, so nothing written there survives a restart. No privileged runtime access means the nodes ship without remote shells or interactive debugging, can’t enable Developer Mode, and have no general-purpose logging; only pre-specified, audited logs and metrics can leave. Verifiable transparency means Apple plans to publish the software image of every production build, record code measurements in an append-only transparency log, and have devices refuse to send data to any node that can’t attest to running logged software.

Requests are also encrypted to specific validated nodes, so load balancers can’t read them, and a third-party OHTTP relay hides the device’s IP address. Apple calls the post an initial overview, with a deeper dive promised once PCC is in beta. So, again, announced and not yet independently checked.

Why care about a server platform you can’t use? Because it spells out what a private cloud AI backend would actually have to promise. In a typical stack, prompts get logged for debugging, TLS terminates at a load balancer, and whoever is on call has a shell. Apple’s post uses almost exactly those examples to explain how privacy promises quietly leak.

Small models outside Apple

Apple isn’t the only one betting on local. In April Microsoft introduced Phi-3, a family of open small language models starting with Phi-3-mini at 3.8B parameters. Microsoft made it available on Azure AI Studio, Hugging Face and Ollama, pointing developers to Ollama for running it locally on their laptops, and optimized it for ONNX Runtime across GPU, CPU and even mobile hardware. The use cases Microsoft lists for small models line up neatly with this post: on-device and offline inference, latency-bound scenarios, and cost-constrained simpler tasks. Microsoft’s technical report adds that a 4-bit quantized Phi-3-mini takes about 1.8GB of memory and ran fully offline on an iPhone 14 at more than 12 tokens per second.

Microsoft is upfront about the weakness too. The same post notes Phi-3 models don’t do as well on factual knowledge benchmarks such as TriviaQA, since a smaller model has less capacity to retain facts. In Microsoft’s feature story on Phi-3, Sebastien Bubeck says there’s still a gap between small models and the big models in the cloud, and maybe always will be. The company’s framing is a portfolio, where some customers need small models, some need big ones, and many will combine both.

The tradeoffs, one at a time

Latency cuts both ways. A local model skips the network round trip, which is the delay Microsoft’s story points to. But a phone generating 30 tokens a second won’t feel instant on long outputs. Short in and short out is where local clearly wins.

Offline is the easy call. If a feature has to work on a plane, the model has to live on the device.

Privacy favors local, and Apple’s post makes the case well: data that exists only on the user’s device isn’t sitting in some central place waiting to be attacked. On a server, privacy depends on what the operators can do, and a promise not to log something is usually one users can’t check.

Battery and memory are where local costs you. Apple’s optimization work is all about memory and power budgets, and Phi-3-mini’s 1.8GB at 4 bits is a decent reference for what a small model asks of a device that’s running everything else too. Bring your own model and you also ship its weights, which means a large download or a large app.

Model quality favors the server. A 3B or 4B model tuned for narrow jobs can be very good at those jobs and weak at knowledge-heavy, open-ended ones.

Cost moves around rather than disappearing. Cloud inference is a per-request bill that grows with usage. Local inference costs you almost nothing per request, but you pay in engineering time, app size, and a fallback for devices that can’t run it.

Questions to ask before you pick

Start with the data. Does the feature touch messages, photos, health, location or anything else personal? Could you explain to a user where that data goes, and back it up? If you can’t get near a bar like no retention and no privileged access, local deserves a hard look.

Then the task. Is it narrow, like classifying, extracting or summarizing something short, or does it need broad world knowledge and long answers? Small models are strongest on the narrow stuff.

Then the devices. How many of your users have hardware that can run a model well? For anything built on Apple Intelligence, people below iPhone 15 Pro or M1 get nothing, so you’ll need a fallback regardless, and that fallback is probably a server.

Then the failure modes. What happens offline, or when the model is slow?

Then the money. What does a request cost in the cloud at the volume you expect, and what does running locally cost in engineering and app size?

For a lot of apps the honest answer will look like Apple’s own design: do the small, private, frequent work on the device, send the hard work to a server, and be clear with users about which is which. What WWDC changed isn’t so much the answer as the bar. If a platform vendor plans to publish its server images and have devices refuse unverified servers, “trust us, it’s private” is going to sound a lot thinner for the rest of us.