ItInsightsWorld Review: Where AI Inference Actually Runs, and What the Teardown Left Out
Every argument about where artificial intelligence should actually execute came back into focus for me through one hardware teardown. In December 2022 an autonomous object-tracking accessory built around a smartphone-integrated camera served as the worked example for framing the cloud-versus-edge decision, and the map drawn then has held up well enough that revisiting it is now more instructive than declaring a winner. The original tensions — roundtrip delay, private imagery, recurring service spend, dependence on connectivity — have not been resolved. They have accumulated new line items.
What the data-center route buys, and what it charges
Shoving required processing, memory and other resources off the device and onto connected hardware remains one of the easiest ways to slim a bill of materials; a cloud server is the usual destination. The counterweights land on suppliers and purchasers at the same time, as the side-by-side reading below shows.
| Pressure point | What happens when processing is offloaded | Why it bites at the product level |
|---|---|---|
| Roundtrip latency | Raw data travels to the cloud for processing and the result travels back | Self-driving vehicles do all their processing locally for a reason |
| Privacy exposure | Uploaded material depends on the data type — baby cam content is the obvious case — and on how well encryption and other protections keep unintended access out | Trust, not just engineering, becomes a purchasing criterion |
| Availability | The service depends on the link staying alive | At the extreme, if the cloud connection fails, the device becomes a paperweight |
| Cost shape | Server usage fees such as AWS plus connectivity bandwidth | A one-time fixed BOM cost is traded for ongoing variable costs that scale with customer count and per-customer use, both for each device and cumulatively across all devices a customer owns |
The grievance list has grown longer, not shorter
It is nearly four years later, and in my view every one of those points still holds. Back then the AI data center — the hyperscaler, in industry shorthand — build-out was still in comparative infancy, and much of the compute capability coming online had been intended for cryptocurrency mining before being redirected to deep learning processing. Today the phenomenon runs in fuller stride, and public backlash is understandable.
- Speculative investment deals, often circular in nature — a hardware supplier backing a data center developer that promises to use that supplier's chips and software — feed frequent stock market volatility and broader bubble market concern.
- Governmental jurisdictions dangle lucrative tax breaks and other incentives at developers, even when any tangible local employment increase lasts only through construction.
- Data center energy demands are substantially incremental, requiring additional power plant facilities built in parallel, with costs assumed by all.
- That incremental energy comes predominantly from hydrocarbon-based sources such as coal and natural gas, with substantive pollution and noise impacts.
- Water demands for cooling are equally sizeable, redirecting and draining aquifer reserves whether the sources are fresh, recycled, or a mix.
- The facilities are massive in size, consuming open space and fields formerly used for various agricultural purposes.
The tethered-device answer, reread
Here is what was written about edge processing in late 2022, and the assessment still holds. Another popular BOM-slimming approach uses a wired or, more commonly, wireless tethered local device with abundant processing, storage, imaging and other resources — a smartphone or tablet. The convenient advantage is that it employs a device already in the consumer's possession, already paid for, and for which any remaining cloud processing bandwidth in the complete solution is also bankrolled. Latency is notably less than with the pure cloud approach, privacy worries are lessened if not fully alleviated, and although the smartphone's connection to the cloud may periodically fail, the connection between smartphone and device generally remains intact.
Two refinements deserve emphasis now. The enhanced privacy is not only general but also specific: it keeps personal data out of the cloud provider's training data set. And the end of ongoing-cost outflows involves not only hardware but also monthly expenses for AI service subscriptions for consumers or tokens for developers.
The downside flagged at the time was the portability burden. Whatever device-offloading application runs on the tethered mobile device must comprehend all common mobile operating systems, predominantly Android and iOS, along with the related iPadOS for tablets and the various devices that run on them — and it must evolve as operating systems and hardware evolve. Otherwise the application will sooner or later stop meaningfully working, as will the device tethered to it.
In defense of that framing, the showcased device depended heavily on a smartphone-based app, so focusing on that nuance made some sense, and the app used the smartphone's built-in image capture and processing resources, avoiding costly redundancy in the accessory. What was overlooked was the growing burden of that processing and the associated need for adequate memory — both nonvolatile for storage and volatile for code execution and temporary data — and how those capacity and speed needs might expand.
The memory bill arrives on the client side
Now we know. The not-unlimited supplies of SSDs, and interestingly HDDs, trading off raw performance and energy consumption for raw capacity and cost-per-bit, along with DRAM, are being redirected to data centers, while required capacities in PCs, smartphones and similar devices for today's edge AI processing functions increase in lockstep. The local footprints line up as follows.
| Embedded AI payload | How it ships | Local storage footprint |
|---|---|---|
| Gemini AI models | Embedded in latest-generation Chrome browser installs | at least 4 GBytes |
| Apple Intelligence | Bundled with iOS 27, where it is now required versus optional with the version 26 O/S predecessor | up to 14 GBytes |
| AI model suite in MacOS 27 | Included, similarly required by default; open source can help | 12-30 GBytes |
Compression is attacking the same constraint. PrismML, which specializes in quantizing and otherwise compressing deep learning models for resource-constrained systems, shrank the Alibaba Qwen3.8 27B open-source model by between 9-10x while still matching 98% of Qwen aggregate benchmark scores; the resultant model was still 5.9 GBytes. Perplexity's Portable Computer, for local on-computer model storage and processing, is 20 GBytes. Storage needs in both cloud and edge scenarios will increase further.
Training and inference pull in opposite directions
One irony is that a not-insignificant percentage of processing resources in data centers is used, at least part-time, not for inference but for training: initial training, periodic retraining to counteract performance decay, iterative training to encompass new and expanded datasets, and transfer learning to adapt frontier models for specific tasks. OpenAI recently reported, for example, that the training run for its latest-generation Astra GPT-6 fully consumed more than 100,000 GPUs. These operations are compute-intensive and highly parallelizable, which explains why GPU-derived architectures have come to the fore in that application.
For inference, only the prefill portion of the total task is parallelizable and therefore GPU-amenable, though not necessarily GPU-optimal. The subsequent decode portion is conversely memory-intensive, from both capacity and bandwidth standpoints, and the latter rationalizes the ascendance of advanced HBM approaches.
Why a wholesale move to the edge remains fantasy
Any claim that inference will transition fully to the edge in all cases is, again in my view, a pipe dream. Autonomous vehicles and other specific scenarios? Sure. But there are plenty of other applications where latency and privacy, which can alternatively be handled via encryption anyway, are not the issue, while the need for access to sizeable trained models is. Servers rule in such cases, as noted in late-2024 follow-up coverage.
Hybrid approaches, in which a base level of inference occurs locally while cloud resources are tapped as needed for heavier functions, will become increasingly common. Other local-ish, or local networked cloud, scenarios are also emerging. NVIDIA's Personal AI Router, PAIR, enables spreading the overall inference, including agents, processing task across multiple LAN-connected computers, including non-NVIDIA Macs. Alternatively, models can be stored on a LAN-shared NAS for common access, although RAM limitations will likely preclude full inference processing there.
Reading the ledger as a demand story
Treat the whole debate as a record of what each placement consumes rather than a contest between venues. The obvious beneficiaries are memory and broader mass storage suppliers; it is ironic that after countless commodity boom-and-bust cycles, some of which I lived through from the supplier side, the semiconductor memory industry may have finally found a path to a predictable and otherwise sustainable business model. Inference chip and IP core suppliers are the other names to watch. GPU-based architectures will likely continue to reign from a training standpoint, if for no other reason than proprietary software tools prevalence, while an ever-increasing percentage of total growth in the AI processing space sits in inference, where toolset lock-ins are more fungible and other parallel-processing, and more memory-centric, architectures may indeed end up more optimal for the necessary tasks. Which venue wins a given workload is still open; what is already visible is that the silicon best placed to survive either answer is the kind that stores as much as it computes.