The night shift: getting the most from local AI means changing when you work

If you spend any time in the forums where people compare local-AI machines, you will have seen the complaint. Someone buys, or more often is thinking about buying, a Mac Studio with 256 or 512GB of unified memory, loads up one of the very large open models it is finally big enough to hold, and reports back that it feels slow. Slow next to a frontier cloud model, certainly, but also slow next to a friend running a leaner model on a single fast GPU. With Apple’s M5 Ultra machines now arriving, the question is being asked again with fresh urgency, and it is worth saying up front that most of the chatter is running ahead of the measurements: comparatively few people own these machines, and fewer still have published the numbers that would settle the argument.

The numbers that do exist tell a consistent story, though, and it is not really about Apple. When people measured one of the 600-billion-parameter-plus models on the previous generation of 512GB Mac Studio, the slow part was not generating the answer, it was reading the question. Prompt processing ran at only a handful of tokens a second, which means a long prompt could sit for minutes before the first word of a reply appeared. Generation then plodded along behind it. By contrast, a lean model of around 30 billion parameters on a single current high-end GPU reads a few-thousand-token prompt in well under a second and then generates faster than you can read. (Both figures move with every software update, which is part of the point: the Mac measurements above have already been improved on by later releases, and nobody has yet published the equivalent for the newest hardware.)

It is tempting to read that as a simple verdict, that the big-memory machine is worse. It is not. It is a machine with a different cost shape, and the mistake is bringing cloud-shaped working habits to it.

Latency is not the only thing worth paying for

A cloud model is tuned for latency. You pay per token, a fast answer to a single request is the product, and the whole experience rewards a particular rhythm: you prompt, you wait a second or two, you read, you prompt again. That rhythm is so normal now that it is invisible. It is also a direct consequence of renting someone else’s hardware by the token.

A large model on your own machine inverts almost all of that. The cost is mostly fixed and already spent: you bought the memory. The marginal cost of another token is as near to zero as makes no difference. What you pay instead is time, in the form of that latency tax on long prompts and large models. Hardware with this shape, high fixed cost and near-zero running cost, has a well-understood economics, and it is the economics of owning rather than renting. Owned compute is cheapest when it is kept busy. A render farm, an overnight build server, a scientific cluster: none of them are left idle during the expensive hours and run flat out only when someone is watching. They are kept full.

Seen that way, “my local model is slow” stops being a complaint about the hardware and becomes a question about scheduling. If answers take minutes rather than seconds, the rational response is not to sit and watch the progress bar. It is to stop watching.

What to queue, and what to keep live

This only works for the right kind of work, and being honest about which kind is the difference between a useful idea and a glib one.

Some tasks tolerate latency happily because nobody is waiting on them. Summarising a large collection of documents. Pulling structured data out of a thousand records. Building embeddings over an archive so you can search it later. A long, multi-step research task with a clear end state. Preparing a dataset. For work like this, whether each step takes two seconds or two minutes is almost irrelevant, because the result is being collected tomorrow, not now.

Other tasks are ruined by latency, and no amount of clever scheduling fixes them. Writing code with an assistant that completes your line as you type. A back-and-forth conversation where each reply shapes your next question. Anything where you are in the loop. For these, a smaller model on fast hardware genuinely is the better tool, and that is a perfectly good reason to buy a fast GPU instead of a big one. Batch working is not a cure for slowness; it is a way of routing the latency-tolerant work to the machine that is cheap and patient, and keeping the latency-critical work somewhere quick.

Compose by day, compute by night

Here is the shift, and it is more of a habit than a feature. Most people who work a standard day think of work as happening inside that day. The idea that you might use the nine-to-five to set up work that runs from five to nine, while you are asleep, is genuinely foreign if you have never run batch jobs. Your working hours become planning hours. You spend them specifying tasks well enough that they can run without you, pointing the machine at them, and leaving. The output is waiting when you come back.

None of this is new. Mainframes ran overnight batches decades ago; animators have always rendered while they slept. What is new is that the machine doing it is yours, sitting under your own desk, and the work is the open-ended, language-shaped work that until recently only a person could do. For the first time, “leave it running overnight” applies to the kind of task you would previously have done yourself, in the day, one at a time.

The honest limits

This asks more of you, not less. A task you will not be present to supervise has to be specified more carefully than one you babysit, and unattended runs need real guardrails: clear permissions, approvals for anything that touches the outside world, and a sensible stop if something goes wrong. Not every job fits the pattern, and forcing one that does not is worse than not bothering. And for latency-critical, bursty, or genuinely frontier-capability work, the cloud still wins outright. Nothing here is an argument for giving it up. It is an argument for noticing that a second machine, with a different cost shape, rewards a different way of working.

I have been building exactly this into Ragtag, my own local-first desktop AI tool, in the form of scheduled tasks that run while you are away, with the permissions and approvals to do so safely. I mention it as disclosure, not a pitch: the capability is not special, and schedulers of one kind or another exist across the ecosystem. What surprised me building it was how much of the value was not in the scheduling at all, but in being forced to think clearly enough about a task to hand it over. That turns out to be the actual skill.

The upgrade is the habit, not the hardware

The people getting the most out of local AI are not, for the most part, the ones with the fastest machines. They are the ones who have stopped expecting every answer immediately, and started treating their hardware like something they own rather than something they rent by the second. The slow prefill on a big model is real. It is also, mostly, a scheduling problem, and scheduling problems have answers. The question local AI quietly asks is not what you are willing to run. It is whether you are willing to change when you work.