Running AI on your own machine: when a local model beats the cloud
Small open models now run well on a laptop. Where they shine, where they fail, and how to design around their limits.
A year ago, running a language model meant sending data to someone else's server. Today a small open model runs comfortably on a laptop, with no API bill, no rate limit and nothing leaving the machine. I run a voice assistant, an SEO monitoring system and a job-search pipeline on local models. Here is what I have learned about when that is the right choice.
Where local models shine
- Privacy. Client documents, inboxes and voice never leave the device. For some clients this is the whole reason to use AI at all.
- Cost at volume. A job that runs every hour over thousands of items costs nothing extra.
- No quotas. No free-tier wall halfway through a run.
- Latency for voice. Wake word, speech-to-text, a small model and text-to-speech can all run on-device, quickly enough for a natural conversation.
Where they fail
Small models are confident and wrong in ways larger models usually are not. In my own tests, a three-billion-parameter model:
- scored a perfect-match job 20 out of 100, with reasoning that contradicted itself;
- invented employment history when asked to write a cover letter;
- labelled bank promotions as job offers.
None of this means local models are useless. It means you should not ask them for open-ended judgement.
Design around the limits
- Ask binary questions. Replace "score this from 0 to 100" with five yes-or-no questions mapped to fixed points. Results became identical across repeated runs.
- Use templates. Let the model fill bounded slots in a fixed structure instead of writing freely.
- Gate before you ask. A cheap relevance filter in code stops irrelevant input from ever reaching the model.
- Keep maths in code. Anomaly detection, trends and scores are pure arithmetic; the model only writes the explanation.
- Retrieve, do not memorise. A small model plus a local vector store of the right documents beats a bigger model guessing from memory.
The hardware gotcha
On Apple silicon, local models run on the GPU and are fast. But when memory is tight, a model can quietly spill onto the CPU and become many times slower, with no error. Check what the runtime reports about where the model is loaded before you blame the model for being slow.
Choosing local or cloud
My rule of thumb: use a local model for private, repetitive and bounded work, and a frontier cloud model for hard reasoning, long documents and anything a customer reads directly. Many of my systems use both: local for the high-volume classification, cloud for the final write-up, each behind the same timeout and fallback.