The interesting story in local AI isn't running a frontier model slowly on your laptop — it's what a small model can do reliably when you give it one narrow job. Tota ships a 2.3 GB language model inside a dictation app: Qwen 3, quantized to 4 bits, running on Apple's MLX framework. Its entire job is cleaning up speech-to-text output. Here's how that works and why each piece was chosen.
TL;DR
The Job: A Cleanup Pass, Not a Chatbot
Whisper turns speech into text remarkably well, but raw transcriptions still carry the artifacts of talking: homophones ("their" for "there"), missing punctuation, stutters and false starts, and lists dictated as one breathless sentence. Smart Formatting is a second pass that fixes exactly those things, plus spoken layout directives — say "new paragraph" or "new list" and the text obeys while the command words disappear.
Framing the problem this narrowly is what makes a small model viable. The input is short (one dictation), the output is tightly constrained (the same text, corrected), and the model needs no world knowledge beyond the sentence in front of it.
Why Qwen 3 4B
The specific model is Qwen3-4B-Instruct-2507 in the mlx-community 4-bit build. Three properties earned it the slot:
- Instruction-following per gigabyte. The 2507 refresh of Qwen 3's 4B instruct variant follows precise editing instructions unusually well for its size — and this task is all instruction-following.
- Apache 2.0 weights. An open license means the model can ship inside a one-time-purchase app. There's no per-token bill to pass on, which is a large part of why Tota costs £19.99 once while cloud dictation apps charge monthly.
- It fits. At 4-bit quantization, 4 billion parameters come to roughly 2.3 GB — a one-time download, and a memory footprint that leaves your Mac usable while it's loaded.
Quantization is the quiet enabler here: storing weights in 4 bits instead of 16 cuts the model to about a quarter size. For open-ended generation the quality cost is measurable; for a constrained correction task it's hard to detect — and any output that does go wrong gets caught downstream.
Why MLX
MLX is Apple's open-source machine learning framework, built specifically around Apple Silicon's unified memory: CPU and GPU share one pool, so tensors don't get copied between devices. For an app that runs a model in the background of your typing life, that architecture matters more than benchmark peaks — the model loads once at launch (in the background, so your first dictation doesn't stall) and sits in unified memory until a transcription needs it, at which point a short prompt runs through the GPU in a fraction of a second.
The mlx-community organisation on Hugging Face maintains quantized MLX conversions of most open models, which is what makes this practical for an indie app: the same afternoon a new Qwen release lands, a 4-bit MLX build usually exists.
Keeping a Small Model Honest
A 4B model will occasionally do something silly — that's the price of small. The design answer is to make silliness cheap: Smart Formatting operates under a "corrects, never rewrites" contract, and every output is checked before it reaches your document. If the cleaned-up text fails safety checks — wrong length, too little overlap with what you actually said, or processing that took too long — it's discarded and the raw transcription is pasted instead. The model also never acts on instructions inside your dictation: saying "delete all my files" types those words; it doesn't obey them. The guardrail design is a post of its own coming soon; the short version is on our AI dictation for Mac page.
What This Adds Up To
One Whisper model for ears, one 4B language model for grammar, both resident in unified memory on the same chip — no account, no API key, no latency to a data centre, and nothing you say leaving the machine (every network call the app can make is listed on the security page). Model size trade-offs on the speech side are covered in Whisper model sizes explained.
Honest note: if what you want is to chat with a local LLM, Tota isn't the tool — LM Studio and Ollama are built for exactly that. Smart Formatting is the other kind of local AI: invisible infrastructure that makes one everyday thing quietly better. Setup details live in the Smart Formatting documentation — it's optional and off by default.
Frequently asked questions
Which model does Tota's Smart Formatting actually run?
The mlx-community 4-bit build of Qwen3-4B-Instruct-2507 — Alibaba's July 2025 instruction-tuned refresh of the 4-billion-parameter Qwen 3, quantized to roughly 2.3 GB and run locally through Apple's MLX framework.
What Mac do I need to run it?
Any Apple Silicon Mac on macOS 14 or later. The model takes about 2.3 GB of disk and a similar amount of unified memory while loaded; 16 GB of RAM is comfortable alongside a full workload, and it runs on 8 GB Macs with less headroom. Smart Formatting is optional and off by default — Tota works fully without it.
Is Qwen 3 actually open source?
The Qwen 3 weights are released under the Apache 2.0 license, which permits commercial use, modification, and redistribution. That open license is what makes it possible to ship the model inside a one-time-purchase app instead of metering a cloud API.
Does Smart Formatting send my text anywhere?
No. The model downloads once (~2.3 GB) and then runs entirely on your Mac — inference happens locally via MLX and works with WiFi off. Your dictation never reaches a server, which is the point of running the model locally in the first place.
Why not use a bigger model?
Because the job is narrow. Cleaning up a transcription — homophones, punctuation, list formatting — doesn't need frontier-model reasoning; it needs fast, reliable instruction-following on a short text. A 4B model does that in a fraction of a second of GPU time, leaves RAM for your actual work, and its occasional failures are caught by safety checks that fall back to the raw transcription.
