How it works
About tiny-ai
Small language models that run in your browser, not on a server. The site sends them to you; your device does the thinking.
What it is
Most chat assistants run on a company's computers: what you type goes to them and the answer comes back. Here the model itself is downloaded once and kept by your browser, then every word of every reply is worked out on your phone or computer, with WebAssembly or your graphics card (WebGPU).
A question, step by step
A small model can't know everything, so it borrows a little expertise per subject. A package is a small correction to every layer of the model (a LoRA update), trained on how a much bigger model (Llama-3.1-8B) answers questions in that subject.
YOUR DEVICE tiny-ai.duckii.dev +-----------------------------------+ +------------------------+ | "why is my sourdough so dense?" | | | | | | | ROUTER | | v | vectors | scores 15 subjects | | embedder (4 MB) -----------------------> | in about 0.1 ms | | | "cooking"| | | +--------------------------------<----| | | v | +------------------------+ | cooking package (0.8 MB) <-------------- files: model, packages | | | download (once, then cached) | v | | model + package --> reply | +-----------------------------------+
- Your question becomes 256 numbers (a vector) on your device, and only those go to the router.
- The router compares them with each subject and answers with the package to use, or none.
- The package downloads once and stays in your browser; the model and the package write the reply together.
The models
Each is Qwen3.5, stored at 4 bits with its vocabulary cut to the tokens English and web code use. The first is what the chat starts with; the others can be chosen in its settings. A phone gets the 2B as a smaller file, its vocabulary cut harder (94K tokens where the full file keeps every ASCII token, 141K), because a phone's browser gives a site about 1 GB of storage and a tab little more in memory; it predicts English the same. Speed is the writing speed of a reply on one desktop (a Ryzen CPU with seven helper threads, or an RTX 4090 with WebGPU); phones are slower.
| Model | Download | Same next word | Perplexity, full → 4 bits | CPU tokens/s | GPU tokens/s |
|---|---|---|---|---|---|
| Qwen3.5-0.8B | 364 MB (329 gzipped) | 88% | 4.74 → 4.93 | 95–101 | 182–190 |
| Qwen3.5-2B | 1,025 MB (920 gzipped) | 90% | 3.83 → 3.90 | 43–51 | 130–133 |
| Qwen3.5-2B (phones) | 890 MB (805 gzipped) | 88% | 3.83 → 3.94 | – | – |
| Qwen3.5-4B (desktops) | 2,267 MB (2,051 gzipped) | 92% | 3.44 → 3.48 | 19–25 | 73–78 |
Each 4-bit model is compared with its own full-precision weights on held-out answers from a much bigger model: how often both pick the same next word, and the perplexity of those answers before and after rounding (lower is better; the 2B predicts them better than the 0.8B by a point, the 4B by more).
Small Qwen3.5 models tend to think in circles: left to themselves, the 0.8B and the 2B end their own thinking only one time in twenty. Each carries a tiny edit to two of its layers (0.02% of its weights, a single direction removed, after Weight-Space Programming) that lets the thinking conclude: the 0.8B now ends it 99 times in 100, the 2B 17 times in 22 within the chat's budget. The 4B doesn't loop and is unedited.
Privacy
- Your text never leaves your device. The router gets vectors, not words.
- A vector isn't the text, but it is an average of word meanings, so roughly which words were used could be worked out from it.
- Downloading a package tells the server its subject, the first time only.
- The router logs which packages it picked and how long it took, nothing else. There are no accounts, cookies or analytics.
- Turn Packages off under the chat and nothing leaves your device at all: the base model answers alone.
How well it works
| Measure | Result |
|---|---|
| The better answer, with packages or without | with 33, without 1, 6 ties |
| Answers that wouldn't mislead someone following them | with 25 of 40, without 1 of 40 |
| Claims in an answer that a fact check finds false | with 1.25 (19%), without 3.09 (42%) |
| TruthfulQA: the true answer picked among common misconceptions | with 29–31%, without 26% |
| Questions sent to the right subject | 89–95% |
| Off-topic questions given no package | 92% |
The first two judged blind: each of 40 typed questions answered both ways, labels hidden, order random. The fact check splits answers to 550 typed questions into claims and has Qwen3-8B judge each on its own.
It is still a small model, and it can be confidently wrong, about law and money too. Packages mostly teach it how to answer a subject, not new facts: on exam questions they move its score by about a point. Check anything that matters.