Skip to content

Field notes · AI

This AI Model Runs Its Middle Layers Three Times. Here's How to Tell If It Will Run on Your Computer

Inkling 2 Mini is an open-weights model built to fit on a laptop. It stores 26.4B parameters but runs its middle layers three times per token. That changes the usual rule for picking a download. We ran the fit and speed math from the real file sizes, and show how to tell a wrong build from a too-small machine from a day-one bug.

A row of 24 blocks in four groups of six. The first and last groups have a single arrow. The two middle groups have a loop arrow marked x3 in the accent color. A counter reads 24 stored, 48 applied.

A field note for anyone who wants a capable AI model running offline on their own machine. The fit and speed table below comes from a small script we ran on our build box this morning against the published file sizes. We did not run the model itself; why is explained below.

The short version

On Nov 20, Thinking Machines Lab released Inkling 2 Mini, an open-weights model under the Apache 2.0 license that they describe as “small enough to run on your laptop.” By Saturday morning it was the top trending model on Hugging Face and over 1,000 points on Hacker News.

It has a twist. Most models run each layer once per word. Inkling 2 Mini runs its middle 12 layers three times. That keeps the download small, but every word still costs more work than the “active parameters” number suggests.

So the honest answer to “will it run on my computer?” takes two questions, not one:

  1. Does it fit? That depends on the file size, which depends on the quant you pick.
  2. Will it feel fast? That depends on how many bytes your machine has to read for every word, and looping raises that number by about 1.6x.

Your first step: check how much memory you have, then pick the file from the table below. On a Mac, Apple menu, About This Mac. On Linux, free -g, plus nvidia-smi if you have an NVIDIA card.

When it fails, ask three questions in order

Day-one local models are rough, and this one already has open bugs. Before you blame your hardware, work through it the way you’d work any outage: rule out the cheap causes first.

  1. Is it the wrong build for this file? An “unknown architecture” or load error usually means a mismatch, not a weak machine. Per Unsloth’s run guide, the GGUF files in the root of their repo use Unsloth’s own looped-layer layout and need their b12588-mix-c0c3de8 llama.cpp build. Mainline llama.cpp reads a different layout, published in the repo’s Mainline_Support folder.
  2. Is the machine too small? Out-of-memory errors, or a system that crawls because it’s swapping to disk, mean the file plus context doesn’t fit. Drop one quant step or shrink the context. Unsloth’s examples use a 32K context. The model supports up to 1M, but that’s a ceiling, not a setting to start with.
  3. Is it a known bug? On NVIDIA, check llama.cpp issue #32710, perplexity NaN on CUDA, and #32707, very slow prefill and crashes on an RTX 5090. Both were open this morning. If your symptom matches, wait a few days for a fix instead of rebuilding your setup.

The habit that saves the most time: on launch week, read the issue tracker before you debug your own box.

A few terms first

  • Parameters are the numbers a model learned. More usually means smarter, and bigger.
  • Mixture of Experts (MoE) means each layer has many small “expert” blocks and only uses a few for each word. The model card says each word goes to 8 of 128 experts. Most of the model sits idle on any one step.
  • Active parameters are the ones actually used for one word.
  • A looped transformerThe neural network architecture behind modern AI language models, using a self-attention mechanism to process all input tokens simultaneously. Read more → reuses the same layers more than once per word instead of stacking more layers.
  • Quantization, or a “quant,” stores each number in fewer bits so the file shrinks. Smaller file, slightly less accurate.
  • GGUF is the file format llama.cpp and many desktop apps use to run local models.

What looping actually changes

Picture a 48-station assembly line. A normal model would need 48 different machines. Inkling 2 Mini owns 24 machines and sends each part through the middle 12 three times. The factory floor is smaller. Each part still takes almost as long to build.

The numbers from the model card:

  • 24 stored layers. The middle 12 run three times with shared weights, for 48 layer applications per word.
  • 26.4B total parameters, 3.4B active.
  • Each extra pass adds about 1.0B “effective active” parameters, so with three passes it’s 5.4B effective active per word.

Here’s why that matters on your machine. When a local model writes text one word at a time, the bottleneckThe single component or resource that limits a system's overall performance, so improving anything else makes little difference until it is fixed. Read more → is usually memory bandwidthThe maximum amount of data that can be transmitted over a network connection in a given period, typically measured in megabits or gigabits per second. Read more →, meaning how fast your RAMRandom Access Memory: the fast, temporary working memory that holds the programs and data a computer is using right now, cleared when power is lost. Read more → or GPUGraphics Processing Unit: a processor with thousands of small cores built for parallel math, used for displaying graphics and increasingly for AI workloads. Read more → memory can feed weights to the processor. Thinking Machines’ own speed chart agrees. It calls its figures a “theoretical ceiling at 614 GB/s … including repeated reads across loops.” Looping saves memory. It doesn’t save reads. So speed tracks the 5.4B number, not the 3.4B one.

Two bars. Stored in memory, 26.4B. Read per token, 3.4B active plus 2.0B from two extra loops for 5.4B, with the 2.0B segment in the accent color and a label reading loop tax 1.59x.
Looping saves memory, not reads. Each word reads about 1.59x what the active number suggests.

One more thing looping doesn’t buy you. The launch post says plainly that looping “doesn’t enable its weights to store more facts.” Knowledge benchmarks didn’t improve with more passes, and the model leans on search and tools to make up for it. If you want trivia from memory, this model’s size still sets the limit.

What we ran: the fit and speed math

Our build box has 15 GB of RAM shared with other work and no GPU, so we did not download or run the model. Real load and tokens per second are untested here. What we did run is the math, using the actual file sizes from the Hugging Face API for unsloth/Inkling-2-Mini-GGUF, read this morning, and the parameter counts from the model card.

The method is simple. Divide file size by total parameters to get average bits per weight. Multiply by the 5.4B effective active parameters to estimate bytes read per word. Divide your memory bandwidth by that to get a speed ceiling. We kept 3 GB free for the OSThe core software that manages a computer's hardware and gives every other program a safe, shared way to use the CPU, memory, storage, and network. Read more →, apps and context, which is our assumption.

Selected rows from the script output at 16, 24 and 32 GB, using the 614 GB/s figure from Thinking Machines’ chart:

quant        file GB bits/wt GB/token ceiling tok/s  fits 16/24/32 GB
UD-Q2_K_XL      9.14    2.77     1.87           328  yes/yes/yes
UD-IQ3_XXS     11.66    3.53     2.39           257  yes/yes/yes
UD-Q3_K_XL     13.63    4.13     2.79           220  no/yes/yes
UD-Q4_K_XL     17.63    5.34     3.61           170  no/yes/yes
UD-Q8_K_XL     22.87    6.93     4.68           131  no/no/yes
loop tax: 5.4B effective / 3.4B active = 1.59x reads per token

How to read it:

  • Fit is the hard gate. On a 16 GB machine with 3 GB kept free, the largest file that fits is UD-IQ3_XXS. At 24 GB you reach UD-Q4_K_XL, which Unsloth recommends. At 32 GB the 8-bit file fits.
  • Speed is a ceiling, not a promise. 614 GB/s is the figure Thinking Machines used for its chart, not your machine. Look up your own memory bandwidth and rerun the script. Real speed will land well below the ceiling.
  • The loop tax is the lesson. At any quant, each word reads about 1.59x what a 3.4B-active model of the same size would read. If you compare this model to another MoE by its “active” number alone, you’ll guess its speed wrong.

When two sources disagree, reconcile them

The model card lists support for Apple silicon with 32 GB or more and recommends 32 GB or more of VRAM on other GPUs. Unsloth’s guide says 16 GB or 24 GB works. That looks like a contradiction until you check what each one is loading. The card describes the official checkpoint, about 20 GB in its native NVFP4 and MXFP8 formats. Unsloth is talking about smaller quantized GGUF files: its table lists about 9 GB for 1-bit, 10 to 11 GB for 2-bit, 13 to 15 GB for 3-bit, 17 to 18 GB for 4-bit and 24 GB for 8-bit. Both are right. They describe different files.

Unsloth’s ranges are more conservative than our 3 GB headroom guess. Trust their table for the purchase decision, and use our script to understand why.

Your first build

The easiest path is Unsloth Desktop for macOS, Windows or Linux: search for Inkling 2 Mini, pick the unsloth/Inkling-2-Mini-GGUF quant that fits, and chat. If you prefer the command line, this is the run command from Unsloth’s guide, with their recommended sampling settings:

unsloth run --model unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_XL \
    --temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0

SwapDisk space the operating system uses as overflow when physical memory (RAM) fills up, at the cost of being much slower than RAM. Read more → UD-Q4_K_XL for the quant your memory allows. The guide also offers Homebrew, WinGet and DockerA platform that packages applications into containers, providing a standardized way to build, ship, and run software consistently across any environment. Read more → installs, plus a curl | sh script. If you use the script, read it before you run it.

Then measure. Note your real tokens per second for one quant and compare it to the ceiling for your bandwidth. The gap tells you how much overhead your setup adds, and it’s a number you can carry to the next model.

Related on BytesNation: why open weights don’t always mean laptop-ready, the opposite case from this one, and how AI actually works if the terms above are new.

Go deeper: the formula, the full table, and the vendor claims

The ceiling formula. tok/s ceiling = bandwidth (GB/s) / (effective active params x bits per weight / 8 / 1e9). Our bits per weight is the file size averaged over all 26.4B weights, so it blurs the difference between expert weights and shared weights, and it ignores reads from the context cache. Treat it as a rough upper bound.

The full file list from the APIA set of rules and protocols that allows different software applications to communicate with each other and share data or functionality. Read more →: UD-IQ1_S 7.76 GB, UD-IQ1_M 8.02, UD-IQ2_XXS 8.49, UD-Q2_K_XL 9.14, UD-IQ3_XXS 11.66, UD-Q3_K_XL 13.63, UD-IQ4_XS 15.73, UD-Q4_K_M 16.61, UD-Q4_K_XL 17.63, UD-Q8_K_XL 22.87. The vision projector (mmproj-F16.gguf) adds about 0.64 GB if you want image input, and the optional DSpark draft model for speculative decoding is about 0.4 GB.

Where the loop sits. Thinking Machines splits the 24 layers into four blocks of six. The first and last run once; the two middle blocks run three times. In their training runs, looping the middle layers beat looping the last layers on 27 of 28 evaluations. Source: the launch post.

Monitorability. Looping lets a model do more reasoning without writing it out. Thinking Machines studied this with a variable-loop research variant released alongside the model. They report that chain of thought got shorter with more loops but stayed legible, and that scores on agentic sabotage tests did not rise. That is the vendor’s own study, not an independent one.

Benchmarks. The launch post reports 48.2% on DeepSWE v1.1 and 46.5% on Vals Index v2.2, and lists limits too: it trails Qwen3.8-27B on multi-turn chat (MultiChallenge 50.0%) and many-constraint instructions (ComplexConstraints v1.1 33.4%). All of these are reported by Thinking Machines. Evaluation setups vary by model, so wait for independent runs before treating rankings as settled.

What we didn’t do. We didn’t use Tinker, Thinking Machines’ hosted platform, because this note is about running locally at no cost.

The script. will_it_fit.py (Python 3.13 standard library, no installs) reads tree.json, the Hugging Face file listing we saved this morning. Run python3 will_it_fit.py --mem-gb 24 --bw-gbs <your bandwidth> --headroom-gb 3. Full output for 16, 24 and 32 GB at 614 GB/s is in run-614gbs.txt, and a 16 GB example at 100 GB/s (a made-up bandwidth to show how the ceiling scales, not a real device) is in run-16gb-100gbs.txt. At 100 GB/s every ceiling drops by about 6x: UD-IQ3_XXS goes from 257 to 42 tok/s.

Scripts

src/will_it_fit.py

#!/usr/bin/env python3
"""will_it_fit.py: rough fit and speed-ceiling math for Inkling 2 Mini GGUFs.
Inputs: Unsloth GGUF file sizes (Hugging Face API, read Nov 21, 2026), model card params.
Usage: python3 will_it_fit.py --mem-gb 16 --bw-gbs 614 --headroom-gb 3
All speeds are theoretical ceilings, not measurements."""
import argparse, json, os
p = argparse.ArgumentParser()
p.add_argument("--mem-gb", type=float, default=16, help="RAM+VRAM or unified memory, GB")
p.add_argument("--bw-gbs", type=float, default=614, help="memory bandwidth, GB/s")
p.add_argument("--headroom-gb", type=float, default=3, help="memory kept for OS, apps, context")
a = p.parse_args()
TOTAL, ACTIVE, EFFECTIVE = 26.4e9, 3.4e9, 5.4e9   # model card
here = os.path.dirname(os.path.abspath(__file__))
tree = json.load(open(os.path.join(here, "tree.json")))
rows = sorted((f["size"], f["path"]) for f in tree
              if f["path"].startswith("Inkling-2-Mini-UD-") and f["path"].endswith(".gguf"))
print(f"memory {a.mem_gb:g} GB, headroom {a.headroom_gb:g} GB, bandwidth {a.bw_gbs:g} GB/s")
print(f"{'quant':<12}{'file GB':>8}{'bits/wt':>8}{'GB/token':>9}{'ceiling tok/s':>14}  fits?")
for size, path in rows:
    q = path.removeprefix("Inkling-2-Mini-").removesuffix(".gguf")
    gb = size / 1e9
    bpw = size * 8 / TOTAL
    read = EFFECTIVE * bpw / 8 / 1e9
    fits = "yes" if gb + a.headroom_gb <= a.mem_gb else "no"
    print(f"{q:<12}{gb:>8.2f}{bpw:>8.2f}{read:>9.2f}{a.bw_gbs/read:>14.0f}  {fits}")
print(f"loop tax: {EFFECTIVE/1e9:.1f}B effective / {ACTIVE/1e9:.1f}B active = {EFFECTIVE/ACTIVE:.2f}x reads per token")

src/tree.json

[{"type":"directory","oid":"b4817e43fe8c0c5b3ed740040384b8b4b28a43c4","size":0,"path":"Mainline_Support"},{"type":"directory","oid":"2c9c32ef60ba28084173d6dfb152a263742901cd","size":0,"path":"assets"},{"type":"directory","oid":"5a9f39f3532e92a0f8e3163f9c9bdd0d60a33a32","size":0,"path":"dspark"},{"type":"file","oid":"d703180958de894044f74981144da803ef36b08d","size":5495,"path":".gitattributes"},{"type":"file","oid":"2ba33dd707e5402336428c7aca01e4c8870c8a07","size":8024441024,"lfs":{"oid":"544e0b2412ba2392272b0510b0c8f53a1ab872b73c2ba4f7488de0d5d61467d9","size":8024441024,"pointerSize":135},"xetHash":"701b3f43f90ba53672f110ddeff34c520e4fa4953192de38e0acb4e77a9a2fe2","path":"Inkling-2-Mini-UD-IQ1_M.gguf"},{"type":"file","oid":"22506acce0f1fe140c05431294f2df27d336059d","size":7764291776,"lfs":{"oid":"c09c028b050c108624682baeb28ac6e0cdb936af021ac637c8c3609f923ea4b8","size":7764291776,"pointerSize":135},"xetHash":"73f674d6e1058ca1a14b29fd7b39f5f12cf0cc9b08bbec95160fe4bbe5e0731b","path":"Inkling-2-Mini-UD-IQ1_S.gguf"},{"type":"file","oid":"5cd08dfce087543203d02e411c7b1888b5fd9889","size":8486236352,"lfs":{"oid":"4fe958f73a667d8082f911a4347e52bc34696a7816cf3d5b9f7c8a81d5c5ceaa","size":8486236352,"pointerSize":135},"xetHash":"fdb95ae81aaf4712b238cf3e6dac2a170a2350cb6d5bffd7712ba9c563052bcd","path":"Inkling-2-Mini-UD-IQ2_XXS.gguf"},{"type":"file","oid":"e49626d46f504ef198a46836d3d7ca5b1e355aa2","size":11663309504,"lfs":{"oid":"bd7ef0865adf119b5183e45e741c629bafa649769b33b5ebf689bb4bcc4690d6","size":11663309504,"pointerSize":136},"xetHash":"642af3bb8bc95d634f3170b443be43db39e55efbc775bad4f3705b7b93aaa59e","path":"Inkling-2-Mini-UD-IQ3_XXS.gguf"},{"type":"file","oid":"529737212a406b0cf3894936f316baea7c57752e","size":15733755232,"lfs":{"oid":"a861d21e7c6d7bc69d8cdcdfb9eb195fd37ef1cd2ab31499db61cd7d30d9b3a1","size":15733755232,"pointerSize":136},"xetHash":"a26c6f78f80def3a77210f79a138cd48667f8227d71c5f37d2f979dacd49c84a","path":"Inkling-2-Mini-UD-IQ4_XS.gguf"},{"type":"file","oid":"2bdc0ecc0b68c38199dd17d8cc20054d112192e9","size":9138715328,"lfs":{"oid":"20292e8142b69535ca88a5d04439d0fdeb4dc698d44c002dca706dcf92901441","size":9138715328,"pointerSize":135},"xetHash":"9ae8597e6a487dab16b166db14fd2ea278193ecc13866592a03afcbac7303ecf","path":"Inkling-2-Mini-UD-Q2_K_XL.gguf"},{"type":"file","oid":"38309706dc2e217c49b913a95cd5644e83ca21ef","size":13631778016,"lfs":{"oid":"f72f396d1b479b14d76346df80baa449c8b1f399653bff38fa4764b359a40048","size":13631778016,"pointerSize":136},"xetHash":"b8706f1a9460037db8e4e6b8d87655c4e977898da2387d21d273a34cdb832e9a","path":"Inkling-2-Mini-UD-Q3_K_XL.gguf"},{"type":"file","oid":"71660105feaf6530a49e452c9628046507f62d95","size":16610543328,"lfs":{"oid":"f90c5a4df44d335ac9f9eea2a15c31d6c5a377b1b3cd9ff7fc2cf123f3cdf7e1","size":16610543328,"pointerSize":136},"xetHash":"c37abe60ccb1fd2519da4070f3110cf8eaff82eb4385964b9d6e3c61ecc3b054","path":"Inkling-2-Mini-UD-Q4_K_M.gguf"},{"type":"file","oid":"42ef2035f008b2b29164c23fdd91880e43ff9cbd","size":17629759456,"lfs":{"oid":"e62d463c135ffe283df8d94b279a96ca145ac128b52ce854478613045bf4de15","size":17629759456,"pointerSize":136},"xetHash":"b635217ed1343da61ca3213332f01e356b952d965a0b7b496d73cea280f3ab38","path":"Inkling-2-Mini-UD-Q4_K_XL.gguf"},{"type":"file","oid":"ca657537b29df69ce22f00f3a6590842352e56f9","size":22867451616,"lfs":{"oid":"dfb04bf37bee457f98800989dedf93ec442ede8636dbea9433992f1e527e9a41","size":22867451616,"pointerSize":136},"xetHash":"29cdffcafccd72c0b019105b20826bab5cf79204f64a72858346218edbec8497","path":"Inkling-2-Mini-UD-Q8_K_XL.gguf"},{"type":"file","oid":"037c70044ea2ab981b912185a099d4a106872b06","size":56290,"path":"README.md"},{"type":"file","oid":"50a885d01d799b63a58ec746c662a32203ab594e","size":402278560,"lfs":{"oid":"2ec78510da837e8fc3be54a14757a0c5fbd2d8d9e85a0d92853fafdf674f3773","size":402278560,"pointerSize":134},"xetHash":"44d6371e1a59e511ba42a0056c79aa3518f5ecbe557790a05d229269c3883319","path":"dspark-Inkling-2-Mini-Q8_0.gguf"},{"type":"file","oid":"336f756c5b70ab4970e683baec7f9e94f4d82157","size":641805760,"lfs":{"oid":"2c165c78790e78a3f45a2d1e9e931083e52ee69d45ff2a0dc9c281fdf1a666e7","size":641805760,"pointerSize":134},"xetHash":"816f5b19f3e8f53976c04408e840db37f7f096af20453babf9e00829656ab92e","path":"mmproj-BF16.gguf"},{"type":"file","oid":"e1406a8d6dde77ff754e1e034e379bed13c4b676","size":639348160,"lfs":{"oid":"3bff75ffeb2d6e0371ba7bd42e30d26a7d98aea681fa44b03b5a20891cfa2c5f","size":639348160,"pointerSize":134},"xetHash":"997a088366f15cca5415148a17fce85361d704041df80b83f62c6215b19a9673","path":"mmproj-F16.gguf"},{"type":"file","oid":"d8f085a11d8673eeff5266cc14f4cd042f315905","size":1273048512,"lfs":{"oid":"fa6f2478812f33876e279b6fe8e9f00ea5ff4e9db40868c98c0a39277a439e25","size":1273048512,"pointerSize":135},"xetHash":"5bf87d04b3aad70f9af63ca8078eda76ed1d8b1cf390a1b969040ec762a74910","path":"mmproj-F32.gguf"}]

src/run-614gbs.txt

memory 16 GB, headroom 3 GB, bandwidth 614 GB/s
quant        file GB bits/wt GB/token ceiling tok/s  fits?
UD-IQ1_S        7.76    2.35     1.59           387  yes
UD-IQ1_M        8.02    2.43     1.64           374  yes
UD-IQ2_XXS      8.49    2.57     1.74           354  yes
UD-Q2_K_XL      9.14    2.77     1.87           328  yes
UD-IQ3_XXS     11.66    3.53     2.39           257  yes
UD-Q3_K_XL     13.63    4.13     2.79           220  no
UD-IQ4_XS      15.73    4.77     3.22           191  no
UD-Q4_K_M      16.61    5.03     3.40           181  no
UD-Q4_K_XL     17.63    5.34     3.61           170  no
UD-Q8_K_XL     22.87    6.93     4.68           131  no
loop tax: 5.4B effective / 3.4B active = 1.59x reads per token

memory 24 GB, headroom 3 GB, bandwidth 614 GB/s
quant        file GB bits/wt GB/token ceiling tok/s  fits?
UD-IQ1_S        7.76    2.35     1.59           387  yes
UD-IQ1_M        8.02    2.43     1.64           374  yes
UD-IQ2_XXS      8.49    2.57     1.74           354  yes
UD-Q2_K_XL      9.14    2.77     1.87           328  yes
UD-IQ3_XXS     11.66    3.53     2.39           257  yes
UD-Q3_K_XL     13.63    4.13     2.79           220  yes
UD-IQ4_XS      15.73    4.77     3.22           191  yes
UD-Q4_K_M      16.61    5.03     3.40           181  yes
UD-Q4_K_XL     17.63    5.34     3.61           170  yes
UD-Q8_K_XL     22.87    6.93     4.68           131  no
loop tax: 5.4B effective / 3.4B active = 1.59x reads per token

memory 32 GB, headroom 3 GB, bandwidth 614 GB/s
quant        file GB bits/wt GB/token ceiling tok/s  fits?
UD-IQ1_S        7.76    2.35     1.59           387  yes
UD-IQ1_M        8.02    2.43     1.64           374  yes
UD-IQ2_XXS      8.49    2.57     1.74           354  yes
UD-Q2_K_XL      9.14    2.77     1.87           328  yes
UD-IQ3_XXS     11.66    3.53     2.39           257  yes
UD-Q3_K_XL     13.63    4.13     2.79           220  yes
UD-IQ4_XS      15.73    4.77     3.22           191  yes
UD-Q4_K_M      16.61    5.03     3.40           181  yes
UD-Q4_K_XL     17.63    5.34     3.61           170  yes
UD-Q8_K_XL     22.87    6.93     4.68           131  yes
loop tax: 5.4B effective / 3.4B active = 1.59x reads per token

src/run-16gb-100gbs.txt

memory 16 GB, headroom 3 GB, bandwidth 100 GB/s
quant        file GB bits/wt GB/token ceiling tok/s  fits?
UD-IQ1_S        7.76    2.35     1.59            63  yes
UD-IQ1_M        8.02    2.43     1.64            61  yes
UD-IQ2_XXS      8.49    2.57     1.74            58  yes
UD-Q2_K_XL      9.14    2.77     1.87            53  yes
UD-IQ3_XXS     11.66    3.53     2.39            42  yes
UD-Q3_K_XL     13.63    4.13     2.79            36  no
UD-IQ4_XS      15.73    4.77     3.22            31  no
UD-Q4_K_M      16.61    5.03     3.40            29  no
UD-Q4_K_XL     17.63    5.34     3.61            28  no
UD-Q8_K_XL     22.87    6.93     4.68            21  no
loop tax: 5.4B effective / 3.4B active = 1.59x reads per token
inkling-2-minilocal-ailooped-transformermixture-of-expertsggufquantizationllama-cppunslothmemory-bandwidthtroubleshooting

← All field notes