Welcome Inkling 2 Mini, Cutting-edge Performance for your Local Setup
Inkling 2 Mini is a compact, loop‑based Mixture‑of‑Experts model that delivers near‑large‑model performance for local setups, with multimodal support, flexible looping, and ready‑to‑use integrations across Transformers, llama.cpp, MLX and SGLang.

Introduction
Inkling 2 Mini is the latest release from Thinking Machines Lab, designed to bring high‑end transformer capabilities to a local environment. By looping a core set of layers three times, the model behaves like a much deeper network while keeping the physical checkpoint small. It supports multimodal inputs (text + images), a 1 M‑token context window, and a range of inference back‑ends, making it a practical choice for developers who want cutting‑edge performance without a cloud subscription.
How the Loop Architecture Works
- Physical vs. logical depth: The model has 24 physical decoder layers, but layers 6‑17 are visited three times, yielding 48 logical layer passes. This looped core reuses the same weights, reducing storage while still delivering the expressive power of a deeper network.
- Mixture‑of‑Experts routing: Each token is routed to 8 out of 128 experts per layer (the larger Inkling 2 Small uses 384 experts). The router normalizes scores and scales the selected weights, keeping inference efficient.
- Cross‑pass messaging: Between loop passes, a small gating controller mixes the current token representation with a learned message from the previous pass. This refinement step lets the model iteratively improve its hidden state without additional parameters.
- Attention design: The model combines sliding‑window attention (1,024‑token windows) with occasional full‑attention layers that use dynamic key/value convolutions, 24 query heads, and 8 KV heads. Relative‑position logits and an output gate further enhance long‑range reasoning.
- Variable‑loop research variant: A companion checkpoint allows you to set the number of loop passes (1‑3) at inference time. Fewer passes speed up generation at a modest quality trade‑off, offering flexibility for resource‑constrained devices.
Getting Started with Popular Toolkits
Transformers (Python)
```python from transformers import AutoModelForMultimodalLM, AutoProcessor modelid = "thinkingmachines/Inkling-2-Mini" processor = AutoProcessor.frompretrained(modelid) model = AutoModelForMultimodalLM.frompretrained(model_id)
messages = [{"role": "user", "content": [{"type": "text", "text": "Why is the sky blue?"}]}] inputs = processor.applychattemplate(messages, tokenize=True, addgenerationprompt=True, returntensors="pt").to(model.device) output = model.generate(**inputs, maxnewtokens=256) print(processor.decode(output[0], skipspecialtokens=False)) ``` For the variable‑loop checkpoint, pass `textconfig={"loopnumpasses": 2}` when loading the model.
Llama.cpp & Llama App
- Quantized GGUF files: 4‑bit (IQ4NL) and 8‑bit (Q80) versions are provided.
- CLI serving:
```bash llama serve -hf ggml-org/Inkling-2-Mini-GGUF:IQ4NL # fixed loop llama serve -hf ggml-org/TML-d24-varloop-research-GGUF:IQ4NL # variable loop ``` The Llama App bundles a local OpenAI‑compatible API, letting you call the model from any editor or script.
MLX (Apple Silicon)
Install the latest `mlx` package and load the model via the standard `mlx.nn` loader. The model runs in bfloat16 by default, taking advantage of the M2/M3 GPU for low‑latency inference.
SGLang (Day‑0 Production)
```bash export MODEL="thinkingmachines/Inkling-2-Mini" python -m sglang.launch_server \ --model-path $MODEL \ --served-model-name inkling \ --attention-backend fa4 \ --bf16-gemm-backend torch \ --dtype bfloat16 \ --tp 1 --page-size 128 \ --host 0.0.0.0 --port 30000 ``` SGLang adds built‑in tool‑calling, streaming, and grammar‑guided generation, making Inkling 2 Mini ready for production workloads on a single GPU.
Practical Tips for Local Deployment
- Memory budgeting: The 4‑bit quantized checkpoint fits comfortably on a 24 GB GPU; the 8‑bit version provides a small accuracy boost at the cost of ~2 GB more VRAM.
- Loop selection: Use the fixed‑loop model for maximum quality. Switch to the variable‑loop checkpoint with `loopnumpasses=1` when you need sub‑second responses on a laptop.
- Reasoning effort: The processor’s `reasoning_effort` flag lets you trade speed for depth (e.g., `medium` ≈ 0.7). Lower values reduce token‑level computation, useful for batch inference.
- Tool calling: Wrap your own functions (weather APIs, database lookups, etc.) and pass them via the `tools=` argument in the chat template. The model will emit a structured tool call that you can execute and feed back.
- Benchmark awareness: While Inkling 2 Mini ranks highly on coding and multimodal benchmarks, always validate on your specific workload—especially if you rely on the variable‑loop mode.
Closing Thoughts
Inkling 2 Mini demonstrates that clever architectural tricks—looped layers, expert routing, and cross‑pass messaging—can deliver large‑model quality without the usual hardware burden. With out‑of‑the‑box support across the major inference stacks, it’s ready for developers, researchers, and hobbyists who want a powerful multimodal model on a single GPU or even a high‑end CPU. Experiment with the loop count, quantization level, and reasoning effort to find the sweet spot for your use case, and you’ll have a versatile AI companion that rivals much larger models while staying comfortably local.
Source
Read original sourceWhy we picked this
Announces a new Hugging Face model for local use, core AI content.