compute budget · 6 GB VRAM

Henrique Echternacht · LinkedIn · GitHub

I made AI run on my 6 GB VRAM laptop.

Models that normally need a data-center GPU, running on a laptop graphics card with 6 GB of memory. I rewrote parts of the engines to get there, then built the apps on top: assistants you can talk to, songs written and sung by a model, tools that see the screen.

Strata Coder · 177B parameters 25 tok/s · 0 tokens

This text appears at 25 tokens per second, the speed of a 177-billion-parameter coding model on a 6 GB laptop GPU. It is far too big for the card, so the engine splits the work between the GPU and system RAM.

Qwen3.6 MoE · 35B parameters 48 tok/s · 0 tokens

This one runs at 48 tokens per second. It started at 28 before my engine changes, which are described further down this page.

largest model on 6 GB: 177B
faster Qwen3.6 decode: 1.7×
faster prompt reading: 3.1×
my own projects: 104

Each section below goes one level deeper. Stop wherever it stops being useful.

Level 1 · The apps

Apps I built around the models

The models are open source. What I add is the software that makes them useful on one laptop: assistants, routers, editors and the glue between models. These are my favourites, shown running locally.

Plate 01 · multi-model system

Omni-call

A phone-style voice call with an AI that runs on my laptop. It listens, answers out loud, and runs errands in the background while you keep talking.

Full-duplex turn taking with barge-in: talk over it and it stops.
A planner turns each spoken turn into parallel tool calls: search, files, Python, reminders.
Speech on the CPU so the whole 6 GB GPU is left for the language model.

Plate 02 · developer tool

Code Flow Canvas

A VS Code extension that draws a Python project as a node graph, ComfyUI-style, and lights up the wires while the code runs. Shown on its sample FastAPI app.

Static analysis maps functions, calls, globals and data flow; refreshes on save.
A live tracer streams calls and shows the last value on every wire.
Pause a call to edit its arguments, or inject your own call into the running program.

Plate 03 · workbench

Project Hub

A Netflix-style catalog of the 119 folders I work in (104 of my own projects, 15 upstream clones) and 55 Python environments, without moving a single folder.

Drop a folder and an AI agent reads it, then files it with a summary and run command.
Moved projects are found again by fingerprint; one click runs any project.
Finds the existing Python environment that already has a project's dependencies.

Plate 04 · model router

freectl

One AI endpoint that pools the free models of many providers. Ask for "default" and the strongest model that is answering right now replies. It also powers the chat on this page.

Finds free models automatically and ranks them by size parsed from their names.
Falls through to the next model on any error, even mid-stream.
Runs on my PC or as a Cloudflare Worker.

Plate 05 · multi-model system

LLM Radio + musicraw

A radio station that never repeats: an AI DJ picks what to play, another model writes the lyrics, and a music model on my PC sings it while the current song plays. musicraw goes the other way: any real song becomes a karaoke video.

Lyrics and audio workers run in parallel and keep about three songs ready.
musicraw re-inserts chorus repeats that lyrics sites leave out, by matching a transcript of the vocals.

little-coder · my extensions
› /voice        talk to the agent, hear its replies
› websearch     cascade over ~25 providers
› webfetch      PDF / HTML → clean Markdown + answer
› python        persistent scratch env (uv)
› /web-setup    reorder providers and keys
› auto-resume   "continue" after a model error
› freectl-model which free model answered
› /lcraw /raw   full prompt or raw server request
› /manage       toggle extensions, skills, tools
› install       one command on a new machine

Plate 06 · agent tooling

little-coder extensions

little-coder is a coding agent built for small local models. I extended it until it fit how I work: voice, web research that keeps working, a Python tool, and Windows fixes.

/voice reuses Omni-call's speech stack: speak a prompt, hear the answer, typing cuts the speech.
Web tools fall through ~25 search and summary providers; failures stay out of the model's context.
Patches for Windows paths and broken JSON edits in the bundled agent.

Plate 07 · optimization

GPUUnlock

Windows tuning that keeps all 6 GB of my laptop's NVIDIA GPU free and awake for AI, by moving everyday apps to the integrated GPU.

A Rust tray app watches per-app GPU memory and moves VRAM-hungry apps to the iGPU, skipping games.
An idle CUDA context keeps the GPU powered for 1.3 W; a plain D3D device let it sleep after ~25 s.
Every registry change is recorded and undone with one click.

Level 2 · What I built

Everything I built, by what it took

Six groups, ordered from the hardest work to the simplest: engine optimizations, systems that wire several models together, fine-tuned and vision models, tooling for coding agents, interfaces, and the infrastructure I run them on. Click anything for its spec sheet.

Desktop tooling experiments

Small Windows utilities for my own use, built to try out low-latency screen capture, input handling and hardware I/O. Only the stack is listed here.

scshtRust + Python GUI · RP2040 microcontroller as a USB device · build pipeline

RustPythonRP2040

Loot splittertray app · clipboard parsing · Rust helper binary

PythonRustpystray

PixelTriggerRust egui · screen sampling · worker threads decoupled from the UI

RusteguiWin32

Olheirocapture card + dxcam frame grabbing · win32 window capture · notebook prototyping

PythondxcamOpenCV

Level 3 · How it's wired

What happens between input and answer

Three systems traced from start to finish. Each box is a separate process or model. The small text under each box shows what it does there.

Omni-call: a voice and video call with a local model

Speech recognition and speech synthesis run on the CPU, so the GPU's 6 GB stay free for the language model. Interrupting it mid-sentence cancels the reply.

Browsermic + camera, voice activity detection
Node serverWebSocket /rt, HTTPS
Parakeet ASRspeech → text, ~170 ms/turn on CPU
LLM routerllama.cpp / Strata, tool calls
Tool agentweb, files, shell, Python, reminders
TTSsentences streamed back as 24 kHz audio

musicraw: YouTube song → karaoke video

Lyrics sites print the chorus once, but singers repeat it. So the transcript is matched against the lyrics, and the missing repeats are put back before alignment.

yt-dlpaudio + title
Lyrics fetchGenius, lyrics.ovh
UVR MDX-Netisolate the vocal stem
Parakeet ASRword-level transcript
expand.pyre-insert repeated choruses
CTC alignword timings → .srt
ffmpegmux to .mkv

This website and everything else at echternacht.dev

This page is on Cloudflare Pages, so it stays up when my PC is off. Apps on other subdomains go through a tunnel to a 12 MB Go router on my PC. When nobody is using an app, the router stops it, and it starts again when someone visits.

Visitorname.echternacht.dev
CloudflareDNS, TLS, Pages
Tunnelcloudflared, outbound only
Go routerlogin, wake on visit, idle stop
Local appseach in a Windows Job Object

Level 4 · Engine room

Changes I made to open-source engines

Most local AI runs on a few C++ engines. I keep my own forks of them, merge upstream regularly, and change what doesn't fit a 6 GB laptop GPU. Line counts are my commits only, not upstream's.

Lines I added and removed per fork

llama.cpp

34 commits · ggml-org/llama.cpp

Hybrid MoE expert cache. The experts the router picks most often (LFU with decay) stay in a VRAM slot pool. Hits run on the GPU, misses on the CPU, and the results are summed. Decode went from 31 to 34.6 tok/s (+11%) on Qwen3.5-MoE.
Weight prefetch on a second CUDA stream, plus a Windows mmap prefetch that makes the model fully resident before loading returns.
Tool-call checkpoint in the server, so the next agent turn resumes right after the reasoning instead of re-reading the prompt.
Reverted upstream's MMQ refactor (#24127), which broke MUL_MAT_ID with CPU-offloaded experts on sm_89 GPUs.
Merged in PrismML's Q2_0, MBZUAI's K2 Horizon model and adaptive KV streaming from other forks.

Strata

30 commits · local AI engine

End result on the 6 GB laptop: Qwen3.6 MoE decode went from 28 to 48 tok/s, and the 177B Strata Coder runs at about 25 tok/s.
Expert load halved: the copy into the arena now fans out over threads, and only the padding gets zero-filled. Load went from 31.5 to 12.8 s and the output bytes are identical.
One staging buffer instead of two cut prompt staging from 0.97 to 0.49 GiB. The freed memory went back to decode slots.
A config that fits in VRAM: 12 GiB of pinned memory, 66K context, no spill to system memory. Decode went from 41 to 46.8 tok/s.
Benchmark harnesses that swept 24 configurations (chart below) and a settings app for the engine switches.

audio.cpp

19 commits · music & speech engine

A durable HTTP job server with cancellable jobs, SSE progress and a record written before the model runs, so crashes stay visible.
YuE2 VRAM fitting: the token budget is cut to fit free memory. Running out of memory now gives a shorter song instead of an error.
An audio-to-score job using the SheetSage2 native runtime.

little-coder

3 commits · coding agent for small models

Web tools rebuilt: GitHub links go to the raw README, PDFs become text, and HTML goes through Readability → Markdown. A provider cascade (Exa → local Strata → DuckDuckGo) can be switched with /web-setup.
write-guard backs up an existing file instead of refusing to write it. auto-resume sends "continue" after an error the agent couldn't recover from.
Compact swap: before sub-agents fan out, the active model switches to a 2B model so the big model's VRAM is free for them.
/lcraw dumps the full assembled prompt and tool schemas for audit. /manage toggles extensions, and a python tool runs a persistent uv scratch env.

HOT-Step-CPP

2 commits · music app

A 6 GB mode. The MERT attention window fell from 3.6 GB to about 0.36 GB.
A q4_0 KV cache for MiniMax-Music3: 81 kB per position instead of 288 kB in f16.

How the hybrid expert cache decides where an expert runs

Each token picks 8 of 256 experts. Experts already in the GPU slot pool (outlined) run there and light up blue. The rest run on the CPU from system RAM and light up orange. Both results are added together, and the cache keeps the most popular experts as the routing changes.

Level 5 · Measurements

Before and after, on the same laptop

RTX 4050 Laptop (6,141 MiB), i7-13620H, 24 GB DDR5. The numbers come from my benchmark logs and commit messages. Hover or tap a chart for exact values.

Every optimization, normalized to its "before"

Bar = how much of the original cost is left. Shorter is better.

Strata: 24 configs, prompt speed vs decode speed

Qwen3.6 MoE, pp2048 / tg128, warm run. Top right is best.

Where the 6 GB ceiling bites

Decode tok/s vs expert layers on the CPU (-ncmoe). Fewer layers on the CPU means more weights in VRAM. Below 32, Windows starts paging VRAM.

Decode stops scaling at 6 threads

tg128 tok/s by CPU threads. Past 6 threads, memory bandwidth is the limit, not compute.

Long context costs little

tg64 tok/s by context depth. Hybrid SSM layers keep the slowdown at 27% at 64K.

Chimp128 vs zstd on float time series

1,000,000 noisy-sine float64 values (8 MB). My Rust codec is 23% smaller, and the round trip is bit-exact.

A negative result I kept: early-exit heads

SmolLM2-135M on SST-2, validation accuracy per epoch. Extra heads trained 25% faster but scored 0.7 pp lower.

Can a frozen vision model judge sprite quality the way people do?

Agreement with 151 human-judged sprite pairs. Coin flip is 50%, and people agree with themselves 92% of the time. No model got close, so sprite review stays manual.

Level 6 · Workbench

How 104 projects stay findable

Project Hub is a catalog I built for my own disk. It indexes projects wherever they are with symlinks, never moves the real folders, and tracks status, stack and duplicates. The charts cover all 119 catalogued folders: my 104 projects plus 15 upstream clones I build against.