Henrique Echternacht · LinkedIn · GitHub

I made AI run on my 6 GB VRAM laptop.

Models that normally need a data-center GPU, running on a laptop graphics card with 6 GB of memory. I rewrote parts of the engines to get there, then built the apps on top: assistants you can talk to, songs written and sung by a model, tools that see the screen.

Strata Coder · 177B parameters 25 tok/s · 0 tokens

This text appears at 25 tokens per second, the speed of a 177-billion-parameter coding model on a 6 GB laptop GPU. It is far too big for the card, so the engine splits the work between the GPU and system RAM.

Qwen3.6 MoE · 35B parameters 48 tok/s · 0 tokens

This one runs at 48 tokens per second. It started at 28 before my engine changes, which are described further down this page.

largest model on 6 GB: 177B
faster Qwen3.6 decode: 1.7×
faster prompt reading: 3.1×
my own projects: 104

Each section below goes one level deeper. Stop wherever it stops being useful.

Level 1 · The apps

Apps I built around the models

The models are open source. What I add is the software that makes them usable on one laptop: interfaces, routers, schedulers and the glue between them. These are screenshots of my apps running locally.

Qwen Image UI edit screen: before and after slider over a generated image, prompt panel with presets and a VRAM budget estimate — **Qwen Image UI.** An image studio that fits in 4 GB of graphics memory. It starts the model servers on demand, rewrites prompts with a vision model, and estimates time and VRAM share before each edit.

Tunnel Router admin panel listing two routes with idle stop, wake on visit and start/stop controls; folders and commands blurred — **Personal Tunnel.** Puts apps on my PC at their own subdomain behind one login. It starts them when someone visits and stops them when they go idle. Paths are blurred.

freectl status page: default fallback cascade and a table of free models with parameters, context, score, latency and state — **freectl.** One endpoint over the free models of many providers. It ranks them by size, tests them, and falls back to the next one on any error.

musicraw lyrics editor: song list, timed lyric lines (blurred), waveform and synthesis controls — **musicraw editor.** Timed lyrics for any song, with a waveform, splice tools and re-synthesis of selected lines. Lyrics are blurred for copyright.

SystemOne UI: media input, text state and a builder for choice, yes/no and score questions — **SystemOne UI.** Ask a decision model typed questions about text, images or live video and read the answer probabilities. It skips frames that haven't changed.

Pencil concept art of an armoured warrior with an axe — **Sprite pipeline.** Concept art to a walking spritesheet, made with an image model I fine-tuned (LoRA on FLUX.2-klein) and scored with metrics I wrote.

Pixel-art spritesheet of the same warrior walking in four directions — **Sprite pipeline.** Concept art to a walking spritesheet, made with an image model I fine-tuned (LoRA on FLUX.2-klein) and scored with metrics I wrote.

Project Hub catalog: category chips, status filters and rows of colored project cards — **Project Hub.** A Netflix-style manager for 119 projects and their Python environments. It classifies new projects with an AI agent, detects moved and duplicated folders, and runs any project in one click.

Level 2 · What I built

The projects, in one sentence each

Grouped by what they do. The small tags show what they are made of.

Desktop tooling experiments

Small Windows utilities for my own use, built to try out low-latency screen capture, input handling and hardware I/O. Only the stack is listed here.

scshtRust + Python GUI · RP2040 microcontroller as a USB device · build pipeline

RustPythonRP2040

Loot splittertray app · clipboard parsing · Rust helper binary

PythonRustpystray

PixelTriggerRust egui · screen sampling · worker threads decoupled from the UI

RusteguiWin32

Olheirocapture card + dxcam frame grabbing · win32 window capture · notebook prototyping

PythondxcamOpenCV

Level 3 · How it's wired

What happens between input and answer

Three systems traced from start to finish. Each box is a separate process or model. The small text under each box shows what it does there.

Omni-call: a voice and video call with a local model

Speech recognition and speech synthesis run on the CPU, so the GPU's 6 GB stay free for the language model. Interrupting it mid-sentence cancels the reply.

Browsermic + camera, voice activity detection
Node serverWebSocket /rt, HTTPS
Parakeet ASRspeech → text, ~170 ms/turn on CPU
LLM routerllama.cpp / Strata, tool calls
Tool agentweb, files, shell, Python, reminders
TTSsentences streamed back as 24 kHz audio

musicraw: YouTube song → karaoke video

Lyrics sites print the chorus once, but singers repeat it. So the transcript is matched against the lyrics, and the missing repeats are put back before alignment.

yt-dlpaudio + title
Lyrics fetchGenius, lyrics.ovh
UVR MDX-Netisolate the vocal stem
Parakeet ASRword-level transcript
expand.pyre-insert repeated choruses
CTC alignword timings → .srt
ffmpegmux to .mkv

This website and everything else at echternacht.dev

This page is on Cloudflare Pages, so it stays up when my PC is off. Apps on other subdomains go through a tunnel to a 12 MB Go router on my PC. When nobody is using an app, the router stops it, and it starts again when someone visits.

Visitorname.echternacht.dev
CloudflareDNS, TLS, Pages
Tunnelcloudflared, outbound only
Go routerlogin, wake on visit, idle stop
Local appseach in a Windows Job Object

Level 4 · Engine room

Changes I made to open-source engines

Most local AI runs on a few C++ engines. I keep my own forks of them, merge upstream regularly, and change what doesn't fit a 6 GB laptop GPU. Line counts are my commits only, not upstream's.

Lines I added and removed per fork

llama.cpp

75 commits · ggml-org/llama.cpp

Hybrid MoE expert cache. The experts the router picks most often (LFU with decay) stay in a VRAM slot pool. Hits run on the GPU, misses on the CPU, and the results are summed. Decode went from 31 to 34.6 tok/s (+11%) on Qwen3.5-MoE.
Weight prefetch on a second CUDA stream, plus a Windows mmap prefetch that makes the model fully resident before loading returns.
Tool-call checkpoint in the server, so the next agent turn resumes right after the reasoning instead of re-reading the prompt.
Reverted upstream's MMQ refactor (#24127), which broke MUL_MAT_ID with CPU-offloaded experts on sm_89 GPUs.
Merged in PrismML's Q2_0, MBZUAI's K2 Horizon model and adaptive KV streaming from other forks.

Strata

30 commits · local AI engine

End result on the 6 GB laptop: Qwen3.6 MoE decode went from 28 to 48 tok/s, and the 177B Strata Coder runs at about 25 tok/s.
Expert load halved: the copy into the arena now fans out over threads, and only the padding gets zero-filled. Load went from 31.5 to 12.8 s and the output bytes are identical.
One staging buffer instead of two cut prompt staging from 0.97 to 0.49 GiB. The freed memory went back to decode slots.
A config that fits in VRAM: 12 GiB of pinned memory, 66K context, no spill to system memory. Decode went from 41 to 46.8 tok/s.
Benchmark harnesses that swept 24 configurations (chart below) and a settings app for the engine switches.

audio.cpp

16 commits · music & speech engine

A durable HTTP job server with cancellable jobs, SSE progress and a record written before the model runs, so crashes stay visible.
YuE2 VRAM fitting: the token budget is cut to fit free memory. Running out of memory now gives a shorter song instead of an error.
An audio-to-score job using the SheetSage2 native runtime.

little-coder

3 commits · coding agent for small models

Web tools rebuilt: GitHub links go to the raw README, PDFs become text, and HTML goes through Readability → Markdown. A provider cascade (Exa → local Strata → DuckDuckGo) can be switched with /web-setup.
write-guard backs up an existing file instead of refusing to write it. auto-resume sends "continue" after an error the agent couldn't recover from.
Compact swap: before sub-agents fan out, the active model switches to a 2B model so the big model's VRAM is free for them.
/lcraw dumps the full assembled prompt and tool schemas for audit. /manage toggles extensions, and a python tool runs a persistent uv scratch env.

HOT-Step-CPP

2 commits · music app

A 6 GB mode. The MERT attention window fell from 3.6 GB to about 0.36 GB.
A q4_0 KV cache for MiniMax-Music3: 81 kB per position instead of 288 kB in f16.

How the hybrid expert cache decides where an expert runs

Each token picks 8 of 256 experts. Experts already in the GPU slot pool (outlined) run there and light up blue. The rest run on the CPU from system RAM and light up orange. Both results are added together, and the cache keeps the most popular experts as the routing changes.

Level 5 · Measurements

Before and after, on the same laptop

RTX 4050 Laptop (6,141 MiB), i7-13620H, 24 GB DDR5. The numbers come from my benchmark logs and commit messages. Hover or tap a chart for exact values.

Every optimization, normalized to its "before"

Bar = how much of the original cost is left. Shorter is better.

Strata: 24 configs, prompt speed vs decode speed

Qwen3.6 MoE, pp2048 / tg128, warm run. Top right is best.

Where the 6 GB ceiling bites

Decode tok/s vs expert layers on the CPU (-ncmoe). Fewer layers on the CPU means more weights in VRAM. Below 32, Windows starts paging VRAM.

Decode stops scaling at 6 threads

tg128 tok/s by CPU threads. Past 6 threads, memory bandwidth is the limit, not compute.

Long context costs little

tg64 tok/s by context depth. Hybrid SSM layers keep the slowdown at 27% at 64K.

Chimp128 vs zstd on float time series

1,000,000 noisy-sine float64 values (8 MB). My Rust codec is 23% smaller, and the round trip is bit-exact.

A negative result I kept: early-exit heads

SmolLM2-135M on SST-2, validation accuracy per epoch. Extra heads trained 25% faster but scored 0.7 pp lower.

Can a frozen vision model judge sprite quality the way people do?

Agreement with 151 human-judged sprite pairs. Coin flip is 50%, and people agree with themselves 92% of the time. No model got close, so sprite review stays manual.

Level 6 · Workbench

How 119 projects stay findable

Project Hub is a catalog I built for my own disk. It indexes projects wherever they are with symlinks, never moves the real folders, and tracks status, stack and duplicates. These charts come from its catalog.