GPU sizing

Setup reads each NVIDIA GPU’s UUID, driver, total and free VRAM and compute capability, and binds hosting to one GPU; memory is not summed across GPUs. It needs driver 580 or newer (CUDA 13) and a GeForce RTX 30 (8.6), 40 (8.9) or 50 (12.0) series card. One release covers all three.

Synoec picks the largest configuration that fits the card’s free memory: the context window, from 262,144 down to 32,768 tokens, in the all-round rk8v4 KV cache, else the same window in the smaller rk4v4, else a shorter window. The estimate counts 7.7 GB of resident text weights, 1.1 GB for the DFlash2 drafter, 1 GB of runtime headroom, 24 (rk8v4) or 16 (rk4v4) bytes per token of window and a 512 MiB desktop floor; the vision encoder borrows the drafter’s memory while it reads an image. These are Synoec’s estimates from the NInfer-all performance tables, not measured on every card.

The server also pins host RAM for its prefix cache, the checkpoints that let a long chat’s next step re-read only what is new. Synoec sizes it from what the machine has left after the GPU share, media buffers and a 10 GB reserve, up to 16 GiB: two full 262k windows. On a 24 GB machine that leaves none, and every step re-reads the chat.

Free VRAM	Starting configuration
8 GB	None: 32,768 tokens do not fit beside the drafter
12 GB	about 65,536 tokens, rk4v4
16 GB	about 196,608 tokens, rk8v4
24 GB or more	262,144 tokens, rk8v4

A load the card refuses tries the next smaller configuration, three times, and the smaller size is saved; a saved size on the same card is kept across restarts. When other apps hold the memory it needs (a game, a video editor, another AI app), loading waits a few seconds for memory another runtime just released, then fails with how much of the card is in use and what to close. The next message to Bonsai loads it again; no Retry is needed.

To move to a newer NInfer release: download its Windows CUDA 13.4 volumes and SHA256SUMS-*.txt, verify them, publish them as a ninfer-<tag> prerelease on the Synoec repository (gh release create ... --prerelease), then change apps/desktop/assets/local-ai/ninfer.json: the upstream tag and commit, the mirror tag, each volume’s name, size, SHA-256 and asset id (from gh api repos/Devkison/Synoec/releases/tags/<tag>), and for a new model its revision, size and SHA-256 from Hugging Face. The manifest validator refuses any volume URL that is not the mirror’s own API asset URL and any model URL that is not that revision’s own file. To rotate the token, replace runtime.mirror.key and ship a new build.