Two small decision models from the basal family, one 16 GB card, no quantization. This post measures basal-omni-pico (0.8B, built on Qwen3.5-0.8B-Base) and basal-2.0-mini (1.5B, built on Bielik-1.5B-v3.0-Instruct) running natively in BF16 on an RTX 4070 Ti SUPER.

basal is an open (Apache-2.0) family of typed-decision models for Polish and English, a project by Remigiusz Kinas. The model reads a state (a message, a document, retrieved passages) and a typed question with allowed answers, and returns a probability for each option in a single forward pass, without generating text. You define the classifier in the request, so there is no training step.

The question here is practical: how much do the fast (CUDA graphs) modes of the engine buy you over eager, what do they cost in VRAM and RAM, and how does the HTTP server behave under concurrent clients. If you want the GGUF / llama.cpp route instead, see the companion article (in Polish).

Accuracy figures come from the built-in basal-bench run (n=44). That is a tiny sample: one decision is worth 0.023, so 0.750 vs 0.727 is a single flipped answer. Treat it as a sanity check, not as a quality ranking.

TL;DR: basal-omni-pico in fast mode is the best option for speed and accuracy: 226.5 dec/s in the offline benchmark, 21.2 decisions/s over HTTP, with accuracy and agreement identical to eager. It needs about 9 GiB of VRAM in serving mode, so it fits a 16 GB card, but with limited headroom.

Technical details

Test environment

Basal inference server 2.8.0, everything native (no containers). Benchmarks: basal-bench batch run (n=44) and a basal-serve concurrency probe on /v1/systemone.

Component	Value
Python	3.14.7 (uv managed venv)
PyTorch	2.14.1+cu130
NVIDIA driver	580.178.04
CUDA	13.0
GPU	NVIDIA GeForce RTX 4070 Ti SUPER, 16 GB VRAM

Unlocked GPU

NVIDIA caps the tensor cores of GeForce cards to half rate for FP16, BF16 and TF32 math with FP32 accumulation, which is what unquantized inference uses. gpu-unlock removes that cap at boot, before the stock NVIDIA driver loads, with no patched driver or VBIOS flash. Support for Ada Lovelace (RTX 40), including the 4070 Ti SUPER, is marked experimental. According to its README, outputs are bit-identical and only the speed changes. Memory-bound decode, INT8, FP8 and FP32 CUDA work should not change, which matches what I measured.

The same script gave very different results on this card before and after the unlock (same script and torch build, 5 runs averaged):

Workload	Locked	Unlocked	Change
GEMM BF16 (fp32 accum)	83.1 TFLOPS	136.7 TFLOPS	+64%
GEMM FP16 (fp32 accum)	82.1 TFLOPS	127.5 TFLOPS	+55%
GEMM TF32 (fp32 accum)	41.4 TFLOPS	63.2 TFLOPS	+53%
GEMM FP32 (CUDA cores)	25.5 TFLOPS	25.5 TFLOPS	0%
GEMM INT8	56.9 TFLOPS	56.6 TFLOPS	0%
Matvec BF16 (decode shape)	558.5 GB/s	562.1 GB/s	+1%

All numbers in this article are from the unlocked card.

Results

1. Offline benchmark (basal-bench)

basal-2.0-mini

Mode	lat1 ms	lat2 ms	lat2 p95 ms	dec/s	VRAM GiB	acc	agree
eager	18.5	18.6	20.3	85.1	4.1	0.750	–
fast-nocompile	9.7	17.0	19.5	100.6	4.5	0.750	1.000
fast	7.9	15.2	17.0	109.1	4.3	0.727	0.977

basal-omni-pico

Mode	lat1 ms	lat2 ms	lat2 p95 ms	dec/s	VRAM GiB	acc	agree
eager	30.7	31.0	34.4	101.9	3.6	0.795	–
fast-nocompile	5.4	10.4	11.0	186.5	4.3	0.795	1.000
fast	4.0	7.8	8.8	226.5	4.3	0.795	1.000

Notes:

• omni-pico fast gives a 2.2x decode speedup over eager (226.5 vs 101.9 dec/s) with accuracy preserved and full agreement.
• omni-pico fast-nocompile and fast match eager accuracy (0.795) exactly.
• 2.0-mini gains less (1.28x) and fast drops agreement to 0.977 and accuracy to 0.727, versus 0.750 for eager. With n=44 that is one answer.
• omni-pico eager falls back to the PyTorch reference causal_conv1d kernel; install causal_conv1d to speed it up. fast-nocompile and fast use the fla convolution, which is a large part of the gap.

2. Serving VRAM / RAM (basal-serve)

VRAM and RAM were read from nvtop while the server was running, during the load test (section 3), so they reflect usage under load. Values are in GiB, rounded to one decimal.

basal-2.0-mini

Mode	VRAM GiB	RAM GiB	Notes
eager	3.8	1.4	ready in 1 s
fast-nocompile	10.9	5.1	attention fa2, ready in 43 s
fast	9.8	4.6	attention fa2, ready in 111 s

basal-omni-pico

Mode	VRAM GiB	RAM GiB	Notes
eager	3.1	1.8	ready in 3 s
fast-nocompile	11.7	5.9	attention fa2, convolution fla, ready in 49 s
fast	9.2	3.5	inductor warns “Not enough SMs to use max_autotune_gemm mode”, ready in 51 s

Notes:

• fast uses less VRAM than fast-nocompile on both models in serving mode.
• Startup in fast mode: 111 s for mini, 51 s for omni-pico.
• In the compiled modes, serving VRAM is about 2.1 to 2.7x the offline benchmark figure (4.3 GiB), so budget for the serving numbers.

Why the compiled modes use more memory: they compile the forward pass with torch.compile and then record CUDA graphs for a grid of static shapes. Every captured graph keeps its buffers in a shared pool that stays allocated, while eager mode allocates activations per forward and frees them. The compile and kernel caches are the likely reason for the higher host RAM (3.5 to 5.9 GiB vs 1.4 to 1.8 GiB for eager), though I didn’t profile it.

3. Concurrency probe (basal-serve, /v1/systemone)

Each model and mode got a fresh server, e.g. for 2.0-mini in fast mode:

uv run basal-serve --model Remek/basal-2.0-mini --mode fast --port 10030

Then basal-loadtest ran against it at 1, 4, 8, 16 and 32 clients with synthetic requests:

for c in 1 4 8 16 32; do
  uv run basal-loadtest --url http://127.0.0.1:10030/v1/systemone \
    --synthetic 3000 --n-seq 20 --n-conc 200 --concurrency $c \
    --out output/conc-fast-$c.json
done

Each run prints decisions_per_s, the aggregate throughput at that concurrency; the sequential p50_ms / p95_ms give the single-client baseline. --synthetic 3000 makes every state about 3k tokens, all of the same shape, so these figures are for ~3k-token inputs.

The other runs only change --model (Remek/basal-omni-pico), --mode (eager, fast-nocompile, fast) and the output file name.

Model	Server	p50 / p95, one client, ms	decisions/s at 1 / 4 / 8 / 16 / 32 clients	GPU memory	RAM
basal-2.0-mini	eager	130 / 153	8.35 / 8.40 / 8.41 / 8.51 / 8.52	3.8 GiB	1.4 GiB
basal-2.0-mini	fast-nocompile	112 / 135	9.41 / 9.85 / 9.86 / 9.91 / 9.78	10.9 GiB	5.1 GiB
basal-2.0-mini	fast	96 / 122	10.99 / 9.86 / 9.88 / 9.82 / 9.83	9.8 GiB	4.6 GiB
basal-omni-pico	eager	85 / 113	12.69 / 13.05 / 13.15 / 13.07 / 13.06	3.1 GiB	1.8 GiB
basal-omni-pico	fast-nocompile	59 / 83	16.18 / 14.40 / 12.55 / 12.67 / 12.98	11.7 GiB	5.9 GiB
basal-omni-pico	fast	44 / 62	21.24 / 20.03 / 18.44 / 18.83 / 18.98	9.2 GiB	3.5 GiB

Notes:

• omni-pico fast is the best mode at every concurrency level (1.67x over eager at one client; 1.32x for mini).
• All modes saturate before concurrency 4; more clients only add latency.
• omni-pico eager stays flat near 13 decisions/s. fast-nocompile drops about 20 to 22 percent from c1 to c8 and c32. fast drops about 11 percent from c1 to c32.
• 2.0-mini fast and fast-nocompile saturate after c1 and hold about 9.9 decisions/s.
• omni-pico is faster in serving than 2.0-mini in every mode and at every concurrency level.
• Zero errors in all runs.

4. Summary

Metric	basal-2.0-mini	basal-omni-pico
Best mode (batch)	fast (109.1 dec/s, acc 0.727)	fast (226.5 dec/s)
Best mode (serving)	fast (10.99 decisions/s, c1)	fast (21.24 decisions/s, c1)
Best accuracy	0.750 (eager, fast-nocompile)	0.795 (all modes)
Min serving VRAM	3.8 GiB (eager)	3.1 GiB (eager)
Serving VRAM in fast	9.8 GiB	9.2 GiB

Which one to pick

basal-omni-pico in fast mode: highest accuracy on this benchmark, highest throughput, full agreement with eager and the lowest serving latency (p50 44 ms, p95 62 ms). If startup time or host RAM matters more than speed, eager starts in 1 to 3 s and needs only about 3 to 4 GiB of VRAM, at roughly 60 percent of the throughput.

Keep the headroom in mind: in serving, the compiled modes take 9 to 12 GiB, which leaves only about 4 to 7 GiB of a 16 GB card. That is enough for a small generative model at best, not for running a larger LLM alongside. Only eager (3 to 4 GiB) leaves real room for that.

---

If you have questions, spot an error in the measurements, or want to compare results on your own hardware, write to me on X: @kskrzypinski.

Photo by Dawid Małecki