Sub-Byte Quantization and Dynamic Weight Streaming: Running 26B LLMs in 2 GB RAM

The On-Device Memory Bottleneck

Running high-parameter Large Language Models (LLMs) locally on consumer hardware has historically been blocked by a hard physical barrier: Unified Memory and VRAM capacity. A standard 26-billion parameter model compiled in half-precision floating-point format (FP16) requires approximately 52 GB of dedicated memory just to hold its weights. Even at standard 4-bit quantization (INT4), the baseline memory requirement floats around 13 GB to 15 GB once activation buffers and Key-Value (KV) caches are allocated.

For developers targetting modern consumer devices like Apple M-series Macs or high-efficiency laptops, holding a 26B model in under 2 GB of RAM seems mathematically impossible at first glance. However, by combining extreme sub-byte non-uniform quantization, layer-wise memory-mapped dynamic streaming, and low-level matrix multiplication kernels optimized for modern GPU register files, software engineers are rewriting the rules of local inference engines.

In this article, we will breakdown the exact low-level engineering techniques required to execute a 26B parameter model within a tight 2 GB RAM footprint without collapsing model perplexity.

Rethinking Quantization: Moving Below 4 Bits

Traditional quantization strategies like GGUF (k-quants) or AWQ focus on mapping weights down to 4-bit integer values (Q4_K_M). To compress a model down to fit within a 2 GB window, the average bit-width per weight must drop to approximately 0.5 to 1.5 bits.

Non-Uniform Sub-Byte Weight Formats

Standard linear quantization uses a fixed scale and zero-point across a block of weights:

$$W_{fp16} \approx S \cdot (W_{quant} - Z)$$

At sub-2-bit precision, uniform quantization introduces massive quantization noise, destroying model reasoning capabilities. Engine architects instead utilize Non-Uniform Vector Quantization and Ternary Representations (e.g., values restricted to ${-1, 0, +1}$).

Codebook Quantization: Instead of mapping weights to integers, weights are clustered into codebooks. A 2-bit index selects one of 4 predefined floating-point centroids learned during post-training quantization (PTQ).
Outlier Channel Preservation: Neural network activations exhibit severe outlier channels that hold disproportionate importance for accuracy. By storing 99% of non-critical weights in 1-bit or 1.5-bit formats while retaining critical outlier weights in 8-bit precision, overall entropy degradation is minimized.

Memory-Mapped Paging and Layer-Wise Streaming

Even with 1.5-bit quantization, a 26B model weighs roughly 4.8 GB. To lower active execution footprint down to 2 GB RAM, engines cannot load the entire model into VRAM at once. Instead, they leverage memory-mapped files (mmap) combined with targeted dynamic layer paging.

How Layer Paging Works

During autoregressive generation, an LLM processes inputs sequentially layer by layer through transformer blocks:

Prefetching: As Layer $N$ executes on the GPU matrix units, Layer $N+1$ is asynchronously preloaded from storage (NVMe / fast SSD) directly into a pinned RAM buffer using OS page tables.
Execution: The GPU computes the self-attention and MLP feed-forward pass for Layer $N$.
Eviction: As soon as Layer $N$ finishes, its weight buffer memory pages are marked as purgeable (madvise(MADV_DONTNEED)), releasing the memory back to the operating system before Layer $N+2$ is fetched.

Because M-series chips share unified memory between the CPU and GPU, zero-copy operations allow direct reads from memory-mapped files straight into GPU-accessible buffers without copying data across peripheral buses.

Optimizing the KV Cache and Attention Mechanisms

Loading weight matrices dynamically solves static storage constraints, but context growth poses another major RAM threat: the Key-Value (KV) cache. At long context lengths, storing attention keys and values for 26 billion parameters can easily consume multiple gigabytes.

To keep the total runtime footprint strictly bounded under 2 GB, state-of-the-art engines apply three primary KV optimizations:

1. Grouped-Query Attention (GQA) & Multi-Query Attention (MQA)

Models configured with GQA share key and value heads across multiple query heads, reducing KV cache memory by a factor equal to the query-to-key ratio (typically 4x to 8x).

2. PagedAttention and Dynamic Quantization

Rather than allocating continuous virtual memory chunks for context vectors, keys and values are stored in non-contiguous physical memory pages (similar to virtual memory in operating systems). Furthermore, KV entries are quantized on the fly from FP16 to INT4 or FP8 before writing to the cache buffer:

$$\text{KV}{compressed} = \text{Quantize}{int4}(K, V)$$

This reduces context memory consumption by 75% with zero impact on layer weight retrieval rates.

Custom Metal Compute Kernels for On-the-Fly Dequantization

When weight bits are packed tightly (e.g., four 2-bit weights packed inside a single uint8_t byte), traditional SIMD execution pipelines suffer from byte-unpacking overhead. Standard GPU linear algebra libraries (like BLAS or standard MPS) expect unpacked float vectors.

To achieve realtime token generation speed (15-20 tokens/sec), modern inference engines write hand-tuned Metal Shading Language (MSL) compute kernels that perform fused weight-dequantization and vector dot-products directly within GPU registers.

Here is a conceptual look at how a custom 2-bit fused matrix-vector multiplication kernel unfolds in Metal:

#include <metal_stdlib>
using namespace metal;

kernel void gemv_2bit_fused(
    device const uint8_t*  packed_weights [[buffer(0)]],
    device const half*     scales         [[buffer(1)]],
    device const half*     inputs         [[buffer(2)]],
    device half*           outputs        [[buffer(3)]],
    uint threadgroup_position_in_grid   [[threadgroup_position_in_grid]],
    uint thread_position_in_threadgroup [[thread_position_in_threadgroup]]
) {
    uint row = threadgroup_position_in_grid.x;
    uint tid = thread_position_in_threadgroup;
    
    half accumulator = 0.0;
    
    // Each byte holds four 2-bit weights
    uint packed_idx = (row * K_DIM / 4) + tid;
    uint8_t byte_val = packed_weights[packed_idx];
    half scale = scales[row];
    
    // Unpack 2-bit values directly inside registers via bit-shift
    for (int i = 0; i < 4; i++) {
        uint8_t raw_2bit = (byte_val >> (i * 2)) & 0x03;
        half weight = (half(raw_2bit) - 1.5h) * scale; // Map {0,1,2,3} -> {-1.5, -0.5, 0.5, 1.5}
        accumulator += weight * inputs[tid * 4 + i];
    }
    
    // Write back threadgroup reduction
    // ... (SIMD shuffle / threadgroup barrier reduction omitted for brevity)
}

By keeping unpacked values isolated inside SIMD registers and avoiding round-trips to global device memory, GPU execution pipelines maintain maximum memory bandwidth saturation while processing highly compressed model layers.

Practical Performance Breakdown

When running a sub-byte engine on an M-series Mac equipped with standard memory bandwidth (e.g., 100 GB/s on base M-chips, 200+ GB/s on Pro/Max chips), performance yields remarkable characteristics:

While streaming layer pages from storage adds a marginal read latency penalty compared to holding all parameters in fast RAM, NVMe throughput speeds exceeding 3,000 MB/s make continuous layer streaming smooth enough for seamless interactive chat and local agent execution.

Conclusion: The Horizon of Sovereign Local AI

The ability to execute a modern 26-billion parameter LLM within 2 GB of available RAM proves that model capability is no longer strictly bound to VRAM size. Through sophisticated non-uniform quantization, asynchronous page-mapped streaming, dynamic context management, and register-fused Metal compute kernels, high-capability frontier models are becoming accessible on everyday, low-spec consumer devices.

As sub-byte optimization frameworks mature, local edge execution will transition from a compromised fallback to the primary interface for secure, private, sovereign AI systems.