Micrologics Logo
Back to Blog
App DevelopmentPublished on July 7, 2026

Hacking E-Paper: Architecting a Low-Latency, Local LLM Interface on the reMarkable Tablet

Discover how to transform a standard reMarkable e-paper tablet into an interactive, AI-powered notebook. This deep-dive tutorial covers low-level Linux input processing, direct framebuffer rendering, and optimized WebSocket pipelines.

The Allure of Conversational E-Paper: Beyond Static Digital Notebooks

For years, E-Ink and electronic paper (EPD) displays have been prized for their paper-like readability, low power consumption, and lack of blue-light emission. However, these devices have traditionally remained passive containers for reading PDFs or taking static handwritten notes. Inspired by the concept of interactive mediumship—where the canvas itself reacts to your writing in real-time, resembling a futuristic digital diary—it is now possible to hack these devices to build an interactive, AI-assisted writing partner.

This guide will walk you through the technical architecture required to turn a reMarkable 2 tablet into an interactive, low-latency canvas connected to a local Large Language Model (LLM). This project bypasses the stock user interface entirely, tapping directly into the Linux operating system (Codex) running on the device to intercept handwriting, process it, and stream LLM responses directly onto the E-Ink screen via the hardware framebuffer.

System Architecture Overview

The reMarkable 2 operates on a custom, minimalist Linux distribution called Codex, powered by an NXP i.MX7 Dual processor (ARM Cortex-A7) paired with 1GB of RAM. Because of the limited on-device computational power, running a modern 7B or 8B parameter model locally on the tablet is unfeasible. Instead, we must design a client-server architecture that splits the workload:

  1. The Client (reMarkable 2): A low-level daemon written in Rust that reads raw stylus events, displays a canvas, streams vector coordinates over a network socket, and renders incoming text streams.
  2. The Server (Local Workstation): A local machine running an inference engine (such as Ollama or llama.cpp) alongside an Optical Character Recognition (OCR) pipeline and a WebSocket server.
+------------------+                    +-------------------------+
|   reMarkable 2   |                    |    Local Workstation    |
|  (Client Daemon) |                    |        (Server)         |
+--------+---------+                    +------------+------------+
         |                                           ^
         |  1. Raw Stylus Events (/dev/input/event0)  |
         |-------------------------------------------|
         |  2. Vector Data Stream (WebSockets)       |
         |------------------------------------------>| 3. OCR Engine
         |                                           |    (Extract text from strokes)
         |                                           | 
         |                                           | 4. Local LLM
         |  5. Text Token Stream (JSON/WebSockets)   |    (Generate response context)
         |<------------------------------------------|
         |                                           
         |  6. Direct Framebuffer Paint (/dev/fb0)   
         v

Capturing Stylus Input via Linux Event Devices

To capture handwriting without the overhead of the stock xochitl interface, we must read directly from the Linux input subsystem. The digitizer on the reMarkable 2 registers as an input event device, typically located at /dev/input/event0 (or event1, depending on the hardware revision).

In Linux, these events are exposed as standard input_event structs defined in <linux/input.h>. In Rust, we can read these events asynchronously to capture coordinates, pen pressure, and hover state. Below is a simplified implementation of the event parser:

use std::fs::File;
use std::io::{Read, Result};
use std::os::unix::fs::FileExt;

#[repr(C)]
#[derive(Debug, Default, Clone, Copy)]
struct InputEvent {
    sec: i64,
    usec: i64,
    type_: u16,
    code: u16,
    value: i32,
}

const EV_ABS: u16 = 0x03;
const ABS_X: u16 = 0x00;
const ABS_Y: u16 = 0x01;
const ABS_PRESSURE: u16 = 0x18;

fn stream_stylus_events() -> Result<()> {
    let mut file = File::open("/dev/input/event0")?;
    let mut buffer = [0u8; 24]; // Size of input_event struct on 64-bit systems

    println!("Listening for stylus input...");
    loop {
        file.read_exact(&mut buffer)?;
        let event: InputEvent = unsafe { std::mem::transmute(buffer) };

        if event.type_ == EV_ABS {
            match event.code {
                ABS_X => println!("X Coord: {}", event.value),
                ABS_Y => println!("Y Coord: {}", event.value),
                ABS_PRESSURE => println!("Pressure: {}", event.value),
                _ => {}
            }
        }
    }
}

As the user writes, we collect these 2D coordinates into stroke paths. Once the pen leaves the screen for a configurable threshold (e.g., 1.5 seconds), the stroke path is finalized and prepared for transmission.

Bypassing the UI: Direct Framebuffer Access and Waveform Optimization

The most challenging aspect of writing software for E-Ink displays is rendering. Traditional screens update at a constant 60Hz or 120Hz by rewriting the entire liquid crystal grid. E-Ink screens, however, rely on physical charged microparticles suspended in microcapsules. Moving these particles requires applying specific electrical voltage pulses—known as waveforms—over time.

On the reMarkable 2, we can interact directly with the framebuffer device /dev/fb0. However, simply writing raw pixel bytes to /dev/fb0 is not enough; we must notify the Electronic Paper Display Controller (EPDC) which regions of the screen have changed and which waveform mode to apply.

E-Ink Waveform Modes

Different waveforms balance speed against image quality:

  1. Direct Update (DU): High speed, low latency (~100-150ms transition time), but produces high ghosting and only supports monochrome black/white. Excellent for drawing lines in real-time.
  2. Grayscale Clear (GC16): Slow transition (~400-800ms), but yields high-quality, 16-level grayscale images with zero ghosting. Perfect for rendering final printed text blocks.

To achieve a natural feel, we use DU for capturing real-time handwriting, and switch to GC16 when rendering the streaming text responses generated by the local LLM.

To instruct the hardware to update, we use ioctl calls on /dev/fb0 to pass update rectangles to the driver. Here is how to configure a custom ioctl update block in Rust:

use std::os::unix::io::AsRawFd;

// Custom ioctl definitions for the reMarkable EPDC driver
const MXCFB_SEND_UPDATE: u32 = 0x4044462E; 

#[repr(C)]
struct MxcfbUpdateData {
    update_region: MxcfbRect,
    waveform_mode: u32,
    update_mode: u32,
    update_marker: u32,
    temp_alpha: i32,
    flags: u32,
    alt_buffer_data: *const std::ffi::c_void,
}

#[repr(C)]
struct MxcfbRect {
    top: u32,
    left: u32,
    width: u32,
    height: u32,
}

fn refresh_screen_region(file: &File, x: u32, y: u32, width: u32, height: u32, mode: u32) {
    let update = MxcfbUpdateData {
        update_region: MxcfbRect { top: y, left: x, width, height },
        waveform_mode: mode, // e.g., WAVEFORM_MODE_DU (2) or WAVEFORM_MODE_GC16 (1)
        update_mode: 0,      // 0 = Partial update, 1 = Full update
        update_marker: 0,
        temp_alpha: 0,
        flags: 0,
        alt_buffer_data: std::ptr::null(),
    };

    unsafe {
        libc::ioctl(file.as_raw_fd(), MXCFB_SEND_UPDATE as libc::ioctl_num_t, &update);
    }
}

Bridging the Tablet to the Local LLM Host

With the on-device input and output pipelines established, we need a robust transport layer to communicate with the local server. A WebSocket connection is ideal here, as it allows full-duplex, low-latency streaming.

The Stroke-to-Text Pipeline (Server-Side)

When the client sends a finished drawing path, the server must convert the strokes into a format that the LLM can interpret. There are two primary approaches:

  1. Stroke-to-Image OCR: Render the coordinate paths onto a temporary 2D bitmap on the server and pass the image to an OCR engine like Tesseract, or directly to a vision-language model (e.g., Llama-3.2-Vision).
  2. Direct Vector Processing: Feed the raw SVG path coordinates to a handwriting recognition model (like MyScript or a custom-trained RNN) which is highly resilient to stylistic variances.

For a fully local, open-source setup, rendering the strokes to a high-contrast PNG and running it through Tesseract OCR yields excellent, highly deterministic results with sub-100ms latency.

Orchestrating the Streaming Inference Loop

Once the written text is extracted, it is formatted into a prompt and pushed to the local LLM. To achieve an immersive, real-time feel, we cannot wait for the entire LLM response to generate before rendering. Instead, we stream the output token-by-token.

Here is a conceptual Python server using websockets and ollama to stream tokens back to the tablet:

import asyncio
import websockets
import json
import ollama

async def handle_connection(websocket, path):
    async for message in websocket:
        data = json.loads(message)
        
        if data["type"] == "STROKE_DATA":
            # Step 1: Render strokes to image and perform OCR
            extracted_text = perform_ocr_on_strokes(data["strokes"])
            print(f"OCR Result: {extracted_text}")
            
            # Step 2: Stream response from local LLM
            stream = ollama.chat(
                model="llama3",
                messages=[{'role': 'user', 'content': extracted_text}],
                stream=True
            )
            
            for chunk in stream:
                token = chunk['message']['content']
                # Send each token immediately to the tablet
                await websocket.send(json.dumps({
                    "type": "TOKEN",
                    "content": token
                }))

async def main():
    server = await websockets.serve(handle_connection, "0.0.0.0", 8765)
    await server.wait_closed()

if __name__ == "__main__":
    asyncio.run(main())

Optimizing the E-Ink Font Engine for Rapid Stream Updates

When the client receives tokens via WebSockets, it must print them to the screen. Because rendering text on a custom framebuffer requires rasterizing fonts manually, we integrate a lightweight library like rusttype to parse TrueType fonts (.ttf) and paint the sub-pixel anti-aliased glyphs directly into the mapped memory of /dev/fb0.

// Pseudocode for writing a character to the memory-mapped framebuffer
fn draw_char(mmap: &mut [u8], x: u32, y: u32, glyph: &RasterizedGlyph, pitch: u32) {
    for row in 0..glyph.height {
        for col in 0..glyph.width {
            let pixel_val = glyph.pixels[(row * glyph.width + col) as usize];
            if pixel_val > 0 {
                let fb_index = ((y + row) * pitch + (x + col)) as usize;
                // Invert grayscale value for E-Ink (0 is white, 255 is black)
                mmap[fb_index] = 255 - pixel_val;
            }
        }
    }
}

To minimize display flashing during streaming:

  1. Maintain a cursor position in memory.
  2. For every incoming token, calculate the bounding box of the new characters.
  3. Write the pixels to the framebuffer.
  4. Trigger an ioctl refresh using WAVEFORM_MODE_DU for the bounding box of only those new characters. This makes the text appear to flow onto the page fluidly and instantly.
  5. Once the LLM finishes streaming (signaled by an end-of-stream socket message), trigger a single, full-screen WAVEFORM_MODE_GC16 refresh. This wipes away any accumulated micro-ghosting and locks the text into crisp, high-contrast, paper-like legibility.

Deployment and Execution

To deploy this architecture on your reMarkable 2:

  1. Enable SSH Access: Go to Settings > Help > Copyrights and Licenses to find your unique root password and IP address.
  2. Set up Toolchain: Cross-compile your Rust client binary from your host system using the target architecture armv7-unknown-linux-gnueabihf.
  3. Stop the Default Interface: Connect via SSH and run systemctl stop xochitl. This releases the locks on the hardware event files and the framebuffer.
  4. Run the Client: Copy your compiled binary to the device via scp and execute it.
  5. Start Writing: Watch as your handwriting is digested by your local server, analyzed by an advanced LLM, and instantly written back onto the physical page before your eyes.
#E-Ink#Rust#Embedded Systems#LLM#Open Source