Memory-Safe Context Switching: Implementing setjmp and longjmp in Fil-C

Introduction to Non-Local Jumps and Memory Safety

In standard C systems programming, the setjmp and longjmp functions provide a mechanism for performing "non-local jumps." Unlike standard functions that adhere to a strict Last-In, First-Out (LIFO) call stack structure, these functions allow execution to jump directly from one deeply nested function back to a previously saved CPU context. This behavior is commonly used for error recovery, cooperative multithreading, and implementing exception handling frameworks in pure C.

However, this power comes with severe memory safety risks. Standard C compilers implement setjmp by saving the CPU's register state—including the stack pointer (SP) and program counter (PC)—directly into a jmp_buf structure. Because standard C does not enforce spatial or temporal memory safety, calling longjmp after the function that initialized the jmp_buf has returned results in catastrophic undefined behavior. The stack pointer is restored to a region of the stack that has already been deallocated or reused, leading to stack-use-after-return vulnerabilities, control-flow hijacking, and arbitrary code execution.

Enter Fil-C, a memory-safe compiler and runtime environment designed by Filip Pizlo to run unmodified C and C++ code with hardware-enforced spatial and temporal safety guarantees. Achieving memory safety in Fil-C requires completely re-architecting how the compiler and runtime handle context switching. This article explores the mechanics of how Fil-C safely implements setjmp and longjmp without sacrificing the performance characteristics expected of C code.

The Anatomy of the Attack Surface in Traditional `setjmp`/`longjmp`

To understand how Fil-C solves this problem, we must first look at the raw assembly-level mechanics of standard context switching. When you call setjmp(env), the runtime library executes assembly instructions similar to this (on x86-64):

movq %rbx, 0(%rdi)   ; Save callee-saved registers into jmp_buf
movq %rbp, 8(%rdi)
movq %r12, 16(%rdi)
movq %r13, 24(%rdi)
movq %r14, 32(%rdi)
movq %r15, 40(%rdi)
leaq 8(%rsp), %rdx   ; Calculate original stack pointer
movq %rdx, 48(%rdi)  ; Save RSP
movq (%rsp), %rdx    ; Get return address
movq %rdx, 56(%rdi)  ; Save RIP
xorl %eax, %eax      ; Return 0
ret

This simple mechanism assumes that the memory layout remains consistent when longjmp is invoked. However, consider the following classic C anti-pattern:

#include <setjmp.h>
#include <stdio.h>

jmp_buf env;

void setup() {
    if (setjmp(env) == 0) {
        printf("Context saved\\n");
    } else {
        printf("Jumped back!\\n");
    }
}

void corrupt() {
    volatile char buffer[128];
    for (int i = 0; i < 128; i++) {
        buffer[i] = 0xAA; // Overwrite stack memory
    }
}

int main() {
    setup();
    corrupt();
    longjmp(env, 1); // Undefined Behavior: stack frame for setup() is gone!
    return 0;
}

When setup() returns, its stack frame is logically deallocated. When corrupt() is called, it reuses the exact same stack region. Calling longjmp in main() restores the stack pointer to the now-destroyed setup() frame. The CPU attempts to resume execution using corrupted stack variables and invalid return addresses, creating an immediate vector for arbitrary code execution.

The Fil-C Memory Model: Fat Pointers and Managed Stacks

Fil-C guarantees memory safety by replacing traditional raw hardware pointers with fat pointers. In Fil-C, a pointer is not a simple 64-bit memory address; it is a 128-bit structure containing:

Address: The current memory location being pointed to.
Base: The start of the allocated memory block.
Bound: The end of the allocated memory block.

Every memory access is validated against these bounds by compiler-generated checks. Furthermore, Fil-C manages allocation lifetimes using a highly optimized, real-time garbage collector (GC) and a shadow stack architecture. Stack frames are not merely offsets on a single contiguous hardware stack; they are tracked objects with defined scopes and lifetimes.

Because pointers cannot be forged or cast arbitrarily back to raw addresses, the compiler can enforce strict validation rules on any memory read or write. This architecture introduces a fundamental challenge: how do you jump across arbitrary execution boundaries when stack frames are strictly isolated, managed objects?

Implementing Safe Context Switching in Fil-C

Fil-C solves the setjmp/longjmp problem by turning the jmp_buf structure into a managed object and validating the target execution context before any jump is executed.

1. The Managed `jmp_buf` Structure

Instead of exposing a raw byte array to the programmer, Fil-C's runtime defines jmp_buf as a compiler-recognized type that holds a safe reference to the current execution frame. This reference is treated as a capability.

typedef struct {
    void* stack_frame_id;      // A safe, unique identifier of the target stack frame
    uintptr_t instruction_ptr;  // Compiler-validated instruction pointer
    // ... safe register state ...
} __filc_jmp_buf;

2. Stack Frame Validation

When setjmp is called, Fil-C records the active stack frame's identifier within the jmp_buf. This identifier is tied to the lifetime of the stack frame. If the function containing setjmp returns, the runtime invalidates the stack frame's identifier.

When longjmp is invoked, the Fil-C runtime performs a crucial safety check:

Frame Existence Check: It queries the active thread's shadow stack to verify whether the target stack_frame_id is still alive and active.
Unwinding Check: It checks if the destination frame is an ancestor of the current frame. You can only jump up the call stack (to a caller), never down into a frame that has already exited.

If the check fails (e.g., because the function that called setjmp has already returned), the Fil-C runtime immediately aborts execution with a descriptive error message (e.g., Panic: longjmp target stack frame no longer active), preventing memory corruption entirely.

3. Preventing Buffer Overflows in `jmp_buf`

Because jmp_buf is a managed structure, its bounds are strictly tracked. If malicious code attempts to overwrite a jmp_buf variable using a buffer overflow vulnerability elsewhere in the program, the compiler's fat pointer checks will catch the out-of-bounds write before it can touch the saved context. Even if an attacker somehow manages to construct a fake jmp_buf inside a block of raw memory, the runtime will reject it because the stack_frame_id pointer inside the fake buffer will lack the necessary cryptographic or capability-based authorization tokens required by Fil-C's execution engine.

Under the Hood: The Unwinding Process

When a valid longjmp is executed, Fil-C does not simply reset the stack pointer register. Doing so would leak resource handles, break garbage collection invariants, and skip critical destructor executions in C++ code compiled alongside C.

Instead, Fil-C performs a structured stack unwind:

Mark-and-Sweep Coordination: The runtime notifies the garbage collector that a range of stack allocations is about to be discarded. This allows the GC to immediately reclaim memory or schedule finalizers.
Destructor Invocation: If the compiler detects cleanup attributes (such as __attribute__((cleanup)) in C or object destructors in C++), it executes them sequentially for every frame being unwound.
Register Restoration: Once the stack is safely wound back to the target frame's state, the runtime restores the saved register context and transfers execution to the instruction pointer stored in the jmp_buf.

This design ensures that longjmp behaves less like an unsafe, chaotic assembly jump and more like a structured exception-handling mechanism.

Performance Implications: The Cost of Safety

Enforcing absolute memory safety during context switches is not free. Traditional setjmp is incredibly fast because it only requires a few assembly mov instructions. Fil-C’s safe implementation introduces overhead due to:

Shadow Stack Traversal: Validating that the destination stack frame is still active requires walking the shadow stack or querying an active frame map.
Fat Pointer Overhead: Reading and writing the jmp_buf requires copying 128-bit fat pointers instead of raw 64-bit registers.
GC Tracking: Registering the context change with the garbage collector to prevent memory leaks.

However, for the vast majority of real-world applications, this overhead is negligible. Non-local jumps are typically used for error recovery paths (which are executed infrequently) rather than tight inner loops. The trade-off—eliminating an entire class of critical, exploitable system vulnerabilities—is a massive net positive for modern secure software engineering.

Conclusion

Fil-C demonstrates that even the most notoriously unsafe features of the C language can be tamed through modern compiler design and runtime verification. By transforming jmp_buf from a raw register dump into a managed execution capability, Fil-C prevents stack-use-after-return exploits and ensures control-flow integrity. As the software industry shifts toward memory-safe systems, compilers like Fil-C bridge the gap, allowing legacy C codebases to achieve modern safety standards without requiring complete rewrites in alternative languages.