Reversing 16-Bit DOS Classics: Demystifying Real-Mode Memory Segmentation and VGA Interrupts

The Resurgence of Retro Reversing

There is a quiet revolution occurring in the systems engineering community. While the software world marches toward containerization, distributed cloud architectures, and 64-bit virtual memory spaces, a dedicated group of reverse engineers is looking backward. Projects aiming to decompile, reverse, and preserve classic 16-bit DOS games—such as the recent push to reverse-engineer MicroProse's classic military flight simulator F-15 Strike Eagle II—have highlighted a fascinating truth: understanding the constraints of legacy hardware makes you a better modern systems architect.

Programming or reversing for MS-DOS requires shedding the safety nets of modern operating systems. There are no virtual memory page tables, no user/kernel mode boundary separation, and no protected memory spaces. If your code writes to the wrong address, you do not get a Segmentation Fault or an Access Violation—you overwrite the Interrupt Vector Table, lock up the CPU, or cause the screen to dissolve into static.

In this technical deep dive, we will explore the core mechanics of 16-bit real-mode assembly, deconstruct the notorious segment-offset memory model, and walk through how these classic engines interacted directly with video hardware using BIOS interrupts and direct memory access.

Real-Mode Memory Segmentation: The 20-Bit Address Space

To understand 16-bit DOS software, you must first grapple with real-mode memory segmentation. When the Intel 8086 processor was designed, it featured 16-bit registers (like AX, BX, CX, DX, SP, BP, SI, and DI). However, Intel wanted the processor to address up to 1 Megabyte (MB) of memory.

To address 1 MB ($2^{20}$ bytes), you need a 20-bit address. To solve this mismatch using 16-bit registers, Intel introduced segmented memory addressing.

Memory in real mode is addressed using a pair of 16-bit values formatted as Segment:Offset. The physical address is calculated using a simple hardware shift-and-add operation:

$$\text{Physical Address} = (\text{Segment} \times 16) + \text{Offset}$$

Or, represented in hexadecimal, you shift the segment value left by one hex digit (4 bits) and add the offset:

$$\text{Physical Address} = (\text{Segment} \ll 4) + \text{Offset}$$

Segment Aliasing

Because of this overlapping design, multiple Segment:Offset combinations can point to the exact same physical address in RAM. This phenomenon is known as segment aliasing. For example, let's look at the physical address 0x12345:

1230:0045 $\rightarrow (0x1230 \times 16) + 0x0045 = 0x12300 + 0x0045 = 0x12345$
1200:0345 $\rightarrow (0x1200 \times 16) + 0x0345 = 0x12000 + 0x0345 = 0x12345$
1000:2345 $\rightarrow (0x1000 \times 16) + 0x02345 = 0x10000 + 0x02345 = 0x12345$

When reverse-engineering a 16-bit DOS executable in tools like Ghidra or IDA Pro, this overlapping behavior can make static analysis incredibly confusing. If a game engine loads a pointer into ES:DI using segment 0x1000 and another routine accesses the same data using segment 0x1200, naive static analysis tools may fail to recognize that both operations are targeting the exact same block of memory.

The Real-Mode Memory Map

Under the MS-DOS memory model, the 1 MB address space is strictly partitioned. This layout is critical for reverse engineers searching for specific game subsystems (such as graphics routines or keyboard input loops):

| Address Range (Hex) | Size | Description | |---------------------|------|-------------| | 00000 - 003FF | 1 KB | Interrupt Vector Table (IVT) | | 00400 - 004FF | 256 B| BIOS Data Area (BDA) | | 00500 - 9FFFF | 638 KB| Conventional Memory (User Program Space / "640KB Barrier") | | A0000 - B7FFF | 96 KB | VGA/EGA Graphics Video RAM (VRAM) | | B8000 - BFFFF | 32 KB | CGA/MDA Text Mode Video RAM | | C0000 - F7FFF | 224 KB| Expansion ROMs (e.g., VGA BIOS) | | F8000 - FFFFF | 32 KB | Motherboard BIOS System ROM |

If you are reversing a DOS game and you see an assembly instruction like mov ax, 0xA000; mov es, ax, you instantly know that the code is preparing to perform direct-to-screen rendering by targeting the VGA Video RAM segment.

Setting the Stage: VGA Mode 13h

Before the advent of dedicated 3D accelerators, standardizing graphics modes was a chaotic frontier. However, in 1987, IBM introduced the VGA standard, bringing with it Mode 13h.

Mode 13h became the absolute gold standard for PC game development in the late 80s and early 90s. It offered:

Resolution: 320 x 200 pixels.
Color Depth: 8-bit (256 simultaneous colors from a palette of 262,144).
Memory Footprint: Exactly $320 \times 200 = 60,000$ bytes.

This 60,000-byte footprint was a stroke of genius because it fits entirely within a single 64 KB memory segment (which can hold up to 65,536 bytes). This meant developers could represent the entire screen as a single contiguous block of memory, mapping pixel coordinates $(X, Y)$ to a linear offset using a simple formula:

$$\text{Offset} = (Y \times 320) + X$$

Entering Mode 13h via BIOS Interrupts

To switch the display from standard text mode to Mode 13h, developers did not write directly to hardware registers. Instead, they invoked the BIOS video interrupt INT 10h with the register AH set to 0x00 (Set Video Mode) and AL set to 0x13 (Mode 13h).

Here is how that looks in x86 Assembly:

mov ah, 0x00    ; Function 00h: Set Video Mode
mov al, 0x13    ; Mode 13h: 320x200, 256 colors
int 0x10        ; Call BIOS Video Services

To return to standard 80x25 text mode before exiting the program back to the DOS prompt, they would invoke the interrupt again with AL set to 0x03:

mov ah, 0x00    ; Function 00h: Set Video Mode
mov al, 0x03    ; Mode 03h: 80x25 Text Mode
int 0x10        ; Call BIOS Video Services

Writing a Custom Real-Mode VGA Renderer

To tie these concepts together, let's look at how we can implement a basic, low-level render loop in C (compiled with a 16-bit compiler like Open Watcom) that bypasses the slow BIOS interrupt system and writes pixels directly to the hardware VRAM.

#include <dos.h>
#include <conio.h>

#define VIDEO_INT           0x10
#define SET_MODE            0x00
#define MODE_13H            0x13
#define MODE_TEXT           0x03

/* Create a far pointer pointing directly to the VGA VRAM segment */
volatile unsigned char far* VGA = (volatile unsigned char far*)0xA0000000L;

/* Switch to 320x200 256-color VGA mode */
void init_vga() {
    union REGS regs;
    regs.h.ah = SET_MODE;
    regs.h.al = MODE_13H;
    int86(VIDEO_INT, &regs, &regs);
}

/* Restore the system to standard DOS text mode */
void close_vga() {
    union REGS regs;
    regs.h.ah = SET_MODE;
    regs.h.al = MODE_TEXT;
    int86(VIDEO_INT, &regs, &regs);
}

/* Draw a pixel directly to the VRAM segment */
void draw_pixel(int x, int y, unsigned char color) {
    if (x >= 0 && x < 320 && y >= 0 && y < 200) {
        /* Calculate linear offset: (Y * width) + X */
        unsigned int offset = (y * 320) + x;
        VGA[offset] = color;
    }
}

int main() {
    int x, y;
    
    init_vga();

    /* Draw a simple gradient pattern across the screen */
    for (y = 0; y < 200; y++) {
        for (x = 0; x < 320; x++) {
            /* Color cycling pattern */
            draw_pixel(x, y, (unsigned char)(x + y));
        }
    }

    /* Wait for a key press (blocking raw keyboard input) */
    getch();

    close_vga();
    return 0;
}

The Mechanics of the `far` Pointer

In the code above, the declaration unsigned char far* VGA is highly critical. The keyword far is a 16-bit compiler extension. It tells the compiler to generate a 32-bit pointer containing both a 16-bit segment and a 16-bit offset, rather than a standard 16-bit near pointer (which only tracks the offset relative to the program's default data segment DS).

The compiler translates VGA[offset] = color into an assembly sequence resembling this:

mov ax, 0xA000
mov es, ax             ; Load the segment register ES with VRAM start
mov di, [bp - 2]       ; Load DI with the calculated offset
mov al, [bp - 4]       ; Load AL with the color byte
mov [es:di], al        ; Write the byte directly to VRAM

Modern Tooling for Reverse-Engineering DOS Binaries

If you are analyzing a binary like F-15 Strike Eagle II or any other 16-bit executable, you cannot rely on standard modern debuggers like GDB or WinDbg. You need a toolchain tailored for real-mode execution.

Ghidra with 16-Bit Real Mode Extension: Ghidra natively supports the x86 16-bit Real Mode processor. When importing a .EXE (usually in the MZ format) or a .COM file, ensure you select x86:LE:16:Real Mode. Ghidra will attempt to resolve segment references, but you must manually define the memory segments if the program dynamically alters segment registers like DS and ES during execution.
DOSBox-X (Debugger Build): Standard DOSBox is designed for gaming, but DOSBox-X offers an incredibly robust, built-in interactive debugger. It allows you to set hardware breakpoints on memory reads/writes (essential for tracking changes to the Interrupt Vector Table), trace execution instruction-by-instruction, and dump specific memory segments directly to your host machine.
IDA Pro: IDA Pro has long been the gold standard for 16-bit reversing. Its auto-analysis handles segment registers remarkably well, tracking segment changes via virtual segment registers (sreg) to dynamically recalculate target offsets as you scroll through the disassembly.

Retrospective: Why Legacy Architecture Still Matters

Reversing 16-bit DOS games isn't merely an exercise in nostalgia; it is a masterclass in resource optimization. When you are limited to 640 KB of system memory and a 4.77 MHz processor, every instruction, memory write, and alignment boundary matters.

Modern paradigms like GPU frame buffering, memory-mapped I/O, and direct memory access (DMA) are direct descendants of the hardware-hacking techniques pioneered by 16-bit game developers. By peeling back the layers of abstraction and exploring the bare metal of real-mode assembly, you gain a profound appreciation for the foundation upon which modern computing is built.