Shadowing the Web: Architecting Single-Binary Self-Contained Web Archivers

The Fragile Web and the Fallacy of "Save Page As"

The modern web is built on a house of cards. A typical web application is not a static document; it is a highly distributed, reactive runtime environment that pulls assets, API payloads, fonts, and scripts from dozens of distinct Content Delivery Networks (CDNs) and origin servers. When you click "Save Page As" in a desktop browser, you quickly realize how broken this model is for preservation. Absolute URLs, dynamic JavaScript imports, CORS policies, and client-side routing immediately break, leaving you with a disjointed skeleton of unstyled HTML.

To solve this, developers are turning to a pattern known as "Web Shadowing." The goal of a web shadow is simple yet architecturally challenging: compile an entire web page, including all its dynamic assets, fonts, media, and client-side execution logic, into a single, self-contained binary. This binary can be distributed, run offline without any network access, and executed with 100% visual and functional fidelity.

In this article, we will dive deep into the systems architecture required to build a single-binary web archiver. We will explore dependency tree resolution, asset inlining, and how to embed a lightweight runtime engine to serve the shadowed site locally without external dependencies.

The Anatomy of Web Shadowing

To package a living, breathing website into a single compiled binary (such as a Go or Rust executable), we must design a multi-stage compilation pipeline. This pipeline mimics a web browser's parser but operates as an offline compiler.

[Target URL] 
     │
     ▼
[1. Headless Browser / Crawler] ──► Captures dynamic DOM & network requests
     │
     ▼
[2. AST Parser & Link Resolver] ──► Rewrites URLs to virtual localized paths
     │
     ▼
[3. Asset Bundler]             ──► Converts images/fonts to Base64 or VFS blocks
     │
     ▼
[4. Compiler Wrapper]          ──► Embeds assets into Go/Rust binary using VFS
     │
     ▼
[Self-Contained Executable]

1. Headless Asset Extraction

Simple HTTP GET requests are no longer sufficient to fetch a webpage's assets. Modern SPAs (Single Page Applications) built with React, Vue, or Svelte render their DOM dynamically using JavaScript.

To capture the true state of a website, our archiver must spin up a headless browser instance (using Chrome DevTools Protocol via Go's chromedp or Rust's headless_chrome). The archiver lets the page execute its initial JS, trigger its API calls, and settle into a stable state. During this phase, we hook into the network layer to intercept every network response, saving the exact payloads of fonts, CSS, images, and API responses.

2. AST Parsing and Dependency Resolution

Once the raw HTML, CSS, and JS are captured, we must resolve the dependency tree. This requires parsing the assets into Abstract Syntax Trees (ASTs):

HTML Parsing: We find all <link>, <script>, <img>, and <iframe> tags.
CSS Parsing: We parse stylesheet rules to find @import statements, url() declarations, and font-face sources.
JS Parsing: We analyze scripts to detect dynamic imports (import()) and fetch calls.

Every external reference must be mapped to a local, virtualized resource identifier.

3. Virtual File System (VFS) vs. Base64 Inlining

To bundle these assets into a single binary, we have two primary architectural paths:

Inlining (Single HTML File): We convert every asset (images, fonts, stylesheets) into Base64 data URIs and inject them directly into a massive HTML file. While simple, this approach breaks down with large media files and triggers severe performance issues due to Base64's 33% size overhead and browser parsing bottlenecks.
Embedded Virtual File System (Single Binary): We preserve the assets as raw binary files and embed them directly into our compiler's binary payload using native embedding directives (like go:embed in Go or include_bytes! in Rust). At runtime, our binary spins up a micro-web server on an ephemeral loopback port or intercepts system-level Webview protocols to serve these assets dynamically.

Architecting the Shadow Bundler in Go

Let’s look at a practical architectural pattern for building a single-binary web archiver in Go. Go is highly suited for this task due to its robust networking stack, fast compilation, and native support for embedding virtual file systems.

Step 1: Defining the Virtual File System

We start by designing an archiver that fetches a page, rewrites its links to point to a virtual structure, and uses Go's embed package to package our local runtime assets.

package main

import (
	"embed"
	"fmt"
	"io/fs"
	"net/http"
	"os"
)

//go:embed templates/* ui/dist/*
var embeddedAssets embed.FS

func main() {
	// Strip the prefix to serve assets from the virtual directory structure
	uiAssets, err := fs.Sub(embeddedAssets, "ui/dist")
	if err != nil {
		panic(err)
	}

	fileServer := http.FileServer(http.FS(uiAssets))

	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		// Set security headers to sandbox the offline application
		w.Header().Set("Content-Security-Policy", "default-src 'self' 'unsafe-inline' 'unsafe-eval' data:")
		w.Header().Set("X-Frame-Options", "DENY")
		fileServer.ServeHTTP(w, r)
	})

	port := ":8080"
	fmt.Printf("Shadow website running offline at http://localhost%s\n", port)
	if err := http.ListenAndServe(port, nil); err != nil {
		os.Exit(1)
	}
}

In a production-grade web shadow compiler, the static assets of the targeted website are not known at compile time of our compiler tool. Instead, the compiler tool reads the target site, constructs a ZIP archive or a custom binary blob of all retrieved resources, and appends this blob to the end of a pre-compiled runner stub executable (a technique similar to self-extracting archives).

Overcoming the Hardest Engineering Challenges

1. Dynamic API Mocking and State Hydration

Many modern websites rely heavily on REST or GraphQL APIs to display content. If you archive the static JS and HTML, the page will load offline, but it will immediately fail when trying to fetch data from api.example.com.

To solve this, a sophisticated web shadow compiler intercepts network requests during the headless crawling phase. It serializes these API responses into a JSON database. When building the final binary, this JSON database is embedded alongside the frontend assets. At runtime, the binary injects a lightweight Service Worker into the browser context. This Service Worker intercepts all outgoing fetch and XMLHttpRequest calls, routing them to the embedded JSON database instead of the real internet.

2. Handling Dynamic Imports and Code Splitting

Modern bundlers (Vite, Webpack) split JavaScript into multiple chunks that are loaded on-demand. If your crawler only captures the homepage, clicking a button that loads a dynamic route will crash the application.

Your crawler must actively traverse the dependency graph. It can do this by parsing the source maps of your JavaScript files (if available) to identify all chunk files, or by forcing the headless browser to trigger all user interactions (clicks, navigations) to force the loading of split chunks before finalizing the archive.

Sandboxing and Security of Offline Binaries

Running untrusted, archived JavaScript locally on your machine introduces significant security risks. If an archived website contains malicious JS (or was compromised before archiving), running it locally could expose sensitive local system data if not properly sandboxed.

When architecting your runtime wrapper, implement these security safeguards:

Strict Content Security Policies (CSP): Prevent the archived site from executing external scripts or exfiltrating data if the machine ever connects to the internet.
Origin Isolation: If using a native desktop window wrapper (like Tauri, Wry, or Webview), ensure that the file system access APIs are completely disabled. The web page must believe it is running in a standard, restricted browser context.
Read-Only Virtual File Systems: The embedded assets should be mounted as a read-only VFS block to prevent any runtime modification of the archived source code.

Conclusion: The Era of Self-Contained Web Artifacts

As the internet becomes increasingly ephemeral, the ability to compile highly complex, interactive web experiences into single, offline-ready binaries is transitioning from a niche archiving requirement to a critical tool for developers, digital preservationists, and security researchers. By combining headless browser instrumentation, AST-driven asset rewriting, and embedded virtual file systems, we can rescue modern web applications from the inevitable decay of the live web.