Files
niklas cff6dae88f Add image gc, stop VMs on kernel panic, harden first boot
- Add `wsbx gc` to delete cached images that no sandbox uses, and
  drop the superseded image automatically when a tag moves to a new
  digest (unless another ref still points at it).
- Boot the guest with panic=1. Virtualization.framework restarts a
  rebooting guest, so init now leaves a marker in the per-launch
  control share and powers off when it is already there. A kernel
  panic (or a reboot inside the VM) now ends the run with exit 1
  instead of hanging or looping forever.
- Report a VM that stops without an exit status as an error.
- Sync the first-boot image unpack before marking the sandbox
  provisioned; a crash right after first boot previously left an
  empty root filesystem.
- Validate kernel, workspace and state-dir paths before creating any
  sandbox files, so rejected runs no longer leave a sandbox behind.
- Ignore .llm-wiki/, commit .gitignore, fix the remaining Clippy
  warning, and document gc and the panic behavior.
2026-10-02 15:33:30 +02:00

7.9 KiB

wsbx — tiny microVM sandboxes for macOS

Goal: docker sandbox-like UX, but every sandbox is a real Linux microVM.

Decisions

Concern Choice Why
Hypervisor Apple Virtualization.framework via vfkit (brew install vfkit) Firecracker needs KVM (Linux only). vfkit is a signed, maintained CLI over VZ — no entitlement/codesign dance for us.
Kernel Kata Containers release kernel (vmlinux.container, arm64). The latest release is found via the GitHub API, then the tarball is streamed and checked against GitHub's published asset sha256, and only the kernel is kept. kernel --update / --version / WSBX_KATA_VERSION Trusted upstream, built for VMs (virtio-pci/blk/net/fs/console built in, no initrd needed). Same kernel Apple's container uses. Override with --kernel.
Base OS Any OCI image (Docker Hub, GHCR, …) pulled by wsbx itself (anonymous token auth, sha256-verified blobs) No Docker needed.
Rootfs build Host flattens layers → one tar (whiteouts applied). Host makes empty ext4 disk with mke2fs (brew e2fsprogs) seeded with a static busybox + init. Guest extracts the tar as root on first boot. Correct uid/gid/devices/hardlinks and case-sensitivity — impossible to get right on an APFS host dir.
Guest init /.wsbx/init (busybox sh as PID 1): mounts, DHCP (VZ NAT), mounts workspace (virtiofs), runs image entrypoint+cmd as image user, powers off. No systemd, boots in ~1s.
Agents Presets over Docker's own docker/sandbox-templates:{claude-code,codex,gemini} images Same images Docker Sandboxes use; multi-arch.

CLI

wsbx kernel                       # download/cache kernel
wsbx pull IMAGE                   # pull + flatten image
wsbx run [opts] IMAGE [-- CMD..]  # bare VM from any image (default: shell)
wsbx claude|codex|gemini [opts]   # agent sandbox for current dir
wsbx ls | wsbx rm NAME...
opts: --name N  --cpus N  --memory MiB  --disk GiB  --workspace DIR  --env K=V  --fresh
  • Workspace dir is shared via virtio-fs at the same absolute path as on the host (like Docker Sandboxes).
  • Sandboxes are persistent (disk kept per name; default name derived from image+workspace). --fresh/rm resets.
  • Agent API key env vars (ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY, …) are forwarded if set.
  • Terminal size + TERM forwarded.

Layout

~/.wsbx/kernel/vmlinux
~/.wsbx/images/<sha256>/{rootfs.tar,config.json}
~/.wsbx/sandboxes/<name>/{disk.img,meta.json,share/run.sh}

Steps

  1. Plan (this file). ✅
  2. ✅ OCI pull: parse ref, token auth, index → linux/arm64 manifest, verified blobs, flatten w/ whiteouts.
  3. ✅ Kernel fetch via curl | tar --zstd (bsdtar streams; no 650 MB file kept).
  4. ✅ Disk creation (mke2fs -d with busybox+init), init script.
  5. ✅ run via vfkit; agents presets; ls/rm.
  6. ✅ Verified: alpine (net/DNS/TLS/workspace rw/exit codes/interactive pty), ubuntu (apt), claude-code template (runs as agent, claude --version). Cold first boot of alpine ≈3.5 s, warm boot ≈0.6 s.

Findings

  • vfkit requires an initrd → empty file works (kernel has everything built in).
  • vfkit virtio-serial,stdio needs a TTY → non-TTY falls back to logFilePath=<our stdout>.
  • VZ virtio-fs: any guest uid can write; files land owned by the host user.
  • Homebrew e2fsprogs lacks libarchive (mke2fs -d x.tar unsupported) → guest-side unpack.

Iteration 2: pi + host tunnel ✅

  • pi: @earendil-works/pi-coding-agent pinned to host version (the old @mariozechner package stopped at 0.73.1).
  • Config: allowlist of ~/.pi/agent items copied to share/stage each start (not the whole dir). The copy is needed because the host skill symlinks point outside ~/.pi and would dangle in the guest. auth.json is opt-in.
  • Tunnel over vsock (vfkit virtio-vsock,listen): no host listener on any network interface, per-sandbox, no port clashes. Guest side is guest/fwd.go (static Go binary, built by build.rs), and it tells run.sh it's ready through a FIFO.
  • Verified: 20 MB transfer over IPv4 and IPv6 loopback with matching hash. wsbx pi -p answered through the local omlx model on host 127.0.0.1:8000 (first run 91 s incl. installs, then 3.8 s).

Iteration 3: de-hardcoding ✅

Was hardcoded Now
KATA_VERSION = "4.2.0", unverified curl GitHub releases API (latest or pinned), asset sha256 verified while streaming, kernel/source.json records version
image tags cached forever re-resolved after 24 h, falls back to the cache when offline
pi package name read from the host's pi install (package.json name + version)
/opt/homebrew/opt/e2fsprogs/sbin brew --prefix <formula> when not on PATH
--cpus 4 --memory 4096 half the host cores / a quarter of host RAM (≤ 8 GiB)
init/dhcp frozen into each disk disk holds a 3-line stub; the real init is sent through the host share on every start

Bugs found along the way: the image tar was attached read-write, so two sandboxes from one image couldn't run together. It's now read-only and only attached until the guest reports it's unpacked. Poweroff raced ext4 lazy-init (WRITE_ZEROES I/O error); root is now remounted read-only first.

Kept on purpose: Kata's in-tarball kernel path, registry-1.docker.io, arm64, hvc0/vda/eth0 (we define the device order), vsock ports, 20 GiB sparse disk, the agent image names and API-key env lists.

Iteration 4: security review fixes ✅

Finding Fix Verified
Critical: guest symlinks in the persistent share → host file overwrite share recreated per start (VM stopped, lock held), deleted after exit; outputs read with no-symlink + size cap; provisioned moved to meta.json guest planted run.sh/init/dhcp/fwd -> victim, exit -> /etc/hosts, plus the killed-wsbx leftover case: victim unchanged
High: comma injection into vfkit args vf() rejects , " and control chars in every interpolated path; workspace overlapping ~/.wsbx refused repo,sharedDir=/Users/niklas rejected
High: automatic API-key forwarding opt-in --keys with hint; ~/.wsbx 0700; run.sh deleted after exit key absent in VM without --keys
Medium: rm ../.. name must be a sanitized sandbox name traversal refused
Medium: auth-realm SSRF https only; public registries use an agent whose resolver refuses non-public IPs unit test
Medium: digest refs not enforced top-level digest compared; all registry digests format-checked; manifest size capped pin OK, bogus pin rejected
Medium: pi copy follows symlinks contained to staged roots (incl. ~/.agents/skills, which pi reads natively), cycle-safe, no special files, 256 MB cap unit test + show-me still arrives
(own) reference/tag/registry validated before any request (alpine@sha256:../../x hit the registry as /manifests/x), busybox must be a regular file, Kata tag validated unit tests

Iteration 5: cleanup ✅

  • wsbx gc: keeps only images whose tag a sandbox uses (freed 1.74 GB here). A tag that moves now deletes the superseded image automatically, unless another ref still points at it.
  • Kernel panics: panic=1 makes the kernel reboot, and VZ turns that into a restart loop. init writes a booted marker into the per-launch share and powers off when it's already there. A real panic (panic_on_oom) now stops in about 2 s with exit 1 and a message. Limit: a panic before the root disk mounts still loops.
  • Found while testing panics: the first-boot unpack wasn't synced before provisioned was recorded, so a crash right after first boot left an empty system. Fixed with a sync before marking.
  • Found while reviewing: path validation ran after disk creation, so rejected runs left a sandbox behind. It now runs first.
  • .gitignore covers .llm-wiki/; Clippy is clean.

Out of scope (add when needed)

Port forwarding, network policy / credential-injecting egress proxy, snapshots, x86_64 hosts, zstd layers.