7.0 KiB
7.0 KiB
wsbx — tiny microVM sandboxes for macOS
Goal: docker sandbox-like UX, but every sandbox is a real Linux microVM.
Decisions
| Concern | Choice | Why |
|---|---|---|
| Hypervisor | Apple Virtualization.framework via vfkit (brew install vfkit) |
Firecracker needs KVM (Linux only). vfkit is a signed, maintained CLI over VZ — no entitlement/codesign dance for us. |
| Kernel | Kata Containers release kernel (vmlinux.container, arm64). The latest release is found via the GitHub API, then the tarball is streamed and checked against GitHub's published asset sha256, and only the kernel is kept. kernel --update / --version / WSBX_KATA_VERSION |
Trusted upstream, built for VMs (virtio-pci/blk/net/fs/console built in, no initrd needed). Same kernel Apple's container uses. Override with --kernel. |
| Base OS | Any OCI image (Docker Hub, GHCR, …) pulled by wsbx itself (anonymous token auth, sha256-verified blobs) | No Docker needed. |
| Rootfs build | Host flattens layers → one tar (whiteouts applied). Host makes empty ext4 disk with mke2fs (brew e2fsprogs) seeded with a static busybox + init. Guest extracts the tar as root on first boot. |
Correct uid/gid/devices/hardlinks and case-sensitivity — impossible to get right on an APFS host dir. |
| Guest init | /.wsbx/init (busybox sh as PID 1): mounts, DHCP (VZ NAT), mounts workspace (virtiofs), runs image entrypoint+cmd as image user, powers off. |
No systemd, boots in ~1s. |
| Agents | Presets over Docker's own docker/sandbox-templates:{claude-code,codex,gemini} images |
Same images Docker Sandboxes use; multi-arch. |
CLI
wsbx kernel # download/cache kernel
wsbx pull IMAGE # pull + flatten image
wsbx run [opts] IMAGE [-- CMD..] # bare VM from any image (default: shell)
wsbx claude|codex|gemini [opts] # agent sandbox for current dir
wsbx ls | wsbx rm NAME...
opts: --name N --cpus N --memory MiB --disk GiB --workspace DIR --env K=V --fresh
- Workspace dir is shared via virtio-fs at the same absolute path as on the host (like Docker Sandboxes).
- Sandboxes are persistent (disk kept per name; default name derived from image+workspace).
--fresh/rmresets. - Agent API key env vars (
ANTHROPIC_API_KEY,OPENAI_API_KEY,GEMINI_API_KEY, …) are forwarded if set. - Terminal size +
TERMforwarded.
Layout
~/.wsbx/kernel/vmlinux
~/.wsbx/images/<sha256>/{rootfs.tar,config.json}
~/.wsbx/sandboxes/<name>/{disk.img,meta.json,share/run.sh}
Steps
- Plan (this file). ✅
- ✅ OCI pull: parse ref, token auth, index → linux/arm64 manifest, verified blobs, flatten w/ whiteouts.
- ✅ Kernel fetch via
curl | tar --zstd(bsdtar streams; no 650 MB file kept). - ✅ Disk creation (
mke2fs -dwith busybox+init), init script. - ✅
runvia vfkit; agents presets; ls/rm. - ✅ Verified: alpine (net/DNS/TLS/workspace rw/exit codes/interactive pty), ubuntu (apt), claude-code template (runs as
agent,claude --version). Cold first boot of alpine ≈3.5 s, warm boot ≈0.6 s.
Findings
- vfkit requires an initrd → empty file works (kernel has everything built in).
- vfkit
virtio-serial,stdioneeds a TTY → non-TTY falls back tologFilePath=<our stdout>. - VZ virtio-fs: any guest uid can write; files land owned by the host user.
- Homebrew e2fsprogs lacks libarchive (
mke2fs -d x.tarunsupported) → guest-side unpack.
Iteration 2: pi + host tunnel ✅
- pi:
@earendil-works/pi-coding-agentpinned to host version (the old@mariozechnerpackage stopped at 0.73.1). - Config: allowlist of
~/.pi/agentitems copied toshare/stageeach start (not the whole dir). The copy is needed because the host skill symlinks point outside~/.piand would dangle in the guest.auth.jsonis opt-in. - Tunnel over vsock (vfkit
virtio-vsock,listen): no host listener on any network interface, per-sandbox, no port clashes. Guest side isguest/fwd.go(static Go binary, built bybuild.rs), and it tellsrun.shit's ready through a FIFO. - Verified: 20 MB transfer over IPv4 and IPv6 loopback with matching hash.
wsbx pi -panswered through the local omlx model on host 127.0.0.1:8000 (first run 91 s incl. installs, then 3.8 s).
Iteration 3: de-hardcoding ✅
| Was hardcoded | Now |
|---|---|
KATA_VERSION = "4.2.0", unverified curl |
GitHub releases API (latest or pinned), asset sha256 verified while streaming, kernel/source.json records version |
| image tags cached forever | re-resolved after 24 h, falls back to the cache when offline |
| pi package name | read from the host's pi install (package.json name + version) |
/opt/homebrew/opt/e2fsprogs/sbin |
brew --prefix <formula> when not on PATH |
--cpus 4 --memory 4096 |
half the host cores / a quarter of host RAM (≤ 8 GiB) |
| init/dhcp frozen into each disk | disk holds a 3-line stub; the real init is sent through the host share on every start |
Bugs found along the way: the image tar was attached read-write, so two sandboxes from one image couldn't run together.
It's now read-only and only attached until the guest reports it's unpacked. Poweroff raced ext4 lazy-init (WRITE_ZEROES
I/O error); root is now remounted read-only first.
Kept on purpose: Kata's in-tarball kernel path, registry-1.docker.io, arm64, hvc0/vda/eth0 (we
define the device order), vsock ports, 20 GiB sparse disk, the agent image names and API-key env lists.
Iteration 4: security review fixes ✅
| Finding | Fix | Verified |
|---|---|---|
| Critical: guest symlinks in the persistent share → host file overwrite | share recreated per start (VM stopped, lock held), deleted after exit; outputs read with no-symlink + size cap; provisioned moved to meta.json |
guest planted run.sh/init/dhcp/fwd -> victim, exit -> /etc/hosts, plus the killed-wsbx leftover case: victim unchanged |
| High: comma injection into vfkit args | vf() rejects , " and control chars in every interpolated path; workspace overlapping ~/.wsbx refused |
repo,sharedDir=/Users/niklas rejected |
| High: automatic API-key forwarding | opt-in --keys with hint; ~/.wsbx 0700; run.sh deleted after exit |
key absent in VM without --keys |
Medium: rm ../.. |
name must be a sanitized sandbox name | traversal refused |
| Medium: auth-realm SSRF | https only; public registries use an agent whose resolver refuses non-public IPs | unit test |
| Medium: digest refs not enforced | top-level digest compared; all registry digests format-checked; manifest size capped | pin OK, bogus pin rejected |
| Medium: pi copy follows symlinks | contained to staged roots (incl. ~/.agents/skills, which pi reads natively), cycle-safe, no special files, 256 MB cap |
unit test + show-me still arrives |
| (own) | reference/tag/registry validated before any request (alpine@sha256:../../x hit the registry as /manifests/x), busybox must be a regular file, Kata tag validated |
unit tests |
Out of scope (add when needed)
Port forwarding, network policy / credential-injecting egress proxy, snapshots, x86_64 hosts, zstd layers.