# wsbx — tiny microVM sandboxes for macOS Goal: `docker sandbox`-like UX, but every sandbox is a real Linux microVM. ## Decisions | Concern | Choice | Why | |---|---|---| | Hypervisor | Apple Virtualization.framework via **vfkit** (`brew install vfkit`) | Firecracker needs KVM (Linux only). vfkit is a signed, maintained CLI over VZ — no entitlement/codesign dance for us. | | Kernel | **Kata Containers** release kernel (`vmlinux.container`, arm64). The latest release is found via the GitHub API, then the tarball is streamed and checked against GitHub's published asset sha256, and only the kernel is kept. `kernel --update` / `--version` / `WSBX_KATA_VERSION` | Trusted upstream, built for VMs (virtio-pci/blk/net/fs/console built in, no initrd needed). Same kernel Apple's `container` uses. Override with `--kernel`. | | Base OS | Any **OCI image** (Docker Hub, GHCR, …) pulled by wsbx itself (anonymous token auth, sha256-verified blobs) | No Docker needed. | | Rootfs build | Host flattens layers → one tar (whiteouts applied). Host makes empty ext4 disk with `mke2fs` (brew e2fsprogs) seeded with a static busybox + init. **Guest extracts the tar as root on first boot.** | Correct uid/gid/devices/hardlinks and case-sensitivity — impossible to get right on an APFS host dir. | | Guest init | `/.wsbx/init` (busybox sh as PID 1): mounts, DHCP (VZ NAT), mounts workspace (virtiofs), runs image entrypoint+cmd as image user, powers off. | No systemd, boots in ~1s. | | Agents | Presets over Docker's own `docker/sandbox-templates:{claude-code,codex,gemini}` images | Same images Docker Sandboxes use; multi-arch. | ## CLI ``` wsbx kernel # download/cache kernel wsbx pull IMAGE # pull + flatten image wsbx run [opts] IMAGE [-- CMD..] # bare VM from any image (default: shell) wsbx claude|codex|gemini [opts] # agent sandbox for current dir wsbx ls | wsbx rm NAME... opts: --name N --cpus N --memory MiB --disk GiB --workspace DIR --env K=V --fresh ``` - Workspace dir is shared via virtio-fs at the **same absolute path** as on the host (like Docker Sandboxes). - Sandboxes are **persistent** (disk kept per name; default name derived from image+workspace). `--fresh`/`rm` resets. - Agent API key env vars (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GEMINI_API_KEY`, …) are forwarded if set. - Terminal size + `TERM` forwarded. ## Layout ``` ~/.wsbx/kernel/vmlinux ~/.wsbx/images//{rootfs.tar,config.json} ~/.wsbx/sandboxes//{disk.img,meta.json,share/run.sh} ``` ## Steps 1. Plan (this file). ✅ 2. ✅ OCI pull: parse ref, token auth, index → linux/arm64 manifest, verified blobs, flatten w/ whiteouts. 3. ✅ Kernel fetch via `curl | tar --zstd` (bsdtar streams; no 650 MB file kept). 4. ✅ Disk creation (`mke2fs -d` with busybox+init), init script. 5. ✅ `run` via vfkit; agents presets; ls/rm. 6. ✅ Verified: alpine (net/DNS/TLS/workspace rw/exit codes/interactive pty), ubuntu (apt), claude-code template (runs as `agent`, `claude --version`). Cold first boot of alpine ≈3.5 s, warm boot ≈0.6 s. ## Findings - vfkit requires an initrd → empty file works (kernel has everything built in). - vfkit `virtio-serial,stdio` needs a TTY → non-TTY falls back to `logFilePath=`. - VZ virtio-fs: any guest uid can write; files land owned by the host user. - Homebrew e2fsprogs lacks libarchive (`mke2fs -d x.tar` unsupported) → guest-side unpack. ## Iteration 2: pi + host tunnel ✅ - pi: `@earendil-works/pi-coding-agent` pinned to host version (the old `@mariozechner` package stopped at 0.73.1). - Config: allowlist of `~/.pi/agent` items copied to `share/stage` each start (not the whole dir). The copy is needed because the host skill symlinks point outside `~/.pi` and would dangle in the guest. `auth.json` is opt-in. - Tunnel over vsock (vfkit `virtio-vsock,listen`): no host listener on any network interface, per-sandbox, no port clashes. Guest side is `guest/fwd.go` (static Go binary, built by `build.rs`), and it tells `run.sh` it's ready through a FIFO. - Verified: 20 MB transfer over IPv4 and IPv6 loopback with matching hash. `wsbx pi -p` answered through the local omlx model on host 127.0.0.1:8000 (first run 91 s incl. installs, then 3.8 s). ## Iteration 3: de-hardcoding ✅ | Was hardcoded | Now | |---|---| | `KATA_VERSION = "4.2.0"`, unverified curl | GitHub releases API (latest or pinned), asset sha256 verified while streaming, `kernel/source.json` records version | | image tags cached forever | re-resolved after 24 h, falls back to the cache when offline | | pi package name | read from the host's `pi` install (`package.json` name + version) | | `/opt/homebrew/opt/e2fsprogs/sbin` | `brew --prefix ` when not on PATH | | `--cpus 4 --memory 4096` | half the host cores / a quarter of host RAM (≤ 8 GiB) | | init/dhcp frozen into each disk | disk holds a 3-line stub; the real init is sent through the host share on every start | Bugs found along the way: the image tar was attached read-write, so two sandboxes from one image couldn't run together. It's now read-only and only attached until the guest reports it's unpacked. Poweroff raced ext4 lazy-init (`WRITE_ZEROES` I/O error); root is now remounted read-only first. Kept on purpose: Kata's in-tarball kernel path, `registry-1.docker.io`, arm64, `hvc0`/`vda`/`eth0` (we define the device order), vsock ports, 20 GiB sparse disk, the agent image names and API-key env lists. ## Iteration 4: security review fixes ✅ | Finding | Fix | Verified | |---|---|---| | Critical: guest symlinks in the persistent share → host file overwrite | share recreated per start (VM stopped, lock held), deleted after exit; outputs read with no-symlink + size cap; `provisioned` moved to meta.json | guest planted `run.sh/init/dhcp/fwd -> victim`, `exit -> /etc/hosts`, plus the killed-wsbx leftover case: victim unchanged | | High: comma injection into vfkit args | `vf()` rejects `,` `"` and control chars in every interpolated path; workspace overlapping `~/.wsbx` refused | `repo,sharedDir=/Users/niklas` rejected | | High: automatic API-key forwarding | opt-in `--keys` with hint; `~/.wsbx` 0700; run.sh deleted after exit | key absent in VM without `--keys` | | Medium: `rm ../..` | name must be a sanitized sandbox name | traversal refused | | Medium: auth-realm SSRF | https only; public registries use an agent whose resolver refuses non-public IPs | unit test | | Medium: digest refs not enforced | top-level digest compared; all registry digests format-checked; manifest size capped | pin OK, bogus pin rejected | | Medium: pi copy follows symlinks | contained to staged roots (incl. `~/.agents/skills`, which pi reads natively), cycle-safe, no special files, 256 MB cap | unit test + `show-me` still arrives | | (own) | reference/tag/registry validated before any request (`alpine@sha256:../../x` hit the registry as `/manifests/x`), busybox must be a regular file, Kata tag validated | unit tests | ## Iteration 5: cleanup ✅ - `wsbx gc`: keeps only images whose tag a sandbox uses (freed 1.74 GB here). A tag that moves now deletes the superseded image automatically, unless another ref still points at it. - Kernel panics: `panic=1` makes the kernel reboot, and VZ turns that into a restart loop. init writes a `booted` marker into the per-launch share and powers off when it's already there. A real panic (`panic_on_oom`) now stops in about 2 s with exit 1 and a message. Limit: a panic before the root disk mounts still loops. - Found while testing panics: the first-boot unpack wasn't synced before `provisioned` was recorded, so a crash right after first boot left an empty system. Fixed with a sync before marking. - Found while reviewing: path validation ran after disk creation, so rejected runs left a sandbox behind. It now runs first. - `.gitignore` covers `.llm-wiki/`; Clippy is clean. ## Out of scope (add when needed) Port forwarding, network policy / credential-injecting egress proxy, snapshots, x86_64 hosts, zstd layers.