Files
woiken-sandboxes/PLAN.md
T
niklas cff6dae88f Add image gc, stop VMs on kernel panic, harden first boot
- Add `wsbx gc` to delete cached images that no sandbox uses, and
  drop the superseded image automatically when a tag moves to a new
  digest (unless another ref still points at it).
- Boot the guest with panic=1. Virtualization.framework restarts a
  rebooting guest, so init now leaves a marker in the per-launch
  control share and powers off when it is already there. A kernel
  panic (or a reboot inside the VM) now ends the run with exit 1
  instead of hanging or looping forever.
- Report a VM that stops without an exit status as an error.
- Sync the first-boot image unpack before marking the sandbox
  provisioned; a crash right after first boot previously left an
  empty root filesystem.
- Validate kernel, workspace and state-dir paths before creating any
  sandbox files, so rejected runs no longer leave a sandbox behind.
- Ignore .llm-wiki/, commit .gitignore, fix the remaining Clippy
  warning, and document gc and the panic behavior.
2026-10-02 15:33:30 +02:00

107 lines
7.9 KiB
Markdown

# wsbx — tiny microVM sandboxes for macOS
Goal: `docker sandbox`-like UX, but every sandbox is a real Linux microVM.
## Decisions
| Concern | Choice | Why |
|---|---|---|
| Hypervisor | Apple Virtualization.framework via **vfkit** (`brew install vfkit`) | Firecracker needs KVM (Linux only). vfkit is a signed, maintained CLI over VZ — no entitlement/codesign dance for us. |
| Kernel | **Kata Containers** release kernel (`vmlinux.container`, arm64). The latest release is found via the GitHub API, then the tarball is streamed and checked against GitHub's published asset sha256, and only the kernel is kept. `kernel --update` / `--version` / `WSBX_KATA_VERSION` | Trusted upstream, built for VMs (virtio-pci/blk/net/fs/console built in, no initrd needed). Same kernel Apple's `container` uses. Override with `--kernel`. |
| Base OS | Any **OCI image** (Docker Hub, GHCR, …) pulled by wsbx itself (anonymous token auth, sha256-verified blobs) | No Docker needed. |
| Rootfs build | Host flattens layers → one tar (whiteouts applied). Host makes empty ext4 disk with `mke2fs` (brew e2fsprogs) seeded with a static busybox + init. **Guest extracts the tar as root on first boot.** | Correct uid/gid/devices/hardlinks and case-sensitivity — impossible to get right on an APFS host dir. |
| Guest init | `/.wsbx/init` (busybox sh as PID 1): mounts, DHCP (VZ NAT), mounts workspace (virtiofs), runs image entrypoint+cmd as image user, powers off. | No systemd, boots in ~1s. |
| Agents | Presets over Docker's own `docker/sandbox-templates:{claude-code,codex,gemini}` images | Same images Docker Sandboxes use; multi-arch. |
## CLI
```
wsbx kernel # download/cache kernel
wsbx pull IMAGE # pull + flatten image
wsbx run [opts] IMAGE [-- CMD..] # bare VM from any image (default: shell)
wsbx claude|codex|gemini [opts] # agent sandbox for current dir
wsbx ls | wsbx rm NAME...
opts: --name N --cpus N --memory MiB --disk GiB --workspace DIR --env K=V --fresh
```
- Workspace dir is shared via virtio-fs at the **same absolute path** as on the host (like Docker Sandboxes).
- Sandboxes are **persistent** (disk kept per name; default name derived from image+workspace). `--fresh`/`rm` resets.
- Agent API key env vars (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GEMINI_API_KEY`, …) are forwarded if set.
- Terminal size + `TERM` forwarded.
## Layout
```
~/.wsbx/kernel/vmlinux
~/.wsbx/images/<sha256>/{rootfs.tar,config.json}
~/.wsbx/sandboxes/<name>/{disk.img,meta.json,share/run.sh}
```
## Steps
1. Plan (this file). ✅
2. ✅ OCI pull: parse ref, token auth, index → linux/arm64 manifest, verified blobs, flatten w/ whiteouts.
3. ✅ Kernel fetch via `curl | tar --zstd` (bsdtar streams; no 650 MB file kept).
4. ✅ Disk creation (`mke2fs -d` with busybox+init), init script.
5. ✅ `run` via vfkit; agents presets; ls/rm.
6. ✅ Verified: alpine (net/DNS/TLS/workspace rw/exit codes/interactive pty), ubuntu (apt), claude-code template (runs as `agent`, `claude --version`).
Cold first boot of alpine ≈3.5 s, warm boot ≈0.6 s.
## Findings
- vfkit requires an initrd → empty file works (kernel has everything built in).
- vfkit `virtio-serial,stdio` needs a TTY → non-TTY falls back to `logFilePath=<our stdout>`.
- VZ virtio-fs: any guest uid can write; files land owned by the host user.
- Homebrew e2fsprogs lacks libarchive (`mke2fs -d x.tar` unsupported) → guest-side unpack.
## Iteration 2: pi + host tunnel ✅
- pi: `@earendil-works/pi-coding-agent` pinned to host version (the old `@mariozechner` package stopped at 0.73.1).
- Config: allowlist of `~/.pi/agent` items copied to `share/stage` each start (not the whole dir). The copy is needed because the
host skill symlinks point outside `~/.pi` and would dangle in the guest. `auth.json` is opt-in.
- Tunnel over vsock (vfkit `virtio-vsock,listen`): no host listener on any network interface, per-sandbox, no port clashes.
Guest side is `guest/fwd.go` (static Go binary, built by `build.rs`), and it tells `run.sh` it's ready through a FIFO.
- Verified: 20 MB transfer over IPv4 and IPv6 loopback with matching hash. `wsbx pi -p` answered through the local omlx model on
host 127.0.0.1:8000 (first run 91 s incl. installs, then 3.8 s).
## Iteration 3: de-hardcoding ✅
| Was hardcoded | Now |
|---|---|
| `KATA_VERSION = "4.2.0"`, unverified curl | GitHub releases API (latest or pinned), asset sha256 verified while streaming, `kernel/source.json` records version |
| image tags cached forever | re-resolved after 24 h, falls back to the cache when offline |
| pi package name | read from the host's `pi` install (`package.json` name + version) |
| `/opt/homebrew/opt/e2fsprogs/sbin` | `brew --prefix <formula>` when not on PATH |
| `--cpus 4 --memory 4096` | half the host cores / a quarter of host RAM (≤ 8 GiB) |
| init/dhcp frozen into each disk | disk holds a 3-line stub; the real init is sent through the host share on every start |
Bugs found along the way: the image tar was attached read-write, so two sandboxes from one image couldn't run together.
It's now read-only and only attached until the guest reports it's unpacked. Poweroff raced ext4 lazy-init (`WRITE_ZEROES`
I/O error); root is now remounted read-only first.
Kept on purpose: Kata's in-tarball kernel path, `registry-1.docker.io`, arm64, `hvc0`/`vda`/`eth0` (we
define the device order), vsock ports, 20 GiB sparse disk, the agent image names and API-key env lists.
## Iteration 4: security review fixes ✅
| Finding | Fix | Verified |
|---|---|---|
| Critical: guest symlinks in the persistent share → host file overwrite | share recreated per start (VM stopped, lock held), deleted after exit; outputs read with no-symlink + size cap; `provisioned` moved to meta.json | guest planted `run.sh/init/dhcp/fwd -> victim`, `exit -> /etc/hosts`, plus the killed-wsbx leftover case: victim unchanged |
| High: comma injection into vfkit args | `vf()` rejects `,` `"` and control chars in every interpolated path; workspace overlapping `~/.wsbx` refused | `repo,sharedDir=/Users/niklas` rejected |
| High: automatic API-key forwarding | opt-in `--keys` with hint; `~/.wsbx` 0700; run.sh deleted after exit | key absent in VM without `--keys` |
| Medium: `rm ../..` | name must be a sanitized sandbox name | traversal refused |
| Medium: auth-realm SSRF | https only; public registries use an agent whose resolver refuses non-public IPs | unit test |
| Medium: digest refs not enforced | top-level digest compared; all registry digests format-checked; manifest size capped | pin OK, bogus pin rejected |
| Medium: pi copy follows symlinks | contained to staged roots (incl. `~/.agents/skills`, which pi reads natively), cycle-safe, no special files, 256 MB cap | unit test + `show-me` still arrives |
| (own) | reference/tag/registry validated before any request (`alpine@sha256:../../x` hit the registry as `/manifests/x`), busybox must be a regular file, Kata tag validated | unit tests |
## Iteration 5: cleanup ✅
- `wsbx gc`: keeps only images whose tag a sandbox uses (freed 1.74 GB here). A tag that moves now deletes the superseded
image automatically, unless another ref still points at it.
- Kernel panics: `panic=1` makes the kernel reboot, and VZ turns that into a restart loop. init writes a `booted` marker into
the per-launch share and powers off when it's already there. A real panic (`panic_on_oom`) now stops in about 2 s with
exit 1 and a message. Limit: a panic before the root disk mounts still loops.
- Found while testing panics: the first-boot unpack wasn't synced before `provisioned` was recorded, so a crash right after
first boot left an empty system. Fixed with a sync before marking.
- Found while reviewing: path validation ran after disk creation, so rejected runs left a sandbox behind. It now runs first.
- `.gitignore` covers `.llm-wiki/`; Clippy is clean.
## Out of scope (add when needed)
Port forwarding, network policy / credential-injecting egress proxy, snapshots, x86_64 hosts, zstd layers.