first commit
This commit is contained in:
@@ -0,0 +1,95 @@
|
||||
# wsbx — tiny microVM sandboxes for macOS
|
||||
|
||||
Goal: `docker sandbox`-like UX, but every sandbox is a real Linux microVM.
|
||||
|
||||
## Decisions
|
||||
|
||||
| Concern | Choice | Why |
|
||||
|---|---|---|
|
||||
| Hypervisor | Apple Virtualization.framework via **vfkit** (`brew install vfkit`) | Firecracker needs KVM (Linux only). vfkit is a signed, maintained CLI over VZ — no entitlement/codesign dance for us. |
|
||||
| Kernel | **Kata Containers** release kernel (`vmlinux.container`, arm64). The latest release is found via the GitHub API, then the tarball is streamed and checked against GitHub's published asset sha256, and only the kernel is kept. `kernel --update` / `--version` / `WSBX_KATA_VERSION` | Trusted upstream, built for VMs (virtio-pci/blk/net/fs/console built in, no initrd needed). Same kernel Apple's `container` uses. Override with `--kernel`. |
|
||||
| Base OS | Any **OCI image** (Docker Hub, GHCR, …) pulled by wsbx itself (anonymous token auth, sha256-verified blobs) | No Docker needed. |
|
||||
| Rootfs build | Host flattens layers → one tar (whiteouts applied). Host makes empty ext4 disk with `mke2fs` (brew e2fsprogs) seeded with a static busybox + init. **Guest extracts the tar as root on first boot.** | Correct uid/gid/devices/hardlinks and case-sensitivity — impossible to get right on an APFS host dir. |
|
||||
| Guest init | `/.wsbx/init` (busybox sh as PID 1): mounts, DHCP (VZ NAT), mounts workspace (virtiofs), runs image entrypoint+cmd as image user, powers off. | No systemd, boots in ~1s. |
|
||||
| Agents | Presets over Docker's own `docker/sandbox-templates:{claude-code,codex,gemini}` images | Same images Docker Sandboxes use; multi-arch. |
|
||||
|
||||
## CLI
|
||||
|
||||
```
|
||||
wsbx kernel # download/cache kernel
|
||||
wsbx pull IMAGE # pull + flatten image
|
||||
wsbx run [opts] IMAGE [-- CMD..] # bare VM from any image (default: shell)
|
||||
wsbx claude|codex|gemini [opts] # agent sandbox for current dir
|
||||
wsbx ls | wsbx rm NAME...
|
||||
opts: --name N --cpus N --memory MiB --disk GiB --workspace DIR --env K=V --fresh
|
||||
```
|
||||
|
||||
- Workspace dir is shared via virtio-fs at the **same absolute path** as on the host (like Docker Sandboxes).
|
||||
- Sandboxes are **persistent** (disk kept per name; default name derived from image+workspace). `--fresh`/`rm` resets.
|
||||
- Agent API key env vars (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GEMINI_API_KEY`, …) are forwarded if set.
|
||||
- Terminal size + `TERM` forwarded.
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
~/.wsbx/kernel/vmlinux
|
||||
~/.wsbx/images/<sha256>/{rootfs.tar,config.json}
|
||||
~/.wsbx/sandboxes/<name>/{disk.img,meta.json,share/run.sh}
|
||||
```
|
||||
|
||||
## Steps
|
||||
|
||||
1. Plan (this file). ✅
|
||||
2. ✅ OCI pull: parse ref, token auth, index → linux/arm64 manifest, verified blobs, flatten w/ whiteouts.
|
||||
3. ✅ Kernel fetch via `curl | tar --zstd` (bsdtar streams; no 650 MB file kept).
|
||||
4. ✅ Disk creation (`mke2fs -d` with busybox+init), init script.
|
||||
5. ✅ `run` via vfkit; agents presets; ls/rm.
|
||||
6. ✅ Verified: alpine (net/DNS/TLS/workspace rw/exit codes/interactive pty), ubuntu (apt), claude-code template (runs as `agent`, `claude --version`).
|
||||
Cold first boot of alpine ≈3.5 s, warm boot ≈0.6 s.
|
||||
|
||||
## Findings
|
||||
- vfkit requires an initrd → empty file works (kernel has everything built in).
|
||||
- vfkit `virtio-serial,stdio` needs a TTY → non-TTY falls back to `logFilePath=<our stdout>`.
|
||||
- VZ virtio-fs: any guest uid can write; files land owned by the host user.
|
||||
- Homebrew e2fsprogs lacks libarchive (`mke2fs -d x.tar` unsupported) → guest-side unpack.
|
||||
|
||||
## Iteration 2: pi + host tunnel ✅
|
||||
- pi: `@earendil-works/pi-coding-agent` pinned to host version (the old `@mariozechner` package stopped at 0.73.1).
|
||||
- Config: allowlist of `~/.pi/agent` items copied to `share/stage` each start (not the whole dir). The copy is needed because the
|
||||
host skill symlinks point outside `~/.pi` and would dangle in the guest. `auth.json` is opt-in.
|
||||
- Tunnel over vsock (vfkit `virtio-vsock,listen`): no host listener on any network interface, per-sandbox, no port clashes.
|
||||
Guest side is `guest/fwd.go` (static Go binary, built by `build.rs`), and it tells `run.sh` it's ready through a FIFO.
|
||||
- Verified: 20 MB transfer over IPv4 and IPv6 loopback with matching hash. `wsbx pi -p` answered through the local omlx model on
|
||||
host 127.0.0.1:8000 (first run 91 s incl. installs, then 3.8 s).
|
||||
|
||||
## Iteration 3: de-hardcoding ✅
|
||||
| Was hardcoded | Now |
|
||||
|---|---|
|
||||
| `KATA_VERSION = "4.2.0"`, unverified curl | GitHub releases API (latest or pinned), asset sha256 verified while streaming, `kernel/source.json` records version |
|
||||
| image tags cached forever | re-resolved after 24 h, falls back to the cache when offline |
|
||||
| pi package name | read from the host's `pi` install (`package.json` name + version) |
|
||||
| `/opt/homebrew/opt/e2fsprogs/sbin` | `brew --prefix <formula>` when not on PATH |
|
||||
| `--cpus 4 --memory 4096` | half the host cores / a quarter of host RAM (≤ 8 GiB) |
|
||||
| init/dhcp frozen into each disk | disk holds a 3-line stub; the real init is sent through the host share on every start |
|
||||
|
||||
Bugs found along the way: the image tar was attached read-write, so two sandboxes from one image couldn't run together.
|
||||
It's now read-only and only attached until the guest reports it's unpacked. Poweroff raced ext4 lazy-init (`WRITE_ZEROES`
|
||||
I/O error); root is now remounted read-only first.
|
||||
|
||||
Kept on purpose: Kata's in-tarball kernel path, `registry-1.docker.io`, arm64, `hvc0`/`vda`/`eth0` (we
|
||||
define the device order), vsock ports, 20 GiB sparse disk, the agent image names and API-key env lists.
|
||||
|
||||
## Iteration 4: security review fixes ✅
|
||||
| Finding | Fix | Verified |
|
||||
|---|---|---|
|
||||
| Critical: guest symlinks in the persistent share → host file overwrite | share recreated per start (VM stopped, lock held), deleted after exit; outputs read with no-symlink + size cap; `provisioned` moved to meta.json | guest planted `run.sh/init/dhcp/fwd -> victim`, `exit -> /etc/hosts`, plus the killed-wsbx leftover case: victim unchanged |
|
||||
| High: comma injection into vfkit args | `vf()` rejects `,` `"` and control chars in every interpolated path; workspace overlapping `~/.wsbx` refused | `repo,sharedDir=/Users/niklas` rejected |
|
||||
| High: automatic API-key forwarding | opt-in `--keys` with hint; `~/.wsbx` 0700; run.sh deleted after exit | key absent in VM without `--keys` |
|
||||
| Medium: `rm ../..` | name must be a sanitized sandbox name | traversal refused |
|
||||
| Medium: auth-realm SSRF | https only; public registries use an agent whose resolver refuses non-public IPs | unit test |
|
||||
| Medium: digest refs not enforced | top-level digest compared; all registry digests format-checked; manifest size capped | pin OK, bogus pin rejected |
|
||||
| Medium: pi copy follows symlinks | contained to staged roots (incl. `~/.agents/skills`, which pi reads natively), cycle-safe, no special files, 256 MB cap | unit test + `show-me` still arrives |
|
||||
| (own) | reference/tag/registry validated before any request (`alpine@sha256:../../x` hit the registry as `/manifests/x`), busybox must be a regular file, Kata tag validated | unit tests |
|
||||
|
||||
## Out of scope (add when needed)
|
||||
Port forwarding, network policy / credential-injecting egress proxy, snapshots, x86_64 hosts, zstd layers.
|
||||
Reference in New Issue
Block a user