[stack] Add system vats #797
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add core infrastructure for system vats - vats that run without compartment isolation directly in the host process. System vats enable the host application to use E() on vat object presences directly. This commit adds: - SystemVatId type and isSystemVatId() type guard - SystemSubclusterConfig and related types for configuring system subclusters - SystemVatSyscall: Handles syscalls from system vats (similar to VatSyscall) - SystemVatHandle: EndpointHandle implementation for system vats (kernel-side) - SystemVatSupervisor: Runtime-agnostic supervisor using liveslots without compartment isolation and with ephemeral (non-persistent) vatstore System vats: - Do NOT execute in a compartment - run directly in host process - Do NOT participate in kernel persistence machinery - Use an ephemeral Map-based vatstore - Receive buildRootObject function directly instead of loading bundles Co-Authored-By: Claude Opus 4.5 <[email protected]>
Add components for managing system subclusters: - KernelFacetService: A service object exposed as vatpower to system subcluster bootstrap vats. Provides privileged kernel operations: - launchSubcluster() - launches dynamic subclusters, returns presences - terminateSubcluster() - reloadSubcluster() - getSubcluster() / getSubclusters() - getStatus() - SystemSubclusterManager: Manages system subcluster lifecycle: - Launches system vats with correct vatpowers - Bootstrap vat receives kernel facet - Coordinates with kernel store for clist management - Tracks active system subclusters and their vats Co-Authored-By: Claude Opus 4.5 <[email protected]>
- Add SystemSubclusterManager to Kernel class
- Update #getEndpoint() to handle system vat IDs (sv${number})
- Add launchSystemSubcluster() and terminateSystemSubcluster() methods
- Rename makeKernelFacetService to makeKernelFacet (utility, not a service)
- Move kernel-facet.ts to vats/ directory (closer to where it is used)
The kernel facet is a utility that creates an object providing privileged
kernel operations to system subcluster bootstrap vats, not a platform-level
service like those in KernelServiceManager.
Co-Authored-By: Claude Opus 4.5 <[email protected]>
Add utilities for creating host subclusters in browser runtime: - makeHostSubcluster() factory function - HostSubclusterConfig, HostSubclusterVat, HostSubclusterResult types - Export new system vat types from @MetaMask/ocap-kernel Co-Authored-By: Claude Opus 4.5 <[email protected]>
- Move kernel-facet.ts from src/vats/ to src/ - Update import paths in SystemSubclusterManager.ts - Add SystemVatId type assertion for template literal Co-Authored-By: Claude Opus 4.5 <[email protected]>
Add comprehensive unit tests for: - SystemVatSyscall: syscall handling, error handling, termination - SystemVatHandle: message/notify delivery, GC, crank results - SystemVatSupervisor: liveslots integration, ephemeral vatstore - kernel-facet: makeKernelFacet utility function - SystemSubclusterManager: subcluster lifecycle, vat ID allocation - makeHostSubcluster: config conversion, kernel integration Also update index.test.ts files to include new exports. Co-Authored-By: Claude Opus 4.5 <[email protected]>
…anager Use direct type annotations on variable declarations instead of relying on type inference for template literal types. Co-Authored-By: Claude <[email protected]>
…otstrap Update kernel-worker to launch a kernel host system subcluster instead of using a plain kernel facade for CapTP. The kernel host vat: - Is a proper system vat with liveslots reference management - Receives the kernel facet as a vatpower - Uses E() to call kernel facet methods - Returns proper presences for dynamic subcluster roots This enables the background to receive E()-callable presences when launching dynamic subclusters through the kernel host. Co-Authored-By: Claude <[email protected]>
- Add KernelFacet type derived from KernelFacetDependencies with launchSubcluster overridden to return KernelFacetLaunchResult - Export KernelFacet and KernelFacetLaunchResult from ocap-kernel index - Update kernel-host-vat.ts to import KernelFacet from ocap-kernel instead of declaring a local duplicate type Co-Authored-By: Claude <[email protected]>
- Consolidate VatSyscall and SystemVatSyscall into a single VatSyscall class that handles both regular and system vats via constructor parameters - Add isActive callback and vatLabel parameters to VatSyscall - Fix CrankResults.terminate.vatId type to accept EndpointId - Document ephemeral KVStore and implement getKeys/getPrefixedKeys - Simplify buildVatNamespace vatPowers merge with clear documentation - Make logger required at type level in SystemSubclusterManagerOptions - Delete EndpointSyscall.ts and SystemVatSyscall.ts (no longer needed) Co-Authored-By: Claude <[email protected]>
The kernel-worker/captp/ directory contained makeKernelCapTP and makeKernelFacade which have been replaced by the kernel host subcluster. CapTP now uses the kernel host vat root directly as the bootstrap object. Files removed: - kernel-captp.ts, kernel-captp.test.ts - kernel-facade.ts, kernel-facade.test.ts - captp.integration.test.ts - index.ts Note: background-captp.ts is kept as it provides the CapTP message utilities used by both kernel-worker and background. Co-Authored-By: Claude <[email protected]>
Add host subcluster utilities for Node.js runtime that enable launching system subclusters with kernel facet access. - Add makeKernelHostSubclusterConfig() to create kernel host vat config - Add makeHostSubcluster() to launch and get root object - Export KernelHostRoot and HostSubclusterResult types - Add comprehensive unit tests Co-Authored-By: Claude Opus 4.5 <[email protected]>
Enable omnium-gatherum to store kref strings and restore presences after restart. KernelFacetLaunchResult now includes rootKref alongside the root SlotValue. The new getVatRoot(kref) method converts stored krefs back to slot values that become presences when marshalled via CapTP/liveslots. Co-Authored-By: Claude Opus 4.5 <[email protected]>
Add test vats for verifying system subcluster functionality: - alice-vat.js: orchestrates third-party handoff - bob-vat.js: creates greeter exos for handoff tests - carol-vat.js: receives and uses exos from other vats - promise-vat.js: tests deferred promises, rejection, pipelining Add e2e test suites: - system-subcluster.test.ts: full test suite for third-party handoff, promise handling, and kref restoration - system-subcluster-simple.test.ts: diagnostic tests to isolate system subcluster launch behavior Note: Tests are work-in-progress for debugging system subcluster launch timing issues. Co-Authored-By: Claude Opus 4.5 <[email protected]>
Replace direct supervisor instantiation with pluggable transport pattern: - Add SystemVatTransport interface with deliver/setSyscallHandler - Runtime creates supervisors, kernel connects via transports - Pass systemSubclusters config to Kernel.make() - Singleton kernel facet shared across system subclusters - Remove KernelHostRoot wrapper, expose KernelFacet directly This enables cross-process system vats (e.g., extension background) where kernel and supervisor run in different processes. Co-Authored-By: Claude <[email protected]>
- Fix queue length cache initialization (was showing 0 when item enqueued) - Fix vref format for system vats (sv0 now uses same format as v0) - Register kernel facet as kernel service for message delivery - Change bootstrap to fire-and-forget to avoid deadlock during Kernel.make() - Pass kernel facet via services parameter instead of separate argument - Add integration tests for system vat with kernel facet E() calls - Change SystemVatSupervisor.start() to throw on error instead of returning string Co-Authored-By: Claude <[email protected]>
…cranks Changed invokeKernelService to not await the service method result. Instead, it uses Promise chaining to resolve the kernel promise when the method eventually completes. This allows service methods to internally use waitForCrank() without causing deadlock - the crank can complete, and the resolution happens in a future turn of the event loop. Key changes: - KernelServiceManager.invokeKernelService() now returns void instead of Promise<void> and uses Promise.resolve().then().catch() for async handling - KernelRouter.#deliverKernelServiceMessage() is now synchronous - Updated tests to use delay() for microtask flushing - Enabled the integration test for E(kernelFacet).getStatus() Co-Authored-By: Claude <[email protected]>
TypeScript's WeakMap-based private field polyfills (used with target ES2020) caused "privateMap.get is not a function" errors when running under SES/lockdown in vitest fork mode. Changing target to ES2022 uses native private fields which work correctly with lockdown. Also fixes: - Bootstrap signature in host-subcluster (kernelFacet is in services) - Test for supervisor.start() throwing instead of returning error - Removes obsolete system-subcluster-simple.test.ts using old API Co-Authored-By: Claude <[email protected]>
- Use vi.waitFor() to poll for kernel facet availability instead of non-existent kernel.run() method - Fix all kernelFacet method calls to use E() since it's a presence - Fix promise rejection test to use rejects.toThrow() - Update host-subcluster deliver function to wait for supervisor using promise kit pattern instead of throwing Co-Authored-By: Claude <[email protected]>
Change system vat connection model from pull-based (kernel reaches out)
to push-based (supervisor initiates connection). This architectural
change ensures the kernel is passive and only accepts pre-declared
connections.
Key changes:
- Rename connectSystemSubcluster() to prepareSystemSubcluster()
- Make prepareSystemSubcluster() synchronous (returns immediately)
- Replace transport.onReady with transport.awaitConnection()
- Add connect() method to HostSubclusterResult
- Bootstrap message sent after all transports' awaitConnection() resolve
New flow:
1. makeHostSubcluster() returns { config, connect, kernelFacetPromise }
2. Kernel.make() calls prepareSystemSubcluster() (non-blocking)
3. After Kernel.make() returns, call hostSubcluster.connect()
4. Supervisor starts and resolves awaitConnection()
5. Kernel sends bootstrap message, kernelFacetPromise resolves
Co-Authored-By: Claude <[email protected]>
The push-based connection model makes the holder unnecessary. Since setSyscallHandler() is called before connect(), we can store the handler directly and pass it to SystemVatSupervisor.make(). Changes: - Remove SyscallHandlerHolder type and makeSyscallHandlerHolder() - SystemVatSupervisor now takes executeSyscall directly - Transport stores syscall handler and passes it on connect() Co-Authored-By: Claude <[email protected]>
- Rename "system subclusters" to "system vats" throughout - Rename SystemSubclusterManager to SystemVatManager - Rename host-subcluster to host-vat with simplified API - Add dynamic system vat registration via kernel facet - Remove DeliveryHelper (didn't achieve DRY savings) - Update Kernel to use systemVats config instead of systemSubclusters Co-Authored-By: Claude <[email protected]>
- Replace prepareStaticSystemVat() and registerDynamicSystemVat() with single registerSystemVat() method - Change Kernel.make() option from systemVats to hostVat for single host vat - Remove StaticSystemVatConfig, DynamicSystemVatConfig, KernelSystemVatsConfig - Rename SystemVatConfig to SystemSubclusterVatConfig (for subclusters) - Implement proper disconnect cleanup: reject pending promises and delete endpoint state - Update all tests for unified API Co-Authored-By: Claude <[email protected]>
Extract the shared #getCrankResults() logic from VatHandle and SystemVatHandle into VatSyscall, eliminating duplicate code. The method accepts an optional deliveryError parameter to support both usage patterns. Co-Authored-By: Claude <[email protected]>
…erns Make VatHandle explicitly pass deliveryError to getCrankResults() like SystemVatHandle does. Remove the deliveryError instance property from VatSyscall since it's no longer needed. Co-Authored-By: Claude <[email protected]>
…tHandle Create abstract base class BaseVatHandle that implements the 6 delivery methods shared between VatHandle and SystemVatHandle. Both classes now extend BaseVatHandle and provide a deliver function via setDeliver(). - Add BaseVatHandle with deliverMessage, deliverNotify, deliverDropExports, deliverRetireExports, deliverRetireImports, deliverBringOutYourDead - Update VatHandle to extend BaseVatHandle - Update SystemVatHandle to extend BaseVatHandle - Export BaseVatHandle, DeliverFn, and DeliveryObject types Co-Authored-By: Claude Opus 4.5 <[email protected]>
…oercion Move DeliveryObject type from BaseVatHandle.ts to types.ts for better discoverability and export it from the package. Remove unnecessary message coercion in SystemVatHandle since our Message type (with optional result) is equivalent to SwingSet's for JSON serialization. Co-Authored-By: Claude <[email protected]>
Replace host-subcluster with host-vat pattern that supports system vat supervisors running in a different process than the kernel. The kernel runs in a Worker, the host vat supervisor runs in the background script, and they communicate over a stream using the optimistic syscall model. Key changes: - Add makeKernelHostVat() for kernel Worker side - Add makeBackgroundHostVat() for background script side - Define typed messages for kernel-supervisor communication - Update kernel-worker to use the new transport pattern - Delete obsolete host-subcluster and kernel-host-vat modules Co-Authored-By: Claude Opus 4.5 <[email protected]>
Replace ad hoc message types with RpcClient/RpcService from @metamask/kernel-rpc-methods, using JSON-RPC 2.0 format for all communication between kernel and supervisor sides. Co-Authored-By: Claude Opus 4.5 <[email protected]>
Replace legacy CapTP connection with makeBackgroundHostVat API: - Fix kernel-worker.ts to pass hostVat.config to Kernel.make() - Simplify makeBackgroundHostVat to use internal buildRootObject - Update background.ts to use new host vat API - kernelFacet is now captured from bootstrap message services Co-Authored-By: Claude Opus 4.5 <[email protected]>
The host vat introduction allocates 2 kernel objects (ko3, ko4) before user vats, shifting all user vat kref assignments by +2. Updates hardcoded kref values in e2e tests accordingly. Co-Authored-By: Claude Opus 4.5 <[email protected]>
Replace legacy CapTP connection with makeBackgroundHostVat API: - Update background.ts to use new host vat API - Update controllers/index.ts to use KernelFacet type - kernelFacet is now captured from bootstrap message services Co-Authored-By: Claude Opus 4.5 <[email protected]>
Rename the public API method from `getCapletRoot` to `getRoot` for a cleaner, more idiomatic interface. Add usage documentation showing how to load, install, and interact with caplets using E(). Co-Authored-By: Claude <[email protected]>
Test that a vat can await a promise created in the host vat. Creates a deferred promise using makePromiseKit, passes it to the promise-vat, resolves it from the host, and verifies the vat receives the value. Co-Authored-By: Claude <[email protected]>
Contributor
…minatedVat Extend cleanupTerminatedVat to handle both regular vats and system vats, eliminating the need for a separate cleanupEndpoint function. The function now automatically detects system vats (IDs starting with 'sv') and skips vat-specific operations: - Terminated vat check (system vats aren't in the terminated list) - Vat KV entries cleanup (system vats have no vatstore) - Terminated vats list removal Also simplify orphaned system vat cleanup in Kernel.ts to scan for system vat c-list keys directly instead of using a separate helper. Co-Authored-By: Claude Opus 4.5 <[email protected]>
Remove all logger?.log() calls from kernel-facet.ts and remove the logger dependency from KernelFacetDependencies. This simplifies the facet by removing verbose logging that adds noise without significant debugging value. - Remove Logger import and logger from dependencies type - Remove logger?.log() calls from all methods - Update tests to remove logging-related test cases - Update Kernel.ts to stop passing logger to kernelFacetDeps Co-Authored-By: Claude <[email protected]>
Member
Author
|
@cursor review |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| // Syscall handler - set by kernel during registerSystemVat() | ||
| let syscallHandler: SystemVatSyscallHandler | null = null; | ||
|
|
||
| const supervisor: SystemVatSupervisor | null = null; |
There was a problem hiding this comment.
Dead supervisor variable never assigned after initialization
Low Severity
The supervisor variable is declared as const and initialized to null, but never reassigned. In the connect() function, the .then() callback only calls supervisorReady.resolve(result) without updating supervisor. This makes the nullish coalescing check supervisor ?? (await supervisorReady.promise) on line 96 dead code since supervisor is always null. The intent appears to be caching the supervisor reference, but the assignment was not implemented.
Additional Locations (2)
| services: { kernelFacet: KernelFacet }, | ||
| ) => { | ||
| kernelFacetKit.resolve(services.kernelFacet); | ||
| }, |
There was a problem hiding this comment.
Missing kernelFacet validation in Node.js bootstrap handler
Low Severity
The Node.js host vat bootstrap handler directly resolves kernelFacetKit with services.kernelFacet without checking if it exists. The browser version in supervisor-side.ts defensively checks if (services.kernelFacet) and rejects with a clear error if not provided. If kernelFacet is ever undefined, the Node.js version would resolve the promise with undefined, causing confusing runtime errors when using the facet later.
Additional Locations (1)
| // Syscall handler - set by kernel during registerSystemVat() | ||
| let syscallHandler: SystemVatSyscallHandler | null = null; | ||
|
|
||
| const supervisor: SystemVatSupervisor | null = null; |
There was a problem hiding this comment.
Unused supervisor variable is dead code
Low Severity
The supervisor variable is declared as const initialized to null and never reassigned. On line 96, the expression supervisor ?? (await supervisorReady.promise) always evaluates to the right-hand side since supervisor is always null. The variable and null-coalescing check serve no purpose.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces system vats - vats that run directly in the runtime process (background/main script) without compartment isolation, enabling privileged kernel operations from the host environment.
Architecture
System vats
KernelFacetin the bootstrap message for privileged operationscleanupTerminatedVatlogic as regular vats, skipping steps not relevant for system vatsKey changes
ocap-kernel): AddSystemVatManager,SystemVatHandle,SystemVatSupervisor, andKernelFacetto manage vats that connect to the kernel via transports rather than being spawned by itKernelFacetproviding privileged operations likelaunchSubcluster(),terminateSubcluster(),getStatus(), etc.nodejs): AddmakeHostVat()utility for easy host vat setupkernel-browser-runtime): Add cross-process system vat supervisor infrastructureextensionandomnium-gatherumbackground scripts, replacing the previous CapTP-based kernel facadecleanupTerminatedVatto handle both regular vats and system vats, automatically detecting system vat IDs (sv*) and skipping vat-specific operations (terminated check, vat KV cleanup, terminated list removal)Test plan
SystemVatManager,SystemVatHandle,SystemVatSupervisor,KernelFacetsystem-vat.test.ts)Note
High Risk
High risk because it introduces a new system-vat execution/transport path, expands endpoint identity semantics (
sv*), and changes kernel service invocation timing and kernel startup cleanup behavior, all of which affect core message delivery and persistence cleanup.Overview
Introduces system vats to
@metamask/ocap-kernel: newSystemVatId/transport config types,SystemVatManagerregistration/disconnect flow,KernelFacet(privileged API + dynamic system-vat registration), andSystemVatHandle/BaseVatHandlefor delivering deliveries and routing syscalls.Updates kernel lifecycle and storage semantics:
Kernel.make()acceptshostVat, kernel startup cleans up orphaned system-vat state,cleanupTerminatedVat/deleteEndpoint/c-list allocation handlesv*endpoints, and kernel service invocation is made non-blocking (noawait) to avoid crank deadlocks.Replaces browser CapTP wiring with a JSON-RPC host-vat layer in
kernel-browser-runtime(newmakeKernelHostVat/makeBackgroundHostVat, kernel worker refactor, CapTP worker code/tests removed) and updates the extension background/global types plus e2e expectations.Adds Node.js
makeHostVat()helper and new end-to-end coverage (including new test vats) to validate subcluster operations, third-party handoff, promise behavior, and kref-to-presence restoration.Written by Cursor Bugbot for commit 7be05e6. This will update automatically on new commits. Configure here.