1# Notable rationale of wazero
2
3## Zero dependencies
4
5Wazero has zero dependencies to differentiate itself from other runtimes which
6have heavy impact usually due to CGO. By avoiding CGO, wazero avoids
7prerequisites such as shared libraries or libc, and lets users keep features
8like cross compilation.
9
10Avoiding go.mod dependencies reduces interference on Go version support, and
11size of a statically compiled binary. However, doing so brings some
12responsibility into the project.
13
14Go's native platform support is good: We don't need platform-specific code to
15get monotonic time, nor do we need much work to implement certain features
16needed by our compiler such as `mmap`. That said, Go does not support all
17common operating systems to the same degree. For example, Go 1.18 includes
18`Mprotect` on Linux and Darwin, but not FreeBSD.
19
20The general tradeoff the project takes from a zero dependency policy is more
21explicit support of platforms (in the compiler runtime), as well a larger and
22more technically difficult codebase.
23
24At some point, we may allow extensions to supply their own platform-specific
25hooks. Until then, one end user impact/tradeoff is some glitches trying
26untested platforms (with the Compiler runtime).
27
28### Why do we use CGO to implement system calls on darwin?
29
30wazero is dependency and CGO free by design. In some cases, we have code that
31can optionally use CGO, but retain a fallback for when that's disabled. The only
32operating system (`GOOS`) we use CGO by default in is `darwin`.
33
34Unlike other operating systems, regardless of `CGO_ENABLED`, Go always uses
35"CGO" mechanisms in the runtime layer of `darwin`. This is explained in
36[Statically linked binaries on Mac OS X](https://developer.apple.com/library/archive/qa/qa1118/_index.html#//apple_ref/doc/uid/DTS10001666):
37
38> Apple does not support statically linked binaries on Mac OS X. A statically
39> linked binary assumes binary compatibility at the kernel system call
40> interface, and we do not make any guarantees on that front. Rather, we strive
41> to ensure binary compatibility in each dynamically linked system library and
42> framework.
43
44This plays to our advantage for system calls that aren't yet exposed in the Go
45standard library, notably `futimens` for nanosecond-precision timestamp
46manipulation.
47
48### Why not x/sys
49
50Going beyond Go's SDK limitations can be accomplished with their [x/sys library](https://pkg.go.dev/golang.org/x/sys/unix).
51For example, this includes `zsyscall_freebsd_amd64.go` missing from the Go SDK.
52
53However, like all dependencies, x/sys is a source of conflict. For example,
54x/sys had to be in order to upgrade to Go 1.18.
55
56If we depended on x/sys, we could get more precise functionality needed for
57features such as clocks or more platform support for the compiler runtime.
58
59That said, formally supporting an operating system may still require testing as
60even use of x/sys can require platform-specifics. For example, [mmap-go](https://github.com/edsrzf/mmap-go)
61uses x/sys, but also mentions limitations, some not surmountable with x/sys
62alone.
63
64Regardless, we may at some point introduce a separate go.mod for users to use
65x/sys as a platform plugin without forcing all users to maintain that
66dependency.
67
68## Project structure
69
70wazero uses internal packages extensively to balance API compatibility desires for end users with the need to safely
71share internals between compilers.
72
73End-user packages include `wazero`, with `Config` structs, `api`, with shared types, and the built-in `wasi` library.
74Everything else is internal.
75
76We put the main program for wazero into a directory of the same name to match conventions used in `go install`,
77notably the name of the folder becomes the binary name. We chose to use `cmd/wazero` as it is common practice
78and less surprising than `wazero/wazero`.
79
80### Internal packages
81
82Most code in wazero is internal, and it is acknowledged that this prevents external implementation of facets such as
83compilers or decoding. It also prevents splitting this code into separate repositories, resulting in a larger monorepo.
84This also adds work as more code needs to be centrally reviewed.
85
86However, the alternative is neither secure nor viable. To allow external implementation would require exporting symbols
87public, such as the `CodeSection`, which can easily create bugs. Moreover, there's a high drift risk for any attempt at
88external implementations, compounded not just by wazero's code organization, but also the fast moving Wasm and WASI
89specifications.
90
91For example, implementing a compiler correctly requires expertise in Wasm, Golang and assembly. This requires deep
92insight into how internals are meant to be structured and the various tiers of testing required for `wazero` to result
93in a high quality experience. Even if someone had these skills, supporting external code would introduce variables which
94are constants in the central one. Supporting an external codebase is harder on the project team, and could starve time
95from the already large burden on the central codebase.
96
97The tradeoffs of internal packages are a larger codebase and responsibility to implement all standard features. It also
98implies thinking about extension more as forking is not viable for reasons above also. The primary mitigation of these
99realities are friendly OSS licensing, high rigor and a collaborative spirit which aim to make contribution in the shared
100codebase productive.
101
102### Avoiding cyclic dependencies
103
104wazero shares constants and interfaces with internal code by a sharing pattern described below:
105* shared interfaces and constants go in one package under root: `api`.
106* user APIs and structs depend on `api` and go into the root package `wazero`.
107 * e.g. `InstantiateModule` -> `/wasm.go` depends on the type `api.Module`.
108* implementation code can also depend on `api` in a corresponding package under `/internal`.
109 * Ex package `wasm` -> `/internal/wasm/*.go` and can depend on the type `api.Module`.
110
111The above guarantees no cyclic dependencies at the cost of having to re-define symbols that exist in both packages.
112For example, if `wasm.Store` is a type the user needs access to, it is narrowed by a cover type in the `wazero`:
113
114```go
115type runtime struct {
116 s *wasm.Store
117}
118```
119
120This is not as bad as it sounds as mutations are only available via configuration. This means exported functions are
121limited to only a few functions.
122
123### Avoiding security bugs
124
125In order to avoid security flaws such as code insertion, nothing in the public API is permitted to write directly to any
126mutable symbol in the internal package. For example, the package `api` is shared with internal code. To ensure
127immutability, the `api` package cannot contain any mutable public symbol, such as a slice or a struct with an exported
128field.
129
130In practice, this means shared functionality like memory mutation need to be implemented by interfaces.
131
132Here are some examples:
133* `api.Memory` protects access by exposing functions like `WriteFloat64Le` instead of exporting a buffer (`[]byte`).
134* There is no exported symbol for the `[]byte` representing the `CodeSection`
135
136Besides security, this practice prevents other bugs and allows centralization of validation logic such as decoding Wasm.
137
138## API Design
139
140### Why is `context.Context` inconsistent?
141
142It may seem strange that only certain API have an initial `context.Context`
143parameter. We originally had a `context.Context` for anything that might be
144traced, but it turned out to be only useful for lifecycle and host functions.
145
146For instruction-scoped aspects like memory updates, a context parameter is too
147fine-grained and also invisible in practice. For example, most users will use
148the compiler engine, and its memory, global or table access will never use go's
149context.
150
151### Why does `api.ValueType` map to uint64?
152
153WebAssembly allows functions to be defined either by the guest or the host,
154with signatures expressed as WebAssembly types. For example, `i32` is a 32-bit
155type which might be interpreted as signed. Function signatures can have zero or
156more parameters or results even if WebAssembly 1.0 allows up to one result.
157
158The guest can export functions, so that the host can call it. In the case of
159wazero, the host is Go and an exported function can be called via
160`api.Function`. `api.Function` allows users to supply parameters and read
161results as a slice of uint64. For example, if there are no results, an empty
162slice is returned. The user can learn the signature via `FunctionDescription`,
163which returns the `api.ValueType` corresponding to each parameter or result.
164`api.ValueType` defines the mapping of WebAssembly types to `uint64` values for
165reason described in this section. The special case of `v128` is also mentioned
166below.
167
168wazero maps each value type to a uint64 values because it holds the largest
169type in WebAssembly 1.0 (i64). A slice allows you to express empty (e.g. a
170nullary signature), for example a start function.
171
172Here's an example of calling a function, noting this syntax works for both a
173signature `(param i32 i32) (result i32)` and `(param i64 i64) (result i64)`
174```go
175x, y := uint64(1), uint64(2)
176results, err := mod.ExportedFunction("add").Call(ctx, x, y)
177if err != nil {
178 log.Panicln(err)
179}
180fmt.Printf("%d + %d = %d\n", x, y, results[0])
181```
182
183WebAssembly does not define an encoding strategy for host defined parameters or
184results. This means the encoding rules above are defined by wazero instead. To
185address this, we clarified mapping both in `api.ValueType` and added helper
186functions like `api.EncodeF64`. This allows users conversions typical in Go
187programming, and utilities to avoid ambiguity and edge cases around casting.
188
189Alternatively, we could have defined a byte buffer based approach and a binary
190encoding of value types in and out. For example, an empty byte slice would mean
191no values, while a non-empty could use a binary encoding for supported values.
192This could work, but it is more difficult for the normal case of i32 and i64.
193It also shares a struggle with the current approach, which is that value types
194were added after WebAssembly 1.0 and not all of them have an encoding. More on
195this below.
196
197In summary, wazero chose an approach for signature mapping because there was
198none, and the one we chose biases towards simplicity with integers and handles
199the rest with documentation and utilities.
200
201#### Post 1.0 value types
202
203Value types added after WebAssembly 1.0 stressed the current model, as some
204have no encoding or are larger than 64 bits. While problematic, these value
205types are not commonly used in exported (extern) functions. However, some
206decisions were made and detailed below.
207
208For example `externref` has no guest representation. wazero chose to map
209references to uint64 as that's the largest value needed to encode a pointer on
210supported platforms. While there are two reference types, `externref` and
211`functype`, the latter is an internal detail of function tables, and the former
212is rarely if ever used in function signatures as of the end of 2022.
213
214The only value larger than 64 bits is used for SIMD (`v128`). Vectorizing via
215host functions is not used as of the end of 2022. Even if it were, it would be
216inefficient vs guest vectorization due to host function overhead. In other
217words, the `v128` value type is unlikely to be in an exported function
218signature. That it requires two uint64 values to encode is an internal detail
219and not worth changing the exported function interface `api.Function`, as doing
220so would break all users.
221
222### Interfaces, not structs
223
224All exported types in public packages, regardless of configuration vs runtime, are interfaces. The primary benefits are
225internal flexibility and avoiding people accidentally mis-initializing by instantiating the types on their own vs using
226the `NewXxx` constructor functions. In other words, there's less support load when things can't be done incorrectly.
227
228Here's an example:
229```go
230rt := &RuntimeConfig{} // not initialized properly (fields are nil which shouldn't be)
231rt := RuntimeConfig{} // not initialized properly (should be a pointer)
232rt := wazero.NewRuntimeConfig() // initialized properly
233```
234
235There are a few drawbacks to this, notably some work for maintainers.
236* Interfaces are decoupled from the structs implementing them, which means the signature has to be repeated twice.
237* Interfaces have to be documented and guarded at time of use, that 3rd party implementations aren't supported.
238* As of Golang 1.21, interfaces are still [not well supported](https://github.com/golang/go/issues/5860) in godoc.
239
240## Config
241
242wazero configures scopes such as Runtime and Module using `XxxConfig` types. For example, `RuntimeConfig` configures
243`Runtime` and `ModuleConfig` configure `Module` (instantiation). In all cases, config types begin defaults and can be
244customized by a user, e.g., selecting features or a module name override.
245
246### Why don't we make each configuration setting return an error?
247No config types create resources that would need to be closed, nor do they return errors on use. This helps reduce
248resource leaks, and makes chaining easier. It makes it possible to parse configuration (ex by parsing yaml) independent
249of validating it.
250
251Instead of:
252```
253cfg, err = cfg.WithFS(fs)
254if err != nil {
255 return err
256}
257cfg, err = cfg.WithName(name)
258if err != nil {
259 return err
260}
261mod, err = rt.InstantiateModuleWithConfig(ctx, code, cfg)
262if err != nil {
263 return err
264}
265```
266
267There's only one call site to handle errors:
268```
269cfg = cfg.WithFS(fs).WithName(name)
270mod, err = rt.InstantiateModuleWithConfig(ctx, code, cfg)
271if err != nil {
272 return err
273}
274```
275
276This allows users one place to look for errors, and also the benefit that if anything internally opens a resource, but
277errs, there's nothing they need to close. In other words, users don't need to track which resources need closing on
278partial error, as that is handled internally by the only code that can read configuration fields.
279
280### Why are configuration immutable?
281While it seems certain scopes like `Runtime` won't repeat within a process, they do, possibly in different goroutines.
282For example, some users create a new runtime for each module, and some re-use the same base module configuration with
283only small updates (ex the name) for each instantiation. Making configuration immutable allows them to be safely used in
284any goroutine.
285
286Since config are immutable, changes apply via return val, similar to `append` in a slice.
287
288For example, both of these are the same sort of error:
289```go
290append(slice, element) // bug as only the return value has the updated slice.
291cfg.WithName(next) // bug as only the return value has the updated name.
292```
293
294Here's an example of correct use: re-assigning explicitly or via chaining.
295```go
296cfg = cfg.WithName(name) // explicit
297
298mod, err = rt.InstantiateModuleWithConfig(ctx, code, cfg.WithName(name)) // implicit
299if err != nil {
300 return err
301}
302```
303
304### Why aren't configuration assigned with option types?
305The option pattern is a familiar one in Go. For example, someone defines a type `func (x X) err` and uses it to update
306the target. For example, you could imagine wazero could choose to make `ModuleConfig` from options vs chaining fields.
307
308Ex instead of:
309```go
310type ModuleConfig interface {
311 WithName(string) ModuleConfig
312 WithFS(fs.FS) ModuleConfig
313}
314
315struct moduleConfig {
316 name string
317 fs fs.FS
318}
319
320func (c *moduleConfig) WithName(name string) ModuleConfig {
321 ret := *c // copy
322 ret.name = name
323 return &ret
324}
325
326func (c *moduleConfig) WithFS(fs fs.FS) ModuleConfig {
327 ret := *c // copy
328 ret.setFS("/", fs)
329 return &ret
330}
331
332config := r.NewModuleConfig().WithFS(fs)
333configDerived := config.WithName("name")
334```
335
336An option function could be defined, then refactor each config method into an name prefixed option function:
337```go
338type ModuleConfig interface {
339}
340struct moduleConfig {
341 name string
342 fs fs.FS
343}
344
345type ModuleConfigOption func(c *moduleConfig)
346
347func ModuleConfigName(name string) ModuleConfigOption {
348 return func(c *moduleConfig) {
349 c.name = name
350 }
351}
352
353func ModuleConfigFS(fs fs.FS) ModuleConfigOption {
354 return func(c *moduleConfig) {
355 c.fs = fs
356 }
357}
358
359func (r *runtime) NewModuleConfig(opts ...ModuleConfigOption) ModuleConfig {
360 ret := newModuleConfig() // defaults
361 for _, opt := range opts {
362 opt(&ret.config)
363 }
364 return ret
365}
366
367func (c *moduleConfig) WithOptions(opts ...ModuleConfigOption) ModuleConfig {
368 ret := *c // copy base config
369 for _, opt := range opts {
370 opt(&ret.config)
371 }
372 return ret
373}
374
375config := r.NewModuleConfig(ModuleConfigFS(fs))
376configDerived := config.WithOptions(ModuleConfigName("name"))
377```
378
379wazero took the path of the former design primarily due to:
380* interfaces provide natural namespaces for their methods, which is more direct than functions with name prefixes.
381* parsing config into function callbacks is more direct vs parsing config into a slice of functions to do the same.
382* in either case derived config is needed and the options pattern is more awkward to achieve that.
383
384There are other reasons such as test and debug being simpler without options: the above list is constrained to conserve
385space. It is accepted that the options pattern is common in Go, which is the main reason for documenting this decision.
386
387### Why aren't config types deeply structured?
388wazero's configuration types cover the two main scopes of WebAssembly use:
389* `RuntimeConfig`: This is the broadest scope, so applies also to compilation
390 and instantiation. e.g. This controls the WebAssembly Specification Version.
391* `ModuleConfig`: This affects modules instantiated after compilation and what
392 resources are allowed. e.g. This defines how or if STDOUT is captured. This
393 also allows sub-configuration of `FSConfig`.
394
395These default to a flat definition each, with lazy sub-configuration only after
396proven to be necessary. A flat structure is easier to work with and is also
397easy to discover. Unlike the option pattern described earlier, more
398configuration in the interface doesn't taint the package namespace, only
399`ModuleConfig`.
400
401We default to a flat structure to encourage simplicity. If we eagerly broke out
402all possible configurations into sub-types (e.g. ClockConfig), it would be hard
403to notice configuration sprawl. By keeping the config flat, it is easy to see
404the cognitive load we may be adding to our users.
405
406In other words, discomfort adding more configuration is a feature, not a bug.
407We should only add new configuration rarely, and before doing so, ensure it
408will be used. In fact, this is why we support using context fields for
409experimental configuration. By letting users practice, we can find out if a
410configuration was a good idea or not before committing to it, and potentially
411sprawling our types.
412
413In reflection, this approach worked well for the nearly 1.5 year period leading
414to version 1.0. We've only had to create a single sub-configuration, `FSConfig`,
415and it was well understood why when it occurred.
416
417## Why does `ModuleConfig.WithStartFunctions` default to `_start`?
418
419We formerly had functions like `StartWASICommand` that would verify
420preconditions and start WASI's `_start` command. However, this caused confusion
421because both many languages compiled a WASI dependency, and many did so
422inconsistently.
423
424The conflict is that exported functions need to use features the language
425runtime provides, such as garbage collection. There's a "chicken-egg problem"
426where `_start` needs to complete in order for exported behavior to work.
427
428For example, unlike `GOOS=wasip1` in Go 1.21, TinyGo's "wasi" target supports
429function exports. So, the only way to use FFI style is via the "wasi" target.
430Not explicitly calling `_start` before an ABI such as wapc-go, would crash, due
431to setup not happening (e.g. to implement `panic`). Other embedders such as
432Envoy also called `_start` for the same reason. To avoid a common problem for
433users unaware of WASI, and also to simplify normal use of WASI (e.g. `main`),
434we added `_start` to `ModuleConfig.WithStartFunctions`.
435
436In cases of multiple initializers, such as in wapc-go, users can override this
437to add the others *after* `_start`. Users who want to explicitly control
438`_start`, such as some of our unit tests, can clear the start functions and
439remove it.
440
441This decision was made in 2022, and holds true in 2023, even with the
442introduction of "wasix". It holds because "wasix" is backwards compatible with
443"wasip1". In the future, there will be other ways to start applications, and
444may not be backwards compatible with "wasip1".
445
446Most notably WASI "Preview 2" is not implemented in a way compatible with
447wasip1. Its start function is likely to be different, and defined in the
448wasi-cli "world". When the design settles, and it is implemented by compilers,
449wazero will attempt to support "wasip2". However, it won't do so in a way that
450breaks existing compilers.
451
452In other words, we won't remove `_start` if "wasip2" continues a path of an
453alternate function name. If we did, we'd break existing users despite our
454compatibility promise saying we don't. The most likely case is that when we
455build-in something incompatible with "wasip1", that start function will be
456added to the start functions list in addition to `_start`.
457
458See http://wasix.org
459See https://github.com/WebAssembly/wasi-cli
460
461## Runtime == Engine+Store
462wazero defines a single user-type which combines the specification concept of `Store` with the unspecified `Engine`
463which manages them.
464
465### Why not multi-store?
466Multi-store isn't supported as the extra tier complicates lifecycle and locking. Moreover, in practice it is unusual for
467there to be an engine that has multiple stores which have multiple modules. More often, it is the case that there is
468either 1 engine with 1 store and multiple modules, or 1 engine with many stores, each having 1 non-host module. In worst
469case, a user can use multiple runtimes until "multi-store" is better understood.
470
471If later, we have demand for multiple stores, that can be accomplished by overload. e.g. `Runtime.InstantiateInStore` or
472`Runtime.Store(name) Store`.
473
474## Exit
475
476### Why do we only return a `sys.ExitError` on a non-zero exit code?
477
478It is reasonable to think an exit error should be returned, even if the code is
479success (zero). Even on success, the module is no longer functional. For
480example, function exports would error later. However, wazero does not. The only
481time `sys.ExitError` is on error (non-zero).
482
483This decision was to improve performance and ergonomics for guests that both
484use WASI (have a `_start` function), and also allow custom exports.
485Specifically, Rust, TinyGo and normal wasi-libc, don't exit the module during
486`_start`. If they did, it would invalidate their function exports. This means
487it is unlikely most compilers will change this behavior.
488
489`GOOS=waspi1` from Go 1.21 does exit during `_start`. However, it doesn't
490support other exports besides `_start`, and `_start` is not defined to be
491called multiple times anyway.
492
493Since `sys.ExitError` is not always returned, we added `Module.IsClosed` for
494defensive checks. This helps integrators avoid calling functions which will
495always fail.
496
497### Why panic with `sys.ExitError` after a host function exits?
498
499Currently, the only portable way to stop processing code is via panic. For
500example, WebAssembly "trap" instructions, such as divide by zero, are
501implemented via panic. This ensures code isn't executed after it.
502
503When code reaches the WASI `proc_exit` instruction, we need to stop processing.
504Regardless of the exit code, any code invoked after exit would be in an
505inconsistent state. This is likely why unreachable instructions are sometimes
506inserted after exit: https://github.com/emscripten-core/emscripten/issues/12322
507
508## WASI
509
510Unfortunately, (WASI Snapshot Preview 1)[https://github.com/WebAssembly/WASI/blob/snapshot-01/phases/snapshot/docs.md] is not formally defined enough, and has APIs with ambiguous semantics.
511This section describes how Wazero interprets and implements the semantics of several WASI APIs that may be interpreted differently by different wasm runtimes.
512Those APIs may affect the portability of a WASI application.
513
514### Why don't we attempt to pass wasi-testsuite on user-defined `fs.FS`?
515
516While most cases work fine on an `os.File` based implementation, we won't
517promise wasi-testsuite compatibility on user defined wrappers of `os.DirFS`.
518The only option for real systems is to use our `sysfs.FS`.
519
520There are a lot of areas where windows behaves differently, despite the
521`os.File` abstraction. This goes well beyond file locking concerns (e.g.
522`EBUSY` errors on open files). For example, errors like `ACCESS_DENIED` aren't
523properly mapped to `EPERM`. There are trickier parts too. `FileInfo.Sys()`
524doesn't return enough information to build inodes needed for WASI. To rebuild
525them requires the full path to the underlying file, not just its directory
526name, and there's no way for us to get that information. At one point we tried,
527but in practice things became tangled and functionality such as read-only
528wrappers became untenable. Finally, there are version-specific behaviors which
529are difficult to maintain even in our own code. For example, go 1.20 opens
530files in a different way than versions before it.
531
532### Why aren't WASI rules enforced?
533
534The [snapshot-01](https://github.com/WebAssembly/WASI/blob/snapshot-01/phases/snapshot/docs.md) version of WASI has a
535number of rules for a "command module", but only the memory export rule is enforced. If a "_start" function exists, it
536is enforced to be the correct signature and succeed, but the export itself isn't enforced. It follows that this means
537exports are not required to be contained to a "_start" function invocation. Finally, the "__indirect_function_table"
538export is also not enforced.
539
540The reason for the exceptions are that implementations aren't following the rules. For example, TinyGo doesn't export
541"__indirect_function_table", so crashing on this would make wazero unable to run TinyGo modules. Similarly, modules
542loaded by wapc-go don't always define a "_start" function. Since "snapshot-01" is not a proper version, and certainly
543not a W3C recommendation, there's no sense in breaking users over matters like this.
544
545### Why is I/O configuration not coupled to WASI?
546
547WebAssembly System Interfaces (WASI) is a formalization of a practice that can be done anyway: Define a host function to
548access a system interface, such as writing to STDOUT. WASI stalled at snapshot-01 and as of early 2023, is being
549rewritten entirely.
550
551This instability implies a need to transition between WASI specs, which places wazero in a position that requires
552decoupling. For example, if code uses two different functions to call `fd_write`, the underlying configuration must be
553centralized and decoupled. Otherwise, calls using the same file descriptor number will end up writing to different
554places.
555
556In short, wazero defined system configuration in `ModuleConfig`, not a WASI type. This allows end-users to switch from
557one spec to another with minimal impact. This has other helpful benefits, as centralized resources are simpler to close
558coherently (ex via `Module.Close`).
559
560In reflection, this worked well as more ABI became usable in wazero.
561
562### Background on `ModuleConfig` design
563
564WebAssembly 1.0 (20191205) specifies some aspects to control isolation between modules ([sandboxing](https://en.wikipedia.org/wiki/Sandbox_(computer_security))).
565For example, `wasm.Memory` has size constraints and each instance of it is isolated from each other. While `wasm.Memory`
566can be shared, by exporting it, it is not exported by default. In fact a WebAssembly Module (Wasm) has no memory by
567default.
568
569While memory is defined in WebAssembly 1.0 (20191205), many aspects are not. Let's use an example of `exec.Cmd` as for
570example, a WebAssembly System Interfaces (WASI) command is implemented as a module with a `_start` function, and in many
571ways acts similar to a process with a `main` function.
572
573To capture "hello world" written to the console (stdout a.k.a. file descriptor 1) in `exec.Cmd`, you would set the
574`Stdout` field accordingly, perhaps to a buffer. In WebAssembly 1.0 (20191205), the only way to perform something like
575this is via a host function (ex `HostModuleFunctionBuilder`) and internally copy memory corresponding to that string
576to a buffer.
577
578WASI implements system interfaces with host functions. Concretely, to write to console, a WASI command `Module` imports
579"fd_write" from "wasi_snapshot_preview1" and calls it with the `fd` parameter set to 1 (STDOUT).
580
581The [snapshot-01](https://github.com/WebAssembly/WASI/blob/snapshot-01/phases/snapshot/docs.md) version of WASI has no
582means to declare configuration, although its function definitions imply configuration for example if fd 1 should exist,
583and if so where should it write. Moreover, snapshot-01 was last updated in late 2020 and the specification is being
584completely rewritten as of early 2022. This means WASI as defined by "snapshot-01" will not clarify aspects like which
585file descriptors are required. While it is possible a subsequent version may, it is too early to tell as no version of
586WASI has reached a stage near W3C recommendation. Even if it did, module authors are not required to only use WASI to
587write to console, as they can define their own host functions, such as they did before WASI existed.
588
589wazero aims to serve Go developers as a primary function, and help them transition between WASI specifications. In
590order to do this, we have to allow top-level configuration. To ensure isolation by default, `ModuleConfig` has WithXXX
591that override defaults to no-op or empty. One `ModuleConfig` instance is used regardless of how many times the same WASI
592functions are imported. The nil defaults allow safe concurrency in these situations, as well lower the cost when they
593are never used. Finally, a one-to-one mapping with `Module` allows the module to close the `ModuleConfig` instead of
594confusing users with another API to close.
595
596Naming, defaults and validation rules of aspects like `STDIN` and `Environ` are intentionally similar to other Go
597libraries such as `exec.Cmd` or `syscall.SetEnv`, and differences called out where helpful. For example, there's no goal
598to emulate any operating system primitive specific to Windows (such as a 'c:\' drive). Moreover, certain defaults
599working with real system calls are neither relevant nor safe to inherit: For example, `exec.Cmd` defaults to read STDIN
600from a real file descriptor ("/dev/null"). Defaulting to this, vs reading `io.EOF`, would be unsafe as it can exhaust
601file descriptors if resources aren't managed properly. In other words, blind copying of defaults isn't wise as it can
602violate isolation or endanger the embedding process. In summary, we try to be similar to normal Go code, but often need
603act differently and document `ModuleConfig` is more about emulating, not necessarily performing real system calls.
604
605## File systems
606
607### Motivation on `sys.FS`
608
609The `sys.FS` abstraction in wazero was created because of limitations in
610`fs.FS`, and `fs.File` in Go. Compilers targeting `wasip1` may access
611functionality that writes new files. The ability to overcome this was requested
612even before wazero was named this, via issue #21 in March 2021.
613
614A month later, golang/go#45757 was raised by someone else on the same topic. As
615of July 2023, this has not resolved to a writeable file system abstraction.
616
617Over the next year more use cases accumulated, consolidated in March 2022 into
618#390. This closed in January 2023 with a milestone of providing more
619functionality, limited to users giving a real directory. This didn't yet expose
620a file abstraction for general purpose use. Internally, this used `os.File`.
621However, a wasm module instance is a virtual machine. Only supporting `os.File`
622breaks sand-boxing use cases. Moreover, `os.File` is not an interface. Even
623though this abstracts functionality, it does allow interception use cases.
624
625Hence, a few days later in January 2023, we had more issues asking to expose an
626abstraction, #1013 and later #1532, on use cases like masking access to files.
627In other words, the use case requests never stopped, and aren't solved by
628exposing only real files.
629
630In summary, the primary motivation for exposing a replacement for `fs.FS` and
631`fs.File` was around repetitive use case requests for years, around
632interception and the ability to create new files, both virtual and real files.
633While some use cases are solved with real files, not all are. Regardless, an
634interface approach is necessary to ensure users can intercept I/O operations.
635
636### Why doesn't `sys.File` have a `Fd()` method?
637
638There are many features we could expose. We could make File expose underlying
639file descriptors in case they are supported, for integration of system calls
640that accept multiple ones, namely `poll` for multiplexing. This special case is
641described in a subsequent section.
642
643As noted above, users have been asking for a file abstraction for over two
644years, and a common answer was to wait. Making users wait is a problem,
645especially so long. Good reasons to make people wait are stabilization. Edge
646case features are not a great reason to hold abstractions from users.
647
648Another reason is implementation difficulty. Go did not attempt to abstract
649file descriptors. For example, unlike `fs.ReadFile` there is no `fs.FdFile`
650interface. Most likely, this is because file descriptors are an implementation
651detail of common features. Programming languages, including Go, do not require
652end users to know about file descriptors. Types such as `fs.File` can be used
653without any knowledge of them. Implementations may or may not have file
654descriptors. For example, in Go, `os.DirFS` has underlying file descriptors
655while `embed.FS` does not.
656
657Despite this, some may want to expose a non-standard interface because
658`os.File` has `Fd() uintptr` to return a file descriptor. Mainly, this is
659handy to integrate with `syscall` package functions (on `GOOS` values that
660declare them). Notice, though that `uintptr` is unsafe and not an abstraction.
661Close inspection will find some `os.File` types internally use `poll.FD`
662instead, yet this is not possible to use abstractly because that type is not
663exposed. For example, `plan9` uses a different type than `poll.FD`. In other
664words, even in real files, `Fd()` is not wholly portable, despite it being
665useful on many operating systems with the `syscall` package.
666
667The reasons above, why Go doesn't abstract `FdFile` interface are a subset of
668reasons why `sys.File` does not. If we exposed `File.Fd()` we not only would
669have to declare all the edge cases that Go describes including impact of
670finalizers, we would have to describe these in terms of virtualized files.
671Then, we would have to reason with this value vs our existing virtualized
672`sys.FileTable`, mapping whatever type we return to keys in that table, also
673in consideration of garbage collection impact. The combination of issues like
674this could lead down a path of not implementing a file system abstraction at
675all, and instead a weak key mapped abstraction of the `syscall` package. Once
676we finished with all the edge cases, we would have lost context of the original
677reason why we started.. simply to allow file write access!
678
679When wazero attempts to do more than what the Go programming language team, it
680has to be carefully evaluated, to:
681* Be possible to implement at least for `os.File` backed files
682* Not be confusing or cognitively hard for virtual file systems and normal use.
683* Affordable: custom code is solely the responsible by the core team, a much
684 smaller group of individuals than who maintain the Go programming language.
685
686Due to problems well known in Go, consideration of the end users who constantly
687ask for basic file system functionality, and the difficulty virtualizing file
688descriptors at multiple levels, we don't expose `Fd()` and likely won't ever
689expose `Fd()` on `sys.File`.
690
691### Why does `sys.File` have a `Poll()` method, while `sys.FS` does not?
692
693wazero exposes `File.Poll` which allows one-at-a-time poll use cases,
694requested by multiple users. This not only includes abstract tests such as
695Go 1.21 `GOOS=wasip1`, but real use cases including python and container2wasm
696repls, as well listen sockets. The main use cases is non-blocking poll on a
697single file. Being a single file, this has no risk of problems such as
698head-of-line blocking, even when emulated.
699
700The main use case of multi-poll are bidirectional network services, something
701not used in `GOOS=wasip1` standard libraries, but could be in the future.
702Moving forward without a multi-poller allows wazero to expose its file system
703abstraction instead of continuing to hold back it back for edge cases. We'll
704continue discussion below regardless, as rationale was requested.
705
706You can loop through multiple `sys.File`, using `File.Poll` to see if an event
707is ready, but there is a head-of-line blocking problem. If a long timeout is
708used, bad luck could have a file that has nothing to read or write before one
709that does. This could cause more blocking than necessary, even if you could
710poll the others just after with a zero timeout. What's worse than this is if
711unlimited blocking was used (`timeout=-1`). The host implementations could use
712goroutines to avoid this, but interrupting a "forever" poll is problematic. All
713of these are reasons to consider a multi-poll API, but do not require exporting
714`File.Fd()`.
715
716Should multi-poll becomes critical, `sys.FS` could expose a `Poll` function
717like below, despite it being the non-portable, complicated if possible to
718implement on all platforms and virtual file systems.
719```go
720ready, errno := fs.Poll([]sys.PollFile{{f1, sys.POLLIN}, {f2, sys.POLLOUT}}, timeoutMillis)
721```
722
723A real filesystem could handle this by using an approach like the internal
724`unix.Poll` function in Go, passing file descriptors on unix platforms, or
725returning `sys.ENOSYS` for unsupported operating systems. Implementation for
726virtual files could have a strategy around timeout to avoid the worst case of
727head-of-line blocking (unlimited timeout).
728
729Let's remember that when designing abstractions, it is not best to add an
730interface for everything. Certainly, Go doesn't, as evidenced by them not
731exposing `poll.FD` in `os.File`! Such a multi-poll could be limited to
732built-in filesystems in the wazero repository, avoiding complexity of trying to
733support and test this abstractly. This would still permit multiplexing for CLI
734users, and also permit single file polling as exists now.
735
736### Why doesn't wazero implement the working directory?
737
738An early design of wazero's API included a `WithWorkDirFS` which allowed
739control over which file a relative path such as "./config.yml" resolved to,
740independent of the root file system. This intended to help separate concerns
741like mutability of files, but it didn't work and was removed.
742
743Compilers that target wasm act differently with regard to the working
744directory. For example, wasi-libc, used by TinyGo,
745tracks working directory changes in compiled wasm instead: initially "/" until
746code calls `chdir`. Zig assumes the first pre-opened file descriptor is the
747working directory.
748
749The only place wazero can standardize a layered concern is via a host function.
750Since WASI doesn't use host functions to track the working directory, we can't
751standardize the storage and initial value of it.
752
753Meanwhile, code may be able to affect the working directory by compiling
754`chdir` into their main function, using an argument or ENV for the initial
755value (possibly `PWD`). Those unable to control the compiled code should only
756use absolute paths in configuration.
757
758See
759* https://github.com/golang/go/blob/go1.20/src/syscall/fs_js.go#L324
760* https://github.com/WebAssembly/wasi-libc/pull/214#issue-673090117
761* https://github.com/ziglang/zig/blob/53a9ee699a35a3d245ab6d1dac1f0687a4dcb42c/src/main.zig#L32
762
763### Why ignore the error returned by io.Reader when n > 1?
764
765Per https://pkg.go.dev/io#Reader, if we receive an error, any bytes read should
766be processed first. At the syscall abstraction (`fd_read`), the caller is the
767processor, so we can't process the bytes inline and also return the error (as
768`EIO`).
769
770Let's assume we want to return the bytes read on error to the caller. This
771implies we at least temporarily ignore the error alongside them. The choice
772remaining is whether to persist the error returned with the read until a
773possible next call, or ignore the error.
774
775If we persist an error returned, it would be coupled to a file descriptor, but
776effectively it is boolean as this case coerces to `EIO`. If we track a "last
777error" on a file descriptor, it could be complicated for a couple reasons
778including whether the error is transient or permanent, or if the error would
779apply to any FD operation, or just read. Finally, there may never be a
780subsequent read as perhaps the bytes leading up to the error are enough to
781satisfy the processor.
782
783This decision boils down to whether or not to track an error bit per file
784descriptor or not. If not, the assumption is that a subsequent operation would
785also error, this time without reading any bytes.
786
787The current opinion is to go with the simplest path, which is to return the
788bytes read and ignore the error the there were any. Assume a subsequent
789operation will err if it needs to. This helps reduce the complexity of the code
790in wazero and also accommodates the scenario where the bytes read are enough to
791satisfy its processor.
792
793### File descriptor allocation strategy
794
795File descriptor allocation currently uses a strategy similar the one implemented
796by unix systems: when opening a file, the lowest unused number is picked.
797
798The WASI standard documents that programs cannot expect that file descriptor
799numbers will be allocated with a lowest-first strategy, and they should instead
800assume the values will be random. Since _random_ is a very imprecise concept in
801computers, we technically satisfying the implementation with the descriptor
802allocation strategy we use in Wazero. We could imagine adding more _randomness_
803to the descriptor selection process, however this should never be used as a
804security measure to prevent applications from guessing the next file number so
805there are no strong incentives to complicate the logic.
806
807### Why does `FSConfig.WithDirMount` not match behaviour with `os.DirFS`?
808
809It may seem that we should require any feature that seems like a standard
810library in Go, to behave the same way as the standard library. Doing so would
811present least surprise to Go developers. In the case of how we handle
812filesystems, we break from that as it is incompatible with the expectations of
813WASI, the most commonly implemented filesystem ABI.
814
815The main reason is that `os.DirFS` is a virtual filesystem abstraction while
816WASI is an abstraction over syscalls. For example, the signature of `fs.Open`
817does not permit use of flags. This creates conflict on what default behaviors
818to take when Go implemented `os.DirFS`. On the other hand, `path_open` can pass
819flags, and in fact tests require them to be honored in specific ways.
820
821This conflict requires us to choose what to be more compatible with, and which
822type of user to surprise the least. We assume there will be more developers
823compiling code to wasm than developers of custom filesystem plugins, and those
824compiling code to wasm will be better served if we are compatible with WASI.
825Hence on conflict, we prefer WASI behavior vs the behavior of `os.DirFS`.
826
827See https://github.com/WebAssembly/wasi-testsuite
828See https://github.com/golang/go/issues/58141
829
830## Why is our `Readdir` function more like Go's `os.File` than POSIX `readdir`?
831
832At one point we attempted to move from a bulk `Readdir` function to something
833more like the POSIX `DIR` struct, exposing functions like `telldir`, `seekdir`
834and `readdir`. However, we chose the design more like `os.File.Readdir`,
835because it performs and fits wasip1 better.
836
837### wasip1/wasix
838
839`fd_readdir` in wasip1 (and so also wasix) is like `getdents` in Linux, not
840`readdir` in POSIX. `getdents` is more like Go's `os.File.Readdir`.
841
842We currently have an internal type `sys.DirentCache` which only is used by
843wasip1 or wasix. When `HostModuleBuilder` adds support for instantiation state,
844we could move this to the `wasi_snapshot_preview1` package. Meanwhile, all
845filesystem code is internal anyway, so this special-case is acceptable.
846
847### wasip2
848
849`directory-entry-stream` in wasi-filesystem preview2 is defined in component
850model, not an ABI, but in wasmtime it is a consuming iterator. A consuming
851iterator is easy to support with anything (like `Readdir(1)`), even if it is
852inefficient as you can neither bulk read nor skip. The implementation of the
853preview1 adapter (uses preview2) confirms this. They use a dirent cache similar
854in some ways to our `sysfs.DirentCache`. As there is no seek concept in
855preview2, they interpret the cookie as numeric and read on repeat entries when
856a cache wasn't available. Note: we currently do not skip-read like this as it
857risks buffering large directories, and no user has requested entries before the
858cache, yet.
859
860Regardless, wasip2 is not complete until the end of 2023. We can defer design
861discussion until after it is stable and after the reference impl wasmtime
862implements it.
863
864See
865 * https://github.com/WebAssembly/wasi-filesystem/blob/ef9fc87c07323a6827632edeb6a7388b31266c8e/example-world.md#directory_entry_stream
866 * https://github.com/bytecodealliance/wasmtime/blob/b741f7c79d72492d17ab8a29c8ffe4687715938e/crates/wasi/src/preview2/preview2/filesystem.rs#L286-L296
867 * https://github.com/bytecodealliance/preview2-prototyping/blob/e4c04bcfbd11c42c27c28984948d501a3e168121/crates/wasi-preview1-component-adapter/src/lib.rs#L2131-L2137
868 * https://github.com/bytecodealliance/preview2-prototyping/blob/e4c04bcfbd11c42c27c28984948d501a3e168121/crates/wasi-preview1-component-adapter/src/lib.rs#L936
869
870### wasip3
871
872`directory-entry-stream` is documented to change significantly in wasip3 moving
873from synchronous to synchronous streams. This is dramatically different than
874POSIX `readdir` which is synchronous.
875
876Regardless, wasip3 is not complete until after wasip2, which means 2024 or
877later. We can defer design discussion until after it is stable and after the
878reference impl wasmtime implements it.
879
880See
881 * https://github.com/WebAssembly/WASI/blob/ddfe3d1dda5d1473f37ecebc552ae20ce5fd319a/docs/WitInWasi.md#Streams
882 * https://docs.google.com/presentation/d/1MNVOZ8hdofO3tI0szg_i-Yoy0N2QPU2C--LzVuoGSlE/edit#slide=id.g1270ef7d5b6_0_662
883
884### How do we implement `Pread` with an `fs.File`?
885
886`ReadAt` is the Go equivalent to `pread`: it does not affect, and is not
887affected by, the underlying file offset. Unfortunately, `io.ReaderAt` is not
888implemented by all `fs.File`. For example, as of Go 1.19, `embed.openFile` does
889not.
890
891The initial implementation of `fd_pread` instead used `Seek`. To avoid a
892regression, we fall back to `io.Seeker` when `io.ReaderAt` is not supported.
893
894This requires obtaining the initial file offset, seeking to the intended read
895offset, and resetting the file offset the initial state. If this final seek
896fails, the file offset is left in an undefined state. This is not thread-safe.
897
898While seeking per read seems expensive, the common case of `embed.openFile` is
899only accessing a single int64 field, which is cheap.
900
901### Pre-opened files
902
903WASI includes `fd_prestat_get` and `fd_prestat_dir_name` functions used to
904learn any directory paths for file descriptors open at initialization time.
905
906For example, `__wasilibc_register_preopened_fd` scans any file descriptors past
907STDERR (1) and invokes `fd_prestat_dir_name` to learn any path prefixes they
908correspond to. Zig's `preopensAlloc` does similar. These pre-open functions are
909not used again after initialization.
910
911wazero supports stdio pre-opens followed by any mounts e.g `.:/`. The guest
912path is a directory and its name, e.g. "/" is returned by `fd_prestat_dir_name`
913for file descriptor 3 (STDERR+1). The first longest match wins on multiple
914pre-opens, which allows a path like "/tmp" to match regardless of order vs "/".
915
916See
917 * https://github.com/WebAssembly/wasi-libc/blob/a02298043ff551ce1157bc2ee7ab74c3bffe7144/libc-bottom-half/sources/preopens.c
918 * https://github.com/ziglang/zig/blob/9cb06f3b8bf9ea6b5e5307711bc97328762d6a1d/lib/std/fs/wasi.zig#L50-L53
919
920### fd_prestat_dir_name
921
922`fd_prestat_dir_name` is a WASI function to return the path of the pre-opened
923directory of a file descriptor. It has the following three parameters, and the
924third `path_len` has ambiguous semantics.
925
926* `fd`: a file descriptor
927* `path`: the offset for the result path
928* `path_len`: In wazero, `FdPrestatDirName` writes the result path string to
929 `path` offset for the exact length of `path_len`.
930
931Wasmer considers `path_len` to be the maximum length instead of the exact
932length that should be written.
933See https://github.com/wasmerio/wasmer/blob/3463c51268ed551933392a4063bd4f8e7498b0f6/lib/wasi/src/syscalls/mod.rs#L764
934
935The semantics in wazero follows that of wasmtime.
936See https://github.com/bytecodealliance/wasmtime/blob/2ca01ae9478f199337cf743a6ab543e8c3f3b238/crates/wasi-common/src/snapshots/preview_1.rs#L578-L582
937
938Their semantics match when `path_len` == the length of `path`, so in practice
939this difference won't matter match.
940
941## fd_readdir
942
943### Why does "wasi_snapshot_preview1" require dot entries when POSIX does not?
944
945In October 2019, WASI project knew requiring dot entries ("." and "..") was not
946documented in preview1, not required by POSIX and problematic to synthesize.
947For example, Windows runtimes backed by `FindNextFileW` could not return these.
948A year later, the tag representing WASI preview 1 (`snapshot-01`) was made.
949This did not include the requested change of making dot entries optional.
950
951The `phases/snapshot/docs.md` document was altered in subsequent years in
952significant ways, often in lock-step with wasmtime or wasi-libc. In January
9532022, `sock_accept` was added to `phases/snapshot/docs.md`, a document later
954renamed to later renamed to `legacy/preview1/docs.md`.
955
956As a result, the ABI and behavior remained unstable: The `snapshot-01` tag was
957not an effective basis of portability. A test suite was requested well before
958this tag, in April 2019. Meanwhile, compliance had no meaning. Developers had
959to track changes to the latest doc, while clarifying with wasi-libc or wasmtime
960behavior. This lack of stability could have permitted a fix to the dot entries
961problem, just as it permitted changes desired by other users.
962
963In November 2022, the wasi-testsuite project began and started solidifying
964expectations. This quickly led to changes in runtimes and the spec doc. WASI
965began importing tests from wasmtime as required behaviors for all runtimes.
966Some changes implied changes to wasi-libc. For example, `readdir` began to
967imply inode fan-outs, which caused performance regressions. Most notably a
968test merged in January required dot entries. Tests were merged without running
969against any runtime, and even when run ad-hoc only against Linux. Hence,
970portability issues mentioned over three years earlier did not trigger any
971failure until wazero (which tests Windows) noticed.
972
973In the same month, wazero requested to revert this change primarily because
974Go does not return them from `os.ReadDir`, and materializing them is
975complicated due to tests also requiring inodes. Moreover, they are discarded by
976not just Go, but other common programming languages. This was rejected by the
977WASI lead for preview1, but considered for the completely different ABI named
978preview2.
979
980In February 2023, the WASI chair declared that new rule requiring preview1 to
981return dot entries "was decided by the subgroup as a whole", citing meeting
982notes. According to these notes, the WASI lead stated incorrectly that POSIX
983conformance required returning dot entries, something it explicitly says are
984optional. In other words, he said filtering them out would make Preview1
985non-conforming, and asked if anyone objects to this. The co-chair was noted to
986say "Because there are existing P1 programs, we shouldn’t make changes like
987this." No other were recorded to say anything.
988
989In summary, preview1 was changed retrospectively to require dot entries and
990preview2 was changed to require their absence. This rule was reverse engineered
991from wasmtime tests, and affirmed on two false premises:
992
993* POSIX compliance requires dot entries
994 * POSIX literally says these are optional
995* WASI cannot make changes because there are existing P1 programs.
996 * Changes to Preview 1 happened before and after this topic.
997
998As of June 2023, wasi-testsuite still only runs on Linux, so compliance of this
999rule on Windows is left to runtimes to decide to validate. The preview2 adapter
1000uses fake cookies zero and one to refer to dot dirents, uses a real inode for
1001the dot(".") entry and zero inode for dot-dot("..").
1002
1003See https://github.com/WebAssembly/wasi-filesystem/issues/3
1004See https://github.com/WebAssembly/WASI/tree/snapshot-01
1005See https://github.com/WebAssembly/WASI/issues/9
1006See https://github.com/WebAssembly/WASI/pull/458
1007See https://github.com/WebAssembly/wasi-testsuite/pull/32
1008See https://github.com/WebAssembly/wasi-libc/pull/345
1009See https://github.com/WebAssembly/wasi-testsuite/issues/52
1010See https://github.com/WebAssembly/WASI/pull/516
1011See https://github.com/WebAssembly/meetings/blob/main/wasi/2023/WASI-02-09.md#should-preview1-fd_readdir-filter-out--and-
1012See https://github.com/bytecodealliance/preview2-prototyping/blob/e4c04bcfbd11c42c27c28984948d501a3e168121/crates/wasi-preview1-component-adapter/src/lib.rs#L1026-L1041
1013
1014### Why are dot (".") and dot-dot ("..") entries problematic?
1015
1016When reading a directory, dot (".") and dot-dot ("..") entries are problematic.
1017For example, Go does not return them from `os.ReadDir`, and materializing them
1018is complicated (at least dot-dot is).
1019
1020A directory entry has stat information in it. The stat information includes
1021inode which is used for comparing file equivalence. In the simple case of dot,
1022we could materialize a special entry to expose the same info as stat on the fd
1023would return. However, doing this and not doing dot-dot would cause confusion,
1024and dot-dot is far more tricky. To back-fill inode information about a parent
1025directory would be costly and subtle. For example, the pre-open (mount) of the
1026directory may be different than its logical parent. This is easy to understand
1027when considering the common case of mounting "/" and "/tmp" as pre-opens. To
1028implement ".." from "/tmp" requires information from a separate pre-open, this
1029includes state to even know the difference. There are easier edge cases as
1030well, such as the decision to not return ".." from a root path. In any case,
1031this should start to explain that faking entries when underlying stdlib doesn't
1032return them is tricky and requires quite a lot of state.
1033
1034Another issue is around the `Dirent.Off` value of a directory entry, sometimes
1035called a "cookie" in Linux man pagers. When the host operating system or
1036library function does not return dot entries, to support functions such as
1037`seekdir`, you still need a value for `Dirent.Off`. Naively, you can synthesize
1038these by choosing sequential offsets zero and one. However, POSIX strictly says
1039offsets should be treated opaquely. The backing filesystem could use these to
1040represent real entries. For example, a directory with one entry could use zero
1041as the `Dirent.Off` value. If you also used zero for the "." dirent, there
1042would be a clash. This means if you synthesize `Dirent.Off` for any entry, you
1043need to synthesize this value for all entries. In practice, the simplest way is
1044using an incrementing number, such as done in the WASI preview2 adapter.
1045
1046Working around these issues causes expense to all users of wazero, so we'd
1047then look to see if that would be justified or not. However, the most common
1048compilers involved in end user questions, as of early 2023 are TinyGo, Rust and
1049Zig. All of these compile code which ignores dot and dot-dot entries. In other
1050words, faking these entries would not only cost our codebase with complexity,
1051but it would also add unnecessary overhead as the values aren't commonly used.
1052
1053The final reason why we might do this, is an end users or a specification
1054requiring us to. As of early 2023, no end user has raised concern over Go and
1055by extension wazero not returning dot and dot-dot. The snapshot-01 spec of WASI
1056does not mention anything on this point. Also, POSIX has the following to say,
1057which summarizes to "these are optional"
1058
1059> The readdir() function shall not return directory entries containing empty names. If entries for dot or dot-dot exist, one entry shall be returned for dot and one entry shall be returned for dot-dot; otherwise, they shall not be returned.
1060
1061Unfortunately, as described above, the WASI project decided in early 2023 to
1062require dot entries in both the spec and the wasi-testsuite. For only this
1063reason, wazero adds overhead to synthesize dot entries despite it being
1064unnecessary for most users.
1065
1066See https://pubs.opengroup.org/onlinepubs/9699919799/functions/readdir.html
1067See https://github.com/golang/go/blob/go1.20/src/os/dir_unix.go#L108-L110
1068See https://github.com/bytecodealliance/preview2-prototyping/blob/e4c04bcfbd11c42c27c28984948d501a3e168121/crates/wasi-preview1-component-adapter/src/lib.rs#L1026-L1041
1069
1070### Why don't we pre-populate an inode for the dot-dot ("..") entry?
1071
1072We only populate an inode for dot (".") because wasi-testsuite requires it, and
1073we likely already have it (because we cache it). We could attempt to populate
1074one for dot-dot (".."), but chose not to.
1075
1076Firstly, wasi-testsuite does not require the inode of dot-dot, possibly because
1077the wasip2 adapter doesn't populate it (but we don't really know why).
1078
1079The only other reason to populate it would be to avoid wasi-libc's stat fanout
1080when it is missing. However, wasi-libc explicitly doesn't fan-out to lstat on
1081the ".." entry on a zero ino.
1082
1083Fetching dot-dot's inode despite the above not only doesn't help wasi-libc, but
1084it also hurts languages that don't use it, such as Go. These languages would
1085pay a stat syscall penalty even if they don't need the inode. In fact, Go
1086discards both dot entries!
1087
1088In summary, there are no significant upsides in attempting to pre-fetch
1089dot-dot's inode, and there are downsides to doing it anyway.
1090
1091See
1092 * https://github.com/WebAssembly/wasi-libc/blob/bd950eb128bff337153de217b11270f948d04bb4/libc-bottom-half/cloudlibc/src/libc/dirent/readdir.c#L87-L94
1093 * https://github.com/WebAssembly/wasi-testsuite/blob/main/tests/rust/src/bin/fd_readdir.rs#L108
1094 * https://github.com/bytecodealliance/preview2-prototyping/blob/e4c04bcfbd11c42c27c28984948d501a3e168121/crates/wasi-preview1-component-adapter/src/lib.rs#L1037
1095
1096### Why don't we require inodes to be non-zero?
1097
1098We don't require a non-zero value for `Dirent.Ino` because doing so can prevent
1099a real one from resolving later via `Stat_t.Ino`.
1100
1101We define `Ino` like `d_ino` in POSIX which doesn't special-case zero. It can
1102be zero for a few reasons:
1103
1104* The file is not a regular file or directory.
1105* The underlying filesystem does not support inodes. e.g. embed:fs
1106* A directory doesn't include inodes, but a later stat can. e.g. Windows
1107* The backend is based on wasi-filesystem (a.k.a wasip2), which has
1108 `directory_entry.inode` optional, and might remove it entirely.
1109
1110There are other downsides to returning a zero inode in widely used compilers:
1111
1112* File equivalence utilities, like `os.SameFile` will not work.
1113* wasi-libc's `wasip1` mode will call `lstat` and attempt to retrieve a
1114 non-zero value (unless the entry is named "..").
1115
1116A new compiler may accidentally skip a `Dirent` with a zero `Ino` if emulating
1117a non-POSIX function and re-using `Dirent.Ino` for `d_fileno`.
1118
1119* Linux `getdents` doesn't define `d_fileno` must be non-zero
1120* BSD `getdirentries` is implementation specific. For example, OpenBSD will
1121 return dirents with a zero `d_fileno`, but Darwin will skip them.
1122
1123The above shouldn't be a problem, even in the case of BSD, because `wasip1` is
1124defined more in terms of `getdents` than `getdirentries`. The bottom half of
1125either should treat `wasip1` (or any similar ABI such as wasix or wasip2) as a
1126different operating system and either use different logic that doesn't skip, or
1127synthesize a fake non-zero `d_fileno` when `d_ino` is zero.
1128
1129However, this has been a problem. Go's `syscall.ParseDirent` utility is shared
1130for all `GOOS=unix`. For simplicity, this abstracts `direntIno` with data from
1131`d_fileno` or `d_ino`, and drops if either are zero, even if `d_fileno` is the
1132only field with zero explicitly defined. This led to a change to special case
1133`GOOS=wasip1` as otherwise virtual files would be unconditionally skipped.
1134
1135In practice, this problem is rather unique due to so many compilers relying on
1136wasi-libc, which tolerates a zero inode. For example, while issues were
1137reported about the performance regression when wasi-libc began doing a fan-out
1138on zero `Dirent.Ino`, no issues were reported about dirents being dropped as a
1139result.
1140
1141In summary, rather than complicating implementation and forcing non-zero inodes
1142for a rare case, we permit zero. We instead document this topic thoroughly, so
1143that emerging compilers can re-use the research and reference it on conflict.
1144We also document that `Ino` should be non-zero, so that users implementing that
1145field will attempt to get it.
1146
1147See
1148 * https://github.com/WebAssembly/wasi-filesystem/pull/81
1149 * https://github.com/WebAssembly/wasi-libc/blob/bd950eb128bff337153de217b11270f948d04bb4/libc-bottom-half/cloudlibc/src/libc/dirent/readdir.c#L87-L94
1150 * https://linux.die.net/man/3/getdents
1151 * https://www.unix.com/man-page/osx/2/getdirentries/
1152 * https://man.openbsd.org/OpenBSD-5.4/getdirentries.2
1153 * https://github.com/golang/go/blob/go1.20/src/syscall/dirent.go#L60-L102
1154 * https://go-review.googlesource.com/c/go/+/507915
1155
1156## sys.Walltime and Nanotime
1157
1158The `sys` package has two function types, `Walltime` and `Nanotime` for real
1159and monotonic clock exports. The naming matches conventions used in Go.
1160
1161```go
1162func time_now() (sec int64, nsec int32, mono int64) {
1163 sec, nsec = walltime()
1164 return sec, nsec, nanotime()
1165}
1166```
1167
1168Splitting functions for wall and clock time allow implementations to choose
1169whether to implement the clock once (as in Go), or split them out.
1170
1171Each can be configured with a `ClockResolution`, although is it usually
1172incorrect as detailed in a sub-heading below. The only reason for exposing this
1173is to satisfy WASI:
1174
1175See https://github.com/WebAssembly/wasi-clocks
1176
1177### Why default to fake time?
1178
1179WebAssembly has an implicit design pattern of capabilities based security. By
1180defaulting to a fake time, we reduce the chance of timing attacks, at the cost
1181of requiring configuration to opt-into real clocks.
1182
1183See https://gruss.cc/files/fantastictimers.pdf for an example attacks.
1184
1185### Why does fake time increase on reading?
1186
1187Both the fake nanotime and walltime increase by 1ms on reading. Particularly in
1188the case of nanotime, this prevents spinning.
1189
1190### Why not `time.Clock`?
1191
1192wazero can't use `time.Clock` as a plugin for clock implementation as it is
1193only substitutable with build flags (`faketime`) and conflates wall and
1194monotonic time in the same call.
1195
1196Go's `time.Clock` was added monotonic time after the fact. For portability with
1197prior APIs, a decision was made to combine readings into the same API call.
1198
1199See https://go.googlesource.com/proposal/+/master/design/12914-monotonic.md
1200
1201WebAssembly time imports do not have the same concern. In fact even Go's
1202imports for clocks split walltime from nanotime readings.
1203
1204See https://github.com/golang/go/blob/go1.20/misc/wasm/wasm_exec.js#L243-L255
1205
1206Finally, Go's clock is not an interface. WebAssembly users who want determinism
1207or security need to be able to substitute an alternative clock implementation
1208from the host process one.
1209
1210### `ClockResolution`
1211
1212A clock's resolution is hardware and OS dependent so requires a system call to retrieve an accurate value.
1213Go does not provide a function for getting resolution, so without CGO we don't have an easy way to get an actual
1214value. For now, we return fixed values of 1us for realtime and 1ns for monotonic, assuming that realtime clocks are
1215often lower precision than monotonic clocks. In the future, this could be improved by having OS+arch specific assembly
1216to make syscalls.
1217
1218For example, Go implements time.Now for linux-amd64 with this [assembly](https://github.com/golang/go/blob/go1.20/src/runtime/time_linux_amd64.s).
1219Because retrieving resolution is not generally called often, unlike getting time, it could be appropriate to only
1220implement the fallback logic that does not use VDSO (executing syscalls in user mode). The syscall for clock_getres
1221is 229 and should be usable. https://pkg.go.dev/syscall#pkg-constants.
1222
1223If implementing similar for Windows, [mingw](https://github.com/mirror/mingw-w64/blob/6a0e9165008f731bccadfc41a59719cf7c8efc02/mingw-w64-libraries/winpthreads/src/clock.c#L77
1224) is often a good source to find the Windows API calls that correspond
1225to a POSIX method.
1226
1227Writing assembly would allow making syscalls without CGO, but comes with the cost that it will require implementations
1228across many combinations of OS and architecture.
1229
1230## sys.Nanosleep
1231
1232All major programming languages have a `sleep` mechanism to block for a
1233duration. Sleep is typically implemented by a WASI `poll_oneoff` relative clock
1234subscription.
1235
1236For example, the below ends up calling `wasi_snapshot_preview1.poll_oneoff`:
1237
1238```zig
1239const std = @import("std");
1240pub fn main() !void {
1241 std.time.sleep(std.time.ns_per_s * 5);
1242}
1243```
1244
1245Besides Zig, this is also the case with TinyGo (`-target=wasi`) and Rust
1246(`--target wasm32-wasi`).
1247
1248We decided to expose `sys.Nanosleep` to allow overriding the implementation
1249used in the common case, even if it isn't used by Go, because this gives an
1250easy and efficient closure over a common program function. We also documented
1251`sys.Nanotime` to warn users that some compilers don't optimize sleep.
1252
1253## sys.Osyield
1254
1255We expose `sys.Osyield`, to allow users to control the behavior of WASI's
1256`sched_yield` without a new build of wazero. This is mainly for parity with
1257all other related features which we allow users to implement, including
1258`sys.Nanosleep`. Unlike others, we don't provide an out-of-box implementation
1259primarily because it will cause performance problems when accessed.
1260
1261For example, the below implementation uses CGO, which might result in a 1us
1262delay per invocation depending on the platform.
1263
1264See https://github.com/golang/go/issues/19409#issuecomment-284788196
1265```go
1266//go:noescape
1267//go:linkname osyield runtime.osyield
1268func osyield()
1269```
1270
1271In practice, a request to customize this is unlikely to happen until other
1272thread based functions are implemented. That said, as of early 2023, there are
1273a few signs of implementation interest and cross-referencing:
1274
1275See https://github.com/WebAssembly/stack-switching/discussions/38
1276See https://github.com/WebAssembly/wasi-threads#what-can-be-skipped
1277See https://slinkydeveloper.com/Kubernetes-controllers-A-New-Hope/
1278
1279## sys.Stat_t
1280
1281We expose `stat` information as `sys.Stat_t`, like `syscall.Stat_t` except
1282defined without build constraints. For example, you can use `sys.Stat_t` on
1283`GOOS=windows` which doesn't define `syscall.Stat_t`.
1284
1285The first use case of this is to return inodes from `fs.FileInfo` without
1286relying on platform-specifics. For example, a user could return `*sys.Stat_t`
1287from `info.Sys()` and define a non-zero inode for a virtual file, or map a
1288real inode to a virtual one.
1289
1290Notable choices per field are listed below, where `sys.Stat_t` is unlike
1291`syscall.Stat_t` on `GOOS=linux`, or needs clarification. One common issue
1292not repeated below is that numeric fields are 64-bit when at least one platform
1293defines it that large. Also, zero values are equivalent to nil or absent.
1294
1295* `Dev` and `Ino` (`Inode`) are both defined unsigned as they are defined
1296 opaque, and most `syscall.Stat_t` also defined them unsigned. There are
1297 separate sections in this document discussing the impact of zero in `Ino`.
1298* `Mode` is defined as a `fs.FileMode` even though that is not defined in POSIX
1299 and will not map to all possible values. This is because the current use is
1300 WASI, which doesn't define any types or features not already supported. By
1301 using `fs.FileMode`, we can re-use routine experience in Go.
1302* `NLink` is unsigned because it is defined that way in `syscall.Stat_t`: there
1303 can never be less than zero links to a file. We suggest defaulting to 1 in
1304 conversions when information is not knowable because at least that many links
1305 exist.
1306* `Size` is signed because it is defined that way in `syscall.Stat_t`: while
1307 regular files and directories will always be non-negative, irregular files
1308 are possibly negative or not defined. Notably sparse files are known to
1309 return negative values.
1310* `Atim`, `Mtim` and `Ctim` are signed because they are defined that way in
1311 `syscall.Stat_t`: Negative values are time before 1970. The resolution is
1312 nanosecond because that's the maximum resolution currently supported in Go.
1313
1314### Why do we use `sys.EpochNanos` instead of `time.Time` or similar?
1315
1316To simplify documentation, we defined a type alias `sys.EpochNanos` for int64.
1317`time.Time` is a data structure, and we could have used this for
1318`syscall.Stat_t` time values. The most important reason we do not is conversion
1319penalty deriving time from common types.
1320
1321The most common ABI used in `wasip2`. This, and compatible ABI such as `wasix`,
1322encode timestamps in memory as a 64-bit number. If we used `time.Time`, we
1323would have to convert an underlying type like `syscall.Timespec` to `time.Time`
1324only to later have to call `.UnixNano()` to convert it back to a 64-bit number.
1325
1326In the future, the component model module "wasi-filesystem" may represent stat
1327timestamps with a type shared with "wasi-clocks", abstractly structured similar
1328to `time.Time`. However, component model intentionally does not define an ABI.
1329It is likely that the canonical ABI for timestamp will be in two parts, but it
1330is not required for it to be intermediately represented this way. A utility
1331like `syscall.NsecToTimespec` could split an int64 so that it could be written
1332to memory as 96 bytes (int64, int32), without allocating a struct.
1333
1334Finally, some may confuse epoch nanoseconds with 32-bit epoch seconds. While
133532-bit epoch seconds has "The year 2038" problem, epoch nanoseconds has
1336"The Year 2262" problem, which is even less concerning for this library. If
1337the Go programming language and wazero exist in the 2200's, we can make a major
1338version increment to adjust the `sys.EpochNanos` approach. Meanwhile, we have
1339faster code.
1340
1341## poll_oneoff
1342
1343`poll_oneoff` is a WASI API for waiting for I/O events on multiple handles.
1344It is conceptually similar to the POSIX `poll(2)` syscall.
1345The name is not `poll`, because it references [“the fact that this function is not efficient
1346when used repeatedly with the same large set of handles”][poll_oneoff].
1347
1348We chose to support this API in a handful of cases that work for regular files
1349and standard input. We currently do not support other types of file descriptors such
1350as socket handles.
1351
1352### Clock Subscriptions
1353
1354As detailed above in [sys.Nanosleep](#sysnanosleep), `poll_oneoff` handles
1355relative clock subscriptions. In our implementation we use `sys.Nanosleep()`
1356for this purpose in most cases, except when polling for interactive input
1357from `os.Stdin` (see more details below).
1358
1359### FdRead and FdWrite Subscriptions
1360
1361When subscribing a file descriptor (except `Stdin`) for reads or writes,
1362the implementation will generally return immediately with success, unless
1363the file descriptor is unknown. The file descriptor is not checked further
1364for new incoming data. Any timeout is cancelled, and the API call is able
1365to return, unless there are subscriptions to `Stdin`: these are handled
1366separately.
1367
1368### FdRead and FdWrite Subscription to Stdin
1369
1370Subscribing `Stdin` for reads (writes make no sense and cause an error),
1371requires extra care: wazero allows to configure a custom reader for `Stdin`.
1372
1373In general, if a custom reader is found, the behavior will be the same
1374as for regular file descriptors: data is assumed to be present and
1375a success is written back to the result buffer.
1376
1377However, if the reader is detected to read from `os.Stdin`,
1378a special code path is followed, invoking `sysfs.poll()`.
1379
1380`sysfs.poll()` is a wrapper for `poll(2)` on POSIX systems,
1381and it is emulated on Windows.
1382
1383### Poll on POSIX
1384
1385On POSIX systems, `poll(2)` allows to wait for incoming data on a file
1386descriptor, and block until either data becomes available or the timeout
1387expires.
1388
1389Usage of `syfs.poll()` is currently only reserved for standard input, because
1390
13911. it is really only necessary to handle interactive input: otherwise,
1392 there is no way in Go to peek from Standard Input without actually
1393 reading (and thus consuming) from it;
1394
13952. if `Stdin` is connected to a pipe, it is ok in most cases to return
1396 with success immediately;
1397
13983. `syfs.poll()` is currently a blocking call, irrespective of goroutines,
1399 because the underlying syscall is; thus, it is better to limit its usage.
1400
1401So, if the subscription is for `os.Stdin` and the handle is detected
1402to correspond to an interactive session, then `sysfs.poll()` will be
1403invoked with a the `Stdin` handle *and* the timeout.
1404
1405This also means that in this specific case, the timeout is uninterruptible,
1406unless data becomes available on `Stdin` itself.
1407
1408### Select on Windows
1409
1410On Windows `sysfs.poll()` cannot be delegated to a single
1411syscall, because there is no single syscall to handle sockets,
1412pipes and regular files.
1413
1414Instead, we emulate its behavior for the cases that are currently
1415of interest.
1416
1417- For regular files, we _always_ report them as ready, as
1418[most operating systems do anyway][async-io-windows].
1419
1420- For pipes, we invoke [`PeekNamedPipe`][peeknamedpipe]
1421for each file handle we detect is a pipe open for reading.
1422We currently ignore pipes open for writing.
1423
1424- Notably, we include also support for sockets using the [WinSock
1425implementation of `poll`][wsapoll], but instead
1426of relying on the timeout argument of the `WSAPoll` function,
1427we set a 0-duration timeout so that it behaves like a peek.
1428
1429This way, we can check for regular files all at once,
1430at the beginning of the function, then we poll pipes and
1431sockets periodically using a cancellable `time.Tick`,
1432which plays nicely with the rest of the Go runtime.
1433
1434### Impact of blocking
1435
1436Because this is a blocking syscall, it will also block the carrier thread of
1437the goroutine, preventing any means to support context cancellation directly.
1438
1439There are ways to obviate this issue. We outline here one idea, that is however
1440not currently implemented. A common approach to support context cancellation is
1441to add a signal file descriptor to the set, e.g. the read-end of a pipe or an
1442eventfd on Linux. When the context is canceled, we may unblock a Select call by
1443writing to the fd, causing it to return immediately. This however requires to
1444do a bit of housekeeping to hide the "special" FD from the end-user.
1445
1446[poll_oneoff]: https://github.com/WebAssembly/wasi-poll#why-is-the-function-called-poll_oneoff
1447[async-io-windows]: https://tinyclouds.org/iocp_links
1448[peeknamedpipe]: https://learn.microsoft.com/en-us/windows/win32/api/namedpipeapi/nf-namedpipeapi-peeknamedpipe
1449[wsapoll]: https://learn.microsoft.com/en-us/windows/win32/api/winsock2/nf-winsock2-wsapoll
1450
1451## Signed encoding of integer global constant initializers
1452
1453wazero treats integer global constant initializers signed as their interpretation is not known at declaration time. For
1454example, there is no signed integer [value type](https://www.w3.org/TR/2019/REC-wasm-core-1-20191205/#value-types%E2%91%A0).
1455
1456To get at the problem, let's use an example.
1457```
1458(global (export "start_epoch") i64 (i64.const 1620216263544))
1459```
1460
1461In both signed and unsigned LEB128 encoding, this value is the same bit pattern. The problem is that some numbers are
1462not. For example, 16256 is `807f` encoded as unsigned, but `80ff00` encoded as signed.
1463
1464While the specification mentions uninterpreted integers are in abstract [unsigned values](https://www.w3.org/TR/2019/REC-wasm-core-1-20191205/#integers%E2%91%A0),
1465the binary encoding is clear that they are encoded [signed](https://www.w3.org/TR/2019/REC-wasm-core-1-20191205/#integers%E2%91%A4).
1466
1467For consistency, we go with signed encoding in the special case of global constant initializers.
1468
1469## Implementation limitations
1470
1471WebAssembly 1.0 (20191205) specification allows runtimes to [limit certain aspects of Wasm module or execution](https://www.w3.org/TR/2019/REC-wasm-core-1-20191205/#a2-implementation-limitations).
1472
1473wazero limitations are imposed pragmatically and described below.
1474
1475### Number of functions in a module
1476
1477The possible number of function instances in [a module](https://www.w3.org/TR/2019/REC-wasm-core-1-20191205/#module-instances%E2%91%A0) is not specified in the WebAssembly specifications since [`funcaddr`](https://www.w3.org/TR/2019/REC-wasm-core-1-20191205/#syntax-funcaddr) corresponding to a function instance in a store can be arbitrary number.
1478wazero limits the maximum function instances to 2^27 as even that number would occupy 1GB in function pointers.
1479
1480That is because not only we _believe_ that all use cases are fine with the limitation, but also we have no way to test wazero runtimes under these unusual circumstances.
1481
1482### Number of function types in a store
1483
1484There's no limitation on the number of function types in [a store](https://www.w3.org/TR/2019/REC-wasm-core-1-20191205/#store%E2%91%A0) according to the spec. In wazero implementation, we assign each function type to a unique ID, and choose to use `uint32` to represent the IDs.
1485Therefore the maximum number of function types a store can have is limited to 2^27 as even that number would occupy 512MB just to reference the function types.
1486
1487This is due to the same reason for the limitation on the number of functions above.
1488
1489### Number of values on the stack in a function
1490
1491While the the spec does not clarify a limitation of function stack values, wazero limits this to 2^27 = 134,217,728.
1492The reason is that we internally represent all the values as 64-bit integers regardless of its types (including f32, f64), and 2^27 values means
14931 GiB = (2^30). 1 GiB is the reasonable for most applications [as we see a Goroutine has 250 MB as a limit on the stack for 32-bit arch](https://github.com/golang/go/blob/go1.20/src/runtime/proc.go#L152-L159), considering that WebAssembly is (currently) 32-bit environment.
1494
1495All the functions are statically analyzed at module instantiation phase, and if a function can potentially reach this limit, an error is returned.
1496
1497### Number of globals in a module
1498
1499Theoretically, a module can declare globals (including imports) up to 2^32 times. However, wazero limits this to 2^27(134,217,728) per module.
1500That is because internally we store globals in a slice with pointer types (meaning 8 bytes on 64-bit platforms), and therefore 2^27 globals
1501means that we have 1 GiB size of slice which seems large enough for most applications.
1502
1503### Number of tables in a module
1504
1505While the the spec says that a module can have up to 2^32 tables, wazero limits this to 2^27 = 134,217,728.
1506One of the reasons is even that number would occupy 1GB in the pointers tables alone. Not only that, we access tables slice by
1507table index by using 32-bit signed offset in the compiler implementation, which means that the table index of 2^27 can reach 2^27 * 8 (pointer size on 64-bit machines) = 2^30 offsets in bytes.
1508
1509We _believe_ that all use cases are fine with the limitation, but also note that we have no way to test wazero runtimes under these unusual circumstances.
1510
1511If a module reaches this limit, an error is returned at the compilation phase.
1512
1513## Compiler engine implementation
1514
1515### Why it's safe to execute runtime-generated machine codes against async Goroutine preemption
1516
1517Goroutine preemption is the mechanism of the Go runtime to switch goroutines contexts on an OS thread.
1518There are two types of preemption: cooperative preemption and async preemption. The former happens, for example,
1519when making a function call, and it is not an issue for our runtime-generated functions as they do not make
1520direct function calls to Go-implemented functions. On the other hand, the latter, async preemption, can be problematic
1521since it tries to interrupt the execution of Goroutine at any point of function, and manipulates CPU register states.
1522
1523Fortunately, our runtime-generated machine codes do not need to take the async preemption into account.
1524All the assembly codes are entered via the trampoline implemented as Go Assembler Function (e.g. [arch_amd64.s](./arch_amd64.s)),
1525and as of Go 1.20, these assembler functions are considered as _unsafe_ for async preemption:
1526- https://github.com/golang/go/blob/go1.20rc1/src/runtime/preempt.go#L406-L407
1527- https://github.com/golang/go/blob/9f0234214473dfb785a5ad84a8fc62a6a395cbc3/src/runtime/traceback.go#L227
1528
1529From the Go runtime point of view, the execution of runtime-generated machine codes is considered as a part of
1530that trampoline function. Therefore, runtime-generated machine code is also correctly considered unsafe for async preemption.
1531
1532## Why context cancellation is handled in Go code rather than native code
1533
1534Since [wazero v1.0.0-pre.9](https://github.com/tetratelabs/wazero/releases/tag/v1.0.0-pre.9), the runtime
1535supports integration with Go contexts to interrupt execution after a timeout, or in response to explicit cancellation.
1536This support is internally implemented as a special opcode `builtinFunctionCheckExitCode` that triggers the execution of
1537a Go function (`ModuleInstance.FailIfClosed`) that atomically checks a sentinel value at strategic points in the code.
1538
1539[It _is indeed_ possible to check the sentinel value directly, without leaving the native world][native_check], thus sparing some cycles;
1540however, because native code never preempts (see section above), this may lead to a state where the other goroutines
1541never get the chance to run, and thus never get the chance to set the sentinel value; effectively preventing
1542cancellation from taking place.
1543
1544[native_check]: https://github.com/tetratelabs/wazero/issues/1409
1545
1546## Golang patterns
1547
1548### Hammer tests
1549Code that uses concurrency primitives, such as locks or atomics, should include "hammer tests", which run large loops
1550inside a bounded amount of goroutines, run by half that many `GOMAXPROCS`. These are named consistently "hammer", so
1551they are easy to find. The name inherits from some existing tests in [golang/go](https://github.com/golang/go/search?q=hammer&type=code).
1552
1553Here is an annotated description of the key pieces of a hammer test:
15541. `P` declares the count of goroutines to use, defaulting to 8 or 4 if `testing.Short`.
1555 * Half this amount are the cores used, and 4 is less than a modern laptop's CPU. This allows multiple "hammer" tests to run in parallel.
15562. `N` declares the scale of work (loop) per goroutine, defaulting to value that finishes in ~0.1s on a modern laptop.
1557 * When in doubt, try 1000 or 100 if `testing.Short`
1558 * Remember, there are multiple hammer tests and CI nodes are slow. Slower tests hurt feedback loops.
15593. `defer runtime.GOMAXPROCS(runtime.GOMAXPROCS(P/2))` makes goroutines switch cores, testing visibility of shared data.
15604. To ensure goroutines execute at the same time, block them with `sync.WaitGroup`, initialized to `Add(P)`.
1561 * `sync.WaitGroup` internally uses `runtime_Semacquire` not available in any other library.
1562 * `sync.WaitGroup.Add` with a negative value can unblock many goroutines at the same time, e.g. without a for loop.
15635. Track goroutines progress via `finished := make(chan int)` where each goroutine in `P` defers `finished <- 1`.
1564 1. Tests use `require.XXX`, so `recover()` into `t.Fail` in a `defer` function before `finished <- 1`.
1565 * This makes it easier to spot larger concurrency problems as you see each failure, not just the first.
1566 2. After the `defer` function, await unblocked, then run the stateful function `N` times in a normal loop.
1567 * This loop should trigger shared state problems as locks or atomics are contended by `P` goroutines.
15686. After all `P` goroutines launch, atomically release all of them with `WaitGroup.Add(-P)`.
15697. Block the runner on goroutine completion, by (`<-finished`) for each `P`.
15708. When all goroutines complete, `return` if `t.Failed()`, otherwise perform follow-up state checks.
1571
1572This is implemented in wazero in [hammer.go](internal/testing/hammer/hammer.go)
1573
1574### Lock-free, cross-goroutine observations of updates
1575
1576How to achieve cross-goroutine reads of a variable are not explicitly defined in https://go.dev/ref/mem. wazero uses
1577atomics to implement this following unofficial practice. For example, a `Close` operation can be guarded to happen only
1578once via compare-and-swap (CAS) against a zero value. When we use this pattern, we consistently use atomics to both
1579read and update the same numeric field.
1580
1581In lieu of formal documentation, we infer this pattern works from other sources (besides tests):
1582 * `sync.WaitGroup` by definition must support calling `Add` from other goroutines. Internally, it uses atomics.
1583 * rsc in golang/go#5045 writes "atomics guarantee sequential consistency among the atomic variables".
1584
1585See https://github.com/golang/go/blob/go1.20/src/sync/waitgroup.go#L64
1586See https://github.com/golang/go/issues/5045#issuecomment-252730563
1587See https://www.youtube.com/watch?v=VmrEG-3bWyM