Bosatsu Documentation

Minimal IO Functions for Basic Tools (core_alpha + Prog)

Status: proposed
Date: 2026-02-20
Issue: https://github.com/johnynek/bosatsu/issues/1744
PR: https://github.com/johnynek/bosatsu/pull/1745

Problem

core_alpha currently exposes only Bosatsu/IO/Std (print, println, stderr variants, and read_stdin_utf8_bytes).

That is enough for simple CLI output/input, but not enough for basic tool-building tasks (files, directories, process spawning, env lookup, and time/sleep).

The target API is a minimal core IO surface adapted to Bosatsu style and implemented uniformly on JVM evaluator runtime, transpiled Python runtime, and C runtime.

Decision Summary

  1. Add a new package Bosatsu/IO/Core in core_alpha containing the file/process/env/time primitives and related IO types.
  2. Keep Bosatsu/Prog as the single effect boundary; every effectful operation returns Prog[IOError, a].
  3. Keep Bosatsu/IO/Error::IOError as the shared error channel.
  4. Keep argv at the Main boundary (Main(args -> ...)) and do not duplicate it in Bosatsu/IO/Core.
  5. Represent Path as an opaque Bosatsu struct Path(to_String: String) and export path_sep + path helpers.
  6. Represent Instant/Duration as opaque Bosatsu structs backed by nanosecond Int fields.
  7. Keep Bosatsu/IO/Std source-compatible and add read_line and read_all_stdin.
  8. Do not add direct exit to Bosatsu/IO/Core; model termination with Main return codes and explicit result types.

Where in core_alpha

New package

  1. test_workspace/Bosatsu/IO/Core.bosatsu

Existing package updates

  1. test_workspace/Bosatsu/IO/Std.bosatsu: wrappers over Core (stdout, stderr, write_text, read_text, flush) plus read_line and read_all_stdin.
  2. test_workspace/core_alpha_conf.json: add Bosatsu/IO/Core to exported_packages.

Runtime implementation files

  1. JVM evaluator: core/src/main/scala/dev/bosatsu/Predef.scala
  2. Python runtime: test_workspace/ProgExt.py, test_workspace/Prog.bosatsu_externals
  3. C runtime: new c_runtime/bosatsu_ext_Bosatsu_l_IO_l_Core.c and .h, plus c_runtime/Makefile

Bosatsu API Shape (adapted)

Naming follows current stdlib style (snake_case), except where a type name appears in the identifier (string_to_Path, path_to_String). Bosatsu/IO/Core exports externals directly (no duplicated *_impl wrappers).

package Bosatsu/IO/Core

from Bosatsu/Prog import Prog
from Bosatsu/IO/Error import IOError

export (
  Path,
  path_sep,
  string_to_Path,
  path_to_String,
  path_join,
  path_parent,
  path_file_name,
  Handle,
  Process,
  Instant,
  Duration,
  duration_from_nanos,
  FileKind(),
  FileStat(),
  OpenMode(),
  Stdio(),
  StdioConfig(),
  SpawnResult(),
  StopResult(),
  stdin,
  stdout,
  stderr,
  read_text,
  write_text,
  flush,
  close,
  open_file,
  list_dir,
  stat,
  mkdir,
  remove,
  rename,
  get_env,
  spawn,
  wait,
  terminate,
  kill,
  poll,
  wait_timeout,
  with_process,
  now_wall,
  now_mono,
  sleep,
)

external path_sep: String

# Opaque in public API because constructor is not exported.
struct Path(to_String: String)

# Lift/projection helpers stay in Bosatsu so callers only depend on Path.
# `None` means the input is not a valid cross-platform path in the
# portable profile described below.
def string_to_Path(s: String) -> Option[Path]
def path_to_String(path: Path) -> String
def path_join(base: Path, child: Path) -> Path
def path_parent(path: Path) -> Option[Path]
def path_file_name(path: Path) -> Option[String]

external type Handle
external type Process

# Opaque in public API because constructors are not exported.
struct Instant(epoch_nanos: Int)
struct Duration(to_nanos: Int)

enum FileKind:
  File
  Dir
  Symlink
  Other

struct FileStat(kind: FileKind, size_bytes: Int, mtime: Instant)

enum OpenMode:
  Read
  WriteTruncate
  Append

enum Stdio:
  Inherit
  Pipe
  Null
  UseHandle(handle: Handle)

struct StdioConfig(stdin: Stdio, stdout: Stdio, stderr: Stdio)

struct SpawnResult(
  proc: Process,
  stdin: Option[Handle],
  stdout: Option[Handle],
  stderr: Option[Handle],
)

enum StopResult:
  StopSent
  AlreadyExited

external stdin: Handle
external stdout: Handle
external stderr: Handle

external def read_text(h: Handle, max_chars: Int) -> Prog[IOError, Option[String]]
external def write_text(h: Handle, s: String) -> Prog[IOError, Unit]
external def flush(h: Handle) -> Prog[IOError, Unit]
external def close(h: Handle) -> Prog[IOError, Unit]
external def open_file(path: Path, mode: OpenMode) -> Prog[IOError, Handle]
external def list_dir(path: Path) -> Prog[IOError, List[Path]]
external def stat(path: Path) -> Prog[IOError, Option[FileStat]]
external def mkdir(path: Path, recursive: Bool) -> Prog[IOError, Unit]
external def remove(path: Path, recursive: Bool) -> Prog[IOError, Unit]
external def rename(from: Path, to: Path) -> Prog[IOError, Unit]
external def get_env(name: String) -> Prog[IOError, Option[String]]
external def spawn(cmd: String, args: List[String], stdio: StdioConfig) -> Prog[IOError, SpawnResult]
external def wait(p: Process) -> Prog[IOError, Int]
external def terminate(p: Process) -> Prog[IOError, StopResult]
external def kill(p: Process) -> Prog[IOError, StopResult]
external def poll(p: Process) -> Prog[IOError, Option[Int]]
external def wait_timeout(p: Process, d: Duration) -> Prog[IOError, Option[Int]]
def with_process[e, a](
  cmd: String,
  args: List[String],
  stdio: StdioConfig,
  grace: Duration,
  on_error: IOError -> Prog[e, a],
  use: SpawnResult -> Prog[e, a],
) -> Prog[e, a]
def duration_from_nanos(nanos: Int) -> Duration
external now_wall: Prog[IOError, Instant]
external now_mono: Prog[IOError, Duration]
external def sleep(d: Duration) -> Prog[IOError, Unit]

Spawned-process lifecycle

terminate and kill operate only on the direct child represented by the Process returned from spawn. terminate requests the backend’s best available normal stop and kill requests its best available forceful stop; neither promise a graceful application shutdown. Each returns AlreadyExited after final status is recorded, or StopSent when it issues a request. These are semantic operations: Bosatsu/IO/Core exposes neither raw signals nor control of descendants, process trees, process groups, or job objects.

poll is nonblocking. wait_timeout returns None when its duration expires without consuming the eventual result. wait, poll, and wait_timeout observe the same recorded, normalized integer status after exit, so repeated wait calls return the same value.

At the low level, a spawn caller owns every handle returned for Stdio.Pipe and must close it. Lifecycle operations do not close or drain those handles. A handle supplied in Stdio.UseHandle remains caller-owned.

For ordinary scoped cleanup, use the Bosatsu-level with_process composition rather than treating it as an external operation:

from Bosatsu/Prog import Prog, raise_error
from Bosatsu/IO/Error import IOError
from Bosatsu/IO/Core import (
  Pipe,
  StdioConfig,
  SpawnResult,
  duration_from_nanos,
  wait,
  with_process,
)

def run_version() -> Prog[IOError, Int]:
  with_process(
    "tool",
    ["--version"],
    StdioConfig(Pipe, Pipe, Pipe),
    duration_from_nanos(1000000000),
    err -> raise_error(err),
    SpawnResult(proc, ...) -> wait(proc),
  )

After a successful spawn, with_process runs use; whether use succeeds or fails, it closes returned stdin first, observes or stops the direct child, waits for the grace duration before escalating to kill when necessary, attempts a final wait, then closes returned stdout and stderr. It closes only pipe handles returned in SpawnResult, never caller-supplied UseHandle resources, and does not drain output. Callers needing a custom drain or close order should keep using the low-level API.

Path representation tradeoff

  1. Chosen shape: native struct Path(to_String: String) with hidden constructor, plus path_sep and helper APIs.
  2. Advantage: callers manipulate Path values without runtime-specific wrapper allocation.
  3. Cost: runtime implementations still convert Path.to_String to platform-native path objects at IO call boundaries.
  4. Alternative considered: external type Path parsed once per value. That can reduce repeated conversion but increases runtime payload complexity and portability risk.
  5. Follow-up option: if profiling shows conversion overhead, keep the same surface API and switch internals to external type Path.

Path parsing and cross-platform behavior

  1. string_to_Path: String -> Option[Path] is intentionally partial because not every String can be used as a path on every target runtime.
  2. POSIX baseline: pathnames are slash-separated byte sequences; NUL is not allowed, slash is the separator, null pathname is invalid, and exactly two leading slashes have implementation-defined meaning.
  3. Java baseline (java.nio.file.FileSystem.getPath): parsing is implementation-dependent, uses platform path rules, and throws InvalidPathException for rejected strings (for example, NUL on UNIX).
  4. Python baseline (pathlib): PurePath parsing is lexical (no filesystem access), while concrete IO operations apply host filesystem validation later.
  5. To get deterministic behavior across macOS, Linux/Unix, and Windows, string_to_Path uses a Bosatsu-level portable parser instead of delegating directly to host-native parsing.
  6. Parsing policy:
  7. Accept separators / and \ in input; normalize stored Path.to_String to /.
  8. Accept roots in these forms: relative (a/b), POSIX absolute (/a/b), Windows drive absolute (C:/a/b), UNC (//server/share/a).
  9. Reject Windows drive-relative form (C:tmp/file) because meaning depends on per-drive current directory.
  10. Reject Windows device namespace prefixes (\\\\?\\, \\\\.\\) in v1.
  11. Reject ambiguous POSIX-like paths that start with exactly // unless they parse as UNC with non-empty server/share components.
  12. Reject strings containing NUL (\\u0000) or control characters \\u0001..\\u001F.
  13. Reject path components containing Windows-reserved characters < > : " | ? * (except the drive colon in C:/...).
  14. Reject Windows reserved device names as components (case-insensitive): CON, PRN, AUX, NUL, COM1..COM9, LPT1..LPT9 (including with extensions like NUL.txt).
  15. Reject components with trailing space or trailing dot to avoid Windows shell/API mismatch.
  16. Keep . and .. as lexical components; do not resolve symlinks or normalize away .. at parse time.
  17. Do not enforce PATH_MAX/NAME_MAX at parse time; those checks remain runtime/filesystem specific.
  18. path_join(base: Path, child: Path) -> Path stays total because both inputs are already validated Path values.
  19. Why Option[Path] instead of total String -> Path: parsing failure is expected for non-portable or malformed inputs, and callers can handle None without exceptions in pure code.
  20. Representative rejected inputs:
  21. "" (empty string)
  22. "a\u0000b"
  23. "C:tmp\\x"
  24. "foo/<bar>"
  25. "NUL.txt"
  26. "dir/ends-with-dot."
  27. Representative accepted flow:
  28. string_to_Path("src") -> Some(child)
  29. string_to_Path("/tmp") -> Some(base)
  30. path_join(base, child) -> /tmp/src
  31. path_join, path_parent, and path_file_name are lexical Bosatsu helpers (not externals); runtime-specific path handling is required only at IO call boundaries such as open_file, stat, and list_dir.
  32. References:
  33. POSIX pathname definition and resolution: https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap03.html#tag_03_271, https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/V1_chap04.html#tag_04_11
  34. Java parsing contract (FileSystem.getPath): https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/nio/file/FileSystem.html#getPath(java.lang.String,java.lang.String...)
  35. Python pathlib lexical behavior and flavor differences: https://docs.python.org/3/library/pathlib.html
  36. Windows naming constraints: https://learn.microsoft.com/en-us/windows/win32/fileio/naming-a-file

Current-program exit tradeoff

  1. This design intentionally omits direct exit for the current Bosatsu program from Bosatsu/IO/Core; it does support stopping a spawned direct child through the lifecycle API above.
  2. Abrupt process termination from inside arbitrary Prog code makes cleanup and structured error handling less safe.
  3. Preferred pattern: return status from Main (Prog[err, Int]), or encode early termination in program types such as Prog[IOError, Result[Int, a]].

Mapping from the requested primitives

  1. stdin -> Bosatsu/IO/Core::stdin
  2. stdout -> Bosatsu/IO/Core::stdout
  3. stderr -> Bosatsu/IO/Core::stderr
  4. readText -> read_text
  5. writeText -> write_text
  6. flush -> flush
  7. close -> close
  8. openFile -> open_file
  9. listDir -> list_dir
  10. stat -> stat
  11. mkdir -> mkdir
  12. remove -> remove
  13. rename -> rename
  14. argv -> Main(args -> ...) argument list (not duplicated in Bosatsu/IO/Core)
  15. getEnv -> get_env
  16. exit -> intentionally omitted; use Main return codes and typed early-termination (Result[Int, a]) instead
  17. spawn -> spawn
  18. wait -> wait
  19. terminate -> terminate
  20. kill -> kill
  21. poll -> poll
  22. waitTimeout -> wait_timeout
  23. nowWall -> now_wall
  24. nowMono -> now_mono
  25. sleep -> sleep

Type mapping in Bosatsu / Predef / core_alpha

  1. String, Int, Bool, List[a], Option[a], Unit map to existing Bosatsu/Predef builtins.
  2. Path maps to opaque struct Path(to_String: String) in Bosatsu/IO/Core with helpers (string_to_Path, path_to_String, path_join, path_parent, path_file_name) and external path_sep.
  3. Int64 sizeBytes maps to Int (Bosatsu Int is arbitrary precision; runtimes convert native 64-bit values).
  4. Instant maps to opaque struct Instant(epoch_nanos: Int).
  5. Duration maps to opaque struct Duration(to_nanos: Int).
  6. FileKind, FileStat, OpenMode, Stdio, StdioConfig, SpawnResult are new enum/struct types in Bosatsu/IO/Core.
  7. Handle and Process map to opaque external type types in Bosatsu/IO/Core.
  8. Program entrypoint args use Bosatsu/Prog::Main(run: List[String] -> forall err. Prog[err, Int]).
  9. Constructors for Path, Instant, and Duration are intentionally hidden from consumers (type exported, constructor not exported).

Runtime semantics (shared contract)

  1. read_text returns None only for EOF; otherwise Some(chunk) where chunk length is 1..max_chars.
  2. read_text(max_chars <= 0) returns InvalidArgument.
  3. list_dir returns child paths sorted by path_to_String for deterministic behavior.
  4. stat returns None for missing path, Some(FileStat(...)) otherwise.
  5. stat.kind is Symlink when path itself is a symlink (lstat-style classification).
  6. remove(recursive = true) removes directory trees without following symlinks.
  7. spawn never invokes a shell; cmd + args are executed directly.
  8. wait is idempotent: once complete, repeated waits return the same exit code; poll and wait_timeout observe that same recorded normalized integer status, and a timed-out wait_timeout does not consume it.
  9. Time precision is nanoseconds.
  10. now_wall returns a wall-clock timestamp value (Instant) whose intended encoding is UNIX epoch nanoseconds.
  11. now_mono returns a monotonic elapsed-time reading (Duration) and is not affected by wall-clock changes.
  12. sleep consumes Duration in nanoseconds and returns InvalidArgument for negative durations.

Bosatsu/IO/Std additions

  1. Add read_line: Prog[IOError, Option[String]].
  2. Add read_all_stdin: Prog[IOError, String].
  3. read_line blocks until newline (\n) or EOF; returns None only when EOF is reached before reading any characters.
  4. read_line strips trailing line ending (\n or \r\n) from returned text.
  5. read_all_stdin reads until EOF (stdin stream closed), not merely until “nothing currently available”.
  6. Both functions are implemented in Bosatsu/IO/Std via repeated Bosatsu/IO/Core::read_text(stdin, chunk_size).

Java/Python/C implementation plan

JVM (Predef.scala)

  1. Extend jvmExternals with all Bosatsu/IO/Core symbols.
  2. Add internal runtime objects:
  3. HandleValue (stdin, stdout, stderr, file, child pipe read/write)
  4. ProcessValue (wrap java.lang.Process, cached exit status)
  5. Implement IO effects using prog_effect dispatch:
  6. Files: java.nio.file.Files + java.io streams
  7. Process: ProcessBuilder + redirected streams
  8. Env: System.getenv
  9. Path conversion: convert Path.to_String to java.nio.file.Path at each filesystem call
  10. Wall clock: java.time.Instant.now() converted to epoch nanoseconds
  11. Monotonic clock: System.nanoTime()
  12. Sleep: Thread.sleep/LockSupport.parkNanos from duration nanoseconds
  13. Keep existing ProgRunResult testability by preserving capture-mode behavior for stdio when run under evaluator tests.

Python (ProgExt.py + externals map)

  1. Add runtime classes for Handle and Process wrapper values (Path/Instant/Duration remain Bosatsu struct values).
  2. Add new ProgExt constructors returning effect(...) thunks for each primitive.
  3. Implement using stdlib:
  4. Files/dirs/stat/remove/rename/path ops: os, pathlib, shutil
  5. Spawn/wait/pipes: subprocess.Popen
  6. Wall clock: time.time_ns
  7. Monotonic clock: time.monotonic_ns
  8. Sleep: time.sleep(nanos / 1_000_000_000.0)
  9. Env: os.environ.get
  10. Update test_workspace/Prog.bosatsu_externals for Bosatsu/IO/Core symbol remapping.

C (c_runtime)

  1. Add bosatsu_ext_Bosatsu_l_IO_l_Core.c/.h and include in build/install targets.
  2. Represent opaque runtime handles/processes as alloc_external(...) payloads (Path/Instant/Duration stay Bosatsu struct values).
  3. Implement using POSIX APIs first (same approach as current IO/Error errno mapping):
  4. Files/dirs/stat/remove/rename/path ops: open/fopen, readdir, lstat, mkdir, unlink/rmdir, rename
  5. Spawn/wait/pipes: fork/execvp/pipe/waitpid (or posix_spawn variant)
  6. Wall clock: clock_gettime(CLOCK_REALTIME, ...)
  7. Monotonic clock: clock_gettime(CLOCK_MONOTONIC, ...)
  8. Sleep: nanosleep
  9. Env: getenv
  10. Reuse and extend current errno-to-IOError mapper (c_runtime/bosatsu_ext_Bosatsu_l_IO_l_Core.c) into shared helpers for IO/Core.

Compatibility and migration

  1. Bosatsu/IO/Std remains source-compatible.
  2. Existing print/println behavior is preserved.
  3. New code can choose either high-level IO/Std or low-level IO/Core.
  4. argv remains available at the Main(args -> ...) boundary.
  5. Program termination should flow through Main return codes rather than IO/Core::exit.
  6. This design assumes the post-#1748 Bosatsu/Prog shape (Prog[err, res] with no reader env).

Tests and conformance

  1. Add package-level evaluator tests in core/src/test/scala/dev/bosatsu/EvaluationTest.scala for each primitive family.
  2. Add Python transpile + execute tests using Prog.bosatsu_externals mappings.
  3. Add C transpile + runtime tests linked against c_runtime for file/process/time operations.
  4. Add cross-runtime parity tests for:
  5. EOF semantics
  6. error tag parity (IOError variants)
  7. spawn stdio pipe behavior
  8. wall/monotonic clock semantics
  9. sleep minimum behavior

Rollout

  1. Land this design doc.
  2. Open issue: add minimal set of IO functions for basic tools.
  3. Implement Bosatsu/IO/Core + runtime bindings in JVM/Python/C.
  4. Rebase Bosatsu/IO/Std on IO/Core wrappers and add read_line / read_all_stdin.
  5. Regenerate core_alpha docs and release a new core_alpha version.
The source code for this page can be found here.

Next: Prog Concurrency API Design