Crate hermit 

Hermit

Hermit runs Linux programs with controlled thread schedules, time and randomness for reproducible testing and record/replay. The Cargo package is hermit-run; its main executable is hermit.

Read the 2022 introduction, Hermit: Deterministic Linux for Controlled Testing and Software Bug-finding for the ideas behind the project, and the current user guide for supported commands and execution guarantees.

Install on Ubuntu 24.04

Use x86_64 Linux and a current stable Rust toolchain from rustup. The stock Ubuntu 24.04 Linux 6.8 kernel is the oldest kernel qualified for the 0.4.1 ptrace release. Linux user, PID and mount namespaces, parent-child ptrace and seccomp filters must be available.

Hermit uses Reverie’s explicit ptracer-thread mode on older kernels. The generic Safeptrace interface for sibling-thread waits requires PIDFD_THREAD, introduced in Linux 6.9, and reports an error when it is unavailable. Linux 5.15 is a design target, with no qualification claim yet; see the compatibility follow-up.

Install the C compiler/linker and native development libraries before Cargo:

sudo apt-get update
sudo apt-get install -y build-essential pkg-config libunwind-dev liblzma-dev
cargo +stable install hermit-run --version 0.4.1 --locked
hermit --version
hermit run -- /bin/echo hello

No extra LZMA linker flags or nightly Rust are required. On other distributions, install the equivalent C toolchain, pkg-config, libunwind and LZMA development packages. The installer also provides hermit-dap and verification-report.

The default installation includes ptrace and KVM. SaBRe support, e9patch preprocessing and LiteInst are off by default and require their explicit Cargo features plus separately supplied runtime artifacts. DBT is available through source builds, rather than this registry release. KVM requires read/write access to /dev/kvm, compatible CPU features, performance counters and an unlimited address-space limit (ulimit -v; it refuses to start under a finite RLIMIT_AS); ptrace is the default execution path.

Ubuntu AppArmor and user namespaces

Ubuntu 24.04 restricts unprivileged user namespaces through AppArmor. A first run can fail with MapUid: EPERM or Setting UID map failed even when kernel.unprivileged_userns_clone=1. Ask the system administrator to allow user namespaces for the installed Hermit executable with a scoped AppArmor profile. Keep the system-wide user-namespace restriction enabled.

Find the executable’s absolute path with readlink -f "$(command -v hermit)". Have the administrator create /etc/apparmor.d/hermit-local with this content, replacing /home/YOUR_USER/.cargo/bin/hermit with that exact path:

abi <abi/4.0>,
include <tunables/global>

profile hermit /home/YOUR_USER/.cargo/bin/hermit flags=(unconfined) {
  userns,
}

Load it and retry:

sudo apparmor_parser -r /etc/apparmor.d/hermit-local
hermit run -- /bin/echo hello

This profile grants the installed Hermit executable user-namespace access while keeping kernel.apparmor_restrict_unprivileged_userns=1. Review the executable path and local security policy before applying it. Reinstalling to a different path requires updating the profile.

Performance counters

Precise preemption uses CPU performance counters. Host security policy and virtual machines can restrict them. Hermit reports when it disables its PMU timeslice; a successful small example in that mode does not qualify CPU-bound or threaded workloads for precise preemption. Check the user guide before changing the host’s perf policy.

Run and verify

hermit run -- /bin/echo hello
hermit run --strict --epoch=2026-01-01T00:00:00Z -- /bin/echo hello
hermit run --strict --epoch=2026-01-01T00:00:00Z --verify --verify-strict -- /bin/echo reproducible < /dev/null

--strict selects strict launch behavior and requires every input of the run on the command line: it refuses a run that would take its virtual-time epoch from the host clock, so pass --epoch or set HERMIT_EPOCH. --verify-strict selects the full canonical observation comparison; ordinary --verify uses a weaker diagnostic comparison. Hermit does not make a changing filesystem or external network reproducible; provide stable inputs for repeatability checks.

Without --strict and without an epoch, virtual time starts at the current host time captured once for the invocation. Hermit reports the exact epoch and reproduce with --epoch=... on stderr, or in the selected --log-file. Reuse that epoch and the same seed for time-dependent reproduction. Explicit --epoch and HERMIT_EPOCH values are preserved; --seed defaults to 0. See the virtual-time behavior notes.

Record and replay

hermit record start -- /bin/echo recorded
hermit record list
hermit replay --autopilot

Autopilot replays the latest recording without GDB. For an interactive replay, install GDB and use hermit replay <record-id>. Run hermit --help and the subcommand’s --help for options, output locations and verification reports.

Registry package identity

The source manifest, including a core-only source build, declares:

[package.metadata.hermit]
build-identity = "source"

When preparing the registry release projection, change that value to "registry" in its manifest and corresponding autocargo package metadata in hermit-cli/BUCK before running cargo package -p hermit-run or cargo publish -p hermit-run. Keep both values at "source" in the ordinary source tree. Cargo preserves this metadata in the packaged manifest. Removing backend features is a separate packaging choice and does not change the build identity. A missing or invalid identity is a build error.

The build script uses this value only for cfg(hermit_registry_build), which selects registry-specific diagnostics. It does not enable backends, change version/build report schemas, or mark optional backend sources as excluded.

License

BSD-3-Clause. See LICENSE.

Re-exports

pub use canonical_verdict::Verdict;

pub use reverie::process;

Modules

app

backend_stats

Hermit-side enablement and reporting for backend statistics.

build_info

Producer values for the shared machine-readable build record.

canonical_verdict

Machine-readable verification reports shared by the Hermit producer and its consumers.

chaos_timeslice

The maximum timeslice that hermit run --chaos --chaos-seeded-timeslice draws from the scheduler seed.

compatibility

Exact compatibility for a separately linked CLI backend executable.

cpu_profile_admission

Admission for intercepted CPU profiles: v3 binds qualified host XSTATE facts; host-no-rdrand requires CPUID faulting without the v3 qualification.

gdbserver

The TCP port a gdbserver serves its one gdb client on, bound in the host’s network namespace.

happens_before

Debug-info resolution for happens-before anchors.

host_input_change

Name a host file that changed during one verification run.

instruction_map

Offline discovery of nondeterministic x86 instructions in ELF binaries.

kvm_address_space

Refuse to start the KVM backend under a finite address-space limit.

liteinst_bootstrap

The shared CLI logging filter policy and the tool identity, configuration schema and effective logging directives carried by the LiteInst bootstrap. The payload is schema-validated and its config fingerprint is compared; it is not cryptographically authenticated. The host resolves the environment once; the guest consumes the same accepted directives without reading its own environment or reformatting field matchers.

liteinst_record

Bounded complete records using stock tracing field formatting.

logdiff_report

Typed machine-readable reports produced by hermit log-diff --json.

nonwaiting_write

Diagnostic writes that never wait for a reader; they live in detcore, whose own stderr writer reads the same flag. Diagnostic writes that never wait for the reader of a descriptor.

physical_exit_watch

Observes guest processes’ physical exits through pidfds, for backends whose processes leave the host some time after their scheduler-granted exit.

preemption_admission

Refuse timer requests that the selected runtime cannot implement.

proc_mount

Report the proc mount’s capabilities after container setup.

verify_run_record

Producer-owned record and raw streams for one retained verification run. Producer-owned wire record and raw streams for one retained execution.

xstate_probe

The startup qualification of --cpu-profile x86-64-v3 (https://github.com/rrnewton/hermit/issues/4243).

Structs

BackendRunFailure

The backend failed while running the guest, after Hermit launched it: the DBT evidence check, a KVM execution failure, a SaBRe guest that never reached Detcore. run --verify records it as run_failed with backend_failure instead of leaving the not_run stamp.

BackendUnavailable

A requested backend could not start because its required runtime support is absent.

Command

A builder for spawning a process.

DetConfig

Configuration options for detcore.

Detcore

The detcore tool and its per-process state.

GuestTimedOut

The guest outlived the wall-clock bound the caller asked hermit to enforce.

HermitCleanupUnconfirmed

Sendable diagnostic identifying an operation retained on its original thread.

HermitData

Holds the context necessary to run high-level hermit functions.

HermitPtraceFailure

A completed failed ptrace run, with its original nonclone primary cause. Secondary backend/cleanup errors never determine its failure classification.

Id

A recording ID.

LiteinstInGuestRefusal

In-guest LiteInst refused a run before starting its guest, because the runtime cannot honour the request or could not monitor the program.

Mount

A mount.

MountProvenance

What a recording takes from the completed recording container’s mount namespace, which a caller without a Hermit container leaves empty.

Namespace

A namespace that may be unshared with Command::unshare.

Output

The output of a finished process.

RecordOptions

Choices hermit record start makes that the recording stores in its metadata.json and that its replays honor.

Recording

The result of recording a command.

SchedulerEndedRun

The run ended with a scheduler terminal outcome (detcore::TerminalOutcome): a deadlock under Hermit’s model (Detcore’s scheduler found every remaining thread blocked with no wake it models), or the run’s virtual clock passed --virtual-timeout. Neither is a guest result. The scheduler printed its report.

SerializableError

⚠️ THE kind FIELD IS THE THIRD AND LAST OF ONE CHAIN OF FLATTENINGS. The same property – the failure’s CLASS – was destroyed at three different boundaries, in series on one path:

Shebang

Unix shebang, see Wikipedia.

SkidOvershootError

A run on a backend that delivers precise timer events (ptrace, e9patch preprocessing on it, or KVM) observed late precise-timer delivery, so Hermit refuses to treat the completed execution as deterministic evidence.

Stdio

Describes what to do with a standard I/O stream for a child process when passed to the stdin, stdout, and stderr methods of Command.

Enums

Backend

Process instrumentation backend used to run a Hermit guest.

ExitStatus

Describes the result of a process after it has exited.

FailureKind

A serializable error. This is useful for sending an error to the parent process. This works by converting an error into a string via its Display implementation. Although we lose type information in the process of converting to a string, this preserves the error message and its error chain.

HermitCleanupStage

The phase whose original owner remains retained.

NamespaceUse

Whether a run creates Hermit’s user, PID and IPC namespaces. A --no-namespace run starts the guest as a plain child and never calls unshare, so the namespace probe says nothing about whether it can run (https://github.com/rrnewton/hermit/issues/4515).

RecordedSchedule

How a recording schedules guest threads. Replay schedules them the same way, so a schedule-dependent failure that one chaos seed provokes replays.

RecoveryRefusal

A refusal that leaves any existing operation owner untouched.

Constants

GUEST_PROGRAM_NOT_EXECUTABLE_EXIT

The guest program exists but cannot be executed as given: 126.

GUEST_PROGRAM_NOT_FOUND_EXIT

The guest program could not be found at all: 127.

HERMIT_DEADLINE_EXIT

A deadline hermit itself was asked to enforce expired: 124.

HERMIT_INTERNAL_FAILURE_EXIT

Exit status for a failure of HERMIT ITSELF, as distinct from the guest’s.

HERMIT_LOG_CAP_EXIT

Hermit’s own log output exceeded --max-log-bytes: 123.

HERMIT_VERIFICATION_DIVERGENCE_EXIT

Exit status for a completed comparison that found a divergence.

LAST

The file in the data directory naming the newest recording a user asked for (hermit record, hermit run --record-networking), which a bare hermit replay replays.

LAST_RUN

The file in the data directory naming the newest recording of any kind, the manifest of a plain hermit run included.

LITEINST_TOOL_RUNTIME_ENV

Environment variable naming the in-guest Detcore preload (libdetcore_liteinst.so) explicitly.

MANIFEST_NAME

The name of the run manifest a recording directory holds (https://github.com/rrnewton/hermit/issues/4271).

RECORDING_METADATA_NAME

The name of the JSON metadata file that is saved for each recording.

RECORD_REPLAY_HASHES_IO_BUFFERS

Whether record/replay verification hashes syscall output buffers, and therefore whether its log comparison can see buffer CONTENT at all.

RECORD_REPLAY_VIRTUALIZES_TIME

Whether record/replay virtualizes time. FALSE, DELIBERATELY, AND THAT IS WHY A GREEN RECORD/REPLAY VERDICT IS NOT A DETERMINISM RESULT.

Traits

RecordOrReplay

Helper trait.

Functions

backend_delivers_preemption_timer

Whether backend’s runtime arms a PMU preemption timer and delivers its events, so --max-timeslice really preempts a thread that makes no syscall: the backend’s stated delivers_preemption_timer_events. ptrace (and e9patch preprocessing, which runs on it) and KVM do. SaBRe accepts a timer request and delivers nothing; DBT and in-guest LiteInst refuse it.

base_mountpoint_mount_ids

Each of BASE_MOUNTPOINTS that the calling thread can open, as the kernel escapes it in mountinfo, with the mount ID of a descriptor held on it: rule 2’s proof of ContainerMountTable::select. The base mountpoints hold no byte the kernel escapes (space, tab, newline or backslash), so each is its own escaped form.

capture_mountinfo_identity_order

Capture the calling thread’s mount namespace’s raw IDs in the exact row/parent order used by Detcore’s canonical mountinfo mapping.

capture_mountinfo_identity_order_keeping

capture_mountinfo_identity_order, keeping every row whose mount ID is in keep (the rows a container mount table shows) whatever its name: a mount Hermit installed is never dropped by the seed filter, so a table showing it validates against the captured order.

current_mount_namespace

The calling thread’s mount namespace (DetConfig::mountinfo_mount_namespace): the kernel’s namespace ID from ioctl(NS_GET_MNTNS_ID) on /proc/thread-self/ns/mnt, and the descriptor’s device and inode numbers. The calling thread’s, not the thread group leader’s: a library caller’s thread can have a mount namespace of its own, and every capture here (/proc/thread-self/mountinfo and fdinfo) reads that same thread’s. A kernel without the ioctl (ENOTTY or EINVAL) yields no ID; any other failure is an error.

forced_site_patching_value

The value of REVERIE_LITEINST_SITE_PATCHING (reverie_liteinst::SITE_PATCHING_ENV) that the guest sees under backend, when Hermit decides it: 0 for in-guest-trap, which exists to run with no syscall site patched, and None for liteinst, which leaves the variable to the guest environment (unset means patching on).

in_guest_guard_router_installed

Whether LiteInst’s SIGTRAP guard router can be installed in the in-guest Tool images of a default-route run of backend configured by config: on an in-guest backend, unless every image runs with syscall site patching off (config.in_guest_site_patching_off, set for --backend=in-guest-trap).

in_guest_refuses_host_timed_producers

Whether Detcore refuses host-timed signal producers in a run of backend (detcore::Config::in_guest_refuses_host_timed_producers, A5 step 3c slice 0b): on Hermit’s default in-guest route, for both in-guest backends, and never on another backend. The in-guest loader route never calls this, so the setting stays off there: today’s behaviour (D13(a), D26).

in_guest_reserved_signals

The signals the in-guest runtime keeps for itself in a run of backend configured by config (detcore::Config::in_guest_reserved_signals), bit n - 1 for signal n, for Hermit’s default in-guest route: SIGSYS, its syscall trap, always; SIGSEGV while it subscribes instructions, when it installs its instruction-fault handler for SIGSEGV; and SIGTRAP while LiteInst’s guard router can be installed, which is site patching on (in_guest_guard_router_installed): never on in-guest-trap, whose config.in_guest_site_patching_off the caller has set first. The subscription is Detcore’s own decision from the same configuration (Tool::subscriptions), which each in-guest Tool image reads to choose its instruction subscriptions (reverie-inguest tool_install.rs:191-195). Empty (0) for every other backend. The in-guest loader route never calls this, so its set stays empty: today’s behaviour there (D13(a)).

in_guest_signal_delivery_gate

The in-guest delivery gate (detcore::Config::in_guest_signal_delivery, D35 of the A5 step 3c design) for a run of backend on Hermit’s default in-guest route, from its final prepared configuration and the launcher’s PID-namespace ownership: an in-guest backend, site patching off, sequentialized threads, no configuration in which the two-phase delivery protocol cannot be kept (detcore::two_phase_refusal: a timeslice, a recorded or replayed schedule, an in-effect pipe hand-off, and the rows no in-guest backend reaches), and a scheduler that owns the guest’s PID namespace (owns_guest_pid_namespace, which the launch sites compute with scheduler_owns_guest_pid_namespace). The in-guest loader route never asks (D26, D43): its configuration keeps the gate off.

launch_on_calling_thread

Makes every later launch in this process run its ptrace session on the calling thread, as launches did before the tracer thread, and leaves the caller’s stack to the caller. For a process that must not gain threads: the hermit CLI runs its launch inside a container child made with a raw clone, whose entry point ends with a thread exit, so a tracer thread still waiting for work would keep that child alive. The CLI launches from its main thread or from the child’s own stack, which have room.

launched_image

The guest’s first image, as its /proc/self/exe link names it (detcore::Config::launch_image): the program the launch executes, found as the launch finds it, then resolved as the kernel resolves an execve (detcore::exec_record::ExecRecord::resolve). Read in the container, whose root is the guest’s. A launch through a launcher names it itself (run_sabre). None when it cannot be found, which leaves the link to the kernel, and the launch reports the missing program itself.

mountinfo_identity_order_keeping_from_raw

The identity order of the mountinfo contents raw (seed rows dropped, as mountinfo_identity_order_from_raw drops them), keeping a seed row whose mount ID is in keep.

output_backend_stdin

Returns the stdin to hand a guest run in an output-capturing backend.

prepare_in_guest_loader_route_config

The configuration of an in-guest run on the loader route (the exec-stop, native-exec and mapped-first-admission launchers of hermit run): the in-guest-trap backend’s preparation with site patching off. It keeps today’s behaviour on that route (D13(a), and D26 of the A5 step 3c design: the route’s own comparison gates any change), so the in-guest runtime’s reserved-signal set stays empty here whatever config carried; only the default route sets it (in_guest_reserved_signals).

record_to

Records to an existing directory. The caller must keep its backing resources alive through any HermitCleanupUnconfirmed recovery. HermitData::record owns its temporary recording directory inside the retained operation.

record_to_with_mountinfo

Records with producer-owned mountinfo provenance. The directory-lifetime obligation is the same as record_to. options are stored in the recording’s metadata.json, and a replay uses the same network, /proc level, CPU profile and schedule.

record_with_output

Records and captures output. Keep the directory’s backing resources alive through any cleanup checkpoint, as for record_to.

record_with_output_with_mountinfo

Records with captured output and exact mountinfo provenance. options are as for record_to_with_mountinfo.

replay_from

Replays an existing recording. Materialization and chroot guards remain with the original runtime through any cleanup checkpoint.

replay_with_gdbserver

Replays with a gdb server.

replay_with_gdbserver_and_mounts

Replays with a gdb server and mounts inside the replay chroot.

replay_with_output

Replays with captured output.

replay_with_output_and_mounts

Replays with captured output and mounts inside the replay chroot.

reserve_kvm_stdin

Saves stdin captured before Rust’s process startup can reuse a closed fd 0.

reserve_output_stdin_snapshot

Records a rewindable snapshot of the process stdin so the output-capturing backends can replay identical input to each run of hermit run --verify.

run

Run the given command as deterministically as possible.

run_with_backend

Run the given command using the selected instrumentation backend.

run_with_backend_timeout

run_with_backend with a hermit-enforced wall-clock bound on the guest.

run_with_output

Variant of run that also captures stdout/stderr.

run_with_output_backend

Variant of run_with_backend that also captures stdout/stderr.

run_with_output_backend_timeout

run_with_output_backend with a hermit-enforced wall-clock bound.

scheduler_owns_guest_pid_namespace

Whether this process, which runs the scheduler of the in-guest run it launches, owns the guests’ PID namespace (A5 step 3c.1, D51 and D52): it is PID 1 of its PID namespace, and every guest it launches descends from it, so is a member of that namespace. When a PID namespace’s init exits, the kernel sends SIGKILL to every task with a PID in the namespace and waits for them (zap_pid_ns_processes, kernel/pid_namespace.c v6.12 lines 170-222). That covers a guest in a nested PID namespace too: a task has a PID in its own namespace and in every ancestor (alloc_pid, kernel/pid.c v6.12 lines 194-253), and setns enters only the caller’s namespace or a descendant of it (pidns_install, kernel/pid_namespace.c lines 390-397), so no guest leaves the namespace. So a refusal that exits the scheduler (GlobalRequest::StoppingRefusal) then leaves no guest process running; the in-guest delivery gate requires that (in_guest_signal_delivery_gate).

thread_mountinfo

The calling thread’s mount table (/proc/thread-self/mountinfo): the rows of the namespace current_mount_namespace names.

unplanned_container_mount_table

The container mount table of a run whose guest runs in the calling thread’s own mount namespace, in which Hermit installed no mount (--no-namespace, or a library caller that built no container, on KVM), and the raw identity order Detcore validates it against, both from one read of the namespace’s rows. Shown are:

validate_tracing_environment

Check the host mechanisms needed by a retained exec-stop launcher or a ptrace run, without assuming which runtime artifact the launcher uses.

Type Aliases

Error

UnplannedMountTable

A container mount table and the raw identity order Detcore validates it against (unplanned_container_mount_table).