Compiler internals: source to output
This is a source-guided manual of the compiler and runtimes present in the public tree. It is not a language specification or a promise of full support on every target. The implementation and the Cargo features of the binary being used determine what is available.
Reading map#
| Page | Purpose |
|---|---|
| Documentation standard | What file headers, public items, and implementation comments must explain. |
| HIR | The source-structured representation, analysis state, and companion facts. |
| MIR | Execution-shaped control flow, storage, validation, and consumers. |
| Adding a target | Static backend integration, traced through the C emitter. |
Source links point into the public faberlang/faber repository at the Faber 1.18.1 release commit, the compiler this site is pinned to. Some shared enums and feature-gated branches mention targets whose implementations do not ship there. Their presence is not evidence that those backends are available; this manual does not describe their internals. For grammar and source examples, use the grammar, the corpus, and the authoring skill.
The pipeline and its owners#
UTF-8 source + reader locale + explicit import context
-> tokens + interner radix-lexer
-> parsed syntax radix-parser / radix-syntax
-> collect -> resolve -> HIR lowering radix-semantic
-> typecheck + post-typecheck analyses radix-semantic
-> coherent analyzed module radix-module driver
|
+-> HIR source emission radix-hir-c / radix-hir-faber
+-> analyzed-HIR artifact radix-hir-fhir + driver assembly
+-> HIR-to-MIR lowering + validation radix-mir-lower / radix-mir
+-> interpretation radix-mir-runner
+-> portable MIR image radix-mir-fmir + package assembly
+-> native object or JIT radix-mir-clifThe diagram describes data flow, not a promise that every command visits every
box. radix-module sequences module compilation; radix-program composes
module analyses into package graphs and artifacts; faber owns user-facing
policy, output writing, external tool invocation, and execution orchestration.
There is no second package-specific parser or typechecker.
1. Establish source context before lexing#
The product resolves reader packs in
faber's locale policy. Package
per-file selection lives in
source_files.rs.
The intended precedence is an explicit reader override, file frontmatter,
package locale, then the product default (en). Diagnostic locale selection is
separate: changing message rendering must not change the reader language.
The module layer receives that policy via Config and Session in
session.rs.
Source preparation and
frontmatter handling
keep the body and its source offsets available. Import contracts are explicit
inputs to driver::compile; they are not guessed by the code generator.
2. Lex and parse#
radix-lexer produces tokens and an
Interner. Symbols belong to that interner. Spans use byte offsets, not
character positions; identifier normalization does not rewrite literal payloads.
Keywords are lexed as identifiers with reader-keyword identity information that
the parser can claim.
radix-parser builds the untyped syntax model in radix-syntax, allocating phase-local node IDs and recovering at syntax boundaries. Lexical errors stop the driver front end; recovered parse diagnostics can be retained while semantic analysis finds additional errors. The final error-bearing analysis must not become emitted output. See the actual continuation checks in the module driver.
3. Analyze and qualify#
radix-semantic owns collection,
resolution, AST-to-HIR lowering, typechecking, and post-typecheck analyses.
The pass registry is the
navigation map; analyze_with_cli in the semantic crate is the execution
coordinator. Collection, resolution, and HIR-lowering hard errors stop this
front half. Later analyses accumulate diagnostics; warnings alone are not
failure. Pass configuration and target qualification matter, so not every
analysis runs with identical policy on every route.
The driver packages a successful analysis as AnalyzedModule: HIR, type and
symbol arenas, resolver metadata, presentation, effect and provenance facts,
import identities, and diagnostics from the same run. analyze_source returns
this without emitting code. compile_analyzed applies warning policy,
qualifies the unit for the selected target, handles CLI entry programs, and
dispatches to output generation. compile joins those two operations. See
HIR for the boundary contract.
4. Select an output family#
- Source output:
HIR dispatch reads
CodegenAnalysis, runs shared rejection gates, and invokes a feature-enabled leaf. C emits one translation unit; the Faber re-emitter also uses the presentation sidecar and output locale. - HIR artifact: driver artifact assembly bridges analyzed state to the FHIR format. This is a codec and integrity boundary, not a source-emitter trait implementation.
- Execution-shaped output:
MIR lowering consumes
AnalyzedModuleApplicationViewand returnsLoweredMirUnit, whoseValidatedMirtoken carries the program and validation context. The MIR chapter follows this into the interpreter, FMIR, and Cranelift.
Target in radix-target owns identity,
canonical spellings, and feature availability. The
service capability table
separately describes check/build/run/package support. An enum variant can exist
while its backend feature is absent. Use faber targets for the current binary's
reported surface; do not infer it from the workspace member list.
5. Write, build, or execute at the product boundary#
Emit commands resolve product locale policy, request compiler output, report diagnostics, and hand the artifact to postprocessing. Compiler leaf emitters do not run target toolchains.
Execution is not synonymous with source emission. In
the run command, an unqualified run
can try Cranelift JIT when that feature and route are eligible, falling back on
a compile-time decline to the interpreter. It does not retry after execution
has begun. --target runner explicitly selects MIR interpretation;
--target clif-jit explicitly selects the JIT route. Runtime output then goes through
the selected engine and host, not through services::output_code.
A concrete program to follow#
The public C fixture
scalars.fab has
English frontmatter, arithmetic, locals, and printing. Its first expression is
print 1 + 2. Follow it without needing a private backend:
- The front end reads its
enpack, interns names, and parses its entry block. - Semantic lowering builds
HirModule::entryand HIR expressions. Analysis establishes types and any folds or store facts before backend dispatch. - On the C path,
Target::HirCreachesCCodegenthrough generate_c_from_analyzed. The fixture's output is pinned byscalars_program_is_pinnedin emit_test.rs. scalars.c shows the first sum asINT64_C(3)passed toprintf, plus the required runtime header. - On the interpreter path, HIR becomes validated MIR. Computation and printing are explicit operations; the runner evaluates them and its Host boundary delivers output. The C fixture is not an assertion that MIR keeps the same constant-folded shape as C.
Optional local probes, not a record of commands run for this manual (they
require a built target/debug/faber):
target/debug/faber lex crates/radix-hir-c/tests/fixtures/run/scalars.fab
target/debug/faber parse crates/radix-hir-c/tests/fixtures/run/scalars.fab
target/debug/faber hir crates/radix-hir-c/tests/fixtures/run/scalars.fab
target/debug/faber mir crates/radix-hir-c/tests/fixtures/run/scalars.fab
target/debug/faber emit -t c crates/radix-hir-c/tests/fixtures/run/scalars.fab
target/debug/faber run --target runner crates/radix-hir-c/tests/fixtures/run/scalars.fabInspection caveat: faber hir uses
the HIR inspection service:
collect/resolve prerequisites followed by HIR lowering, not the full typecheck
pipeline. Missing expression types in that dump are not evidence that emitted
code received untyped input. MIR inspection takes the analyzed route.
Packages add a graph, not another language front end#
Discovery,
manifest loading, and
package analysis establish the
source units and import graph in AnalyzedProgram. AnalysisSeams supplies
product-owned inputs such as library resolution.
Artifact planning decides
whether a selected output family has a supported package route.
For MIR packages, the package driver prepares analysis, links reachable definitions, lowers units, merges identities and arenas, and validates the result before lending it to a consumer. Linking and remapping are first stops for a failure that occurs only across module boundaries. FMIR assembly/loading lives in image.rs; its wire contract is owned by the separate format crate. A single-file C emitter does not imply package C support.
Public crate map#
This table covers the current public compiler/product/support crates, not every member of a larger development workspace. Each link is an implementation entry point, not a claim that all historical crate-header text is current.
| Crate or runtime package | Responsibility |
|---|---|
| radix-lexer | Tokens, lexing, symbols/interner, reader keyword identities. |
| radix-syntax | Parsed syntax nodes and traversal surface. |
| radix-parser | Recursive-descent parsing and syntax recovery. |
| radix-diagnostics | Shared diagnostic transport, arguments, and source spans. |
| radix-types | Semantic type/index arenas and definition identity. |
| radix-intrinsics | Compiler-owned receiver intrinsic catalog. |
| radix-semantic | Resolution, HIR lowering, typechecking, analyses, locale and diagnostic catalogs. |
| radix-hir | HIR nodes, visitors, and neutral HIR fact carriers; not the lowering passes. |
| radix-mir | MIR nodes, validation, layout/control-flow helpers, dumps, and analysis. |
| radix-mir-lower | Analyzed HIR to validated MIR and lowering metadata. |
| radix-target | Target identity and feature-gate vocabulary. |
| radix-codegen-shared | HIR emitter trait, text/naming utilities, shared rejection gates. |
| radix-hir-c | Fail-closed HIR-to-C source emitter. |
| radix-hir-faber | Faber source re-emission for formatting and conversion. |
| radix-hir-fhir | Analyzed-HIR wire schemas, codecs, integrity checks. |
| radix-mir-fmir | Portable MIR image schemas, text/binary codecs, integrity checks. |
| radix-mir-runner | MIR interpreter and execution APIs. |
| radix-mir-clif | Cranelift native-object and JIT emission. |
faber-host-clif (runtime/clif) | Host runtime and symbols used by Cranelift output. |
| radix-runtime-contract | Shared runtime display, traps, shape, and scalar contracts. |
| radix-abi | Compiler-owned host ABI symbol and wire-code tables. |
| radix-module | Module driver, services, output dispatch, and leaf re-exports. |
| radix-program | Package graph analysis, linking, and artifact assembly. |
| radix | Library-only facade over module and program APIs; no binary. |
| faber | Product library and CLI package: commands, package policy, tooling, execution. |
| test | Test inventory, selection, and MIR-runner case execution. |
| faber-leaf | Native effects such as filesystem, console, process, clock, and network routes. |
| handroll | Std-only primitives, hashing, entropy, and arbitrary-precision integers. |
| radix-hygiene | Production-source budget scans and structural hygiene checks. |
Outside Cargo crates, Norma supplies Faber library source, locale packs supply reader/message data, and the C runtime supplies header-only support for C output.
Where to look when something goes wrong#
Locate the earliest incorrect representation or missing fact; do not repair an upstream type or identity failure by guessing in a backend.
| Symptom | First investigation |
|---|---|
| Wrong locale, keyword, or literal interpretation | Product locale policy, per-file source selection, lexer, then parser. Compare lex and parse inspections. |
| Incorrect diagnostic location | Source peeling/body offsets and driver line adjustment; then diagnostic conversion/rendering in radix-module. |
| Unresolved definition, wrong type, or invalid inferred shape | Semantic collect/resolve/lower/typecheck and the matching HIR/type arena. |
| Correct semantics but lost comments or changed authored form | HirPresentation, presentation witnesses, and the Faber re-emitter. |
| Checks but cannot lower | MIR lowerer error issue/span, then validator category; distinguish unsupported lowering from malformed produced MIR. faber check --lower is the product probe. |
| Correct MIR but wrong interpreted result | Runner value/statement execution and Host implementation; inspect runtime contract helpers for traps/display. |
| Correct single unit but wrong package | Import contracts, package link/remap, merged type/symbol arenas, and metadata carriage. |
| Named target unavailable | Feature forwarding, Target::is_available, and the product gate, not emitter expression logic. |
| Wrong generated C | C expression/statement/type/naming modules and the runtime header; compare the focused pinned/rejection fixture before host execution. |
| Correct emitted artifact but build/link failure | Product postprocessing/build orchestration, runtime dependencies, and the external toolchain; not parser recovery. |
Limits of this manual: it is a structural guide, not a measured target coverage matrix. No cross-target equivalence, performance, platform availability, or complete device execution support is established here. Treat those as unverified unless the relevant focused tests and host-specific proofs have run.