* Writing an x86-64 OS (mostly) from scratch in Rust I’ve always been fascinated by the idea of bootstrapping. Combined with a curiosity for learning about how things work, this has lead to me often asking the question: if all I had were my hands, how far into the “tech-tree” of humanity could I get? The answer, unfortunately, is likely not very far at all. Certainly, I would never be able to put together any sort of machine that resembles what we today understand to be a computer. The term “bootstrapping” famously comes from the saying “pull oneself up by one’s bootstraps”, which is rarely understood for what it actually is: an impossible task, a paradox. So, when tackling the problem of bootstrapping anything computer related, it is important to pick ones battles, and know where to start and what the goal is. For most people, for economic reasons, this means starting with a computer and a good bit of software already in place. Still, there are many paths one can take to either fulfil ones curiosity about how things work, or to feel the satisfaction of having something that are built by yourself. One might, from there, attempt to bootstrap a compiler (typically a simple C compiler), starting from nothing but the simplest hand-written binary blob that can turn a text file containing hex digits into a binary executable, and then build on that, step by step, every successive tool becoming more complex and more ergonomic, until, ex-nihil, a C compiler has appeared which is capable of building itself, as well as old versions of binutils and the gcc toolchain. From there, it’s possible to build every piece of software found on the modern Linux system. The journey that I will hopefully document in this blog, both learning my self and hopefully acting as a reference for others also interested in the same topic, will be the building of a simple operating system for the average modern computer running the x86-64 architecture. Instead of, how the little purist voice in my head demands, starting, as described above, from nothing and writing my own compiler and language, I will instead use the awesome Rust programming language. This is both because writing an assembler and multiple compilers from scratch is a lot of very tedious work, and because Rust is, in my opinion, the best language available for the task at hand. * Initial Setup https://os.phil-opp.com/ https://wiki.osdev.org/Limine_Bare_Bones We will start by setting up a development environment using the equally awesome nix package manager, which allows us to declaratively specify which software we need and have them made available to us on any system where nix is available. We will be using nix flakes, and our initial flake.nix will look like this: #+begin_src nix { description = "An x86 kernel written in Rust"; inputs = { nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable"; rust-overlays.url = "github:oxalica/rust-overlay"; flake-utils.url = "github:numtide/flake-utils"; }; outputs = { nixpkgs, flake-utils, rust-overlays, ...}: flake-utils.lib.eachDefaultSystem (system: let overlays = [ (import rust-overlays) ]; pkgs = import nixpkgs { inherit system overlays; config.allowUnfree = true; }; rust-override = { extensions = ["rust-analyzer" "rust-src"]; targets = [ "x86_64-unknown-linux-gnu" ]; }; rust-pkg = pkgs.rust-bin.nightly.latest.default.override rust-override; in with pkgs; { devShells = { default = mkShell rec { name = "rust-osdev"; nativeBuildInputs = [ pkg-config rust-pkg clang gcc mold ]; buildInputs = [ udev stdenv.cc.cc.lib ]; LD_LIBRARY_PATH = lib.makeLibraryPath buildInputs; }; }; }); } #+end_src This is a lot of code to essentially just say that we would like to have the latest nightly version of Rust as well as a standard collection of build tools available. We specifically add the =rust-src= component to the Rust derivation because we will need to build the standard library ourselves in a later step. To automatically load the proper environment when entering the project directory, we will also create a .envrc file with the following contents: #+begin_src bash use flake . #+end_src and then run the following command to allow direnv to load the environment: #+begin_src bash direnv allow #+end_src Now we can start writing the first bit of rust code: #+begin_src bash cargo new --bin kernel #+end_src Most software will at some point want to make use of or work together in some way with other software running on the same machine, including most certainly the kernel, the very thing we are trying to replace. In Rust, the mechanisms for interfacing with other software or the kernel are provided by the standard library =std=. =std= itself depends on a number of libraries which are not available to us, in our bare-metal, operating system-less environment. To tell the rust compiler that we don’t want to use the standard library, we can use the crate-level attribute =no_std=; we will still have access to a large portion of the =std= library via the =core= and =alloc= libraries, which =std= re-exports. A typical Rust program starts in the =main= function, but this is not actually where the final program itself starts running from. Before the users =main= function, things like environment variables and command line arguments are set up. Since there are no command line arguments or environment variables for our kernel, and because the scaffolding around the special =main= function won’t be available for us either, we will need to tell the compiler additionally that we have no =main= function. This is done with the crate-level attribute =no_main=. We will need to provide the entry point for the linker ourselves, which is canonically named =_start=. We annotate this function with the unsafe =no_mangle= attribute which tells the compiler to keep the name of the function as is so that the linker can find it. We could also have used the =#[export_name = "_start"]= attribute and called the function something else, such as =kernel_main=. For now, our =_start= function will do nothing, which we can accomplish by “spinning” in an infinite loop. #+begin_src rust #![no_std] #![no_main] #[unsafe(no_mangle)] fn _start() -> ! { loop {} } #+end_src As is, attempting to build this code will throw linker errors, because cargo will still attempt to link with its own =_start= function and the aforementioned scaffolding around the =main= function. To tell cargo off, we will define our own custom target specification file which will also allow us to conveniently special-case the build for our kernel via cargo's configuration later. #+begin_src json { "arch": "x86_64", "code-model": "kernel", "cpu": "x86-64", "crt-objects-fallback": "false", "data-layout": "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-i128:128-f80:128-n8:16:32:64-S128", "disable-redzone": true, "target-endian": "little", "features": "-mmx,-sse,-sse2,-sse3,-ssse3,-sse4.1,-sse4.2,-avx,-avx2,+soft-float", "linker": "rust-lld", "linker-flavor": "ld.lld", "llvm-target": "x86_64-unknown-none-elf", "max-atomic-width": 64, "panic-strategy": "abort", "plt-by-default": false, "os": "none", "executables": true, "relro-level": "full", "rustc-abi": "softfloat", "stack-probes": { "kind": "inline" }, "supported-sanitizers": [ "kcfi", "kernel-address" ], "target-pointer-width": 64, "target-c-int-width": 32 } #+end_src Most of these fields are taken from the the =x86_64-unknown-none-elf= target, which is easily retrieved by running the following command: #+begin_src bash rustc -Z unstable-options --print target-spec-json --target x86_64-unknown-none #+end_src More options available in the current rustc can be found by running the following command with a nightly version of rustc: #+begin_src bash rustc -Zunstable-options --print target-spec-json-schema #+end_src Notably, I have not copied the =*position-independent-executables= fields from the =x86_64-unknown-none-elf= target, because we will later use a linker script to place our kernel in the higher-half of the address space and modify the layout of our sections manually, and position independent executables complicate this. If we now attempt to build our kernel, cargo will now complain that it cannot find the =core= crate since it is not available for our custom target: #+begin_src bash cargo -Zjson-target-spec build --target x86_64-unknown-kernel.json #+end_src Even though we’ve told the compiler we don’t want the standard library, we will still have to build it for our target to make available the =core= and =alloc= libraries. Since we added the =rust-src= component when installing Rust via nix, we can now simply add a =.cargo/config.toml= file to tell cargo to build =std= for us and select the appropriate features: #+begin_src toml [unstable] json-target-spec = true # lets us specify a custom target specification file build-std-features = ["compiler-builtins-mem"] build-std = ["core", "compiler_builtins"] #+end_src Because we plan on using builtin functions like =memcpy=, we tell cargo to build the =compiler_builtins= crate with the =mem= feature enabled. Unstable fields in =cargo/config.toml= are unstable cargo features that can otherwise be enabled by passing the =-Z= flag to cargo, as we did above for the =json-target-spec= feature. A list of unstable cargo features can be found [[https://doc.rust-lang.org/cargo/reference/unstable.html][here]]. =build-std= takes as argument a list of crates to build from the standard library. We only want =core= and =compiler_builtins=, but in the future we may want to add =alloc=. Any crate that is part of the library workspace can be specified here, though some make no sense to specify, or even break cargo. =core= implies =compiler_builtins=, but since =build-std= is an unstable feature and may change in the future, we explicitly specify both to be safe. =build-std-features= takes a list of features to pass to the crates specified in =build-std=. For a complete list of crates and features, I recommend exploring the =library= directory of the Rust source code. Note that not only the crates specified in the =members= field of a workspace =Cargo.toml= are members, but also any path dependencies. Rust permits the user to abort the program at any time by calling the =panic!= macro, or a function that “panics” internally. Typically, the mechanism by which this happens, as well as the more complex behaviour of unwinding the stack and catching unwinds, is provided by the =std=. Since we’ve opted out of using =std=, we will need to provide a panic handler for rust to call in case of a panic. Unwinding is already disabled by the =panic-strategy= field in our target specification file. For the time being, we will again simply spin in an infinite loop whenever a panic occurs: #+begin_src rust #[panic_handler] fn panic(_info: &core::panic::PanicInfo) -> ! { loop {} } #+end_src Both the panic handler and our entry point have so far made use of the unstable =never= type =!=, which can be used stabily only as the return type of functions that never return. =never= is Rust’s uninstantiable type which can coerce to any other type: it is the subtype of every other type. =!= is returned by a handful of expressions that either affect control flow such as =return expr=, =continue= or =break expr= and so mean that any assignment of the value of such an expression becomes unreachable, or expressions that never return such as =panic!= or a =loop= with no break. By annotating our functions with the =!= return type, the compiler will ensure that we can’t accidentally write code that does return: because =!= is uninstantiable, no value can coerce to it, and so any expression that yields a value would cause a compiler error. Finally, our kernel builds without further errors. Since cargo produces an ELF binary, we can even try it out it by running #+begin_src bash cargo run --target x86_64-unknown-kernel.json #+end_src Of course, all it will do is nothing, indefinitely. The next step will be to get our kernel to run in a virtual machine, which will stand in for a real computer and permit us to start doing anything proper kernel-like that would be forbidden in user-space. There are a number of strategies for getting a computer to run our kernel, but we will be using the limine bootloader, as it greatly simplifies the code required to get our kernel running, as well as starting us off already in 64-bit mode and mapped into the higher-half of the address space. Since the launch of Windows 8 in 2012, Microsoft has required that all computers ship with UEFI firmware; as a consequence, almost all modern x86-64 computers will have UEFI firmware. Another strategy would be to act as a UEFI application and letting the firmware load our kernel without bothering with a bootloader, however UEFI is a very complex and heavy specification, requiring a lot more code to get to the fun part of writing our operating system, so instead we will let limine deal with the UEFI part of booting. Using either limine or UEFI also means we skip over the legacy BIOS boot process which involves setting up the CPU in 16-bit real mode, and switching to 32-bit protected mode before finally switching to 64-bit long mode, which is what we are interested in. We will need to add a few additional dependencies to our =flake.nix=: #+begin_src nix nativeBuildInputs = [ ... qemu # the virtual machine OVMF # UEFI firmware for qemu limine-full # the bootloader mtools # tools for creating a FAT filesystem image ]; # path to the UEFI firmware binary exposed as an environment variable OVMF_PATH = OVMF.fd; #+end_src We will create a =run.sh= script to automate hide away the details of launching the virtual machine and setting up the bootloader and kernel image: #+begin_src bash #!/usr/bin/env bash set -e bin="$1" canonicalised=$(realpath -- "$bin") parent=$(dirname -- "$canonicalised") bin_name=$(basename -- "$exe") bin_name=${bin_name%.*} image="$parent/$bin_name.img" cat > /tmp/limine.conf < /tmp/limine.conf < ! { loop { hlt(); } } #+end_src For a detailed explanation of the =asm!= macro, see the [[https://doc.rust-lang.org/nightly/reference/inline-assembly.html][Rust reference]]. In our =main.rs=, we can now replace the infinite loop with a call to our new =halt_loop= function: #+begin_src rust #[panic_handler] fn panic(_info: &core::panic::PanicInfo) -> ! { kernel::x86_64::halt_loop() } #[unsafe(no_mangle)] fn _start() -> ! { kernel::x86_64::halt_loop() } #+end_src * Getting the Framebuffer https://wiki.osdev.org/Limine_Bare_Bones#Compiling_the_kernel https://github.com/Limine-Bootloader/limine-protocol/blob/trunk/PROTOCOL.md The limine protocol, which we’ve chosen to use for our bootloader, allows our kernel to make requests at compile time, which the bootloader will then fulfil at boot time. One of the requests we can make is for the Framebuffer with which we can draw pixels to the screen. Limine requests generally have the following structure: #+begin_src rust #[repr(C)] pub struct Request { magic: [u64; 2], id: [u64; 2], revision: u64, response: UnsafeCell>>>, request: U, } impl Request { pub const MAGIC: [u64; 2] = [0xc7b1dd30df4c8b88, 0x0a82e883a194f07b]; pub const fn new_raw(id: [u64; 2], revision: u64, request: U) -> Self { Self { magic: Self::MAGIC, id, revision, response: UnsafeCell::new(None), request, } } pub fn response(&self) -> Option<&Response> { unsafe { self.response .get() .cast::>>>() .read_volatile() } .map(|ptr| unsafe { ptr.as_ref() }) } } #+end_src A magic 128-bit number, which is the same for all requests, is followed by a 128-bit request ID which is unique to each request type. For the Framebuffer, the request ID, expressed as two 64-bit integers, is =[0x9d5827dcd881dd75, 0xa3148604f6fab11b]=. What follows is a 64-bit revision number for the request, which allows single requests to change shape without having to change the revision of the entire protocol. For the Framebuffer, there are currently two revisions, 0 and 1, but we are interested only in the payload of revision 1. In the case that there are Framebuffers available, limine will write a pointer to the response structure into the =response= field of the request. Some requests may require additional data to be read by the bootloader, which are appended to the end of the request structure, here represented by the generic type parameter =U=. The response structure is very simple, containing only the revision number of the response the bootloader provided, followed by the request-dependent payload, which in the case of the Framebuffer is a count, and a pointer to an array of pointers to the available Framebuffers: #+begin_src rust #[repr(C)] pub struct Response { revision: u64, response: T, } #[repr(C)] pub struct FramebufferResponse { count: u64, framebuffers: *const *const Framebuffer, } unsafe impl Send for FramebufferResponse {} unsafe impl Sync for FramebufferResponse {} impl FramebufferRequest { pub const ID: [u64; 2] = [0x9d5827dcd881dd75, 0xa3148604f6fab11b]; pub const fn new() -> Self { Self::new_raw(Self::ID, 0, ()) } pub fn framebuffers<'a>(&self) -> &'a [&'a Framebuffer] { // SAFETY: limine responses are guaranteed to be valid for the lifetime of the memory mapping. self.response() .map(|response| unsafe { core::slice::from_raw_parts( response .response .framebuffers .cast::<&'a Framebuffer>(), response.response.count as usize, ) }) .unwrap_or(&[]) } } #+end_src Limine requests, for any revision versions of the protocol greater than 1, must be placed inbetween two special magic markers in the mapping of the kernel binary. This way, Limine will know where to find the requests and hopefully not accidentally mistake some other data in the kernel binary for a request. #+begin_src rust #[repr(transparent)] pub struct RequestsStartMarker([u64; 4]); pub const REQUESTS_START_MARKER: RequestsStartMarker = RequestsStartMarker([ 0xf6b8f4b39de7d1ae, 0xfab91a6940fcb9cf, 0x785c6ed015d3e316, 0x181e920a7852b9d9, ]); #[repr(transparent)] pub struct RequestsEndMarker([u64; 2]); pub const REQUESTS_END_MARKER: RequestsEndMarker = RequestsEndMarker([0xadc0e0531bb10d03, 0x9572709f31764c62]); #+end_src Typically, items in Rust are placed into sections (e.g. =.text=, =.data=, =.rodata=, =.bss=) according to their type and usage. Functions are placed in the =.text= section, which is usually marked as executable and read-only, meaning that, just like attempting to write to a =const= variable, attempting to write to a function will cause a fault. Sometimes, one might still want to write to a function however, and then Rust offers the =link_section= attribute to place items into arbitrary sections. A linker or other binary manipulation tools (found in the =binutils= collection of tools) can be used to inform the loader to mark the memory of that section as, for example, writable. #+begin_src rust #[used] #[unsafe(link_section = ".limine_requests_start")] static LIMINE_REQUESTS_START: RequestsStartMarker = REQUESTS_START_MARKER; #[used] #[unsafe(link_section = ".limine_requests")] pub static FRAMEBUFFER_REQUEST: FramebufferRequest = FramebufferRequest::new(); #[used] #[unsafe(link_section = ".limine_requests_end")] static LIMINE_REQUESTS_END: RequestsEndMarker = REQUESTS_END_MARKER; #+end_src Since we don’t reference the =LIMINE_REQUESTS_START= and =LIMINE_REQUESTS_END= markers anywhere in our code, we tell the compiler and clippy that they are nevertheless used with the =used= attribute. Rust already knows to place our requests into read-write memory, however the order of items within a section cannot be controlled from within Rust, and so we will use a linker script to ensure that all requests are placed between the start and end markers. The following script is mostly taken from the =osdev.org= wiki entry on the [[https://wiki.osdev.org/Limine_Bare_Bones#Compiling_the_kernel][Limine Bare Bones example]]. #+begin_src linkerscript OUTPUT_FORMAT(elf64-x86-64) ENTRY(_start) PHDRS { limine_requests PT_LOAD; text PT_LOAD; rodata PT_LOAD; data PT_LOAD; } SECTIONS { . = 0xffffffff80000000; .limine_requests : { KEEP(*(.limine_requests_start)) KEEP(*(.limine_requests)) KEEP(*(.limine_requests_end)) } :limine_requests . = ALIGN(CONSTANT(MAXPAGESIZE)); .text : { *(.text .text.*) } :text . = ALIGN(CONSTANT(MAXPAGESIZE)); .rodata : { *(.rodata .rodata.*) } :rodata . = ALIGN(CONSTANT(MAXPAGESIZE)); .data : { *(.data .data.*) } :data .bss : { *(.bss .bss.*) *(COMMON) } :data /DISCARD/ : { *(.eh_frame*) *(.note .note.*) } } #+end_src We take the opportunity to place our kernel explicitly into the higher-half of the address space, starting at =0xffffffff80000000=. The most important bit with respect to the limine requests is the ordering of the =KEEP= directives in the =.limine_requests= section. After that, we need to make sure that none of the sections with different permissions (read, write, execute) are placed in the same page. In addition to the Framebuffer, another request we will make use of is the Base Revision request, which tells us whether the bootloader supports the revision of the protocol we are using. #+begin_src rust #[repr(C)] pub struct BaseRevision(UnsafeCell<[u64; 3]>); unsafe impl Send for BaseRevision {} unsafe impl Sync for BaseRevision {} impl BaseRevision { pub const fn from_revision(revision: u64) -> Self { Self(UnsafeCell::new([ 0xf9562b2d5c95a6c8, 0x6a7b384944536bdc, revision, ])) } pub fn is_supported(&self) -> bool { unsafe { self.0.get().cast::().add(2).read_volatile() == 0 } } pub fn base_revision(&self) -> u64 { unsafe { self.0.get().cast::().add(2).read_volatile() } } } #+end_src Even though we don’t ever require mutable access to the internal array, we use an =UnsafeCell= to tell the compiler that the value of the array may have changed at any time and are not what it might expect them to be. This makes the reads of the array, which require unsafe code to access the contents of the interior mutabily cell, perfectly safe as we don’t violate Rust’s aliasing rules. Rust also assumes by default that anything containing an =UnsafeCell= is not safe to share between threads, and while our program is currently single-threaded, since we only ever read from the cell anyway, we can safely tell Rust our type is =Sync=. This is actually required for the =BaseRevision= request to be used in a static variable, as Rust has no way of enforcing that all code that accesses the variable is running on the same thread. The same is true also for the =Request= type, except that we have to bound the impl of =Sync= on =T= and =U= being =Sync= as well. ** Using the Framebuffer http://www.petesqbsite.com/sections/tutorials/tuts/vbe3.pdf https://www.ctyme.com/intr/rb-0274.htm#Table82 https://github.com/Limine-Bootloader/Limine/blob/v12.x/common/drivers/vbe.c https://github.com/Limine-Bootloader/Limine/blob/v12.x/common/drivers/gop.c Framebuffers have the following layout: #+begin_src rust #[repr(C)] pub struct Framebuffer { pub addr: *mut u8, pub width: u64, pub height: u64, pub pitch: u64, pub bpp: u16, pub memory_model: u8, pub red_mask_size: u8, pub red_mask_shift: u8, pub green_mask_size: u8, pub green_mask_shift: u8, pub blue_mask_size: u8, pub blue_mask_shift: u8, pub reserved: [u8; 7], pub edid_size: u64, pub edid: *mut u8, } #+end_src The limine documentation on this type is somewhat sparse, but the fields are mostly self-explanatory. Relevant to us are the =addr= field, and the fields describing how the (sub)pixels =addr= points to are laid out in memory. =width= and =height= describe the dimensions of the framebuffer in pixels, =pitch= describes how many bytes are used for each row of pixels, and =bpp= describes how many bits are used for each pixel. =bpp= can be assumed to be a multiple of 8. We will additionally assume that =bpp= is always at most 64. =memory_model= is currently always =1=, meaning =LIMINE_FRAMEBUFFER_RGB=, but might in the future allow for other memory models in order to support [[https://www.ctyme.com/intr/rb-0274.htm#Table82][VESA SVGA memory model]] values other than Direct Color. The =mask_size= and =mask_shift= fields describe how the respective color channels are to be organised with a =bpp / 8= byte memory region. To draw an rgb-colored pixel at a given position =(x, y)= then, we will define the following function: #+begin_src rust pub struct Color(pub [u8; 3]); impl Framebuffer { /// # Safety /// the caller must ensure they have exclusive access to the region of the /// framebuffer they are writing to. pub unsafe fn draw_pixel(&self, x: u64, y: u64, color: Color) { assert!(self.bpp <= 64); let bytes_per_pixel = self.bpp.div_ceil(8) as usize; let [r, g, b] = color.0; let mask_color = |c: u8, bits: u8, shift: u8| { let mask = (1 << bits) - 1; (c as u64 & mask) << shift }; let mut pixel = 0u64; pixel |= mask_color(r, self.red_mask_size, self.red_mask_shift); pixel |= mask_color(g, self.green_mask_size, self.green_mask_shift); pixel |= mask_color(b, self.blue_mask_size, self.blue_mask_shift); let pixel = pixel.to_ne_bytes(); let dst = unsafe { self.addr .add((y * self.pitch + x * bytes_per_pixel as u64) as usize) }; unsafe { super::volatile_copy(pixel.as_ptr(), dst, bytes_per_pixel) } } } #+end_src The method is =unsafe=, since we cannot guarantee that the call site is the only code currently attempting to write to the framebuffer yet. =volatile_copy= is a simple utility function mirroring =core::ptr::copy=, but performing a volatile write to the destination; this is necessary because =addr= might be device mapped memory. Inside of our =_start= function, we can now bring all of this together to draw a pretty pattern to the screen: #+begin_src rust #[unsafe(no_mangle)] fn _start() -> ! { assert!(limine_requests::LIMINE_BASE_REVISION.is_supported()); let fb = limine_requests::FRAMEBUFFER_REQUEST .framebuffers() .first() .expect("No framebuffer found"); for y in 0..fb.height { for x in 0..fb.width { let r = (x * 255 / fb.width) as u8; let b = (y * 255 / fb.height) as u8; // SAFETY: We have exclusive access to the framebuffer. unsafe { fb.draw_pixel(x, y, kernel::limine::Color([r, 0, b])); } } } kernel::x86_64::halt_loop() } #+end_src * Interlude 2: Sync We’ve managed to get something on the screen and confirmed that our kernel is running and working as intended, but what if something had gone wrong? We have used assertions of some kind multiple times, to enforce invariants our code relies on, or to =expect= the presence of a framebuffer. If those assertions fail, rust will call our panic handler and abort the intended execution of our kernel, but our panic handler currently just silently swallows the panic message and spins. If we want some way to communicated with the host, the most straightforward way is to use a serial port, just like we told limine to do for its error output. Before that, however, I want to tackle a problem that we’ve just encountered with the framebuffer, and which will be even more relevant for the serial port: synchronisation. One of Rust’s core strengths, besides its enforcement of memory safety, is its robust type and trait system to protect against concurrency bugs. Currently, our kernel is single-threaded, so the compilers pedantism about the thread-safety of the framebuffer request feels like a nuisance, but since we do plan on eventually running on multiple cores and threads, it is better seen as an opportinity to build the necessary framework from the very beginning. Whether we want to use global singletons or pass around one large context object, we will eventually want to share access to specific resources requiring explicit synchronisation between threads or cores, and for that we will need primitives like mutexes and locks. In this section we will implement a simple spinlock-based =Mutex=, a =Once= type for synchronising one-time initialisation, and the =OnceLock= and =LazyLock= types built on top of =Once=. All of these types are typically found in the =std::sync= module, but those implementations rely on the operating system to provide the necessary synchronisation primitives such as futexes (fast user mutexes). Alternatively, the =spin= crate provides a spinlock-based implementations of these types with a similar API which will be functionally identical to the implementations we will be writing. ** =Mutex= The purpose of a mutex is to provide exclusive access to a resource, or to ensure that two critical sections on different threads agree on the state of a resource. A critical section is a region of code in which a shared resource is read from or written to. The primitive upon which mutexes are built, is the atomic compare-and-swap operation. Our mutex will be a simple struct consisting of an =AtomicBool= capable of performing this operation, and the familiar =UnsafeCell= containing the resource we want to protect with the mutex: #+begin_src rust pub struct SpinMutex { lock: AtomicBool, data: UnsafeCell, } #+end_src A mutex will start out in a known unlocked state (=false= in our case). When a thread wants access to the resource, it will attempt to acquire the lock by exchanging the expected =false= value with =true=. When performing any atomic operation in Rust, we must specify the memory ordering of the operation, or a sub-operation, according to the C++11 memory model. Both the processor and the compiler are allowed to reorder the individual instructions of our program according to their respective memory models, and even if we targeted a hypothetical processor with a totally sequentially consistent memory model, it would still be necessary to provide the compiler with the correct memory ordering since our Rust program does not actually run on the processor, but rather the abstract Rust machine which is essentially emulated on the target processor. In our case, wanting “two threads to agree on the state of a resource” means that in the case that we successfully exchange =false= for =true=, we want to =Acquire= all write-operations that happened before the respective =false= write on some other thread that last accessed our resource. In the case that we failed to exchange =false= for =true= (because some other thread currently holds the lock), the memory ordering can be =Relaxed= because we don’t care about receiving partial updates to the resource we won’t be accessing until the other thread has released the lock. #+begin_src rust impl SpinMutex { pub fn try_lock(&self) -> bool { self.lock .compare_exchange(false, true, Ordering::Acquire, Ordering::Relaxed) .is_ok() } pub fn try_lock_weak(&self) -> bool { self.lock .compare_exchange_weak(false, true, Ordering::Acquire, Ordering::Relaxed) .is_ok() } pub fn lock(&self) -> SpinMutexGuard<'_, T> { while !self.try_lock() { core::hint::spin_loop(); } SpinMutexGuard { mutex: self } } /// # Safety /// The caller must be the logical owner of the locked mutex. pub unsafe fn unlock(&self) { self.lock.store(false, Ordering::Release); } } #+end_src The =try_lock_weak= variant of =try_lock= usesa variant of the compare-and-swap operation that is allowed to fail spuriously, meaning it may return =Err(_)= even if the current value of the atomic is =false=, but may result in better performance overall on some architectures. As can be seen on Compiler Explorer, on our platform, there is no difference in the assembly emitted. Raymond Chen of The Old New Thing has a short post explaining the difference on a platform that does care: [[https://devblogs.microsoft.com/oldnewthing/20180329-00/?p=98375][ARM]]. For a Rust oriented explanation, there is an [[https://mara.nl/atomics/hardware.html][entire chapter]] on the topic of atomics by Mara Bos. Atomics are a very complex topic and often hard to reason about, easily demonstrated by a note from Mara’s chapter which points out that the behaviour noted by Raymond Chen is actually no longer the case with the =cas= instruction of the =ARMv8.1= architecture. The =SpinMutexGuard= type is a simple wrapper which holds a reference to the mutex and implements the =Deref= and =DerefMut= traits for the wrapped resource =T=. When dropped, ==SpinMutexGuard= automatically releases the lock on the mutex: #+begin_src rust pub struct SpinMutexGuard<'a, T: ?Sized + 'a> { mutex: &'a SpinMutex, } impl core::ops::Deref for SpinMutexGuard<'_, T> { type Target = T; fn deref(&self) -> &Self::Target { unsafe { &*self.mutex.data.get() } } } impl core::ops::DerefMut for SpinMutexGuard<'_, T> { fn deref_mut(&mut self) -> &mut Self::Target { unsafe { &mut *self.mutex.data.get() } } } impl Drop for SpinMutexGuard<'_, T> { fn drop(&mut self) { unsafe { self.mutex.unlock() } } } #+end_src If we recall, the whole point of implementing the =Mutex= type was to be able to share resources between threads. However, since our =SpinMutex= type contains an =UnsafeCell=, Rust has automatically determined that it never implements the =Sync= trait. Since we know that all access to the resource is properly synchronised by the mutex, we can safely inform the compiler that our type is indeed =Sync=: #+begin_src rust unsafe impl Send for SpinMutex {} // SAFETY: SpinMutex properly synchronises access to T, but T must be Send // to be safely shared between threads. unsafe impl Sync for SpinMutex {} // SAFETY: &SpinMutexGuard cannot yield a &mut T, and the SpinMutexGuard // holds the lock for the duration of its lifetime, so it is safe to share // between threads. unsafe impl Sync for SpinMutexGuard<'_, T> {} // SAFETY: SpinMutex may be unlocked on any thread, so it is safe to send // the guard to another thread as long as T is Send. unsafe impl Send for SpinMutexGuard<'_, T> {} #+end_src ** =Once= The =Once= type is a synchronisation primitive that acts as a barrier separating the before and after scopes of some initialisation code. To completely describe the states of a =Once=, we need more values than a simple boolean: #+begin_src rust const UNINITIALIZED: u8 = 0; const INITIALIZING: u8 = 1; const INITIALIZED: u8 = 2; const POISONED: u8 = 3; pub struct Once { state: AtomicU8, } #+end_src The need for the =POISONED= and =INITIALIZING= states will become clear in a moment. To check if the =Once= has been initialised, we can simply check if the state is =INITIALIZED=. Since we care about write operations that happened during the initialisation, we =Acquire= them when loading the state: #+begin_src rust pub fn is_completed(&self) -> bool { self.state.load(Ordering::Acquire) == INITIALIZED } #+end_src When we want to run some initialisation code, we first load and match on the current state: if the =Once= is already initialised, we don’t want to run anything, and can simply return. If the state is =UNINITIALIZED=, we attempt another compare-and-swap operation to change the state from =UNINITIALIZED= to =INITIALIZING=, and if we succeed, we run a user-provided closure. If we fail the compare-and-swap, we will want to make sure to nevertheless =Acquire= any initialisation related write-operations that might have happened in the meantime on a different thread: if the compare-and-swap failed because the state has changed to =INITIALIZED=, we can now simply return. Since our kernel does not allow for recovering from panics through unwinding, we don’t care that panics might occur during the user-provided initialisation closure causing our =Once= to remain in the =INITIALIZING= state, however, there is still value in permitting the user to poison the =Once= as a result of, for example, a fallible initialisation closure. We let the user decide how a poisoned =Once= should be handled: the =ignore_poison= parameter to the method decides whether a previously poisoned =Once= panics or attempts to run the initialisation closure again. Finally, if the state is =INITIALIZING=, we spin until it changes to some other state. #+begin_src rust pub fn call_once(&self, ignore_poison: bool, f: F) where F: FnOnce(&OnceState), { let mut state = self.state.load(Ordering::Acquire); loop { match state { INITIALIZED => return, POISONED if !ignore_poison => { panic!("Once instance has previously been poisoned") } POISONED | UNINITIALIZED => { match self.state.compare_exchange( state, INITIALIZING, Ordering::Acquire, // if we get `Err(INITIALIZED)`, we want to have // acquired what the lock is protecting. Ordering::Acquire, ) { Err(new) => { state = new; continue; } Ok(_) => { // even though we don't have unwinding, for // completeness sake we'll poison the lock if // the closure panics. let guard = crate::drop_guard! { self.state.store(POISONED, Ordering::Release) }; let state = OnceState { poisoned: state == POISONED, state_to_set: Cell::new(INITIALIZED), }; f(&state); guard.forget(); self.state .store(state.state_to_set.take(), Ordering::Release); return; } } } _ => { assert_eq!(state, INITIALIZING); loop { state = self.state.load(Ordering::Acquire); if state != INITIALIZING { break; } core::hint::spin_loop(); } } } } } #+end_src The =OnceState= struct passed to the closure lets the caller check whether the =Once= was previously poisoned, and to set the state to =POISONED= rather than =INITIALIZED= if the initialisation closure fails. Though not necessary for our non-unwinding kernel, a drop guard is used to additionally poison the =Once= should the closure cause a panic and unwind. the =drop_guard!= macro is a simple utility macro that will be useful not only as an idiomatic rusty =PanicGuard=, but also in other situations where we might want to run some code on the short-circuiting path of a function using the =?= operator. #+begin_src rust #[macro_export] macro_rules! drop_guard { ($($stmts:stmt)*) => { { struct __DropGuard(::core::mem::ManuallyDrop); impl __DropGuard { #[allow(dead_code)] fn forget(self) { let mut this = ::core::mem::ManuallyDrop::new(self); unsafe { ManuallyDrop::drop(&mut this.0); } } } impl Drop for __DropGuard { fn drop(&mut self) { unsafe { ::core::ptr::read(&*self.0)() } } } __DropGuard(::core::mem::ManuallyDrop::new(|| { $($stmts)* })) } }; } #+end_src The astute reader might have noticed that most of the code in the =call_once= method can only ever be executed once: we can likely improve the performance by offering a fast path for the common case of the =Once= already being initialised. Since we will, in most cases, also know whether or not we want to panic or attempt initialisation on a poisoned =Once=, we can provide two separate methods for the two cases: #+begin_src rust pub fn call_once(&self, f: F) where F: FnOnce(&OnceState), { if self.is_completed() { return; } self.call_once_slow(false, f); } pub fn call_once_force(&self, f: F) where F: FnOnce(&OnceState), { if self.is_completed() { return; } self.call_once_slow(true, f); } #+end_src We’ve renamed the original =call_once= method to =call_once_slow=, and annotated it with the =#[cold]= attribute to tell the compiler that any branch that calls this method is unlikely to be taken. #+begin_src rust #[cold] pub fn call_once_slow(&self, ignore_poison: bool, f: F) where F: FnOnce(&OnceState), { ... } #+end_src ** =OnceLock= & =LazyLock= Whereas a =Once= is a simple flag, the =OnceLock= builds upon it to explicitly scope the initialisation of a wrapped resource: #+begin_src rust pub struct OnceLock { once: Once, data: UnsafeCell>, } #+end_src Since our resource of type =T= starts out uninitialised but we still need to keep enough space around for it, we make use of the =MaybeUninit= type inside of an =UnsafeCell=. The most important method of =OnceLock= is =initialize=, which takes a fallible closure to initialise the resource: #+begin_src rust pub fn initialize(&self, f: F) -> Result<(), E> where F: FnOnce() -> Result, { let mut res: Result<(), E> = Ok(()); let cell = unsafe { &mut *self.data.get() }; self.once.call_once_force(|state| match f() { Ok(value) => { unsafe { cell.as_mut_ptr().write(value) }; } Err(e) => { state.poison(); res = Err(e); } }); res } #+end_src The rest of the type is various helper methods to access the underlying resource and to make common patterns more convenient. In the =Drop= implementation, we make sure to only drop the resource if it was actually initialised: #+begin_src rust impl Drop for OnceLock { fn drop(&mut self) { if self.is_completed() { unsafe { self.data.get().as_mut_unchecked().assume_init_drop() } } } } #+end_src The =LazyLock= type is a more special variant of =OnceLock=, in which the initialisation closure is always the same, infallible, and provided at construction time: #+begin_src rust pub struct LazyLock T> { once: Once, f: UnsafeCell>, t: UnsafeCell>, } #+end_src Since we don’t have to provide a closure in order to unconditionally receive a reference to the initialised resource, =LazyLock= can implement the =Deref= and =DerefMut= traits to provide more ergonomic access to the wrapped value. Here, we take advantage of the fact that =Once::call_once= will panic if a previous initialisation poisoned the =Once= in order to guarantee that we only ever take the initialisation closure out of the =ManuallyDrop= once: #+begin_src rust impl T> LazyLock { fn force(this: &Self) -> &T { this.once.call_once(|_| { // SAFETY: because `call_once` will panic if poisoned, this // closure will only be called once. let f = unsafe { ManuallyDrop::take(&mut *this.f.get()) }; let val = f(); unsafe { (&mut *this.t.get()).write(val) }; }); unsafe { (&*this.t.get()).assume_init_ref() } } } impl T> core::ops::Deref for LazyLock { type Target = T; fn deref(&self) -> &Self::Target { LazyLock::force(self) } } #+end_src In the =Drop= implementation of =LazyLock=, we make sure only to drop the initialisation closure or the value if we can say for certain that they are still present and initialised, respectively: #+begin_src rust impl Drop for LazyLock { fn drop(&mut self) { match self.once.state() { UNINITIALIZED => unsafe { ManuallyDrop::drop(self.f.get_mut()); }, INITIALIZED => unsafe { (&mut *self.t.get()).assume_init_drop(); }, _ => {} } } } #+end_src Once again, we have to explicitly implement the =Sync= trait for both =OnceLock= and =LazyLock=: since neither type contains any way for a user to create a mutable reference to the wrapped resource without a mutable reference to the lock itself, we can safely implement =Sync= for both types as long as the wrapped resource is also =Sync=. #+begin_src rust unsafe impl Sync for OnceLock {} unsafe impl Sync for LazyLock {} #+end_src In the case of =LazyLock=, the bound =F: Send= is sufficient, since the initialisation closure is only ever called once.