curiOS/blog/intro.org at ba089a5169e8b86c3550544f95169d97a9e6f5e6

janis/curiOS

Fork 0

janis ba089a5169

elaborate/update json in blog

2026-07-17 16:22:42 +02:00

20 KiB

Raw Blame History

Writing an x86-64 OS (mostly) from scratch in Rust
Initial Setup
Interlude: Welcome to kernel-land
Getting the Framebuffer

Writing an x86-64 OS (mostly) from scratch in Rust

I’ve always been fascinated by the idea of bootstrapping. Combined with a curiosity for learning about how things work, this has lead to me often asking the question: if all I had were my hands, how far into the “tech-tree” of humanity could I get? The answer, unfortunately, is likely not very far at all. Certainly, I would never be able to put together any sort of machine that resembles what we today understand to be a computer. The term “bootstrapping” famously comes from the saying “pull oneself up by one’s bootstraps”, which is rarely understood for what it actually is: an impossible task, a paradox. So, when tackling the problem of bootstrapping anything computer related, it is important to pick ones battles, and know where to start and what the goal is. For most people, for economic reasons, this means starting with a computer and a good bit of software already in place. Still, there are many paths one can take to either fulfil ones curiosity about how things work, or to feel the satisfaction of having something that are built by yourself. One might, from there, attempt to bootstrap a compiler (typically a simple C compiler), starting from nothing but the simplest hand-written binary blob that can turn a text file containing hex digits into a binary executable, and then build on that, step by step, every successive tool becoming more complex and more ergonomic, until, ex-nihil, a C compiler has appeared which is capable of building itself, as well as old versions of binutils and the gcc toolchain. From there, it’s possible to build every piece of software found on the modern Linux system. The journey that I will hopefully document in this blog, both learning my self and hopefully acting as a reference for others also interested in the same topic, will be the building of a simple operating system for the average modern computer running the x86-64 architecture. Instead of, how the little purist voice in my head demands, starting, as described above, from nothing and writing my own compiler and language, I will instead use the awesome Rust programming language. This is both because writing an assembler and multiple compilers from scratch is a lot of very tedious work, and because Rust is, in my opinion, the best language available for the task at hand.

Initial Setup

We will start by setting up a development environment using the equally awesome nix package manager, which allows us to declaratively specify which software we need and have them made available to us on any system where nix is available.

We will be using nix flakes, and our initial flake.nix will look like this:

{
  description = "An x86 kernel written in Rust";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    rust-overlays.url = "github:oxalica/rust-overlay";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = { nixpkgs, flake-utils, rust-overlays, ...}: 
    flake-utils.lib.eachDefaultSystem (system: let
      overlays = [
        (import rust-overlays)
      ];
      pkgs = import nixpkgs {
        inherit system overlays;
        config.allowUnfree = true;
      };

      rust-override = {
        extensions = ["rust-analyzer" "rust-src"];
        targets = [ "x86_64-unknown-linux-gnu" ];
      };

      rust-pkg = pkgs.rust-bin.nightly.latest.default.override rust-override;
    in with pkgs; {
      devShells = {
        default = mkShell rec {
          name = "rust-osdev";

          nativeBuildInputs = [
            pkg-config
            rust-pkg
            clang
            gcc
            mold
          ];

          buildInputs = [
            udev stdenv.cc.cc.lib
          ];

          LD_LIBRARY_PATH = lib.makeLibraryPath buildInputs;
        };
      };
    });
}

This is a lot of code to essentially just say that we would like to have the latest nightly version of Rust as well as a standard collection of build tools available.

To automatically load the proper environment when entering the project directory, we will also create a .envrc file with the following contents:

use flake .

and then run the following command to allow direnv to load the environment:

direnv allow

Now we can start writing the first bit of rust code:

cargo new --bin kernel

Most software will at some point want to make use of or work together in some way with other software running on the same machine, including most certainly the kernel, the very thing we are trying to replace. In Rust, the mechanisms for interfacing with other software or the kernel are provided by the standard library std. std itself depends on a number of libraries which are not available to us, in our bare-metal, operating system-less environment. To tell the rust compiler that we don’t want to use the standard library, we can use the crate-level attribute no_std; we will still have access to a large portion of the std library via the core and alloc libraries, which std typically re-exports.

A typical Rust program starts in the main function, but this is not actually where the final program itself starts running from. Before the users main function, things like environment variables and command line arguments are set up.

Since there are no command line arguments or environment variables for our kernel, and because the scaffolding around the special main function won’t be available for us either, we will need to tell the compiler additionally that we have no main function. This is done with the crate-level attribute no_main. We will need to provide the entry point for the linker ourselves, which is canonically named _start. We annotate this function with the unsafe no_mangle attribute which tells the compiler to keep the name of the function as is so that the linker can find it. We could also have used the #[export_name = "_start"] attribute and called the function something else, such as kernel_main.

For now, our _start function will do nothing, which we can accomplish by “spinning” in an infinite loop.

#![no_std]
#![no_main]

#[unsafe(no_mangle)]
fn _start() -> ! {
    loop {}
}

As is, attempting to build this code will throw linker errors, because cargo will still attempt to link with its own _start function and the aforementioned scaffolding around the main function. To tell cargo off, we will define our own custom target specification file which will also allow us to conveniently special-case the build for our kernel via cargos configuration later.

{
    "arch": "x86_64",
    "code-model": "kernel",
    "cpu": "x86-64",
    "crt-objects-fallback": "false",
    "data-layout": "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-i128:128-f80:128-n8:16:32:64-S128",
    "disable-redzone": true,
    "target-endian": "little",
    "features": "-mmx,-sse,-sse2,-sse3,-ssse3,-sse4.1,-sse4.2,-avx,-avx2,+soft-float",
    "linker": "rust-lld",
    "linker-flavor": "ld.lld",
    "llvm-target": "x86_64-unknown-none-elf",
    "max-atomic-width": 64,
    "panic-strategy": "abort",
    "plt-by-default": false,
    "os": "none",
    "executables": true,
    "relro-level": "full",
    "rustc-abi": "softfloat",
    "stack-probes": {
        "kind": "inline"
    },
    "supported-sanitizers": [
        "kcfi",
        "kernel-address"
    ],
    "target-pointer-width": 64,
    "target-c-int-width": 32
}

Most of these fields are taken from the the x86_64-unknown-none-elf target, which is easily retrieved by running the following command:

rustc -Z unstable-options --print target-spec-json --target x86_64-unknown-none

More options available in the current rustc can be found by running the following command with a nightly version of rustc:

rustc -Zunstable-options --print target-spec-json-schema

Notably, I have not copied the *position-independent-executables fields from the x86_64-unknown-none-elf target, because we will later use a linker script to place our kernel in the higher-half of the address space and modify the layout of our sections manually, and position independent executables complicate this.

If we now attempt to build our kernel, cargo will now complain that it cannot find the core crate since it is not available for our custom target:

cargo -Zjson-target-spec build --target x86_64-unknown-kernel.json

Even though we’ve told the compiler we don’t want the standard library, we will still have to build it for our target to make available the core and alloc libraries. Since we added the rust-src component when installing Rust via nix, we can now simply add a .cargo/config.toml file to tell cargo to build std for us and select the appropriate features:

[unstable]
json-target-spec = true # lets us specify a custom target specification file
build-std = ["core", "compiler_builtins"]

Rust permits the user to abort the program at any time by calling the panic! macro, or a function that “panics” internally. Typically, the mechanism by which this happens, as well as the more complex behaviour of unwinding the stack and catching unwinds, is provided by the std. Since we’ve opted out of using std, we will need to provide a panic handler for rust to call in case of a panic. Unwinding is already disabled by the panic-strategy field in our target specification file.

For the time being, we will again simply spin in an infinite loop whenever a panic occurs:

#[panic_handler]
fn panic(_info: &core::panic::PanicInfo) -> ! {
    loop {}
}

Both the panic handler and our entry point have so far made use of the unstable never type !, which can be used stabily only as the return type of functions that never return. never is Rust’s uninstantiable type which can coerce to any other type: it is the subtype of every other type. ! is returned by a handful of expressions that either affect control flow such as return expr, continue or break expr and so mean that any assignment of the value of such an expression becomes unreachable, or expressions that never return such as panic! or a loop with no break.

By annotating our functions with the ! return type, the compiler will ensure that we can’t accidentally write code that does return: because ! is uninstantiable, no value can coerce to it, and so any expression that yields a value would cause a compiler error.

Finally, our kernel builds without further errors. Since cargo produces an ELF binary, we can even try it out it by running

cargo run --target x86_64-unknown-kernel.json

Of course, all it will do is nothing, indefinitely.

The next step will be to get our kernel to run in a virtual machine, which will stand in for a real computer and permit us to start doing anything proper kernel-like that would be forbidden in user-space.

There are a number of strategies for getting a computer to run our kernel, but we will be using the limine bootloader, as it greatly simplifies the code required to get our kernel running, as well as starting us off already in 64-bit mode and mapped into the higher-half of the address space.

Since the launch of Windows 8 in 2012, Microsoft has required that all computers ship with UEFI firmware; as a consequence, almost all modern x86-64 computers will have UEFI firmware. Another strategy would be to act as a UEFI application and letting the firmware load our kernel without bothering with a bootloader, however UEFI is a very complex and heavy specification, requiring a lot more code to get to the fun part of writing our operating system, so instead we will let limine deal with the UEFI part of booting.

Using either limine or UEFI also means we skip over the legacy BIOS boot process which involves setting up the CPU in 16-bit real mode, and switching to 32-bit protected mode before finally switching to 64-bit long mode, which is the mode we are interested in.

We will need to add a few additional dependencies to our flake.nix:

nativeBuildInputs = [
  ...
  qemu # the virtual machine
  OVMF # UEFI firmware for qemu
  limine-full # the bootloader
  mtools # tools for creating a FAT filesystem image
];

# path to the UEFI firmware binary exposed as an environment variable
OVMF_PATH = OVMF.fd;

We will create a run.sh script to automate hide away the details of launching the virtual machine and setting up the bootloader and kernel image:

#!/usr/bin/env bash

set -e

bin="$1"

canonicalised=$(realpath -- "$bin")
parent=$(dirname -- "$canonicalised")
bin_name=$(basename -- "$exe")
bin_name=${bin_name%.*}

image="$parent/$bin_name.img"


cat > /tmp/limine.conf <<EOF
timeout: 0
serial: yes
serial_baudrate: 9600

/Kernel
    protocol: limine
    path: boot():/boot/kernel.elf
EOF

if [[ "$canonicalised" -nt "$image" ]]; then
    echo "Creating image $image"

    limine_datadir="$(limine --print-datadir)"
    limine_uefi_path="$limine_datadir/BOOTX64.EFI"

    dd if=/dev/zero of="$image" bs=1M count=64
    mformat -i "$image" -F ::

    mmd -i "$image" ::/boot
    mmd -i "$image" ::/EFI
    mmd -i "$image" ::/EFI/BOOT

    mcopy -i "$image" /tmp/limine.conf ::/boot/limine.conf
    mcopy -i "$image" "$canonicalised" ::/boot/kernel.elf
    mcopy -i "$image" "$limine_uefi_path" ::/EFI/BOOT/BOOTX64.EFI
fi

qemu-system-x86_64 \
    -drive file="$image",format=raw \
    -machine q35,accel=kvm -enable-kvm \
    -drive if=pflash,format=raw,readonly=on,file="$OVMF_PATH/FV/OVMF_CODE.fd" \
    -serial stdio \
    -vga std

Lets go through the script step by step: The first like after the shebang that tells the loader to use bash as the interpreter is set -e, which means the script will exit immediately if any command returns a failure exit code. This means that, if we for example don't have limine installed, the script won't try to create the image or run qemu with an incomplete setup.

Our script currently takes a single argument, the path to the kernel binary. Cargo allows us to set a "runner" command in the .cargo/config.toml file, which it will call with the path to the binary as the first argument instead of the calling the binary itself. We will make use of this feature to automatically start our kernel in qemu with the help of cargo.

We take the first argument and extract the proper parent directory and the filename and use both to construct the path to the image file we create in the next step:

bin="$1"

canonicalised=$(realpath -- "$bin")
parent=$(dirname -- "$canonicalised")
bin_name=$(basename -- "$exe")

image="$parent/$bin_name.img"

Next, we check if a file exists at the path we constructed for the image file, and whether it is older than the kernel binary:

if [[ "$canonicalised" -nt "$image" ]]; then

If so, we will want to (re)create the image, starting with the limine.conf file the limine bootloader will read to know where to find our kernel binary:

cat > /tmp/limine.conf <<EOF
timeout: 0
serial: yes
serial_baudrate: 9600

/Kernel
    protocol: limine
    path: boot():/boot/kernel.elf
EOF

We also tell limine to use the serial port for output, which means any error messages will be printed in the terminal we run our script (or cargo run ) in.

Next, we create a 64 MiB image file filled with zeroes, and format it with mformat to create a FAT32 filesystem. After creating the directory structure using mmd, we copy the relevant files, including the BOOTX64.EFI bootloader binary from limine into the respective paths in the image file using mcopy.

limine_datadir="$(limine --print-datadir)"
limine_uefi_path="$limine_datadir/BOOTX64.EFI"

dd if=/dev/zero of="$image" bs=1M count=64
mformat -i "$image" -F ::

mmd -i "$image" ::/boot
mmd -i "$image" ::/EFI
mmd -i "$image" ::/EFI/BOOT

mcopy -i "$image" /tmp/limine.conf ::/boot/limine.conf
mcopy -i "$image" "$canonicalised" ::/boot/kernel.elf
mcopy -i "$image" "$limine_uefi_path" ::/EFI/BOOT/BOOTX64.EFI

Finally, we run qemu with the image attached as a drive:

qemu-system-x86_64 \
    -drive file="$image",format=raw \
    -machine q35,accel=kvm -enable-kvm \
    -drive if=pflash,format=raw,readonly=on,file="$OVMF_PATH/FV/OVMF_CODE.fd" \
    -serial stdio

We tell qemu to use the q35 machine type, which is a more modern virtual motherboard and is noticably faster at startup in my experience. Additionally, we tell qemu to use KVM acceleration, which will allow the virtual machine to run at near-native speed by utilising the CPUs hardware virtualization features rather than emulating the CPU in software. This requires that the host CPU supports virtualization and that KVM is enabled in the host linux system. We also attach the UEFI firmware and the serial port we've told limine to write to.

Finally, we make a few changes to our .cargo/config.toml file to tell cargo to use our run.sh script as the runner for our kernel binary:

[build]
target = "x86_64-unknown-kernel.json"

[target.x86_64-unknown-kernel]
runner = "./run.sh"

This tells cargo to use our custom target specification file by default when building, and to use our run.sh script as the runner for any binary built for our custom target.

If we now run cargo run, cargo will build our kernel and automatically start qemu. Unfortunately, since our kernel currently does nothing, this is hard to discern. Therefore, our next goal will be to get some output from our kernel, which we will do by making use of limine’s framebuffer.

Interlude: Welcome to kernel-land

If you have an aggressive LSP client, or you ran the cargo clippy command on our current crate, you may have noticed a warning telling you that an empty loop wastes CPU cycles. This is true, and is in fact the very reason why we use it! However, now that we have arrived in kernel-land, we can actually do a little better: the hlt instruction which, according to the Intel manual, stops execution of a logical processor until further notice. Further notice here may be an interrupt, which means we will want to run the hlt instruction in a loop. hlt is one of a handful of privileged instructions that when attempted to be executed in user-space will cause a general protection fault and terminate the program.

We start by creating a lib.rs file in the src directory, which will contain our kernel library code. Just line in our main.rs, we will need to tell the compiler that we want to opt out of the standard library:

#![no_std]

Since the hlt instruction is specific to the x86 architecture and not available in the core::arch::x86_64 module of the core library, we will need to write our own wrapper around some inline assembly to execute the instruction. We create a new x86_64 module in our new library to house this and future x86-64 specific code:

#![no_std]

pbu mod x86_64;

#![cfg(target_arch = "x86_64")]

#[inline]
pub fn hlt() {
  unsafe {
      core::arch::asm!("hlt", options(nomem, nostack, preserves_flags));
  }
}

pub fn halt_loop() -> ! {
  loop {
      hlt();
  }
}

For a detailed explanation of the asm! macro, see the Rust reference.

In our main.rs, we can now replace the infinite loop with a call to our new halt_loop function:

#[panic_handler]
fn panic(_info: &core::panic::PanicInfo) -> ! {
  kernel::x86_64::halt_loop()
}

#[unsafe(no_mangle)]
fn _start() -> ! {
  kernel::x86_64::halt_loop()
}

20 KiB Raw Blame History Unescape Escape

Writing an x86-64 OS (mostly) from scratch in Rust

Initial Setup

Interlude: Welcome to kernel-land

Getting the Framebuffer

20 KiB

Raw Blame History