post 1 partial

blog/intro.org

@@ -0,0 +1,261 @@
+* Writing an x86-64 OS (mostly) from scratch in Rust
+
+  I’ve always been fascinated by the idea of bootstrapping.
+  Combined with a curiosity for learning about how things work, this has lead to me often asking the question: if all I had were my hands, how far into the “tech-tree” of humanity could I get? The answer, unfortunately, is likely not very far at all. Certainly, I would never be able to put together any sort of machine that resembles what we today understand to be a computer.
+  The term “bootstrapping” famously comes from the saying “pull oneself up by one’s bootstraps”, which is rarely understood for what it actually is: an impossible task, a paradox.
+  So, when tackling the problem of bootstrapping anything computer related, it is important to pick ones battles, and know where to start and what the goal is.
+  For most people, for economic reasons, this means starting with a computer and a good bit of software already in place.
+  Still, there are many paths one can take to either fulfil ones curiosity about how things work, or to feel the satisfaction of having something that are built by yourself.
+  One might, from there, attempt to bootstrap a compiler (typically a simple C compiler), starting from nothing but the simplest hand-written binary blob that can turn a text file containing hex digits into a binary executable, and then build on that, step by step, every successive tool becoming more complex and more ergonomic, until, ex-nihil, a C compiler has appeared which is capable of building itself, as well as old versions of binutils and the gcc toolchain. From there, it’s possible to build every piece of software found on the modern Linux system.
+The journey that I will hopefully document in this blog, both learning my self and hopefully acting as a reference for others also interested in the same topic, will be the building of a simple operating system for the average modern computer running the x86-64 architecture.
+  Instead of, how the little purist voice in my head demands, starting, as described above, from nothing and writing my own compiler and language, I will instead use the awesome Rust programming language.
+This is both because writing an assembler and multiple compilers from scratch is a lot of very tedious work, and because Rust is, in my opinion, the best language available for the task at hand.
+
+* Initial Setup
+
+  We will start by setting up a development environment using the equally awesome nix package manager, which allows us to declaratively specify which software we need and have them made available to us on any system where nix is available.
+
+  We will be using nix flakes, and our initial flake.nix will look like this:
+  #+begin_src nix
+  {
+    description = "An x86 kernel written in Rust";
+
+    inputs = {
+      nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
+      rust-overlays.url = "github:oxalica/rust-overlay";
+      flake-utils.url = "github:numtide/flake-utils";
+    };
+
+    outputs = { nixpkgs, flake-utils, rust-overlays, ...}: 
+      flake-utils.lib.eachDefaultSystem (system: let
+        overlays = [
+          (import rust-overlays)
+        ];
+        pkgs = import nixpkgs {
+          inherit system overlays;
+          config.allowUnfree = true;
+        };
+
+        rust-override = {
+          extensions = ["rust-analyzer" "rust-src"];
+          targets = [ "x86_64-unknown-linux-gnu" ];
+        };
+
+        rust-pkg = pkgs.rust-bin.nightly.latest.default.override rust-override;
+      in with pkgs; {
+        devShells = {
+          default = mkShell rec {
+            name = "rust-osdev";
+
+            nativeBuildInputs = [
+              pkg-config
+              rust-pkg
+              clang
+              gcc
+              mold
+            ];
+
+            buildInputs = [
+              udev stdenv.cc.cc.lib
+            ];
+
+            LD_LIBRARY_PATH = lib.makeLibraryPath buildInputs;
+          };
+        };
+      });
+  }
+  #+end_src
+This is a lot of code to essentially just say that we would like to have the latest nightly version of Rust as well as a standard collection of build tools available.
+
+To automatically load the proper environment when entering the project directory, we will also create a .envrc file with the following contents:
+  #+begin_src bash
+use flake .
+  #+end_src
+and then run the following command to allow direnv to load the environment:
+  #+begin_src bash
+direnv allow
+  #+end_src
+
+Now we can start writing the first bit of rust code:
+  #+begin_src bash
+cargo new --bin kernel
+  #+end_src
+
+
+Most software will at some point want to make use of or work together in some way with other software running on the same machine, including most certainly the kernel, the very thing we are trying to replace.
+In Rust, the mechanisms for interfacing with other software or the kernel are provided by the standard library =std=. =std= itself depends on a number of libraries which are not available to us, in our bare-metal, operating system-less environment.
+To tell the rust compiler that we don’t want to use the standard library, we can use the crate-level attribute =no_std=; we will still have access to a large portion of the =std= library via the =core= and =alloc= libraries, which =std= typically re-exports.
+
+A typical Rust program starts in the =main= function, but this is not actually where the final program itself starts running from. Before the users =main= function, things like environment variables and command line arguments are set up.
+
+Since there are no command line arguments or environment variables for our kernel, and because the scaffolding around the special =main= function won’t be available for us either, we will need to tell the compiler additionally that we have no =main= function. This is done with the crate-level attribute =no_main=.
+We will need to provide the entry point for the linker ourselves, which is canonically named =_start=.
+We annotate this function with the unsafe =no_mangle= attribute which tells the compiler to keep the name of the function as is so that the linker can find it.
+We could also have used the =#[export_name = "_start"]= attribute and called the function something else, such as =kernel_main=.
+
+For now, our =_start= function will do nothing, which we can accomplish by “spinning” in an infinite loop.
+
+#+begin_src rust
+#![no_std]
+#![no_main]
+
+#[unsafe(no_mangle)]
+fn _start() -> ! {
+    loop {}
+}
+#+end_src
+
+As is, attempting to build this code will throw linker errors, because cargo will still attempt to link with its own =_start= function and the aforementioned scaffolding around the =main= function. To tell cargo off, we will define our own custom target specification file which will also allow us to conveniently special-case the build for our kernel via cargos configuration later.
+
+#+begin_src json
+{
+  "arch": "x86_64",
+  "code-model": "kernel",
+  "cpu": "x86-64",
+  "crt-objects-fallback": "false",
+  "data-layout": "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-i128:128-f80:128-n8:16:32:64-S128",
+  "disable-redzone": true,
+  "target-endian": "little",
+  "features": "-mmx,-sse,-sse2,-sse3,-ssse3,-sse4.1,-sse4.2,-avx,-avx2,+soft-float",
+  "linker": "mold",
+  "linker-flavor": "gnu",
+  "llvm-target": "x86_64-unknown-none-elf",
+  "max-atomic-width": 64,
+  "panic-strategy": "abort",
+  "plt-by-default": false,
+  "position-independent-executables": true,
+  "relro-level": "full",
+  "rustc-abi": "softfloat",
+  "stack-probes": {
+    "kind": "inline"
+  },
+  "static-position-independent-executables": true,
+  "supported-sanitizers": [
+    "kcfi",
+    "kernel-address"
+  ],
+  "target-pointer-width": 64
+}
+#+end_src
+
+Most of these fields are taken from the the =x86_64-unknown-none-elf= target, which is easily retrieved by running the following command:
+  #+begin_src bash
+rustc -Z unstable-options --print target-spec-json --target x86_64-unknown-none
+  #+end_src
+
+More options available in the current rustc can be found by running the following command with a nightly version of rustc:
+  #+begin_src bash
+rustc -Zunstable-options --print target-spec-json-schema
+  #+end_src
+
+If we now attempt to build our kernel, cargo will now complain that it cannot find the =core= crate since it is not available for our custom target:
+  #+begin_src bash
+cargo -Zjson-target-spec build --target x86_64-unknown-kernel.json
+  #+end_src
+
+
+  Even though we’ve told the compiler we don’t want the standard library, we will still have to build it for our target to make available the =core= and =alloc= libraries.
+  Since we added the =rust-src= component when installing Rust via nix, we can now simply add a =.cargo/config.toml= file to tell cargo to build =std= for us and select the appropriate features:
+  #+begin_src toml
+[unstable]
+json-target-spec = true # lets us specify a custom target specification file
+build-std = ["core", "compiler_builtins"]
+#+end_src
+
+Rust permits the user to abort the program at any time by calling the =panic!= macro, or a function that “panics” internally. Typically, the mechanism by which this happens, as well as the more complex behaviour of unwinding the stack and catching unwinds, is provided by the =std=. Since we’ve opted out of using =std=, we will need to provide a panic handler for rust to call in case of a panic. Unwinding is already disabled by the =panic-strategy= field in our target specification file.
+
+For the time being, we will again simply spin in an infinite loop whenever a panic occurs:
+  #+begin_src rust
+  #[panic_handler]
+  fn panic(_info: &core::panic::PanicInfo) -> ! {
+      loop {}
+  }
+#+end_src
+
+Both the panic handler and our entry point have so far made use of the unstable =never= type =!=, which can be used stabily only as the return type of functions that never return. =never= is Rust’s uninstantiable type which can coerce to any other type: it is the subtype of every other type.
+=!= is returned by a handful of expressions that either affect control flow such as =return expr=, =continue= or =break expr= and so mean that any assignment of the value of such an expression becomes unreachable, or expressions that never return such as =panic!= or a =loop= with no break.
+
+By annotating our functions with the =!= return type, the compiler will ensure that we can’t accidentally write code that does return: because =!= is uninstantiable, no value can coerce to it, and so any expression that yields a value would cause a compiler error.
+
+Finally, our kernel builds without further errors.
+Since cargo produces an ELF binary, we can even try it out it by running
+  #+begin_src bash
+cargo run --target x86_64-unknown-kernel.json
+ #+end_src
+Of course, all it will do is nothing, indefinitely.
+
+The next step will be to get our kernel to run in a virtual machine, which will stand in for a real computer and permit us to start doing anything proper kernel-like that would be forbidden in user-space.
+
+There are a number of strategies for getting a computer to run our kernel, but we will be using the limine bootloader, as it greatly simplifies the code required to get our kernel running, as well as starting us off already in 64-bit mode and mapped into the higher-half of the address space.
+
+Since the launch of Windows 8 in 2012, Microsoft has required that all computers ship with UEFI firmware; as a consequence, almost all modern x86-64 computers will have UEFI firmware.
+Another strategy would be to act as a UEFI application and letting the firmware load our kernel without bothering with a bootloader, however UEFI is a very complex and heavy specification, requiring a lot more code to get to the fun part of writing our operating system, so instead we will let limine deal with the UEFI part of booting.
+
+Using either limine or UEFI also means we skip over the legacy BIOS boot process which involves setting up the CPU in 16-bit real mode, and switching to 32-bit protected mode before finally switching to 64-bit long mode, which is the mode we are interested in.
+
+We will need to add a few additional dependencies to our =flake.nix=:
+  #+begin_src nix
+  nativeBuildInputs = [
+    ...
+    qemu # the virtual machine
+    OVMF # UEFI firmware for qemu
+    limine-full # the bootloader
+    mtools # tools for creating a FAT filesystem image
+  ];
+
+  # path to the UEFI firmware binary exposed as an environment variable
+  OVMF_PATH = OVMF.fd;
+  #+end_src
+
+We will create a =run.sh= script to automate hide away the details of launching the virtual machine and setting up the bootloader and kernel image:
+  #+begin_src bash
+  #!/usr/bin/env bash
+
+  set -e
+
+  bin="$1"
+
+  canonicalised=$(realpath -- "$bin")
+  parent=$(dirname -- "$canonicalised")
+  bin_name=$(basename -- "$exe")
+  bin_name=${bin_name%.*}
+
+  image="$parent/$bin_name.img"
+
+
+  cat > /tmp/limine.conf <<EOF
+  timeout: 0
+  serial: yes
+  serial_baudrate: 9600
+
+  /Kernel
+      protocol: limine
+      path: boot():/boot/kernel.elf
+  EOF
+
+  if [[ "$canonicalised" -nt "$image" ]]; then
+      echo "Creating image $image"
+
+      limine_datadir="$(limine --print-datadir)"
+      limine_uefi_path="$limine_datadir/BOOTX64.EFI"
+
+      dd if=/dev/zero of="$image" bs=1M count=64
+      mformat -i "$image" -F ::
+
+      mmd -i "$image" ::/boot
+      mmd -i "$image" ::/EFI
+      mmd -i "$image" ::/EFI/BOOT
+
+      mcopy -i "$image" /tmp/limine.conf ::/boot/limine.conf
+      mcopy -i "$image" "$canonicalised" ::/boot/kernel.elf
+      mcopy -i "$image" "$limine_uefi_path" ::/EFI/BOOT/BOOTX64.EFI
+  fi
+
+  qemu-system-x86_64 \
+      -drive file="$image",format=raw \
+      -machine q35,accel=kvm -enable-kvm \
+      -drive if=pflash,format=raw,readonly=on,file="$OVMF_PATH/FV/OVMF_CODE.fd" \
+      -serial stdio \
+      -vga std
+      
+ #+end_src

flake.nix

@@ -35,6 +35,7 @@
             gcc
             mold
 
+            limine-full
             qemu
             OVMF
             mtools

kernel/.cargo/config.toml

@@ -0,0 +1,12 @@
+[profile.dev]
+codegen-backend = "llvm"
+
+[unstable]
+json-target-spec = true # lets us specify a custom target specification file
+build-std = ["core", "compiler_builtins"]
+
+[build]
+target = "x86_64-unknown-kernel.json"
+
+[target.x86_64-unknown-kernel]
+runner = "./run.sh"

kernel/Cargo.lock

@@ -0,0 +1,7 @@
+# This file is automatically @generated by Cargo.
+# It is not intended for manual editing.
+version = 4
+
+[[package]]
+name = "kernel"
+version = "0.1.0"

kernel/Cargo.toml

@@ -0,0 +1,6 @@
+[package]
+name = "kernel"
+version = "0.1.0"
+edition = "2024"
+
+[dependencies]

kernel/run.sh

@@ -0,0 +1,48 @@
+#!/usr/bin/env bash
+
+set -e
+
+bin="$1"
+
+canonicalised="$(realpath -- "$bin")"
+parent="$(dirname -- "$canonicalised")"
+bin_name="$(basename -- "$canonicalised")"
+
+image="$parent/$bin_name.img"
+
+
+cat > /tmp/limine.conf <<EOF
+timeout: 0
+serial: yes
+serial_baudrate: 9600
+
+/Kernel
+    protocol: limine
+    path: boot():/boot/kernel.elf
+EOF
+
+if [[ "$canonicalised" -nt "$image" ]]; then
+    echo "Creating image $image"
+
+    limine_datadir="$(limine --print-datadir)"
+    limine_uefi_path="$limine_datadir/BOOTX64.EFI"
+
+    dd if=/dev/zero of="$image" bs=1M count=64
+    mformat -i "$image" -F ::
+
+    mmd -i "$image" ::/boot
+    mmd -i "$image" ::/EFI
+    mmd -i "$image" ::/EFI/BOOT
+
+    mcopy -i "$image" /tmp/limine.conf ::/boot/limine.conf
+    mcopy -i "$image" "$canonicalised" ::/boot/kernel.elf
+    mcopy -i "$image" "$limine_uefi_path" ::/EFI/BOOT/BOOTX64.EFI
+fi
+
+qemu-system-x86_64 \
+    -drive file="$image",format=raw \
+    -machine q35,accel=kvm -enable-kvm \
+    -drive if=pflash,format=raw,readonly=on,file="$OVMF_PATH/FV/OVMF_CODE.fd" \
+    -serial stdio \
+    -vga std
+

kernel/src/main.rs

@@ -0,0 +1,12 @@
+#![no_std]
+#![no_main]
+
+#[panic_handler]
+fn panic(_info: &core::panic::PanicInfo) -> ! {
+    loop {}
+}
+
+#[unsafe(no_mangle)]
+fn _start() -> ! {
+    loop {}
+}

kernel/x86_64-unknown-kernel.json

@@ -0,0 +1,28 @@
+{
+  "arch": "x86_64",
+  "code-model": "kernel",
+  "cpu": "x86-64",
+  "crt-objects-fallback": "false",
+  "data-layout": "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-i128:128-f80:128-n8:16:32:64-S128",
+  "disable-redzone": true,
+  "target-endian": "little",
+  "features": "-mmx,-sse,-sse2,-sse3,-ssse3,-sse4.1,-sse4.2,-avx,-avx2,+soft-float",
+  "linker": "mold",
+  "linker-flavor": "gnu",
+  "llvm-target": "x86_64-unknown-none-elf",
+  "max-atomic-width": 64,
+  "panic-strategy": "abort",
+  "plt-by-default": false,
+  "position-independent-executables": true,
+  "relro-level": "full",
+  "rustc-abi": "softfloat",
+  "stack-probes": {
+    "kind": "inline"
+  },
+  "static-position-independent-executables": true,
+  "supported-sanitizers": [
+    "kcfi",
+    "kernel-address"
+  ],
+  "target-pointer-width": 64
+}