diff --git a/blog/intro.org b/blog/intro.org index 242e159..3e0ec57 100644 --- a/blog/intro.org +++ b/blog/intro.org @@ -369,7 +369,7 @@ Therefore, our next goal will be to get some output from our kernel, which we wi * Interlude: Welcome to kernel-land If you have an aggressive LSP client, or you ran the =cargo clippy= command on our current crate, you may have noticed a warning telling you that an empty loop wastes CPU cycles. This is true, and is in fact the very reason why we use it! - However, now that we have arrived in kernel-land, we can actually do a little better: the =hlt= instruction which, according to the Intel manual, stops execution of a logical processor until further notice. + However, now that we have arrived in kernel-land, we can actually do a little better: the =hlt= instruction, which, according to the Intel manual, stops execution of a logical processor until further notice. Further notice here may be an interrupt, which means we will want to run the =hlt= instruction in a loop. =hlt= is one of a handful of privileged instructions that when attempted to be executed in user-space will cause a general protection fault and terminate the program. @@ -385,7 +385,7 @@ Therefore, our next goal will be to get some output from our kernel, which we wi #+begin_src rust #![no_std] -pbu mod x86_64; +pub mod x86_64; #+end_src #+begin_src rust @@ -735,3 +735,412 @@ The same is true also for the =Request= type, except that we have to bound the i kernel::x86_64::halt_loop() } #+end_src + +* Interlude 2: Sync + We’ve managed to get something on the screen and confirmed that our kernel is running and working as intended, but what if something had gone wrong? + + We have used assertions of some kind multiple times, to enforce invariants our code relies on, or to =expect= the presence of a framebuffer. + + If those assertions fail, rust will call our panic handler and abort the intended execution of our kernel, but our panic handler currently just silently swallows the panic message and spins. + + If we want some way to communicated with the host, the most straightforward way is to use a serial port, just like we told limine to do for its error output. + + Before that, however, I want to tackle a problem that we’ve just encountered with the framebuffer, and which will be even more relevant for the serial port: synchronisation. + + One of Rust’s core strengths, besides its enforcement of memory safety, is its robust type and trait system to protect against concurrency bugs. + + Currently, our kernel is single-threaded, so the compilers pedantism about the thread-safety of the framebuffer request feels like a nuisance, but since we do plan on eventually running on multiple cores and threads, it is better seen as an opportinity to build the necessary framework from the very beginning. + + Whether we want to use global singletons or pass around one large context object, we will eventually want to share access to specific resources requiring explicit synchronisation between threads or cores, and for that we will need primitives like mutexes and locks. + + In this section we will implement a simple spinlock-based =Mutex=, a =Once= type for synchronising one-time initialisation, and the =OnceLock= and =LazyLock= types built on top of =Once=. + All of these types are typically found in the =std::sync= module, but those implementations rely on the operating system to provide the necessary synchronisation primitives such as futexes (fast user mutexes). + Alternatively, the =spin= crate provides a spinlock-based implementations of these types with a similar API which will be functionally identical to the implementations we will be writing. + +** =Mutex= + The purpose of a mutex is to provide exclusive access to a resource, or to ensure that two critical sections on different threads agree on the state of a resource. + A critical section is a region of code in which a shared resource is read from or written to. + + The primitive upon which mutexes are built, is the atomic compare-and-swap operation. + Our mutex will be a simple struct consisting of an =AtomicBool= capable of performing this operation, and the familiar =UnsafeCell= containing the resource we want to protect with the mutex: + #+begin_src rust +pub struct SpinMutex { + lock: AtomicBool, + data: UnsafeCell, +} + #+end_src + +A mutex will start out in a known unlocked state (=false= in our case). +When a thread wants access to the resource, it will attempt to acquire the lock by exchanging the expected =false= value with =true=. + +When performing any atomic operation in Rust, we must specify the memory ordering of the operation, or a sub-operation, according to the C++11 memory model. + +Both the processor and the compiler are allowed to reorder the individual instructions of our program according to their respective memory models, and even if we targeted a hypothetical processor with a totally sequentially consistent memory model, it would still be necessary to provide the compiler with the correct memory ordering since our Rust program does not actually run on the processor, but rather the abstract Rust machine which is essentially emulated on the target processor. + +In our case, wanting “two threads to agree on the state of a resource” means that in the case that we successfully exchange =false= for =true=, we want to =Acquire= all write-operations that happened before the respective =false= write on some other thread that last accessed our resource. +In the case that we failed to exchange =false= for =true= (because some other thread currently holds the lock), the memory ordering can be =Relaxed= because we don’t care about receiving partial updates to the resource we won’t be accessing until the other thread has released the lock. + +#+begin_src rust +impl SpinMutex { + pub fn try_lock(&self) -> bool { + self.lock + .compare_exchange(false, true, Ordering::Acquire, Ordering::Relaxed) + .is_ok() + } + + pub fn try_lock_weak(&self) -> bool { + self.lock + .compare_exchange_weak(false, true, Ordering::Acquire, Ordering::Relaxed) + .is_ok() + } + + pub fn lock(&self) -> SpinMutexGuard<'_, T> { + while !self.try_lock() { + core::hint::spin_loop(); + } + SpinMutexGuard { mutex: self } + } + + /// # Safety + /// The caller must be the logical owner of the locked mutex. + pub unsafe fn unlock(&self) { + self.lock.store(false, Ordering::Release); + } +} +#+end_src + +The =try_lock_weak= variant of =try_lock= usesa variant of the compare-and-swap operation that is allowed to fail spuriously, meaning it may return =Err(_)= even if the current value of the atomic is =false=, but may result in better performance overall on some architectures. +As can be seen on Compiler Explorer, on our platform, there is no difference in the assembly emitted. + +Raymond Chen of The Old New Thing has a short post explaining the difference on a platform that does care: [[https://devblogs.microsoft.com/oldnewthing/20180329-00/?p=98375][ARM]]. + +For a Rust oriented explanation, there is an [[https://mara.nl/atomics/hardware.html][entire chapter]] on the topic of atomics by Mara Bos. +Atomics are a very complex topic and often hard to reason about, easily demonstrated by a note from Mara’s chapter which points out that the behaviour noted by Raymond Chen is actually no longer the case with the =cas= instruction of the =ARMv8.1= architecture. + +The =SpinMutexGuard= type is a simple wrapper which holds a reference to the mutex and implements the =Deref= and =DerefMut= traits for the wrapped resource =T=. + +When dropped, ==SpinMutexGuard= automatically releases the lock on the mutex: +#+begin_src rust +pub struct SpinMutexGuard<'a, T: ?Sized + 'a> { + mutex: &'a SpinMutex, +} + +impl core::ops::Deref for SpinMutexGuard<'_, T> { + type Target = T; + + fn deref(&self) -> &Self::Target { + unsafe { &*self.mutex.data.get() } + } +} + +impl core::ops::DerefMut for SpinMutexGuard<'_, T> { + fn deref_mut(&mut self) -> &mut Self::Target { + unsafe { &mut *self.mutex.data.get() } + } +} + +impl Drop for SpinMutexGuard<'_, T> { + fn drop(&mut self) { + unsafe { self.mutex.unlock() } + } +} +#+end_src + +If we recall, the whole point of implementing the =Mutex= type was to be able to share resources between threads. +However, since our =SpinMutex= type contains an =UnsafeCell=, Rust has automatically determined that it never implements the =Sync= trait. + +Since we know that all access to the resource is properly synchronised by the mutex, we can safely inform the compiler that our type is indeed =Sync=: +#+begin_src rust +unsafe impl Send for SpinMutex {} +// SAFETY: SpinMutex properly synchronises access to T, but T must be Send +// to be safely shared between threads. +unsafe impl Sync for SpinMutex {} + +// SAFETY: &SpinMutexGuard cannot yield a &mut T, and the SpinMutexGuard +// holds the lock for the duration of its lifetime, so it is safe to share +// between threads. +unsafe impl Sync for SpinMutexGuard<'_, T> {} +// SAFETY: SpinMutex may be unlocked on any thread, so it is safe to send +// the guard to another thread as long as T is Send. +unsafe impl Send for SpinMutexGuard<'_, T> {} +#+end_src + + +** =Once= +The =Once= type is a synchronisation primitive that acts as a barrier separating the before and after scopes of some initialisation code. +To completely describe the states of a =Once=, we need more values than a simple boolean: +#+begin_src rust +const UNINITIALIZED: u8 = 0; +const INITIALIZING: u8 = 1; +const INITIALIZED: u8 = 2; +const POISONED: u8 = 3; + +pub struct Once { + state: AtomicU8, +} +#+end_src +The need for the =POISONED= and =INITIALIZING= states will become clear in a moment. + +To check if the =Once= has been initialised, we can simply check if the state is =INITIALIZED=. +Since we care about write operations that happened during the initialisation, we =Acquire= them when loading the state: +#+begin_src rust +pub fn is_completed(&self) -> bool { + self.state.load(Ordering::Acquire) == INITIALIZED +} +#+end_src + + +When we want to run some initialisation code, we first load and match on the current state: +if the =Once= is already initialised, we don’t want to run anything, and can simply return. +If the state is =UNINITIALIZED=, we attempt another compare-and-swap operation to change the state from =UNINITIALIZED= to =INITIALIZING=, and if we succeed, we run a user-provided closure. +If we fail the compare-and-swap, we will want to make sure to nevertheless =Acquire= any initialisation related write-operations that might have happened in the meantime on a different thread: if the compare-and-swap failed because the state has changed to =INITIALIZED=, we can now simply return. + +Since our kernel does not allow for recovering from panics through unwinding, we don’t care that panics might occur during the user-provided initialisation closure causing our =Once= to remain in the =INITIALIZING= state, however, there is still value in permitting the user to poison the =Once= as a result of, for example, a fallible initialisation closure. +We let the user decide how a poisoned =Once= should be handled: the =ignore_poison= parameter to the method decides whether a previously poisoned =Once= panics or attempts to run the initialisation closure again. + +Finally, if the state is =INITIALIZING=, we spin until it changes to some other state. +#+begin_src rust +pub fn call_once(&self, ignore_poison: bool, f: F) +where + F: FnOnce(&OnceState), +{ + let mut state = self.state.load(Ordering::Acquire); + loop { + match state { + INITIALIZED => return, + POISONED if !ignore_poison => { + panic!("Once instance has previously been poisoned") + } + POISONED | UNINITIALIZED => { + match self.state.compare_exchange( + state, + INITIALIZING, + Ordering::Acquire, + // if we get `Err(INITIALIZED)`, we want to have + // acquired what the lock is protecting. + Ordering::Acquire, + ) { + Err(new) => { + state = new; + continue; + } + Ok(_) => { + // even though we don't have unwinding, for + // completeness sake we'll poison the lock if + // the closure panics. + let guard = crate::drop_guard! { + self.state.store(POISONED, Ordering::Release) + }; + + let state = OnceState { + poisoned: state == POISONED, + state_to_set: Cell::new(INITIALIZED), + }; + + f(&state); + guard.forget(); + + self.state + .store(state.state_to_set.take(), Ordering::Release); + return; + } + } + } + _ => { + assert_eq!(state, INITIALIZING); + loop { + state = self.state.load(Ordering::Acquire); + if state != INITIALIZING { + break; + } + core::hint::spin_loop(); + } + } + } + } +} +#+end_src + +The =OnceState= struct passed to the closure lets the caller check whether the =Once= was previously poisoned, and to set the state to =POISONED= rather than =INITIALIZED= if the initialisation closure fails. + +Though not necessary for our non-unwinding kernel, a drop guard is used to additionally poison the =Once= should the closure cause a panic and unwind. + +the =drop_guard!= macro is a simple utility macro that will be useful not only as an idiomatic rusty =PanicGuard=, but also in other situations where we might want to run some code on the short-circuiting path of a function using the =?= operator. +#+begin_src rust +#[macro_export] +macro_rules! drop_guard { + ($($stmts:stmt)*) => { + { + struct __DropGuard(::core::mem::ManuallyDrop); + + impl __DropGuard { + #[allow(dead_code)] + fn forget(self) { + let mut this = ::core::mem::ManuallyDrop::new(self); + unsafe { + ManuallyDrop::drop(&mut this.0); + } + } + } + + impl Drop for __DropGuard { + fn drop(&mut self) { + unsafe { ::core::ptr::read(&*self.0)() } + } + } + + __DropGuard(::core::mem::ManuallyDrop::new(|| { + $($stmts)* + })) + } + }; +} +#+end_src + +The astute reader might have noticed that most of the code in the =call_once= method can only ever be executed once: we can likely improve the performance by offering a fast path for the common case of the =Once= already being initialised. +Since we will, in most cases, also know whether or not we want to panic or attempt initialisation on a poisoned =Once=, we can provide two separate methods for the two cases: +#+begin_src rust +pub fn call_once(&self, f: F) +where + F: FnOnce(&OnceState), +{ + if self.is_completed() { + return; + } + + self.call_once_slow(false, f); +} + +pub fn call_once_force(&self, f: F) +where + F: FnOnce(&OnceState), +{ + if self.is_completed() { + return; + } + + self.call_once_slow(true, f); +} +#+end_src + +We’ve renamed the original =call_once= method to =call_once_slow=, and annotated it with the =#[cold]= attribute to tell the compiler that any branch that calls this method is unlikely to be taken. +#+begin_src rust +#[cold] +pub fn call_once_slow(&self, ignore_poison: bool, f: F) +where + F: FnOnce(&OnceState), +{ ... } +#+end_src + + +** =OnceLock= & =LazyLock= + + Whereas a =Once= is a simple flag, the =OnceLock= builds upon it to explicitly scope the initialisation of a wrapped resource: + #+begin_src rust +pub struct OnceLock { + once: Once, + data: UnsafeCell>, +} + #+end_src + +Since our resource of type =T= starts out uninitialised but we still need to keep enough space around for it, we make use of the =MaybeUninit= type inside of an =UnsafeCell=. + +The most important method of =OnceLock= is =initialize=, which takes a fallible closure to initialise the resource: +#+begin_src rust +pub fn initialize(&self, f: F) -> Result<(), E> +where + F: FnOnce() -> Result, +{ + let mut res: Result<(), E> = Ok(()); + let cell = unsafe { &mut *self.data.get() }; + + self.once.call_once_force(|state| match f() { + Ok(value) => { + unsafe { cell.as_mut_ptr().write(value) }; + } + Err(e) => { + state.poison(); + res = Err(e); + } + }); + + res +} +#+end_src + +The rest of the type is various helper methods to access the underlying resource and to make common patterns more convenient. + +In the =Drop= implementation, we make sure to only drop the resource if it was actually initialised: +#+begin_src rust +impl Drop for OnceLock { + fn drop(&mut self) { + if self.is_completed() { + unsafe { self.data.get().as_mut_unchecked().assume_init_drop() } + } + } +} +#+end_src + + +The =LazyLock= type is a more special variant of =OnceLock=, in which the initialisation closure is always the same, infallible, and provided at construction time: +#+begin_src rust +pub struct LazyLock T> { + once: Once, + f: UnsafeCell>, + t: UnsafeCell>, +} +#+end_src + +Since we don’t have to provide a closure in order to unconditionally receive a reference to the initialised resource, =LazyLock= can implement the =Deref= and =DerefMut= traits to provide more ergonomic access to the wrapped value. + +Here, we take advantage of the fact that =Once::call_once= will panic if a previous initialisation poisoned the =Once= in order to guarantee that we only ever take the initialisation closure out of the =ManuallyDrop= once: +#+begin_src rust +impl T> LazyLock { + + fn force(this: &Self) -> &T { + this.once.call_once(|_| { + // SAFETY: because `call_once` will panic if poisoned, this + // closure will only be called once. + let f = unsafe { ManuallyDrop::take(&mut *this.f.get()) }; + let val = f(); + unsafe { (&mut *this.t.get()).write(val) }; + }); + + unsafe { (&*this.t.get()).assume_init_ref() } + } +} + +impl T> core::ops::Deref for LazyLock { + type Target = T; + + fn deref(&self) -> &Self::Target { + LazyLock::force(self) + } +} +#+end_src + +In the =Drop= implementation of =LazyLock=, we make sure only to drop the initialisation closure or the value if we can say for certain that they are still present and initialised, respectively: +#+begin_src rust +impl Drop for LazyLock { + fn drop(&mut self) { + match self.once.state() { + UNINITIALIZED => unsafe { + ManuallyDrop::drop(self.f.get_mut()); + }, + INITIALIZED => unsafe { + (&mut *self.t.get()).assume_init_drop(); + }, + _ => {} + } + } +} +#+end_src + +Once again, we have to explicitly implement the =Sync= trait for both =OnceLock= and =LazyLock=: since neither type contains any way for a user to create a mutable reference to the wrapped resource without a mutable reference to the lock itself, we can safely implement =Sync= for both types as long as the wrapped resource is also =Sync=. +#+begin_src rust +unsafe impl Sync for OnceLock {} + +unsafe impl Sync for LazyLock {} +#+end_src + +In the case of =LazyLock=, the bound =F: Send= is sufficient, since the initialisation closure is only ever called once. diff --git a/kernel/src/sync/mod.rs b/kernel/src/sync/mod.rs index 498f9ab..f828c7d 100644 --- a/kernel/src/sync/mod.rs +++ b/kernel/src/sync/mod.rs @@ -34,10 +34,17 @@ mod spin_mutex { } unsafe impl Send for SpinMutex {} + // SAFETY: SpinMutex properly synchronises access to T, but T must be Send + // to be safely shared between threads. unsafe impl Sync for SpinMutex {} - unsafe impl Send for SpinMutexGuard<'_, T> where for<'a> &'a mut T: Send {} - unsafe impl Sync for SpinMutexGuard<'_, T> where for<'a> &'a mut T: Sync {} + // SAFETY: &SpinMutexGuard cannot yield a &mut T, and the SpinMutexGuard + // holds the lock for the duration of its lifetime, so it is safe to share + // between threads. + unsafe impl Sync for SpinMutexGuard<'_, T> {} + // SAFETY: SpinMutex may be unlocked on any thread, so it is safe to send + // the guard to another thread as long as T is Send. + unsafe impl Send for SpinMutexGuard<'_, T> {} impl SpinMutex { pub const fn new(data: T) -> Self { @@ -50,13 +57,26 @@ mod spin_mutex { pub fn into_inner(self) -> T { self.data.into_inner() } - - pub fn as_mut_ptr(&self) -> *mut T { - self.data.get() - } } impl SpinMutex { + pub fn as_mut_ptr(&self) -> *mut T { + self.data.get() + } + + /// # Safety + /// The caller must be the logical owner of the locked mutex. + #[expect(clippy::mut_from_ref, reason = "unsafe mutex api")] + pub unsafe fn get_mut_unchecked(&self) -> &mut T { + unsafe { &mut *self.data.get() } + } + + /// # Safety + /// The caller must be the logical owner of the locked mutex. + pub unsafe fn get_unchecked(&self) -> &T { + unsafe { &*self.data.get() } + } + pub fn try_lock(&self) -> bool { self.lock .compare_exchange(false, true, Ordering::Acquire, Ordering::Relaxed) @@ -94,7 +114,7 @@ mod once { const UNINITIALIZED: u8 = 0; const INITIALIZING: u8 = 1; const INITIALIZED: u8 = 2; - const PANICKED: u8 = 3; + const POISONED: u8 = 3; pub struct OnceState { poisoned: bool, @@ -106,7 +126,7 @@ mod once { self.poisoned } pub fn poison(&self) { - self.state_to_set.set(PANICKED); + self.state_to_set.set(POISONED); } } @@ -116,9 +136,7 @@ mod once { const impl Default for Once { fn default() -> Self { - Self { - state: AtomicU8::new(UNINITIALIZED), - } + Self::new() } } @@ -172,10 +190,10 @@ mod once { loop { match state { INITIALIZED => return, - PANICKED if !ignore_poison => { + POISONED if !ignore_poison => { panic!("Once instance has previously been poisoned") } - PANICKED | UNINITIALIZED => { + POISONED | UNINITIALIZED => { match self.state.compare_exchange( state, INITIALIZING, @@ -193,11 +211,11 @@ mod once { // completeness sake we'll poison the lock if // the closure panics. let guard = crate::drop_guard! { - self.state.store(PANICKED, Ordering::Release) + self.state.store(POISONED, Ordering::Release) }; let state = OnceState { - poisoned: state == PANICKED, + poisoned: state == POISONED, state_to_set: Cell::new(INITIALIZED), }; @@ -230,7 +248,6 @@ mod once { data: UnsafeCell>, } - unsafe impl Send for OnceLock {} unsafe impl Sync for OnceLock {} const impl Default for OnceLock { @@ -299,7 +316,7 @@ mod once { match self.once.state() { INITIALIZED => return Some(unsafe { self.get_unchecked() }), INITIALIZING => core::hint::spin_loop(), - UNINITIALIZED | PANICKED => return None, + UNINITIALIZED | POISONED => return None, _ => unreachable!(), } }