From 5e4bc0ccf08b93637aa02621864e1b203a471a2d Mon Sep 17 00:00:00 2001 From: janis Date: Tue, 27 Oct 2026 01:57:27 +0100 Subject: [PATCH] rendergraph: correct transient-resource aliasing with pooled placement Lifetimes derive from the real dependency ordering (retained DAG + compile-time happens-before closure), not command-index intervals. Aliasing is restricted to the single-queue contract; alias reuse is synchronized with memory/image barriers anchored at first-use commands. - renderer_device: aliasable shared allocations (CAN_ALIAS), memory- requirements queries incl. dedicated hints, offset-aware bind_buffer/image_memory2, grouped bind under one allocator lock - renderer_resources: unallocated resource creation, requirements cached in Allocation::Unmanaged (OnceLock), MemoryGroup pooling with intersecting memoryTypeBits, bind-time validation - renderer_rendergraph: retained dependency DAG, happens_before closure, batch lifetimes, Placement type, plan_placements with pooled aliasing (first-fit over offset-sorted ranges, pool classes split by bufferImageGranularity), alias-barrier construction, placement/bind error variants - perf: quadratic compile/resolve paths eliminated (per-resource indexing, frontier caching, backward-scan prune, cached recording index, recorder scratch reuse); single requirements query per resource through plan/validate/bind - design plan + GPU-gated validation tests --- Cargo.toml | 1 + crates/renderer/AGENTS.md | 2 +- crates/renderer/src/render_graph/resources.rs | 12 +- crates/renderer_device/src/device.rs | 456 +++- crates/renderer_device/src/lib.rs | 33 + crates/renderer_device/src/testing.rs | 119 +- crates/renderer_rendergraph/Cargo.toml | 1 + crates/renderer_rendergraph/src/graph.rs | 2417 +++++++++++++++-- crates/renderer_rendergraph/src/lib.rs | 5 +- crates/renderer_rendergraph/src/recorder.rs | 211 +- crates/renderer_rendergraph/src/resources.rs | 8 +- .../tests/graph_validation.rs | 241 +- crates/renderer_resources/AGENTS.md | 14 +- crates/renderer_resources/src/buffers.rs | 290 +- crates/renderer_resources/src/images.rs | 103 + crates/renderer_resources/src/lib.rs | 1 + crates/renderer_resources/src/memory.rs | 411 +++ .../rendergraph-transient-aliasing.md | 202 ++ 18 files changed, 4223 insertions(+), 304 deletions(-) create mode 100644 crates/renderer_resources/src/memory.rs create mode 100644 design-plans/rendergraph-transient-aliasing.md diff --git a/Cargo.toml b/Cargo.toml index a260040..a90d647 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -49,6 +49,7 @@ arrayvec = "0.7.6" tinyvec = "1.8" indexmap = "2" petgraph = "0.7" +fixedbitset = "0.5" # itertools = "0.14.0" ahash = "0.8" diff --git a/crates/renderer/AGENTS.md b/crates/renderer/AGENTS.md index 194f067..0affb66 100644 --- a/crates/renderer/AGENTS.md +++ b/crates/renderer/AGENTS.md @@ -123,7 +123,7 @@ - The `egui` pre-pass currently packs texture updates into a staging image/buffer and registers graph work; texture-free deltas and upload ordering deserve explicit coverage when changing the integration. - The private font-atlas image is CPU memory, not a Vulkan `images::Image`; do not conflate the similarly named `Image` types when editing `text.rs`. - `weak_vec` moved to `renderer_resources::util` and is not re-exported by `renderer`; it prunes dead weak references on capacity pressure. Image-side view tracking should remain weak to avoid ownership cycles. -- Render graph IDs are distinct: legacy `GraphResourceId` wraps `u32`; the newer internal `resources::ResourceId` wraps `u64`. +- Render graph IDs are distinct: legacy `GraphResourceId` wraps `u32`; the newer internal `resources::ResourceId` also wraps `u32` (constructed via `ResourceId::from_u32`). - Public `commands::traits::CommandBufferExt` contains raw recording convenience methods; those methods assume valid command-buffer state and do not all enforce it at runtime. - `SwapchainConfiguration.image_count` is treated as the additional desired in-flight count during validation, while creation asks for minimum surface image count plus that configured value. The fallback-configuration helper assigns a total count. - The supported surface capability extension query can filter present modes based on the selected mode; its behavior depends on optional `InstanceExtensions` support. diff --git a/crates/renderer/src/render_graph/resources.rs b/crates/renderer/src/render_graph/resources.rs index 993092a..9cee8f6 100644 --- a/crates/renderer/src/render_graph/resources.rs +++ b/crates/renderer/src/render_graph/resources.rs @@ -228,7 +228,17 @@ mod impls { #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)] #[repr(transparent)] -pub struct ResourceId(pub u64); +pub struct ResourceId(u32); + +impl ResourceId { + pub fn from_u32(index: u32) -> Self { + Self(index) + } + + pub fn index(self) -> usize { + self.0 as usize + } +} #[derive(Debug, Clone, Copy)] pub enum ResourceRange { diff --git a/crates/renderer_device/src/device.rs b/crates/renderer_device/src/device.rs index b1b98ff..1f0d10a 100644 --- a/crates/renderer_device/src/device.rs +++ b/crates/renderer_device/src/device.rs @@ -4,7 +4,7 @@ use std::{ ffi::CStr, ops::{Deref, DerefMut}, ptr::NonNull, - sync::{Arc, Weak}, + sync::{Arc, OnceLock, Weak}, }; use ash::{ext, khr, prelude::VkResult, vk}; @@ -206,11 +206,78 @@ pub struct AllocationDesc<'a> { pub strategy: AllocationStrategy, } +/// Vulkan memory requirements for a buffer or image. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct MemoryRequirements { + pub size: vk::DeviceSize, + pub alignment: vk::DeviceSize, + pub memory_type_bits: u32, + pub requires_dedicated_allocation: bool, + pub prefers_dedicated_allocation: bool, +} + +/// A buffer or image to bind into a shared allocation at an allocation-local offset. +#[derive(Debug, Clone, Copy)] +pub enum MemoryBinding { + Buffer { + buffer: vk::Buffer, + offset: vk::DeviceSize, + requirements: MemoryRequirements, + }, + Image { + image: vk::Image, + offset: vk::DeviceSize, + requirements: MemoryRequirements, + }, +} + +/// A failed grouped bind and the number of earlier members that remain bound. +#[derive(Debug)] +pub struct GroupBindError { + pub error: crate::Error, + pub bound_count: usize, +} + +/// Returns the first memory type allowed by `memory_type_bits` with all requested properties. +fn find_memory_type_index( + memory_properties: &vk::PhysicalDeviceMemoryProperties, + memory_type_bits: u32, + property_flags: vk::MemoryPropertyFlags, +) -> Result { + (0..memory_properties.memory_type_count) + .find(|&index| { + memory_type_bits & (1 << index) != 0 + && memory_properties.memory_types[index as usize] + .property_flags + .contains(property_flags) + }) + .ok_or(Error::NoCompatibleMemoryType { + memory_type_bits, + property_flags, + }) +} + +impl From for MemoryRequirements { + fn from(requirements: vk::MemoryRequirements) -> Self { + Self { + size: requirements.size, + alignment: requirements.alignment, + memory_type_bits: requirements.memory_type_bits, + requires_dedicated_allocation: false, + prefers_dedicated_allocation: false, + } + } +} + #[derive(Debug)] pub enum Allocation { Owned(DeviceObject), Shared(Arc>), - Unmanaged, + /// An unbound resource. Requirements are a pure function of its Vulkan handle, so + /// resources can safely cache the first query here for their lifetime. + Unmanaged { + requirements: OnceLock, + }, } impl Allocation { @@ -218,7 +285,7 @@ impl Allocation { match self { Allocation::Owned(obj) => Some(obj), Allocation::Shared(arc) => Some(arc.as_ref()), - Allocation::Unmanaged => None, + Allocation::Unmanaged { .. } => None, } } @@ -226,7 +293,7 @@ impl Allocation { match self { Allocation::Owned(obj) => Some(obj), Allocation::Shared(arc) => Arc::get_mut(arc).map(DerefMut::deref_mut), - Allocation::Unmanaged => None, + Allocation::Unmanaged { .. } => None, } } } @@ -524,11 +591,24 @@ impl PhysicalDeviceInfo { let mut allocator_info = vk_mem::AllocatorCreateInfo::new(&instance.inner.raw, &device, self.pdev); + allocator_info.preferred_large_heap_block_size = 64 * 1024 * 1024; allocator_info.vulkan_api_version = instance .inner .api_version .min(self.properties.device_api_version); + + if allocator_info.vulkan_api_version < vk::API_VERSION_1_1 { + if enabled_extensions.contains(&khr::dedicated_allocation::NAME) + && enabled_extensions.contains(&khr::get_memory_requirements2::NAME) + { + allocator_info.flags |= vk_mem::AllocatorCreateFlags::KHR_DEDICATED_ALLOCATION; + } + if enabled_extensions.contains(&khr::bind_memory2::NAME) { + allocator_info.flags |= vk_mem::AllocatorCreateFlags::KHR_BIND_MEMORY2; + } + } + let allocator = unsafe { vk_mem::Allocator::new(allocator_info) }?; let pipeline_cache = PipelineCache::new(&device, &self).unwrap(); @@ -656,7 +736,366 @@ where } } +impl Device { + /// Allocates aliasable memory compatible with `requirements` and `property_flags`. + /// + /// The returned `Arc` is an allocation handle intended to be transferred into each bound + /// resource's `Allocation::Shared`. Binding consumes one Arc reference; the allocation is + /// freed by its `DeviceObject` when the last bound resource drops. This helper does not + /// retain a reference after a successful bind. + pub fn allocate_memory( + &self, + requirements: &MemoryRequirements, + property_flags: vk::MemoryPropertyFlags, + ) -> Result>> { + let allocation = self + .0 + .allocate_aliasable_memory(requirements, property_flags)?; + Ok(Arc::new(DeviceObject::new(self.clone(), allocation))) + } +} + impl DeviceInner { + fn api_version(&self) -> u32 { + self.instance + .inner + .api_version + .min(self.adapter.properties.device_api_version) + } + + fn has_enabled_extension(&self, name: &CStr) -> bool { + self.enabled_extensions.contains(&name) + } + + fn has_memory2_extension_set(&self) -> bool { + self.has_enabled_extension(khr::get_memory_requirements2::NAME) + && self.has_enabled_extension(khr::bind_memory2::NAME) + && self.has_enabled_extension(khr::dedicated_allocation::NAME) + } + + fn require_offset_binding_support(&self) -> Result<()> { + if self.api_version() < vk::API_VERSION_1_1 && !self.has_memory2_extension_set() { + return Err(Error::OffsetBindingRequiresApi11); + } + Ok(()) + } + + /// Finds a compatible memory type satisfying `property_flags`. + pub fn find_memory_type_index( + &self, + requirements: &MemoryRequirements, + property_flags: vk::MemoryPropertyFlags, + ) -> Result { + find_memory_type_index( + &self.adapter.properties.memory_properties, + requirements.memory_type_bits, + property_flags, + ) + } + + /// Queries buffer memory requirements, including dedicated-allocation hints when available. + pub fn buffer_memory_requirements(&self, buffer: vk::Buffer) -> Result { + let api11 = self.api_version() >= vk::API_VERSION_1_1; + + let get_requirements2 = + api11 || self.has_enabled_extension(khr::get_memory_requirements2::NAME); + let dedicated_available = + api11 || self.has_enabled_extension(khr::dedicated_allocation::NAME); + + if !get_requirements2 { + let requirements = unsafe { self.raw.get_buffer_memory_requirements(buffer) }; + return Ok(requirements.into()); + } + + let mut dedicated = vk::MemoryDedicatedRequirements::default(); + let mut requirements = vk::MemoryRequirements2::default(); + + if dedicated_available { + requirements = requirements.push_next(&mut dedicated); + } + + let info = vk::BufferMemoryRequirementsInfo2::default().buffer(buffer); + unsafe { + if api11 { + self.raw + .get_buffer_memory_requirements2(&info, &mut requirements); + } else { + khr::get_memory_requirements2::Device::new(&self.instance.inner.raw, &self.raw) + .get_buffer_memory_requirements2(&info, &mut requirements); + } + } + + Ok(MemoryRequirements { + size: requirements.memory_requirements.size, + alignment: requirements.memory_requirements.alignment, + memory_type_bits: requirements.memory_requirements.memory_type_bits, + requires_dedicated_allocation: dedicated_available + && dedicated.requires_dedicated_allocation == vk::TRUE, + prefers_dedicated_allocation: dedicated_available + && dedicated.prefers_dedicated_allocation == vk::TRUE, + }) + } + + /// Queries image memory requirements, including dedicated-allocation hints when available. + pub fn image_memory_requirements(&self, image: vk::Image) -> Result { + let api11 = self.api_version() >= vk::API_VERSION_1_1; + + let get_requirements2 = + api11 || self.has_enabled_extension(khr::get_memory_requirements2::NAME); + let dedicated_available = + api11 || self.has_enabled_extension(khr::dedicated_allocation::NAME); + + if !get_requirements2 { + let requirements = unsafe { self.raw.get_image_memory_requirements(image) }; + return Ok(requirements.into()); + } + + let mut dedicated = vk::MemoryDedicatedRequirements::default(); + let mut requirements = vk::MemoryRequirements2::default(); + if dedicated_available { + requirements = requirements.push_next(&mut dedicated); + } + + let info = vk::ImageMemoryRequirementsInfo2::default().image(image); + + unsafe { + if api11 { + self.raw + .get_image_memory_requirements2(&info, &mut requirements); + } else { + khr::get_memory_requirements2::Device::new(&self.instance.inner.raw, &self.raw) + .get_image_memory_requirements2(&info, &mut requirements); + } + } + + Ok(MemoryRequirements { + size: requirements.memory_requirements.size, + alignment: requirements.memory_requirements.alignment, + memory_type_bits: requirements.memory_requirements.memory_type_bits, + requires_dedicated_allocation: dedicated_available + && dedicated.requires_dedicated_allocation == vk::TRUE, + prefers_dedicated_allocation: dedicated_available + && dedicated.prefers_dedicated_allocation == vk::TRUE, + }) + } + + /// Allocates aliasable memory compatible with `requirements` and `property_flags`. + /// + /// Resources that require a dedicated allocation cannot use this resource-independent path. + fn allocate_aliasable_memory( + &self, + requirements: &MemoryRequirements, + property_flags: vk::MemoryPropertyFlags, + ) -> Result { + use vk_mem::{Alloc as _, AllocationCreateFlags, AllocationCreateInfo}; + + if requirements.requires_dedicated_allocation { + return Err(Error::DedicatedAllocationRequiresResource); + } + + let memory_type_index = self.find_memory_type_index(requirements, property_flags)?; + let create_info = AllocationCreateInfo { + flags: AllocationCreateFlags::CAN_ALIAS, + required_flags: property_flags, + memory_type_bits: 1 << memory_type_index, + ..Default::default() + }; + + let raw_requirements = vk::MemoryRequirements { + size: requirements.size, + alignment: requirements.alignment, + memory_type_bits: requirements.memory_type_bits, + }; + + let allocator = self.allocator.lock(); + let raw = unsafe { allocator.allocate_memory(&raw_requirements, &create_info) }?; + + let allocation_info = allocator.get_allocation_info2(&raw); + let dedicated_memory = allocation_info.dedicated_memory; + let allocation_info = allocation_info.allocation_info; + let mapped = NonNull::new(allocation_info.mapped_data.cast::()).map(SendSyncPtr); + + let allocation = GpuAllocation { + raw, + memory: allocation_info.device_memory, + offset: allocation_info.offset, + size: allocation_info.size, + memory_type_index: allocation_info.memory_type, + mapped, + dedicated_memory, + }; + Ok(allocation) + } + + /// Binds a buffer to aliasable memory at an allocation-local offset. + /// + /// On success, ownership of `allocation` transfers to the returned `Allocation::Shared`; + /// store it in the bound resource and drop any caller-side references. The allocation is + /// freed when the last bound resource drops. The resource must be destroyed before its + /// allocation ownership is dropped. `requirements` must be the buffer's own requirements, + /// as returned by the buffer requirements query. If the supplied requirements are wrong, + /// the Vulkan bind (`vkBindBufferMemory2` via VMA) remains the authoritative validation. + /// Calls are externally synchronized by VMA. + pub fn bind_buffer_at_offset( + &self, + buffer: vk::Buffer, + requirements: &MemoryRequirements, + allocation: Arc>, + local_offset: vk::DeviceSize, + ) -> Result { + self.require_offset_binding_support()?; + self.validate_shared_binding(&allocation, local_offset, *requirements)?; + + unsafe { + self.allocator.lock().bind_buffer_memory2( + &allocation.raw, + local_offset, + buffer, + core::ptr::null(), + )?; + } + + Ok(Allocation::Shared(allocation)) + } + + /// Binds an image to aliasable memory at an allocation-local offset. + /// + /// On success, ownership of `allocation` transfers to the returned `Allocation::Shared`; + /// store it in the bound resource and drop any caller-side references. The allocation is + /// freed when the last bound resource drops. The resource must be destroyed before its + /// allocation ownership is dropped. `requirements` must be the image's own requirements, + /// as returned by the image requirements query. If the supplied requirements are wrong, + /// the Vulkan bind (`vkBindImageMemory2` via VMA) remains the authoritative validation. + /// Calls are externally synchronized by VMA. + pub fn bind_image_at_offset( + &self, + image: vk::Image, + requirements: &MemoryRequirements, + allocation: Arc>, + local_offset: vk::DeviceSize, + ) -> Result { + self.require_offset_binding_support()?; + self.validate_shared_binding(&allocation, local_offset, *requirements)?; + + unsafe { + self.allocator.lock().bind_image_memory2( + &allocation.raw, + local_offset, + image, + core::ptr::null(), + )?; + } + + Ok(Allocation::Shared(allocation)) + } + + /// Binds a group of buffers and images into one aliasable allocation. + /// + /// Every member is validated before any Vulkan bind is attempted. Binds then run in slice + /// order under one allocator lock; if a bind fails, the first error is returned and earlier + /// successful binds remain bound, matching repeated single-resource binds. `bound_count` + /// identifies that successfully bound prefix so callers can retain the allocation for it. + pub fn bind_group_at_offset( + &self, + allocation: &Arc>, + bindings: &[MemoryBinding], + ) -> core::result::Result<(), GroupBindError> { + self.require_offset_binding_support() + .map_err(|error| GroupBindError { + error, + bound_count: 0, + })?; + + for binding in bindings { + let (offset, requirements) = match binding { + MemoryBinding::Buffer { + offset, + requirements, + .. + } + | MemoryBinding::Image { + offset, + requirements, + .. + } => (*offset, *requirements), + }; + self.validate_shared_binding(allocation, offset, requirements) + .map_err(|error| GroupBindError { + error, + bound_count: 0, + })?; + } + + let allocator = self.allocator.lock(); + for (bound_count, binding) in bindings.iter().enumerate() { + unsafe { + match binding { + MemoryBinding::Buffer { buffer, offset, .. } => { + allocator + .bind_buffer_memory2( + &allocation.raw, + *offset, + *buffer, + core::ptr::null(), + ) + .map_err(|error| GroupBindError { + error: error.into(), + bound_count, + })?; + } + MemoryBinding::Image { image, offset, .. } => { + allocator + .bind_image_memory2(&allocation.raw, *offset, *image, core::ptr::null()) + .map_err(|error| GroupBindError { + error: error.into(), + bound_count, + })?; + } + } + } + } + + Ok(()) + } + + fn validate_shared_binding( + &self, + allocation: &DeviceObject, + local_offset: vk::DeviceSize, + requirements: MemoryRequirements, + ) -> Result<()> { + if !core::ptr::eq(Arc::as_ptr(&allocation.device().0), self) { + return Err(Error::AllocationDeviceMismatch); + } + + if requirements.requires_dedicated_allocation { + return Err(Error::DedicatedAllocationRequiresResource); + } + + let gpu_allocation: &GpuAllocation = allocation; + if requirements.memory_type_bits & (1 << gpu_allocation.memory_type_index()) == 0 { + return Err(Error::IncompatibleAllocationMemoryType { + memory_type_index: gpu_allocation.memory_type_index(), + memory_type_bits: requirements.memory_type_bits, + }); + } + + let end = local_offset.checked_add(requirements.size); + if requirements.alignment == 0 + || !local_offset.is_multiple_of(requirements.alignment) + || end.is_none_or(|end| end > gpu_allocation.size()) + { + return Err(Error::InvalidAllocationOffset { + offset: local_offset, + alignment: requirements.alignment, + required_size: requirements.size, + allocation_size: gpu_allocation.size(), + }); + } + + Ok(()) + } + fn allocation_create_info( &self, desc: &AllocationDesc<'_>, @@ -679,6 +1118,7 @@ impl DeviceInner { MemoryLocation::CpuToGpu | MemoryLocation::GpuToCpu ) { let memory_properties = &self.adapter.properties.memory_properties; + let coherent_host_types = memory_properties .memory_types .iter() @@ -691,9 +1131,11 @@ impl DeviceInner { ) }) .fold(0_u32, |bits, (index, _)| bits | (1 << index)); + if coherent_host_types == 0 { return Err(vk::Result::ERROR_FEATURE_NOT_PRESENT.into()); } + info.memory_type_bits = coherent_host_types; info.flags |= AllocationCreateFlags::MAPPED; } @@ -703,9 +1145,11 @@ impl DeviceInner { .inner .api_version .min(self.adapter.properties.device_api_version); + if desc.strategy == AllocationStrategy::Dedicated && api_version < vk::API_VERSION_1_1 { return Err(Error::DedicatedAllocationRequiresApi11); } + if desc.strategy == AllocationStrategy::Dedicated { info.flags |= AllocationCreateFlags::DEDICATED_MEMORY; } @@ -722,11 +1166,13 @@ impl DeviceInner { let create_info = self.allocation_create_info(&desc)?; let allocator = self.allocator.lock(); + let raw = unsafe { allocator.allocate_memory_for_buffer(buffer, &create_info) }?; let allocation_info = allocator.get_allocation_info2(&raw); let dedicated_memory = allocation_info.dedicated_memory; let allocation_info = allocation_info.allocation_info; let mapped = NonNull::new(allocation_info.mapped_data.cast::()).map(SendSyncPtr); + Ok(GpuAllocation { raw, memory: allocation_info.device_memory, @@ -747,11 +1193,13 @@ impl DeviceInner { let create_info = self.allocation_create_info(&desc)?; let allocator = self.allocator.lock(); + let raw = unsafe { allocator.allocate_memory_for_image(image, &create_info) }?; let allocation_info = allocator.get_allocation_info2(&raw); let dedicated_memory = allocation_info.dedicated_memory; let allocation_info = allocation_info.allocation_info; let mapped = NonNull::new(allocation_info.mapped_data.cast::()).map(SendSyncPtr); + Ok(GpuAllocation { raw, memory: allocation_info.device_memory, diff --git a/crates/renderer_device/src/lib.rs b/crates/renderer_device/src/lib.rs index 8aed0d6..a06be0a 100644 --- a/crates/renderer_device/src/lib.rs +++ b/crates/renderer_device/src/lib.rs @@ -83,6 +83,39 @@ pub enum Error { "Dedicated allocation requires Vulkan 1.1 or both VK_KHR_dedicated_allocation and VK_KHR_get_memory_requirements2 extensions." )] DedicatedAllocationRequiresApi11, + #[error( + "Offset-aware memory binding requires Vulkan 1.1 or VK_KHR_get_memory_requirements2, VK_KHR_bind_memory2, and VK_KHR_dedicated_allocation to be enabled." + )] + OffsetBindingRequiresApi11, + #[error( + "No memory type matches requirements mask {memory_type_bits:#010x} and required flags {property_flags:?}." + )] + NoCompatibleMemoryType { + memory_type_bits: u32, + property_flags: vk::MemoryPropertyFlags, + }, + #[error( + "Memory allocation requirements request a dedicated allocation, which cannot be allocated without its resource." + )] + DedicatedAllocationRequiresResource, + #[error( + "Allocation offset {offset} is invalid: alignment is {alignment}, requested size is {required_size}, allocation size is {allocation_size}." + )] + InvalidAllocationOffset { + offset: u64, + alignment: u64, + required_size: u64, + allocation_size: u64, + }, + #[error( + "Allocation memory type {memory_type_index} is not allowed by requirements mask {memory_type_bits:#010x}." + )] + IncompatibleAllocationMemoryType { + memory_type_index: u32, + memory_type_bits: u32, + }, + #[error("The allocation and resource belong to different Vulkan devices.")] + AllocationDeviceMismatch, #[error(transparent)] Io(#[from] std::io::Error), #[error( diff --git a/crates/renderer_device/src/testing.rs b/crates/renderer_device/src/testing.rs index bbf901d..54e024e 100644 --- a/crates/renderer_device/src/testing.rs +++ b/crates/renderer_device/src/testing.rs @@ -282,7 +282,9 @@ unsafe extern "system" fn capture_callback( #[cfg(test)] mod tests { use super::*; - use crate::device::{AllocationDesc, AllocationStrategy, DeviceObject, MemoryLocation}; + use crate::device::{ + Allocation, AllocationDesc, AllocationStrategy, DeviceObject, MemoryLocation, + }; #[test] fn instance_and_device_creation() { @@ -326,6 +328,121 @@ mod tests { test.assert_no_validation_errors(); } + #[test] + fn aliasable_buffer_allocation_binds_at_zero_under_validation() { + let Some(test) = TestVulkan::acquire() else { + return; + }; + let device = test.device(); + let create_info = vk::BufferCreateInfo::default() + .size(256) + .usage(vk::BufferUsageFlags::TRANSFER_SRC | vk::BufferUsageFlags::TRANSFER_DST) + .sharing_mode(vk::SharingMode::EXCLUSIVE); + let buffer = unsafe { device.dev().create_buffer(&create_info, None) } + .expect("create aliasable buffer"); + let buffer = DeviceObject::new(device.clone(), buffer); + let requirements = device + .buffer_memory_requirements(*buffer) + .expect("query buffer requirements"); + let memory = device + .allocate_memory(&requirements, vk::MemoryPropertyFlags::empty()) + .expect("allocate aliasable buffer memory"); + let binding = device + .bind_buffer_at_offset(*buffer, &requirements, memory, 0) + .expect("bind aliasable buffer memory"); + assert!(matches!(binding, Allocation::Shared(_))); + + drop(buffer); + drop(binding); + test.assert_no_validation_errors(); + } + + #[test] + fn aliasable_image_allocation_binds_at_zero_under_validation() { + let Some(test) = TestVulkan::acquire() else { + return; + }; + let device = test.device(); + let create_info = vk::ImageCreateInfo::default() + .image_type(vk::ImageType::TYPE_2D) + .format(vk::Format::R8G8B8A8_UNORM) + .extent(vk::Extent3D { + width: 16, + height: 16, + depth: 1, + }) + .mip_levels(1) + .array_layers(1) + .samples(vk::SampleCountFlags::TYPE_1) + .tiling(vk::ImageTiling::OPTIMAL) + .usage(vk::ImageUsageFlags::TRANSFER_DST) + .sharing_mode(vk::SharingMode::EXCLUSIVE) + .initial_layout(vk::ImageLayout::UNDEFINED); + let image = unsafe { device.dev().create_image(&create_info, None) } + .expect("create aliasable image"); + let image = DeviceObject::new(device.clone(), image); + let requirements = device + .image_memory_requirements(*image) + .expect("query image requirements"); + let memory = device + .allocate_memory(&requirements, vk::MemoryPropertyFlags::empty()) + .expect("allocate aliasable image memory"); + let binding = device + .bind_image_at_offset(*image, &requirements, memory, 0) + .expect("bind aliasable image memory"); + assert!(matches!(binding, Allocation::Shared(_))); + + drop(image); + drop(binding); + test.assert_no_validation_errors(); + } + + #[test] + fn aliasable_binding_rejects_out_of_bounds_and_unaligned_offsets() { + let Some(test) = TestVulkan::acquire() else { + return; + }; + let device = test.device(); + let create_info = vk::BufferCreateInfo::default() + .size(256) + .usage(vk::BufferUsageFlags::TRANSFER_DST) + .sharing_mode(vk::SharingMode::EXCLUSIVE); + + let make_buffer = || { + let raw = unsafe { device.dev().create_buffer(&create_info, None) } + .expect("create offset validation buffer"); + DeviceObject::new(device.clone(), raw) + }; + let buffer = make_buffer(); + let requirements = device + .buffer_memory_requirements(*buffer) + .expect("query offset validation requirements"); + let memory = device + .allocate_memory(&requirements, vk::MemoryPropertyFlags::empty()) + .expect("allocate offset validation memory"); + assert!(matches!( + device.bind_buffer_at_offset(*buffer, &requirements, memory, requirements.size), + Err(crate::Error::InvalidAllocationOffset { .. }) + )); + drop(buffer); + + let buffer = make_buffer(); + let requirements = device + .buffer_memory_requirements(*buffer) + .expect("query unaligned validation requirements"); + if requirements.alignment > 1 { + let memory = device + .allocate_memory(&requirements, vk::MemoryPropertyFlags::empty()) + .expect("allocate unaligned validation memory"); + assert!(matches!( + device.bind_buffer_at_offset(*buffer, &requirements, memory, 1), + Err(crate::Error::InvalidAllocationOffset { .. }) + )); + } + drop(buffer); + test.assert_no_validation_errors(); + } + #[test] fn validation_capture_works() { let Some(test) = TestVulkan::acquire() else { diff --git a/crates/renderer_rendergraph/Cargo.toml b/crates/renderer_rendergraph/Cargo.toml index 1436fcf..47f92d0 100644 --- a/crates/renderer_rendergraph/Cargo.toml +++ b/crates/renderer_rendergraph/Cargo.toml @@ -5,6 +5,7 @@ edition = "2024" [dependencies] ash.workspace = true +fixedbitset.workspace = true renderer_device = { path = "../renderer_device" } renderer_resources = { path = "../renderer_resources" } ash-formats = { git = "https://github.com/janis-bhm/ash-formats", branch = "main" } diff --git a/crates/renderer_rendergraph/src/graph.rs b/crates/renderer_rendergraph/src/graph.rs index fa1918d..e03d034 100644 --- a/crates/renderer_rendergraph/src/graph.rs +++ b/crates/renderer_rendergraph/src/graph.rs @@ -1,3 +1,15 @@ +//! Render-graph compilation derives command dependencies from conflicting resource +//! accesses and computes their transitive happens-before closure. Transient +//! resources may share memory only when every terminal use of one precedes every +//! initial use of the other; device-dependent placement uses separate +//! linear/optimal pools and inserts alias barriers at the new occupant's first +//! use. +//! +//! Aliasing safety assumes all graph commands are recorded into one command +//! buffer that is submitted to one queue. Cross-queue aliasing is unsupported: +//! when command-to-queue assignment is introduced, alias transitions between +//! queues must also be represented by semaphore edges. + use std::{ cmp::Reverse, collections::{BTreeSet, BinaryHeap, HashMap, VecDeque}, @@ -5,12 +17,12 @@ use std::{ }; use ash::vk; -use ash_formats::FormatExt; +use fixedbitset::FixedBitSet; use renderer_resources::{ Device, buffers::{Buffer as BufferResource, BufferDesc}, descriptors::DescriptorResourceBinding, - device::{AllocationStrategy, QueueFlags}, + device::{AllocationStrategy, MemoryLocation, QueueFlags}, images::{Image as ImageResource, ImageDesc, ImageSubresourceExt}, pipeline::DescriptorSet, }; @@ -36,6 +48,28 @@ pub enum GraphError { ConflictingDrawLayouts(ResourceId), ConflictingDescriptorInitialStates, ConflictingDescriptorDrawLayouts, + PlacementFailed(ResourceId), + BindFailed(ResourceId), + InvalidPooledImageLayout(ResourceId), +} + +impl From<(ResourceId, renderer_resources::memory::Error)> for GraphError { + fn from((resource, error): (ResourceId, renderer_resources::memory::Error)) -> Self { + match error { + renderer_resources::memory::Error::Bind(_) + | renderer_resources::memory::Error::MisalignedOffset { .. } + | renderer_resources::memory::Error::IncompatibleMemoryType { .. } + | renderer_resources::memory::Error::OutOfBounds { .. } + | renderer_resources::memory::Error::AlreadyBound => Self::BindFailed(resource), + _ => Self::PlacementFailed(resource), + } + } +} + +impl From<(ResourceId, renderer_resources::Error)> for GraphError { + fn from((resource, _error): (ResourceId, renderer_resources::Error)) -> Self { + Self::PlacementFailed(resource) + } } #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -57,40 +91,64 @@ pub struct BarrierResource { pub queue_transfer: Option, } +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct MemoryBarrier { + pub src_stage: vk::PipelineStageFlags2, + pub src_access: vk::AccessFlags2, + pub dst_stage: vk::PipelineStageFlags2, + pub dst_access: vk::AccessFlags2, +} + #[derive(Debug, Clone)] pub struct PipelineBarrier { pub before_command: CommandIndex, + pub memory_barriers: Vec, pub resources: Vec, } #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub struct ResourceLifetime { - pub first_command: CommandIndex, - pub last_command: CommandIndex, - /// Placeholder allocator binding; resources with disjoint lifetimes can - /// share an offset when their descriptor sizes permit it. - pub memory_offset: u64, + pub first_batch: usize, + pub last_batch: usize, + pub placement: Placement, + /// Imported resources participate in analysis, but never in transient packing. + pub imported: bool, } -impl ResourceLifetime { - fn overlaps(&self, other: &Self) -> bool { - self.first_command <= other.last_command && other.first_command <= self.last_command - } +/// Placement metadata reserved for the device-dependent packing phase. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct Placement { + pub offset: u64, + pub size: u64, + pub alignment: u64, + pub pool: Option, + pub aliased: bool, +} + +/// One command's access scope for a resource, retained for alias barriers. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct ResourceUse { + pub command: CommandIndex, + pub stage: vk::PipelineStageFlags2, + pub access: vk::AccessFlags2, } /// The resource-level interference graph used by transient-resource aliasing. -/// An edge means the two resources are live at the same command index. +/// An edge means no safe ordering between the resources was proven. #[derive(Debug, Clone, Default)] pub struct ResourceInterference { edges: Vec>, } impl ResourceInterference { - /// Build a symmetric interference graph from inclusive resource lifetimes. + /// Build a conservative interference graph from inclusive batch intervals. + /// + /// This interval-only constructor is useful for reporting/tests. The compiler + /// uses command reachability instead; batch separation is not an ordering proof. pub fn from_lifetimes( lifetimes: impl IntoIterator, ) -> Self { - let mut max_id = ResourceId(0); + let mut max_id = ResourceId::from_u32(0); let mut resources: Vec<_> = lifetimes .into_iter() .inspect(|(id, _)| { @@ -102,7 +160,7 @@ impl ResourceInterference { let mut edges = vec![BTreeSet::new(); max_id.index() + 1]; - resources.sort_unstable_by_key(|&(_, lifetime)| lifetime.first_command); + resources.sort_unstable_by_key(|&(_, lifetime)| lifetime.first_batch); let mut resources = VecDeque::from(resources); let mut index = 0; @@ -111,7 +169,7 @@ impl ResourceInterference { let mut j = 0; while j < index { - if resources[j].1.last_command < lifetime.first_command { + if resources[j].1.last_batch < lifetime.first_batch { resources.swap_remove_front(j); index -= 1; } else { @@ -121,9 +179,9 @@ impl ResourceInterference { index += 1; - // resources is no longer sorted in the interval ..index, but because of the invariant `last_command >= first_command`, it is still partitioned by `a.last_command <= b.first_command`. - let active = resources - .partition_point(|&(_, other)| other.first_command <= lifetime.last_command); + // Active entries are partitioned by their inclusive batch intervals. + let active = + resources.partition_point(|&(_, other)| other.first_batch <= lifetime.last_batch); for &(other_id, _) in resources.range(..active) { edges[id.index()].insert(other_id); @@ -134,12 +192,62 @@ impl ResourceInterference { Self { edges } } - /// Returns whether both resources are live at any common command index. + /// Returns whether the resources interfere and therefore must not share memory. pub fn interferes(&self, a: ResourceId, b: ResourceId) -> bool { self.edges[a.index()].contains(&b) } } +/// Compact transitive-closure rows, with one bit for every command index. +/// Built once during `compile()` by walking the Kahn batches in reverse order. +#[derive(Debug, Clone, Default)] +pub struct HappensBefore { + rows: Vec, +} + +impl HappensBefore { + fn from_dag(edges: &[Vec], batches: &[Vec]) -> Self { + let mut closure = Self { + rows: (0..edges.len()) + .map(|_| FixedBitSet::with_capacity(edges.len())) + .collect(), + }; + + // Process nodes in reverse topological order so each successor row is complete. + for command in batches + .iter() + .flatten() + .rev() + .map(|command| command.index()) + { + for &successor in &edges[command] { + closure.rows[command].insert(successor); + if command != successor { + let (command_row, successor_row) = if command < successor { + let (before_successor, from_successor) = + closure.rows.split_at_mut(successor); + (&mut before_successor[command], &from_successor[0]) + } else { + let (before_command, from_command) = closure.rows.split_at_mut(command); + (&mut from_command[0], &before_command[successor]) + }; + command_row.union_with(successor_row); + } + } + } + closure + } + + fn contains(&self, from: usize, to: usize) -> bool { + self.rows.get(from).is_some_and(|row| row.contains(to)) + } + + /// Whether command `from` transitively precedes command `to`. + pub fn happens_before(&self, from: CommandIndex, to: CommandIndex) -> bool { + self.contains(from.index(), to.index()) + } +} + /// Deterministic Kahn topological-sort state for command dependencies. struct Kahn { edges: Vec>, @@ -201,8 +309,11 @@ impl Kahn { } } +/// A graph of resource uses whose compiled ordering also governs transient +/// resource aliasing. Aliasing is currently safe only under the single-queue +/// submission contract documented at the module level. pub struct RenderGraph { - next_id: u64, + next_id: u32, resources: Vec, imported_buffers_by_ptr: HashMap, imported_buffers_by_raw: HashMap, @@ -225,7 +336,7 @@ impl RenderGraph { } fn add_resource(&mut self, desc: ResourceDesc) -> ResourceId { - let id = ResourceId(self.next_id); + let id = ResourceId::from_u32(self.next_id); self.next_id = self.next_id.checked_add(1).expect("resource id overflow"); self.resources.push(desc); id @@ -381,7 +492,7 @@ impl RenderGraph { let (access, _) = descriptor_access(kind) .ok_or_else(|| fail(format!("unsupported descriptor type {kind:?}")))?; staged_accesses.push(DescriptorResourceAccess::Buffer { - buffer: Buffer(ResourceId(u64::MAX)), + buffer: Buffer(ResourceId::from_u32(u32::MAX)), range, stage, access, @@ -408,7 +519,7 @@ impl RenderGraph { .ok_or_else(|| fail(format!("unsupported descriptor type {kind:?}")))?; let layout = forced_layout.unwrap_or(*layout); staged_accesses.push(DescriptorResourceAccess::Texture { - texture: Texture(ResourceId(u64::MAX)), + texture: Texture(ResourceId::from_u32(u32::MAX)), range: *range, layout, stage, @@ -434,7 +545,7 @@ impl RenderGraph { let (access, _) = descriptor_access(kind) .ok_or_else(|| fail(format!("unsupported descriptor type {kind:?}")))?; staged_accesses.push(DescriptorResourceAccess::Buffer { - buffer: Buffer(ResourceId(u64::MAX)), + buffer: Buffer(ResourceId::from_u32(u32::MAX)), range: BufferRange { offset: start, size, @@ -613,6 +724,11 @@ impl RenderGraph { self.commands.push(command) } + /// Compile dependencies, their transitive happens-before closure, and + /// lifetimes for transient-alias analysis. Placement-specific alias barriers + /// are added by `CompiledGraph::resolve`. The closure is derived from + /// resource-hazard edges; batch membership alone is not proof that uses are + /// ordered. pub fn compile(self) -> Result { // Check for conflicting draw layouts before building the dependency graph. for accesses in self.commands.side_effects.command_accesses() { @@ -640,21 +756,41 @@ impl RenderGraph { let command_count = self.commands.len(); let accesses = self.commands.side_effects.command_accesses(); let compiler = GraphCompiler::new(&self.resources, accesses, command_count); - let batches = compiler.topo_batches()?; + let (batches, dependency_edges) = compiler.topo_batches_and_dependencies()?; compiler.validate_resources()?; let barriers = compiler.build_barriers(&batches); - let (lifetimes, interference) = compiler.bind_lifetimes(); + let happens_before = HappensBefore::from_dag(&dependency_edges, &batches); + let uses_by_resource = compiler.resource_uses(); + let (lifetimes, interference) = + compiler.bind_lifetimes(&batches, &happens_before, &uses_by_resource); + let ordinary_barrier_count = barriers.len(); + let mut barriers = barriers; + + compiler.build_alias_barriers( + &mut barriers, + &lifetimes, + &uses_by_resource, + &happens_before, + ); + + let barrier_indices_by_command = grouped_barrier_indices(&barriers); Ok(CompiledGraph { resources: self.resources, resolved: HashMap::new(), + placements_resolved: false, commands: self.commands, batches, barriers, + barrier_indices_by_command, + ordinary_barrier_count, lifetimes, interference, + dependency_edges, + happens_before, + uses_by_resource, }) } } @@ -680,24 +816,59 @@ impl<'a> GraphCompiler<'a> { fn dependencies(&self) -> Kahn { let mut kahn = Kahn::new(self.command_count); + let mut accesses_by_resource: HashMap)>> = + HashMap::new(); + + for (command, accesses) in self.accesses.iter().enumerate() { + for access in accesses { + let resource_accesses = accesses_by_resource.entry(access.resource).or_default(); + if let Some((last_command, scopes)) = resource_accesses.last_mut() + && *last_command == command + { + scopes.push(access.scope); + } else { + resource_accesses.push((command, vec![access.scope])); + } + } + } + + let mut priors_by_command = vec![Vec::new(); self.command_count]; + for resource_accesses in accesses_by_resource.values() { + for (current_position, (current, current_scopes)) in + resource_accesses.iter().enumerate() + { + for (prior, prior_scopes) in &resource_accesses[..current_position] { + if prior_scopes.iter().any(|prior_scope| { + current_scopes + .iter() + .any(|current_scope| prior_scope.conflicts(current_scope)) + }) { + priors_by_command[*current].push(*prior); + } + } + } + } for current in 0..self.command_count { - for prior in 0..current { - let dependency = self.accesses[prior].iter().any(|earlier| { - self.accesses[current].iter().any(|later| { - earlier.resource == later.resource && earlier.scope.conflicts(&later.scope) - }) - }); - if dependency { - kahn.add_edge(prior, current); - } + priors_by_command[current].sort_unstable(); + priors_by_command[current].dedup(); + for &prior in &priors_by_command[current] { + kahn.add_edge(prior, current); } } kahn } fn topo_batches(&self) -> Result>, GraphError> { + self.topo_batches_and_dependencies() + .map(|(batches, _)| batches) + } + + fn topo_batches_and_dependencies( + &self, + ) -> Result<(Vec>, Vec>), GraphError> { let mut kahn = self.dependencies(); + let dependency_edges = kahn.edges.clone(); let mut batches = Vec::new(); while !kahn.is_complete() { let batch = kahn.pop_batch(); @@ -711,7 +882,7 @@ impl<'a> GraphCompiler<'a> { .collect(); batches.push(batch); } - Ok(batches) + Ok((batches, dependency_edges)) } fn validate_resources(&self) -> Result<(), GraphError> { @@ -747,28 +918,43 @@ impl<'a> GraphCompiler<'a> { } let mut result: Vec = Vec::new(); + let mut prior_accesses_by_resource: HashMap< + ResourceId, + Vec<(CommandIndex, crate::recorder::ResourceAccess)>, + > = HashMap::new(); + let mut prior_access_summaries: HashMap = HashMap::new(); - for (ordered_index, &command_index) in batches.iter().flatten().enumerate() { + for &command_index in batches.iter().flatten() { let mut barrier_resources = Vec::new(); for current in &self.accesses[command_index.index()] { // The most recent conflicting access is the complete source scope // for this barrier; prior barrier masks are not carried forward. - let previous = batches - .iter() + let read_only = crate::resources::access_flag_is_read(current.scope.access) + && !access_flag_is_write(current.scope.access); + let can_skip_history_scan = read_only + && prior_access_summaries + .get(¤t.resource) + .is_some_and(|summary| { + !summary.has_write + && !summary.has_non_read_only + && match current.scope.subresource { + SubresourceScopeInner::Buffer(_) => true, + SubresourceScopeInner::Texture { layout, .. } => { + summary.uniform_layout == Some(layout) + } + } + }); + let previous = (!can_skip_history_scan) + .then(|| prior_accesses_by_resource.get(¤t.resource)) .flatten() - .take(ordered_index) - .filter_map(|&prior_command| { - self.accesses[prior_command.index()] + .and_then(|prior_accesses| { + prior_accesses .iter() .rev() - .find(|prior| { - prior.resource == current.resource - && prior.scope.conflicts(¤t.scope) - }) - .map(|prior| (prior_command, prior)) + .find(|(_, prior)| prior.scope.conflicts(¤t.scope)) }) - .last(); + .map(|(command, access)| (*command, *access)); if let Some((previous_command, previous)) = previous { debug_assert!( @@ -795,8 +981,8 @@ impl<'a> GraphCompiler<'a> { continue; } - let desc = &self.resources[current.resource.0 as usize]; - let initial = initial_states[current.resource.0 as usize]; + let desc = &self.resources[current.resource.index()]; + let initial = initial_states[current.resource.index()]; let (old_layout, new_layout) = Self::layouts_from_state(initial, current.scope); @@ -827,14 +1013,215 @@ impl<'a> GraphCompiler<'a> { if !barrier_resources.is_empty() { result.push(PipelineBarrier { before_command: command_index, + memory_barriers: Vec::new(), resources: barrier_resources, }); } + + for access in &self.accesses[command_index.index()] { + prior_accesses_by_resource + .entry(access.resource) + .or_default() + .push((command_index, *access)); + + let summary = prior_access_summaries.entry(access.resource).or_default(); + let is_read = crate::resources::access_flag_is_read(access.scope.access); + let is_write = access_flag_is_write(access.scope.access); + summary.has_write |= is_write; + summary.has_non_read_only |= !is_read || is_write; + if let SubresourceScopeInner::Texture { layout, .. } = access.scope.subresource { + if let Some(uniform_layout) = summary.uniform_layout + && uniform_layout != layout + { + summary.uniform_layout = None; + summary.texture_layouts_diverged = true; + } else if summary.uniform_layout.is_none() && !summary.texture_layouts_diverged + { + summary.uniform_layout = Some(layout); + } + } + } } result } + fn build_alias_barriers( + &self, + barriers: &mut Vec, + lifetimes: &HashMap, + uses_by_resource: &HashMap>, + happens_before: &HappensBefore, + ) { + let mut barrier_index_by_command = HashMap::new(); + for (index, barrier) in barriers.iter().enumerate() { + barrier_index_by_command + .entry(barrier.before_command) + .or_insert(index); + } + + let mut resources_by_pool: HashMap> = HashMap::new(); + for (&id, lifetime) in lifetimes { + if let Some(pool) = lifetime.placement.pool { + resources_by_pool.entry(pool).or_default().push(id); + } + } + + let mut new_resources = lifetimes + .iter() + .filter_map(|(&id, lifetime)| { + (lifetime.placement.aliased && !lifetime.imported).then_some(id) + }) + .collect::>(); + new_resources.sort_unstable(); + + for new_id in new_resources { + let Some(new_lifetime) = lifetimes.get(&new_id) else { + continue; + }; + let Some(new_uses) = uses_by_resource.get(&new_id) else { + continue; + }; + let new_first_uses = initial_uses(new_uses, happens_before); + if new_first_uses.is_empty() { + continue; + } + + let mut old_resources = new_lifetime + .placement + .pool + .and_then(|pool| resources_by_pool.get(&pool)) + .into_iter() + .flatten() + .filter_map(|&old_id| { + let old_lifetime = lifetimes.get(&old_id)?; + (old_id != new_id + && !old_lifetime.imported + && placement_ranges_overlap(old_lifetime.placement, new_lifetime.placement)) + .then_some(old_id) + }) + .collect::>(); + old_resources.sort_unstable(); + + let old_terminal_uses = old_resources + .iter() + .map(|old_id| { + uses_by_resource + .get(old_id) + .map(|uses| terminal_uses(uses, happens_before)) + .unwrap_or_default() + }) + .collect::>(); + + for &first_use in &new_first_uses { + let mut old_stage = vk::PipelineStageFlags2::NONE; + let mut old_access = vk::AccessFlags2::NONE; + let mut old_image_writes = false; + let mut has_old_image = false; + let mut merged_src_stage = vk::PipelineStageFlags2::NONE; + let mut merged_src_access = vk::AccessFlags2::NONE; + let mut has_old_buffer = false; + + for old_terminal_uses in &old_terminal_uses { + if old_terminal_uses.is_empty() + || !old_terminal_uses.iter().all(|terminal| { + happens_before.happens_before(terminal.command, first_use.command) + }) + { + continue; + } + + let (src_stage, src_access) = aggregate_resource_uses(old_terminal_uses); + match &self.resources[new_id.index()] { + ResourceDesc::CreatedBuffer(_) => { + has_old_buffer = true; + merged_src_stage |= src_stage; + merged_src_access |= if old_terminal_uses + .iter() + .any(|use_| access_flag_is_write(use_.access)) + { + src_access + } else { + vk::AccessFlags2::NONE + }; + } + ResourceDesc::CreatedTexture(_) => { + has_old_image = true; + old_stage |= src_stage; + old_access |= src_access; + old_image_writes |= old_terminal_uses + .iter() + .any(|use_| access_flag_is_write(use_.access)); + } + ResourceDesc::ImportedBuffer { .. } + | ResourceDesc::ImportedTexture { .. } => {} + } + } + + if has_old_image { + let barrier = pipeline_barrier_at_command_indexed( + barriers, + &mut barrier_index_by_command, + first_use.command, + ); + let mut found_image_barrier = false; + for image_barrier in barrier + .resources + .iter_mut() + .filter(|resource| resource.id == new_id) + { + found_image_barrier = true; + image_barrier.src_stage = old_stage; + image_barrier.src_access = if old_image_writes { + old_access + } else { + vk::AccessFlags2::NONE + }; + image_barrier.dst_stage = first_use.stage; + image_barrier.dst_access = first_use.access; + image_barrier.old_layout = Some(vk::ImageLayout::UNDEFINED); + } + if !found_image_barrier { + barrier.resources.push(BarrierResource { + id: new_id, + range: scope_range(first_use_scope(first_use, self.accesses, new_id)), + src_stage: old_stage, + src_access: if old_image_writes { + old_access + } else { + vk::AccessFlags2::NONE + }, + dst_stage: first_use.stage, + dst_access: first_use.access, + old_layout: Some(vk::ImageLayout::UNDEFINED), + new_layout: match first_use_scope(first_use, self.accesses, new_id) + .subresource + { + SubresourceScopeInner::Texture { layout, .. } => Some(layout), + SubresourceScopeInner::Buffer(_) => None, + }, + queue_transfer: None, + }); + } + } + + if has_old_buffer { + let barrier = pipeline_barrier_at_command_indexed( + barriers, + &mut barrier_index_by_command, + first_use.command, + ); + barrier.memory_barriers.push(MemoryBarrier { + src_stage: merged_src_stage, + src_access: merged_src_access, + dst_stage: first_use.stage, + dst_access: first_use.access, + }); + } + } + } + } + fn layouts( previous: SubresourceScope, current: SubresourceScope, @@ -859,68 +1246,496 @@ impl<'a> GraphCompiler<'a> { (initial.layout, current_layout) } - fn bind_lifetimes(&self) -> (HashMap, ResourceInterference) { + fn bind_lifetimes( + &self, + batches: &[Vec], + happens_before: &HappensBefore, + uses_by_resource: &HashMap>, + ) -> (HashMap, ResourceInterference) { + let mut batch_for_command = vec![usize::MAX; self.command_count]; + for (batch_index, batch) in batches.iter().enumerate() { + for command in batch { + batch_for_command[command.index()] = batch_index; + } + } + let mut lifetimes = HashMap::new(); for (command, command_accesses) in self.accesses.iter().enumerate() { for access in command_accesses { lifetimes .entry(access.resource) .and_modify(|lifetime: &mut ResourceLifetime| { - lifetime.last_command = command.into() + lifetime.first_batch = lifetime.first_batch.min(batch_for_command[command]); + lifetime.last_batch = lifetime.last_batch.max(batch_for_command[command]); }) .or_insert(ResourceLifetime { - first_command: command.into(), - last_command: command.into(), - memory_offset: 0, + first_batch: batch_for_command[command], + last_batch: batch_for_command[command], + placement: Placement { + offset: 0, + size: match &self.resources[access.resource.index()] { + ResourceDesc::CreatedBuffer(_) + | ResourceDesc::CreatedTexture(_) => 0, + ResourceDesc::ImportedBuffer { .. } + | ResourceDesc::ImportedTexture { .. } => 0, + }, + alignment: 1, + pool: None, + aliased: false, + }, + imported: !self.resources[access.resource.index()].is_created(), }); } } - let interference = ResourceInterference::from_lifetimes( - lifetimes - .iter() - // TODO: consider resource usage across multiple queues. - // when two resources do not overlap in time but their last-first commands are not connected via an edge in the dag, they must be marked as interfering. - .filter(|&(id, _)| self.resources[id.0 as usize].is_created()) - .map(|(&id, &lifetime)| (id, lifetime)), - ); + let interference = + ResourceInterference::from_resource_uses(uses_by_resource, happens_before); - // Skeleton aliasing policy: resources with non-overlapping lifetimes may - // share a heap offset. The graph test is equivalent to the previous - // strict `last_use < first_command` check (intervals are inclusive). - // Actual size/alignment-aware Vulkan allocation is TODO. - let mut slots: Vec<(u64, ResourceId, u64)> = Vec::new(); // (offset, last occupant, capacity) - let mut next_offset = 0; - let mut ids: Vec<_> = lifetimes.keys().copied().collect(); - ids.sort_by_key(|id| lifetimes[id].first_command); - for id in ids { - let lifetime = lifetimes.get_mut(&id).expect("lifetime key exists"); - let desc = &self.resources[id.0 as usize]; - let size = match desc { - ResourceDesc::CreatedBuffer(desc) => desc.size, - ResourceDesc::CreatedTexture(desc) => texture_size_estimate(desc), - ResourceDesc::ImportedBuffer { .. } | ResourceDesc::ImportedTexture { .. } => 0, - }; - if desc.is_created() - && let Some((offset, last_use, capacity)) = slots - .iter_mut() - .find(|(_, last, cap)| !interference.interferes(*last, id) && *cap >= size) - { - lifetime.memory_offset = *offset; - *last_use = id; - let _ = capacity; - } else { - lifetime.memory_offset = next_offset; - if desc.is_created() { - slots.push((next_offset, id, size)); - next_offset = next_offset.saturating_add(size); + (lifetimes, interference) + } + + fn resource_uses(&self) -> HashMap> { + let mut uses_by_resource: HashMap> = HashMap::new(); + for (command, command_accesses) in self.accesses.iter().enumerate() { + for access in command_accesses { + let resource_uses = uses_by_resource.entry(access.resource).or_default(); + if let Some(resource_use) = resource_uses.last_mut() + && resource_use.command.index() == command + { + resource_use.stage |= access.scope.stage; + resource_use.access |= access.scope.access; + } else { + resource_uses.push(ResourceUse { + command: command.into(), + stage: access.scope.stage, + access: access.scope.access, + }); } } } - (lifetimes, interference) + uses_by_resource } } +#[derive(Default)] +struct PriorAccessSummary { + has_write: bool, + has_non_read_only: bool, + uniform_layout: Option, + texture_layouts_diverged: bool, +} + +impl ResourceInterference { + fn from_resource_uses( + uses_by_resource: &HashMap>, + happens_before: &HappensBefore, + ) -> Self { + let mut result = Self { + edges: vec![ + BTreeSet::new(); + uses_by_resource + .keys() + .map(|id| id.index()) + .max() + .unwrap_or(0) + + 1 + ], + }; + let resource_ids = uses_by_resource.keys().copied().collect::>(); + let mut frontiers: Vec, Vec<&ResourceUse>)>> = + vec![None; result.edges.len()]; + for &resource in &resource_ids { + let uses = &uses_by_resource[&resource]; + frontiers[resource.index()] = Some(( + terminal_uses(uses, happens_before), + initial_uses(uses, happens_before), + )); + } + + for (index, &a) in resource_ids.iter().enumerate() { + for &b in &resource_ids[index + 1..] { + let (terminal_a, initial_a) = frontiers[a.index()] + .as_ref() + .expect("resource frontier was computed"); + let (terminal_b, initial_b) = frontiers[b.index()] + .as_ref() + .expect("resource frontier was computed"); + if !resource_use_sets_alias_in_order(&terminal_a, &initial_b, happens_before) + && !resource_use_sets_alias_in_order(&terminal_b, &initial_a, happens_before) + { + result.edges[a.index()].insert(b); + result.edges[b.index()].insert(a); + } + } + } + result + } +} + +fn terminal_uses<'a>( + uses: &'a [ResourceUse], + happens_before: &HappensBefore, +) -> Vec<&'a ResourceUse> { + uses.iter() + .filter(|candidate| { + !uses.iter().any(|other| { + candidate.command != other.command + && happens_before.contains(candidate.command.index(), other.command.index()) + }) + }) + .collect() +} + +fn initial_uses<'a>( + uses: &'a [ResourceUse], + happens_before: &HappensBefore, +) -> Vec<&'a ResourceUse> { + uses.iter() + .filter(|candidate| { + !uses.iter().any(|other| { + candidate.command != other.command + && happens_before.contains(other.command.index(), candidate.command.index()) + }) + }) + .collect() +} + +/// Returns whether every terminal use of A precedes every initial use of B. +fn resources_alias_in_order( + uses_a: &[ResourceUse], + uses_b: &[ResourceUse], + happens_before: &HappensBefore, +) -> bool { + aliasing_permitted(uses_a, uses_b, happens_before) +} + +/// Returns whether every terminal use of A precedes every initial use of B. +/// Empty use sets are rejected rather than treated as vacuously safe. This +/// ordering proves alias safety only under the current single-queue submission +/// contract; if command-to-queue assignment is added, cross-queue alias +/// transitions must also be semaphore edges. +/// +/// The graph currently records all commands into one command buffer. It has no +/// command-to-queue assignment, so cross-queue aliasing is out of scope. +pub fn aliasing_permitted( + uses_a: &[ResourceUse], + uses_b: &[ResourceUse], + happens_before: &HappensBefore, +) -> bool { + let terminal_a = terminal_uses(uses_a, happens_before); + let initial_b = initial_uses(uses_b, happens_before); + resource_use_sets_alias_in_order(&terminal_a, &initial_b, happens_before) +} + +fn resource_use_sets_alias_in_order( + terminal_a: &[&ResourceUse], + initial_b: &[&ResourceUse], + happens_before: &HappensBefore, +) -> bool { + !terminal_a.is_empty() + && !initial_b.is_empty() + && terminal_a.iter().all(|terminal| { + initial_b + .iter() + .all(|initial| happens_before.happens_before(terminal.command, initial.command)) + }) +} + +fn aggregate_resource_uses(uses: &[&ResourceUse]) -> (vk::PipelineStageFlags2, vk::AccessFlags2) { + uses.iter().fold( + (vk::PipelineStageFlags2::NONE, vk::AccessFlags2::NONE), + |(stage, access), resource_use| (stage | resource_use.stage, access | resource_use.access), + ) +} + +fn placement_ranges_overlap(a: Placement, b: Placement) -> bool { + if a.pool.is_none() || a.pool != b.pool || a.size == 0 || b.size == 0 { + return false; + } + let (Some(a_end), Some(b_end)) = (a.offset.checked_add(a.size), b.offset.checked_add(b.size)) + else { + return false; + }; + a.offset < b_end && b.offset < a_end +} + +fn is_host_visible_location(location: MemoryLocation) -> bool { + matches!( + location, + MemoryLocation::CpuToGpu | MemoryLocation::GpuToCpu + ) +} + +fn is_poolable( + location: MemoryLocation, + strategy: AllocationStrategy, + requirements: renderer_resources::memory::MemoryRequirements, +) -> bool { + !is_host_visible_location(location) + && strategy != AllocationStrategy::Dedicated + && !requirements.requires_dedicated_allocation +} + +fn allocation_strategy_for( + requirements: renderer_resources::memory::MemoryRequirements, +) -> AllocationStrategy { + if requirements.requires_dedicated_allocation + || requirements.size >= DEDICATED_ALLOCATION_THRESHOLD + { + AllocationStrategy::Dedicated + } else { + AllocationStrategy::AllocatorManaged + } +} + +fn allocation_strategy_for_size( + requested_size: u64, + requirements: renderer_resources::memory::MemoryRequirements, +) -> AllocationStrategy { + if requested_size >= DEDICATED_ALLOCATION_THRESHOLD { + AllocationStrategy::Dedicated + } else { + allocation_strategy_for(requirements) + } +} + +fn pool_class_for_image(tiling: vk::ImageTiling) -> PoolClass { + if tiling == vk::ImageTiling::LINEAR { + PoolClass::Linear + } else { + PoolClass::Optimal + } +} + +fn validate_pooled_image_layout( + id: ResourceId, + desc: &GraphTextureDesc, + requirements: renderer_resources::memory::MemoryRequirements, +) -> Result<(), GraphError> { + if is_poolable( + desc.mem_location, + allocation_strategy_for(requirements), + requirements, + ) && desc.layout != vk::ImageLayout::UNDEFINED + { + return Err(GraphError::InvalidPooledImageLayout(id)); + } + Ok(()) +} + +fn memory_property_flags(location: MemoryLocation) -> vk::MemoryPropertyFlags { + match location { + MemoryLocation::CpuToGpu | MemoryLocation::GpuToCpu => { + vk::MemoryPropertyFlags::HOST_VISIBLE | vk::MemoryPropertyFlags::HOST_COHERENT + } + MemoryLocation::GpuOnly => vk::MemoryPropertyFlags::DEVICE_LOCAL, + MemoryLocation::Unknown => vk::MemoryPropertyFlags::empty(), + } +} + +fn pool_accepts_member( + pool: &PlacementPool, + class: PoolClass, + property_flags: vk::MemoryPropertyFlags, + requirements: renderer_resources::memory::MemoryRequirements, +) -> bool { + pool.class == class + && pool.property_flags == property_flags + && requirements.memory_type_bits & (1 << pool.memory_type_index) != 0 +} + +fn should_plan_created_resource( + id: ResourceId, + lifetimes: &HashMap, + resolved: &HashMap, +) -> bool { + lifetimes + .get(&id) + .is_some_and(|lifetime| !lifetime.imported) + && !resolved.contains_key(&id) +} + +fn set_placement( + lifetimes: &mut HashMap, + id: ResourceId, + requirements: renderer_resources::memory::MemoryRequirements, + offset: u64, + pool: Option, + aliased: bool, +) { + if let Some(lifetime) = lifetimes.get_mut(&id) { + lifetime.placement = Placement { + offset, + size: requirements.size, + alignment: requirements.alignment, + pool, + aliased, + }; + } +} + +fn byte_ranges_overlap(a_offset: u64, a_size: u64, b_offset: u64, b_size: u64) -> bool { + let (Some(a_end), Some(b_end)) = (a_offset.checked_add(a_size), b_offset.checked_add(b_size)) + else { + return true; + }; + a_offset < b_end && b_offset < a_end +} + +fn checked_alignment_lcm(lhs: u64, rhs: u64) -> Option { + if lhs == 0 || rhs == 0 { + return None; + } + let (mut a, mut b) = (lhs, rhs); + while b != 0 { + (a, b) = (b, a % b); + } + lhs.checked_div(a)?.checked_mul(rhs) +} + +fn first_fit_placement( + id: ResourceId, + size: u64, + alignment: u64, + occupied: &[(ResourceId, u64)], + lifetimes: &HashMap, + interference: &ResourceInterference, + uses_by_resource: &HashMap>, + happens_before: &HappensBefore, +) -> Option { + if alignment == 0 || size == 0 { + return None; + } + + // Validate all lifetimes up front to preserve the missing-lifetime failure behavior. + for (other_id, _) in occupied { + lifetimes.get(other_id)?; + } + + let mut candidate = 0_u64; + let mut range_index = 0; + while range_index < occupied.len() { + let aligned = candidate + .checked_add(alignment - 1)? + .checked_div(alignment)? + .checked_mul(alignment)?; + let candidate_end = aligned.checked_add(size)?; + let (other_id, other_offset) = occupied[range_index]; + let other_size = lifetimes.get(&other_id)?.placement.size; + let other_end = other_offset.checked_add(other_size); + + if other_end.is_some_and(|end| end <= aligned) { + range_index += 1; + continue; + } + if other_offset >= candidate_end && other_end.is_some() { + return Some(aligned); + } + if !byte_ranges_overlap(aligned, size, other_offset, other_size) { + range_index += 1; + continue; + } + + let ordered = uses_by_resource + .get(&id) + .zip(uses_by_resource.get(&other_id)) + .is_some_and(|(new_uses, old_uses)| { + !interference.interferes(id, other_id) + && (resources_alias_in_order(new_uses, old_uses, happens_before) + || resources_alias_in_order(old_uses, new_uses, happens_before)) + }); + + if ordered { + range_index += 1; + continue; + } + + let next_offset = other_end?; + if next_offset <= aligned { + return None; + } + candidate = next_offset; + } + + let aligned = candidate + .checked_add(alignment - 1)? + .checked_div(alignment)? + .checked_mul(alignment)?; + aligned.checked_add(size)?; + Some(aligned) +} + +fn image_desc_for_graph_texture( + desc: &GraphTextureDesc, + accesses: &[crate::recorder::ResourceAccess], + queue_families: QueueFlags, + alloc_scheme: AllocationStrategy, + flags: vk::ImageCreateFlags, +) -> ImageDesc { + ImageDesc { + flags, + name: desc.name.clone(), + format: desc.format, + kind: desc.kind, + mip_levels: desc.mip_levels, + array_layers: desc.array_layers, + samples: desc.samples, + extent: desc.extent, + tiling: desc.tiling, + usage: derive_image_usage(accesses, desc.extra_usage), + queue_families, + layout: desc.layout, + mem_location: desc.mem_location, + alloc_scheme, + } +} + +fn first_use_scope<'a>( + resource_use: &ResourceUse, + accesses: &'a [Vec], + resource: ResourceId, +) -> SubresourceScope { + accesses[resource_use.command.index()] + .iter() + .find(|access| access.resource == resource) + .expect("resource use has a matching declared access") + .scope +} + +fn pipeline_barrier_at_command( + barriers: &mut Vec, + command: CommandIndex, +) -> &mut PipelineBarrier { + let index = barriers + .iter() + .position(|barrier| barrier.before_command == command) + .unwrap_or_else(|| { + barriers.push(PipelineBarrier { + before_command: command, + memory_barriers: Vec::new(), + resources: Vec::new(), + }); + barriers.len() - 1 + }); + &mut barriers[index] +} + +fn pipeline_barrier_at_command_indexed<'a>( + barriers: &'a mut Vec, + index_by_command: &mut HashMap, + command: CommandIndex, +) -> &'a mut PipelineBarrier { + let index = *index_by_command.entry(command).or_insert_with(|| { + barriers.push(PipelineBarrier { + before_command: command, + memory_barriers: Vec::new(), + resources: Vec::new(), + }); + barriers.len() - 1 + }); + &mut barriers[index] +} + impl Default for RenderGraph { fn default() -> Self { Self::new() @@ -1180,14 +1995,50 @@ fn shader_stages_to_pipeline(stages: vk::ShaderStageFlags) -> Option, resolved: HashMap, + placements_resolved: bool, commands: CommandList, batches: Vec>, barriers: Vec, + barrier_indices_by_command: HashMap>, + ordinary_barrier_count: usize, lifetimes: HashMap, interference: ResourceInterference, + dependency_edges: Vec>, + happens_before: HappensBefore, + uses_by_resource: HashMap>, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum PoolClass { + Linear, + Optimal, +} + +struct PendingPlacement { + id: ResourceId, + resource: ResolvedResource, + requirements: renderer_resources::memory::MemoryRequirements, + class: PoolClass, + property_flags: vk::MemoryPropertyFlags, +} + +struct PlacementPool { + id: u32, + class: PoolClass, + memory_type_index: u32, + property_flags: vk::MemoryPropertyFlags, + size: u64, + members: Vec<(ResourceId, u64)>, + occupied: Vec<(ResourceId, u64)>, } enum ResolvedResource { @@ -1212,67 +2063,432 @@ impl CompiledGraph { &self.interference } - /// Create the graph's transient resources. Compilation remains independent - /// of a Vulkan device, while command recording requires this resolution step - /// for any created resources used by commands. - pub fn resolve(&mut self, device: &Device) -> renderer_resources::Result<()> { + /// Whether command `a` is transitively ordered before command `b` by + /// resource-hazard edges. Commands in the same batch are not ordered. + pub fn happens_before(&self, a: CommandIndex, b: CommandIndex) -> bool { + self.happens_before.contains(a.index(), b.index()) + } + + /// The compile-time transitive closure of resource-hazard ordering. + pub fn happens_before_closure(&self) -> &HappensBefore { + &self.happens_before + } + + /// The retained command adjacency used to build the compile-time closure. + pub fn dependency_edges(&self) -> &[Vec] { + &self.dependency_edges + } + + /// Per-resource command uses, with stage/access masks ORed across accesses + /// to that resource within each command. + pub fn resource_uses(&self, resource: ResourceId) -> Option<&[ResourceUse]> { + self.uses_by_resource.get(&resource).map(Vec::as_slice) + } + + /// Replace placement-specific alias barriers while retaining compile-time ordinary barriers. + fn rebuild_barriers(&mut self) { + let compiler = GraphCompiler::new( + &self.resources, + self.commands.side_effects.command_accesses(), + self.commands.len(), + ); + self.barriers.truncate(self.ordinary_barrier_count); + compiler.build_alias_barriers( + &mut self.barriers, + &self.lifetimes, + &self.uses_by_resource, + &self.happens_before, + ); + self.barrier_indices_by_command = grouped_barrier_indices(&self.barriers); + } + + /// Tests whether two use sets can safely reuse a placement in the supplied + /// direction. Stage/access masks are retained for alias barriers; ordering + /// is proven from command ids through the dependency closure. + pub fn aliasing_permitted(&self, uses_a: &[ResourceUse], uses_b: &[ResourceUse]) -> bool { + resources_alias_in_order(uses_a, uses_b, &self.happens_before) + } + + /// Tests whether two resources can reuse a placement in either safe order. + pub fn resources_can_alias(&self, a: ResourceId, b: ResourceId) -> bool { + let (Some(uses_a), Some(uses_b)) = (self.resource_uses(a), self.resource_uses(b)) else { + return false; + }; + self.aliasing_permitted(uses_a, uses_b) || self.aliasing_permitted(uses_b, uses_a) + } + + /// Create and bind used transient resources and finalize alias barriers. + /// + /// Aliasing safety assumes all graph commands are submitted from one command + /// buffer to one queue. Cross-queue aliasing is unsupported; when + /// per-command queue assignment exists, alias transitions between queues + /// must become semaphore edges as well as the corresponding memory barriers. + pub fn resolve(&mut self, device: &Device) -> Result<(), GraphError> { + if self.plan_placements(device)? { + self.rebuild_barriers(); + } + Ok(()) + } + + /// Stage resource creation and placements locally; publish only after all binds succeed. + fn plan_placements(&mut self, device: &Device) -> Result { + if self.placements_resolved { + let has_unresolved_created_resource = + self.resources.iter().enumerate().any(|(index, desc)| { + matches!( + desc, + ResourceDesc::CreatedBuffer(_) | ResourceDesc::CreatedTexture(_) + ) && self + .lifetimes + .contains_key(&ResourceId::from_u32(index as u32)) + && !self + .resolved + .contains_key(&ResourceId::from_u32(index as u32)) + }); + if !has_unresolved_created_resource { + return Ok(false); + } + } + + let command_accesses = self.commands.side_effects.command_accesses(); + let mut accesses_by_resource = vec![Vec::new(); self.resources.len()]; + for access in command_accesses.iter().flatten() { + accesses_by_resource[access.resource.index()].push(*access); + } + let mut staged_resources = Vec::new(); + let mut staged_lifetimes = self.lifetimes.clone(); + let mut pooled = Vec::new(); + + // Create only resources with actual graph uses; unused transients have no lifetime. for (index, desc) in self.resources.iter().enumerate() { - let id = ResourceId(index as u64); - if self.resolved.contains_key(&id) { + let id = ResourceId::from_u32(index as u32); + if !should_plan_created_resource(id, &staged_lifetimes, &self.resolved) { continue; } - let accesses = accesses_for_resource(id, self.commands.side_effects.command_accesses()); - let resource = match desc { + + let accesses = accesses_by_resource[id.index()].as_slice(); + match desc { ResourceDesc::CreatedBuffer(desc) => { - let usage = derive_buffer_usage(&accesses, desc.extra_usage); - let queue_families = - actual_queue_families(device, derive_queue_capabilities(&accesses)); - let resource_size = desc.size; - ResolvedResource::Buffer(Arc::new(BufferResource::new( - device.clone(), - BufferDesc { - flags: vk::BufferCreateFlags::empty(), - name: desc.name.clone(), - size: desc.size, - usage, - queue_families, - mem_location: desc.mem_location, - alloc_scheme: allocation_strategy(resource_size), - }, - )?)) + let desc = BufferDesc { + flags: vk::BufferCreateFlags::empty(), + name: desc.name.clone(), + size: desc.size, + usage: derive_buffer_usage(accesses, desc.extra_usage), + queue_families: actual_queue_families( + device, + derive_queue_capabilities(accesses), + ), + mem_location: desc.mem_location, + // Explicit allocation strategy is selected after the requirements query. + alloc_scheme: AllocationStrategy::AllocatorManaged, + }; + + let resource = BufferResource::new_unallocated(device.clone(), desc.clone()) + .map_err(|_| GraphError::PlacementFailed(id))?; + + let requirements = resource + .memory_requirements() + .map_err(|_| GraphError::PlacementFailed(id))?; + + if !is_poolable( + desc.mem_location, + allocation_strategy_for_size(desc.size, requirements), + requirements, + ) { + drop(resource); + + let resource = BufferResource::new( + device.clone(), + BufferDesc { + alloc_scheme: allocation_strategy_for_size(desc.size, requirements), + ..desc + }, + ) + .map_err(|_| GraphError::PlacementFailed(id))?; + + set_placement(&mut staged_lifetimes, id, requirements, 0, None, false); + + staged_resources.push((id, ResolvedResource::Buffer(Arc::new(resource)))); + continue; + } + + pooled.push(PendingPlacement { + id, + resource: ResolvedResource::Buffer(Arc::new(resource)), + requirements, + class: PoolClass::Linear, + property_flags: memory_property_flags(desc.mem_location), + }); + + continue; } ResourceDesc::CreatedTexture(desc) => { - let usage = derive_image_usage(&accesses, desc.extra_usage); - let queue_families = - actual_queue_families(device, derive_queue_capabilities(&accesses)); - let resource_size = texture_size_estimate(desc); - ResolvedResource::Image(Arc::new(ImageResource::new( - device.clone(), - ImageDesc { - flags: vk::ImageCreateFlags::empty(), - name: desc.name.clone(), - format: desc.format, - kind: desc.kind, - mip_levels: desc.mip_levels, - array_layers: desc.array_layers, - samples: desc.samples, - extent: desc.extent, - tiling: desc.tiling, - usage, - queue_families, - layout: desc.layout, - mem_location: desc.mem_location, - alloc_scheme: allocation_strategy(resource_size), - }, - )?)) + let image_desc = ImageDesc { + // Candidates start alias-capable; non-poolable candidates are recreated + // without the alias flag below. + flags: vk::ImageCreateFlags::ALIAS, + name: desc.name.clone(), + format: desc.format, + kind: desc.kind, + mip_levels: desc.mip_levels, + array_layers: desc.array_layers, + samples: desc.samples, + extent: desc.extent, + tiling: desc.tiling, + usage: derive_image_usage(accesses, desc.extra_usage), + queue_families: actual_queue_families( + device, + derive_queue_capabilities(accesses), + ), + // Query requirements from a Vulkan-valid undefined candidate; pooled + // eligibility requires the graph's declared initial layout be undefined. + layout: vk::ImageLayout::UNDEFINED, + mem_location: desc.mem_location, + alloc_scheme: AllocationStrategy::AllocatorManaged, + }; + + let candidate = ImageResource::new_unallocated(device.clone(), image_desc) + .map_err(|_| GraphError::PlacementFailed(id))?; + + let requirements = candidate + .memory_requirements() + .map_err(|_| GraphError::PlacementFailed(id))?; + + if !is_poolable( + desc.mem_location, + allocation_strategy_for(requirements), + requirements, + ) { + // TODO: Consider reusing the candidate + drop(candidate); + + let resource = ImageResource::new( + device.clone(), + image_desc_for_graph_texture( + desc, + accesses, + actual_queue_families(device, derive_queue_capabilities(accesses)), + allocation_strategy_for_size( + desc.extent.width.into(), + requirements, + ), + vk::ImageCreateFlags::empty(), + ), + ) + .map_err(|_| GraphError::PlacementFailed(id))?; + + let requirements = resource + .memory_requirements() + .map_err(|_| GraphError::PlacementFailed(id))?; + + set_placement(&mut staged_lifetimes, id, requirements, 0, None, false); + + staged_resources.push((id, ResolvedResource::Image(Arc::new(resource)))); + + continue; + } + + validate_pooled_image_layout(id, desc, requirements)?; + + pooled.push(PendingPlacement { + id, + resource: ResolvedResource::Image(Arc::new(candidate)), + requirements, + class: pool_class_for_image(desc.tiling), + property_flags: memory_property_flags(desc.mem_location), + }); + + continue; } ResourceDesc::ImportedBuffer { .. } | ResourceDesc::ImportedTexture { .. } => { continue; } + } + } + + // Add resources to class/type-specific pools with intersection semantics. + let mut pools: Vec = Vec::new(); + pooled.sort_by_key(|member| member.id); + + for member in pooled { + let compatible_pool = pools.iter().position(|pool| { + pool_accepts_member( + pool, + member.class, + member.property_flags, + member.requirements, + ) + }); + + let pool_index = if let Some(pool_index) = compatible_pool { + pool_index + } else { + let memory_type_index = device + .find_memory_type_index(&member.requirements, member.property_flags) + .map_err(|_| GraphError::PlacementFailed(member.id))?; + + let id = u32::try_from(pools.len()) + .map_err(|_| GraphError::PlacementFailed(member.id))?; + + pools.push(PlacementPool { + id, + class: member.class, + memory_type_index, + property_flags: member.property_flags, + size: 0, + members: Vec::new(), + occupied: Vec::new(), + }); + + pools.len() - 1 }; + + let offset = first_fit_placement( + member.id, + member.requirements.size, + member.requirements.alignment, + &pools[pool_index].occupied, + &staged_lifetimes, + &self.interference, + &self.uses_by_resource, + &self.happens_before, + ) + .ok_or(GraphError::PlacementFailed(member.id))?; + + let end = offset + .checked_add(member.requirements.size) + .ok_or(GraphError::PlacementFailed(member.id))?; + + let pool = &mut pools[pool_index]; + pool.size = pool.size.max(end); + + let mut intersects_unsafe_lifetime = false; + let overlapping = pool + .members + .iter() + .filter_map(|(other_id, other_offset)| { + let overlaps = byte_ranges_overlap( + offset, + member.requirements.size, + *other_offset, + staged_lifetimes[other_id].placement.size, + ); + + if !overlaps { + return None; + } + + let is_ordered = self + .uses_by_resource + .get(&member.id) + .zip(self.uses_by_resource.get(other_id)) + .is_some_and(|(new_uses, existing_uses)| { + !self.interference.interferes(member.id, *other_id) + && (resources_alias_in_order( + new_uses, + existing_uses, + &self.happens_before, + ) || resources_alias_in_order( + existing_uses, + new_uses, + &self.happens_before, + )) + }); + + if !is_ordered { + intersects_unsafe_lifetime = true; + } + + Some(*other_id) + }) + .collect::>(); + + if intersects_unsafe_lifetime { + return Err(GraphError::PlacementFailed(member.id)); + } + + pool.members.push((member.id, offset)); + let occupied_index = pool + .occupied + .binary_search_by_key(&(offset, member.id), |(id, offset)| (*offset, *id)) + .unwrap_or_else(|index| index); + pool.occupied.insert(occupied_index, (member.id, offset)); + + for other_id in &overlapping { + if let Some(lifetime) = staged_lifetimes.get_mut(other_id) { + lifetime.placement.aliased = true; + } + } + + set_placement( + &mut staged_lifetimes, + member.id, + member.requirements, + offset, + Some(pool.id), + !overlapping.is_empty(), + ); + + staged_resources.push((member.id, member.resource)); + } + + // Allocate one shared block per pool, then transfer references to every bound member. + let mut staged_indices = HashMap::with_capacity(staged_resources.len()); + for (index, (id, _)) in staged_resources.iter().enumerate() { + staged_indices.insert(*id, index); + } + + for pool in pools { + let first_id = pool.members[0].0; + + let alignment = pool + .members + .iter() + .try_fold(1_u64, |alignment, (id, _)| { + checked_alignment_lcm(alignment, staged_lifetimes[id].placement.alignment) + }) + .ok_or(GraphError::PlacementFailed(first_id))?; + + let requirements = renderer_resources::memory::MemoryRequirements { + size: pool.size, + alignment, + memory_type_bits: 1 << pool.memory_type_index, + requires_dedicated_allocation: false, + prefers_dedicated_allocation: false, + }; + + let allocation = device + .allocate_memory(&requirements, pool.property_flags) + .map_err(|_| GraphError::PlacementFailed(first_id))?; + + for (id, _) in &pool.members { + let resource_index = *staged_indices.get(id).expect("every pool member is staged"); + let resource = &mut staged_resources[resource_index]; + + match &mut resource.1 { + ResolvedResource::Buffer(buffer) => Arc::get_mut(buffer) + .expect("new pooled buffer has a unique owner") + .bind_allocation(allocation.clone(), staged_lifetimes[id].placement.offset) + .map_err(|error| GraphError::from((*id, error)))?, + + ResolvedResource::Image(image) => Arc::get_mut(image) + .expect("new pooled image has a unique owner") + .bind_allocation(allocation.clone(), staged_lifetimes[id].placement.offset) + .map_err(|error| GraphError::from((*id, error)))?, + } + } + } + + // Commit only after every creation, allocation, and bind has succeeded. + let changed = !self.placements_resolved || !staged_resources.is_empty(); + for (id, resource) in staged_resources { self.resolved.insert(id, resource); } - Ok(()) + + self.lifetimes = staged_lifetimes; + self.placements_resolved = true; + + Ok(changed) } /// Record the compiled order into a real Vulkan command buffer. Imported @@ -1286,10 +2502,11 @@ impl CompiledGraph { self.commands.preflight()?; for (index, desc) in self.resources.iter().enumerate() { - let id = ResourceId(index as u64); + let id = ResourceId::from_u32(index as u32); if !self.lifetimes.contains_key(&id) { continue; } + match desc { ResourceDesc::ImportedBuffer { resource, .. } => { recorder.buffers.insert(id, resource.raw()); @@ -1312,9 +2529,16 @@ impl CompiledGraph { } } - // TODO: `batches` are currently recorded sequentially into one command buffer; - // record per-batch command buffers and submit sequentially/in parallel, using split barriers/events or timeline semaphores for overlap. - for step in recording_steps(&self.batches, &self.barriers) { + // Batches currently record into one command buffer, which must be + // submitted to one queue for current aliasing safety. TODO: separate + // per-batch command buffers may be submitted sequentially on that queue; + // parallel/cross-queue execution requires command-to-queue assignment + // and semaphore edges for every cross-queue alias transition. + for step in recording_steps_cached( + &self.batches, + &self.barriers, + &self.barrier_indices_by_command, + ) { match step { // TODO: Merge `PipelineBarrier.resources` across commands or a batch to reduce vkCmdPipelineBarrier2 calls. RecordingStep::Barrier(barrier) => recorder.pipeline_barrier(barrier), @@ -1336,27 +2560,53 @@ fn recording_steps<'a>( batches: &'a [Vec], barriers: &'a [PipelineBarrier], ) -> impl Iterator> + 'a { + let mut barriers_by_command: HashMap> = HashMap::new(); + for barrier in barriers { + barriers_by_command + .entry(barrier.before_command) + .or_default() + .push(barrier); + } + batches.iter().flatten().flat_map(move |&command_index| { - barriers - .iter() - .filter(move |barrier| barrier.before_command == command_index) + barriers_by_command + .remove(&command_index) + .unwrap_or_default() + .into_iter() .map(RecordingStep::Barrier) .chain(std::iter::once(RecordingStep::Command(command_index))) }) } -/// The skeleton switches to a dedicated allocation at 64 MiB. This is a -/// placeholder heuristic until resource lifetime/aliasing allocation is real. -const DEDICATED_ALLOCATION_THRESHOLD: u64 = 64 * 1024 * 1024; - -fn allocation_strategy(size: u64) -> AllocationStrategy { - if size >= DEDICATED_ALLOCATION_THRESHOLD { - AllocationStrategy::Dedicated - } else { - AllocationStrategy::AllocatorManaged +fn grouped_barrier_indices(barriers: &[PipelineBarrier]) -> HashMap> { + let mut barriers_by_command: HashMap> = HashMap::new(); + for (index, barrier) in barriers.iter().enumerate() { + barriers_by_command + .entry(barrier.before_command) + .or_default() + .push(index); } + barriers_by_command } +fn recording_steps_cached<'a>( + batches: &'a [Vec], + barriers: &'a [PipelineBarrier], + barrier_indices_by_command: &'a HashMap>, +) -> impl Iterator> + 'a { + batches.iter().flatten().flat_map(move |&command_index| { + barrier_indices_by_command + .get(&command_index) + .into_iter() + .flatten() + .map(move |&index| RecordingStep::Barrier(&barriers[index])) + .chain(std::iter::once(RecordingStep::Command(command_index))) + }) +} + +/// Very large resources bypass transient pools and use a dedicated allocation. +const DEDICATED_ALLOCATION_THRESHOLD: u64 = 64 * 1024 * 1024; + fn accesses_for_resource( id: ResourceId, command_accesses: &[Vec], @@ -1540,31 +2790,6 @@ fn actual_queue_families(device: &Device, capabilities: QueueFlags) -> QueueFlag } } -fn texture_size_estimate(desc: &GraphTextureDesc) -> u64 { - let texels_per_block = u64::from(desc.format.texels_per_block()).max(1); - let block_size = u64::from(desc.format.block_size()); - let mut size = 0u64; - let (mut width, mut height, mut depth) = ( - u64::from(desc.extent.width), - u64::from(desc.extent.height), - u64::from(desc.extent.depth), - ); - for _ in 0..desc.mip_levels { - size = size.saturating_add( - width - .saturating_mul(height) - .saturating_mul(depth) - .div_ceil(texels_per_block) - .saturating_mul(block_size), - ); - width = (width / 2).max(1); - height = (height / 2).max(1); - depth = (depth / 2).max(1); - } - size.saturating_mul(u64::from(desc.array_layers)) - .saturating_mul(u64::from(desc.samples.as_raw().max(1))) -} - #[cfg(test)] mod tests { use super::*; @@ -1672,6 +2897,328 @@ mod tests { )) } + #[test] + fn pool_classes_separate_linear_buffers_from_optimal_images() { + let pool = PlacementPool { + id: 0, + class: PoolClass::Linear, + memory_type_index: 0, + property_flags: vk::MemoryPropertyFlags::DEVICE_LOCAL, + size: 64, + members: Vec::new(), + occupied: Vec::new(), + }; + let requirements = renderer_resources::memory::MemoryRequirements { + size: 16, + alignment: 8, + memory_type_bits: 1, + requires_dedicated_allocation: false, + prefers_dedicated_allocation: false, + }; + assert_eq!( + pool_class_for_image(vk::ImageTiling::LINEAR), + PoolClass::Linear + ); + assert_eq!( + pool_class_for_image(vk::ImageTiling::OPTIMAL), + PoolClass::Optimal + ); + assert_ne!(PoolClass::Linear, PoolClass::Optimal); + assert!(!pool_accepts_member( + &pool, + pool_class_for_image(vk::ImageTiling::OPTIMAL), + vk::MemoryPropertyFlags::DEVICE_LOCAL, + requirements, + )); + } + + #[test] + fn pool_groups_require_a_common_memory_type_for_every_member() { + let requirements = |bits| renderer_resources::memory::MemoryRequirements { + size: 16, + alignment: 8, + memory_type_bits: bits, + requires_dedicated_allocation: false, + prefers_dedicated_allocation: false, + }; + let pool = PlacementPool { + id: 0, + class: PoolClass::Linear, + memory_type_index: 1, + property_flags: vk::MemoryPropertyFlags::DEVICE_LOCAL, + size: 16, + members: vec![(ResourceId::from_u32(0), 0)], + occupied: vec![(ResourceId::from_u32(0), 0)], + }; + assert!(pool_accepts_member( + &pool, + PoolClass::Linear, + vk::MemoryPropertyFlags::DEVICE_LOCAL, + requirements(0b0010), + )); + assert!(!pool_accepts_member( + &pool, + PoolClass::Linear, + vk::MemoryPropertyFlags::DEVICE_LOCAL, + requirements(0b0001), + )); + } + + #[test] + fn first_fit_checks_every_occupied_range_and_finds_earliest_gap() { + let id = ResourceId::from_u32(2); + let occupied = [(ResourceId::from_u32(0), 0), (ResourceId::from_u32(1), 16)]; + let lifetimes = HashMap::from([ + (ResourceId::from_u32(0), placed_lifetime(0, 32, false)), + (ResourceId::from_u32(1), placed_lifetime(16, 32, false)), + (id, placed_lifetime(0, 0, false)), + ]); + let uses = HashMap::from([ + ( + ResourceId::from_u32(0), + vec![ResourceUse { + command: CommandIndex(0), + stage: vk::PipelineStageFlags2::COPY, + access: vk::AccessFlags2::TRANSFER_WRITE, + }], + ), + ( + ResourceId::from_u32(1), + vec![ResourceUse { + command: CommandIndex(1), + stage: vk::PipelineStageFlags2::COPY, + access: vk::AccessFlags2::TRANSFER_WRITE, + }], + ), + ( + id, + vec![ResourceUse { + command: CommandIndex(2), + stage: vk::PipelineStageFlags2::COPY, + access: vk::AccessFlags2::TRANSFER_WRITE, + }], + ), + ]); + let closure = HappensBefore::from_dag( + &[vec![2], vec![2], vec![]], + &[ + vec![CommandIndex(0), CommandIndex(1)], + vec![CommandIndex(2)], + ], + ); + let mut interference = ResourceInterference { + edges: vec![BTreeSet::new(); 3], + }; + for old in [ResourceId::from_u32(0), ResourceId::from_u32(1)] { + interference.edges[id.index()].insert(old); + interference.edges[old.index()].insert(id); + } + + assert_eq!( + first_fit_placement( + id, + 32, + 16, + &occupied, + &lifetimes, + &interference, + &uses, + &closure, + ), + Some(48) + ); + } + + #[test] + fn first_fit_advances_past_all_unsafe_intersecting_ranges() { + let candidate = ResourceId::from_u32(2); + let occupied = [(ResourceId::from_u32(0), 0), (ResourceId::from_u32(1), 16)]; + let lifetimes = HashMap::from([ + (ResourceId::from_u32(0), placed_lifetime(0, 32, false)), + (ResourceId::from_u32(1), placed_lifetime(16, 32, false)), + (candidate, placed_lifetime(0, 0, false)), + ]); + let uses = HashMap::from([ + ( + ResourceId::from_u32(0), + vec![ResourceUse { + command: CommandIndex(0), + stage: vk::PipelineStageFlags2::COPY, + access: vk::AccessFlags2::TRANSFER_WRITE, + }], + ), + ( + ResourceId::from_u32(1), + vec![ResourceUse { + command: CommandIndex(1), + stage: vk::PipelineStageFlags2::COPY, + access: vk::AccessFlags2::TRANSFER_WRITE, + }], + ), + ( + candidate, + vec![ResourceUse { + command: CommandIndex(2), + stage: vk::PipelineStageFlags2::COPY, + access: vk::AccessFlags2::TRANSFER_WRITE, + }], + ), + ]); + let closure = HappensBefore::from_dag( + &[vec![], vec![], vec![]], + &[vec![CommandIndex(0), CommandIndex(1), CommandIndex(2)]], + ); + let mut interference = ResourceInterference { + edges: vec![BTreeSet::new(); 3], + }; + for old in [ResourceId::from_u32(0), ResourceId::from_u32(1)] { + interference.edges[candidate.index()].insert(old); + interference.edges[old.index()].insert(candidate); + } + + assert_eq!( + first_fit_placement( + candidate, + 16, + 16, + &occupied, + &lifetimes, + &interference, + &uses, + &closure, + ), + Some(48) + ); + } + + #[test] + fn dedicated_and_host_visible_resources_are_not_poolable() { + let requirements = renderer_resources::memory::MemoryRequirements { + size: 128, + alignment: 16, + memory_type_bits: 1, + requires_dedicated_allocation: false, + prefers_dedicated_allocation: false, + }; + assert!(!is_poolable( + MemoryLocation::GpuOnly, + AllocationStrategy::Dedicated, + requirements, + )); + assert!(!is_poolable( + MemoryLocation::CpuToGpu, + AllocationStrategy::AllocatorManaged, + requirements, + )); + assert!(!is_poolable( + MemoryLocation::GpuToCpu, + AllocationStrategy::AllocatorManaged, + requirements, + )); + assert!(!is_poolable( + MemoryLocation::GpuOnly, + AllocationStrategy::AllocatorManaged, + renderer_resources::memory::MemoryRequirements { + requires_dedicated_allocation: true, + ..requirements + }, + )); + assert!(is_poolable( + MemoryLocation::GpuOnly, + AllocationStrategy::AllocatorManaged, + requirements, + )); + } + + #[test] + fn allocator_managed_strategy_remains_poolable_until_requirements_select_dedicated() { + let requirements = renderer_resources::memory::MemoryRequirements { + size: 128, + alignment: 16, + memory_type_bits: 1, + requires_dedicated_allocation: false, + prefers_dedicated_allocation: false, + }; + assert_eq!( + allocation_strategy_for(requirements), + AllocationStrategy::AllocatorManaged + ); + assert_eq!( + allocation_strategy_for(renderer_resources::memory::MemoryRequirements { + requires_dedicated_allocation: true, + ..requirements + }), + AllocationStrategy::Dedicated + ); + assert_eq!( + allocation_strategy_for(renderer_resources::memory::MemoryRequirements { + size: DEDICATED_ALLOCATION_THRESHOLD, + ..requirements + }), + AllocationStrategy::Dedicated + ); + } + + #[test] + fn unused_transients_have_no_lifetime_and_are_not_candidates() { + let unused = ResourceId::from_u32(0); + let used = ResourceId::from_u32(1); + let resources = [ + ResourceDesc::CreatedBuffer(GraphBufferDesc { + size: 1024, + ..Default::default() + }), + ResourceDesc::CreatedBuffer(GraphBufferDesc::default()), + ]; + let accesses = vec![vec![resource_access_for(used)]]; + let compiler = GraphCompiler::new(&resources, &accesses, 1); + let batches = compiler.topo_batches().unwrap(); + let closure = HappensBefore::from_dag(&[vec![]], &batches); + let uses = compiler.resource_uses(); + let (lifetimes, _) = compiler.bind_lifetimes(&batches, &closure, &uses); + assert!(!lifetimes.contains_key(&unused)); + assert!(lifetimes.contains_key(&used)); + assert!(!should_plan_created_resource( + unused, + &lifetimes, + &HashMap::new(), + )); + assert!(should_plan_created_resource( + used, + &lifetimes, + &HashMap::new(), + )); + } + + #[test] + fn pooled_image_layout_validation_rejects_non_undefined_but_allows_dedicated() { + let id = ResourceId::from_u32(7); + let desc = GraphTextureDesc { + layout: vk::ImageLayout::GENERAL, + ..Default::default() + }; + let requirements = renderer_resources::memory::MemoryRequirements { + size: 128, + alignment: 16, + memory_type_bits: 1, + requires_dedicated_allocation: false, + prefers_dedicated_allocation: false, + }; + assert_eq!( + validate_pooled_image_layout(id, &desc, requirements), + Err(GraphError::InvalidPooledImageLayout(id)) + ); + validate_pooled_image_layout( + id, + &desc, + renderer_resources::memory::MemoryRequirements { + size: DEDICATED_ALLOCATION_THRESHOLD, + ..requirements + }, + ) + .expect("non-pooled images retain the existing initial-layout behavior"); + } + fn buffer_entry( binding: u32, kind: vk::DescriptorType, @@ -2031,6 +3578,7 @@ mod tests { }); let mut compiled = graph.compile().unwrap(); + assert!(compiled.resource_lifetimes()[&imported.id()].imported); assert!(compiled.barriers().iter().any(|barrier| { barrier.before_command == CommandIndex(0) && !barrier.resources.is_empty() })); @@ -2491,27 +4039,48 @@ mod tests { fn interference_tracks_overlapping_inclusive_lifetimes() { let lifetimes = HashMap::from([ ( - ResourceId(0), + ResourceId::from_u32(0), ResourceLifetime { - first_command: CommandIndex(0), - last_command: CommandIndex(2), - memory_offset: 0, + first_batch: 0, + last_batch: 2, + placement: Placement { + offset: 0, + size: 1, + alignment: 1, + pool: None, + aliased: false, + }, + imported: false, }, ), ( - ResourceId(1), + ResourceId::from_u32(1), ResourceLifetime { - first_command: CommandIndex(2), - last_command: CommandIndex(3), - memory_offset: 0, + first_batch: 2, + last_batch: 3, + placement: Placement { + offset: 0, + size: 1, + alignment: 1, + pool: None, + aliased: false, + }, + imported: false, }, ), ( - ResourceId(2), + ResourceId::from_u32(2), ResourceLifetime { - first_command: CommandIndex(4), - last_command: CommandIndex(5), - memory_offset: 0, + first_batch: 4, + last_batch: 5, + placement: Placement { + offset: 0, + size: 1, + alignment: 1, + pool: None, + aliased: false, + }, + imported: false, }, ), ]); @@ -2519,14 +4088,14 @@ mod tests { lifetimes.iter().map(|(&id, &lifetime)| (id, lifetime)), ); - assert!(interference.interferes(ResourceId(0), ResourceId(1))); - assert!(interference.interferes(ResourceId(1), ResourceId(0))); - assert!(!interference.interferes(ResourceId(0), ResourceId(2))); - assert!(!interference.interferes(ResourceId(1), ResourceId(2))); + assert!(interference.interferes(ResourceId::from_u32(0), ResourceId::from_u32(1))); + assert!(interference.interferes(ResourceId::from_u32(1), ResourceId::from_u32(0))); + assert!(!interference.interferes(ResourceId::from_u32(0), ResourceId::from_u32(2))); + assert!(!interference.interferes(ResourceId::from_u32(1), ResourceId::from_u32(2))); } #[test] - fn aliasing_reuses_non_interfering_slot_and_separates_interfering_resources() { + fn lifetime_analysis_tracks_batches_and_command_level_interference() { let resources = vec![ ResourceDesc::CreatedBuffer(GraphBufferDesc { size: 8, @@ -2542,25 +4111,145 @@ mod tests { }), ]; let accesses = vec![ - vec![resource_access_for(ResourceId(0))], + vec![resource_access_for(ResourceId::from_u32(0))], vec![ - resource_access_for(ResourceId(0)), - resource_access_for(ResourceId(1)), + resource_access_for(ResourceId::from_u32(0)), + resource_access_for(ResourceId::from_u32(1)), ], - vec![resource_access_for(ResourceId(0))], - vec![resource_access_for(ResourceId(2))], + vec![resource_access_for(ResourceId::from_u32(0))], + vec![resource_access_for(ResourceId::from_u32(2))], ]; let compiler = GraphCompiler::new(&resources, &accesses, accesses.len()); - let (lifetimes, interference) = compiler.bind_lifetimes(); + let (batches, edges) = compiler.topo_batches_and_dependencies().unwrap(); + let happens_before = HappensBefore::from_dag(&edges, &batches); + let uses_by_resource = compiler.resource_uses(); + let (lifetimes, interference) = + compiler.bind_lifetimes(&batches, &happens_before, &uses_by_resource); - let long_lived_offset = lifetimes[&ResourceId(0)].memory_offset; - let overlapping_offset = lifetimes[&ResourceId(1)].memory_offset; - let later_offset = lifetimes[&ResourceId(2)].memory_offset; - assert!(interference.interferes(ResourceId(0), ResourceId(1))); - assert_ne!(long_lived_offset, overlapping_offset); - assert!(!interference.interferes(ResourceId(0), ResourceId(2))); - assert!(!interference.interferes(ResourceId(1), ResourceId(2))); - assert!(later_offset == long_lived_offset || later_offset == overlapping_offset); + assert_eq!(lifetimes[&ResourceId::from_u32(0)].first_batch, 0); + assert_eq!(lifetimes[&ResourceId::from_u32(0)].last_batch, 2); + assert!(interference.interferes(ResourceId::from_u32(0), ResourceId::from_u32(1))); + assert!(interference.interferes(ResourceId::from_u32(0), ResourceId::from_u32(2))); + assert!(interference.interferes(ResourceId::from_u32(1), ResourceId::from_u32(2))); + } + + #[test] + fn happens_before_closure_covers_direct_and_transitive_edges() { + let edges = vec![vec![1], vec![2], vec![], vec![]]; + let batches = vec![ + vec![CommandIndex(0), CommandIndex(3)], + vec![CommandIndex(1)], + vec![CommandIndex(2)], + ]; + let happens_before = HappensBefore::from_dag(&edges, &batches); + + assert!(happens_before.happens_before(CommandIndex(0), CommandIndex(1))); + assert!(happens_before.happens_before(CommandIndex(0), CommandIndex(2))); + assert!(happens_before.happens_before(CommandIndex(1), CommandIndex(2))); + assert!(!happens_before.happens_before(CommandIndex(2), CommandIndex(0))); + assert!(!happens_before.happens_before(CommandIndex(0), CommandIndex(3))); + assert!(!happens_before.happens_before(CommandIndex(0), CommandIndex(0))); + } + + #[test] + fn same_batch_resource_uses_interfere() { + let edges = vec![vec![], vec![]]; + let batches = vec![vec![CommandIndex(0), CommandIndex(1)]]; + let happens_before = HappensBefore::from_dag(&edges, &batches); + let a = [ResourceUse { + command: CommandIndex(0), + stage: vk::PipelineStageFlags2::COPY, + access: vk::AccessFlags2::TRANSFER_WRITE, + }]; + let b = [ResourceUse { + command: CommandIndex(1), + stage: vk::PipelineStageFlags2::COPY, + access: vk::AccessFlags2::TRANSFER_WRITE, + }]; + + let use_sets = HashMap::from([ + (ResourceId::from_u32(0), a.to_vec()), + (ResourceId::from_u32(1), b.to_vec()), + ]); + let interference = ResourceInterference::from_resource_uses(&use_sets, &happens_before); + assert!(interference.interferes(ResourceId::from_u32(0), ResourceId::from_u32(1))); + assert!(!resources_alias_in_order(&a, &b, &happens_before)); + assert!(!resources_alias_in_order(&b, &a, &happens_before)); + } + + #[test] + fn unordered_distinct_batches_interfere_even_without_interval_overlap() { + // This reproduces the plan's ready-set shape: 0 and 2 are batch-ready; + // 1 depends on 0, but 2 has no edge to 1. + let edges = vec![vec![1], vec![], vec![]]; + let batches = vec![ + vec![CommandIndex(0), CommandIndex(2)], + vec![CommandIndex(1)], + ]; + let happens_before = HappensBefore::from_dag(&edges, &batches); + let resource_a = [ResourceUse { + command: CommandIndex(2), + stage: vk::PipelineStageFlags2::COPY, + access: vk::AccessFlags2::TRANSFER_WRITE, + }]; + let resource_b = [ResourceUse { + command: CommandIndex(1), + stage: vk::PipelineStageFlags2::COPY, + access: vk::AccessFlags2::TRANSFER_WRITE, + }]; + + assert!(!happens_before.happens_before(CommandIndex(2), CommandIndex(1))); + assert!(!resources_alias_in_order( + &resource_a, + &resource_b, + &happens_before, + )); + let use_sets = HashMap::from([ + (ResourceId::from_u32(0), resource_a.to_vec()), + (ResourceId::from_u32(1), resource_b.to_vec()), + ]); + let interference = ResourceInterference::from_resource_uses(&use_sets, &happens_before); + assert!(interference.interferes(ResourceId::from_u32(0), ResourceId::from_u32(1))); + } + + #[test] + fn dependency_chain_proves_alias_reuse_safe() { + let resources = (0..5) + .map(|_| { + ResourceDesc::CreatedBuffer(GraphBufferDesc { + size: 8, + ..Default::default() + }) + }) + .collect::>(); + let accesses = vec![ + vec![ + resource_access_for(ResourceId::from_u32(0)), + resource_access_for(ResourceId::from_u32(2)), + ], + vec![ + resource_access_for(ResourceId::from_u32(2)), + resource_access_for(ResourceId::from_u32(3)), + ], + vec![ + resource_access_for(ResourceId::from_u32(3)), + resource_access_for(ResourceId::from_u32(1)), + ], + ]; + let compiler = GraphCompiler::new(&resources, &accesses, accesses.len()); + let (batches, edges) = compiler.topo_batches_and_dependencies().unwrap(); + let happens_before = HappensBefore::from_dag(&edges, &batches); + let uses = compiler.resource_uses(); + + assert!(happens_before.happens_before(CommandIndex(0), CommandIndex(2))); + assert!(resources_alias_in_order( + &uses[&ResourceId::from_u32(0)], + &uses[&ResourceId::from_u32(1)], + &happens_before, + )); + let uses_a = &uses[&ResourceId::from_u32(0)]; + let uses_b = &uses[&ResourceId::from_u32(1)]; + assert!(aliasing_permitted(uses_a, uses_b, &happens_before)); } #[test] @@ -2600,6 +4289,7 @@ mod tests { assert!(compiled.barriers().is_empty()); assert_eq!(compiled.resource_lifetimes().len(), 2); assert!(compiled.interference().interferes(src.id(), dst.id())); + assert!(!compiled.resource_lifetimes()[&src.id()].imported); } #[test] @@ -2826,6 +4516,8 @@ mod tests { ); assert!(compiled.resource_lifetimes().contains_key(&uniform.id())); assert!(compiled.resource_lifetimes().contains_key(&texture.id())); + assert!(!compiled.resource_lifetimes()[&uniform.id()].imported); + assert!(!compiled.resource_lifetimes()[&texture.id()].imported); assert!( compiled .interference() @@ -2990,7 +4682,7 @@ mod tests { #[test] fn write_read_write_emits_barriers_from_each_most_recent_conflict() { - let resource = ResourceId(0); + let resource = ResourceId::from_u32(0); let resources = [ResourceDesc::CreatedBuffer(GraphBufferDesc { size: 64, ..Default::default() @@ -3055,9 +4747,275 @@ mod tests { ); } + #[test] + fn alias_buffer_barrier_aggregates_terminal_write_and_first_use_scopes() { + let old = ResourceId::from_u32(0); + let new = ResourceId::from_u32(1); + let resources = [ + ResourceDesc::CreatedBuffer(GraphBufferDesc { + size: 64, + ..Default::default() + }), + ResourceDesc::CreatedBuffer(GraphBufferDesc { + size: 64, + ..Default::default() + }), + ]; + let accesses = vec![ + vec![ + scoped_access( + old, + 0, + 64, + vk::PipelineStageFlags2::COPY, + vk::AccessFlags2::TRANSFER_WRITE, + ), + scoped_access( + old, + 0, + 64, + vk::PipelineStageFlags2::COMPUTE_SHADER, + vk::AccessFlags2::SHADER_STORAGE_WRITE, + ), + ], + vec![scoped_access( + new, + 0, + 64, + vk::PipelineStageFlags2::FRAGMENT_SHADER, + vk::AccessFlags2::UNIFORM_READ, + )], + ]; + let compiler = GraphCompiler::new(&resources, &accesses, accesses.len()); + let batches = vec![vec![CommandIndex(0)], vec![CommandIndex(1)]]; + let happens_before = HappensBefore::from_dag(&[vec![1], vec![]], &batches); + let uses = compiler.resource_uses(); + let lifetimes = HashMap::from([ + (old, placed_lifetime(0, 64, false)), + (new, placed_lifetime(0, 64, true)), + ]); + let mut barriers = compiler.build_barriers(&batches); + + compiler.build_alias_barriers(&mut barriers, &lifetimes, &uses, &happens_before); + + let barrier = barriers + .iter() + .find(|barrier| barrier.before_command == CommandIndex(1)) + .unwrap(); + assert_eq!(barrier.memory_barriers.len(), 1); + let memory = barrier.memory_barriers[0]; + assert_eq!( + memory.src_stage, + vk::PipelineStageFlags2::COPY | vk::PipelineStageFlags2::COMPUTE_SHADER + ); + assert_eq!( + memory.src_access, + vk::AccessFlags2::TRANSFER_WRITE | vk::AccessFlags2::SHADER_STORAGE_WRITE + ); + assert_eq!(memory.dst_stage, vk::PipelineStageFlags2::FRAGMENT_SHADER); + assert_eq!(memory.dst_access, vk::AccessFlags2::UNIFORM_READ); + } + + #[test] + fn alias_buffer_read_only_source_uses_execution_only_memory_barrier() { + let old = ResourceId::from_u32(0); + let new = ResourceId::from_u32(1); + let resources = [ + ResourceDesc::CreatedBuffer(GraphBufferDesc::default()), + ResourceDesc::CreatedBuffer(GraphBufferDesc::default()), + ]; + let accesses = vec![ + vec![scoped_access( + old, + 0, + 8, + vk::PipelineStageFlags2::VERTEX_SHADER, + vk::AccessFlags2::SHADER_READ, + )], + vec![scoped_access( + new, + 0, + 8, + vk::PipelineStageFlags2::COPY, + vk::AccessFlags2::TRANSFER_WRITE, + )], + ]; + let compiler = GraphCompiler::new(&resources, &accesses, accesses.len()); + let batches = vec![vec![CommandIndex(0)], vec![CommandIndex(1)]]; + let happens_before = HappensBefore::from_dag(&[vec![1], vec![]], &batches); + let uses = compiler.resource_uses(); + let lifetimes = HashMap::from([ + (old, placed_lifetime(0, 8, false)), + (new, placed_lifetime(0, 8, true)), + ]); + let mut barriers = compiler.build_barriers(&batches); + + compiler.build_alias_barriers(&mut barriers, &lifetimes, &uses, &happens_before); + + let memory = barriers + .iter() + .find(|barrier| barrier.before_command == CommandIndex(1)) + .unwrap() + .memory_barriers[0]; + assert_eq!(memory.src_stage, vk::PipelineStageFlags2::VERTEX_SHADER); + assert_eq!(memory.src_access, vk::AccessFlags2::NONE); + assert_eq!(memory.dst_access, vk::AccessFlags2::TRANSFER_WRITE); + } + + #[test] + fn alias_image_scope_is_folded_into_undefined_layout_transition() { + let old = ResourceId::from_u32(0); + let new = ResourceId::from_u32(1); + let resources = [ + ResourceDesc::CreatedTexture(GraphTextureDesc::default()), + ResourceDesc::CreatedTexture(GraphTextureDesc::default()), + ]; + let range = texture_range(vk::ImageAspectFlags::COLOR, 0, 1, 0, 1); + let accesses = vec![ + vec![texture_scoped_access( + old, + range, + vk::ImageLayout::GENERAL, + vk::PipelineStageFlags2::COLOR_ATTACHMENT_OUTPUT, + vk::AccessFlags2::COLOR_ATTACHMENT_WRITE, + )], + vec![texture_scoped_access( + new, + range, + vk::ImageLayout::SHADER_READ_ONLY_OPTIMAL, + vk::PipelineStageFlags2::FRAGMENT_SHADER, + vk::AccessFlags2::SHADER_SAMPLED_READ, + )], + ]; + let compiler = GraphCompiler::new(&resources, &accesses, accesses.len()); + let batches = vec![vec![CommandIndex(0)], vec![CommandIndex(1)]]; + let happens_before = HappensBefore::from_dag(&[vec![1], vec![]], &batches); + let uses = compiler.resource_uses(); + let lifetimes = HashMap::from([ + (old, placed_lifetime(0, 128, false)), + (new, placed_lifetime(0, 128, true)), + ]); + let mut barriers = compiler.build_barriers(&batches); + + compiler.build_alias_barriers(&mut barriers, &lifetimes, &uses, &happens_before); + + let barrier = barriers + .iter() + .find(|barrier| barrier.before_command == CommandIndex(1)) + .unwrap(); + assert!(barrier.memory_barriers.is_empty()); + let image = barrier + .resources + .iter() + .find(|resource| resource.id == new) + .unwrap(); + assert_eq!(image.old_layout, Some(vk::ImageLayout::UNDEFINED)); + assert_eq!( + image.src_stage, + vk::PipelineStageFlags2::COLOR_ATTACHMENT_OUTPUT + ); + assert_eq!(image.src_access, vk::AccessFlags2::COLOR_ATTACHMENT_WRITE); + assert_eq!(image.dst_stage, vk::PipelineStageFlags2::FRAGMENT_SHADER); + } + + #[test] + fn alias_range_barriers_include_every_intersecting_occupant() { + let old_ids = [ + ResourceId::from_u32(0), + ResourceId::from_u32(1), + ResourceId::from_u32(2), + ResourceId::from_u32(3), + ]; + let new = ResourceId::from_u32(4); + let resources = (0..5) + .map(|_| ResourceDesc::CreatedBuffer(GraphBufferDesc::default())) + .collect::>(); + let old_stages = [ + vk::PipelineStageFlags2::VERTEX_SHADER, + vk::PipelineStageFlags2::FRAGMENT_SHADER, + vk::PipelineStageFlags2::COMPUTE_SHADER, + vk::PipelineStageFlags2::COPY, + ]; + let mut accesses = old_ids + .iter() + .zip(old_stages) + .map(|(&id, stage)| { + vec![scoped_access( + id, + 0, + 8, + stage, + vk::AccessFlags2::TRANSFER_WRITE, + )] + }) + .collect::>(); + accesses.push(vec![scoped_access( + new, + 0, + 8, + vk::PipelineStageFlags2::COMPUTE_SHADER, + vk::AccessFlags2::SHADER_WRITE, + )]); + let compiler = GraphCompiler::new(&resources, &accesses, accesses.len()); + let batches = (0..5) + .map(|index| vec![CommandIndex(index)]) + .collect::>(); + let edges = vec![vec![4], vec![4], vec![4], vec![4], vec![]]; + let happens_before = HappensBefore::from_dag(&edges, &batches); + let uses = compiler.resource_uses(); + let lifetimes = HashMap::from([ + (old_ids[0], placed_lifetime(0, 8, false)), + (old_ids[1], placed_lifetime(4, 8, false)), + (old_ids[2], placed_lifetime(8, 8, false)), + (old_ids[3], placed_lifetime(32, 8, false)), + (new, placed_lifetime(8, 16, true)), + ]); + let mut barriers = compiler.build_barriers(&batches); + + compiler.build_alias_barriers(&mut barriers, &lifetimes, &uses, &happens_before); + + let barrier = barriers + .iter() + .find(|barrier| barrier.before_command == CommandIndex(4)) + .unwrap(); + // Source masks are unioned across all intersecting occupants into one barrier. + assert_eq!(barrier.memory_barriers.len(), 1); + assert_eq!( + barrier.memory_barriers[0].src_stage, + vk::PipelineStageFlags2::FRAGMENT_SHADER | vk::PipelineStageFlags2::COMPUTE_SHADER + ); + assert_eq!( + barrier.memory_barriers[0].src_access, + vk::AccessFlags2::TRANSFER_WRITE + ); + assert_eq!( + barrier.memory_barriers[0].dst_stage, + vk::PipelineStageFlags2::COMPUTE_SHADER + ); + assert_eq!( + barrier.memory_barriers[0].dst_access, + vk::AccessFlags2::SHADER_WRITE + ); + } + + fn placed_lifetime(offset: u64, size: u64, aliased: bool) -> ResourceLifetime { + ResourceLifetime { + first_batch: 0, + last_batch: 0, + placement: Placement { + offset, + size, + alignment: 1, + pool: Some(0), + aliased, + }, + imported: false, + } + } + #[test] fn imported_texture_read_with_different_layout_gets_initial_transition() { - let resource = ResourceId(0); + let resource = ResourceId::from_u32(0); let resources = [ResourceDesc::CreatedTexture(GraphTextureDesc { format: vk::Format::R8G8B8A8_UNORM, mip_levels: 2, @@ -3109,7 +5067,7 @@ mod tests { #[test] fn overlapping_texture_reads_with_different_layouts_get_layout_transition() { - let resource = ResourceId(0); + let resource = ResourceId::from_u32(0); let resources = [ResourceDesc::CreatedTexture(GraphTextureDesc { format: vk::Format::R8G8B8A8_UNORM, ..Default::default() @@ -3157,7 +5115,7 @@ mod tests { #[test] fn overlapping_pure_reads_without_layout_change_need_no_barrier() { - let resource = ResourceId(0); + let resource = ResourceId::from_u32(0); let resources = [ResourceDesc::CreatedBuffer(GraphBufferDesc { size: 64, ..Default::default() @@ -3186,7 +5144,7 @@ mod tests { #[test] fn write_before_contained_read_emits_barrier_before_read() { - let resource = ResourceId(0); + let resource = ResourceId::from_u32(0); let resources = [ResourceDesc::CreatedBuffer(GraphBufferDesc { size: 64, ..Default::default() @@ -3310,10 +5268,12 @@ mod tests { fn recording_steps_place_barriers_immediately_before_their_target_command() { let first_barrier = PipelineBarrier { before_command: CommandIndex(1), + memory_barriers: Vec::new(), resources: Vec::new(), }; let second_barrier = PipelineBarrier { before_command: CommandIndex(1), + memory_barriers: Vec::new(), resources: Vec::new(), }; let barriers = [first_barrier, second_barrier]; @@ -3438,6 +5398,8 @@ mod tests { vec![CommandIndex(2)] ] ); + assert!(compiled.happens_before(CommandIndex(0), CommandIndex(2))); + assert!(compiled.resources_can_alias(a.id(), d.id())); for barrier in compiled.barriers() { let source_command = CommandIndex(barrier.before_command.index() - 1); let source_batch = compiled @@ -3455,7 +5417,7 @@ mod tests { } #[test] - fn texture_aliasing_size_uses_format_block_size() { + fn texture_lifetime_size_waits_for_device_requirements() { let mut graph = RenderGraph::new(); let texture = graph.create_texture(GraphTextureDesc { format: vk::Format::R8G8B8A8_UNORM, @@ -3494,13 +5456,12 @@ mod tests { ); let compiled = graph.compile().expect("graph compiles"); - let texture_offset = compiled.resource_lifetimes()[&texture.id()].memory_offset; - let buffer_offset = compiled.resource_lifetimes()[&buffer.id()].memory_offset; - // Equal first-use indices may be assigned in either order by the - // lifetime map, but the image's estimated 64-byte size is respected. - assert!( - texture_offset == 0 && buffer_offset == 64 || buffer_offset == 0 && texture_offset == 8 - ); + let texture_placement = compiled.resource_lifetimes()[&texture.id()].placement; + let buffer_placement = compiled.resource_lifetimes()[&buffer.id()].placement; + assert_eq!(texture_placement.size, 0); + assert_eq!(buffer_placement.size, 0); + assert_eq!(texture_placement.offset, 0); + assert_eq!(buffer_placement.offset, 0); } #[test] @@ -3639,7 +5600,7 @@ mod tests { access: vk::AccessFlags2, ) -> crate::recorder::ResourceAccess { crate::recorder::ResourceAccess { - resource: ResourceId(0), + resource: ResourceId::from_u32(0), scope: SubresourceScope { stage, access, diff --git a/crates/renderer_rendergraph/src/lib.rs b/crates/renderer_rendergraph/src/lib.rs index 2bb4501..a84f97b 100644 --- a/crates/renderer_rendergraph/src/lib.rs +++ b/crates/renderer_rendergraph/src/lib.rs @@ -6,7 +6,10 @@ pub mod graph; pub mod recorder; pub mod resources; -pub use graph::{CompiledGraph, GraphError, RenderGraph, ResourceInterference}; +pub use graph::{ + CompiledGraph, GraphError, HappensBefore, MemoryBarrier, Placement, RenderGraph, + ResourceInterference, ResourceLifetime, ResourceUse, aliasing_permitted, +}; pub use recorder::{CommandIndex, CommandList, CommandRecorder, SideEffectMap}; pub use resources::{ Buffer, BufferRange, BufferSlice, DescriptorResourceAccess, DescriptorResourceIdentity, diff --git a/crates/renderer_rendergraph/src/recorder.rs b/crates/renderer_rendergraph/src/recorder.rs index 6235940..c06f2e7 100644 --- a/crates/renderer_rendergraph/src/recorder.rs +++ b/crates/renderer_rendergraph/src/recorder.rs @@ -1,4 +1,4 @@ -use std::collections::BTreeMap; +use std::{cell::RefCell, collections::BTreeMap}; use ash::vk; @@ -163,6 +163,17 @@ pub struct CommandRecorder<'a> { pub images: BTreeMap, } +#[derive(Default)] +struct BarrierScratch { + memory: Vec>, + buffers: Vec>, + images: Vec>, +} + +thread_local! { + static BARRIER_SCRATCH: RefCell = RefCell::new(BarrierScratch::default()); +} + impl<'a> CommandRecorder<'a> { pub fn new(device: &'a ash::Device, command_buffer: vk::CommandBuffer) -> Self { Self { @@ -188,60 +199,92 @@ impl<'a> CommandRecorder<'a> { } pub fn pipeline_barrier(&self, barrier: &super::graph::PipelineBarrier) { - let buffer_barriers: Vec<_> = barrier - .resources - .iter() - .filter_map(|resource| match resource.range { - Subresource::Buffer(BufferRange { offset, size }) => { - self.buffers.get(&resource.id).map(|&buffer| { - let (src, dst) = if let Some(queue_transfer) = resource.queue_transfer { - (queue_transfer.src.index(), queue_transfer.dst.index()) - } else { - (vk::QUEUE_FAMILY_IGNORED, vk::QUEUE_FAMILY_IGNORED) - }; - - vk::BufferMemoryBarrier2::default() - .src_stage_mask(resource.src_stage) - .src_access_mask(resource.src_access) - .dst_stage_mask(resource.dst_stage) - .dst_access_mask(resource.dst_access) - .src_queue_family_index(src) - .dst_queue_family_index(dst) - .buffer(buffer) - .offset(offset) - .size(size) - }) - } - Subresource::Texture(_) => None, - }) - .collect(); - - let image_barriers: Vec<_> = barrier - .resources - .iter() - .filter_map(|resource| match resource.range { - Subresource::Texture(range) => self.images.get(&resource.id).map(|&image| { - vk::ImageMemoryBarrier2::default() + BARRIER_SCRATCH.with(|scratch| { + let mut scratch = scratch.borrow_mut(); + scratch.memory.clear(); + scratch + .memory + .extend(barrier.memory_barriers.iter().map(|resource| { + vk::MemoryBarrier2::default() .src_stage_mask(resource.src_stage) .src_access_mask(resource.src_access) .dst_stage_mask(resource.dst_stage) .dst_access_mask(resource.dst_access) - .old_layout(resource.old_layout.unwrap_or(vk::ImageLayout::UNDEFINED)) - .new_layout(resource.new_layout.unwrap_or(vk::ImageLayout::GENERAL)) - .image(image) - .subresource_range(range) - }), - Subresource::Buffer(_) => None, - }) - .collect(); + })); - let dependency = vk::DependencyInfo::default() - .buffer_memory_barriers(&buffer_barriers) - .image_memory_barriers(&image_barriers); - unsafe { - self.device - .cmd_pipeline_barrier2(self.command_buffer, &dependency); - } + scratch.buffers.clear(); + scratch + .buffers + .extend( + barrier + .resources + .iter() + .filter_map(|resource| match resource.range { + Subresource::Buffer(BufferRange { offset, size }) => { + self.buffers.get(&resource.id).map(|&buffer| { + let (src, dst) = + if let Some(queue_transfer) = resource.queue_transfer { + (queue_transfer.src.index(), queue_transfer.dst.index()) + } else { + (vk::QUEUE_FAMILY_IGNORED, vk::QUEUE_FAMILY_IGNORED) + }; + + vk::BufferMemoryBarrier2::default() + .src_stage_mask(resource.src_stage) + .src_access_mask(resource.src_access) + .dst_stage_mask(resource.dst_stage) + .dst_access_mask(resource.dst_access) + .src_queue_family_index(src) + .dst_queue_family_index(dst) + .buffer(buffer) + .offset(offset) + .size(size) + }) + } + Subresource::Texture(_) => None, + }), + ); + + scratch.images.clear(); + scratch + .images + .extend( + barrier + .resources + .iter() + .filter_map(|resource| match resource.range { + Subresource::Texture(range) => { + self.images.get(&resource.id).map(|&image| { + vk::ImageMemoryBarrier2::default() + .src_stage_mask(resource.src_stage) + .src_access_mask(resource.src_access) + .dst_stage_mask(resource.dst_stage) + .dst_access_mask(resource.dst_access) + .old_layout( + resource + .old_layout + .unwrap_or(vk::ImageLayout::UNDEFINED), + ) + .new_layout( + resource.new_layout.unwrap_or(vk::ImageLayout::GENERAL), + ) + .image(image) + .subresource_range(range) + }) + } + Subresource::Buffer(_) => None, + }), + ); + + let dependency = vk::DependencyInfo::default() + .memory_barriers(&scratch.memory) + .buffer_memory_barriers(&scratch.buffers) + .image_memory_barriers(&scratch.images); + unsafe { + self.device + .cmd_pipeline_barrier2(self.command_buffer, &dependency); + } + }); } } @@ -255,6 +298,58 @@ pub(crate) fn scope_range(scope: SubresourceScope) -> Subresource { #[cfg(test)] mod tests { use super::*; + use crate::graph::{MemoryBarrier, PipelineBarrier}; + use std::sync::atomic::{AtomicU32, AtomicU64, Ordering}; + + static MEMORY_BARRIER_COUNT: AtomicU32 = AtomicU32::new(0); + static MEMORY_BARRIER_SRC_STAGE: AtomicU64 = AtomicU64::new(0); + + unsafe extern "system" fn capture_pipeline_barrier( + _device: vk::Device, + _command_buffer: vk::CommandBuffer, + dependency: *const vk::DependencyInfo<'_>, + ) { + let dependency = unsafe { &*dependency }; + MEMORY_BARRIER_COUNT.store(dependency.memory_barrier_count, Ordering::SeqCst); + let memory = unsafe { &*dependency.p_memory_barriers }; + MEMORY_BARRIER_SRC_STAGE.store(memory.src_stage_mask.as_raw(), Ordering::SeqCst); + } + + #[test] + fn pipeline_barrier_emits_memory_barriers_in_dependency_info() { + let device = unsafe { + ash::Device::load_with( + |name| { + if name.to_bytes() == b"vkCmdPipelineBarrier2" { + capture_pipeline_barrier as *const () as *const std::ffi::c_void + } else { + std::ptr::null() + } + }, + vk::Device::null(), + ) + }; + let barrier = PipelineBarrier { + before_command: CommandIndex(0), + memory_barriers: vec![MemoryBarrier { + src_stage: vk::PipelineStageFlags2::COMPUTE_SHADER, + src_access: vk::AccessFlags2::SHADER_STORAGE_WRITE, + dst_stage: vk::PipelineStageFlags2::FRAGMENT_SHADER, + dst_access: vk::AccessFlags2::SHADER_READ, + }], + resources: Vec::new(), + }; + MEMORY_BARRIER_COUNT.store(0, Ordering::SeqCst); + MEMORY_BARRIER_SRC_STAGE.store(0, Ordering::SeqCst); + + CommandRecorder::new(&device, vk::CommandBuffer::null()).pipeline_barrier(&barrier); + + assert_eq!(MEMORY_BARRIER_COUNT.load(Ordering::SeqCst), 1); + assert_eq!( + MEMORY_BARRIER_SRC_STAGE.load(Ordering::SeqCst), + vk::PipelineStageFlags2::COMPUTE_SHADER.as_raw() + ); + } fn read_scope( offset: u64, @@ -275,7 +370,7 @@ mod tests { let command = map.begin_command(); map.insert( command, - ResourceId(0), + ResourceId::from_u32(0), read_scope( 0, 64, @@ -285,7 +380,7 @@ mod tests { ); map.insert( command, - ResourceId(0), + ResourceId::from_u32(0), read_scope( 16, 16, @@ -316,7 +411,7 @@ mod tests { let command = reverse.begin_command(); reverse.insert( command, - ResourceId(1), + ResourceId::from_u32(1), read_scope( 16, 16, @@ -326,7 +421,7 @@ mod tests { ); reverse.insert( command, - ResourceId(1), + ResourceId::from_u32(1), read_scope( 0, 64, @@ -360,7 +455,7 @@ mod tests { let command = map.begin_command(); map.insert( command, - ResourceId(0), + ResourceId::from_u32(0), read_scope( 0, 16, @@ -370,7 +465,7 @@ mod tests { ); map.insert( command, - ResourceId(0), + ResourceId::from_u32(0), read_scope( 8, 16, @@ -388,7 +483,7 @@ mod tests { let command = map.begin_command(); map.insert( command, - ResourceId(0), + ResourceId::from_u32(0), read_scope( 0, 64, @@ -398,7 +493,7 @@ mod tests { ); map.insert( command, - ResourceId(0), + ResourceId::from_u32(0), read_scope( 16, 16, diff --git a/crates/renderer_rendergraph/src/resources.rs b/crates/renderer_rendergraph/src/resources.rs index 677f56e..bab85f0 100644 --- a/crates/renderer_rendergraph/src/resources.rs +++ b/crates/renderer_rendergraph/src/resources.rs @@ -75,10 +75,14 @@ impl Default for GraphTextureDesc { #[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)] #[repr(transparent)] -pub struct ResourceId(pub u64); +pub struct ResourceId(u32); impl ResourceId { - pub(crate) fn index(self) -> usize { + pub fn from_u32(index: u32) -> Self { + Self(index) + } + + pub fn index(self) -> usize { self.0 as usize } } diff --git a/crates/renderer_rendergraph/tests/graph_validation.rs b/crates/renderer_rendergraph/tests/graph_validation.rs index 7835451..806f68f 100644 --- a/crates/renderer_rendergraph/tests/graph_validation.rs +++ b/crates/renderer_rendergraph/tests/graph_validation.rs @@ -7,12 +7,98 @@ use renderer_device::{ }; use renderer_rendergraph::{ RenderGraph, + commands::Command, commands::CopyBuffer, - recorder::CommandRecorder, - resources::{BufferRange, ResourceState}, + recorder::{CommandIndex, CommandRecorder, RecordError, RecordedCommand, SideEffectMap}, + resources::{ + BufferRange, GraphBufferDesc, GraphTextureDesc, ResourceId, ResourceState, + SubresourceScope, SubresourceScopeInner, + }, }; use renderer_resources::buffers::{Buffer, BufferDesc}; +struct ScopedNoop(Vec<(ResourceId, SubresourceScope)>); + +impl Command for ScopedNoop { + fn side_effects(&self, command_index: CommandIndex, map: &mut SideEffectMap) { + for &(resource, scope) in &self.0 { + map.insert(command_index, resource, scope); + } + } +} + +impl RecordedCommand for ScopedNoop { + fn apply(&mut self, _recorder: &mut CommandRecorder<'_>) -> Result<(), RecordError> { + Ok(()) + } +} + +fn buffer_scope() -> SubresourceScope { + SubresourceScope { + stage: vk::PipelineStageFlags2::COPY, + access: vk::AccessFlags2::TRANSFER_WRITE, + subresource: SubresourceScopeInner::Buffer(BufferRange { + offset: 0, + size: 256, + }), + } +} + +fn image_scope() -> SubresourceScope { + SubresourceScope { + stage: vk::PipelineStageFlags2::COMPUTE_SHADER, + access: vk::AccessFlags2::SHADER_STORAGE_WRITE, + subresource: SubresourceScopeInner::Texture { + range: vk::ImageSubresourceRange { + aspect_mask: vk::ImageAspectFlags::COLOR, + base_mip_level: 0, + level_count: 1, + base_array_layer: 0, + layer_count: 1, + }, + layout: vk::ImageLayout::GENERAL, + }, + } +} + +fn submit_graph(test: &TestVulkan, graph: &mut renderer_rendergraph::CompiledGraph) { + let device = test.device(); + let pool_info = vk::CommandPoolCreateInfo::default() + .queue_family_index(device.queues.graphics().family.index.index()); + let pool = unsafe { device.dev().create_command_pool(&pool_info, None) } + .expect("create alias test command pool"); + let allocation_info = vk::CommandBufferAllocateInfo::default() + .command_pool(pool) + .level(vk::CommandBufferLevel::PRIMARY) + .command_buffer_count(1); + let command_buffer = unsafe { device.dev().allocate_command_buffers(&allocation_info) } + .expect("allocate alias test command buffer")[0]; + unsafe { + device + .dev() + .begin_command_buffer(command_buffer, &vk::CommandBufferBeginInfo::default()) + .expect("begin alias test command buffer"); + } + let mut recorder = CommandRecorder::new(device.dev(), command_buffer); + graph + .record_commands(&mut recorder) + .expect("record alias graph"); + unsafe { + device + .dev() + .end_command_buffer(command_buffer) + .expect("end alias test command buffer"); + let command_buffers = [command_buffer]; + let submit = vk::SubmitInfo::default().command_buffers(&command_buffers); + device + .dev() + .queue_submit(device.queues.graphics().raw, &[submit], vk::Fence::null()) + .expect("submit alias graph"); + } + device.wait_idle().expect("wait for alias graph idle"); + unsafe { device.dev().destroy_command_pool(pool, None) }; +} + #[test] fn imported_buffer_copy_records_under_validation() { let Some(test) = TestVulkan::acquire() else { @@ -91,3 +177,154 @@ fn imported_buffer_copy_records_under_validation() { device.wait_idle().expect("wait for graph device idle"); test.assert_no_validation_errors(); } + +#[test] +fn resolved_buffers_alias_and_submit_with_memory_barrier() { + let Some(test) = TestVulkan::acquire() else { + return; + }; + let device = test.device(); + if device.enabled_features().core13.synchronization2 != vk::TRUE { + eprintln!("Skipping graph aliasing: synchronization2 is unsupported"); + return; + } + let bridge = Arc::new( + Buffer::new( + device.clone(), + BufferDesc { + size: 256, + usage: vk::BufferUsageFlags::TRANSFER_DST, + mem_location: MemoryLocation::CpuToGpu, + ..Default::default() + }, + ) + .expect("create dependency bridge buffer"), + ); + let mut graph = RenderGraph::new(); + let first = graph.create_buffer(GraphBufferDesc { + size: 256, + ..Default::default() + }); + let second = graph.create_buffer(GraphBufferDesc { + size: 256, + ..Default::default() + }); + let bridge = graph + .import_buffer(bridge, ResourceState::default()) + .expect("import dependency bridge"); + graph.push(ScopedNoop(vec![ + (first.id(), buffer_scope()), + (bridge.id(), buffer_scope()), + ])); + graph.push(ScopedNoop(vec![ + (bridge.id(), buffer_scope()), + (second.id(), buffer_scope()), + ])); + + let mut compiled = graph.compile().expect("compile buffer alias graph"); + compiled + .resolve(device) + .expect("resolve buffer alias graph"); + compiled + .resolve(device) + .expect("resolving an already-resolved graph is idempotent"); + let first_placement = compiled.resource_lifetimes()[&first.id()].placement; + let second_placement = compiled.resource_lifetimes()[&second.id()].placement; + assert_eq!(first_placement.pool, second_placement.pool); + assert_eq!(first_placement.offset, second_placement.offset); + assert!(first_placement.aliased && second_placement.aliased); + assert!(compiled.barriers().iter().any(|barrier| { + barrier.before_command == CommandIndex(1) && !barrier.memory_barriers.is_empty() + })); + + submit_graph(&test, &mut compiled); + test.assert_no_validation_errors(); + drop(compiled); + device + .wait_idle() + .expect("wait before shared allocation teardown"); + test.assert_no_validation_errors(); +} + +#[test] +fn resolved_images_alias_and_submit_with_undefined_transition() { + let Some(test) = TestVulkan::acquire() else { + return; + }; + let device = test.device(); + if device.enabled_features().core13.synchronization2 != vk::TRUE { + eprintln!("Skipping graph aliasing: synchronization2 is unsupported"); + return; + } + let bridge = Arc::new( + Buffer::new( + device.clone(), + BufferDesc { + size: 256, + usage: vk::BufferUsageFlags::TRANSFER_DST, + mem_location: MemoryLocation::CpuToGpu, + ..Default::default() + }, + ) + .expect("create dependency bridge buffer"), + ); + let mut graph = RenderGraph::new(); + let first = graph.create_texture(GraphTextureDesc { + format: vk::Format::R8G8B8A8_UNORM, + extent: vk::Extent3D { + width: 16, + height: 16, + depth: 1, + }, + extra_usage: vk::ImageUsageFlags::STORAGE, + ..Default::default() + }); + let second = graph.create_texture(GraphTextureDesc { + format: vk::Format::R8G8B8A8_UNORM, + extent: vk::Extent3D { + width: 16, + height: 16, + depth: 1, + }, + extra_usage: vk::ImageUsageFlags::STORAGE, + ..Default::default() + }); + let bridge = graph + .import_buffer(bridge, ResourceState::default()) + .expect("import dependency bridge"); + graph.push(ScopedNoop(vec![ + (first.id(), image_scope()), + (bridge.id(), buffer_scope()), + ])); + graph.push(ScopedNoop(vec![ + (bridge.id(), buffer_scope()), + (second.id(), image_scope()), + ])); + + let mut compiled = graph.compile().expect("compile image alias graph"); + compiled.resolve(device).expect("resolve image alias graph"); + compiled + .resolve(device) + .expect("resolving an already-resolved graph is idempotent"); + let first_placement = compiled.resource_lifetimes()[&first.id()].placement; + let second_placement = compiled.resource_lifetimes()[&second.id()].placement; + assert_eq!(first_placement.pool, second_placement.pool); + assert_eq!(first_placement.offset, second_placement.offset); + assert!(first_placement.aliased && second_placement.aliased); + assert!(compiled.barriers().iter().any(|barrier| { + barrier.before_command == CommandIndex(1) + && barrier.resources.iter().any(|resource| { + resource.id == second.id() + && resource.old_layout == Some(vk::ImageLayout::UNDEFINED) + && resource.src_access == vk::AccessFlags2::SHADER_STORAGE_WRITE + }) + })); + + submit_graph(&test, &mut compiled); + test.assert_no_validation_errors(); + drop(compiled); + device + .wait_idle() + .expect("wait before shared allocation teardown"); + test.assert_no_validation_errors(); +} diff --git a/crates/renderer_resources/AGENTS.md b/crates/renderer_resources/AGENTS.md index 7b10b3a..a9a9adb 100644 --- a/crates/renderer_resources/AGENTS.md +++ b/crates/renderer_resources/AGENTS.md @@ -4,19 +4,21 @@ `renderer_resources` owns Vulkan buffer, image, view, descriptor, sampler, shader-module, and pipeline construction on the `renderer_device` layer. It handles resource memory and handle lifetimes, but contains no window, swapchain, or presentation orchestration; those remain in `renderer`. It must not depend on `renderer`. ## Module map -- `lib.rs`: re-exports `Device`, `Error`, `Result`, `device`, and `make_extension` from `renderer_device`; declares the public resource modules, including descriptor metadata and validation. -- `buffers.rs`: `BufferDesc` and `Buffer`; Vulkan buffer creation, GPU allocation/binding, raw access, and host mapping. -- `images.rs`: `ImageDesc`, `Image`, `ImageViewDesc`, `ImageView`, subresource ranges, extents, offsets, and queue ownership. `ImageInner` distinguishes allocated images from borrowed swapchain images. +- `lib.rs`: re-exports `Device`, `Error`, `Result`, `device`, and `make_extension` from `renderer_device`; declares public buffer, image, memory, pipeline, utility, and descriptor modules. +- `buffers.rs`: `BufferDesc` and `Buffer`; allocated and unallocated Vulkan buffer creation, memory requirement query/binding, raw access, and host mapping. +- `images.rs`: `ImageDesc`, `Image`, `ImageViewDesc`, `ImageView`, subresource ranges, extents, offsets, and queue ownership. `ImageInner` distinguishes allocated/unallocated images from borrowed swapchain images. +- `memory.rs`: re-exports device `MemoryRequirements`; exposes memory validation errors, `MemoryMember`, and `MemoryGroup` for aligned shared buffer/image allocations. - `pipeline.rs`: shader-stage and graphics/compute pipeline descriptions; `DescriptorPool`, `DescriptorSetLayout`, `PipelineLayout`, `Sampler`, `ShaderModule`, and `Pipeline`. - `descriptors.rs`: managed descriptor binding/resource metadata, validation plans, errors, and pool-generation helpers; it remains independent of render-graph types. - `util.rs`: `eq_f32`, `ord_f32`, `hash_f32`, `F32`, and the crate-private `weak_vec` module / `WeakVec` used by image view tracking. ## Key types and API - These modules are public from `renderer_resources`; the crate root re-exports the core device types and `device` module from `renderer_device` for resource construction. -- `BufferDesc` configures Vulkan buffer flags, size, usage, queue families, `renderer_device::device::MemoryLocation`, allocation strategy, and optional debug name. `Buffer::new` requests a VMA allocation through `DeviceInner::allocate_for_buffer` and binds it through `bind_buffer_memory`; `map` and `map_mut` return mapped slices when available. +- `BufferDesc` configures Vulkan buffer flags, size, usage, queue families, `renderer_device::device::MemoryLocation`, allocation strategy, and optional debug name. `Buffer::new` requests a VMA allocation through `DeviceInner::allocate_for_buffer` and binds it through `bind_buffer_memory`; `Buffer::new_unallocated` creates only the Vulkan handle. Its `memory_requirements` includes dedicated-allocation hints, and `bind_allocation(Arc>, offset)` validates then binds shared memory. `map` and `map_mut` return mapped slices when available. - `Buffer::raw` returns the Vulkan handle and `len` returns the configured size. Mapping is bounded to the descriptor size, not the VMA allocation block size. CPU-mapped locations select host-visible, host-coherent memory. - `ImageDesc` configures image type, format, extent, layers/mips, samples, tiling, usage, initial layout, queue sharing, and allocation strategy/location. `Image::new` uses the device's VMA allocation and image-binding helpers. -- `Image::new` creates an allocated image. `Image::new_with_allocation` accepts an explicit `renderer_device::device::Allocation` and validates size and memory-type compatibility. +- `Image::new` creates an allocated image; `Image::new_unallocated` creates an unbound image. `memory_requirements` includes dedicated-allocation hints, and `bind_allocation(Arc>, offset)` validates then binds shared memory. `Image::new_with_allocation` accepts an explicit `renderer_device::device::Allocation` and validates size and memory-type compatibility. +- `MemoryGroup::new(memory_type_index, property_flags, size, alignment)` fixes the shared pool's memory type and capacity. `add(MemoryMember::Buffer(&mut buffer))` / `add(MemoryMember::Image(&mut image))` queries requirements, checks that the selected type is allowed for every member, and packs each at an aligned offset; `allocate(&device)` creates aliasable memory, then consuming `bind_all()` transfers allocation references into the resources. The group does not retain its own allocation after binding. - `Image::from_swapchain_raw(image, device, format, extent, usage)` builds an image wrapper for an image owned by a swapchain; renderer calls it from `Swapchain::acquire_image`. - `ImageViewDesc` describes a view's type, format, component mapping, aspect, mip range, and layer range. `Image::create_view` validates compatibility and returns `ManuallyDrop`. - `Image::create_view` resolves remaining mip/layer sentinels against the parent image, defaults an undefined view format to the image format, and checks image/view kind and format compatibility. @@ -46,6 +48,8 @@ - `from_swapchain_raw` records a 2D, one-mip, one-layer, single-sample, optimal-tiled image with PRESENT queue sharing and UNDEFINED initial layout; the caller supplies format, extent, and usage. - `from_swapchain_raw` retains a `Device` for views and device queries, but not ownership of the image handle or swapchain lifetime; the renderer's frame/swapchain token must keep the swapchain alive. - Allocated images and buffers use `DeviceObject` for Vulkan handles and allocation ownership. Keep resource destruction ordered before freeing bound memory. +- Unallocated resources use `Allocation::Unmanaged` until bound. Shared bind APIs are one-shot; they validate offset alignment, allowed memory type, and allocation bounds before calling the device layer. Keep `DeviceObject` on the resource after a successful bind so Vulkan handles are destroyed before their allocation reference. +- `MemoryGroup` only accepts members whose `memory_type_bits` include its selected type (the intersection across all members); it rejects dedicated-required resources. Group offsets use aligned bump packing, and allocation ownership transfers into each bound member at `bind_all`. - `Buffer` equality compares the raw buffer handle; `Image` equality compares the raw image handle. Descriptor equality is separate from resource object identity. - `Allocation::Owned` owns a device-managed allocation; `Shared` shares it through `Arc` and permits mutable access only when uniquely held; `Unmanaged` carries no allocation. - `ImageView` retains `Arc` so the source image outlives the view. `Image::create_view` returns `ManuallyDrop`; callers must transfer/drop it deliberately. diff --git a/crates/renderer_resources/src/buffers.rs b/crates/renderer_resources/src/buffers.rs index ffe6efd..2f3636f 100644 --- a/crates/renderer_resources/src/buffers.rs +++ b/crates/renderer_resources/src/buffers.rs @@ -126,6 +126,98 @@ impl Buffer { Ok(buffer) } + /// Creates a Vulkan buffer without allocating or binding memory. + pub fn new_unallocated(device: Device, desc: BufferDesc) -> crate::Result { + let buffer = Self::new_raw(device, &desc)?; + + Ok(Self { + buffer: BufferHandle::Owned(buffer), + desc, + alloc: Allocation::Unmanaged { + requirements: std::sync::OnceLock::new(), + }, + }) + } + + /// Returns the Vulkan memory requirements, including dedicated-allocation hints. + pub fn memory_requirements(&self) -> crate::memory::Result { + let Some(buffer) = self.owned() else { + return Err(crate::memory::Error::RequirementQuery( + crate::Error::Unspecified, + )); + }; + + let requirements_cache = match &self.alloc { + Allocation::Unmanaged { requirements } => Some(requirements), + Allocation::Owned(_) | Allocation::Shared(_) => None, + }; + + if let Some(requirements) = requirements_cache.and_then(std::sync::OnceLock::get) { + return Ok(*requirements); + } + + let requirements = buffer + .device() + .buffer_memory_requirements(**buffer) + .map_err(crate::memory::Error::RequirementQuery)?; + + if let Some(cache) = requirements_cache { + let _ = cache.set(requirements); + } + + Ok(requirements) + } + + /// Binds this buffer into an aliasable allocation at an allocation-local offset. + /// + /// Requirements are queried once and cached in the resource. The Vulkan bind remains the + /// authoritative validation. + pub fn bind_allocation( + &mut self, + allocation: Arc>, + offset: u64, + ) -> crate::memory::Result<()> { + if !matches!(self.alloc, Allocation::Unmanaged { .. }) { + return Err(crate::memory::Error::AlreadyBound); + } + + let requirements = self.memory_requirements()?; + crate::memory::validate_binding(&requirements, allocation.as_ref(), offset)?; + let device = self.owned().expect("owned buffer").device(); + + self.alloc = device + .bind_buffer_at_offset(self.raw(), &requirements, allocation, offset) + .map_err(crate::memory::Error::Bind)?; + + Ok(()) + } + + pub(crate) fn validate_unallocated(&self) -> crate::memory::Result<()> { + if matches!(self.alloc, Allocation::Unmanaged { .. }) { + Ok(()) + } else { + Err(crate::memory::Error::AlreadyBound) + } + } + + pub(crate) fn validate_device(&self, device: &Device) -> crate::memory::Result<()> { + if self.owned().expect("unallocated buffer is owned").device() == device { + Ok(()) + } else { + Err(crate::memory::Error::Bind( + crate::Error::AllocationDeviceMismatch, + )) + } + } + + pub(crate) fn set_shared_allocation( + &mut self, + allocation: Arc>, + ) { + debug_assert!(matches!(self.alloc, Allocation::Unmanaged { .. })); + self.alloc = Allocation::Shared(allocation); + } + fn new_raw(device: Device, desc: &BufferDesc) -> crate::Result> { let queue_families = device.queues.family_indices(desc.queue_families); @@ -192,7 +284,9 @@ impl Buffer { size, ..Default::default() }, - alloc: Allocation::Unmanaged, + alloc: Allocation::Unmanaged { + requirements: std::sync::OnceLock::new(), + }, } } } @@ -303,6 +397,200 @@ mod allocation_validation_tests { } } +#[cfg(all(test, feature = "testing"))] +mod unallocated_memory_tests { + use ash::vk; + use renderer_device::testing::TestVulkan; + + use super::{Buffer, BufferDesc}; + use crate::{ + device::MemoryLocation, + images::{Image, ImageDesc}, + memory::{Error, MemoryGroup, MemoryMember}, + }; + + fn buffer_desc() -> BufferDesc { + BufferDesc { + size: 256, + usage: vk::BufferUsageFlags::TRANSFER_SRC | vk::BufferUsageFlags::TRANSFER_DST, + mem_location: MemoryLocation::GpuOnly, + ..Default::default() + } + } + + #[test] + fn unallocated_resources_query_requirements_and_bind() { + let test = TestVulkan::expect(); + let device = test.device().clone(); + let mut buffer = Buffer::new_unallocated(device.clone(), buffer_desc()) + .expect("create unallocated buffer"); + let buffer_requirements = buffer.memory_requirements().expect("buffer requirements"); + assert!(buffer_requirements.size >= buffer.len()); + assert!(matches!( + &buffer.alloc, + crate::device::Allocation::Unmanaged { .. } + )); + + let mut image = Image::new_unallocated( + device.clone(), + ImageDesc { + format: vk::Format::R8G8B8A8_UNORM, + extent: vk::Extent3D { + width: 16, + height: 16, + depth: 1, + }, + usage: vk::ImageUsageFlags::TRANSFER_DST, + ..Default::default() + }, + ) + .expect("create unallocated image"); + let image_requirements = image.memory_requirements().expect("image requirements"); + assert!(image_requirements.size > 0); + assert!( + image + .allocation() + .is_some_and(|allocation| allocation.allocation().is_none()) + ); + + let buffer_memory = device + .allocate_memory(&buffer_requirements, vk::MemoryPropertyFlags::empty()) + .expect("allocate buffer memory"); + buffer + .bind_allocation(buffer_memory, 0) + .expect("bind buffer at offset zero"); + assert!(matches!( + &buffer.alloc, + crate::device::Allocation::Shared(_) + )); + + let image_memory = device + .allocate_memory(&image_requirements, vk::MemoryPropertyFlags::empty()) + .expect("allocate image memory"); + image + .bind_allocation(image_memory, 0) + .expect("bind image at offset zero"); + assert!(matches!( + image.allocation(), + Some(crate::device::Allocation::Shared(_)) + )); + drop(image); + drop(buffer); + test.device().wait_idle().expect("wait for device idle"); + test.assert_no_validation_errors(); + } + + #[test] + fn bind_rejects_invalid_offsets_and_rebinding() { + let test = TestVulkan::expect(); + let device = test.device().clone(); + + let mut buffer = Buffer::new_unallocated(device.clone(), buffer_desc()) + .expect("create buffer for offset checks"); + let requirements = buffer.memory_requirements().expect("buffer requirements"); + let allocation = device + .allocate_memory(&requirements, vk::MemoryPropertyFlags::empty()) + .expect("allocate offset-check memory"); + if requirements.alignment > 1 { + assert!(matches!( + buffer.bind_allocation(allocation.clone(), 1), + Err(Error::MisalignedOffset { .. }) + )); + } + assert!(matches!( + buffer.bind_allocation(allocation.clone(), requirements.size), + Err(Error::OutOfBounds { .. }) + )); + buffer + .bind_allocation(allocation.clone(), 0) + .expect("first buffer bind"); + assert!(matches!( + buffer.bind_allocation(allocation, 0), + Err(Error::AlreadyBound) + )); + + let allocated = device + .allocate_memory(&requirements, vk::MemoryPropertyFlags::empty()) + .expect("allocate for synthetic type-bits validation"); + let impossible = crate::memory::MemoryRequirements { + memory_type_bits: 0, + ..requirements + }; + assert!(matches!( + crate::memory::validate_binding(&impossible, allocated.as_ref(), 0), + Err(Error::IncompatibleMemoryType { .. }) + )); + + drop(buffer); + test.device().wait_idle().expect("wait for device idle"); + test.assert_no_validation_errors(); + } + + #[test] + fn memory_group_packs_and_binds_two_buffers() { + let test = TestVulkan::expect(); + let device = test.device().clone(); + let mut first = + Buffer::new_unallocated(device.clone(), buffer_desc()).expect("create first buffer"); + let mut second = + Buffer::new_unallocated(device.clone(), buffer_desc()).expect("create second buffer"); + let first_requirements = first.memory_requirements().expect("first requirements"); + let second_requirements = second.memory_requirements().expect("second requirements"); + let common_bits = + first_requirements.memory_type_bits & second_requirements.memory_type_bits; + assert_ne!(common_bits, 0, "the test adapter needs a common type"); + let memory_type_index = common_bits.trailing_zeros(); + let alignment = first_requirements + .alignment + .max(second_requirements.alignment); + let first_end = first_requirements.size; + let expected_second_offset = first_end + .checked_add(alignment - 1) + .expect("offset arithmetic") + / alignment + * alignment; + let size = expected_second_offset + second_requirements.size; + let mut group = MemoryGroup::new( + memory_type_index, + vk::MemoryPropertyFlags::empty(), + size, + alignment, + ) + .expect("create memory group"); + let first_offset = group + .add(MemoryMember::Buffer(&mut first)) + .expect("pack first buffer"); + let second_offset = group + .add(MemoryMember::Buffer(&mut second)) + .expect("pack second buffer"); + assert_eq!(first_offset, 0); + assert_eq!(second_offset, expected_second_offset); + assert_eq!(second_offset % alignment, 0); + group.allocate(&device).expect("allocate shared memory"); + group.bind_all().expect("bind group members"); + assert!(matches!(&first.alloc, crate::device::Allocation::Shared(_))); + assert!(matches!( + &second.alloc, + crate::device::Allocation::Shared(_) + )); + assert_eq!( + first + .alloc + .allocation() + .map(crate::device::GpuAllocation::memory), + second + .alloc + .allocation() + .map(crate::device::GpuAllocation::memory), + "both buffers retain the shared Vulkan memory block" + ); + drop(first); + drop(second); + test.device().wait_idle().expect("wait for device idle"); + test.assert_no_validation_errors(); + } +} + /// Byte range selected by a buffer view. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub struct BufferRange { diff --git a/crates/renderer_resources/src/images.rs b/crates/renderer_resources/src/images.rs index e677182..4adb734 100644 --- a/crates/renderer_resources/src/images.rs +++ b/crates/renderer_resources/src/images.rs @@ -206,6 +206,109 @@ impl Image { ) } + /// Creates a Vulkan image without allocating or binding memory. + pub fn new_unallocated(device: Device, desc: ImageDesc) -> crate::Result { + let image = Self::new_raw(device, &desc)?; + Ok(Self { + image: ImageInner::Allocated( + image, + Allocation::Unmanaged { + requirements: std::sync::OnceLock::new(), + }, + ), + desc, + views: Default::default(), + }) + } + + /// Returns the Vulkan memory requirements, including dedicated-allocation hints. + pub fn memory_requirements(&self) -> crate::memory::Result { + let ImageInner::Allocated(image, allocation) = &self.image else { + return Err(crate::memory::Error::RequirementQuery( + crate::Error::Unspecified, + )); + }; + let requirements_cache = match allocation { + Allocation::Unmanaged { requirements } => Some(requirements), + Allocation::Owned(_) | Allocation::Shared(_) => None, + }; + + if let Some(requirements) = requirements_cache.and_then(std::sync::OnceLock::get) { + return Ok(*requirements); + } + + let requirements = image + .device() + .image_memory_requirements(**image) + .map_err(crate::memory::Error::RequirementQuery)?; + + if let Some(cache) = requirements_cache { + let _ = cache.set(requirements); + } + + Ok(requirements) + } + + /// Binds this image into an aliasable allocation at an allocation-local offset. + /// + /// Requirements are queried once and cached in the resource. The Vulkan bind remains the + /// authoritative validation. + pub fn bind_allocation( + &mut self, + allocation: Arc>, + offset: u64, + ) -> crate::memory::Result<()> { + let ImageInner::Allocated(_, current_allocation) = &self.image else { + return Err(crate::memory::Error::AlreadyBound); + }; + if !matches!(current_allocation, Allocation::Unmanaged { .. }) { + return Err(crate::memory::Error::AlreadyBound); + } + + let requirements = self.memory_requirements()?; + crate::memory::validate_binding(&requirements, allocation.as_ref(), offset)?; + let ImageInner::Allocated(image, current_allocation) = &mut self.image else { + unreachable!("image kind checked above") + }; + *current_allocation = image + .device() + .bind_image_at_offset(**image, &requirements, allocation, offset) + .map_err(crate::memory::Error::Bind)?; + Ok(()) + } + + pub(crate) fn validate_unallocated(&self) -> crate::memory::Result<()> { + match &self.image { + ImageInner::Allocated(_, Allocation::Unmanaged { .. }) => Ok(()), + ImageInner::Allocated(_, _) | ImageInner::Swapchain(_, _) => { + Err(crate::memory::Error::AlreadyBound) + } + #[cfg(feature = "testing")] + ImageInner::Testing(_) => Err(crate::memory::Error::AlreadyBound), + } + } + + pub(crate) fn validate_device(&self, device: &Device) -> crate::memory::Result<()> { + if self.image.device() == device { + Ok(()) + } else { + Err(crate::memory::Error::Bind( + crate::Error::AllocationDeviceMismatch, + )) + } + } + + pub(crate) fn set_shared_allocation( + &mut self, + allocation: Arc>, + ) { + let ImageInner::Allocated(_, current_allocation) = &mut self.image else { + unreachable!("group members are owned images") + }; + debug_assert!(matches!(current_allocation, Allocation::Unmanaged { .. })); + *current_allocation = Allocation::Shared(allocation); + } + fn new_with_allocation_unchecked( device: Device, image: DeviceObject, diff --git a/crates/renderer_resources/src/lib.rs b/crates/renderer_resources/src/lib.rs index e9dd614..1c85f7b 100644 --- a/crates/renderer_resources/src/lib.rs +++ b/crates/renderer_resources/src/lib.rs @@ -3,5 +3,6 @@ pub use renderer_device::{Device, Error, Result, device, make_extension}; pub mod buffers; pub mod descriptors; pub mod images; +pub mod memory; pub mod pipeline; pub mod util; diff --git a/crates/renderer_resources/src/memory.rs b/crates/renderer_resources/src/memory.rs new file mode 100644 index 0000000..da36ebe --- /dev/null +++ b/crates/renderer_resources/src/memory.rs @@ -0,0 +1,411 @@ +//! Shared aliasable memory allocation helpers for buffers and images. + +use std::sync::Arc; + +use ash::vk; + +use crate::{ + Device, + buffers::Buffer, + device::{DeviceObject, GpuAllocation, MemoryBinding}, + images::Image, +}; + +/// Vulkan memory requirements returned by the device layer. +pub use crate::device::MemoryRequirements; + +/// Errors from resource requirement queries, validation, and shared binding. +#[derive(Debug)] +pub enum Error { + RequirementQuery(crate::Error), + Bind(crate::Error), + Allocation(crate::Error), + MisalignedOffset { + offset: u64, + alignment: u64, + }, + IncompatibleMemoryType { + memory_type_index: u32, + memory_type_bits: u32, + }, + OutOfBounds { + offset: u64, + end: u64, + allocation_size: u64, + }, + AlreadyBound, + AlreadyAllocated, + RequiresDedicatedAllocation, + GroupMemoryTypeMismatch { + memory_type_index: u32, + }, + GroupCapacityExceeded { + requested_end: u64, + capacity: u64, + }, + EmptyGroup, + NotAllocated, + InvalidGroupRequirements, + OffsetOverflow, +} + +impl core::fmt::Display for Error { + fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { + match self { + Self::RequirementQuery(error) => { + write!(f, "failed to query resource memory requirements: {error}") + } + Self::Bind(error) => write!(f, "failed to bind resource memory: {error}"), + Self::Allocation(error) => write!(f, "failed to allocate group memory: {error}"), + Self::MisalignedOffset { offset, alignment } => write!( + f, + "allocation offset {offset} is not aligned to {alignment}" + ), + Self::IncompatibleMemoryType { + memory_type_index, + memory_type_bits, + } => write!( + f, + "allocation memory type {memory_type_index} is not allowed by mask {memory_type_bits:#010x}" + ), + Self::OutOfBounds { + offset, + end, + allocation_size, + } => write!( + f, + "resource range [{offset}, {end}) does not fit allocation size {allocation_size}" + ), + Self::AlreadyBound => f.write_str("resource is already bound to an allocation"), + Self::AlreadyAllocated => f.write_str("memory group has already been allocated"), + Self::RequiresDedicatedAllocation => f.write_str( + "resources requiring dedicated allocations cannot be added to a memory group", + ), + Self::GroupMemoryTypeMismatch { memory_type_index } => write!( + f, + "resource memory requirements do not include memory type {memory_type_index}" + ), + Self::GroupCapacityExceeded { + requested_end, + capacity, + } => write!( + f, + "memory group capacity exceeded: requested end {requested_end}, capacity {capacity}" + ), + Self::EmptyGroup => f.write_str("memory group has no members"), + Self::NotAllocated => f.write_str("memory group has not been allocated"), + Self::InvalidGroupRequirements => { + f.write_str("memory group size or alignment is invalid") + } + Self::OffsetOverflow => f.write_str("memory group offset arithmetic overflowed"), + } + } +} + +impl std::error::Error for Error { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + match self { + Self::RequirementQuery(error) | Self::Bind(error) | Self::Allocation(error) => { + Some(error) + } + _ => None, + } + } +} + +pub type Result = core::result::Result; + +pub(crate) fn validate_binding( + requirements: &MemoryRequirements, + allocation: &DeviceObject, + offset: u64, +) -> Result<()> { + let alignment = requirements.alignment; + if alignment == 0 || !offset.is_multiple_of(alignment) { + return Err(Error::MisalignedOffset { offset, alignment }); + } + + let memory_type_index = allocation.memory_type_index(); + if requirements.memory_type_bits & (1 << memory_type_index) == 0 { + return Err(Error::IncompatibleMemoryType { + memory_type_index, + memory_type_bits: requirements.memory_type_bits, + }); + } + + let end = offset + .checked_add(requirements.size) + .ok_or(Error::OutOfBounds { + offset, + end: u64::MAX, + allocation_size: allocation.size(), + })?; + + if end > allocation.size() { + return Err(Error::OutOfBounds { + offset, + end, + allocation_size: allocation.size(), + }); + } + + Ok(()) +} + +enum Member<'a> { + Buffer { + resource: &'a mut Buffer, + offset: u64, + }, + Image { + resource: &'a mut Image, + offset: u64, + }, +} + +/// Packs resources into a single aliasable allocation with a fixed memory type. +/// +/// Members are exclusively borrowed until `bind_all` consumes the group. Their +/// allocation ownership is then transferred to each resource; the group retains +/// no allocation handle after binding. +pub struct MemoryGroup<'a> { + memory_type_index: u32, + property_flags: vk::MemoryPropertyFlags, + size: u64, + alignment: u64, + next_offset: u64, + members: Vec>, + allocation: Option>>, +} + +impl<'a> MemoryGroup<'a> { + /// Creates an empty group for a fixed memory type and allocation capacity. + pub fn new( + memory_type_index: u32, + property_flags: vk::MemoryPropertyFlags, + size: u64, + alignment: u64, + ) -> Result { + if memory_type_index >= u32::BITS || size == 0 || alignment == 0 { + return Err(Error::InvalidGroupRequirements); + } + Ok(Self { + memory_type_index, + property_flags, + size, + alignment, + next_offset: 0, + members: Vec::new(), + allocation: None, + }) + } + + /// Adds a buffer, placing it at the next suitably aligned offset. + pub fn add_buffer(&mut self, resource: &'a mut Buffer) -> Result { + if self.allocation.is_some() { + return Err(Error::AlreadyAllocated); + } + + let requirements = resource.memory_requirements()?; + self.add_member_type_check(&requirements)?; + let offset = self.next_member_offset(&requirements)?; + + let next_offset = offset + .checked_add(requirements.size) + .ok_or(Error::OffsetOverflow)?; + + self.members.push(Member::Buffer { resource, offset }); + + self.next_offset = next_offset; + Ok(offset) + } + + /// Adds either resource kind through a uniform entry point. + pub fn add(&mut self, member: MemoryMember<'a>) -> Result { + match member { + MemoryMember::Buffer(resource) => self.add_buffer(resource), + MemoryMember::Image(resource) => self.add_image(resource), + } + } + + /// Adds an image, placing it at the next suitably aligned offset. + pub fn add_image(&mut self, resource: &'a mut Image) -> Result { + if self.allocation.is_some() { + return Err(Error::AlreadyAllocated); + } + + let requirements = resource.memory_requirements()?; + self.add_member_type_check(&requirements)?; + let offset = self.next_member_offset(&requirements)?; + + let next_offset = offset + .checked_add(requirements.size) + .ok_or(Error::OffsetOverflow)?; + + self.members.push(Member::Image { resource, offset }); + + self.next_offset = next_offset; + Ok(offset) + } + + /// Allocates the group's shared backing memory. + pub fn allocate(&mut self, device: &Device) -> Result<()> { + if self.members.is_empty() { + return Err(Error::EmptyGroup); + } + + if self.allocation.is_some() { + return Err(Error::AlreadyAllocated); + } + + let requirements = MemoryRequirements { + size: self.size, + alignment: self.alignment, + memory_type_bits: 1 << self.memory_type_index, + requires_dedicated_allocation: false, + prefers_dedicated_allocation: false, + }; + + let allocation = device + .allocate_memory(&requirements, self.property_flags) + .map_err(Error::Allocation)?; + + for member in &self.members { + let (requirements, offset) = match member { + Member::Buffer { resource, offset } => (resource.memory_requirements()?, *offset), + Member::Image { resource, offset } => (resource.memory_requirements()?, *offset), + }; + validate_binding(&requirements, allocation.as_ref(), offset)?; + } + + self.allocation = Some(allocation); + + Ok(()) + } + + /// Binds every member and transfers the allocation reference into each resource. + pub fn bind_all(mut self) -> Result<()> { + let allocation = self.allocation.take().ok_or(Error::NotAllocated)?; + + // Validate resources first to preserve the same resource-level error mapping as + // individual binding, then prepare the ordered device batch from cached requirements. + let mut bindings = Vec::with_capacity(self.members.len()); + for member in &self.members { + match member { + Member::Buffer { resource, offset } => { + resource.validate_unallocated()?; + let requirements = resource.memory_requirements()?; + validate_binding(&requirements, allocation.as_ref(), *offset)?; + resource.validate_device(allocation.device())?; + bindings.push(MemoryBinding::Buffer { + buffer: resource.raw(), + offset: *offset, + requirements, + }); + } + Member::Image { resource, offset } => { + resource.validate_unallocated()?; + let requirements = resource.memory_requirements()?; + validate_binding(&requirements, allocation.as_ref(), *offset)?; + resource.validate_device(allocation.device())?; + bindings.push(MemoryBinding::Image { + image: resource.raw(), + offset: *offset, + requirements, + }); + } + } + } + + if let Err(bind_error) = allocation + .device() + .bind_group_at_offset(&allocation, &bindings) + { + // A Vulkan bind failure leaves its earlier prefix bound; retain its allocation just + // as repeated `bind_allocation` calls would have done. + for (index, member) in self.members.into_iter().enumerate() { + if index >= bind_error.bound_count { + break; + } + match member { + Member::Buffer { resource, .. } => { + resource.set_shared_allocation(allocation.clone()); + } + Member::Image { resource, .. } => { + resource.set_shared_allocation(allocation.clone()); + } + } + } + return Err(Error::Bind(bind_error.error)); + } + + for member in self.members { + match member { + Member::Buffer { resource, .. } => { + resource.set_shared_allocation(allocation.clone()); + } + Member::Image { resource, .. } => { + resource.set_shared_allocation(allocation.clone()); + } + } + } + + // `allocation` drops here. Only successfully bound resources retain it. + Ok(()) + } + + fn add_member_type_check(&self, requirements: &MemoryRequirements) -> Result<()> { + if requirements.requires_dedicated_allocation { + return Err(Error::RequiresDedicatedAllocation); + } + if requirements.memory_type_bits & (1 << self.memory_type_index) == 0 { + return Err(Error::GroupMemoryTypeMismatch { + memory_type_index: self.memory_type_index, + }); + } + Ok(()) + } + + fn next_member_offset(&self, requirements: &MemoryRequirements) -> Result { + if requirements.alignment == 0 { + return Err(Error::InvalidGroupRequirements); + } + let alignment = checked_lcm(requirements.alignment, self.alignment)?; + let offset = (self + .next_offset + .checked_add(alignment - 1) + .ok_or(Error::OffsetOverflow)? + / alignment) + .checked_mul(alignment) + .ok_or(Error::OffsetOverflow)?; + let end = offset + .checked_add(requirements.size) + .ok_or(Error::OffsetOverflow)?; + if end > self.size { + return Err(Error::GroupCapacityExceeded { + requested_end: end, + capacity: self.size, + }); + } + Ok(offset) + } +} + +fn checked_lcm(lhs: u64, rhs: u64) -> Result { + let gcd = { + let (mut lhs, mut rhs) = (lhs, rhs); + while rhs != 0 { + (lhs, rhs) = (rhs, lhs % rhs); + } + lhs + }; + lhs.checked_div(gcd) + .and_then(|lhs| lhs.checked_mul(rhs)) + .ok_or(Error::OffsetOverflow) +} + +/// A resource accepted by [`MemoryGroup::add`]. +pub enum MemoryMember<'a> { + Buffer(&'a mut Buffer), + Image(&'a mut Image), +} diff --git a/design-plans/rendergraph-transient-aliasing.md b/design-plans/rendergraph-transient-aliasing.md new file mode 100644 index 0000000..3cc3250 --- /dev/null +++ b/design-plans/rendergraph-transient-aliasing.md @@ -0,0 +1,202 @@ +# Plan: Correct Transient-Resource Aliasing in `renderer_rendergraph` + +Status: proposed — rev 3 (code-verified; rev-2 review + follow-up comments applied) +Date: 2026-10-26 +Scope: `crates/renderer_rendergraph`, `crates/renderer_resources`, `crates/renderer_device` + +## Goal + +Fix the broken transient-resource aliasing in the rendergraph: lifetimes must be +derived from the graph's real ordering (dependency DAG + batches, not command +indices), alias reuse must be synchronized with barriers, and the resources +crate must gain the infrastructure to compute correct memory requirements so +correctly sized pools can be allocated and bound to graph-created resources. + +## Diagnosis (verified against code) + +| # | Defect | Evidence | +|---|--------|----------| +| 1 | Lifetimes are command-index intervals (`first_command`/`last_command`), but commands within a `topo_batches` batch are conceptually concurrent — two resources used by different commands of the same batch are wrongly treated as non-overlapping | `graph.rs:66-79, 862-877` | +| 2 | Batch index is **not** a happens-before proof: the DAG only contains edges for conflicting same-resource accesses and is discarded after `topo_batches`; e.g. test `independent_commands_share_a_batch...` produces batches `[0,2],[1]` with no edge `2→1` | `graph.rs:643-657, 683-697, 699-714` | +| 3 | Queue transfers unmodeled: `QueueTransfer` exists but every barrier sets `None`; no command→queue assignment exists anywhere (`derive_queue_capabilities` is a resource-wide stage union, not a per-command queue) | `graph.rs:41-58, 792, 822, 1461-1541` | +| 4 | No alias-safety barriers: `build_barriers` only tracks prior uses of the *same* resource; nothing orders last use of alias A before first use of alias B sharing memory | `graph.rs:728-835` | +| 5 | Packing uses `texture_size_estimate`/`desc.size`, ignoring real `VkMemoryRequirements`, alignment, memory-type bits, `bufferImageGranularity`, and dedicated-allocation requirements; offsets are never bound to resources | `graph.rs:888-919, 1543-1588` | +| 6 | No offset-aware bind path: `bind_buffer_memory`/`bind_image_memory` bind whole VMA allocations; `Allocation::Shared(Arc>)` exists but is unused for pooling | `device.rs:209-214, 766-778` | + +## Design decisions (post-review) + +### D1 — Happens-before, not batch numbers + +Retain the dependency adjacency from `GraphCompiler::dependencies()` in +`CompiledGraph` and add an explicit `happens_before(a, b)` reachability query. +Alias A→B is permitted only if *every* terminal use of A happens-before *every* +initial use of B. Batch intervals are computed (for reporting/limits) but never +used as the safety criterion. Same-batch uses always interfere. + +`happens_before` is computed as a **transitive closure at compile time**: one +bitset per command (`Vec`-style reachability rows, or bitset +columns if that packs better), built once in `compile()` from the Kahn +adjacency. Queries are then O(1) and deterministic. Rationale: the command +count is bounded by the recorded command list (thousands at most), closure is +O(V·E/64) once, and per-query BFS would be re-run per alias-candidate pair +during placement for no benefit. Note the closure's semantics: edges encode +only conflicting same-resource accesses, so reachability means exactly +"provably ordered via a resource-hazard chain" — precisely the aliasing-safety +criterion. Commands mutually unreachable (including same-batch pairs) interfere. + +### D2 — Phase split + +`compile()` stays device-free: dependency order, batch lifetimes, interference. +A new device-dependent `plan_placements(device)` (called from `resolve`) does +requirements queries, pool placement, alias-barrier construction, and binding. +Tests that call `bind_lifetimes()` directly are updated. + +### D3 — Conservative queue policy + +There is no command→queue assignment today and all commands record into one +command buffer. Aliasing is restricted to the single-queue case; any pair whose +accesses can't be proven ordered on one queue interferes. Cross-queue aliasing +(with ownership transfers/semaphores) is explicitly out of scope and documented +as a follow-up — no fake `QueueTransfer` support. + +### D4 — Alias synchronization done correctly + +- For aliased **buffers**: emit a `VkMemoryBarrier2` (new `MemoryBarrier` + + `PipelineBarrier.memory_barriers`) before the new occupant's *first-use + command* (a concrete command, not "batch start"), with `src` = old occupant's + aggregated terminal stage/access (writes; execution-only if read-only), + `dst` = new occupant's aggregated first-use scope. +- For aliased **images**: fold the old occupant's terminal scope into the new + image's first-use image barrier (`srcStageMask`/`srcAccessMask` = old scope, + `oldLayout = UNDEFINED`) — a bare memory barrier does not chain with a + following `srcStageMask=NONE` layout transition. +- Aggregate multi-scope accesses per command (OR all stage/access for the + resource at that command). +- Placement is range-based: a new resource's `[offset, offset+size)` is tested + against **all** occupied ranges; barriers include every old occupant + intersecting the range. + +### D5 — Vulkan aliasing correctness details + +- `bufferImageGranularity`: **separate pools per class** (linear-only and + optimal-only) is the primary strategy — first-fit stays simple and no + padding logic is needed. Intra-pool offsets are rounded only to each + resource's `alignment`. (Intra-pool interleaving with granularity-driven + offset rounding is a deliberate non-goal; revisit only if pool-class split + waste ever shows up in practice.) +- Set `VK_IMAGE_CREATE_ALIAS_BIT` on pooled images. +- Exclude from pools: `requiresDedicatedAllocation` resources, explicit + `AllocationStrategy::Dedicated` (incl. the 64 MiB heuristic), and + host-visible (`CpuToGpu`/`GpuToCpu`) resources — `Buffer::map()` returns + slices from the allocation base and would silently map wrong bytes unless + suballocation offsets are tracked. +- Created-but-unused resources: skip creation entirely (or allocate outside the + pool) — decided as "skip unused transients", with a test. +- Pooled images must have `GraphTextureDesc.layout == UNDEFINED`: + `plan_placements` rejects non-UNDEFINED declared initial layouts on pooled + images with a graph error (a created image's declared initial layout cannot + be honored by reused memory, whose contents are always undefined at first + use). Non-pooled images keep today's behavior. + +### D6 — Device/resources infra (new APIs, not existing ones) + +- `renderer_device`: `allocate_memory(requirements, …)` via + `vk_mem::Alloc::allocate_memory` with `CAN_ALIAS`; `find_memory_type_index`; + `bind_buffer_at_offset`/`bind_image_at_offset` via `bind_*_memory2` with + `allocation_local_offset`; require API ≥1.1 or enable the full + `get_memory_requirements2`/`bind_memory2`/`dedicated_allocation` extension + set with a clear error otherwise (same guard pattern as the existing + dedicated-allocation check in `device.rs`); define shared-allocation + ownership/cleanup: at bind time the group **transfers** the + `Arc>` into each bound resource's + `Allocation::Shared` and drops its own reference — the underlying memory is + freed when the last bound resource drops (via `DeviceObject` drop), matching + the existing `Owned`/`Shared` semantics. The group never retains a post-bind + handle. +- `renderer_resources`: symmetric `Buffer::new_unallocated` / + `Image::new_unallocated`, `memory_requirements()` (incl. + `VkMemoryDedicatedRequirements`), `bind_allocation(alloc, offset)` with + alignment/type-bits/bounds validation; a `memory` module with + `MemoryRequirements` + `MemoryGroup` (bind → `Allocation::Shared`). Pool + groups are keyed by a memory type that is in **every member's** + `memoryTypeBits` (intersection), not by any single member's individually + best type; a member whose bits exclude the group's type moves to another + group or its own allocation. New `renderer_resources::Error` variants for + failed requirement queries / bind-time validation (alignment, type-bits, + bounds); `renderer_rendergraph` maps these into new `GraphError` variants + (e.g. `PlacementFailed(ResourceId)`, `BindFailed(ResourceId)`) — bind/placement + failures surface through `resolve`/`plan_placements`'s existing + `renderer_resources::Result` boundary and map at the call site. No graph + types leak into this crate. + +## Task breakdown (ordered, dependency-explicit) + +1. **`renderer_device`**: memory-type query, aliasable shared allocation, + offset-aware bind, API/extension guards, error variants, ownership docs. + Tests with `TestVulkan` (`--features testing`). +2. **`renderer_resources`**: unbound create + requirements + bind APIs for + buffer *and* image; `memory` module (`MemoryRequirements`, `MemoryGroup`); + update crate AGENTS.md. Existing `Buffer::new`/`Image::new` unchanged. +3. **`renderer_rendergraph` (analysis)**: retain dependency DAG in + `CompiledGraph`; `happens_before` query; batch-mapped + `ResourceLifetime {first_batch, last_batch}`; replace `memory_offset: u64` + with a typed `Placement {offset, size, alignment, pool, aliased}`; imported + resources tracked but excluded from packing; queue-conservatism rule per D3. +4. **`renderer_rendergraph` (synchronization)**: `MemoryBarrier` + + `PipelineBarrier.memory_barriers`; recorder emits `vk::MemoryBarrier2` in + `DependencyInfo` (anchored via the existing `recording_steps` + `before_command` mechanism, like today's image/barrier steps); alias-barrier + construction per D4 (buffer and image variants, aggregation, + first-use-command anchoring); new `GraphError` variants for placement/bind + failures per D6, mapped from `renderer_resources::Error` at the + `resolve` boundary. +5. **`renderer_rendergraph` (placement)**: `plan_placements(device)` **replaces + `resolve()`'s current eager create+allocate loop** (graph.rs:1218-1276) for + pooled resources — create unbound, query requirements, group by a memory + type common to every member's `memoryTypeBits`, granularity-aware first-fit + over ranges (D4/D5, pool-per-class), allocate groups, bind, store + placements in `CompiledGraph`. Non-poolable resources (dedicated, + host-visible, imports) keep an allocate-and-bind-whole path equivalent to + today's. "Skip unused transients" lands here: created resources absent from + `lifetimes` are never created (no behavior change needed in + `record_commands`, which already skips ids absent from `lifetimes` at + graph.rs:1290). Transactional and idempotent; remove + `texture_size_estimate` from allocation decisions. +6. **Docs/tests**: update stale TODOs (`graph.rs:882`, `1315-1319`), document + the single-queue submission contract and the future multi-queue invariant + (alias transitions become semaphore edges). + +## Verification + +- Device-free unit tests: same-batch interference; unordered-but-distinct-batch + interference; proven-ordered reuse succeeds; granularity boundary (buffer + next to optimal image); range-overlap first-fit; dedicated/host-visible + exclusion; unused-transient skip; pooled-image non-UNDEFINED initial layout + rejected; alias-barrier scope correctness incl. read-only old occupant and + image `UNDEFINED` transition chaining. +- GPU-gated (`tests/graph_validation.rs`): buffer-buffer and image-image alias + reuse through the real barrier path, submit, `device_wait_idle`, + `assert_no_validation_errors`; teardown of shared allocation after all + resources drop. +- Commands: `cargo check --workspace`, `cargo test -p renderer_rendergraph`, + `cargo test -p renderer_resources --features testing`, + `cargo test -p renderer_device --features testing` (from workspace root, + pinned toolchain). + +## Review history + +- Drafted independently by deepseek/planner and luna/planner; cross-reviewed by + deepseek/reviewer and luna/reviewer (both verdicts: CHANGES REQUIRED). +- Blockers folded into this revision: batch-index ≠ happens-before (D1), + compile/resolve phase split (D2), `bufferImageGranularity` handling (D5), + alias-barrier/layout-transition chaining (D4), offset-aware bind APIs are new + work not existing capability (D6), queue model restricted to single-queue + (D3). +- Rev 3: diagnosis re-verified against code and `vk-mem 0.5.0` sources + (all six rows and the D6 API names confirmed). Follow-up comments folded in: + `happens_before` = compile-time transitive closure (D1), pool-per-class + granularity strategy picked (D5), pooled-image UNDEFINED initial-layout + rule (D5), pool groups keyed by intersecting `memoryTypeBits` (D6), + `MemoryGroup` transfers the `Arc` at bind time instead of retaining it (D6), + placement/bind error surfacing through `GraphError` (D6/task 4), + `plan_placements` explicitly replaces `resolve`'s eager loop (task 5).