rendergraph: correct transient-resource aliasing with pooled placement
Lifetimes derive from the real dependency ordering (retained DAG + compile-time happens-before closure), not command-index intervals. Aliasing is restricted to the single-queue contract; alias reuse is synchronized with memory/image barriers anchored at first-use commands. - renderer_device: aliasable shared allocations (CAN_ALIAS), memory- requirements queries incl. dedicated hints, offset-aware bind_buffer/image_memory2, grouped bind under one allocator lock - renderer_resources: unallocated resource creation, requirements cached in Allocation::Unmanaged (OnceLock), MemoryGroup pooling with intersecting memoryTypeBits, bind-time validation - renderer_rendergraph: retained dependency DAG, happens_before closure, batch lifetimes, Placement type, plan_placements with pooled aliasing (first-fit over offset-sorted ranges, pool classes split by bufferImageGranularity), alias-barrier construction, placement/bind error variants - perf: quadratic compile/resolve paths eliminated (per-resource indexing, frontier caching, backward-scan prune, cached recording index, recorder scratch reuse); single requirements query per resource through plan/validate/bind - design plan + GPU-gated validation tests
This commit is contained in:
parent
e116851b90
commit
5e4bc0ccf0
|
|
@ -49,6 +49,7 @@ arrayvec = "0.7.6"
|
||||||
tinyvec = "1.8"
|
tinyvec = "1.8"
|
||||||
indexmap = "2"
|
indexmap = "2"
|
||||||
petgraph = "0.7"
|
petgraph = "0.7"
|
||||||
|
fixedbitset = "0.5"
|
||||||
# itertools = "0.14.0"
|
# itertools = "0.14.0"
|
||||||
ahash = "0.8"
|
ahash = "0.8"
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -123,7 +123,7 @@
|
||||||
- The `egui` pre-pass currently packs texture updates into a staging image/buffer and registers graph work; texture-free deltas and upload ordering deserve explicit coverage when changing the integration.
|
- The `egui` pre-pass currently packs texture updates into a staging image/buffer and registers graph work; texture-free deltas and upload ordering deserve explicit coverage when changing the integration.
|
||||||
- The private font-atlas image is CPU memory, not a Vulkan `images::Image`; do not conflate the similarly named `Image` types when editing `text.rs`.
|
- The private font-atlas image is CPU memory, not a Vulkan `images::Image`; do not conflate the similarly named `Image` types when editing `text.rs`.
|
||||||
- `weak_vec` moved to `renderer_resources::util` and is not re-exported by `renderer`; it prunes dead weak references on capacity pressure. Image-side view tracking should remain weak to avoid ownership cycles.
|
- `weak_vec` moved to `renderer_resources::util` and is not re-exported by `renderer`; it prunes dead weak references on capacity pressure. Image-side view tracking should remain weak to avoid ownership cycles.
|
||||||
- Render graph IDs are distinct: legacy `GraphResourceId` wraps `u32`; the newer internal `resources::ResourceId` wraps `u64`.
|
- Render graph IDs are distinct: legacy `GraphResourceId` wraps `u32`; the newer internal `resources::ResourceId` also wraps `u32` (constructed via `ResourceId::from_u32`).
|
||||||
- Public `commands::traits::CommandBufferExt` contains raw recording convenience methods; those methods assume valid command-buffer state and do not all enforce it at runtime.
|
- Public `commands::traits::CommandBufferExt` contains raw recording convenience methods; those methods assume valid command-buffer state and do not all enforce it at runtime.
|
||||||
- `SwapchainConfiguration.image_count` is treated as the additional desired in-flight count during validation, while creation asks for minimum surface image count plus that configured value. The fallback-configuration helper assigns a total count.
|
- `SwapchainConfiguration.image_count` is treated as the additional desired in-flight count during validation, while creation asks for minimum surface image count plus that configured value. The fallback-configuration helper assigns a total count.
|
||||||
- The supported surface capability extension query can filter present modes based on the selected mode; its behavior depends on optional `InstanceExtensions` support.
|
- The supported surface capability extension query can filter present modes based on the selected mode; its behavior depends on optional `InstanceExtensions` support.
|
||||||
|
|
|
||||||
|
|
@ -228,7 +228,17 @@ mod impls {
|
||||||
|
|
||||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
|
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
|
||||||
#[repr(transparent)]
|
#[repr(transparent)]
|
||||||
pub struct ResourceId(pub u64);
|
pub struct ResourceId(u32);
|
||||||
|
|
||||||
|
impl ResourceId {
|
||||||
|
pub fn from_u32(index: u32) -> Self {
|
||||||
|
Self(index)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn index(self) -> usize {
|
||||||
|
self.0 as usize
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
#[derive(Debug, Clone, Copy)]
|
#[derive(Debug, Clone, Copy)]
|
||||||
pub enum ResourceRange {
|
pub enum ResourceRange {
|
||||||
|
|
|
||||||
|
|
@ -4,7 +4,7 @@ use std::{
|
||||||
ffi::CStr,
|
ffi::CStr,
|
||||||
ops::{Deref, DerefMut},
|
ops::{Deref, DerefMut},
|
||||||
ptr::NonNull,
|
ptr::NonNull,
|
||||||
sync::{Arc, Weak},
|
sync::{Arc, OnceLock, Weak},
|
||||||
};
|
};
|
||||||
|
|
||||||
use ash::{ext, khr, prelude::VkResult, vk};
|
use ash::{ext, khr, prelude::VkResult, vk};
|
||||||
|
|
@ -206,11 +206,78 @@ pub struct AllocationDesc<'a> {
|
||||||
pub strategy: AllocationStrategy,
|
pub strategy: AllocationStrategy,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Vulkan memory requirements for a buffer or image.
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||||
|
pub struct MemoryRequirements {
|
||||||
|
pub size: vk::DeviceSize,
|
||||||
|
pub alignment: vk::DeviceSize,
|
||||||
|
pub memory_type_bits: u32,
|
||||||
|
pub requires_dedicated_allocation: bool,
|
||||||
|
pub prefers_dedicated_allocation: bool,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A buffer or image to bind into a shared allocation at an allocation-local offset.
|
||||||
|
#[derive(Debug, Clone, Copy)]
|
||||||
|
pub enum MemoryBinding {
|
||||||
|
Buffer {
|
||||||
|
buffer: vk::Buffer,
|
||||||
|
offset: vk::DeviceSize,
|
||||||
|
requirements: MemoryRequirements,
|
||||||
|
},
|
||||||
|
Image {
|
||||||
|
image: vk::Image,
|
||||||
|
offset: vk::DeviceSize,
|
||||||
|
requirements: MemoryRequirements,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A failed grouped bind and the number of earlier members that remain bound.
|
||||||
|
#[derive(Debug)]
|
||||||
|
pub struct GroupBindError {
|
||||||
|
pub error: crate::Error,
|
||||||
|
pub bound_count: usize,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the first memory type allowed by `memory_type_bits` with all requested properties.
|
||||||
|
fn find_memory_type_index(
|
||||||
|
memory_properties: &vk::PhysicalDeviceMemoryProperties,
|
||||||
|
memory_type_bits: u32,
|
||||||
|
property_flags: vk::MemoryPropertyFlags,
|
||||||
|
) -> Result<u32> {
|
||||||
|
(0..memory_properties.memory_type_count)
|
||||||
|
.find(|&index| {
|
||||||
|
memory_type_bits & (1 << index) != 0
|
||||||
|
&& memory_properties.memory_types[index as usize]
|
||||||
|
.property_flags
|
||||||
|
.contains(property_flags)
|
||||||
|
})
|
||||||
|
.ok_or(Error::NoCompatibleMemoryType {
|
||||||
|
memory_type_bits,
|
||||||
|
property_flags,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
impl From<vk::MemoryRequirements> for MemoryRequirements {
|
||||||
|
fn from(requirements: vk::MemoryRequirements) -> Self {
|
||||||
|
Self {
|
||||||
|
size: requirements.size,
|
||||||
|
alignment: requirements.alignment,
|
||||||
|
memory_type_bits: requirements.memory_type_bits,
|
||||||
|
requires_dedicated_allocation: false,
|
||||||
|
prefers_dedicated_allocation: false,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
#[derive(Debug)]
|
#[derive(Debug)]
|
||||||
pub enum Allocation {
|
pub enum Allocation {
|
||||||
Owned(DeviceObject<GpuAllocation>),
|
Owned(DeviceObject<GpuAllocation>),
|
||||||
Shared(Arc<DeviceObject<GpuAllocation>>),
|
Shared(Arc<DeviceObject<GpuAllocation>>),
|
||||||
Unmanaged,
|
/// An unbound resource. Requirements are a pure function of its Vulkan handle, so
|
||||||
|
/// resources can safely cache the first query here for their lifetime.
|
||||||
|
Unmanaged {
|
||||||
|
requirements: OnceLock<MemoryRequirements>,
|
||||||
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
impl Allocation {
|
impl Allocation {
|
||||||
|
|
@ -218,7 +285,7 @@ impl Allocation {
|
||||||
match self {
|
match self {
|
||||||
Allocation::Owned(obj) => Some(obj),
|
Allocation::Owned(obj) => Some(obj),
|
||||||
Allocation::Shared(arc) => Some(arc.as_ref()),
|
Allocation::Shared(arc) => Some(arc.as_ref()),
|
||||||
Allocation::Unmanaged => None,
|
Allocation::Unmanaged { .. } => None,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -226,7 +293,7 @@ impl Allocation {
|
||||||
match self {
|
match self {
|
||||||
Allocation::Owned(obj) => Some(obj),
|
Allocation::Owned(obj) => Some(obj),
|
||||||
Allocation::Shared(arc) => Arc::get_mut(arc).map(DerefMut::deref_mut),
|
Allocation::Shared(arc) => Arc::get_mut(arc).map(DerefMut::deref_mut),
|
||||||
Allocation::Unmanaged => None,
|
Allocation::Unmanaged { .. } => None,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -524,11 +591,24 @@ impl PhysicalDeviceInfo {
|
||||||
|
|
||||||
let mut allocator_info =
|
let mut allocator_info =
|
||||||
vk_mem::AllocatorCreateInfo::new(&instance.inner.raw, &device, self.pdev);
|
vk_mem::AllocatorCreateInfo::new(&instance.inner.raw, &device, self.pdev);
|
||||||
|
|
||||||
allocator_info.preferred_large_heap_block_size = 64 * 1024 * 1024;
|
allocator_info.preferred_large_heap_block_size = 64 * 1024 * 1024;
|
||||||
allocator_info.vulkan_api_version = instance
|
allocator_info.vulkan_api_version = instance
|
||||||
.inner
|
.inner
|
||||||
.api_version
|
.api_version
|
||||||
.min(self.properties.device_api_version);
|
.min(self.properties.device_api_version);
|
||||||
|
|
||||||
|
if allocator_info.vulkan_api_version < vk::API_VERSION_1_1 {
|
||||||
|
if enabled_extensions.contains(&khr::dedicated_allocation::NAME)
|
||||||
|
&& enabled_extensions.contains(&khr::get_memory_requirements2::NAME)
|
||||||
|
{
|
||||||
|
allocator_info.flags |= vk_mem::AllocatorCreateFlags::KHR_DEDICATED_ALLOCATION;
|
||||||
|
}
|
||||||
|
if enabled_extensions.contains(&khr::bind_memory2::NAME) {
|
||||||
|
allocator_info.flags |= vk_mem::AllocatorCreateFlags::KHR_BIND_MEMORY2;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
let allocator = unsafe { vk_mem::Allocator::new(allocator_info) }?;
|
let allocator = unsafe { vk_mem::Allocator::new(allocator_info) }?;
|
||||||
|
|
||||||
let pipeline_cache = PipelineCache::new(&device, &self).unwrap();
|
let pipeline_cache = PipelineCache::new(&device, &self).unwrap();
|
||||||
|
|
@ -656,7 +736,366 @@ where
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
impl Device {
|
||||||
|
/// Allocates aliasable memory compatible with `requirements` and `property_flags`.
|
||||||
|
///
|
||||||
|
/// The returned `Arc` is an allocation handle intended to be transferred into each bound
|
||||||
|
/// resource's `Allocation::Shared`. Binding consumes one Arc reference; the allocation is
|
||||||
|
/// freed by its `DeviceObject` when the last bound resource drops. This helper does not
|
||||||
|
/// retain a reference after a successful bind.
|
||||||
|
pub fn allocate_memory(
|
||||||
|
&self,
|
||||||
|
requirements: &MemoryRequirements,
|
||||||
|
property_flags: vk::MemoryPropertyFlags,
|
||||||
|
) -> Result<Arc<DeviceObject<GpuAllocation>>> {
|
||||||
|
let allocation = self
|
||||||
|
.0
|
||||||
|
.allocate_aliasable_memory(requirements, property_flags)?;
|
||||||
|
Ok(Arc::new(DeviceObject::new(self.clone(), allocation)))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
impl DeviceInner {
|
impl DeviceInner {
|
||||||
|
fn api_version(&self) -> u32 {
|
||||||
|
self.instance
|
||||||
|
.inner
|
||||||
|
.api_version
|
||||||
|
.min(self.adapter.properties.device_api_version)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn has_enabled_extension(&self, name: &CStr) -> bool {
|
||||||
|
self.enabled_extensions.contains(&name)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn has_memory2_extension_set(&self) -> bool {
|
||||||
|
self.has_enabled_extension(khr::get_memory_requirements2::NAME)
|
||||||
|
&& self.has_enabled_extension(khr::bind_memory2::NAME)
|
||||||
|
&& self.has_enabled_extension(khr::dedicated_allocation::NAME)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn require_offset_binding_support(&self) -> Result<()> {
|
||||||
|
if self.api_version() < vk::API_VERSION_1_1 && !self.has_memory2_extension_set() {
|
||||||
|
return Err(Error::OffsetBindingRequiresApi11);
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Finds a compatible memory type satisfying `property_flags`.
|
||||||
|
pub fn find_memory_type_index(
|
||||||
|
&self,
|
||||||
|
requirements: &MemoryRequirements,
|
||||||
|
property_flags: vk::MemoryPropertyFlags,
|
||||||
|
) -> Result<u32> {
|
||||||
|
find_memory_type_index(
|
||||||
|
&self.adapter.properties.memory_properties,
|
||||||
|
requirements.memory_type_bits,
|
||||||
|
property_flags,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Queries buffer memory requirements, including dedicated-allocation hints when available.
|
||||||
|
pub fn buffer_memory_requirements(&self, buffer: vk::Buffer) -> Result<MemoryRequirements> {
|
||||||
|
let api11 = self.api_version() >= vk::API_VERSION_1_1;
|
||||||
|
|
||||||
|
let get_requirements2 =
|
||||||
|
api11 || self.has_enabled_extension(khr::get_memory_requirements2::NAME);
|
||||||
|
let dedicated_available =
|
||||||
|
api11 || self.has_enabled_extension(khr::dedicated_allocation::NAME);
|
||||||
|
|
||||||
|
if !get_requirements2 {
|
||||||
|
let requirements = unsafe { self.raw.get_buffer_memory_requirements(buffer) };
|
||||||
|
return Ok(requirements.into());
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut dedicated = vk::MemoryDedicatedRequirements::default();
|
||||||
|
let mut requirements = vk::MemoryRequirements2::default();
|
||||||
|
|
||||||
|
if dedicated_available {
|
||||||
|
requirements = requirements.push_next(&mut dedicated);
|
||||||
|
}
|
||||||
|
|
||||||
|
let info = vk::BufferMemoryRequirementsInfo2::default().buffer(buffer);
|
||||||
|
unsafe {
|
||||||
|
if api11 {
|
||||||
|
self.raw
|
||||||
|
.get_buffer_memory_requirements2(&info, &mut requirements);
|
||||||
|
} else {
|
||||||
|
khr::get_memory_requirements2::Device::new(&self.instance.inner.raw, &self.raw)
|
||||||
|
.get_buffer_memory_requirements2(&info, &mut requirements);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(MemoryRequirements {
|
||||||
|
size: requirements.memory_requirements.size,
|
||||||
|
alignment: requirements.memory_requirements.alignment,
|
||||||
|
memory_type_bits: requirements.memory_requirements.memory_type_bits,
|
||||||
|
requires_dedicated_allocation: dedicated_available
|
||||||
|
&& dedicated.requires_dedicated_allocation == vk::TRUE,
|
||||||
|
prefers_dedicated_allocation: dedicated_available
|
||||||
|
&& dedicated.prefers_dedicated_allocation == vk::TRUE,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Queries image memory requirements, including dedicated-allocation hints when available.
|
||||||
|
pub fn image_memory_requirements(&self, image: vk::Image) -> Result<MemoryRequirements> {
|
||||||
|
let api11 = self.api_version() >= vk::API_VERSION_1_1;
|
||||||
|
|
||||||
|
let get_requirements2 =
|
||||||
|
api11 || self.has_enabled_extension(khr::get_memory_requirements2::NAME);
|
||||||
|
let dedicated_available =
|
||||||
|
api11 || self.has_enabled_extension(khr::dedicated_allocation::NAME);
|
||||||
|
|
||||||
|
if !get_requirements2 {
|
||||||
|
let requirements = unsafe { self.raw.get_image_memory_requirements(image) };
|
||||||
|
return Ok(requirements.into());
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut dedicated = vk::MemoryDedicatedRequirements::default();
|
||||||
|
let mut requirements = vk::MemoryRequirements2::default();
|
||||||
|
if dedicated_available {
|
||||||
|
requirements = requirements.push_next(&mut dedicated);
|
||||||
|
}
|
||||||
|
|
||||||
|
let info = vk::ImageMemoryRequirementsInfo2::default().image(image);
|
||||||
|
|
||||||
|
unsafe {
|
||||||
|
if api11 {
|
||||||
|
self.raw
|
||||||
|
.get_image_memory_requirements2(&info, &mut requirements);
|
||||||
|
} else {
|
||||||
|
khr::get_memory_requirements2::Device::new(&self.instance.inner.raw, &self.raw)
|
||||||
|
.get_image_memory_requirements2(&info, &mut requirements);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(MemoryRequirements {
|
||||||
|
size: requirements.memory_requirements.size,
|
||||||
|
alignment: requirements.memory_requirements.alignment,
|
||||||
|
memory_type_bits: requirements.memory_requirements.memory_type_bits,
|
||||||
|
requires_dedicated_allocation: dedicated_available
|
||||||
|
&& dedicated.requires_dedicated_allocation == vk::TRUE,
|
||||||
|
prefers_dedicated_allocation: dedicated_available
|
||||||
|
&& dedicated.prefers_dedicated_allocation == vk::TRUE,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Allocates aliasable memory compatible with `requirements` and `property_flags`.
|
||||||
|
///
|
||||||
|
/// Resources that require a dedicated allocation cannot use this resource-independent path.
|
||||||
|
fn allocate_aliasable_memory(
|
||||||
|
&self,
|
||||||
|
requirements: &MemoryRequirements,
|
||||||
|
property_flags: vk::MemoryPropertyFlags,
|
||||||
|
) -> Result<GpuAllocation> {
|
||||||
|
use vk_mem::{Alloc as _, AllocationCreateFlags, AllocationCreateInfo};
|
||||||
|
|
||||||
|
if requirements.requires_dedicated_allocation {
|
||||||
|
return Err(Error::DedicatedAllocationRequiresResource);
|
||||||
|
}
|
||||||
|
|
||||||
|
let memory_type_index = self.find_memory_type_index(requirements, property_flags)?;
|
||||||
|
let create_info = AllocationCreateInfo {
|
||||||
|
flags: AllocationCreateFlags::CAN_ALIAS,
|
||||||
|
required_flags: property_flags,
|
||||||
|
memory_type_bits: 1 << memory_type_index,
|
||||||
|
..Default::default()
|
||||||
|
};
|
||||||
|
|
||||||
|
let raw_requirements = vk::MemoryRequirements {
|
||||||
|
size: requirements.size,
|
||||||
|
alignment: requirements.alignment,
|
||||||
|
memory_type_bits: requirements.memory_type_bits,
|
||||||
|
};
|
||||||
|
|
||||||
|
let allocator = self.allocator.lock();
|
||||||
|
let raw = unsafe { allocator.allocate_memory(&raw_requirements, &create_info) }?;
|
||||||
|
|
||||||
|
let allocation_info = allocator.get_allocation_info2(&raw);
|
||||||
|
let dedicated_memory = allocation_info.dedicated_memory;
|
||||||
|
let allocation_info = allocation_info.allocation_info;
|
||||||
|
let mapped = NonNull::new(allocation_info.mapped_data.cast::<u8>()).map(SendSyncPtr);
|
||||||
|
|
||||||
|
let allocation = GpuAllocation {
|
||||||
|
raw,
|
||||||
|
memory: allocation_info.device_memory,
|
||||||
|
offset: allocation_info.offset,
|
||||||
|
size: allocation_info.size,
|
||||||
|
memory_type_index: allocation_info.memory_type,
|
||||||
|
mapped,
|
||||||
|
dedicated_memory,
|
||||||
|
};
|
||||||
|
Ok(allocation)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Binds a buffer to aliasable memory at an allocation-local offset.
|
||||||
|
///
|
||||||
|
/// On success, ownership of `allocation` transfers to the returned `Allocation::Shared`;
|
||||||
|
/// store it in the bound resource and drop any caller-side references. The allocation is
|
||||||
|
/// freed when the last bound resource drops. The resource must be destroyed before its
|
||||||
|
/// allocation ownership is dropped. `requirements` must be the buffer's own requirements,
|
||||||
|
/// as returned by the buffer requirements query. If the supplied requirements are wrong,
|
||||||
|
/// the Vulkan bind (`vkBindBufferMemory2` via VMA) remains the authoritative validation.
|
||||||
|
/// Calls are externally synchronized by VMA.
|
||||||
|
pub fn bind_buffer_at_offset(
|
||||||
|
&self,
|
||||||
|
buffer: vk::Buffer,
|
||||||
|
requirements: &MemoryRequirements,
|
||||||
|
allocation: Arc<DeviceObject<GpuAllocation>>,
|
||||||
|
local_offset: vk::DeviceSize,
|
||||||
|
) -> Result<Allocation> {
|
||||||
|
self.require_offset_binding_support()?;
|
||||||
|
self.validate_shared_binding(&allocation, local_offset, *requirements)?;
|
||||||
|
|
||||||
|
unsafe {
|
||||||
|
self.allocator.lock().bind_buffer_memory2(
|
||||||
|
&allocation.raw,
|
||||||
|
local_offset,
|
||||||
|
buffer,
|
||||||
|
core::ptr::null(),
|
||||||
|
)?;
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(Allocation::Shared(allocation))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Binds an image to aliasable memory at an allocation-local offset.
|
||||||
|
///
|
||||||
|
/// On success, ownership of `allocation` transfers to the returned `Allocation::Shared`;
|
||||||
|
/// store it in the bound resource and drop any caller-side references. The allocation is
|
||||||
|
/// freed when the last bound resource drops. The resource must be destroyed before its
|
||||||
|
/// allocation ownership is dropped. `requirements` must be the image's own requirements,
|
||||||
|
/// as returned by the image requirements query. If the supplied requirements are wrong,
|
||||||
|
/// the Vulkan bind (`vkBindImageMemory2` via VMA) remains the authoritative validation.
|
||||||
|
/// Calls are externally synchronized by VMA.
|
||||||
|
pub fn bind_image_at_offset(
|
||||||
|
&self,
|
||||||
|
image: vk::Image,
|
||||||
|
requirements: &MemoryRequirements,
|
||||||
|
allocation: Arc<DeviceObject<GpuAllocation>>,
|
||||||
|
local_offset: vk::DeviceSize,
|
||||||
|
) -> Result<Allocation> {
|
||||||
|
self.require_offset_binding_support()?;
|
||||||
|
self.validate_shared_binding(&allocation, local_offset, *requirements)?;
|
||||||
|
|
||||||
|
unsafe {
|
||||||
|
self.allocator.lock().bind_image_memory2(
|
||||||
|
&allocation.raw,
|
||||||
|
local_offset,
|
||||||
|
image,
|
||||||
|
core::ptr::null(),
|
||||||
|
)?;
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(Allocation::Shared(allocation))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Binds a group of buffers and images into one aliasable allocation.
|
||||||
|
///
|
||||||
|
/// Every member is validated before any Vulkan bind is attempted. Binds then run in slice
|
||||||
|
/// order under one allocator lock; if a bind fails, the first error is returned and earlier
|
||||||
|
/// successful binds remain bound, matching repeated single-resource binds. `bound_count`
|
||||||
|
/// identifies that successfully bound prefix so callers can retain the allocation for it.
|
||||||
|
pub fn bind_group_at_offset(
|
||||||
|
&self,
|
||||||
|
allocation: &Arc<DeviceObject<GpuAllocation>>,
|
||||||
|
bindings: &[MemoryBinding],
|
||||||
|
) -> core::result::Result<(), GroupBindError> {
|
||||||
|
self.require_offset_binding_support()
|
||||||
|
.map_err(|error| GroupBindError {
|
||||||
|
error,
|
||||||
|
bound_count: 0,
|
||||||
|
})?;
|
||||||
|
|
||||||
|
for binding in bindings {
|
||||||
|
let (offset, requirements) = match binding {
|
||||||
|
MemoryBinding::Buffer {
|
||||||
|
offset,
|
||||||
|
requirements,
|
||||||
|
..
|
||||||
|
}
|
||||||
|
| MemoryBinding::Image {
|
||||||
|
offset,
|
||||||
|
requirements,
|
||||||
|
..
|
||||||
|
} => (*offset, *requirements),
|
||||||
|
};
|
||||||
|
self.validate_shared_binding(allocation, offset, requirements)
|
||||||
|
.map_err(|error| GroupBindError {
|
||||||
|
error,
|
||||||
|
bound_count: 0,
|
||||||
|
})?;
|
||||||
|
}
|
||||||
|
|
||||||
|
let allocator = self.allocator.lock();
|
||||||
|
for (bound_count, binding) in bindings.iter().enumerate() {
|
||||||
|
unsafe {
|
||||||
|
match binding {
|
||||||
|
MemoryBinding::Buffer { buffer, offset, .. } => {
|
||||||
|
allocator
|
||||||
|
.bind_buffer_memory2(
|
||||||
|
&allocation.raw,
|
||||||
|
*offset,
|
||||||
|
*buffer,
|
||||||
|
core::ptr::null(),
|
||||||
|
)
|
||||||
|
.map_err(|error| GroupBindError {
|
||||||
|
error: error.into(),
|
||||||
|
bound_count,
|
||||||
|
})?;
|
||||||
|
}
|
||||||
|
MemoryBinding::Image { image, offset, .. } => {
|
||||||
|
allocator
|
||||||
|
.bind_image_memory2(&allocation.raw, *offset, *image, core::ptr::null())
|
||||||
|
.map_err(|error| GroupBindError {
|
||||||
|
error: error.into(),
|
||||||
|
bound_count,
|
||||||
|
})?;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
fn validate_shared_binding(
|
||||||
|
&self,
|
||||||
|
allocation: &DeviceObject<GpuAllocation>,
|
||||||
|
local_offset: vk::DeviceSize,
|
||||||
|
requirements: MemoryRequirements,
|
||||||
|
) -> Result<()> {
|
||||||
|
if !core::ptr::eq(Arc::as_ptr(&allocation.device().0), self) {
|
||||||
|
return Err(Error::AllocationDeviceMismatch);
|
||||||
|
}
|
||||||
|
|
||||||
|
if requirements.requires_dedicated_allocation {
|
||||||
|
return Err(Error::DedicatedAllocationRequiresResource);
|
||||||
|
}
|
||||||
|
|
||||||
|
let gpu_allocation: &GpuAllocation = allocation;
|
||||||
|
if requirements.memory_type_bits & (1 << gpu_allocation.memory_type_index()) == 0 {
|
||||||
|
return Err(Error::IncompatibleAllocationMemoryType {
|
||||||
|
memory_type_index: gpu_allocation.memory_type_index(),
|
||||||
|
memory_type_bits: requirements.memory_type_bits,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
let end = local_offset.checked_add(requirements.size);
|
||||||
|
if requirements.alignment == 0
|
||||||
|
|| !local_offset.is_multiple_of(requirements.alignment)
|
||||||
|
|| end.is_none_or(|end| end > gpu_allocation.size())
|
||||||
|
{
|
||||||
|
return Err(Error::InvalidAllocationOffset {
|
||||||
|
offset: local_offset,
|
||||||
|
alignment: requirements.alignment,
|
||||||
|
required_size: requirements.size,
|
||||||
|
allocation_size: gpu_allocation.size(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
fn allocation_create_info(
|
fn allocation_create_info(
|
||||||
&self,
|
&self,
|
||||||
desc: &AllocationDesc<'_>,
|
desc: &AllocationDesc<'_>,
|
||||||
|
|
@ -679,6 +1118,7 @@ impl DeviceInner {
|
||||||
MemoryLocation::CpuToGpu | MemoryLocation::GpuToCpu
|
MemoryLocation::CpuToGpu | MemoryLocation::GpuToCpu
|
||||||
) {
|
) {
|
||||||
let memory_properties = &self.adapter.properties.memory_properties;
|
let memory_properties = &self.adapter.properties.memory_properties;
|
||||||
|
|
||||||
let coherent_host_types = memory_properties
|
let coherent_host_types = memory_properties
|
||||||
.memory_types
|
.memory_types
|
||||||
.iter()
|
.iter()
|
||||||
|
|
@ -691,9 +1131,11 @@ impl DeviceInner {
|
||||||
)
|
)
|
||||||
})
|
})
|
||||||
.fold(0_u32, |bits, (index, _)| bits | (1 << index));
|
.fold(0_u32, |bits, (index, _)| bits | (1 << index));
|
||||||
|
|
||||||
if coherent_host_types == 0 {
|
if coherent_host_types == 0 {
|
||||||
return Err(vk::Result::ERROR_FEATURE_NOT_PRESENT.into());
|
return Err(vk::Result::ERROR_FEATURE_NOT_PRESENT.into());
|
||||||
}
|
}
|
||||||
|
|
||||||
info.memory_type_bits = coherent_host_types;
|
info.memory_type_bits = coherent_host_types;
|
||||||
info.flags |= AllocationCreateFlags::MAPPED;
|
info.flags |= AllocationCreateFlags::MAPPED;
|
||||||
}
|
}
|
||||||
|
|
@ -703,9 +1145,11 @@ impl DeviceInner {
|
||||||
.inner
|
.inner
|
||||||
.api_version
|
.api_version
|
||||||
.min(self.adapter.properties.device_api_version);
|
.min(self.adapter.properties.device_api_version);
|
||||||
|
|
||||||
if desc.strategy == AllocationStrategy::Dedicated && api_version < vk::API_VERSION_1_1 {
|
if desc.strategy == AllocationStrategy::Dedicated && api_version < vk::API_VERSION_1_1 {
|
||||||
return Err(Error::DedicatedAllocationRequiresApi11);
|
return Err(Error::DedicatedAllocationRequiresApi11);
|
||||||
}
|
}
|
||||||
|
|
||||||
if desc.strategy == AllocationStrategy::Dedicated {
|
if desc.strategy == AllocationStrategy::Dedicated {
|
||||||
info.flags |= AllocationCreateFlags::DEDICATED_MEMORY;
|
info.flags |= AllocationCreateFlags::DEDICATED_MEMORY;
|
||||||
}
|
}
|
||||||
|
|
@ -722,11 +1166,13 @@ impl DeviceInner {
|
||||||
|
|
||||||
let create_info = self.allocation_create_info(&desc)?;
|
let create_info = self.allocation_create_info(&desc)?;
|
||||||
let allocator = self.allocator.lock();
|
let allocator = self.allocator.lock();
|
||||||
|
|
||||||
let raw = unsafe { allocator.allocate_memory_for_buffer(buffer, &create_info) }?;
|
let raw = unsafe { allocator.allocate_memory_for_buffer(buffer, &create_info) }?;
|
||||||
let allocation_info = allocator.get_allocation_info2(&raw);
|
let allocation_info = allocator.get_allocation_info2(&raw);
|
||||||
let dedicated_memory = allocation_info.dedicated_memory;
|
let dedicated_memory = allocation_info.dedicated_memory;
|
||||||
let allocation_info = allocation_info.allocation_info;
|
let allocation_info = allocation_info.allocation_info;
|
||||||
let mapped = NonNull::new(allocation_info.mapped_data.cast::<u8>()).map(SendSyncPtr);
|
let mapped = NonNull::new(allocation_info.mapped_data.cast::<u8>()).map(SendSyncPtr);
|
||||||
|
|
||||||
Ok(GpuAllocation {
|
Ok(GpuAllocation {
|
||||||
raw,
|
raw,
|
||||||
memory: allocation_info.device_memory,
|
memory: allocation_info.device_memory,
|
||||||
|
|
@ -747,11 +1193,13 @@ impl DeviceInner {
|
||||||
|
|
||||||
let create_info = self.allocation_create_info(&desc)?;
|
let create_info = self.allocation_create_info(&desc)?;
|
||||||
let allocator = self.allocator.lock();
|
let allocator = self.allocator.lock();
|
||||||
|
|
||||||
let raw = unsafe { allocator.allocate_memory_for_image(image, &create_info) }?;
|
let raw = unsafe { allocator.allocate_memory_for_image(image, &create_info) }?;
|
||||||
let allocation_info = allocator.get_allocation_info2(&raw);
|
let allocation_info = allocator.get_allocation_info2(&raw);
|
||||||
let dedicated_memory = allocation_info.dedicated_memory;
|
let dedicated_memory = allocation_info.dedicated_memory;
|
||||||
let allocation_info = allocation_info.allocation_info;
|
let allocation_info = allocation_info.allocation_info;
|
||||||
let mapped = NonNull::new(allocation_info.mapped_data.cast::<u8>()).map(SendSyncPtr);
|
let mapped = NonNull::new(allocation_info.mapped_data.cast::<u8>()).map(SendSyncPtr);
|
||||||
|
|
||||||
Ok(GpuAllocation {
|
Ok(GpuAllocation {
|
||||||
raw,
|
raw,
|
||||||
memory: allocation_info.device_memory,
|
memory: allocation_info.device_memory,
|
||||||
|
|
|
||||||
|
|
@ -83,6 +83,39 @@ pub enum Error {
|
||||||
"Dedicated allocation requires Vulkan 1.1 or both VK_KHR_dedicated_allocation and VK_KHR_get_memory_requirements2 extensions."
|
"Dedicated allocation requires Vulkan 1.1 or both VK_KHR_dedicated_allocation and VK_KHR_get_memory_requirements2 extensions."
|
||||||
)]
|
)]
|
||||||
DedicatedAllocationRequiresApi11,
|
DedicatedAllocationRequiresApi11,
|
||||||
|
#[error(
|
||||||
|
"Offset-aware memory binding requires Vulkan 1.1 or VK_KHR_get_memory_requirements2, VK_KHR_bind_memory2, and VK_KHR_dedicated_allocation to be enabled."
|
||||||
|
)]
|
||||||
|
OffsetBindingRequiresApi11,
|
||||||
|
#[error(
|
||||||
|
"No memory type matches requirements mask {memory_type_bits:#010x} and required flags {property_flags:?}."
|
||||||
|
)]
|
||||||
|
NoCompatibleMemoryType {
|
||||||
|
memory_type_bits: u32,
|
||||||
|
property_flags: vk::MemoryPropertyFlags,
|
||||||
|
},
|
||||||
|
#[error(
|
||||||
|
"Memory allocation requirements request a dedicated allocation, which cannot be allocated without its resource."
|
||||||
|
)]
|
||||||
|
DedicatedAllocationRequiresResource,
|
||||||
|
#[error(
|
||||||
|
"Allocation offset {offset} is invalid: alignment is {alignment}, requested size is {required_size}, allocation size is {allocation_size}."
|
||||||
|
)]
|
||||||
|
InvalidAllocationOffset {
|
||||||
|
offset: u64,
|
||||||
|
alignment: u64,
|
||||||
|
required_size: u64,
|
||||||
|
allocation_size: u64,
|
||||||
|
},
|
||||||
|
#[error(
|
||||||
|
"Allocation memory type {memory_type_index} is not allowed by requirements mask {memory_type_bits:#010x}."
|
||||||
|
)]
|
||||||
|
IncompatibleAllocationMemoryType {
|
||||||
|
memory_type_index: u32,
|
||||||
|
memory_type_bits: u32,
|
||||||
|
},
|
||||||
|
#[error("The allocation and resource belong to different Vulkan devices.")]
|
||||||
|
AllocationDeviceMismatch,
|
||||||
#[error(transparent)]
|
#[error(transparent)]
|
||||||
Io(#[from] std::io::Error),
|
Io(#[from] std::io::Error),
|
||||||
#[error(
|
#[error(
|
||||||
|
|
|
||||||
|
|
@ -282,7 +282,9 @@ unsafe extern "system" fn capture_callback(
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod tests {
|
mod tests {
|
||||||
use super::*;
|
use super::*;
|
||||||
use crate::device::{AllocationDesc, AllocationStrategy, DeviceObject, MemoryLocation};
|
use crate::device::{
|
||||||
|
Allocation, AllocationDesc, AllocationStrategy, DeviceObject, MemoryLocation,
|
||||||
|
};
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn instance_and_device_creation() {
|
fn instance_and_device_creation() {
|
||||||
|
|
@ -326,6 +328,121 @@ mod tests {
|
||||||
test.assert_no_validation_errors();
|
test.assert_no_validation_errors();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn aliasable_buffer_allocation_binds_at_zero_under_validation() {
|
||||||
|
let Some(test) = TestVulkan::acquire() else {
|
||||||
|
return;
|
||||||
|
};
|
||||||
|
let device = test.device();
|
||||||
|
let create_info = vk::BufferCreateInfo::default()
|
||||||
|
.size(256)
|
||||||
|
.usage(vk::BufferUsageFlags::TRANSFER_SRC | vk::BufferUsageFlags::TRANSFER_DST)
|
||||||
|
.sharing_mode(vk::SharingMode::EXCLUSIVE);
|
||||||
|
let buffer = unsafe { device.dev().create_buffer(&create_info, None) }
|
||||||
|
.expect("create aliasable buffer");
|
||||||
|
let buffer = DeviceObject::new(device.clone(), buffer);
|
||||||
|
let requirements = device
|
||||||
|
.buffer_memory_requirements(*buffer)
|
||||||
|
.expect("query buffer requirements");
|
||||||
|
let memory = device
|
||||||
|
.allocate_memory(&requirements, vk::MemoryPropertyFlags::empty())
|
||||||
|
.expect("allocate aliasable buffer memory");
|
||||||
|
let binding = device
|
||||||
|
.bind_buffer_at_offset(*buffer, &requirements, memory, 0)
|
||||||
|
.expect("bind aliasable buffer memory");
|
||||||
|
assert!(matches!(binding, Allocation::Shared(_)));
|
||||||
|
|
||||||
|
drop(buffer);
|
||||||
|
drop(binding);
|
||||||
|
test.assert_no_validation_errors();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn aliasable_image_allocation_binds_at_zero_under_validation() {
|
||||||
|
let Some(test) = TestVulkan::acquire() else {
|
||||||
|
return;
|
||||||
|
};
|
||||||
|
let device = test.device();
|
||||||
|
let create_info = vk::ImageCreateInfo::default()
|
||||||
|
.image_type(vk::ImageType::TYPE_2D)
|
||||||
|
.format(vk::Format::R8G8B8A8_UNORM)
|
||||||
|
.extent(vk::Extent3D {
|
||||||
|
width: 16,
|
||||||
|
height: 16,
|
||||||
|
depth: 1,
|
||||||
|
})
|
||||||
|
.mip_levels(1)
|
||||||
|
.array_layers(1)
|
||||||
|
.samples(vk::SampleCountFlags::TYPE_1)
|
||||||
|
.tiling(vk::ImageTiling::OPTIMAL)
|
||||||
|
.usage(vk::ImageUsageFlags::TRANSFER_DST)
|
||||||
|
.sharing_mode(vk::SharingMode::EXCLUSIVE)
|
||||||
|
.initial_layout(vk::ImageLayout::UNDEFINED);
|
||||||
|
let image = unsafe { device.dev().create_image(&create_info, None) }
|
||||||
|
.expect("create aliasable image");
|
||||||
|
let image = DeviceObject::new(device.clone(), image);
|
||||||
|
let requirements = device
|
||||||
|
.image_memory_requirements(*image)
|
||||||
|
.expect("query image requirements");
|
||||||
|
let memory = device
|
||||||
|
.allocate_memory(&requirements, vk::MemoryPropertyFlags::empty())
|
||||||
|
.expect("allocate aliasable image memory");
|
||||||
|
let binding = device
|
||||||
|
.bind_image_at_offset(*image, &requirements, memory, 0)
|
||||||
|
.expect("bind aliasable image memory");
|
||||||
|
assert!(matches!(binding, Allocation::Shared(_)));
|
||||||
|
|
||||||
|
drop(image);
|
||||||
|
drop(binding);
|
||||||
|
test.assert_no_validation_errors();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn aliasable_binding_rejects_out_of_bounds_and_unaligned_offsets() {
|
||||||
|
let Some(test) = TestVulkan::acquire() else {
|
||||||
|
return;
|
||||||
|
};
|
||||||
|
let device = test.device();
|
||||||
|
let create_info = vk::BufferCreateInfo::default()
|
||||||
|
.size(256)
|
||||||
|
.usage(vk::BufferUsageFlags::TRANSFER_DST)
|
||||||
|
.sharing_mode(vk::SharingMode::EXCLUSIVE);
|
||||||
|
|
||||||
|
let make_buffer = || {
|
||||||
|
let raw = unsafe { device.dev().create_buffer(&create_info, None) }
|
||||||
|
.expect("create offset validation buffer");
|
||||||
|
DeviceObject::new(device.clone(), raw)
|
||||||
|
};
|
||||||
|
let buffer = make_buffer();
|
||||||
|
let requirements = device
|
||||||
|
.buffer_memory_requirements(*buffer)
|
||||||
|
.expect("query offset validation requirements");
|
||||||
|
let memory = device
|
||||||
|
.allocate_memory(&requirements, vk::MemoryPropertyFlags::empty())
|
||||||
|
.expect("allocate offset validation memory");
|
||||||
|
assert!(matches!(
|
||||||
|
device.bind_buffer_at_offset(*buffer, &requirements, memory, requirements.size),
|
||||||
|
Err(crate::Error::InvalidAllocationOffset { .. })
|
||||||
|
));
|
||||||
|
drop(buffer);
|
||||||
|
|
||||||
|
let buffer = make_buffer();
|
||||||
|
let requirements = device
|
||||||
|
.buffer_memory_requirements(*buffer)
|
||||||
|
.expect("query unaligned validation requirements");
|
||||||
|
if requirements.alignment > 1 {
|
||||||
|
let memory = device
|
||||||
|
.allocate_memory(&requirements, vk::MemoryPropertyFlags::empty())
|
||||||
|
.expect("allocate unaligned validation memory");
|
||||||
|
assert!(matches!(
|
||||||
|
device.bind_buffer_at_offset(*buffer, &requirements, memory, 1),
|
||||||
|
Err(crate::Error::InvalidAllocationOffset { .. })
|
||||||
|
));
|
||||||
|
}
|
||||||
|
drop(buffer);
|
||||||
|
test.assert_no_validation_errors();
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn validation_capture_works() {
|
fn validation_capture_works() {
|
||||||
let Some(test) = TestVulkan::acquire() else {
|
let Some(test) = TestVulkan::acquire() else {
|
||||||
|
|
|
||||||
|
|
@ -5,6 +5,7 @@ edition = "2024"
|
||||||
|
|
||||||
[dependencies]
|
[dependencies]
|
||||||
ash.workspace = true
|
ash.workspace = true
|
||||||
|
fixedbitset.workspace = true
|
||||||
renderer_device = { path = "../renderer_device" }
|
renderer_device = { path = "../renderer_device" }
|
||||||
renderer_resources = { path = "../renderer_resources" }
|
renderer_resources = { path = "../renderer_resources" }
|
||||||
ash-formats = { git = "https://github.com/janis-bhm/ash-formats", branch = "main" }
|
ash-formats = { git = "https://github.com/janis-bhm/ash-formats", branch = "main" }
|
||||||
|
|
|
||||||
File diff suppressed because it is too large
Load diff
|
|
@ -6,7 +6,10 @@ pub mod graph;
|
||||||
pub mod recorder;
|
pub mod recorder;
|
||||||
pub mod resources;
|
pub mod resources;
|
||||||
|
|
||||||
pub use graph::{CompiledGraph, GraphError, RenderGraph, ResourceInterference};
|
pub use graph::{
|
||||||
|
CompiledGraph, GraphError, HappensBefore, MemoryBarrier, Placement, RenderGraph,
|
||||||
|
ResourceInterference, ResourceLifetime, ResourceUse, aliasing_permitted,
|
||||||
|
};
|
||||||
pub use recorder::{CommandIndex, CommandList, CommandRecorder, SideEffectMap};
|
pub use recorder::{CommandIndex, CommandList, CommandRecorder, SideEffectMap};
|
||||||
pub use resources::{
|
pub use resources::{
|
||||||
Buffer, BufferRange, BufferSlice, DescriptorResourceAccess, DescriptorResourceIdentity,
|
Buffer, BufferRange, BufferSlice, DescriptorResourceAccess, DescriptorResourceIdentity,
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
use std::collections::BTreeMap;
|
use std::{cell::RefCell, collections::BTreeMap};
|
||||||
|
|
||||||
use ash::vk;
|
use ash::vk;
|
||||||
|
|
||||||
|
|
@ -163,6 +163,17 @@ pub struct CommandRecorder<'a> {
|
||||||
pub images: BTreeMap<ResourceId, vk::Image>,
|
pub images: BTreeMap<ResourceId, vk::Image>,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[derive(Default)]
|
||||||
|
struct BarrierScratch {
|
||||||
|
memory: Vec<vk::MemoryBarrier2<'static>>,
|
||||||
|
buffers: Vec<vk::BufferMemoryBarrier2<'static>>,
|
||||||
|
images: Vec<vk::ImageMemoryBarrier2<'static>>,
|
||||||
|
}
|
||||||
|
|
||||||
|
thread_local! {
|
||||||
|
static BARRIER_SCRATCH: RefCell<BarrierScratch> = RefCell::new(BarrierScratch::default());
|
||||||
|
}
|
||||||
|
|
||||||
impl<'a> CommandRecorder<'a> {
|
impl<'a> CommandRecorder<'a> {
|
||||||
pub fn new(device: &'a ash::Device, command_buffer: vk::CommandBuffer) -> Self {
|
pub fn new(device: &'a ash::Device, command_buffer: vk::CommandBuffer) -> Self {
|
||||||
Self {
|
Self {
|
||||||
|
|
@ -188,13 +199,31 @@ impl<'a> CommandRecorder<'a> {
|
||||||
}
|
}
|
||||||
|
|
||||||
pub fn pipeline_barrier(&self, barrier: &super::graph::PipelineBarrier) {
|
pub fn pipeline_barrier(&self, barrier: &super::graph::PipelineBarrier) {
|
||||||
let buffer_barriers: Vec<_> = barrier
|
BARRIER_SCRATCH.with(|scratch| {
|
||||||
|
let mut scratch = scratch.borrow_mut();
|
||||||
|
scratch.memory.clear();
|
||||||
|
scratch
|
||||||
|
.memory
|
||||||
|
.extend(barrier.memory_barriers.iter().map(|resource| {
|
||||||
|
vk::MemoryBarrier2::default()
|
||||||
|
.src_stage_mask(resource.src_stage)
|
||||||
|
.src_access_mask(resource.src_access)
|
||||||
|
.dst_stage_mask(resource.dst_stage)
|
||||||
|
.dst_access_mask(resource.dst_access)
|
||||||
|
}));
|
||||||
|
|
||||||
|
scratch.buffers.clear();
|
||||||
|
scratch
|
||||||
|
.buffers
|
||||||
|
.extend(
|
||||||
|
barrier
|
||||||
.resources
|
.resources
|
||||||
.iter()
|
.iter()
|
||||||
.filter_map(|resource| match resource.range {
|
.filter_map(|resource| match resource.range {
|
||||||
Subresource::Buffer(BufferRange { offset, size }) => {
|
Subresource::Buffer(BufferRange { offset, size }) => {
|
||||||
self.buffers.get(&resource.id).map(|&buffer| {
|
self.buffers.get(&resource.id).map(|&buffer| {
|
||||||
let (src, dst) = if let Some(queue_transfer) = resource.queue_transfer {
|
let (src, dst) =
|
||||||
|
if let Some(queue_transfer) = resource.queue_transfer {
|
||||||
(queue_transfer.src.index(), queue_transfer.dst.index())
|
(queue_transfer.src.index(), queue_transfer.dst.index())
|
||||||
} else {
|
} else {
|
||||||
(vk::QUEUE_FAMILY_IGNORED, vk::QUEUE_FAMILY_IGNORED)
|
(vk::QUEUE_FAMILY_IGNORED, vk::QUEUE_FAMILY_IGNORED)
|
||||||
|
|
@ -213,35 +242,49 @@ impl<'a> CommandRecorder<'a> {
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
Subresource::Texture(_) => None,
|
Subresource::Texture(_) => None,
|
||||||
})
|
}),
|
||||||
.collect();
|
);
|
||||||
|
|
||||||
let image_barriers: Vec<_> = barrier
|
scratch.images.clear();
|
||||||
|
scratch
|
||||||
|
.images
|
||||||
|
.extend(
|
||||||
|
barrier
|
||||||
.resources
|
.resources
|
||||||
.iter()
|
.iter()
|
||||||
.filter_map(|resource| match resource.range {
|
.filter_map(|resource| match resource.range {
|
||||||
Subresource::Texture(range) => self.images.get(&resource.id).map(|&image| {
|
Subresource::Texture(range) => {
|
||||||
|
self.images.get(&resource.id).map(|&image| {
|
||||||
vk::ImageMemoryBarrier2::default()
|
vk::ImageMemoryBarrier2::default()
|
||||||
.src_stage_mask(resource.src_stage)
|
.src_stage_mask(resource.src_stage)
|
||||||
.src_access_mask(resource.src_access)
|
.src_access_mask(resource.src_access)
|
||||||
.dst_stage_mask(resource.dst_stage)
|
.dst_stage_mask(resource.dst_stage)
|
||||||
.dst_access_mask(resource.dst_access)
|
.dst_access_mask(resource.dst_access)
|
||||||
.old_layout(resource.old_layout.unwrap_or(vk::ImageLayout::UNDEFINED))
|
.old_layout(
|
||||||
.new_layout(resource.new_layout.unwrap_or(vk::ImageLayout::GENERAL))
|
resource
|
||||||
|
.old_layout
|
||||||
|
.unwrap_or(vk::ImageLayout::UNDEFINED),
|
||||||
|
)
|
||||||
|
.new_layout(
|
||||||
|
resource.new_layout.unwrap_or(vk::ImageLayout::GENERAL),
|
||||||
|
)
|
||||||
.image(image)
|
.image(image)
|
||||||
.subresource_range(range)
|
.subresource_range(range)
|
||||||
}),
|
|
||||||
Subresource::Buffer(_) => None,
|
|
||||||
})
|
})
|
||||||
.collect();
|
}
|
||||||
|
Subresource::Buffer(_) => None,
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
|
||||||
let dependency = vk::DependencyInfo::default()
|
let dependency = vk::DependencyInfo::default()
|
||||||
.buffer_memory_barriers(&buffer_barriers)
|
.memory_barriers(&scratch.memory)
|
||||||
.image_memory_barriers(&image_barriers);
|
.buffer_memory_barriers(&scratch.buffers)
|
||||||
|
.image_memory_barriers(&scratch.images);
|
||||||
unsafe {
|
unsafe {
|
||||||
self.device
|
self.device
|
||||||
.cmd_pipeline_barrier2(self.command_buffer, &dependency);
|
.cmd_pipeline_barrier2(self.command_buffer, &dependency);
|
||||||
}
|
}
|
||||||
|
});
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -255,6 +298,58 @@ pub(crate) fn scope_range(scope: SubresourceScope) -> Subresource {
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod tests {
|
mod tests {
|
||||||
use super::*;
|
use super::*;
|
||||||
|
use crate::graph::{MemoryBarrier, PipelineBarrier};
|
||||||
|
use std::sync::atomic::{AtomicU32, AtomicU64, Ordering};
|
||||||
|
|
||||||
|
static MEMORY_BARRIER_COUNT: AtomicU32 = AtomicU32::new(0);
|
||||||
|
static MEMORY_BARRIER_SRC_STAGE: AtomicU64 = AtomicU64::new(0);
|
||||||
|
|
||||||
|
unsafe extern "system" fn capture_pipeline_barrier(
|
||||||
|
_device: vk::Device,
|
||||||
|
_command_buffer: vk::CommandBuffer,
|
||||||
|
dependency: *const vk::DependencyInfo<'_>,
|
||||||
|
) {
|
||||||
|
let dependency = unsafe { &*dependency };
|
||||||
|
MEMORY_BARRIER_COUNT.store(dependency.memory_barrier_count, Ordering::SeqCst);
|
||||||
|
let memory = unsafe { &*dependency.p_memory_barriers };
|
||||||
|
MEMORY_BARRIER_SRC_STAGE.store(memory.src_stage_mask.as_raw(), Ordering::SeqCst);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pipeline_barrier_emits_memory_barriers_in_dependency_info() {
|
||||||
|
let device = unsafe {
|
||||||
|
ash::Device::load_with(
|
||||||
|
|name| {
|
||||||
|
if name.to_bytes() == b"vkCmdPipelineBarrier2" {
|
||||||
|
capture_pipeline_barrier as *const () as *const std::ffi::c_void
|
||||||
|
} else {
|
||||||
|
std::ptr::null()
|
||||||
|
}
|
||||||
|
},
|
||||||
|
vk::Device::null(),
|
||||||
|
)
|
||||||
|
};
|
||||||
|
let barrier = PipelineBarrier {
|
||||||
|
before_command: CommandIndex(0),
|
||||||
|
memory_barriers: vec![MemoryBarrier {
|
||||||
|
src_stage: vk::PipelineStageFlags2::COMPUTE_SHADER,
|
||||||
|
src_access: vk::AccessFlags2::SHADER_STORAGE_WRITE,
|
||||||
|
dst_stage: vk::PipelineStageFlags2::FRAGMENT_SHADER,
|
||||||
|
dst_access: vk::AccessFlags2::SHADER_READ,
|
||||||
|
}],
|
||||||
|
resources: Vec::new(),
|
||||||
|
};
|
||||||
|
MEMORY_BARRIER_COUNT.store(0, Ordering::SeqCst);
|
||||||
|
MEMORY_BARRIER_SRC_STAGE.store(0, Ordering::SeqCst);
|
||||||
|
|
||||||
|
CommandRecorder::new(&device, vk::CommandBuffer::null()).pipeline_barrier(&barrier);
|
||||||
|
|
||||||
|
assert_eq!(MEMORY_BARRIER_COUNT.load(Ordering::SeqCst), 1);
|
||||||
|
assert_eq!(
|
||||||
|
MEMORY_BARRIER_SRC_STAGE.load(Ordering::SeqCst),
|
||||||
|
vk::PipelineStageFlags2::COMPUTE_SHADER.as_raw()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
fn read_scope(
|
fn read_scope(
|
||||||
offset: u64,
|
offset: u64,
|
||||||
|
|
@ -275,7 +370,7 @@ mod tests {
|
||||||
let command = map.begin_command();
|
let command = map.begin_command();
|
||||||
map.insert(
|
map.insert(
|
||||||
command,
|
command,
|
||||||
ResourceId(0),
|
ResourceId::from_u32(0),
|
||||||
read_scope(
|
read_scope(
|
||||||
0,
|
0,
|
||||||
64,
|
64,
|
||||||
|
|
@ -285,7 +380,7 @@ mod tests {
|
||||||
);
|
);
|
||||||
map.insert(
|
map.insert(
|
||||||
command,
|
command,
|
||||||
ResourceId(0),
|
ResourceId::from_u32(0),
|
||||||
read_scope(
|
read_scope(
|
||||||
16,
|
16,
|
||||||
16,
|
16,
|
||||||
|
|
@ -316,7 +411,7 @@ mod tests {
|
||||||
let command = reverse.begin_command();
|
let command = reverse.begin_command();
|
||||||
reverse.insert(
|
reverse.insert(
|
||||||
command,
|
command,
|
||||||
ResourceId(1),
|
ResourceId::from_u32(1),
|
||||||
read_scope(
|
read_scope(
|
||||||
16,
|
16,
|
||||||
16,
|
16,
|
||||||
|
|
@ -326,7 +421,7 @@ mod tests {
|
||||||
);
|
);
|
||||||
reverse.insert(
|
reverse.insert(
|
||||||
command,
|
command,
|
||||||
ResourceId(1),
|
ResourceId::from_u32(1),
|
||||||
read_scope(
|
read_scope(
|
||||||
0,
|
0,
|
||||||
64,
|
64,
|
||||||
|
|
@ -360,7 +455,7 @@ mod tests {
|
||||||
let command = map.begin_command();
|
let command = map.begin_command();
|
||||||
map.insert(
|
map.insert(
|
||||||
command,
|
command,
|
||||||
ResourceId(0),
|
ResourceId::from_u32(0),
|
||||||
read_scope(
|
read_scope(
|
||||||
0,
|
0,
|
||||||
16,
|
16,
|
||||||
|
|
@ -370,7 +465,7 @@ mod tests {
|
||||||
);
|
);
|
||||||
map.insert(
|
map.insert(
|
||||||
command,
|
command,
|
||||||
ResourceId(0),
|
ResourceId::from_u32(0),
|
||||||
read_scope(
|
read_scope(
|
||||||
8,
|
8,
|
||||||
16,
|
16,
|
||||||
|
|
@ -388,7 +483,7 @@ mod tests {
|
||||||
let command = map.begin_command();
|
let command = map.begin_command();
|
||||||
map.insert(
|
map.insert(
|
||||||
command,
|
command,
|
||||||
ResourceId(0),
|
ResourceId::from_u32(0),
|
||||||
read_scope(
|
read_scope(
|
||||||
0,
|
0,
|
||||||
64,
|
64,
|
||||||
|
|
@ -398,7 +493,7 @@ mod tests {
|
||||||
);
|
);
|
||||||
map.insert(
|
map.insert(
|
||||||
command,
|
command,
|
||||||
ResourceId(0),
|
ResourceId::from_u32(0),
|
||||||
read_scope(
|
read_scope(
|
||||||
16,
|
16,
|
||||||
16,
|
16,
|
||||||
|
|
|
||||||
|
|
@ -75,10 +75,14 @@ impl Default for GraphTextureDesc {
|
||||||
|
|
||||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
|
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
|
||||||
#[repr(transparent)]
|
#[repr(transparent)]
|
||||||
pub struct ResourceId(pub u64);
|
pub struct ResourceId(u32);
|
||||||
|
|
||||||
impl ResourceId {
|
impl ResourceId {
|
||||||
pub(crate) fn index(self) -> usize {
|
pub fn from_u32(index: u32) -> Self {
|
||||||
|
Self(index)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn index(self) -> usize {
|
||||||
self.0 as usize
|
self.0 as usize
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -7,12 +7,98 @@ use renderer_device::{
|
||||||
};
|
};
|
||||||
use renderer_rendergraph::{
|
use renderer_rendergraph::{
|
||||||
RenderGraph,
|
RenderGraph,
|
||||||
|
commands::Command,
|
||||||
commands::CopyBuffer,
|
commands::CopyBuffer,
|
||||||
recorder::CommandRecorder,
|
recorder::{CommandIndex, CommandRecorder, RecordError, RecordedCommand, SideEffectMap},
|
||||||
resources::{BufferRange, ResourceState},
|
resources::{
|
||||||
|
BufferRange, GraphBufferDesc, GraphTextureDesc, ResourceId, ResourceState,
|
||||||
|
SubresourceScope, SubresourceScopeInner,
|
||||||
|
},
|
||||||
};
|
};
|
||||||
use renderer_resources::buffers::{Buffer, BufferDesc};
|
use renderer_resources::buffers::{Buffer, BufferDesc};
|
||||||
|
|
||||||
|
struct ScopedNoop(Vec<(ResourceId, SubresourceScope)>);
|
||||||
|
|
||||||
|
impl Command for ScopedNoop {
|
||||||
|
fn side_effects(&self, command_index: CommandIndex, map: &mut SideEffectMap) {
|
||||||
|
for &(resource, scope) in &self.0 {
|
||||||
|
map.insert(command_index, resource, scope);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl RecordedCommand for ScopedNoop {
|
||||||
|
fn apply(&mut self, _recorder: &mut CommandRecorder<'_>) -> Result<(), RecordError> {
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn buffer_scope() -> SubresourceScope {
|
||||||
|
SubresourceScope {
|
||||||
|
stage: vk::PipelineStageFlags2::COPY,
|
||||||
|
access: vk::AccessFlags2::TRANSFER_WRITE,
|
||||||
|
subresource: SubresourceScopeInner::Buffer(BufferRange {
|
||||||
|
offset: 0,
|
||||||
|
size: 256,
|
||||||
|
}),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn image_scope() -> SubresourceScope {
|
||||||
|
SubresourceScope {
|
||||||
|
stage: vk::PipelineStageFlags2::COMPUTE_SHADER,
|
||||||
|
access: vk::AccessFlags2::SHADER_STORAGE_WRITE,
|
||||||
|
subresource: SubresourceScopeInner::Texture {
|
||||||
|
range: vk::ImageSubresourceRange {
|
||||||
|
aspect_mask: vk::ImageAspectFlags::COLOR,
|
||||||
|
base_mip_level: 0,
|
||||||
|
level_count: 1,
|
||||||
|
base_array_layer: 0,
|
||||||
|
layer_count: 1,
|
||||||
|
},
|
||||||
|
layout: vk::ImageLayout::GENERAL,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn submit_graph(test: &TestVulkan, graph: &mut renderer_rendergraph::CompiledGraph) {
|
||||||
|
let device = test.device();
|
||||||
|
let pool_info = vk::CommandPoolCreateInfo::default()
|
||||||
|
.queue_family_index(device.queues.graphics().family.index.index());
|
||||||
|
let pool = unsafe { device.dev().create_command_pool(&pool_info, None) }
|
||||||
|
.expect("create alias test command pool");
|
||||||
|
let allocation_info = vk::CommandBufferAllocateInfo::default()
|
||||||
|
.command_pool(pool)
|
||||||
|
.level(vk::CommandBufferLevel::PRIMARY)
|
||||||
|
.command_buffer_count(1);
|
||||||
|
let command_buffer = unsafe { device.dev().allocate_command_buffers(&allocation_info) }
|
||||||
|
.expect("allocate alias test command buffer")[0];
|
||||||
|
unsafe {
|
||||||
|
device
|
||||||
|
.dev()
|
||||||
|
.begin_command_buffer(command_buffer, &vk::CommandBufferBeginInfo::default())
|
||||||
|
.expect("begin alias test command buffer");
|
||||||
|
}
|
||||||
|
let mut recorder = CommandRecorder::new(device.dev(), command_buffer);
|
||||||
|
graph
|
||||||
|
.record_commands(&mut recorder)
|
||||||
|
.expect("record alias graph");
|
||||||
|
unsafe {
|
||||||
|
device
|
||||||
|
.dev()
|
||||||
|
.end_command_buffer(command_buffer)
|
||||||
|
.expect("end alias test command buffer");
|
||||||
|
let command_buffers = [command_buffer];
|
||||||
|
let submit = vk::SubmitInfo::default().command_buffers(&command_buffers);
|
||||||
|
device
|
||||||
|
.dev()
|
||||||
|
.queue_submit(device.queues.graphics().raw, &[submit], vk::Fence::null())
|
||||||
|
.expect("submit alias graph");
|
||||||
|
}
|
||||||
|
device.wait_idle().expect("wait for alias graph idle");
|
||||||
|
unsafe { device.dev().destroy_command_pool(pool, None) };
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn imported_buffer_copy_records_under_validation() {
|
fn imported_buffer_copy_records_under_validation() {
|
||||||
let Some(test) = TestVulkan::acquire() else {
|
let Some(test) = TestVulkan::acquire() else {
|
||||||
|
|
@ -91,3 +177,154 @@ fn imported_buffer_copy_records_under_validation() {
|
||||||
device.wait_idle().expect("wait for graph device idle");
|
device.wait_idle().expect("wait for graph device idle");
|
||||||
test.assert_no_validation_errors();
|
test.assert_no_validation_errors();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn resolved_buffers_alias_and_submit_with_memory_barrier() {
|
||||||
|
let Some(test) = TestVulkan::acquire() else {
|
||||||
|
return;
|
||||||
|
};
|
||||||
|
let device = test.device();
|
||||||
|
if device.enabled_features().core13.synchronization2 != vk::TRUE {
|
||||||
|
eprintln!("Skipping graph aliasing: synchronization2 is unsupported");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let bridge = Arc::new(
|
||||||
|
Buffer::new(
|
||||||
|
device.clone(),
|
||||||
|
BufferDesc {
|
||||||
|
size: 256,
|
||||||
|
usage: vk::BufferUsageFlags::TRANSFER_DST,
|
||||||
|
mem_location: MemoryLocation::CpuToGpu,
|
||||||
|
..Default::default()
|
||||||
|
},
|
||||||
|
)
|
||||||
|
.expect("create dependency bridge buffer"),
|
||||||
|
);
|
||||||
|
let mut graph = RenderGraph::new();
|
||||||
|
let first = graph.create_buffer(GraphBufferDesc {
|
||||||
|
size: 256,
|
||||||
|
..Default::default()
|
||||||
|
});
|
||||||
|
let second = graph.create_buffer(GraphBufferDesc {
|
||||||
|
size: 256,
|
||||||
|
..Default::default()
|
||||||
|
});
|
||||||
|
let bridge = graph
|
||||||
|
.import_buffer(bridge, ResourceState::default())
|
||||||
|
.expect("import dependency bridge");
|
||||||
|
graph.push(ScopedNoop(vec![
|
||||||
|
(first.id(), buffer_scope()),
|
||||||
|
(bridge.id(), buffer_scope()),
|
||||||
|
]));
|
||||||
|
graph.push(ScopedNoop(vec![
|
||||||
|
(bridge.id(), buffer_scope()),
|
||||||
|
(second.id(), buffer_scope()),
|
||||||
|
]));
|
||||||
|
|
||||||
|
let mut compiled = graph.compile().expect("compile buffer alias graph");
|
||||||
|
compiled
|
||||||
|
.resolve(device)
|
||||||
|
.expect("resolve buffer alias graph");
|
||||||
|
compiled
|
||||||
|
.resolve(device)
|
||||||
|
.expect("resolving an already-resolved graph is idempotent");
|
||||||
|
let first_placement = compiled.resource_lifetimes()[&first.id()].placement;
|
||||||
|
let second_placement = compiled.resource_lifetimes()[&second.id()].placement;
|
||||||
|
assert_eq!(first_placement.pool, second_placement.pool);
|
||||||
|
assert_eq!(first_placement.offset, second_placement.offset);
|
||||||
|
assert!(first_placement.aliased && second_placement.aliased);
|
||||||
|
assert!(compiled.barriers().iter().any(|barrier| {
|
||||||
|
barrier.before_command == CommandIndex(1) && !barrier.memory_barriers.is_empty()
|
||||||
|
}));
|
||||||
|
|
||||||
|
submit_graph(&test, &mut compiled);
|
||||||
|
test.assert_no_validation_errors();
|
||||||
|
drop(compiled);
|
||||||
|
device
|
||||||
|
.wait_idle()
|
||||||
|
.expect("wait before shared allocation teardown");
|
||||||
|
test.assert_no_validation_errors();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn resolved_images_alias_and_submit_with_undefined_transition() {
|
||||||
|
let Some(test) = TestVulkan::acquire() else {
|
||||||
|
return;
|
||||||
|
};
|
||||||
|
let device = test.device();
|
||||||
|
if device.enabled_features().core13.synchronization2 != vk::TRUE {
|
||||||
|
eprintln!("Skipping graph aliasing: synchronization2 is unsupported");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let bridge = Arc::new(
|
||||||
|
Buffer::new(
|
||||||
|
device.clone(),
|
||||||
|
BufferDesc {
|
||||||
|
size: 256,
|
||||||
|
usage: vk::BufferUsageFlags::TRANSFER_DST,
|
||||||
|
mem_location: MemoryLocation::CpuToGpu,
|
||||||
|
..Default::default()
|
||||||
|
},
|
||||||
|
)
|
||||||
|
.expect("create dependency bridge buffer"),
|
||||||
|
);
|
||||||
|
let mut graph = RenderGraph::new();
|
||||||
|
let first = graph.create_texture(GraphTextureDesc {
|
||||||
|
format: vk::Format::R8G8B8A8_UNORM,
|
||||||
|
extent: vk::Extent3D {
|
||||||
|
width: 16,
|
||||||
|
height: 16,
|
||||||
|
depth: 1,
|
||||||
|
},
|
||||||
|
extra_usage: vk::ImageUsageFlags::STORAGE,
|
||||||
|
..Default::default()
|
||||||
|
});
|
||||||
|
let second = graph.create_texture(GraphTextureDesc {
|
||||||
|
format: vk::Format::R8G8B8A8_UNORM,
|
||||||
|
extent: vk::Extent3D {
|
||||||
|
width: 16,
|
||||||
|
height: 16,
|
||||||
|
depth: 1,
|
||||||
|
},
|
||||||
|
extra_usage: vk::ImageUsageFlags::STORAGE,
|
||||||
|
..Default::default()
|
||||||
|
});
|
||||||
|
let bridge = graph
|
||||||
|
.import_buffer(bridge, ResourceState::default())
|
||||||
|
.expect("import dependency bridge");
|
||||||
|
graph.push(ScopedNoop(vec![
|
||||||
|
(first.id(), image_scope()),
|
||||||
|
(bridge.id(), buffer_scope()),
|
||||||
|
]));
|
||||||
|
graph.push(ScopedNoop(vec![
|
||||||
|
(bridge.id(), buffer_scope()),
|
||||||
|
(second.id(), image_scope()),
|
||||||
|
]));
|
||||||
|
|
||||||
|
let mut compiled = graph.compile().expect("compile image alias graph");
|
||||||
|
compiled.resolve(device).expect("resolve image alias graph");
|
||||||
|
compiled
|
||||||
|
.resolve(device)
|
||||||
|
.expect("resolving an already-resolved graph is idempotent");
|
||||||
|
let first_placement = compiled.resource_lifetimes()[&first.id()].placement;
|
||||||
|
let second_placement = compiled.resource_lifetimes()[&second.id()].placement;
|
||||||
|
assert_eq!(first_placement.pool, second_placement.pool);
|
||||||
|
assert_eq!(first_placement.offset, second_placement.offset);
|
||||||
|
assert!(first_placement.aliased && second_placement.aliased);
|
||||||
|
assert!(compiled.barriers().iter().any(|barrier| {
|
||||||
|
barrier.before_command == CommandIndex(1)
|
||||||
|
&& barrier.resources.iter().any(|resource| {
|
||||||
|
resource.id == second.id()
|
||||||
|
&& resource.old_layout == Some(vk::ImageLayout::UNDEFINED)
|
||||||
|
&& resource.src_access == vk::AccessFlags2::SHADER_STORAGE_WRITE
|
||||||
|
})
|
||||||
|
}));
|
||||||
|
|
||||||
|
submit_graph(&test, &mut compiled);
|
||||||
|
test.assert_no_validation_errors();
|
||||||
|
drop(compiled);
|
||||||
|
device
|
||||||
|
.wait_idle()
|
||||||
|
.expect("wait before shared allocation teardown");
|
||||||
|
test.assert_no_validation_errors();
|
||||||
|
}
|
||||||
|
|
|
||||||
|
|
@ -4,19 +4,21 @@
|
||||||
`renderer_resources` owns Vulkan buffer, image, view, descriptor, sampler, shader-module, and pipeline construction on the `renderer_device` layer. It handles resource memory and handle lifetimes, but contains no window, swapchain, or presentation orchestration; those remain in `renderer`. It must not depend on `renderer`.
|
`renderer_resources` owns Vulkan buffer, image, view, descriptor, sampler, shader-module, and pipeline construction on the `renderer_device` layer. It handles resource memory and handle lifetimes, but contains no window, swapchain, or presentation orchestration; those remain in `renderer`. It must not depend on `renderer`.
|
||||||
|
|
||||||
## Module map
|
## Module map
|
||||||
- `lib.rs`: re-exports `Device`, `Error`, `Result`, `device`, and `make_extension` from `renderer_device`; declares the public resource modules, including descriptor metadata and validation.
|
- `lib.rs`: re-exports `Device`, `Error`, `Result`, `device`, and `make_extension` from `renderer_device`; declares public buffer, image, memory, pipeline, utility, and descriptor modules.
|
||||||
- `buffers.rs`: `BufferDesc` and `Buffer`; Vulkan buffer creation, GPU allocation/binding, raw access, and host mapping.
|
- `buffers.rs`: `BufferDesc` and `Buffer`; allocated and unallocated Vulkan buffer creation, memory requirement query/binding, raw access, and host mapping.
|
||||||
- `images.rs`: `ImageDesc`, `Image`, `ImageViewDesc`, `ImageView`, subresource ranges, extents, offsets, and queue ownership. `ImageInner` distinguishes allocated images from borrowed swapchain images.
|
- `images.rs`: `ImageDesc`, `Image`, `ImageViewDesc`, `ImageView`, subresource ranges, extents, offsets, and queue ownership. `ImageInner` distinguishes allocated/unallocated images from borrowed swapchain images.
|
||||||
|
- `memory.rs`: re-exports device `MemoryRequirements`; exposes memory validation errors, `MemoryMember`, and `MemoryGroup` for aligned shared buffer/image allocations.
|
||||||
- `pipeline.rs`: shader-stage and graphics/compute pipeline descriptions; `DescriptorPool`, `DescriptorSetLayout`, `PipelineLayout`, `Sampler`, `ShaderModule`, and `Pipeline`.
|
- `pipeline.rs`: shader-stage and graphics/compute pipeline descriptions; `DescriptorPool`, `DescriptorSetLayout`, `PipelineLayout`, `Sampler`, `ShaderModule`, and `Pipeline`.
|
||||||
- `descriptors.rs`: managed descriptor binding/resource metadata, validation plans, errors, and pool-generation helpers; it remains independent of render-graph types.
|
- `descriptors.rs`: managed descriptor binding/resource metadata, validation plans, errors, and pool-generation helpers; it remains independent of render-graph types.
|
||||||
- `util.rs`: `eq_f32`, `ord_f32`, `hash_f32`, `F32`, and the crate-private `weak_vec` module / `WeakVec` used by image view tracking.
|
- `util.rs`: `eq_f32`, `ord_f32`, `hash_f32`, `F32`, and the crate-private `weak_vec` module / `WeakVec` used by image view tracking.
|
||||||
|
|
||||||
## Key types and API
|
## Key types and API
|
||||||
- These modules are public from `renderer_resources`; the crate root re-exports the core device types and `device` module from `renderer_device` for resource construction.
|
- These modules are public from `renderer_resources`; the crate root re-exports the core device types and `device` module from `renderer_device` for resource construction.
|
||||||
- `BufferDesc` configures Vulkan buffer flags, size, usage, queue families, `renderer_device::device::MemoryLocation`, allocation strategy, and optional debug name. `Buffer::new` requests a VMA allocation through `DeviceInner::allocate_for_buffer` and binds it through `bind_buffer_memory`; `map` and `map_mut` return mapped slices when available.
|
- `BufferDesc` configures Vulkan buffer flags, size, usage, queue families, `renderer_device::device::MemoryLocation`, allocation strategy, and optional debug name. `Buffer::new` requests a VMA allocation through `DeviceInner::allocate_for_buffer` and binds it through `bind_buffer_memory`; `Buffer::new_unallocated` creates only the Vulkan handle. Its `memory_requirements` includes dedicated-allocation hints, and `bind_allocation(Arc<DeviceObject<GpuAllocation>>, offset)` validates then binds shared memory. `map` and `map_mut` return mapped slices when available.
|
||||||
- `Buffer::raw` returns the Vulkan handle and `len` returns the configured size. Mapping is bounded to the descriptor size, not the VMA allocation block size. CPU-mapped locations select host-visible, host-coherent memory.
|
- `Buffer::raw` returns the Vulkan handle and `len` returns the configured size. Mapping is bounded to the descriptor size, not the VMA allocation block size. CPU-mapped locations select host-visible, host-coherent memory.
|
||||||
- `ImageDesc` configures image type, format, extent, layers/mips, samples, tiling, usage, initial layout, queue sharing, and allocation strategy/location. `Image::new` uses the device's VMA allocation and image-binding helpers.
|
- `ImageDesc` configures image type, format, extent, layers/mips, samples, tiling, usage, initial layout, queue sharing, and allocation strategy/location. `Image::new` uses the device's VMA allocation and image-binding helpers.
|
||||||
- `Image::new` creates an allocated image. `Image::new_with_allocation` accepts an explicit `renderer_device::device::Allocation` and validates size and memory-type compatibility.
|
- `Image::new` creates an allocated image; `Image::new_unallocated` creates an unbound image. `memory_requirements` includes dedicated-allocation hints, and `bind_allocation(Arc<DeviceObject<GpuAllocation>>, offset)` validates then binds shared memory. `Image::new_with_allocation` accepts an explicit `renderer_device::device::Allocation` and validates size and memory-type compatibility.
|
||||||
|
- `MemoryGroup::new(memory_type_index, property_flags, size, alignment)` fixes the shared pool's memory type and capacity. `add(MemoryMember::Buffer(&mut buffer))` / `add(MemoryMember::Image(&mut image))` queries requirements, checks that the selected type is allowed for every member, and packs each at an aligned offset; `allocate(&device)` creates aliasable memory, then consuming `bind_all()` transfers allocation references into the resources. The group does not retain its own allocation after binding.
|
||||||
- `Image::from_swapchain_raw(image, device, format, extent, usage)` builds an image wrapper for an image owned by a swapchain; renderer calls it from `Swapchain::acquire_image`.
|
- `Image::from_swapchain_raw(image, device, format, extent, usage)` builds an image wrapper for an image owned by a swapchain; renderer calls it from `Swapchain::acquire_image`.
|
||||||
- `ImageViewDesc` describes a view's type, format, component mapping, aspect, mip range, and layer range. `Image::create_view` validates compatibility and returns `ManuallyDrop<ImageView>`.
|
- `ImageViewDesc` describes a view's type, format, component mapping, aspect, mip range, and layer range. `Image::create_view` validates compatibility and returns `ManuallyDrop<ImageView>`.
|
||||||
- `Image::create_view` resolves remaining mip/layer sentinels against the parent image, defaults an undefined view format to the image format, and checks image/view kind and format compatibility.
|
- `Image::create_view` resolves remaining mip/layer sentinels against the parent image, defaults an undefined view format to the image format, and checks image/view kind and format compatibility.
|
||||||
|
|
@ -46,6 +48,8 @@
|
||||||
- `from_swapchain_raw` records a 2D, one-mip, one-layer, single-sample, optimal-tiled image with PRESENT queue sharing and UNDEFINED initial layout; the caller supplies format, extent, and usage.
|
- `from_swapchain_raw` records a 2D, one-mip, one-layer, single-sample, optimal-tiled image with PRESENT queue sharing and UNDEFINED initial layout; the caller supplies format, extent, and usage.
|
||||||
- `from_swapchain_raw` retains a `Device` for views and device queries, but not ownership of the image handle or swapchain lifetime; the renderer's frame/swapchain token must keep the swapchain alive.
|
- `from_swapchain_raw` retains a `Device` for views and device queries, but not ownership of the image handle or swapchain lifetime; the renderer's frame/swapchain token must keep the swapchain alive.
|
||||||
- Allocated images and buffers use `DeviceObject` for Vulkan handles and allocation ownership. Keep resource destruction ordered before freeing bound memory.
|
- Allocated images and buffers use `DeviceObject` for Vulkan handles and allocation ownership. Keep resource destruction ordered before freeing bound memory.
|
||||||
|
- Unallocated resources use `Allocation::Unmanaged` until bound. Shared bind APIs are one-shot; they validate offset alignment, allowed memory type, and allocation bounds before calling the device layer. Keep `DeviceObject<GpuAllocation>` on the resource after a successful bind so Vulkan handles are destroyed before their allocation reference.
|
||||||
|
- `MemoryGroup` only accepts members whose `memory_type_bits` include its selected type (the intersection across all members); it rejects dedicated-required resources. Group offsets use aligned bump packing, and allocation ownership transfers into each bound member at `bind_all`.
|
||||||
- `Buffer` equality compares the raw buffer handle; `Image` equality compares the raw image handle. Descriptor equality is separate from resource object identity.
|
- `Buffer` equality compares the raw buffer handle; `Image` equality compares the raw image handle. Descriptor equality is separate from resource object identity.
|
||||||
- `Allocation::Owned` owns a device-managed allocation; `Shared` shares it through `Arc` and permits mutable access only when uniquely held; `Unmanaged` carries no allocation.
|
- `Allocation::Owned` owns a device-managed allocation; `Shared` shares it through `Arc` and permits mutable access only when uniquely held; `Unmanaged` carries no allocation.
|
||||||
- `ImageView` retains `Arc<Image>` so the source image outlives the view. `Image::create_view` returns `ManuallyDrop<ImageView>`; callers must transfer/drop it deliberately.
|
- `ImageView` retains `Arc<Image>` so the source image outlives the view. `Image::create_view` returns `ManuallyDrop<ImageView>`; callers must transfer/drop it deliberately.
|
||||||
|
|
|
||||||
|
|
@ -126,6 +126,98 @@ impl Buffer {
|
||||||
Ok(buffer)
|
Ok(buffer)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Creates a Vulkan buffer without allocating or binding memory.
|
||||||
|
pub fn new_unallocated(device: Device, desc: BufferDesc) -> crate::Result<Self> {
|
||||||
|
let buffer = Self::new_raw(device, &desc)?;
|
||||||
|
|
||||||
|
Ok(Self {
|
||||||
|
buffer: BufferHandle::Owned(buffer),
|
||||||
|
desc,
|
||||||
|
alloc: Allocation::Unmanaged {
|
||||||
|
requirements: std::sync::OnceLock::new(),
|
||||||
|
},
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the Vulkan memory requirements, including dedicated-allocation hints.
|
||||||
|
pub fn memory_requirements(&self) -> crate::memory::Result<crate::memory::MemoryRequirements> {
|
||||||
|
let Some(buffer) = self.owned() else {
|
||||||
|
return Err(crate::memory::Error::RequirementQuery(
|
||||||
|
crate::Error::Unspecified,
|
||||||
|
));
|
||||||
|
};
|
||||||
|
|
||||||
|
let requirements_cache = match &self.alloc {
|
||||||
|
Allocation::Unmanaged { requirements } => Some(requirements),
|
||||||
|
Allocation::Owned(_) | Allocation::Shared(_) => None,
|
||||||
|
};
|
||||||
|
|
||||||
|
if let Some(requirements) = requirements_cache.and_then(std::sync::OnceLock::get) {
|
||||||
|
return Ok(*requirements);
|
||||||
|
}
|
||||||
|
|
||||||
|
let requirements = buffer
|
||||||
|
.device()
|
||||||
|
.buffer_memory_requirements(**buffer)
|
||||||
|
.map_err(crate::memory::Error::RequirementQuery)?;
|
||||||
|
|
||||||
|
if let Some(cache) = requirements_cache {
|
||||||
|
let _ = cache.set(requirements);
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(requirements)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Binds this buffer into an aliasable allocation at an allocation-local offset.
|
||||||
|
///
|
||||||
|
/// Requirements are queried once and cached in the resource. The Vulkan bind remains the
|
||||||
|
/// authoritative validation.
|
||||||
|
pub fn bind_allocation(
|
||||||
|
&mut self,
|
||||||
|
allocation: Arc<DeviceObject<crate::device::GpuAllocation>>,
|
||||||
|
offset: u64,
|
||||||
|
) -> crate::memory::Result<()> {
|
||||||
|
if !matches!(self.alloc, Allocation::Unmanaged { .. }) {
|
||||||
|
return Err(crate::memory::Error::AlreadyBound);
|
||||||
|
}
|
||||||
|
|
||||||
|
let requirements = self.memory_requirements()?;
|
||||||
|
crate::memory::validate_binding(&requirements, allocation.as_ref(), offset)?;
|
||||||
|
let device = self.owned().expect("owned buffer").device();
|
||||||
|
|
||||||
|
self.alloc = device
|
||||||
|
.bind_buffer_at_offset(self.raw(), &requirements, allocation, offset)
|
||||||
|
.map_err(crate::memory::Error::Bind)?;
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(crate) fn validate_unallocated(&self) -> crate::memory::Result<()> {
|
||||||
|
if matches!(self.alloc, Allocation::Unmanaged { .. }) {
|
||||||
|
Ok(())
|
||||||
|
} else {
|
||||||
|
Err(crate::memory::Error::AlreadyBound)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(crate) fn validate_device(&self, device: &Device) -> crate::memory::Result<()> {
|
||||||
|
if self.owned().expect("unallocated buffer is owned").device() == device {
|
||||||
|
Ok(())
|
||||||
|
} else {
|
||||||
|
Err(crate::memory::Error::Bind(
|
||||||
|
crate::Error::AllocationDeviceMismatch,
|
||||||
|
))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(crate) fn set_shared_allocation(
|
||||||
|
&mut self,
|
||||||
|
allocation: Arc<DeviceObject<crate::device::GpuAllocation>>,
|
||||||
|
) {
|
||||||
|
debug_assert!(matches!(self.alloc, Allocation::Unmanaged { .. }));
|
||||||
|
self.alloc = Allocation::Shared(allocation);
|
||||||
|
}
|
||||||
|
|
||||||
fn new_raw(device: Device, desc: &BufferDesc) -> crate::Result<DeviceObject<vk::Buffer>> {
|
fn new_raw(device: Device, desc: &BufferDesc) -> crate::Result<DeviceObject<vk::Buffer>> {
|
||||||
let queue_families = device.queues.family_indices(desc.queue_families);
|
let queue_families = device.queues.family_indices(desc.queue_families);
|
||||||
|
|
||||||
|
|
@ -192,7 +284,9 @@ impl Buffer {
|
||||||
size,
|
size,
|
||||||
..Default::default()
|
..Default::default()
|
||||||
},
|
},
|
||||||
alloc: Allocation::Unmanaged,
|
alloc: Allocation::Unmanaged {
|
||||||
|
requirements: std::sync::OnceLock::new(),
|
||||||
|
},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -303,6 +397,200 @@ mod allocation_validation_tests {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[cfg(all(test, feature = "testing"))]
|
||||||
|
mod unallocated_memory_tests {
|
||||||
|
use ash::vk;
|
||||||
|
use renderer_device::testing::TestVulkan;
|
||||||
|
|
||||||
|
use super::{Buffer, BufferDesc};
|
||||||
|
use crate::{
|
||||||
|
device::MemoryLocation,
|
||||||
|
images::{Image, ImageDesc},
|
||||||
|
memory::{Error, MemoryGroup, MemoryMember},
|
||||||
|
};
|
||||||
|
|
||||||
|
fn buffer_desc() -> BufferDesc {
|
||||||
|
BufferDesc {
|
||||||
|
size: 256,
|
||||||
|
usage: vk::BufferUsageFlags::TRANSFER_SRC | vk::BufferUsageFlags::TRANSFER_DST,
|
||||||
|
mem_location: MemoryLocation::GpuOnly,
|
||||||
|
..Default::default()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn unallocated_resources_query_requirements_and_bind() {
|
||||||
|
let test = TestVulkan::expect();
|
||||||
|
let device = test.device().clone();
|
||||||
|
let mut buffer = Buffer::new_unallocated(device.clone(), buffer_desc())
|
||||||
|
.expect("create unallocated buffer");
|
||||||
|
let buffer_requirements = buffer.memory_requirements().expect("buffer requirements");
|
||||||
|
assert!(buffer_requirements.size >= buffer.len());
|
||||||
|
assert!(matches!(
|
||||||
|
&buffer.alloc,
|
||||||
|
crate::device::Allocation::Unmanaged { .. }
|
||||||
|
));
|
||||||
|
|
||||||
|
let mut image = Image::new_unallocated(
|
||||||
|
device.clone(),
|
||||||
|
ImageDesc {
|
||||||
|
format: vk::Format::R8G8B8A8_UNORM,
|
||||||
|
extent: vk::Extent3D {
|
||||||
|
width: 16,
|
||||||
|
height: 16,
|
||||||
|
depth: 1,
|
||||||
|
},
|
||||||
|
usage: vk::ImageUsageFlags::TRANSFER_DST,
|
||||||
|
..Default::default()
|
||||||
|
},
|
||||||
|
)
|
||||||
|
.expect("create unallocated image");
|
||||||
|
let image_requirements = image.memory_requirements().expect("image requirements");
|
||||||
|
assert!(image_requirements.size > 0);
|
||||||
|
assert!(
|
||||||
|
image
|
||||||
|
.allocation()
|
||||||
|
.is_some_and(|allocation| allocation.allocation().is_none())
|
||||||
|
);
|
||||||
|
|
||||||
|
let buffer_memory = device
|
||||||
|
.allocate_memory(&buffer_requirements, vk::MemoryPropertyFlags::empty())
|
||||||
|
.expect("allocate buffer memory");
|
||||||
|
buffer
|
||||||
|
.bind_allocation(buffer_memory, 0)
|
||||||
|
.expect("bind buffer at offset zero");
|
||||||
|
assert!(matches!(
|
||||||
|
&buffer.alloc,
|
||||||
|
crate::device::Allocation::Shared(_)
|
||||||
|
));
|
||||||
|
|
||||||
|
let image_memory = device
|
||||||
|
.allocate_memory(&image_requirements, vk::MemoryPropertyFlags::empty())
|
||||||
|
.expect("allocate image memory");
|
||||||
|
image
|
||||||
|
.bind_allocation(image_memory, 0)
|
||||||
|
.expect("bind image at offset zero");
|
||||||
|
assert!(matches!(
|
||||||
|
image.allocation(),
|
||||||
|
Some(crate::device::Allocation::Shared(_))
|
||||||
|
));
|
||||||
|
drop(image);
|
||||||
|
drop(buffer);
|
||||||
|
test.device().wait_idle().expect("wait for device idle");
|
||||||
|
test.assert_no_validation_errors();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn bind_rejects_invalid_offsets_and_rebinding() {
|
||||||
|
let test = TestVulkan::expect();
|
||||||
|
let device = test.device().clone();
|
||||||
|
|
||||||
|
let mut buffer = Buffer::new_unallocated(device.clone(), buffer_desc())
|
||||||
|
.expect("create buffer for offset checks");
|
||||||
|
let requirements = buffer.memory_requirements().expect("buffer requirements");
|
||||||
|
let allocation = device
|
||||||
|
.allocate_memory(&requirements, vk::MemoryPropertyFlags::empty())
|
||||||
|
.expect("allocate offset-check memory");
|
||||||
|
if requirements.alignment > 1 {
|
||||||
|
assert!(matches!(
|
||||||
|
buffer.bind_allocation(allocation.clone(), 1),
|
||||||
|
Err(Error::MisalignedOffset { .. })
|
||||||
|
));
|
||||||
|
}
|
||||||
|
assert!(matches!(
|
||||||
|
buffer.bind_allocation(allocation.clone(), requirements.size),
|
||||||
|
Err(Error::OutOfBounds { .. })
|
||||||
|
));
|
||||||
|
buffer
|
||||||
|
.bind_allocation(allocation.clone(), 0)
|
||||||
|
.expect("first buffer bind");
|
||||||
|
assert!(matches!(
|
||||||
|
buffer.bind_allocation(allocation, 0),
|
||||||
|
Err(Error::AlreadyBound)
|
||||||
|
));
|
||||||
|
|
||||||
|
let allocated = device
|
||||||
|
.allocate_memory(&requirements, vk::MemoryPropertyFlags::empty())
|
||||||
|
.expect("allocate for synthetic type-bits validation");
|
||||||
|
let impossible = crate::memory::MemoryRequirements {
|
||||||
|
memory_type_bits: 0,
|
||||||
|
..requirements
|
||||||
|
};
|
||||||
|
assert!(matches!(
|
||||||
|
crate::memory::validate_binding(&impossible, allocated.as_ref(), 0),
|
||||||
|
Err(Error::IncompatibleMemoryType { .. })
|
||||||
|
));
|
||||||
|
|
||||||
|
drop(buffer);
|
||||||
|
test.device().wait_idle().expect("wait for device idle");
|
||||||
|
test.assert_no_validation_errors();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn memory_group_packs_and_binds_two_buffers() {
|
||||||
|
let test = TestVulkan::expect();
|
||||||
|
let device = test.device().clone();
|
||||||
|
let mut first =
|
||||||
|
Buffer::new_unallocated(device.clone(), buffer_desc()).expect("create first buffer");
|
||||||
|
let mut second =
|
||||||
|
Buffer::new_unallocated(device.clone(), buffer_desc()).expect("create second buffer");
|
||||||
|
let first_requirements = first.memory_requirements().expect("first requirements");
|
||||||
|
let second_requirements = second.memory_requirements().expect("second requirements");
|
||||||
|
let common_bits =
|
||||||
|
first_requirements.memory_type_bits & second_requirements.memory_type_bits;
|
||||||
|
assert_ne!(common_bits, 0, "the test adapter needs a common type");
|
||||||
|
let memory_type_index = common_bits.trailing_zeros();
|
||||||
|
let alignment = first_requirements
|
||||||
|
.alignment
|
||||||
|
.max(second_requirements.alignment);
|
||||||
|
let first_end = first_requirements.size;
|
||||||
|
let expected_second_offset = first_end
|
||||||
|
.checked_add(alignment - 1)
|
||||||
|
.expect("offset arithmetic")
|
||||||
|
/ alignment
|
||||||
|
* alignment;
|
||||||
|
let size = expected_second_offset + second_requirements.size;
|
||||||
|
let mut group = MemoryGroup::new(
|
||||||
|
memory_type_index,
|
||||||
|
vk::MemoryPropertyFlags::empty(),
|
||||||
|
size,
|
||||||
|
alignment,
|
||||||
|
)
|
||||||
|
.expect("create memory group");
|
||||||
|
let first_offset = group
|
||||||
|
.add(MemoryMember::Buffer(&mut first))
|
||||||
|
.expect("pack first buffer");
|
||||||
|
let second_offset = group
|
||||||
|
.add(MemoryMember::Buffer(&mut second))
|
||||||
|
.expect("pack second buffer");
|
||||||
|
assert_eq!(first_offset, 0);
|
||||||
|
assert_eq!(second_offset, expected_second_offset);
|
||||||
|
assert_eq!(second_offset % alignment, 0);
|
||||||
|
group.allocate(&device).expect("allocate shared memory");
|
||||||
|
group.bind_all().expect("bind group members");
|
||||||
|
assert!(matches!(&first.alloc, crate::device::Allocation::Shared(_)));
|
||||||
|
assert!(matches!(
|
||||||
|
&second.alloc,
|
||||||
|
crate::device::Allocation::Shared(_)
|
||||||
|
));
|
||||||
|
assert_eq!(
|
||||||
|
first
|
||||||
|
.alloc
|
||||||
|
.allocation()
|
||||||
|
.map(crate::device::GpuAllocation::memory),
|
||||||
|
second
|
||||||
|
.alloc
|
||||||
|
.allocation()
|
||||||
|
.map(crate::device::GpuAllocation::memory),
|
||||||
|
"both buffers retain the shared Vulkan memory block"
|
||||||
|
);
|
||||||
|
drop(first);
|
||||||
|
drop(second);
|
||||||
|
test.device().wait_idle().expect("wait for device idle");
|
||||||
|
test.assert_no_validation_errors();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Byte range selected by a buffer view.
|
/// Byte range selected by a buffer view.
|
||||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||||
pub struct BufferRange {
|
pub struct BufferRange {
|
||||||
|
|
|
||||||
|
|
@ -206,6 +206,109 @@ impl Image {
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Creates a Vulkan image without allocating or binding memory.
|
||||||
|
pub fn new_unallocated(device: Device, desc: ImageDesc) -> crate::Result<Self> {
|
||||||
|
let image = Self::new_raw(device, &desc)?;
|
||||||
|
Ok(Self {
|
||||||
|
image: ImageInner::Allocated(
|
||||||
|
image,
|
||||||
|
Allocation::Unmanaged {
|
||||||
|
requirements: std::sync::OnceLock::new(),
|
||||||
|
},
|
||||||
|
),
|
||||||
|
desc,
|
||||||
|
views: Default::default(),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the Vulkan memory requirements, including dedicated-allocation hints.
|
||||||
|
pub fn memory_requirements(&self) -> crate::memory::Result<crate::memory::MemoryRequirements> {
|
||||||
|
let ImageInner::Allocated(image, allocation) = &self.image else {
|
||||||
|
return Err(crate::memory::Error::RequirementQuery(
|
||||||
|
crate::Error::Unspecified,
|
||||||
|
));
|
||||||
|
};
|
||||||
|
let requirements_cache = match allocation {
|
||||||
|
Allocation::Unmanaged { requirements } => Some(requirements),
|
||||||
|
Allocation::Owned(_) | Allocation::Shared(_) => None,
|
||||||
|
};
|
||||||
|
|
||||||
|
if let Some(requirements) = requirements_cache.and_then(std::sync::OnceLock::get) {
|
||||||
|
return Ok(*requirements);
|
||||||
|
}
|
||||||
|
|
||||||
|
let requirements = image
|
||||||
|
.device()
|
||||||
|
.image_memory_requirements(**image)
|
||||||
|
.map_err(crate::memory::Error::RequirementQuery)?;
|
||||||
|
|
||||||
|
if let Some(cache) = requirements_cache {
|
||||||
|
let _ = cache.set(requirements);
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(requirements)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Binds this image into an aliasable allocation at an allocation-local offset.
|
||||||
|
///
|
||||||
|
/// Requirements are queried once and cached in the resource. The Vulkan bind remains the
|
||||||
|
/// authoritative validation.
|
||||||
|
pub fn bind_allocation(
|
||||||
|
&mut self,
|
||||||
|
allocation: Arc<DeviceObject<crate::device::GpuAllocation>>,
|
||||||
|
offset: u64,
|
||||||
|
) -> crate::memory::Result<()> {
|
||||||
|
let ImageInner::Allocated(_, current_allocation) = &self.image else {
|
||||||
|
return Err(crate::memory::Error::AlreadyBound);
|
||||||
|
};
|
||||||
|
if !matches!(current_allocation, Allocation::Unmanaged { .. }) {
|
||||||
|
return Err(crate::memory::Error::AlreadyBound);
|
||||||
|
}
|
||||||
|
|
||||||
|
let requirements = self.memory_requirements()?;
|
||||||
|
crate::memory::validate_binding(&requirements, allocation.as_ref(), offset)?;
|
||||||
|
let ImageInner::Allocated(image, current_allocation) = &mut self.image else {
|
||||||
|
unreachable!("image kind checked above")
|
||||||
|
};
|
||||||
|
*current_allocation = image
|
||||||
|
.device()
|
||||||
|
.bind_image_at_offset(**image, &requirements, allocation, offset)
|
||||||
|
.map_err(crate::memory::Error::Bind)?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(crate) fn validate_unallocated(&self) -> crate::memory::Result<()> {
|
||||||
|
match &self.image {
|
||||||
|
ImageInner::Allocated(_, Allocation::Unmanaged { .. }) => Ok(()),
|
||||||
|
ImageInner::Allocated(_, _) | ImageInner::Swapchain(_, _) => {
|
||||||
|
Err(crate::memory::Error::AlreadyBound)
|
||||||
|
}
|
||||||
|
#[cfg(feature = "testing")]
|
||||||
|
ImageInner::Testing(_) => Err(crate::memory::Error::AlreadyBound),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(crate) fn validate_device(&self, device: &Device) -> crate::memory::Result<()> {
|
||||||
|
if self.image.device() == device {
|
||||||
|
Ok(())
|
||||||
|
} else {
|
||||||
|
Err(crate::memory::Error::Bind(
|
||||||
|
crate::Error::AllocationDeviceMismatch,
|
||||||
|
))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(crate) fn set_shared_allocation(
|
||||||
|
&mut self,
|
||||||
|
allocation: Arc<DeviceObject<crate::device::GpuAllocation>>,
|
||||||
|
) {
|
||||||
|
let ImageInner::Allocated(_, current_allocation) = &mut self.image else {
|
||||||
|
unreachable!("group members are owned images")
|
||||||
|
};
|
||||||
|
debug_assert!(matches!(current_allocation, Allocation::Unmanaged { .. }));
|
||||||
|
*current_allocation = Allocation::Shared(allocation);
|
||||||
|
}
|
||||||
|
|
||||||
fn new_with_allocation_unchecked(
|
fn new_with_allocation_unchecked(
|
||||||
device: Device,
|
device: Device,
|
||||||
image: DeviceObject<vk::Image>,
|
image: DeviceObject<vk::Image>,
|
||||||
|
|
|
||||||
|
|
@ -3,5 +3,6 @@ pub use renderer_device::{Device, Error, Result, device, make_extension};
|
||||||
pub mod buffers;
|
pub mod buffers;
|
||||||
pub mod descriptors;
|
pub mod descriptors;
|
||||||
pub mod images;
|
pub mod images;
|
||||||
|
pub mod memory;
|
||||||
pub mod pipeline;
|
pub mod pipeline;
|
||||||
pub mod util;
|
pub mod util;
|
||||||
|
|
|
||||||
411
crates/renderer_resources/src/memory.rs
Normal file
411
crates/renderer_resources/src/memory.rs
Normal file
|
|
@ -0,0 +1,411 @@
|
||||||
|
//! Shared aliasable memory allocation helpers for buffers and images.
|
||||||
|
|
||||||
|
use std::sync::Arc;
|
||||||
|
|
||||||
|
use ash::vk;
|
||||||
|
|
||||||
|
use crate::{
|
||||||
|
Device,
|
||||||
|
buffers::Buffer,
|
||||||
|
device::{DeviceObject, GpuAllocation, MemoryBinding},
|
||||||
|
images::Image,
|
||||||
|
};
|
||||||
|
|
||||||
|
/// Vulkan memory requirements returned by the device layer.
|
||||||
|
pub use crate::device::MemoryRequirements;
|
||||||
|
|
||||||
|
/// Errors from resource requirement queries, validation, and shared binding.
|
||||||
|
#[derive(Debug)]
|
||||||
|
pub enum Error {
|
||||||
|
RequirementQuery(crate::Error),
|
||||||
|
Bind(crate::Error),
|
||||||
|
Allocation(crate::Error),
|
||||||
|
MisalignedOffset {
|
||||||
|
offset: u64,
|
||||||
|
alignment: u64,
|
||||||
|
},
|
||||||
|
IncompatibleMemoryType {
|
||||||
|
memory_type_index: u32,
|
||||||
|
memory_type_bits: u32,
|
||||||
|
},
|
||||||
|
OutOfBounds {
|
||||||
|
offset: u64,
|
||||||
|
end: u64,
|
||||||
|
allocation_size: u64,
|
||||||
|
},
|
||||||
|
AlreadyBound,
|
||||||
|
AlreadyAllocated,
|
||||||
|
RequiresDedicatedAllocation,
|
||||||
|
GroupMemoryTypeMismatch {
|
||||||
|
memory_type_index: u32,
|
||||||
|
},
|
||||||
|
GroupCapacityExceeded {
|
||||||
|
requested_end: u64,
|
||||||
|
capacity: u64,
|
||||||
|
},
|
||||||
|
EmptyGroup,
|
||||||
|
NotAllocated,
|
||||||
|
InvalidGroupRequirements,
|
||||||
|
OffsetOverflow,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl core::fmt::Display for Error {
|
||||||
|
fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
|
||||||
|
match self {
|
||||||
|
Self::RequirementQuery(error) => {
|
||||||
|
write!(f, "failed to query resource memory requirements: {error}")
|
||||||
|
}
|
||||||
|
Self::Bind(error) => write!(f, "failed to bind resource memory: {error}"),
|
||||||
|
Self::Allocation(error) => write!(f, "failed to allocate group memory: {error}"),
|
||||||
|
Self::MisalignedOffset { offset, alignment } => write!(
|
||||||
|
f,
|
||||||
|
"allocation offset {offset} is not aligned to {alignment}"
|
||||||
|
),
|
||||||
|
Self::IncompatibleMemoryType {
|
||||||
|
memory_type_index,
|
||||||
|
memory_type_bits,
|
||||||
|
} => write!(
|
||||||
|
f,
|
||||||
|
"allocation memory type {memory_type_index} is not allowed by mask {memory_type_bits:#010x}"
|
||||||
|
),
|
||||||
|
Self::OutOfBounds {
|
||||||
|
offset,
|
||||||
|
end,
|
||||||
|
allocation_size,
|
||||||
|
} => write!(
|
||||||
|
f,
|
||||||
|
"resource range [{offset}, {end}) does not fit allocation size {allocation_size}"
|
||||||
|
),
|
||||||
|
Self::AlreadyBound => f.write_str("resource is already bound to an allocation"),
|
||||||
|
Self::AlreadyAllocated => f.write_str("memory group has already been allocated"),
|
||||||
|
Self::RequiresDedicatedAllocation => f.write_str(
|
||||||
|
"resources requiring dedicated allocations cannot be added to a memory group",
|
||||||
|
),
|
||||||
|
Self::GroupMemoryTypeMismatch { memory_type_index } => write!(
|
||||||
|
f,
|
||||||
|
"resource memory requirements do not include memory type {memory_type_index}"
|
||||||
|
),
|
||||||
|
Self::GroupCapacityExceeded {
|
||||||
|
requested_end,
|
||||||
|
capacity,
|
||||||
|
} => write!(
|
||||||
|
f,
|
||||||
|
"memory group capacity exceeded: requested end {requested_end}, capacity {capacity}"
|
||||||
|
),
|
||||||
|
Self::EmptyGroup => f.write_str("memory group has no members"),
|
||||||
|
Self::NotAllocated => f.write_str("memory group has not been allocated"),
|
||||||
|
Self::InvalidGroupRequirements => {
|
||||||
|
f.write_str("memory group size or alignment is invalid")
|
||||||
|
}
|
||||||
|
Self::OffsetOverflow => f.write_str("memory group offset arithmetic overflowed"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::error::Error for Error {
|
||||||
|
fn source(&self) -> Option<&(dyn std::error::Error + 'static)> {
|
||||||
|
match self {
|
||||||
|
Self::RequirementQuery(error) | Self::Bind(error) | Self::Allocation(error) => {
|
||||||
|
Some(error)
|
||||||
|
}
|
||||||
|
_ => None,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub type Result<T> = core::result::Result<T, Error>;
|
||||||
|
|
||||||
|
pub(crate) fn validate_binding(
|
||||||
|
requirements: &MemoryRequirements,
|
||||||
|
allocation: &DeviceObject<GpuAllocation>,
|
||||||
|
offset: u64,
|
||||||
|
) -> Result<()> {
|
||||||
|
let alignment = requirements.alignment;
|
||||||
|
if alignment == 0 || !offset.is_multiple_of(alignment) {
|
||||||
|
return Err(Error::MisalignedOffset { offset, alignment });
|
||||||
|
}
|
||||||
|
|
||||||
|
let memory_type_index = allocation.memory_type_index();
|
||||||
|
if requirements.memory_type_bits & (1 << memory_type_index) == 0 {
|
||||||
|
return Err(Error::IncompatibleMemoryType {
|
||||||
|
memory_type_index,
|
||||||
|
memory_type_bits: requirements.memory_type_bits,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
let end = offset
|
||||||
|
.checked_add(requirements.size)
|
||||||
|
.ok_or(Error::OutOfBounds {
|
||||||
|
offset,
|
||||||
|
end: u64::MAX,
|
||||||
|
allocation_size: allocation.size(),
|
||||||
|
})?;
|
||||||
|
|
||||||
|
if end > allocation.size() {
|
||||||
|
return Err(Error::OutOfBounds {
|
||||||
|
offset,
|
||||||
|
end,
|
||||||
|
allocation_size: allocation.size(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
enum Member<'a> {
|
||||||
|
Buffer {
|
||||||
|
resource: &'a mut Buffer,
|
||||||
|
offset: u64,
|
||||||
|
},
|
||||||
|
Image {
|
||||||
|
resource: &'a mut Image,
|
||||||
|
offset: u64,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Packs resources into a single aliasable allocation with a fixed memory type.
|
||||||
|
///
|
||||||
|
/// Members are exclusively borrowed until `bind_all` consumes the group. Their
|
||||||
|
/// allocation ownership is then transferred to each resource; the group retains
|
||||||
|
/// no allocation handle after binding.
|
||||||
|
pub struct MemoryGroup<'a> {
|
||||||
|
memory_type_index: u32,
|
||||||
|
property_flags: vk::MemoryPropertyFlags,
|
||||||
|
size: u64,
|
||||||
|
alignment: u64,
|
||||||
|
next_offset: u64,
|
||||||
|
members: Vec<Member<'a>>,
|
||||||
|
allocation: Option<Arc<DeviceObject<GpuAllocation>>>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl<'a> MemoryGroup<'a> {
|
||||||
|
/// Creates an empty group for a fixed memory type and allocation capacity.
|
||||||
|
pub fn new(
|
||||||
|
memory_type_index: u32,
|
||||||
|
property_flags: vk::MemoryPropertyFlags,
|
||||||
|
size: u64,
|
||||||
|
alignment: u64,
|
||||||
|
) -> Result<Self> {
|
||||||
|
if memory_type_index >= u32::BITS || size == 0 || alignment == 0 {
|
||||||
|
return Err(Error::InvalidGroupRequirements);
|
||||||
|
}
|
||||||
|
Ok(Self {
|
||||||
|
memory_type_index,
|
||||||
|
property_flags,
|
||||||
|
size,
|
||||||
|
alignment,
|
||||||
|
next_offset: 0,
|
||||||
|
members: Vec::new(),
|
||||||
|
allocation: None,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Adds a buffer, placing it at the next suitably aligned offset.
|
||||||
|
pub fn add_buffer(&mut self, resource: &'a mut Buffer) -> Result<u64> {
|
||||||
|
if self.allocation.is_some() {
|
||||||
|
return Err(Error::AlreadyAllocated);
|
||||||
|
}
|
||||||
|
|
||||||
|
let requirements = resource.memory_requirements()?;
|
||||||
|
self.add_member_type_check(&requirements)?;
|
||||||
|
let offset = self.next_member_offset(&requirements)?;
|
||||||
|
|
||||||
|
let next_offset = offset
|
||||||
|
.checked_add(requirements.size)
|
||||||
|
.ok_or(Error::OffsetOverflow)?;
|
||||||
|
|
||||||
|
self.members.push(Member::Buffer { resource, offset });
|
||||||
|
|
||||||
|
self.next_offset = next_offset;
|
||||||
|
Ok(offset)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Adds either resource kind through a uniform entry point.
|
||||||
|
pub fn add(&mut self, member: MemoryMember<'a>) -> Result<u64> {
|
||||||
|
match member {
|
||||||
|
MemoryMember::Buffer(resource) => self.add_buffer(resource),
|
||||||
|
MemoryMember::Image(resource) => self.add_image(resource),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Adds an image, placing it at the next suitably aligned offset.
|
||||||
|
pub fn add_image(&mut self, resource: &'a mut Image) -> Result<u64> {
|
||||||
|
if self.allocation.is_some() {
|
||||||
|
return Err(Error::AlreadyAllocated);
|
||||||
|
}
|
||||||
|
|
||||||
|
let requirements = resource.memory_requirements()?;
|
||||||
|
self.add_member_type_check(&requirements)?;
|
||||||
|
let offset = self.next_member_offset(&requirements)?;
|
||||||
|
|
||||||
|
let next_offset = offset
|
||||||
|
.checked_add(requirements.size)
|
||||||
|
.ok_or(Error::OffsetOverflow)?;
|
||||||
|
|
||||||
|
self.members.push(Member::Image { resource, offset });
|
||||||
|
|
||||||
|
self.next_offset = next_offset;
|
||||||
|
Ok(offset)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Allocates the group's shared backing memory.
|
||||||
|
pub fn allocate(&mut self, device: &Device) -> Result<()> {
|
||||||
|
if self.members.is_empty() {
|
||||||
|
return Err(Error::EmptyGroup);
|
||||||
|
}
|
||||||
|
|
||||||
|
if self.allocation.is_some() {
|
||||||
|
return Err(Error::AlreadyAllocated);
|
||||||
|
}
|
||||||
|
|
||||||
|
let requirements = MemoryRequirements {
|
||||||
|
size: self.size,
|
||||||
|
alignment: self.alignment,
|
||||||
|
memory_type_bits: 1 << self.memory_type_index,
|
||||||
|
requires_dedicated_allocation: false,
|
||||||
|
prefers_dedicated_allocation: false,
|
||||||
|
};
|
||||||
|
|
||||||
|
let allocation = device
|
||||||
|
.allocate_memory(&requirements, self.property_flags)
|
||||||
|
.map_err(Error::Allocation)?;
|
||||||
|
|
||||||
|
for member in &self.members {
|
||||||
|
let (requirements, offset) = match member {
|
||||||
|
Member::Buffer { resource, offset } => (resource.memory_requirements()?, *offset),
|
||||||
|
Member::Image { resource, offset } => (resource.memory_requirements()?, *offset),
|
||||||
|
};
|
||||||
|
validate_binding(&requirements, allocation.as_ref(), offset)?;
|
||||||
|
}
|
||||||
|
|
||||||
|
self.allocation = Some(allocation);
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Binds every member and transfers the allocation reference into each resource.
|
||||||
|
pub fn bind_all(mut self) -> Result<()> {
|
||||||
|
let allocation = self.allocation.take().ok_or(Error::NotAllocated)?;
|
||||||
|
|
||||||
|
// Validate resources first to preserve the same resource-level error mapping as
|
||||||
|
// individual binding, then prepare the ordered device batch from cached requirements.
|
||||||
|
let mut bindings = Vec::with_capacity(self.members.len());
|
||||||
|
for member in &self.members {
|
||||||
|
match member {
|
||||||
|
Member::Buffer { resource, offset } => {
|
||||||
|
resource.validate_unallocated()?;
|
||||||
|
let requirements = resource.memory_requirements()?;
|
||||||
|
validate_binding(&requirements, allocation.as_ref(), *offset)?;
|
||||||
|
resource.validate_device(allocation.device())?;
|
||||||
|
bindings.push(MemoryBinding::Buffer {
|
||||||
|
buffer: resource.raw(),
|
||||||
|
offset: *offset,
|
||||||
|
requirements,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
Member::Image { resource, offset } => {
|
||||||
|
resource.validate_unallocated()?;
|
||||||
|
let requirements = resource.memory_requirements()?;
|
||||||
|
validate_binding(&requirements, allocation.as_ref(), *offset)?;
|
||||||
|
resource.validate_device(allocation.device())?;
|
||||||
|
bindings.push(MemoryBinding::Image {
|
||||||
|
image: resource.raw(),
|
||||||
|
offset: *offset,
|
||||||
|
requirements,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if let Err(bind_error) = allocation
|
||||||
|
.device()
|
||||||
|
.bind_group_at_offset(&allocation, &bindings)
|
||||||
|
{
|
||||||
|
// A Vulkan bind failure leaves its earlier prefix bound; retain its allocation just
|
||||||
|
// as repeated `bind_allocation` calls would have done.
|
||||||
|
for (index, member) in self.members.into_iter().enumerate() {
|
||||||
|
if index >= bind_error.bound_count {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
match member {
|
||||||
|
Member::Buffer { resource, .. } => {
|
||||||
|
resource.set_shared_allocation(allocation.clone());
|
||||||
|
}
|
||||||
|
Member::Image { resource, .. } => {
|
||||||
|
resource.set_shared_allocation(allocation.clone());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return Err(Error::Bind(bind_error.error));
|
||||||
|
}
|
||||||
|
|
||||||
|
for member in self.members {
|
||||||
|
match member {
|
||||||
|
Member::Buffer { resource, .. } => {
|
||||||
|
resource.set_shared_allocation(allocation.clone());
|
||||||
|
}
|
||||||
|
Member::Image { resource, .. } => {
|
||||||
|
resource.set_shared_allocation(allocation.clone());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// `allocation` drops here. Only successfully bound resources retain it.
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
fn add_member_type_check(&self, requirements: &MemoryRequirements) -> Result<()> {
|
||||||
|
if requirements.requires_dedicated_allocation {
|
||||||
|
return Err(Error::RequiresDedicatedAllocation);
|
||||||
|
}
|
||||||
|
if requirements.memory_type_bits & (1 << self.memory_type_index) == 0 {
|
||||||
|
return Err(Error::GroupMemoryTypeMismatch {
|
||||||
|
memory_type_index: self.memory_type_index,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
fn next_member_offset(&self, requirements: &MemoryRequirements) -> Result<u64> {
|
||||||
|
if requirements.alignment == 0 {
|
||||||
|
return Err(Error::InvalidGroupRequirements);
|
||||||
|
}
|
||||||
|
let alignment = checked_lcm(requirements.alignment, self.alignment)?;
|
||||||
|
let offset = (self
|
||||||
|
.next_offset
|
||||||
|
.checked_add(alignment - 1)
|
||||||
|
.ok_or(Error::OffsetOverflow)?
|
||||||
|
/ alignment)
|
||||||
|
.checked_mul(alignment)
|
||||||
|
.ok_or(Error::OffsetOverflow)?;
|
||||||
|
let end = offset
|
||||||
|
.checked_add(requirements.size)
|
||||||
|
.ok_or(Error::OffsetOverflow)?;
|
||||||
|
if end > self.size {
|
||||||
|
return Err(Error::GroupCapacityExceeded {
|
||||||
|
requested_end: end,
|
||||||
|
capacity: self.size,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
Ok(offset)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn checked_lcm(lhs: u64, rhs: u64) -> Result<u64> {
|
||||||
|
let gcd = {
|
||||||
|
let (mut lhs, mut rhs) = (lhs, rhs);
|
||||||
|
while rhs != 0 {
|
||||||
|
(lhs, rhs) = (rhs, lhs % rhs);
|
||||||
|
}
|
||||||
|
lhs
|
||||||
|
};
|
||||||
|
lhs.checked_div(gcd)
|
||||||
|
.and_then(|lhs| lhs.checked_mul(rhs))
|
||||||
|
.ok_or(Error::OffsetOverflow)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A resource accepted by [`MemoryGroup::add`].
|
||||||
|
pub enum MemoryMember<'a> {
|
||||||
|
Buffer(&'a mut Buffer),
|
||||||
|
Image(&'a mut Image),
|
||||||
|
}
|
||||||
202
design-plans/rendergraph-transient-aliasing.md
Normal file
202
design-plans/rendergraph-transient-aliasing.md
Normal file
|
|
@ -0,0 +1,202 @@
|
||||||
|
# Plan: Correct Transient-Resource Aliasing in `renderer_rendergraph`
|
||||||
|
|
||||||
|
Status: proposed — rev 3 (code-verified; rev-2 review + follow-up comments applied)
|
||||||
|
Date: 2026-10-26
|
||||||
|
Scope: `crates/renderer_rendergraph`, `crates/renderer_resources`, `crates/renderer_device`
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Fix the broken transient-resource aliasing in the rendergraph: lifetimes must be
|
||||||
|
derived from the graph's real ordering (dependency DAG + batches, not command
|
||||||
|
indices), alias reuse must be synchronized with barriers, and the resources
|
||||||
|
crate must gain the infrastructure to compute correct memory requirements so
|
||||||
|
correctly sized pools can be allocated and bound to graph-created resources.
|
||||||
|
|
||||||
|
## Diagnosis (verified against code)
|
||||||
|
|
||||||
|
| # | Defect | Evidence |
|
||||||
|
|---|--------|----------|
|
||||||
|
| 1 | Lifetimes are command-index intervals (`first_command`/`last_command`), but commands within a `topo_batches` batch are conceptually concurrent — two resources used by different commands of the same batch are wrongly treated as non-overlapping | `graph.rs:66-79, 862-877` |
|
||||||
|
| 2 | Batch index is **not** a happens-before proof: the DAG only contains edges for conflicting same-resource accesses and is discarded after `topo_batches`; e.g. test `independent_commands_share_a_batch...` produces batches `[0,2],[1]` with no edge `2→1` | `graph.rs:643-657, 683-697, 699-714` |
|
||||||
|
| 3 | Queue transfers unmodeled: `QueueTransfer` exists but every barrier sets `None`; no command→queue assignment exists anywhere (`derive_queue_capabilities` is a resource-wide stage union, not a per-command queue) | `graph.rs:41-58, 792, 822, 1461-1541` |
|
||||||
|
| 4 | No alias-safety barriers: `build_barriers` only tracks prior uses of the *same* resource; nothing orders last use of alias A before first use of alias B sharing memory | `graph.rs:728-835` |
|
||||||
|
| 5 | Packing uses `texture_size_estimate`/`desc.size`, ignoring real `VkMemoryRequirements`, alignment, memory-type bits, `bufferImageGranularity`, and dedicated-allocation requirements; offsets are never bound to resources | `graph.rs:888-919, 1543-1588` |
|
||||||
|
| 6 | No offset-aware bind path: `bind_buffer_memory`/`bind_image_memory` bind whole VMA allocations; `Allocation::Shared(Arc<DeviceObject<GpuAllocation>>)` exists but is unused for pooling | `device.rs:209-214, 766-778` |
|
||||||
|
|
||||||
|
## Design decisions (post-review)
|
||||||
|
|
||||||
|
### D1 — Happens-before, not batch numbers
|
||||||
|
|
||||||
|
Retain the dependency adjacency from `GraphCompiler::dependencies()` in
|
||||||
|
`CompiledGraph` and add an explicit `happens_before(a, b)` reachability query.
|
||||||
|
Alias A→B is permitted only if *every* terminal use of A happens-before *every*
|
||||||
|
initial use of B. Batch intervals are computed (for reporting/limits) but never
|
||||||
|
used as the safety criterion. Same-batch uses always interfere.
|
||||||
|
|
||||||
|
`happens_before` is computed as a **transitive closure at compile time**: one
|
||||||
|
bitset per command (`Vec<FixedBitSet>`-style reachability rows, or bitset
|
||||||
|
columns if that packs better), built once in `compile()` from the Kahn
|
||||||
|
adjacency. Queries are then O(1) and deterministic. Rationale: the command
|
||||||
|
count is bounded by the recorded command list (thousands at most), closure is
|
||||||
|
O(V·E/64) once, and per-query BFS would be re-run per alias-candidate pair
|
||||||
|
during placement for no benefit. Note the closure's semantics: edges encode
|
||||||
|
only conflicting same-resource accesses, so reachability means exactly
|
||||||
|
"provably ordered via a resource-hazard chain" — precisely the aliasing-safety
|
||||||
|
criterion. Commands mutually unreachable (including same-batch pairs) interfere.
|
||||||
|
|
||||||
|
### D2 — Phase split
|
||||||
|
|
||||||
|
`compile()` stays device-free: dependency order, batch lifetimes, interference.
|
||||||
|
A new device-dependent `plan_placements(device)` (called from `resolve`) does
|
||||||
|
requirements queries, pool placement, alias-barrier construction, and binding.
|
||||||
|
Tests that call `bind_lifetimes()` directly are updated.
|
||||||
|
|
||||||
|
### D3 — Conservative queue policy
|
||||||
|
|
||||||
|
There is no command→queue assignment today and all commands record into one
|
||||||
|
command buffer. Aliasing is restricted to the single-queue case; any pair whose
|
||||||
|
accesses can't be proven ordered on one queue interferes. Cross-queue aliasing
|
||||||
|
(with ownership transfers/semaphores) is explicitly out of scope and documented
|
||||||
|
as a follow-up — no fake `QueueTransfer` support.
|
||||||
|
|
||||||
|
### D4 — Alias synchronization done correctly
|
||||||
|
|
||||||
|
- For aliased **buffers**: emit a `VkMemoryBarrier2` (new `MemoryBarrier` +
|
||||||
|
`PipelineBarrier.memory_barriers`) before the new occupant's *first-use
|
||||||
|
command* (a concrete command, not "batch start"), with `src` = old occupant's
|
||||||
|
aggregated terminal stage/access (writes; execution-only if read-only),
|
||||||
|
`dst` = new occupant's aggregated first-use scope.
|
||||||
|
- For aliased **images**: fold the old occupant's terminal scope into the new
|
||||||
|
image's first-use image barrier (`srcStageMask`/`srcAccessMask` = old scope,
|
||||||
|
`oldLayout = UNDEFINED`) — a bare memory barrier does not chain with a
|
||||||
|
following `srcStageMask=NONE` layout transition.
|
||||||
|
- Aggregate multi-scope accesses per command (OR all stage/access for the
|
||||||
|
resource at that command).
|
||||||
|
- Placement is range-based: a new resource's `[offset, offset+size)` is tested
|
||||||
|
against **all** occupied ranges; barriers include every old occupant
|
||||||
|
intersecting the range.
|
||||||
|
|
||||||
|
### D5 — Vulkan aliasing correctness details
|
||||||
|
|
||||||
|
- `bufferImageGranularity`: **separate pools per class** (linear-only and
|
||||||
|
optimal-only) is the primary strategy — first-fit stays simple and no
|
||||||
|
padding logic is needed. Intra-pool offsets are rounded only to each
|
||||||
|
resource's `alignment`. (Intra-pool interleaving with granularity-driven
|
||||||
|
offset rounding is a deliberate non-goal; revisit only if pool-class split
|
||||||
|
waste ever shows up in practice.)
|
||||||
|
- Set `VK_IMAGE_CREATE_ALIAS_BIT` on pooled images.
|
||||||
|
- Exclude from pools: `requiresDedicatedAllocation` resources, explicit
|
||||||
|
`AllocationStrategy::Dedicated` (incl. the 64 MiB heuristic), and
|
||||||
|
host-visible (`CpuToGpu`/`GpuToCpu`) resources — `Buffer::map()` returns
|
||||||
|
slices from the allocation base and would silently map wrong bytes unless
|
||||||
|
suballocation offsets are tracked.
|
||||||
|
- Created-but-unused resources: skip creation entirely (or allocate outside the
|
||||||
|
pool) — decided as "skip unused transients", with a test.
|
||||||
|
- Pooled images must have `GraphTextureDesc.layout == UNDEFINED`:
|
||||||
|
`plan_placements` rejects non-UNDEFINED declared initial layouts on pooled
|
||||||
|
images with a graph error (a created image's declared initial layout cannot
|
||||||
|
be honored by reused memory, whose contents are always undefined at first
|
||||||
|
use). Non-pooled images keep today's behavior.
|
||||||
|
|
||||||
|
### D6 — Device/resources infra (new APIs, not existing ones)
|
||||||
|
|
||||||
|
- `renderer_device`: `allocate_memory(requirements, …)` via
|
||||||
|
`vk_mem::Alloc::allocate_memory` with `CAN_ALIAS`; `find_memory_type_index`;
|
||||||
|
`bind_buffer_at_offset`/`bind_image_at_offset` via `bind_*_memory2` with
|
||||||
|
`allocation_local_offset`; require API ≥1.1 or enable the full
|
||||||
|
`get_memory_requirements2`/`bind_memory2`/`dedicated_allocation` extension
|
||||||
|
set with a clear error otherwise (same guard pattern as the existing
|
||||||
|
dedicated-allocation check in `device.rs`); define shared-allocation
|
||||||
|
ownership/cleanup: at bind time the group **transfers** the
|
||||||
|
`Arc<DeviceObject<GpuAllocation>>` into each bound resource's
|
||||||
|
`Allocation::Shared` and drops its own reference — the underlying memory is
|
||||||
|
freed when the last bound resource drops (via `DeviceObject` drop), matching
|
||||||
|
the existing `Owned`/`Shared` semantics. The group never retains a post-bind
|
||||||
|
handle.
|
||||||
|
- `renderer_resources`: symmetric `Buffer::new_unallocated` /
|
||||||
|
`Image::new_unallocated`, `memory_requirements()` (incl.
|
||||||
|
`VkMemoryDedicatedRequirements`), `bind_allocation(alloc, offset)` with
|
||||||
|
alignment/type-bits/bounds validation; a `memory` module with
|
||||||
|
`MemoryRequirements` + `MemoryGroup` (bind → `Allocation::Shared`). Pool
|
||||||
|
groups are keyed by a memory type that is in **every member's**
|
||||||
|
`memoryTypeBits` (intersection), not by any single member's individually
|
||||||
|
best type; a member whose bits exclude the group's type moves to another
|
||||||
|
group or its own allocation. New `renderer_resources::Error` variants for
|
||||||
|
failed requirement queries / bind-time validation (alignment, type-bits,
|
||||||
|
bounds); `renderer_rendergraph` maps these into new `GraphError` variants
|
||||||
|
(e.g. `PlacementFailed(ResourceId)`, `BindFailed(ResourceId)`) — bind/placement
|
||||||
|
failures surface through `resolve`/`plan_placements`'s existing
|
||||||
|
`renderer_resources::Result` boundary and map at the call site. No graph
|
||||||
|
types leak into this crate.
|
||||||
|
|
||||||
|
## Task breakdown (ordered, dependency-explicit)
|
||||||
|
|
||||||
|
1. **`renderer_device`**: memory-type query, aliasable shared allocation,
|
||||||
|
offset-aware bind, API/extension guards, error variants, ownership docs.
|
||||||
|
Tests with `TestVulkan` (`--features testing`).
|
||||||
|
2. **`renderer_resources`**: unbound create + requirements + bind APIs for
|
||||||
|
buffer *and* image; `memory` module (`MemoryRequirements`, `MemoryGroup`);
|
||||||
|
update crate AGENTS.md. Existing `Buffer::new`/`Image::new` unchanged.
|
||||||
|
3. **`renderer_rendergraph` (analysis)**: retain dependency DAG in
|
||||||
|
`CompiledGraph`; `happens_before` query; batch-mapped
|
||||||
|
`ResourceLifetime {first_batch, last_batch}`; replace `memory_offset: u64`
|
||||||
|
with a typed `Placement {offset, size, alignment, pool, aliased}`; imported
|
||||||
|
resources tracked but excluded from packing; queue-conservatism rule per D3.
|
||||||
|
4. **`renderer_rendergraph` (synchronization)**: `MemoryBarrier` +
|
||||||
|
`PipelineBarrier.memory_barriers`; recorder emits `vk::MemoryBarrier2` in
|
||||||
|
`DependencyInfo` (anchored via the existing `recording_steps`
|
||||||
|
`before_command` mechanism, like today's image/barrier steps); alias-barrier
|
||||||
|
construction per D4 (buffer and image variants, aggregation,
|
||||||
|
first-use-command anchoring); new `GraphError` variants for placement/bind
|
||||||
|
failures per D6, mapped from `renderer_resources::Error` at the
|
||||||
|
`resolve` boundary.
|
||||||
|
5. **`renderer_rendergraph` (placement)**: `plan_placements(device)` **replaces
|
||||||
|
`resolve()`'s current eager create+allocate loop** (graph.rs:1218-1276) for
|
||||||
|
pooled resources — create unbound, query requirements, group by a memory
|
||||||
|
type common to every member's `memoryTypeBits`, granularity-aware first-fit
|
||||||
|
over ranges (D4/D5, pool-per-class), allocate groups, bind, store
|
||||||
|
placements in `CompiledGraph`. Non-poolable resources (dedicated,
|
||||||
|
host-visible, imports) keep an allocate-and-bind-whole path equivalent to
|
||||||
|
today's. "Skip unused transients" lands here: created resources absent from
|
||||||
|
`lifetimes` are never created (no behavior change needed in
|
||||||
|
`record_commands`, which already skips ids absent from `lifetimes` at
|
||||||
|
graph.rs:1290). Transactional and idempotent; remove
|
||||||
|
`texture_size_estimate` from allocation decisions.
|
||||||
|
6. **Docs/tests**: update stale TODOs (`graph.rs:882`, `1315-1319`), document
|
||||||
|
the single-queue submission contract and the future multi-queue invariant
|
||||||
|
(alias transitions become semaphore edges).
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
- Device-free unit tests: same-batch interference; unordered-but-distinct-batch
|
||||||
|
interference; proven-ordered reuse succeeds; granularity boundary (buffer
|
||||||
|
next to optimal image); range-overlap first-fit; dedicated/host-visible
|
||||||
|
exclusion; unused-transient skip; pooled-image non-UNDEFINED initial layout
|
||||||
|
rejected; alias-barrier scope correctness incl. read-only old occupant and
|
||||||
|
image `UNDEFINED` transition chaining.
|
||||||
|
- GPU-gated (`tests/graph_validation.rs`): buffer-buffer and image-image alias
|
||||||
|
reuse through the real barrier path, submit, `device_wait_idle`,
|
||||||
|
`assert_no_validation_errors`; teardown of shared allocation after all
|
||||||
|
resources drop.
|
||||||
|
- Commands: `cargo check --workspace`, `cargo test -p renderer_rendergraph`,
|
||||||
|
`cargo test -p renderer_resources --features testing`,
|
||||||
|
`cargo test -p renderer_device --features testing` (from workspace root,
|
||||||
|
pinned toolchain).
|
||||||
|
|
||||||
|
## Review history
|
||||||
|
|
||||||
|
- Drafted independently by deepseek/planner and luna/planner; cross-reviewed by
|
||||||
|
deepseek/reviewer and luna/reviewer (both verdicts: CHANGES REQUIRED).
|
||||||
|
- Blockers folded into this revision: batch-index ≠ happens-before (D1),
|
||||||
|
compile/resolve phase split (D2), `bufferImageGranularity` handling (D5),
|
||||||
|
alias-barrier/layout-transition chaining (D4), offset-aware bind APIs are new
|
||||||
|
work not existing capability (D6), queue model restricted to single-queue
|
||||||
|
(D3).
|
||||||
|
- Rev 3: diagnosis re-verified against code and `vk-mem 0.5.0` sources
|
||||||
|
(all six rows and the D6 API names confirmed). Follow-up comments folded in:
|
||||||
|
`happens_before` = compile-time transitive closure (D1), pool-per-class
|
||||||
|
granularity strategy picked (D5), pooled-image UNDEFINED initial-layout
|
||||||
|
rule (D5), pool groups keyed by intersecting `memoryTypeBits` (D6),
|
||||||
|
`MemoryGroup` transfers the `Arc` at bind time instead of retaining it (D6),
|
||||||
|
placement/bind error surfacing through `GraphError` (D6/task 4),
|
||||||
|
`plan_placements` explicitly replaces `resolve`'s eager loop (task 5).
|
||||||
Loading…
Reference in a new issue