vidya/design-plans/gpu-allocator-to-vma-migration.md

# Migration Plan: gpu-allocator → VMA via vk-mem

Status: proposed — rev 2 (Luna review fixes applied)
Date: 2026-10-26
Scope: workspace-wide memory-allocator swap

## Goal

Replace the `gpu-allocator` crate with VMA (Vulkan Memory Allocator) via the
`vk-mem` bindings crate, preserving current allocation behavior (buffer/image
create/bind/destroy, mapping, dedicated allocations) and the existing real-Vulkan
tests.

## Architecture decision

### Requirements

- Hard: remove `gpu-allocator`; use `vk-mem`/VMA; preserve the dependency
  direction `bevy_vulkan_render` → `renderer` → `renderer_resources` →
  `renderer_device`; keep buffer/image create/bind/destroy and mapping behavior;
  keep real-Vulkan tests working.
- Nice: dedicated allocations, allocation naming/stats, defragmentation (not
  required now).

### Constraints

- Nightly toolchain, edition 2024.
- `renderer_resources` must not depend on `renderer`.
- `renderer_device` owns device/allocator state.
- `renderer_resources`/`renderer` should not expose `vk_mem` types.
- The `testing` feature gates real-device tests.
- No per-frame allocation is acceptable on current paths.
- `renderer_rendergraph` is also a workspace member and directly uses
  `gpu_allocator::MemoryLocation`, so it must be included (it is not mentioned in
  the workspace guide).

### Current architecture

- Deps: root `Cargo.toml:45` git `gpu-allocator` (line 44 has a commented
  `# vk-mem = "0.5.0"`); `renderer_device/Cargo.toml:12`,
  `renderer_resources/Cargo.toml:12`, `renderer/Cargo.toml:27`,
  `renderer_rendergraph/Cargo.toml:8`. `Cargo.lock:3423-3435` is the git package;
  `Cargo.lock:3409-3420` is the registry copy pulled by `wgpu-hal` via the Bevy
  dev-dependency.
- Allocator state: `crates/renderer_device/src/device.rs:151`
  `pub alloc2: Mutex<gpu_allocator::vulkan::Allocator>`, initialized at
  `:441-456` (`AllocatorCreateDesc`, `AllocationSizes::new(8MB,64MB)` with 256MB
  maxima). `GpuAllocation` is the private alias at `:111`; `AllocationStrategy`
  at `:114-124`; `Allocation::{Owned,Shared,Unmanaged}` at `:126-147`. Free path
  is `ExternallyManagedObject<Device> for GpuAllocation` at `:1074-1077`, which
  calls `alloc2.lock().free(...)`.
- Errors: `crates/renderer_device/src/lib.rs:81`
  `GpuAllocatorError(#[from] gpu_allocator::AllocationError)`.
- Buffer flow: `crates/renderer_resources/src/buffers.rs:4` imports
  `AllocationScheme`; `:20,68` `MemoryLocation`; `:108-122` allocates from
  `alloc2`; `:128-130` binds with raw `vkBindBufferMemory`; `:137` wraps in
  `Allocation::Owned(DeviceObject<GpuAllocation>)`; `:164-181` maps via
  `mapped_slice`/`mapped_slice_mut`.
- Image flow: `crates/renderer_resources/src/images.rs:16` imports VMA-named
  `AllocationCreateDesc/Scheme`; `:35,126,289` `MemoryLocation`; `:191-205`
  allocates and wraps; `:217-223` binds; `:244-260` validates size/memory-type
  for `new_with_allocation`; `:418-428` exposes allocation.
- Dedicated logic: `buffers.rs:116-121` uses
  `AllocationScheme::DedicatedBuffer(buffer)`; `images.rs:196-199` uses
  `DedicatedImage(image)`; `renderer_rendergraph/src/graph.rs:1352-1357` chooses
  dedicated at 64 MiB.
- Direct test allocations: `crates/renderer_device/src/testing.rs:312-319` and
  `:351-357` allocate/free directly through `device.alloc2`.
- Other type consumers: `renderer/src/rendering/mod.rs:8,56,77,87,97`;
  `renderer/src/egui.rs:8,74-75,140,157,510,531,538,545,555`;
  `renderer/src/render_graph/graph_builder.rs:51,60,69,87,168-183`;
  `renderer_rendergraph/src/resources.rs:16,27,46,69`; tests
  `renderer_resources/tests/managed_set_validation.rs:26` and
  `renderer_rendergraph/tests/graph_validation.rs:31`.
- No stats/reporting or defragmentation call sites exist today. `Buffer`/`Image`
  drop field order: buffer/image handle first, then allocation
  (`buffers.rs:74-78`, `images.rs:144-148`).

### Alternatives considered

1. **Swap allocator inside `renderer_device`, add a private wrapper and a
   workspace-owned `MemoryLocation` (chosen)**: least invasive, upper crates stay
   VMA-free.
2. Re-export `vk_mem` types as public resource API: least code, but leaks VMA
   into `renderer_resources`/`renderer` and locks the workspace API to `vk-mem`.
3. Use `vk_mem::create_buffer/create_image` directly in `renderer_resources`:
   fewer Vulkan calls and idiomatic VMA, but larger ownership refactor and
   couples the resource crate to `vk_mem`.
4. Add a memory-backend trait: easy A/B testing, but unnecessary indirection and
   lifetime complexity for a one-time swap.

### Decision rationale (feasibility verified)

- `vk-mem 0.5.0` bundles **VMA 3.3.0** and depends on **ash 0.38** (workspace ash
  is `0.38.0+1.3.281`), so there is one ash and no duplicate-ash problem.
- `vk_mem::Allocator` is `Send + Sync` and internally synchronized unless
  `EXTERNALLY_SYNCHRONIZED` is set; keep the `Mutex`.
- `vk_mem::Allocation` is `Copy, Send, Sync` and has no inherent
  `size/memory/offset` accessors, so the wrapper must retain `AllocationInfo`.

## Target design

- `renderer_device::device::MemoryLocation { Unknown, GpuOnly, CpuToGpu,
  GpuToCpu }` with `Default = Unknown`, `Copy/Eq/Hash`; use it as the public
  `mem_location` type.
- `pub struct GpuAllocation { raw: vk_mem::Allocation, memory: vk::DeviceMemory,
  offset: u64, size: u64, memory_type_index: u32, mapped: Option<SendSyncPtr> }`.
  `SendSyncPtr` is a private `NonNull<u8>` wrapper with unsafe `Send/Sync`,
  mirroring `gpu_allocator`'s pointer wrapper. Implement `Debug`,
  `memory()/offset()/size()/memory_type_index()/mapped_slice()/mapped_slice_mut()`.
- `pub struct AllocationDesc<'a> { name: &'a str, location: MemoryLocation,
  strategy: AllocationStrategy }` and
  `DeviceInner::{allocate_for_buffer, allocate_for_image, bind_buffer_memory,
  bind_image_memory}`. Note the call shapes: `allocate_memory_for_buffer/image`
  live on the `vk_mem::Alloc` trait (import `vk_mem::Alloc as _`) and are
  `unsafe` in `vk-mem 0.5.0`; `bind_buffer_memory`/`bind_image_memory` are
  inherent (also `unsafe`) `Allocator` methods, not trait methods.
- `DeviceInner.alloc2: Mutex<vk_mem::Allocator>` remains the single allocator
  owner. Keep `Allocation::{Owned(DeviceObject<GpuAllocation>),
  Shared(Arc<...>), Unmanaged}` unchanged in shape.
- `renderer_resources`/`renderer`/`renderer_rendergraph` use only
  `renderer_device::device::MemoryLocation`, `Allocation`, and the new `Device`
  helpers; no `vk_mem` imports outside `renderer_device`.

Suggested interface:

```rust
impl DeviceInner {
    pub fn allocate_for_buffer(&self, buffer: vk::Buffer, desc: AllocationDesc<'_>) -> Result<GpuAllocation>;
    pub fn allocate_for_image(&self, image: vk::Image, desc: AllocationDesc<'_>) -> Result<GpuAllocation>;
    pub fn bind_buffer_memory(&self, alloc: &GpuAllocation, buffer: vk::Buffer) -> Result<()>;
    pub fn bind_image_memory(&self, alloc: &GpuAllocation, image: vk::Image) -> Result<()>;
}
```

## Dependency changes

- Root `Cargo.toml`: add `vk-mem = "0.5.0"` to `[workspace.dependencies]` (the
  commented line at `:44` anticipates this); remove the git `gpu-allocator` entry
  in phase 3.
- `renderer_device/Cargo.toml`: add `vk-mem = { workspace = true }`; remove
  `gpu-allocator` in phase 2 (after the swap compiles).
- `renderer_resources/Cargo.toml`, `renderer/Cargo.toml`,
  `renderer_rendergraph/Cargo.toml`: remove `gpu-allocator` in phase 3.
- Ash alignment: `vk-mem 0.5.0` pins ash 0.38; an ash upgrade requires a
  matching `vk-mem` release or fork.

## API mapping

| gpu-allocator | VMA / `vk-mem` |
|---|---|
| `AllocatorCreateDesc` instance/device/physical_device | `AllocatorCreateInfo::new(&instance, &device, physical_device)` |
| `debug_settings`, allocation sizes | VMA compile-time debug macros; `preferred_large_heap_block_size` only (loss of min/max block tuning) |
| `buffer_device_address` | `AllocatorCreateFlags::BUFFER_DEVICE_ADDRESS` (off here) |
| `MemoryLocation::{Unknown,GpuOnly,CpuToGpu,GpuToCpu}` | `MemoryUsage::{Unknown,GpuOnly,CpuToGpu,GpuToCpu}` |
| `AllocationScheme::GpuAllocatorManaged` | default allocation (`AllocationCreateFlags::empty()`) |
| `DedicatedBuffer(buffer)` / `DedicatedImage(image)` | `DEDICATED_MEMORY` + `allocate_memory_for_buffer/image` (passes `VkMemoryDedicatedAllocateInfo` when API version ≥ 1.1) |
| `AllocationCreateDesc.linear` | implicit: use `allocate_memory_for_buffer/image`, not generic allocation |
| `Allocation.memory()/offset()/size()/memory_type_index()` | `AllocationInfo.device_memory/offset/size/memory_type` captured at allocation |
| `mapped_slice()/mapped_slice_mut()` | `AllocationInfo.mapped_data` from `MAPPED` allocation, or `Allocator::map_memory` |
| `Allocator::free(allocation)` | `Allocator::free_memory(&mut allocation)` |
| `bind_buffer_memory`/`bind_image_memory` | `Allocator::{bind_buffer_memory, bind_image_memory}` |
| stats report | `calculate_statistics()` / `get_heap_budgets()`; no allocation names in `vk-mem 0.5.0` |
| debug names | unchanged: `DeviceObject::new_debug_named` sets Vulkan object names; VMA allocation names are not exposed |

## Migration phases

### Phase 0 — baseline

Run `cargo check --workspace`, `cargo test -p renderer_device --features
testing`, inspect `git status`. No edits.

### Phase 1 — dependency/type scaffolding (non-breaking)

Add `vk-mem = "0.5.0"` to root `[workspace.dependencies]`; add
`vk-mem = { workspace = true }` to `renderer_device/Cargo.toml`; add
`MemoryLocation`, `GpuAllocation` skeleton, and `AllocationDesc` in
`crates/renderer_device/src/device.rs` while leaving the old allocator
untouched.

Verify: `cargo check -p renderer_device`.

### Phase 2 — atomic allocator swap (one compile unit)

In `crates/renderer_device/src/device.rs` replace `alloc2`/creation/
`GpuAllocation` alias/free impl with VMA; remove `GpuAllocatorError` from
`crates/renderer_device/src/lib.rs:81`; rewrite `testing.rs:312-319,351-357`;
in `crates/renderer_resources/src/buffers.rs` and `images.rs` replace
`alloc2.allocate` + raw bind with `allocate_for_buffer/image` + `bind_*`, adding
handle guards in the same pass (raw handle wrapped in `DeviceObject` immediately
after creation, allocation ownership established before binding — see
implementation notes), but temporarily keep the public fields as
`gpu_allocator::MemoryLocation` and convert at the call site.

Verify: `cargo check --workspace` and `cargo test -p renderer_device --features
testing`.

### Phase 3 — public type/dependency cleanup

Change `BufferDesc.mem_location`/`ImageDesc.mem_location` to
`renderer_device::device::MemoryLocation`; update `renderer/src/rendering/mod.rs`,
`renderer/src/egui.rs`, `renderer/src/render_graph/graph_builder.rs`,
`renderer_rendergraph/src/resources.rs`, and both tests
(`renderer_resources/tests/managed_set_validation.rs`,
`renderer_rendergraph/tests/graph_validation.rs`); remove `gpu-allocator` from
`renderer_resources`, `renderer`, `renderer_rendergraph`, and root manifests.

Verify: `cargo check --workspace`; `rg -n 'gpu_allocator|gpu-allocator' --glob
'!Cargo.lock'` should be empty.

### Phase 4 — hardening/docs

- Ensure `GpuAllocation` is `Send + Sync`.
- Add handle guards: wrap raw buffer/image handles in `DeviceObject` right after
  creation and establish allocation ownership before binding, so allocation and
  bind failures never leak (see implementation notes).
- Add a validation test for `CpuToGpu` mapping that asserts the allocated memory
  type is `HOST_VISIBLE | HOST_COHERENT` and round-trips a CPU write through
  `map_mut`, plus dedicated buffers/images.
- Update `crates/renderer_device/AGENTS.md`,
  `crates/renderer_resources/AGENTS.md`, `crates/renderer/AGENTS.md`, and the
  root workspace guide to say VMA/`vk-mem`.
- Optionally rename `alloc2` to `allocator`.

Verify: full checklist below.

## Implementation notes

- Allocator creation: `unsafe { vk_mem::Allocator::new(create_info) }?`; set
  `preferred_large_heap_block_size` to ~64 MiB (VMA's default is 256 MiB).
  `vulkan_api_version` must not exceed the instance's requested version or the
  selected physical device's supported version: record the instance request
  (add `pub api_version: u32` to `InstanceInner` when building `ApplicationInfo`
  in `instance.rs:126-135`) and clamp it with the physical device's
  `limits.api_version` at device selection, passing the effective minimum (a
  safe floor is 1.1 when both sides support it, else 1.0). Dedicated-allocation
  structs require API ≥ 1.1; on Vulkan 1.0 devices require
  `VK_KHR_dedicated_allocation` (+ `VK_KHR_get_memory_requirements2`) at device
  creation or skip `DEDICATED_MEMORY` there.
- Do not enable `EXTERNALLY_SYNCHRONIZED`; keep the `Mutex`.
- Allocation flags: `MAPPED` for `CpuToGpu`/`GpuToCpu`, plus `DEDICATED_MEMORY`
  for `AllocationStrategy::Dedicated`.
- **Coherence policy (required)**: VMA's `CpuToGpu`/`GpuToCpu` usages guarantee
  host-visible memory, not host-coherent memory, but the current mapping API
  (`mapped_slice`/`mapped_slice_mut`) has no flush/invalidate step. Require
  coherent memory: compute the memory-type bitmask of types with
  `HOST_VISIBLE | HOST_COHERENT` from the physical device's memory properties
  and set `AllocationCreateInfo.memory_type_bits` for CPU-mapped allocations;
  fail the allocation if no such type exists (stricter than gpu-allocator,
  which silently falls back to non-coherent types). If a non-coherent-only
  device must be supported later, switch to explicit
  `Allocator::flush_allocation`/`invalidate_allocation` around map windows and
  change the API contract accordingly — do not ship a silent fallback.
- **Handle guards before allocation**: the buffer/image flows create raw Vulkan
  handles before allocating (`buffers.rs:108-130`, `images.rs:191-223`); wrap
  each raw handle in `DeviceObject` immediately after creation and establish
  `Allocation` ownership (`DeviceObject<GpuAllocation>`) before binding, so
  every failure path — allocation failure or bind failure — releases the handle
  and the allocation in the correct order (destroy handle first, then
  `free_memory`). No raw handle may escape a fallible span unowned.
- Free with `free_memory` after the buffer/image handle is destroyed; do not
  also call `destroy_buffer/destroy_image` for the same allocation.

## Validation checklist

- `cargo check -p renderer_device`, `-p renderer_resources`, `-p renderer`,
  `-p renderer_rendergraph`, then `cargo check --workspace`.
- `cargo test -p renderer_device` and `cargo test -p renderer_device --features
  testing` (real Vulkan/validation; skips if unavailable).
- `cargo test -p renderer_resources` and `cargo test -p renderer_rendergraph`
  (both exercise `TestVulkan` through their dev-dependency features).
- `cargo tree -d | rg '^ash'` should show one ash. The lockfile has two
  `gpu-allocator 0.28.0` packages (registry + git), so `cargo tree -i
  gpu-allocator` is ambiguous; use source-qualified package IDs:
  `cargo tree -i 'registry+https://github.com/rust-lang/crates.io-index#gpu-allocator@0.28.0'`
  should list only `wgpu-hal` via the Bevy dev-dependency, and
  `cargo tree -i 'https://github.com/janis-bhm/gpu-allocator#gpu-allocator@0.28.0'`
  should be empty after phase 3.
  `rg -n 'gpu_allocator|gpu-allocator' --glob '!Cargo.lock'` should return
  nothing.
- Re-run the baseline Vulkan smoke tests and, if a display is available,
  `cargo build --examples`.

## Risks & gotchas

- **FFI/lifetimes**: `vk_mem::Allocator` is a raw-pointer owner; it must outlive
  every `Allocation`. The existing `DeviceInner` field order (`alloc2` before
  `_drop`) is correct; do not move the allocator after `_drop`. `GpuAllocation`
  must not be `Copy` and must free exactly once through
  `DeviceObject`/`ExternallyManagedObject`.
- **Thread safety**: `AllocationInfo.mapped_data` is `*mut c_void`, which is not
  `Send/Sync`; storing it directly breaks `Arc<Buffer>` thread-safety. Use a
  private `SendSyncPtr` or unsafe `Send/Sync` wrapper and add a compile-time
  `assert_send_sync::<Buffer>()`/`Image` test.
- **Mapping differs**: `gpu-allocator` mapped host-visible blocks
  opportunistically; VMA maps only with `MAPPED`. `CpuToGpu`/`GpuToCpu` keep
  working; `GpuOnly` mapping on integrated GPUs may now return `None`.
  **Coherence**: VMA only guarantees host-visible (not host-coherent) memory
  for CPU usages; this plan requires coherent types via `memory_type_bits`
  (see implementation notes) so the existing flush-free `map`/`map_mut` usage
  stays correct. Non-coherent support would need explicit
  `flush_allocation`/`invalidate_allocation`; current code does not
  flush/invalidate either.
- **Dedicated allocation**: use `allocate_memory_for_buffer/image` with
  `DEDICATED_MEMORY`; generic `allocate_memory` would create a separate block
  without dedicated buffer/image info. `VkMemoryDedicatedAllocateInfo` is only
  emitted when the runtime API version is ≥ 1.1 (or the KHR dedicated
  extensions are enabled); pass the clamped effective version per the
  implementation notes and require the extensions on Vulkan 1.0 devices.
- **OOM handling**: `gpu_allocator::AllocationError` variants disappear; VMA
  returns `vk::Result` (`ERROR_OUT_OF_DEVICE_MEMORY`,
  `ERROR_FEATURE_NOT_PRESENT`, etc.), already handled by `Error::Result`. No
  call site pattern-matches the old variant.
- **Build environment**: `vk-mem 0.5.0` compiles VMA C++ (`wrapper.cpp`, C++17,
  links stdc++ on Linux); ensure CI has a C++ compiler. It pins ash 0.38; an ash
  upgrade requires a matching `vk-mem` release/fork. The crate is less actively
  maintained than `gpu-allocator`.
- **Lost features**: no `vk_mem` allocation naming setter
  (`vmaSetAllocationName`) and no `build_stats_string` in 0.5.0; VMA
  stats/heap budgets remain available. Defragmentation is intentionally out of
  scope; if enabled later, cached `memory/offset/mapped_data` in the wrapper
  must be re-queried.
- **Pre-existing hazard**: `Image::new_with_allocation` still creates the raw
  image before validation (`images.rs:235-260`); the handle-guard requirement
  (implementation notes) covers this by owning the raw image in `DeviceObject`
  from creation onward.