1//
2// Copyright (c) 2017-2022 Advanced Micro Devices, Inc. All rights reserved.
3//
4// Permission is hereby granted, free of charge, to any person obtaining a copy
5// of this software and associated documentation files (the "Software"), to deal
6// in the Software without restriction, including without limitation the rights
7// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
8// copies of the Software, and to permit persons to whom the Software is
9// furnished to do so, subject to the following conditions:
10//
11// The above copyright notice and this permission notice shall be included in
12// all copies or substantial portions of the Software.
13//
14// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
15// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
16// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
17// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
18// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
19// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
20// THE SOFTWARE.
21//
22
23#ifndef AMD_VULKAN_MEMORY_ALLOCATOR_H
24#define AMD_VULKAN_MEMORY_ALLOCATOR_H
25
26/** \mainpage Vulkan Memory Allocator
27
28<b>Version 3.0.1 (2022-05-26)</b>
29
30Copyright (c) 2017-2022 Advanced Micro Devices, Inc. All rights reserved. \n
31License: MIT
32
33<b>API documentation divided into groups:</b> [Modules](modules.html)
34
35\section main_table_of_contents Table of contents
36
37- <b>User guide</b>
38 - \subpage quick_start
39 - [Project setup](@ref quick_start_project_setup)
40 - [Initialization](@ref quick_start_initialization)
41 - [Resource allocation](@ref quick_start_resource_allocation)
42 - \subpage choosing_memory_type
43 - [Usage](@ref choosing_memory_type_usage)
44 - [Required and preferred flags](@ref choosing_memory_type_required_preferred_flags)
45 - [Explicit memory types](@ref choosing_memory_type_explicit_memory_types)
46 - [Custom memory pools](@ref choosing_memory_type_custom_memory_pools)
47 - [Dedicated allocations](@ref choosing_memory_type_dedicated_allocations)
48 - \subpage memory_mapping
49 - [Mapping functions](@ref memory_mapping_mapping_functions)
50 - [Persistently mapped memory](@ref memory_mapping_persistently_mapped_memory)
51 - [Cache flush and invalidate](@ref memory_mapping_cache_control)
52 - \subpage staying_within_budget
53 - [Querying for budget](@ref staying_within_budget_querying_for_budget)
54 - [Controlling memory usage](@ref staying_within_budget_controlling_memory_usage)
55 - \subpage resource_aliasing
56 - \subpage custom_memory_pools
57 - [Choosing memory type index](@ref custom_memory_pools_MemTypeIndex)
58 - [Linear allocation algorithm](@ref linear_algorithm)
59 - [Free-at-once](@ref linear_algorithm_free_at_once)
60 - [Stack](@ref linear_algorithm_stack)
61 - [Double stack](@ref linear_algorithm_double_stack)
62 - [Ring buffer](@ref linear_algorithm_ring_buffer)
63 - \subpage defragmentation
64 - \subpage statistics
65 - [Numeric statistics](@ref statistics_numeric_statistics)
66 - [JSON dump](@ref statistics_json_dump)
67 - \subpage allocation_annotation
68 - [Allocation user data](@ref allocation_user_data)
69 - [Allocation names](@ref allocation_names)
70 - \subpage virtual_allocator
71 - \subpage debugging_memory_usage
72 - [Memory initialization](@ref debugging_memory_usage_initialization)
73 - [Margins](@ref debugging_memory_usage_margins)
74 - [Corruption detection](@ref debugging_memory_usage_corruption_detection)
75 - \subpage opengl_interop
76- \subpage usage_patterns
77 - [GPU-only resource](@ref usage_patterns_gpu_only)
78 - [Staging copy for upload](@ref usage_patterns_staging_copy_upload)
79 - [Readback](@ref usage_patterns_readback)
80 - [Advanced data uploading](@ref usage_patterns_advanced_data_uploading)
81 - [Other use cases](@ref usage_patterns_other_use_cases)
82- \subpage configuration
83 - [Pointers to Vulkan functions](@ref config_Vulkan_functions)
84 - [Custom host memory allocator](@ref custom_memory_allocator)
85 - [Device memory allocation callbacks](@ref allocation_callbacks)
86 - [Device heap memory limit](@ref heap_memory_limit)
87- <b>Extension support</b>
88 - \subpage vk_khr_dedicated_allocation
89 - \subpage enabling_buffer_device_address
90 - \subpage vk_ext_memory_priority
91 - \subpage vk_amd_device_coherent_memory
92- \subpage general_considerations
93 - [Thread safety](@ref general_considerations_thread_safety)
94 - [Versioning and compatibility](@ref general_considerations_versioning_and_compatibility)
95 - [Validation layer warnings](@ref general_considerations_validation_layer_warnings)
96 - [Allocation algorithm](@ref general_considerations_allocation_algorithm)
97 - [Features not supported](@ref general_considerations_features_not_supported)
98
99\section main_see_also See also
100
101- [**Product page on GPUOpen**](https://gpuopen.com/gaming-product/vulkan-memory-allocator/)
102- [**Source repository on GitHub**](https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator)
103
104\defgroup group_init Library initialization
105
106\brief API elements related to the initialization and management of the entire library, especially #VmaAllocator object.
107
108\defgroup group_alloc Memory allocation
109
110\brief API elements related to the allocation, deallocation, and management of Vulkan memory, buffers, images.
111Most basic ones being: vmaCreateBuffer(), vmaCreateImage().
112
113\defgroup group_virtual Virtual allocator
114
115\brief API elements related to the mechanism of \ref virtual_allocator - using the core allocation algorithm
116for user-defined purpose without allocating any real GPU memory.
117
118\defgroup group_stats Statistics
119
120\brief API elements that query current status of the allocator, from memory usage, budget, to full dump of the internal state in JSON format.
121See documentation chapter: \ref statistics.
122*/
123
124
125#ifdef __cplusplus
126extern "C" {
127#endif
128
129#ifndef VULKAN_H_
130 #include <vulkan/vulkan.h>
131#endif
132
133// Define this macro to declare maximum supported Vulkan version in format AAABBBCCC,
134// where AAA = major, BBB = minor, CCC = patch.
135// If you want to use version > 1.0, it still needs to be enabled via VmaAllocatorCreateInfo::vulkanApiVersion.
136#if !defined(VMA_VULKAN_VERSION)
137 #if defined(VK_VERSION_1_3)
138 #define VMA_VULKAN_VERSION 1003000
139 #elif defined(VK_VERSION_1_2)
140 #define VMA_VULKAN_VERSION 1002000
141 #elif defined(VK_VERSION_1_1)
142 #define VMA_VULKAN_VERSION 1001000
143 #else
144 #define VMA_VULKAN_VERSION 1000000
145 #endif
146#endif
147
148#if defined(__ANDROID__) && defined(VK_NO_PROTOTYPES) && VMA_STATIC_VULKAN_FUNCTIONS
149 extern PFN_vkGetInstanceProcAddr vkGetInstanceProcAddr;
150 extern PFN_vkGetDeviceProcAddr vkGetDeviceProcAddr;
151 extern PFN_vkGetPhysicalDeviceProperties vkGetPhysicalDeviceProperties;
152 extern PFN_vkGetPhysicalDeviceMemoryProperties vkGetPhysicalDeviceMemoryProperties;
153 extern PFN_vkAllocateMemory vkAllocateMemory;
154 extern PFN_vkFreeMemory vkFreeMemory;
155 extern PFN_vkMapMemory vkMapMemory;
156 extern PFN_vkUnmapMemory vkUnmapMemory;
157 extern PFN_vkFlushMappedMemoryRanges vkFlushMappedMemoryRanges;
158 extern PFN_vkInvalidateMappedMemoryRanges vkInvalidateMappedMemoryRanges;
159 extern PFN_vkBindBufferMemory vkBindBufferMemory;
160 extern PFN_vkBindImageMemory vkBindImageMemory;
161 extern PFN_vkGetBufferMemoryRequirements vkGetBufferMemoryRequirements;
162 extern PFN_vkGetImageMemoryRequirements vkGetImageMemoryRequirements;
163 extern PFN_vkCreateBuffer vkCreateBuffer;
164 extern PFN_vkDestroyBuffer vkDestroyBuffer;
165 extern PFN_vkCreateImage vkCreateImage;
166 extern PFN_vkDestroyImage vkDestroyImage;
167 extern PFN_vkCmdCopyBuffer vkCmdCopyBuffer;
168 #if VMA_VULKAN_VERSION >= 1001000
169 extern PFN_vkGetBufferMemoryRequirements2 vkGetBufferMemoryRequirements2;
170 extern PFN_vkGetImageMemoryRequirements2 vkGetImageMemoryRequirements2;
171 extern PFN_vkBindBufferMemory2 vkBindBufferMemory2;
172 extern PFN_vkBindImageMemory2 vkBindImageMemory2;
173 extern PFN_vkGetPhysicalDeviceMemoryProperties2 vkGetPhysicalDeviceMemoryProperties2;
174 #endif // #if VMA_VULKAN_VERSION >= 1001000
175#endif // #if defined(__ANDROID__) && VMA_STATIC_VULKAN_FUNCTIONS && VK_NO_PROTOTYPES
176
177#if !defined(VMA_DEDICATED_ALLOCATION)
178 #if VK_KHR_get_memory_requirements2 && VK_KHR_dedicated_allocation
179 #define VMA_DEDICATED_ALLOCATION 1
180 #else
181 #define VMA_DEDICATED_ALLOCATION 0
182 #endif
183#endif
184
185#if !defined(VMA_BIND_MEMORY2)
186 #if VK_KHR_bind_memory2
187 #define VMA_BIND_MEMORY2 1
188 #else
189 #define VMA_BIND_MEMORY2 0
190 #endif
191#endif
192
193#if !defined(VMA_MEMORY_BUDGET)
194 #if VK_EXT_memory_budget && (VK_KHR_get_physical_device_properties2 || VMA_VULKAN_VERSION >= 1001000)
195 #define VMA_MEMORY_BUDGET 1
196 #else
197 #define VMA_MEMORY_BUDGET 0
198 #endif
199#endif
200
201// Defined to 1 when VK_KHR_buffer_device_address device extension or equivalent core Vulkan 1.2 feature is defined in its headers.
202#if !defined(VMA_BUFFER_DEVICE_ADDRESS)
203 #if VK_KHR_buffer_device_address || VMA_VULKAN_VERSION >= 1002000
204 #define VMA_BUFFER_DEVICE_ADDRESS 1
205 #else
206 #define VMA_BUFFER_DEVICE_ADDRESS 0
207 #endif
208#endif
209
210// Defined to 1 when VK_EXT_memory_priority device extension is defined in Vulkan headers.
211#if !defined(VMA_MEMORY_PRIORITY)
212 #if VK_EXT_memory_priority
213 #define VMA_MEMORY_PRIORITY 1
214 #else
215 #define VMA_MEMORY_PRIORITY 0
216 #endif
217#endif
218
219// Defined to 1 when VK_KHR_external_memory device extension is defined in Vulkan headers.
220#if !defined(VMA_EXTERNAL_MEMORY)
221 #if VK_KHR_external_memory
222 #define VMA_EXTERNAL_MEMORY 1
223 #else
224 #define VMA_EXTERNAL_MEMORY 0
225 #endif
226#endif
227
228// Define these macros to decorate all public functions with additional code,
229// before and after returned type, appropriately. This may be useful for
230// exporting the functions when compiling VMA as a separate library. Example:
231// #define VMA_CALL_PRE __declspec(dllexport)
232// #define VMA_CALL_POST __cdecl
233#ifndef VMA_CALL_PRE
234 #define VMA_CALL_PRE
235#endif
236#ifndef VMA_CALL_POST
237 #define VMA_CALL_POST
238#endif
239
240// Define this macro to decorate pointers with an attribute specifying the
241// length of the array they point to if they are not null.
242//
243// The length may be one of
244// - The name of another parameter in the argument list where the pointer is declared
245// - The name of another member in the struct where the pointer is declared
246// - The name of a member of a struct type, meaning the value of that member in
247// the context of the call. For example
248// VMA_LEN_IF_NOT_NULL("VkPhysicalDeviceMemoryProperties::memoryHeapCount"),
249// this means the number of memory heaps available in the device associated
250// with the VmaAllocator being dealt with.
251#ifndef VMA_LEN_IF_NOT_NULL
252 #define VMA_LEN_IF_NOT_NULL(len)
253#endif
254
255// The VMA_NULLABLE macro is defined to be _Nullable when compiling with Clang.
256// see: https://clang.llvm.org/docs/AttributeReference.html#nullable
257#ifndef VMA_NULLABLE
258 #ifdef __clang__
259 #define VMA_NULLABLE _Nullable
260 #else
261 #define VMA_NULLABLE
262 #endif
263#endif
264
265// The VMA_NOT_NULL macro is defined to be _Nonnull when compiling with Clang.
266// see: https://clang.llvm.org/docs/AttributeReference.html#nonnull
267#ifndef VMA_NOT_NULL
268 #ifdef __clang__
269 #define VMA_NOT_NULL _Nonnull
270 #else
271 #define VMA_NOT_NULL
272 #endif
273#endif
274
275// If non-dispatchable handles are represented as pointers then we can give
276// then nullability annotations
277#ifndef VMA_NOT_NULL_NON_DISPATCHABLE
278 #if defined(__LP64__) || defined(_WIN64) || (defined(__x86_64__) && !defined(__ILP32__) ) || defined(_M_X64) || defined(__ia64) || defined (_M_IA64) || defined(__aarch64__) || defined(__powerpc64__)
279 #define VMA_NOT_NULL_NON_DISPATCHABLE VMA_NOT_NULL
280 #else
281 #define VMA_NOT_NULL_NON_DISPATCHABLE
282 #endif
283#endif
284
285#ifndef VMA_NULLABLE_NON_DISPATCHABLE
286 #if defined(__LP64__) || defined(_WIN64) || (defined(__x86_64__) && !defined(__ILP32__) ) || defined(_M_X64) || defined(__ia64) || defined (_M_IA64) || defined(__aarch64__) || defined(__powerpc64__)
287 #define VMA_NULLABLE_NON_DISPATCHABLE VMA_NULLABLE
288 #else
289 #define VMA_NULLABLE_NON_DISPATCHABLE
290 #endif
291#endif
292
293#ifndef VMA_STATS_STRING_ENABLED
294 #define VMA_STATS_STRING_ENABLED 1
295#endif
296
297////////////////////////////////////////////////////////////////////////////////
298////////////////////////////////////////////////////////////////////////////////
299//
300// INTERFACE
301//
302////////////////////////////////////////////////////////////////////////////////
303////////////////////////////////////////////////////////////////////////////////
304
305// Sections for managing code placement in file, only for development purposes e.g. for convenient folding inside an IDE.
306#ifndef _VMA_ENUM_DECLARATIONS
307
308/**
309\addtogroup group_init
310@{
311*/
312
313/// Flags for created #VmaAllocator.
314typedef enum VmaAllocatorCreateFlagBits
315{
316 /** \brief Allocator and all objects created from it will not be synchronized internally, so you must guarantee they are used from only one thread at a time or synchronized externally by you.
317
318 Using this flag may increase performance because internal mutexes are not used.
319 */
320 VMA_ALLOCATOR_CREATE_EXTERNALLY_SYNCHRONIZED_BIT = 0x00000001,
321 /** \brief Enables usage of VK_KHR_dedicated_allocation extension.
322
323 The flag works only if VmaAllocatorCreateInfo::vulkanApiVersion `== VK_API_VERSION_1_0`.
324 When it is `VK_API_VERSION_1_1`, the flag is ignored because the extension has been promoted to Vulkan 1.1.
325
326 Using this extension will automatically allocate dedicated blocks of memory for
327 some buffers and images instead of suballocating place for them out of bigger
328 memory blocks (as if you explicitly used #VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT
329 flag) when it is recommended by the driver. It may improve performance on some
330 GPUs.
331
332 You may set this flag only if you found out that following device extensions are
333 supported, you enabled them while creating Vulkan device passed as
334 VmaAllocatorCreateInfo::device, and you want them to be used internally by this
335 library:
336
337 - VK_KHR_get_memory_requirements2 (device extension)
338 - VK_KHR_dedicated_allocation (device extension)
339
340 When this flag is set, you can experience following warnings reported by Vulkan
341 validation layer. You can ignore them.
342
343 > vkBindBufferMemory(): Binding memory to buffer 0x2d but vkGetBufferMemoryRequirements() has not been called on that buffer.
344 */
345 VMA_ALLOCATOR_CREATE_KHR_DEDICATED_ALLOCATION_BIT = 0x00000002,
346 /**
347 Enables usage of VK_KHR_bind_memory2 extension.
348
349 The flag works only if VmaAllocatorCreateInfo::vulkanApiVersion `== VK_API_VERSION_1_0`.
350 When it is `VK_API_VERSION_1_1`, the flag is ignored because the extension has been promoted to Vulkan 1.1.
351
352 You may set this flag only if you found out that this device extension is supported,
353 you enabled it while creating Vulkan device passed as VmaAllocatorCreateInfo::device,
354 and you want it to be used internally by this library.
355
356 The extension provides functions `vkBindBufferMemory2KHR` and `vkBindImageMemory2KHR`,
357 which allow to pass a chain of `pNext` structures while binding.
358 This flag is required if you use `pNext` parameter in vmaBindBufferMemory2() or vmaBindImageMemory2().
359 */
360 VMA_ALLOCATOR_CREATE_KHR_BIND_MEMORY2_BIT = 0x00000004,
361 /**
362 Enables usage of VK_EXT_memory_budget extension.
363
364 You may set this flag only if you found out that this device extension is supported,
365 you enabled it while creating Vulkan device passed as VmaAllocatorCreateInfo::device,
366 and you want it to be used internally by this library, along with another instance extension
367 VK_KHR_get_physical_device_properties2, which is required by it (or Vulkan 1.1, where this extension is promoted).
368
369 The extension provides query for current memory usage and budget, which will probably
370 be more accurate than an estimation used by the library otherwise.
371 */
372 VMA_ALLOCATOR_CREATE_EXT_MEMORY_BUDGET_BIT = 0x00000008,
373 /**
374 Enables usage of VK_AMD_device_coherent_memory extension.
375
376 You may set this flag only if you:
377
378 - found out that this device extension is supported and enabled it while creating Vulkan device passed as VmaAllocatorCreateInfo::device,
379 - checked that `VkPhysicalDeviceCoherentMemoryFeaturesAMD::deviceCoherentMemory` is true and set it while creating the Vulkan device,
380 - want it to be used internally by this library.
381
382 The extension and accompanying device feature provide access to memory types with
383 `VK_MEMORY_PROPERTY_DEVICE_COHERENT_BIT_AMD` and `VK_MEMORY_PROPERTY_DEVICE_UNCACHED_BIT_AMD` flags.
384 They are useful mostly for writing breadcrumb markers - a common method for debugging GPU crash/hang/TDR.
385
386 When the extension is not enabled, such memory types are still enumerated, but their usage is illegal.
387 To protect from this error, if you don't create the allocator with this flag, it will refuse to allocate any memory or create a custom pool in such memory type,
388 returning `VK_ERROR_FEATURE_NOT_PRESENT`.
389 */
390 VMA_ALLOCATOR_CREATE_AMD_DEVICE_COHERENT_MEMORY_BIT = 0x00000010,
391 /**
392 Enables usage of "buffer device address" feature, which allows you to use function
393 `vkGetBufferDeviceAddress*` to get raw GPU pointer to a buffer and pass it for usage inside a shader.
394
395 You may set this flag only if you:
396
397 1. (For Vulkan version < 1.2) Found as available and enabled device extension
398 VK_KHR_buffer_device_address.
399 This extension is promoted to core Vulkan 1.2.
400 2. Found as available and enabled device feature `VkPhysicalDeviceBufferDeviceAddressFeatures::bufferDeviceAddress`.
401
402 When this flag is set, you can create buffers with `VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT` using VMA.
403 The library automatically adds `VK_MEMORY_ALLOCATE_DEVICE_ADDRESS_BIT` to
404 allocated memory blocks wherever it might be needed.
405
406 For more information, see documentation chapter \ref enabling_buffer_device_address.
407 */
408 VMA_ALLOCATOR_CREATE_BUFFER_DEVICE_ADDRESS_BIT = 0x00000020,
409 /**
410 Enables usage of VK_EXT_memory_priority extension in the library.
411
412 You may set this flag only if you found available and enabled this device extension,
413 along with `VkPhysicalDeviceMemoryPriorityFeaturesEXT::memoryPriority == VK_TRUE`,
414 while creating Vulkan device passed as VmaAllocatorCreateInfo::device.
415
416 When this flag is used, VmaAllocationCreateInfo::priority and VmaPoolCreateInfo::priority
417 are used to set priorities of allocated Vulkan memory. Without it, these variables are ignored.
418
419 A priority must be a floating-point value between 0 and 1, indicating the priority of the allocation relative to other memory allocations.
420 Larger values are higher priority. The granularity of the priorities is implementation-dependent.
421 It is automatically passed to every call to `vkAllocateMemory` done by the library using structure `VkMemoryPriorityAllocateInfoEXT`.
422 The value to be used for default priority is 0.5.
423 For more details, see the documentation of the VK_EXT_memory_priority extension.
424 */
425 VMA_ALLOCATOR_CREATE_EXT_MEMORY_PRIORITY_BIT = 0x00000040,
426
427 VMA_ALLOCATOR_CREATE_FLAG_BITS_MAX_ENUM = 0x7FFFFFFF
428} VmaAllocatorCreateFlagBits;
429/// See #VmaAllocatorCreateFlagBits.
430typedef VkFlags VmaAllocatorCreateFlags;
431
432/** @} */
433
434/**
435\addtogroup group_alloc
436@{
437*/
438
439/// \brief Intended usage of the allocated memory.
440typedef enum VmaMemoryUsage
441{
442 /** No intended memory usage specified.
443 Use other members of VmaAllocationCreateInfo to specify your requirements.
444 */
445 VMA_MEMORY_USAGE_UNKNOWN = 0,
446 /**
447 \deprecated Obsolete, preserved for backward compatibility.
448 Prefers `VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT`.
449 */
450 VMA_MEMORY_USAGE_GPU_ONLY = 1,
451 /**
452 \deprecated Obsolete, preserved for backward compatibility.
453 Guarantees `VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT` and `VK_MEMORY_PROPERTY_HOST_COHERENT_BIT`.
454 */
455 VMA_MEMORY_USAGE_CPU_ONLY = 2,
456 /**
457 \deprecated Obsolete, preserved for backward compatibility.
458 Guarantees `VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT`, prefers `VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT`.
459 */
460 VMA_MEMORY_USAGE_CPU_TO_GPU = 3,
461 /**
462 \deprecated Obsolete, preserved for backward compatibility.
463 Guarantees `VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT`, prefers `VK_MEMORY_PROPERTY_HOST_CACHED_BIT`.
464 */
465 VMA_MEMORY_USAGE_GPU_TO_CPU = 4,
466 /**
467 \deprecated Obsolete, preserved for backward compatibility.
468 Prefers not `VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT`.
469 */
470 VMA_MEMORY_USAGE_CPU_COPY = 5,
471 /**
472 Lazily allocated GPU memory having `VK_MEMORY_PROPERTY_LAZILY_ALLOCATED_BIT`.
473 Exists mostly on mobile platforms. Using it on desktop PC or other GPUs with no such memory type present will fail the allocation.
474
475 Usage: Memory for transient attachment images (color attachments, depth attachments etc.), created with `VK_IMAGE_USAGE_TRANSIENT_ATTACHMENT_BIT`.
476
477 Allocations with this usage are always created as dedicated - it implies #VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT.
478 */
479 VMA_MEMORY_USAGE_GPU_LAZILY_ALLOCATED = 6,
480 /**
481 Selects best memory type automatically.
482 This flag is recommended for most common use cases.
483
484 When using this flag, if you want to map the allocation (using vmaMapMemory() or #VMA_ALLOCATION_CREATE_MAPPED_BIT),
485 you must pass one of the flags: #VMA_ALLOCATION_CREATE_HOST_ACCESS_SEQUENTIAL_WRITE_BIT or #VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT
486 in VmaAllocationCreateInfo::flags.
487
488 It can be used only with functions that let the library know `VkBufferCreateInfo` or `VkImageCreateInfo`, e.g.
489 vmaCreateBuffer(), vmaCreateImage(), vmaFindMemoryTypeIndexForBufferInfo(), vmaFindMemoryTypeIndexForImageInfo()
490 and not with generic memory allocation functions.
491 */
492 VMA_MEMORY_USAGE_AUTO = 7,
493 /**
494 Selects best memory type automatically with preference for GPU (device) memory.
495
496 When using this flag, if you want to map the allocation (using vmaMapMemory() or #VMA_ALLOCATION_CREATE_MAPPED_BIT),
497 you must pass one of the flags: #VMA_ALLOCATION_CREATE_HOST_ACCESS_SEQUENTIAL_WRITE_BIT or #VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT
498 in VmaAllocationCreateInfo::flags.
499
500 It can be used only with functions that let the library know `VkBufferCreateInfo` or `VkImageCreateInfo`, e.g.
501 vmaCreateBuffer(), vmaCreateImage(), vmaFindMemoryTypeIndexForBufferInfo(), vmaFindMemoryTypeIndexForImageInfo()
502 and not with generic memory allocation functions.
503 */
504 VMA_MEMORY_USAGE_AUTO_PREFER_DEVICE = 8,
505 /**
506 Selects best memory type automatically with preference for CPU (host) memory.
507
508 When using this flag, if you want to map the allocation (using vmaMapMemory() or #VMA_ALLOCATION_CREATE_MAPPED_BIT),
509 you must pass one of the flags: #VMA_ALLOCATION_CREATE_HOST_ACCESS_SEQUENTIAL_WRITE_BIT or #VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT
510 in VmaAllocationCreateInfo::flags.
511
512 It can be used only with functions that let the library know `VkBufferCreateInfo` or `VkImageCreateInfo`, e.g.
513 vmaCreateBuffer(), vmaCreateImage(), vmaFindMemoryTypeIndexForBufferInfo(), vmaFindMemoryTypeIndexForImageInfo()
514 and not with generic memory allocation functions.
515 */
516 VMA_MEMORY_USAGE_AUTO_PREFER_HOST = 9,
517
518 VMA_MEMORY_USAGE_MAX_ENUM = 0x7FFFFFFF
519} VmaMemoryUsage;
520
521/// Flags to be passed as VmaAllocationCreateInfo::flags.
522typedef enum VmaAllocationCreateFlagBits
523{
524 /** \brief Set this flag if the allocation should have its own memory block.
525
526 Use it for special, big resources, like fullscreen images used as attachments.
527 */
528 VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT = 0x00000001,
529
530 /** \brief Set this flag to only try to allocate from existing `VkDeviceMemory` blocks and never create new such block.
531
532 If new allocation cannot be placed in any of the existing blocks, allocation
533 fails with `VK_ERROR_OUT_OF_DEVICE_MEMORY` error.
534
535 You should not use #VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT and
536 #VMA_ALLOCATION_CREATE_NEVER_ALLOCATE_BIT at the same time. It makes no sense.
537 */
538 VMA_ALLOCATION_CREATE_NEVER_ALLOCATE_BIT = 0x00000002,
539 /** \brief Set this flag to use a memory that will be persistently mapped and retrieve pointer to it.
540
541 Pointer to mapped memory will be returned through VmaAllocationInfo::pMappedData.
542
543 It is valid to use this flag for allocation made from memory type that is not
544 `HOST_VISIBLE`. This flag is then ignored and memory is not mapped. This is
545 useful if you need an allocation that is efficient to use on GPU
546 (`DEVICE_LOCAL`) and still want to map it directly if possible on platforms that
547 support it (e.g. Intel GPU).
548 */
549 VMA_ALLOCATION_CREATE_MAPPED_BIT = 0x00000004,
550 /** \deprecated Preserved for backward compatibility. Consider using vmaSetAllocationName() instead.
551
552 Set this flag to treat VmaAllocationCreateInfo::pUserData as pointer to a
553 null-terminated string. Instead of copying pointer value, a local copy of the
554 string is made and stored in allocation's `pName`. The string is automatically
555 freed together with the allocation. It is also used in vmaBuildStatsString().
556 */
557 VMA_ALLOCATION_CREATE_USER_DATA_COPY_STRING_BIT = 0x00000020,
558 /** Allocation will be created from upper stack in a double stack pool.
559
560 This flag is only allowed for custom pools created with #VMA_POOL_CREATE_LINEAR_ALGORITHM_BIT flag.
561 */
562 VMA_ALLOCATION_CREATE_UPPER_ADDRESS_BIT = 0x00000040,
563 /** Create both buffer/image and allocation, but don't bind them together.
564 It is useful when you want to bind yourself to do some more advanced binding, e.g. using some extensions.
565 The flag is meaningful only with functions that bind by default: vmaCreateBuffer(), vmaCreateImage().
566 Otherwise it is ignored.
567
568 If you want to make sure the new buffer/image is not tied to the new memory allocation
569 through `VkMemoryDedicatedAllocateInfoKHR` structure in case the allocation ends up in its own memory block,
570 use also flag #VMA_ALLOCATION_CREATE_CAN_ALIAS_BIT.
571 */
572 VMA_ALLOCATION_CREATE_DONT_BIND_BIT = 0x00000080,
573 /** Create allocation only if additional device memory required for it, if any, won't exceed
574 memory budget. Otherwise return `VK_ERROR_OUT_OF_DEVICE_MEMORY`.
575 */
576 VMA_ALLOCATION_CREATE_WITHIN_BUDGET_BIT = 0x00000100,
577 /** \brief Set this flag if the allocated memory will have aliasing resources.
578
579 Usage of this flag prevents supplying `VkMemoryDedicatedAllocateInfoKHR` when #VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT is specified.
580 Otherwise created dedicated memory will not be suitable for aliasing resources, resulting in Vulkan Validation Layer errors.
581 */
582 VMA_ALLOCATION_CREATE_CAN_ALIAS_BIT = 0x00000200,
583 /**
584 Requests possibility to map the allocation (using vmaMapMemory() or #VMA_ALLOCATION_CREATE_MAPPED_BIT).
585
586 - If you use #VMA_MEMORY_USAGE_AUTO or other `VMA_MEMORY_USAGE_AUTO*` value,
587 you must use this flag to be able to map the allocation. Otherwise, mapping is incorrect.
588 - If you use other value of #VmaMemoryUsage, this flag is ignored and mapping is always possible in memory types that are `HOST_VISIBLE`.
589 This includes allocations created in \ref custom_memory_pools.
590
591 Declares that mapped memory will only be written sequentially, e.g. using `memcpy()` or a loop writing number-by-number,
592 never read or accessed randomly, so a memory type can be selected that is uncached and write-combined.
593
594 \warning Violating this declaration may work correctly, but will likely be very slow.
595 Watch out for implicit reads introduced by doing e.g. `pMappedData[i] += x;`
596 Better prepare your data in a local variable and `memcpy()` it to the mapped pointer all at once.
597 */
598 VMA_ALLOCATION_CREATE_HOST_ACCESS_SEQUENTIAL_WRITE_BIT = 0x00000400,
599 /**
600 Requests possibility to map the allocation (using vmaMapMemory() or #VMA_ALLOCATION_CREATE_MAPPED_BIT).
601
602 - If you use #VMA_MEMORY_USAGE_AUTO or other `VMA_MEMORY_USAGE_AUTO*` value,
603 you must use this flag to be able to map the allocation. Otherwise, mapping is incorrect.
604 - If you use other value of #VmaMemoryUsage, this flag is ignored and mapping is always possible in memory types that are `HOST_VISIBLE`.
605 This includes allocations created in \ref custom_memory_pools.
606
607 Declares that mapped memory can be read, written, and accessed in random order,
608 so a `HOST_CACHED` memory type is required.
609 */
610 VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT = 0x00000800,
611 /**
612 Together with #VMA_ALLOCATION_CREATE_HOST_ACCESS_SEQUENTIAL_WRITE_BIT or #VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT,
613 it says that despite request for host access, a not-`HOST_VISIBLE` memory type can be selected
614 if it may improve performance.
615
616 By using this flag, you declare that you will check if the allocation ended up in a `HOST_VISIBLE` memory type
617 (e.g. using vmaGetAllocationMemoryProperties()) and if not, you will create some "staging" buffer and
618 issue an explicit transfer to write/read your data.
619 To prepare for this possibility, don't forget to add appropriate flags like
620 `VK_BUFFER_USAGE_TRANSFER_DST_BIT`, `VK_BUFFER_USAGE_TRANSFER_SRC_BIT` to the parameters of created buffer or image.
621 */
622 VMA_ALLOCATION_CREATE_HOST_ACCESS_ALLOW_TRANSFER_INSTEAD_BIT = 0x00001000,
623 /** Allocation strategy that chooses smallest possible free range for the allocation
624 to minimize memory usage and fragmentation, possibly at the expense of allocation time.
625 */
626 VMA_ALLOCATION_CREATE_STRATEGY_MIN_MEMORY_BIT = 0x00010000,
627 /** Allocation strategy that chooses first suitable free range for the allocation -
628 not necessarily in terms of the smallest offset but the one that is easiest and fastest to find
629 to minimize allocation time, possibly at the expense of allocation quality.
630 */
631 VMA_ALLOCATION_CREATE_STRATEGY_MIN_TIME_BIT = 0x00020000,
632 /** Allocation strategy that chooses always the lowest offset in available space.
633 This is not the most efficient strategy but achieves highly packed data.
634 Used internally by defragmentation, not recomended in typical usage.
635 */
636 VMA_ALLOCATION_CREATE_STRATEGY_MIN_OFFSET_BIT = 0x00040000,
637 /** Alias to #VMA_ALLOCATION_CREATE_STRATEGY_MIN_MEMORY_BIT.
638 */
639 VMA_ALLOCATION_CREATE_STRATEGY_BEST_FIT_BIT = VMA_ALLOCATION_CREATE_STRATEGY_MIN_MEMORY_BIT,
640 /** Alias to #VMA_ALLOCATION_CREATE_STRATEGY_MIN_TIME_BIT.
641 */
642 VMA_ALLOCATION_CREATE_STRATEGY_FIRST_FIT_BIT = VMA_ALLOCATION_CREATE_STRATEGY_MIN_TIME_BIT,
643 /** A bit mask to extract only `STRATEGY` bits from entire set of flags.
644 */
645 VMA_ALLOCATION_CREATE_STRATEGY_MASK =
646 VMA_ALLOCATION_CREATE_STRATEGY_MIN_MEMORY_BIT |
647 VMA_ALLOCATION_CREATE_STRATEGY_MIN_TIME_BIT |
648 VMA_ALLOCATION_CREATE_STRATEGY_MIN_OFFSET_BIT,
649
650 VMA_ALLOCATION_CREATE_FLAG_BITS_MAX_ENUM = 0x7FFFFFFF
651} VmaAllocationCreateFlagBits;
652/// See #VmaAllocationCreateFlagBits.
653typedef VkFlags VmaAllocationCreateFlags;
654
655/// Flags to be passed as VmaPoolCreateInfo::flags.
656typedef enum VmaPoolCreateFlagBits
657{
658 /** \brief Use this flag if you always allocate only buffers and linear images or only optimal images out of this pool and so Buffer-Image Granularity can be ignored.
659
660 This is an optional optimization flag.
661
662 If you always allocate using vmaCreateBuffer(), vmaCreateImage(),
663 vmaAllocateMemoryForBuffer(), then you don't need to use it because allocator
664 knows exact type of your allocations so it can handle Buffer-Image Granularity
665 in the optimal way.
666
667 If you also allocate using vmaAllocateMemoryForImage() or vmaAllocateMemory(),
668 exact type of such allocations is not known, so allocator must be conservative
669 in handling Buffer-Image Granularity, which can lead to suboptimal allocation
670 (wasted memory). In that case, if you can make sure you always allocate only
671 buffers and linear images or only optimal images out of this pool, use this flag
672 to make allocator disregard Buffer-Image Granularity and so make allocations
673 faster and more optimal.
674 */
675 VMA_POOL_CREATE_IGNORE_BUFFER_IMAGE_GRANULARITY_BIT = 0x00000002,
676
677 /** \brief Enables alternative, linear allocation algorithm in this pool.
678
679 Specify this flag to enable linear allocation algorithm, which always creates
680 new allocations after last one and doesn't reuse space from allocations freed in
681 between. It trades memory consumption for simplified algorithm and data
682 structure, which has better performance and uses less memory for metadata.
683
684 By using this flag, you can achieve behavior of free-at-once, stack,
685 ring buffer, and double stack.
686 For details, see documentation chapter \ref linear_algorithm.
687 */
688 VMA_POOL_CREATE_LINEAR_ALGORITHM_BIT = 0x00000004,
689
690 /** Bit mask to extract only `ALGORITHM` bits from entire set of flags.
691 */
692 VMA_POOL_CREATE_ALGORITHM_MASK =
693 VMA_POOL_CREATE_LINEAR_ALGORITHM_BIT,
694
695 VMA_POOL_CREATE_FLAG_BITS_MAX_ENUM = 0x7FFFFFFF
696} VmaPoolCreateFlagBits;
697/// Flags to be passed as VmaPoolCreateInfo::flags. See #VmaPoolCreateFlagBits.
698typedef VkFlags VmaPoolCreateFlags;
699
700/// Flags to be passed as VmaDefragmentationInfo::flags.
701typedef enum VmaDefragmentationFlagBits
702{
703 /* \brief Use simple but fast algorithm for defragmentation.
704 May not achieve best results but will require least time to compute and least allocations to copy.
705 */
706 VMA_DEFRAGMENTATION_FLAG_ALGORITHM_FAST_BIT = 0x1,
707 /* \brief Default defragmentation algorithm, applied also when no `ALGORITHM` flag is specified.
708 Offers a balance between defragmentation quality and the amount of allocations and bytes that need to be moved.
709 */
710 VMA_DEFRAGMENTATION_FLAG_ALGORITHM_BALANCED_BIT = 0x2,
711 /* \brief Perform full defragmentation of memory.
712 Can result in notably more time to compute and allocations to copy, but will achieve best memory packing.
713 */
714 VMA_DEFRAGMENTATION_FLAG_ALGORITHM_FULL_BIT = 0x4,
715 /** \brief Use the most roboust algorithm at the cost of time to compute and number of copies to make.
716 Only available when bufferImageGranularity is greater than 1, since it aims to reduce
717 alignment issues between different types of resources.
718 Otherwise falls back to same behavior as #VMA_DEFRAGMENTATION_FLAG_ALGORITHM_FULL_BIT.
719 */
720 VMA_DEFRAGMENTATION_FLAG_ALGORITHM_EXTENSIVE_BIT = 0x8,
721
722 /// A bit mask to extract only `ALGORITHM` bits from entire set of flags.
723 VMA_DEFRAGMENTATION_FLAG_ALGORITHM_MASK =
724 VMA_DEFRAGMENTATION_FLAG_ALGORITHM_FAST_BIT |
725 VMA_DEFRAGMENTATION_FLAG_ALGORITHM_BALANCED_BIT |
726 VMA_DEFRAGMENTATION_FLAG_ALGORITHM_FULL_BIT |
727 VMA_DEFRAGMENTATION_FLAG_ALGORITHM_EXTENSIVE_BIT,
728
729 VMA_DEFRAGMENTATION_FLAG_BITS_MAX_ENUM = 0x7FFFFFFF
730} VmaDefragmentationFlagBits;
731/// See #VmaDefragmentationFlagBits.
732typedef VkFlags VmaDefragmentationFlags;
733
734/// Operation performed on single defragmentation move. See structure #VmaDefragmentationMove.
735typedef enum VmaDefragmentationMoveOperation
736{
737 /// Buffer/image has been recreated at `dstTmpAllocation`, data has been copied, old buffer/image has been destroyed. `srcAllocation` should be changed to point to the new place. This is the default value set by vmaBeginDefragmentationPass().
738 VMA_DEFRAGMENTATION_MOVE_OPERATION_COPY = 0,
739 /// Set this value if you cannot move the allocation. New place reserved at `dstTmpAllocation` will be freed. `srcAllocation` will remain unchanged.
740 VMA_DEFRAGMENTATION_MOVE_OPERATION_IGNORE = 1,
741 /// Set this value if you decide to abandon the allocation and you destroyed the buffer/image. New place reserved at `dstTmpAllocation` will be freed, along with `srcAllocation`, which will be destroyed.
742 VMA_DEFRAGMENTATION_MOVE_OPERATION_DESTROY = 2,
743} VmaDefragmentationMoveOperation;
744
745/** @} */
746
747/**
748\addtogroup group_virtual
749@{
750*/
751
752/// Flags to be passed as VmaVirtualBlockCreateInfo::flags.
753typedef enum VmaVirtualBlockCreateFlagBits
754{
755 /** \brief Enables alternative, linear allocation algorithm in this virtual block.
756
757 Specify this flag to enable linear allocation algorithm, which always creates
758 new allocations after last one and doesn't reuse space from allocations freed in
759 between. It trades memory consumption for simplified algorithm and data
760 structure, which has better performance and uses less memory for metadata.
761
762 By using this flag, you can achieve behavior of free-at-once, stack,
763 ring buffer, and double stack.
764 For details, see documentation chapter \ref linear_algorithm.
765 */
766 VMA_VIRTUAL_BLOCK_CREATE_LINEAR_ALGORITHM_BIT = 0x00000001,
767
768 /** \brief Bit mask to extract only `ALGORITHM` bits from entire set of flags.
769 */
770 VMA_VIRTUAL_BLOCK_CREATE_ALGORITHM_MASK =
771 VMA_VIRTUAL_BLOCK_CREATE_LINEAR_ALGORITHM_BIT,
772
773 VMA_VIRTUAL_BLOCK_CREATE_FLAG_BITS_MAX_ENUM = 0x7FFFFFFF
774} VmaVirtualBlockCreateFlagBits;
775/// Flags to be passed as VmaVirtualBlockCreateInfo::flags. See #VmaVirtualBlockCreateFlagBits.
776typedef VkFlags VmaVirtualBlockCreateFlags;
777
778/// Flags to be passed as VmaVirtualAllocationCreateInfo::flags.
779typedef enum VmaVirtualAllocationCreateFlagBits
780{
781 /** \brief Allocation will be created from upper stack in a double stack pool.
782
783 This flag is only allowed for virtual blocks created with #VMA_VIRTUAL_BLOCK_CREATE_LINEAR_ALGORITHM_BIT flag.
784 */
785 VMA_VIRTUAL_ALLOCATION_CREATE_UPPER_ADDRESS_BIT = VMA_ALLOCATION_CREATE_UPPER_ADDRESS_BIT,
786 /** \brief Allocation strategy that tries to minimize memory usage.
787 */
788 VMA_VIRTUAL_ALLOCATION_CREATE_STRATEGY_MIN_MEMORY_BIT = VMA_ALLOCATION_CREATE_STRATEGY_MIN_MEMORY_BIT,
789 /** \brief Allocation strategy that tries to minimize allocation time.
790 */
791 VMA_VIRTUAL_ALLOCATION_CREATE_STRATEGY_MIN_TIME_BIT = VMA_ALLOCATION_CREATE_STRATEGY_MIN_TIME_BIT,
792 /** Allocation strategy that chooses always the lowest offset in available space.
793 This is not the most efficient strategy but achieves highly packed data.
794 */
795 VMA_VIRTUAL_ALLOCATION_CREATE_STRATEGY_MIN_OFFSET_BIT = VMA_ALLOCATION_CREATE_STRATEGY_MIN_OFFSET_BIT,
796 /** \brief A bit mask to extract only `STRATEGY` bits from entire set of flags.
797
798 These strategy flags are binary compatible with equivalent flags in #VmaAllocationCreateFlagBits.
799 */
800 VMA_VIRTUAL_ALLOCATION_CREATE_STRATEGY_MASK = VMA_ALLOCATION_CREATE_STRATEGY_MASK,
801
802 VMA_VIRTUAL_ALLOCATION_CREATE_FLAG_BITS_MAX_ENUM = 0x7FFFFFFF
803} VmaVirtualAllocationCreateFlagBits;
804/// Flags to be passed as VmaVirtualAllocationCreateInfo::flags. See #VmaVirtualAllocationCreateFlagBits.
805typedef VkFlags VmaVirtualAllocationCreateFlags;
806
807/** @} */
808
809#endif // _VMA_ENUM_DECLARATIONS
810
811#ifndef _VMA_DATA_TYPES_DECLARATIONS
812
813/**
814\addtogroup group_init
815@{ */
816
817/** \struct VmaAllocator
818\brief Represents main object of this library initialized.
819
820Fill structure #VmaAllocatorCreateInfo and call function vmaCreateAllocator() to create it.
821Call function vmaDestroyAllocator() to destroy it.
822
823It is recommended to create just one object of this type per `VkDevice` object,
824right after Vulkan is initialized and keep it alive until before Vulkan device is destroyed.
825*/
826VK_DEFINE_HANDLE(VmaAllocator)
827
828/** @} */
829
830/**
831\addtogroup group_alloc
832@{
833*/
834
835/** \struct VmaPool
836\brief Represents custom memory pool
837
838Fill structure VmaPoolCreateInfo and call function vmaCreatePool() to create it.
839Call function vmaDestroyPool() to destroy it.
840
841For more information see [Custom memory pools](@ref choosing_memory_type_custom_memory_pools).
842*/
843VK_DEFINE_HANDLE(VmaPool)
844
845/** \struct VmaAllocation
846\brief Represents single memory allocation.
847
848It may be either dedicated block of `VkDeviceMemory` or a specific region of a bigger block of this type
849plus unique offset.
850
851There are multiple ways to create such object.
852You need to fill structure VmaAllocationCreateInfo.
853For more information see [Choosing memory type](@ref choosing_memory_type).
854
855Although the library provides convenience functions that create Vulkan buffer or image,
856allocate memory for it and bind them together,
857binding of the allocation to a buffer or an image is out of scope of the allocation itself.
858Allocation object can exist without buffer/image bound,
859binding can be done manually by the user, and destruction of it can be done
860independently of destruction of the allocation.
861
862The object also remembers its size and some other information.
863To retrieve this information, use function vmaGetAllocationInfo() and inspect
864returned structure VmaAllocationInfo.
865*/
866VK_DEFINE_HANDLE(VmaAllocation)
867
868/** \struct VmaDefragmentationContext
869\brief An opaque object that represents started defragmentation process.
870
871Fill structure #VmaDefragmentationInfo and call function vmaBeginDefragmentation() to create it.
872Call function vmaEndDefragmentation() to destroy it.
873*/
874VK_DEFINE_HANDLE(VmaDefragmentationContext)
875
876/** @} */
877
878/**
879\addtogroup group_virtual
880@{
881*/
882
883/** \struct VmaVirtualAllocation
884\brief Represents single memory allocation done inside VmaVirtualBlock.
885
886Use it as a unique identifier to virtual allocation within the single block.
887
888Use value `VK_NULL_HANDLE` to represent a null/invalid allocation.
889*/
890VK_DEFINE_NON_DISPATCHABLE_HANDLE(VmaVirtualAllocation);
891
892/** @} */
893
894/**
895\addtogroup group_virtual
896@{
897*/
898
899/** \struct VmaVirtualBlock
900\brief Handle to a virtual block object that allows to use core allocation algorithm without allocating any real GPU memory.
901
902Fill in #VmaVirtualBlockCreateInfo structure and use vmaCreateVirtualBlock() to create it. Use vmaDestroyVirtualBlock() to destroy it.
903For more information, see documentation chapter \ref virtual_allocator.
904
905This object is not thread-safe - should not be used from multiple threads simultaneously, must be synchronized externally.
906*/
907VK_DEFINE_HANDLE(VmaVirtualBlock)
908
909/** @} */
910
911/**
912\addtogroup group_init
913@{
914*/
915
916/// Callback function called after successful vkAllocateMemory.
917typedef void (VKAPI_PTR* PFN_vmaAllocateDeviceMemoryFunction)(
918 VmaAllocator VMA_NOT_NULL allocator,
919 uint32_t memoryType,
920 VkDeviceMemory VMA_NOT_NULL_NON_DISPATCHABLE memory,
921 VkDeviceSize size,
922 void* VMA_NULLABLE pUserData);
923
924/// Callback function called before vkFreeMemory.
925typedef void (VKAPI_PTR* PFN_vmaFreeDeviceMemoryFunction)(
926 VmaAllocator VMA_NOT_NULL allocator,
927 uint32_t memoryType,
928 VkDeviceMemory VMA_NOT_NULL_NON_DISPATCHABLE memory,
929 VkDeviceSize size,
930 void* VMA_NULLABLE pUserData);
931
932/** \brief Set of callbacks that the library will call for `vkAllocateMemory` and `vkFreeMemory`.
933
934Provided for informative purpose, e.g. to gather statistics about number of
935allocations or total amount of memory allocated in Vulkan.
936
937Used in VmaAllocatorCreateInfo::pDeviceMemoryCallbacks.
938*/
939typedef struct VmaDeviceMemoryCallbacks
940{
941 /// Optional, can be null.
942 PFN_vmaAllocateDeviceMemoryFunction VMA_NULLABLE pfnAllocate;
943 /// Optional, can be null.
944 PFN_vmaFreeDeviceMemoryFunction VMA_NULLABLE pfnFree;
945 /// Optional, can be null.
946 void* VMA_NULLABLE pUserData;
947} VmaDeviceMemoryCallbacks;
948
949/** \brief Pointers to some Vulkan functions - a subset used by the library.
950
951Used in VmaAllocatorCreateInfo::pVulkanFunctions.
952*/
953typedef struct VmaVulkanFunctions
954{
955 /// Required when using VMA_DYNAMIC_VULKAN_FUNCTIONS.
956 PFN_vkGetInstanceProcAddr VMA_NULLABLE vkGetInstanceProcAddr;
957 /// Required when using VMA_DYNAMIC_VULKAN_FUNCTIONS.
958 PFN_vkGetDeviceProcAddr VMA_NULLABLE vkGetDeviceProcAddr;
959 PFN_vkGetPhysicalDeviceProperties VMA_NULLABLE vkGetPhysicalDeviceProperties;
960 PFN_vkGetPhysicalDeviceMemoryProperties VMA_NULLABLE vkGetPhysicalDeviceMemoryProperties;
961 PFN_vkAllocateMemory VMA_NULLABLE vkAllocateMemory;
962 PFN_vkFreeMemory VMA_NULLABLE vkFreeMemory;
963 PFN_vkMapMemory VMA_NULLABLE vkMapMemory;
964 PFN_vkUnmapMemory VMA_NULLABLE vkUnmapMemory;
965 PFN_vkFlushMappedMemoryRanges VMA_NULLABLE vkFlushMappedMemoryRanges;
966 PFN_vkInvalidateMappedMemoryRanges VMA_NULLABLE vkInvalidateMappedMemoryRanges;
967 PFN_vkBindBufferMemory VMA_NULLABLE vkBindBufferMemory;
968 PFN_vkBindImageMemory VMA_NULLABLE vkBindImageMemory;
969 PFN_vkGetBufferMemoryRequirements VMA_NULLABLE vkGetBufferMemoryRequirements;
970 PFN_vkGetImageMemoryRequirements VMA_NULLABLE vkGetImageMemoryRequirements;
971 PFN_vkCreateBuffer VMA_NULLABLE vkCreateBuffer;
972 PFN_vkDestroyBuffer VMA_NULLABLE vkDestroyBuffer;
973 PFN_vkCreateImage VMA_NULLABLE vkCreateImage;
974 PFN_vkDestroyImage VMA_NULLABLE vkDestroyImage;
975 PFN_vkCmdCopyBuffer VMA_NULLABLE vkCmdCopyBuffer;
976#if VMA_DEDICATED_ALLOCATION || VMA_VULKAN_VERSION >= 1001000
977 /// Fetch "vkGetBufferMemoryRequirements2" on Vulkan >= 1.1, fetch "vkGetBufferMemoryRequirements2KHR" when using VK_KHR_dedicated_allocation extension.
978 PFN_vkGetBufferMemoryRequirements2KHR VMA_NULLABLE vkGetBufferMemoryRequirements2KHR;
979 /// Fetch "vkGetImageMemoryRequirements2" on Vulkan >= 1.1, fetch "vkGetImageMemoryRequirements2KHR" when using VK_KHR_dedicated_allocation extension.
980 PFN_vkGetImageMemoryRequirements2KHR VMA_NULLABLE vkGetImageMemoryRequirements2KHR;
981#endif
982#if VMA_BIND_MEMORY2 || VMA_VULKAN_VERSION >= 1001000
983 /// Fetch "vkBindBufferMemory2" on Vulkan >= 1.1, fetch "vkBindBufferMemory2KHR" when using VK_KHR_bind_memory2 extension.
984 PFN_vkBindBufferMemory2KHR VMA_NULLABLE vkBindBufferMemory2KHR;
985 /// Fetch "vkBindImageMemory2" on Vulkan >= 1.1, fetch "vkBindImageMemory2KHR" when using VK_KHR_bind_memory2 extension.
986 PFN_vkBindImageMemory2KHR VMA_NULLABLE vkBindImageMemory2KHR;
987#endif
988#if VMA_MEMORY_BUDGET || VMA_VULKAN_VERSION >= 1001000
989 PFN_vkGetPhysicalDeviceMemoryProperties2KHR VMA_NULLABLE vkGetPhysicalDeviceMemoryProperties2KHR;
990#endif
991#if VMA_VULKAN_VERSION >= 1003000
992 /// Fetch from "vkGetDeviceBufferMemoryRequirements" on Vulkan >= 1.3, but you can also fetch it from "vkGetDeviceBufferMemoryRequirementsKHR" if you enabled extension VK_KHR_maintenance4.
993 PFN_vkGetDeviceBufferMemoryRequirements VMA_NULLABLE vkGetDeviceBufferMemoryRequirements;
994 /// Fetch from "vkGetDeviceImageMemoryRequirements" on Vulkan >= 1.3, but you can also fetch it from "vkGetDeviceImageMemoryRequirementsKHR" if you enabled extension VK_KHR_maintenance4.
995 PFN_vkGetDeviceImageMemoryRequirements VMA_NULLABLE vkGetDeviceImageMemoryRequirements;
996#endif
997} VmaVulkanFunctions;
998
999/// Description of a Allocator to be created.
1000typedef struct VmaAllocatorCreateInfo
1001{
1002 /// Flags for created allocator. Use #VmaAllocatorCreateFlagBits enum.
1003 VmaAllocatorCreateFlags flags;
1004 /// Vulkan physical device.
1005 /** It must be valid throughout whole lifetime of created allocator. */
1006 VkPhysicalDevice VMA_NOT_NULL physicalDevice;
1007 /// Vulkan device.
1008 /** It must be valid throughout whole lifetime of created allocator. */
1009 VkDevice VMA_NOT_NULL device;
1010 /// Preferred size of a single `VkDeviceMemory` block to be allocated from large heaps > 1 GiB. Optional.
1011 /** Set to 0 to use default, which is currently 256 MiB. */
1012 VkDeviceSize preferredLargeHeapBlockSize;
1013 /// Custom CPU memory allocation callbacks. Optional.
1014 /** Optional, can be null. When specified, will also be used for all CPU-side memory allocations. */
1015 const VkAllocationCallbacks* VMA_NULLABLE pAllocationCallbacks;
1016 /// Informative callbacks for `vkAllocateMemory`, `vkFreeMemory`. Optional.
1017 /** Optional, can be null. */
1018 const VmaDeviceMemoryCallbacks* VMA_NULLABLE pDeviceMemoryCallbacks;
1019 /** \brief Either null or a pointer to an array of limits on maximum number of bytes that can be allocated out of particular Vulkan memory heap.
1020
1021 If not NULL, it must be a pointer to an array of
1022 `VkPhysicalDeviceMemoryProperties::memoryHeapCount` elements, defining limit on
1023 maximum number of bytes that can be allocated out of particular Vulkan memory
1024 heap.
1025
1026 Any of the elements may be equal to `VK_WHOLE_SIZE`, which means no limit on that
1027 heap. This is also the default in case of `pHeapSizeLimit` = NULL.
1028
1029 If there is a limit defined for a heap:
1030
1031 - If user tries to allocate more memory from that heap using this allocator,
1032 the allocation fails with `VK_ERROR_OUT_OF_DEVICE_MEMORY`.
1033 - If the limit is smaller than heap size reported in `VkMemoryHeap::size`, the
1034 value of this limit will be reported instead when using vmaGetMemoryProperties().
1035
1036 Warning! Using this feature may not be equivalent to installing a GPU with
1037 smaller amount of memory, because graphics driver doesn't necessary fail new
1038 allocations with `VK_ERROR_OUT_OF_DEVICE_MEMORY` result when memory capacity is
1039 exceeded. It may return success and just silently migrate some device memory
1040 blocks to system RAM. This driver behavior can also be controlled using
1041 VK_AMD_memory_overallocation_behavior extension.
1042 */
1043 const VkDeviceSize* VMA_NULLABLE VMA_LEN_IF_NOT_NULL("VkPhysicalDeviceMemoryProperties::memoryHeapCount") pHeapSizeLimit;
1044
1045 /** \brief Pointers to Vulkan functions. Can be null.
1046
1047 For details see [Pointers to Vulkan functions](@ref config_Vulkan_functions).
1048 */
1049 const VmaVulkanFunctions* VMA_NULLABLE pVulkanFunctions;
1050 /** \brief Handle to Vulkan instance object.
1051
1052 Starting from version 3.0.0 this member is no longer optional, it must be set!
1053 */
1054 VkInstance VMA_NOT_NULL instance;
1055 /** \brief Optional. The highest version of Vulkan that the application is designed to use.
1056
1057 It must be a value in the format as created by macro `VK_MAKE_VERSION` or a constant like: `VK_API_VERSION_1_1`, `VK_API_VERSION_1_0`.
1058 The patch version number specified is ignored. Only the major and minor versions are considered.
1059 It must be less or equal (preferably equal) to value as passed to `vkCreateInstance` as `VkApplicationInfo::apiVersion`.
1060 Only versions 1.0, 1.1, 1.2, 1.3 are supported by the current implementation.
1061 Leaving it initialized to zero is equivalent to `VK_API_VERSION_1_0`.
1062 */
1063 uint32_t vulkanApiVersion;
1064#if VMA_EXTERNAL_MEMORY
1065 /** \brief Either null or a pointer to an array of external memory handle types for each Vulkan memory type.
1066
1067 If not NULL, it must be a pointer to an array of `VkPhysicalDeviceMemoryProperties::memoryTypeCount`
1068 elements, defining external memory handle types of particular Vulkan memory type,
1069 to be passed using `VkExportMemoryAllocateInfoKHR`.
1070
1071 Any of the elements may be equal to 0, which means not to use `VkExportMemoryAllocateInfoKHR` on this memory type.
1072 This is also the default in case of `pTypeExternalMemoryHandleTypes` = NULL.
1073 */
1074 const VkExternalMemoryHandleTypeFlagsKHR* VMA_NULLABLE VMA_LEN_IF_NOT_NULL("VkPhysicalDeviceMemoryProperties::memoryTypeCount") pTypeExternalMemoryHandleTypes;
1075#endif // #if VMA_EXTERNAL_MEMORY
1076} VmaAllocatorCreateInfo;
1077
1078/// Information about existing #VmaAllocator object.
1079typedef struct VmaAllocatorInfo
1080{
1081 /** \brief Handle to Vulkan instance object.
1082
1083 This is the same value as has been passed through VmaAllocatorCreateInfo::instance.
1084 */
1085 VkInstance VMA_NOT_NULL instance;
1086 /** \brief Handle to Vulkan physical device object.
1087
1088 This is the same value as has been passed through VmaAllocatorCreateInfo::physicalDevice.
1089 */
1090 VkPhysicalDevice VMA_NOT_NULL physicalDevice;
1091 /** \brief Handle to Vulkan device object.
1092
1093 This is the same value as has been passed through VmaAllocatorCreateInfo::device.
1094 */
1095 VkDevice VMA_NOT_NULL device;
1096} VmaAllocatorInfo;
1097
1098/** @} */
1099
1100/**
1101\addtogroup group_stats
1102@{
1103*/
1104
1105/** \brief Calculated statistics of memory usage e.g. in a specific memory type, heap, custom pool, or total.
1106
1107These are fast to calculate.
1108See functions: vmaGetHeapBudgets(), vmaGetPoolStatistics().
1109*/
1110typedef struct VmaStatistics
1111{
1112 /** \brief Number of `VkDeviceMemory` objects - Vulkan memory blocks allocated.
1113 */
1114 uint32_t blockCount;
1115 /** \brief Number of #VmaAllocation objects allocated.
1116
1117 Dedicated allocations have their own blocks, so each one adds 1 to `allocationCount` as well as `blockCount`.
1118 */
1119 uint32_t allocationCount;
1120 /** \brief Number of bytes allocated in `VkDeviceMemory` blocks.
1121
1122 \note To avoid confusion, please be aware that what Vulkan calls an "allocation" - a whole `VkDeviceMemory` object
1123 (e.g. as in `VkPhysicalDeviceLimits::maxMemoryAllocationCount`) is called a "block" in VMA, while VMA calls
1124 "allocation" a #VmaAllocation object that represents a memory region sub-allocated from such block, usually for a single buffer or image.
1125 */
1126 VkDeviceSize blockBytes;
1127 /** \brief Total number of bytes occupied by all #VmaAllocation objects.
1128
1129 Always less or equal than `blockBytes`.
1130 Difference `(blockBytes - allocationBytes)` is the amount of memory allocated from Vulkan
1131 but unused by any #VmaAllocation.
1132 */
1133 VkDeviceSize allocationBytes;
1134} VmaStatistics;
1135
1136/** \brief More detailed statistics than #VmaStatistics.
1137
1138These are slower to calculate. Use for debugging purposes.
1139See functions: vmaCalculateStatistics(), vmaCalculatePoolStatistics().
1140
1141Previous version of the statistics API provided averages, but they have been removed
1142because they can be easily calculated as:
1143
1144\code
1145VkDeviceSize allocationSizeAvg = detailedStats.statistics.allocationBytes / detailedStats.statistics.allocationCount;
1146VkDeviceSize unusedBytes = detailedStats.statistics.blockBytes - detailedStats.statistics.allocationBytes;
1147VkDeviceSize unusedRangeSizeAvg = unusedBytes / detailedStats.unusedRangeCount;
1148\endcode
1149*/
1150typedef struct VmaDetailedStatistics
1151{
1152 /// Basic statistics.
1153 VmaStatistics statistics;
1154 /// Number of free ranges of memory between allocations.
1155 uint32_t unusedRangeCount;
1156 /// Smallest allocation size. `VK_WHOLE_SIZE` if there are 0 allocations.
1157 VkDeviceSize allocationSizeMin;
1158 /// Largest allocation size. 0 if there are 0 allocations.
1159 VkDeviceSize allocationSizeMax;
1160 /// Smallest empty range size. `VK_WHOLE_SIZE` if there are 0 empty ranges.
1161 VkDeviceSize unusedRangeSizeMin;
1162 /// Largest empty range size. 0 if there are 0 empty ranges.
1163 VkDeviceSize unusedRangeSizeMax;
1164} VmaDetailedStatistics;
1165
1166/** \brief General statistics from current state of the Allocator -
1167total memory usage across all memory heaps and types.
1168
1169These are slower to calculate. Use for debugging purposes.
1170See function vmaCalculateStatistics().
1171*/
1172typedef struct VmaTotalStatistics
1173{
1174 VmaDetailedStatistics memoryType[VK_MAX_MEMORY_TYPES];
1175 VmaDetailedStatistics memoryHeap[VK_MAX_MEMORY_HEAPS];
1176 VmaDetailedStatistics total;
1177} VmaTotalStatistics;
1178
1179/** \brief Statistics of current memory usage and available budget for a specific memory heap.
1180
1181These are fast to calculate.
1182See function vmaGetHeapBudgets().
1183*/
1184typedef struct VmaBudget
1185{
1186 /** \brief Statistics fetched from the library.
1187 */
1188 VmaStatistics statistics;
1189 /** \brief Estimated current memory usage of the program, in bytes.
1190
1191 Fetched from system using VK_EXT_memory_budget extension if enabled.
1192
1193 It might be different than `statistics.blockBytes` (usually higher) due to additional implicit objects
1194 also occupying the memory, like swapchain, pipelines, descriptor heaps, command buffers, or
1195 `VkDeviceMemory` blocks allocated outside of this library, if any.
1196 */
1197 VkDeviceSize usage;
1198 /** \brief Estimated amount of memory available to the program, in bytes.
1199
1200 Fetched from system using VK_EXT_memory_budget extension if enabled.
1201
1202 It might be different (most probably smaller) than `VkMemoryHeap::size[heapIndex]` due to factors
1203 external to the program, decided by the operating system.
1204 Difference `budget - usage` is the amount of additional memory that can probably
1205 be allocated without problems. Exceeding the budget may result in various problems.
1206 */
1207 VkDeviceSize budget;
1208} VmaBudget;
1209
1210/** @} */
1211
1212/**
1213\addtogroup group_alloc
1214@{
1215*/
1216
1217/** \brief Parameters of new #VmaAllocation.
1218
1219To be used with functions like vmaCreateBuffer(), vmaCreateImage(), and many others.
1220*/
1221typedef struct VmaAllocationCreateInfo
1222{
1223 /// Use #VmaAllocationCreateFlagBits enum.
1224 VmaAllocationCreateFlags flags;
1225 /** \brief Intended usage of memory.
1226
1227 You can leave #VMA_MEMORY_USAGE_UNKNOWN if you specify memory requirements in other way. \n
1228 If `pool` is not null, this member is ignored.
1229 */
1230 VmaMemoryUsage usage;
1231 /** \brief Flags that must be set in a Memory Type chosen for an allocation.
1232
1233 Leave 0 if you specify memory requirements in other way. \n
1234 If `pool` is not null, this member is ignored.*/
1235 VkMemoryPropertyFlags requiredFlags;
1236 /** \brief Flags that preferably should be set in a memory type chosen for an allocation.
1237
1238 Set to 0 if no additional flags are preferred. \n
1239 If `pool` is not null, this member is ignored. */
1240 VkMemoryPropertyFlags preferredFlags;
1241 /** \brief Bitmask containing one bit set for every memory type acceptable for this allocation.
1242
1243 Value 0 is equivalent to `UINT32_MAX` - it means any memory type is accepted if
1244 it meets other requirements specified by this structure, with no further
1245 restrictions on memory type index. \n
1246 If `pool` is not null, this member is ignored.
1247 */
1248 uint32_t memoryTypeBits;
1249 /** \brief Pool that this allocation should be created in.
1250
1251 Leave `VK_NULL_HANDLE` to allocate from default pool. If not null, members:
1252 `usage`, `requiredFlags`, `preferredFlags`, `memoryTypeBits` are ignored.
1253 */
1254 VmaPool VMA_NULLABLE pool;
1255 /** \brief Custom general-purpose pointer that will be stored in #VmaAllocation, can be read as VmaAllocationInfo::pUserData and changed using vmaSetAllocationUserData().
1256
1257 If #VMA_ALLOCATION_CREATE_USER_DATA_COPY_STRING_BIT is used, it must be either
1258 null or pointer to a null-terminated string. The string will be then copied to
1259 internal buffer, so it doesn't need to be valid after allocation call.
1260 */
1261 void* VMA_NULLABLE pUserData;
1262 /** \brief A floating-point value between 0 and 1, indicating the priority of the allocation relative to other memory allocations.
1263
1264 It is used only when #VMA_ALLOCATOR_CREATE_EXT_MEMORY_PRIORITY_BIT flag was used during creation of the #VmaAllocator object
1265 and this allocation ends up as dedicated or is explicitly forced as dedicated using #VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT.
1266 Otherwise, it has the priority of a memory block where it is placed and this variable is ignored.
1267 */
1268 float priority;
1269} VmaAllocationCreateInfo;
1270
1271/// Describes parameter of created #VmaPool.
1272typedef struct VmaPoolCreateInfo
1273{
1274 /** \brief Vulkan memory type index to allocate this pool from.
1275 */
1276 uint32_t memoryTypeIndex;
1277 /** \brief Use combination of #VmaPoolCreateFlagBits.
1278 */
1279 VmaPoolCreateFlags flags;
1280 /** \brief Size of a single `VkDeviceMemory` block to be allocated as part of this pool, in bytes. Optional.
1281
1282 Specify nonzero to set explicit, constant size of memory blocks used by this
1283 pool.
1284
1285 Leave 0 to use default and let the library manage block sizes automatically.
1286 Sizes of particular blocks may vary.
1287 In this case, the pool will also support dedicated allocations.
1288 */
1289 VkDeviceSize blockSize;
1290 /** \brief Minimum number of blocks to be always allocated in this pool, even if they stay empty.
1291
1292 Set to 0 to have no preallocated blocks and allow the pool be completely empty.
1293 */
1294 size_t minBlockCount;
1295 /** \brief Maximum number of blocks that can be allocated in this pool. Optional.
1296
1297 Set to 0 to use default, which is `SIZE_MAX`, which means no limit.
1298
1299 Set to same value as VmaPoolCreateInfo::minBlockCount to have fixed amount of memory allocated
1300 throughout whole lifetime of this pool.
1301 */
1302 size_t maxBlockCount;
1303 /** \brief A floating-point value between 0 and 1, indicating the priority of the allocations in this pool relative to other memory allocations.
1304
1305 It is used only when #VMA_ALLOCATOR_CREATE_EXT_MEMORY_PRIORITY_BIT flag was used during creation of the #VmaAllocator object.
1306 Otherwise, this variable is ignored.
1307 */
1308 float priority;
1309 /** \brief Additional minimum alignment to be used for all allocations created from this pool. Can be 0.
1310
1311 Leave 0 (default) not to impose any additional alignment. If not 0, it must be a power of two.
1312 It can be useful in cases where alignment returned by Vulkan by functions like `vkGetBufferMemoryRequirements` is not enough,
1313 e.g. when doing interop with OpenGL.
1314 */
1315 VkDeviceSize minAllocationAlignment;
1316 /** \brief Additional `pNext` chain to be attached to `VkMemoryAllocateInfo` used for every allocation made by this pool. Optional.
1317
1318 Optional, can be null. If not null, it must point to a `pNext` chain of structures that can be attached to `VkMemoryAllocateInfo`.
1319 It can be useful for special needs such as adding `VkExportMemoryAllocateInfoKHR`.
1320 Structures pointed by this member must remain alive and unchanged for the whole lifetime of the custom pool.
1321
1322 Please note that some structures, e.g. `VkMemoryPriorityAllocateInfoEXT`, `VkMemoryDedicatedAllocateInfoKHR`,
1323 can be attached automatically by this library when using other, more convenient of its features.
1324 */
1325 void* VMA_NULLABLE pMemoryAllocateNext;
1326} VmaPoolCreateInfo;
1327
1328/** @} */
1329
1330/**
1331\addtogroup group_alloc
1332@{
1333*/
1334
1335/// Parameters of #VmaAllocation objects, that can be retrieved using function vmaGetAllocationInfo().
1336typedef struct VmaAllocationInfo
1337{
1338 /** \brief Memory type index that this allocation was allocated from.
1339
1340 It never changes.
1341 */
1342 uint32_t memoryType;
1343 /** \brief Handle to Vulkan memory object.
1344
1345 Same memory object can be shared by multiple allocations.
1346
1347 It can change after the allocation is moved during \ref defragmentation.
1348 */
1349 VkDeviceMemory VMA_NULLABLE_NON_DISPATCHABLE deviceMemory;
1350 /** \brief Offset in `VkDeviceMemory` object to the beginning of this allocation, in bytes. `(deviceMemory, offset)` pair is unique to this allocation.
1351
1352 You usually don't need to use this offset. If you create a buffer or an image together with the allocation using e.g. function
1353 vmaCreateBuffer(), vmaCreateImage(), functions that operate on these resources refer to the beginning of the buffer or image,
1354 not entire device memory block. Functions like vmaMapMemory(), vmaBindBufferMemory() also refer to the beginning of the allocation
1355 and apply this offset automatically.
1356
1357 It can change after the allocation is moved during \ref defragmentation.
1358 */
1359 VkDeviceSize offset;
1360 /** \brief Size of this allocation, in bytes.
1361
1362 It never changes.
1363
1364 \note Allocation size returned in this variable may be greater than the size
1365 requested for the resource e.g. as `VkBufferCreateInfo::size`. Whole size of the
1366 allocation is accessible for operations on memory e.g. using a pointer after
1367 mapping with vmaMapMemory(), but operations on the resource e.g. using
1368 `vkCmdCopyBuffer` must be limited to the size of the resource.
1369 */
1370 VkDeviceSize size;
1371 /** \brief Pointer to the beginning of this allocation as mapped data.
1372
1373 If the allocation hasn't been mapped using vmaMapMemory() and hasn't been
1374 created with #VMA_ALLOCATION_CREATE_MAPPED_BIT flag, this value is null.
1375
1376 It can change after call to vmaMapMemory(), vmaUnmapMemory().
1377 It can also change after the allocation is moved during \ref defragmentation.
1378 */
1379 void* VMA_NULLABLE pMappedData;
1380 /** \brief Custom general-purpose pointer that was passed as VmaAllocationCreateInfo::pUserData or set using vmaSetAllocationUserData().
1381
1382 It can change after call to vmaSetAllocationUserData() for this allocation.
1383 */
1384 void* VMA_NULLABLE pUserData;
1385 /** \brief Custom allocation name that was set with vmaSetAllocationName().
1386
1387 It can change after call to vmaSetAllocationName() for this allocation.
1388
1389 Another way to set custom name is to pass it in VmaAllocationCreateInfo::pUserData with
1390 additional flag #VMA_ALLOCATION_CREATE_USER_DATA_COPY_STRING_BIT set [DEPRECATED].
1391 */
1392 const char* VMA_NULLABLE pName;
1393} VmaAllocationInfo;
1394
1395/** \brief Parameters for defragmentation.
1396
1397To be used with function vmaBeginDefragmentation().
1398*/
1399typedef struct VmaDefragmentationInfo
1400{
1401 /// \brief Use combination of #VmaDefragmentationFlagBits.
1402 VmaDefragmentationFlags flags;
1403 /** \brief Custom pool to be defragmented.
1404
1405 If null then default pools will undergo defragmentation process.
1406 */
1407 VmaPool VMA_NULLABLE pool;
1408 /** \brief Maximum numbers of bytes that can be copied during single pass, while moving allocations to different places.
1409
1410 `0` means no limit.
1411 */
1412 VkDeviceSize maxBytesPerPass;
1413 /** \brief Maximum number of allocations that can be moved during single pass to a different place.
1414
1415 `0` means no limit.
1416 */
1417 uint32_t maxAllocationsPerPass;
1418} VmaDefragmentationInfo;
1419
1420/// Single move of an allocation to be done for defragmentation.
1421typedef struct VmaDefragmentationMove
1422{
1423 /// Operation to be performed on the allocation by vmaEndDefragmentationPass(). Default value is #VMA_DEFRAGMENTATION_MOVE_OPERATION_COPY. You can modify it.
1424 VmaDefragmentationMoveOperation operation;
1425 /// Allocation that should be moved.
1426 VmaAllocation VMA_NOT_NULL srcAllocation;
1427 /** \brief Temporary allocation pointing to destination memory that will replace `srcAllocation`.
1428
1429 \warning Do not store this allocation in your data structures! It exists only temporarily, for the duration of the defragmentation pass,
1430 to be used for binding new buffer/image to the destination memory using e.g. vmaBindBufferMemory().
1431 vmaEndDefragmentationPass() will destroy it and make `srcAllocation` point to this memory.
1432 */
1433 VmaAllocation VMA_NOT_NULL dstTmpAllocation;
1434} VmaDefragmentationMove;
1435
1436/** \brief Parameters for incremental defragmentation steps.
1437
1438To be used with function vmaBeginDefragmentationPass().
1439*/
1440typedef struct VmaDefragmentationPassMoveInfo
1441{
1442 /// Number of elements in the `pMoves` array.
1443 uint32_t moveCount;
1444 /** \brief Array of moves to be performed by the user in the current defragmentation pass.
1445
1446 Pointer to an array of `moveCount` elements, owned by VMA, created in vmaBeginDefragmentationPass(), destroyed in vmaEndDefragmentationPass().
1447
1448 For each element, you should:
1449
1450 1. Create a new buffer/image in the place pointed by VmaDefragmentationMove::dstMemory + VmaDefragmentationMove::dstOffset.
1451 2. Copy data from the VmaDefragmentationMove::srcAllocation e.g. using `vkCmdCopyBuffer`, `vkCmdCopyImage`.
1452 3. Make sure these commands finished executing on the GPU.
1453 4. Destroy the old buffer/image.
1454
1455 Only then you can finish defragmentation pass by calling vmaEndDefragmentationPass().
1456 After this call, the allocation will point to the new place in memory.
1457
1458 Alternatively, if you cannot move specific allocation, you can set VmaDefragmentationMove::operation to #VMA_DEFRAGMENTATION_MOVE_OPERATION_IGNORE.
1459
1460 Alternatively, if you decide you want to completely remove the allocation:
1461
1462 1. Destroy its buffer/image.
1463 2. Set VmaDefragmentationMove::operation to #VMA_DEFRAGMENTATION_MOVE_OPERATION_DESTROY.
1464
1465 Then, after vmaEndDefragmentationPass() the allocation will be freed.
1466 */
1467 VmaDefragmentationMove* VMA_NULLABLE VMA_LEN_IF_NOT_NULL(moveCount) pMoves;
1468} VmaDefragmentationPassMoveInfo;
1469
1470/// Statistics returned for defragmentation process in function vmaEndDefragmentation().
1471typedef struct VmaDefragmentationStats
1472{
1473 /// Total number of bytes that have been copied while moving allocations to different places.
1474 VkDeviceSize bytesMoved;
1475 /// Total number of bytes that have been released to the system by freeing empty `VkDeviceMemory` objects.
1476 VkDeviceSize bytesFreed;
1477 /// Number of allocations that have been moved to different places.
1478 uint32_t allocationsMoved;
1479 /// Number of empty `VkDeviceMemory` objects that have been released to the system.
1480 uint32_t deviceMemoryBlocksFreed;
1481} VmaDefragmentationStats;
1482
1483/** @} */
1484
1485/**
1486\addtogroup group_virtual
1487@{
1488*/
1489
1490/// Parameters of created #VmaVirtualBlock object to be passed to vmaCreateVirtualBlock().
1491typedef struct VmaVirtualBlockCreateInfo
1492{
1493 /** \brief Total size of the virtual block.
1494
1495 Sizes can be expressed in bytes or any units you want as long as you are consistent in using them.
1496 For example, if you allocate from some array of structures, 1 can mean single instance of entire structure.
1497 */
1498 VkDeviceSize size;
1499
1500 /** \brief Use combination of #VmaVirtualBlockCreateFlagBits.
1501 */
1502 VmaVirtualBlockCreateFlags flags;
1503
1504 /** \brief Custom CPU memory allocation callbacks. Optional.
1505
1506 Optional, can be null. When specified, they will be used for all CPU-side memory allocations.
1507 */
1508 const VkAllocationCallbacks* VMA_NULLABLE pAllocationCallbacks;
1509} VmaVirtualBlockCreateInfo;
1510
1511/// Parameters of created virtual allocation to be passed to vmaVirtualAllocate().
1512typedef struct VmaVirtualAllocationCreateInfo
1513{
1514 /** \brief Size of the allocation.
1515
1516 Cannot be zero.
1517 */
1518 VkDeviceSize size;
1519 /** \brief Required alignment of the allocation. Optional.
1520
1521 Must be power of two. Special value 0 has the same meaning as 1 - means no special alignment is required, so allocation can start at any offset.
1522 */
1523 VkDeviceSize alignment;
1524 /** \brief Use combination of #VmaVirtualAllocationCreateFlagBits.
1525 */
1526 VmaVirtualAllocationCreateFlags flags;
1527 /** \brief Custom pointer to be associated with the allocation. Optional.
1528
1529 It can be any value and can be used for user-defined purposes. It can be fetched or changed later.
1530 */
1531 void* VMA_NULLABLE pUserData;
1532} VmaVirtualAllocationCreateInfo;
1533
1534/// Parameters of an existing virtual allocation, returned by vmaGetVirtualAllocationInfo().
1535typedef struct VmaVirtualAllocationInfo
1536{
1537 /** \brief Offset of the allocation.
1538
1539 Offset at which the allocation was made.
1540 */
1541 VkDeviceSize offset;
1542 /** \brief Size of the allocation.
1543
1544 Same value as passed in VmaVirtualAllocationCreateInfo::size.
1545 */
1546 VkDeviceSize size;
1547 /** \brief Custom pointer associated with the allocation.
1548
1549 Same value as passed in VmaVirtualAllocationCreateInfo::pUserData or to vmaSetVirtualAllocationUserData().
1550 */
1551 void* VMA_NULLABLE pUserData;
1552} VmaVirtualAllocationInfo;
1553
1554/** @} */
1555
1556#endif // _VMA_DATA_TYPES_DECLARATIONS
1557
1558#ifndef _VMA_FUNCTION_HEADERS
1559
1560/**
1561\addtogroup group_init
1562@{
1563*/
1564
1565/// Creates #VmaAllocator object.
1566VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateAllocator(
1567 const VmaAllocatorCreateInfo* VMA_NOT_NULL pCreateInfo,
1568 VmaAllocator VMA_NULLABLE* VMA_NOT_NULL pAllocator);
1569
1570/// Destroys allocator object.
1571VMA_CALL_PRE void VMA_CALL_POST vmaDestroyAllocator(
1572 VmaAllocator VMA_NULLABLE allocator);
1573
1574/** \brief Returns information about existing #VmaAllocator object - handle to Vulkan device etc.
1575
1576It might be useful if you want to keep just the #VmaAllocator handle and fetch other required handles to
1577`VkPhysicalDevice`, `VkDevice` etc. every time using this function.
1578*/
1579VMA_CALL_PRE void VMA_CALL_POST vmaGetAllocatorInfo(
1580 VmaAllocator VMA_NOT_NULL allocator,
1581 VmaAllocatorInfo* VMA_NOT_NULL pAllocatorInfo);
1582
1583/**
1584PhysicalDeviceProperties are fetched from physicalDevice by the allocator.
1585You can access it here, without fetching it again on your own.
1586*/
1587VMA_CALL_PRE void VMA_CALL_POST vmaGetPhysicalDeviceProperties(
1588 VmaAllocator VMA_NOT_NULL allocator,
1589 const VkPhysicalDeviceProperties* VMA_NULLABLE* VMA_NOT_NULL ppPhysicalDeviceProperties);
1590
1591/**
1592PhysicalDeviceMemoryProperties are fetched from physicalDevice by the allocator.
1593You can access it here, without fetching it again on your own.
1594*/
1595VMA_CALL_PRE void VMA_CALL_POST vmaGetMemoryProperties(
1596 VmaAllocator VMA_NOT_NULL allocator,
1597 const VkPhysicalDeviceMemoryProperties* VMA_NULLABLE* VMA_NOT_NULL ppPhysicalDeviceMemoryProperties);
1598
1599/**
1600\brief Given Memory Type Index, returns Property Flags of this memory type.
1601
1602This is just a convenience function. Same information can be obtained using
1603vmaGetMemoryProperties().
1604*/
1605VMA_CALL_PRE void VMA_CALL_POST vmaGetMemoryTypeProperties(
1606 VmaAllocator VMA_NOT_NULL allocator,
1607 uint32_t memoryTypeIndex,
1608 VkMemoryPropertyFlags* VMA_NOT_NULL pFlags);
1609
1610/** \brief Sets index of the current frame.
1611*/
1612VMA_CALL_PRE void VMA_CALL_POST vmaSetCurrentFrameIndex(
1613 VmaAllocator VMA_NOT_NULL allocator,
1614 uint32_t frameIndex);
1615
1616/** @} */
1617
1618/**
1619\addtogroup group_stats
1620@{
1621*/
1622
1623/** \brief Retrieves statistics from current state of the Allocator.
1624
1625This function is called "calculate" not "get" because it has to traverse all
1626internal data structures, so it may be quite slow. Use it for debugging purposes.
1627For faster but more brief statistics suitable to be called every frame or every allocation,
1628use vmaGetHeapBudgets().
1629
1630Note that when using allocator from multiple threads, returned information may immediately
1631become outdated.
1632*/
1633VMA_CALL_PRE void VMA_CALL_POST vmaCalculateStatistics(
1634 VmaAllocator VMA_NOT_NULL allocator,
1635 VmaTotalStatistics* VMA_NOT_NULL pStats);
1636
1637/** \brief Retrieves information about current memory usage and budget for all memory heaps.
1638
1639\param allocator
1640\param[out] pBudgets Must point to array with number of elements at least equal to number of memory heaps in physical device used.
1641
1642This function is called "get" not "calculate" because it is very fast, suitable to be called
1643every frame or every allocation. For more detailed statistics use vmaCalculateStatistics().
1644
1645Note that when using allocator from multiple threads, returned information may immediately
1646become outdated.
1647*/
1648VMA_CALL_PRE void VMA_CALL_POST vmaGetHeapBudgets(
1649 VmaAllocator VMA_NOT_NULL allocator,
1650 VmaBudget* VMA_NOT_NULL VMA_LEN_IF_NOT_NULL("VkPhysicalDeviceMemoryProperties::memoryHeapCount") pBudgets);
1651
1652/** @} */
1653
1654/**
1655\addtogroup group_alloc
1656@{
1657*/
1658
1659/**
1660\brief Helps to find memoryTypeIndex, given memoryTypeBits and VmaAllocationCreateInfo.
1661
1662This algorithm tries to find a memory type that:
1663
1664- Is allowed by memoryTypeBits.
1665- Contains all the flags from pAllocationCreateInfo->requiredFlags.
1666- Matches intended usage.
1667- Has as many flags from pAllocationCreateInfo->preferredFlags as possible.
1668
1669\return Returns VK_ERROR_FEATURE_NOT_PRESENT if not found. Receiving such result
1670from this function or any other allocating function probably means that your
1671device doesn't support any memory type with requested features for the specific
1672type of resource you want to use it for. Please check parameters of your
1673resource, like image layout (OPTIMAL versus LINEAR) or mip level count.
1674*/
1675VMA_CALL_PRE VkResult VMA_CALL_POST vmaFindMemoryTypeIndex(
1676 VmaAllocator VMA_NOT_NULL allocator,
1677 uint32_t memoryTypeBits,
1678 const VmaAllocationCreateInfo* VMA_NOT_NULL pAllocationCreateInfo,
1679 uint32_t* VMA_NOT_NULL pMemoryTypeIndex);
1680
1681/**
1682\brief Helps to find memoryTypeIndex, given VkBufferCreateInfo and VmaAllocationCreateInfo.
1683
1684It can be useful e.g. to determine value to be used as VmaPoolCreateInfo::memoryTypeIndex.
1685It internally creates a temporary, dummy buffer that never has memory bound.
1686*/
1687VMA_CALL_PRE VkResult VMA_CALL_POST vmaFindMemoryTypeIndexForBufferInfo(
1688 VmaAllocator VMA_NOT_NULL allocator,
1689 const VkBufferCreateInfo* VMA_NOT_NULL pBufferCreateInfo,
1690 const VmaAllocationCreateInfo* VMA_NOT_NULL pAllocationCreateInfo,
1691 uint32_t* VMA_NOT_NULL pMemoryTypeIndex);
1692
1693/**
1694\brief Helps to find memoryTypeIndex, given VkImageCreateInfo and VmaAllocationCreateInfo.
1695
1696It can be useful e.g. to determine value to be used as VmaPoolCreateInfo::memoryTypeIndex.
1697It internally creates a temporary, dummy image that never has memory bound.
1698*/
1699VMA_CALL_PRE VkResult VMA_CALL_POST vmaFindMemoryTypeIndexForImageInfo(
1700 VmaAllocator VMA_NOT_NULL allocator,
1701 const VkImageCreateInfo* VMA_NOT_NULL pImageCreateInfo,
1702 const VmaAllocationCreateInfo* VMA_NOT_NULL pAllocationCreateInfo,
1703 uint32_t* VMA_NOT_NULL pMemoryTypeIndex);
1704
1705/** \brief Allocates Vulkan device memory and creates #VmaPool object.
1706
1707\param allocator Allocator object.
1708\param pCreateInfo Parameters of pool to create.
1709\param[out] pPool Handle to created pool.
1710*/
1711VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreatePool(
1712 VmaAllocator VMA_NOT_NULL allocator,
1713 const VmaPoolCreateInfo* VMA_NOT_NULL pCreateInfo,
1714 VmaPool VMA_NULLABLE* VMA_NOT_NULL pPool);
1715
1716/** \brief Destroys #VmaPool object and frees Vulkan device memory.
1717*/
1718VMA_CALL_PRE void VMA_CALL_POST vmaDestroyPool(
1719 VmaAllocator VMA_NOT_NULL allocator,
1720 VmaPool VMA_NULLABLE pool);
1721
1722/** @} */
1723
1724/**
1725\addtogroup group_stats
1726@{
1727*/
1728
1729/** \brief Retrieves statistics of existing #VmaPool object.
1730
1731\param allocator Allocator object.
1732\param pool Pool object.
1733\param[out] pPoolStats Statistics of specified pool.
1734*/
1735VMA_CALL_PRE void VMA_CALL_POST vmaGetPoolStatistics(
1736 VmaAllocator VMA_NOT_NULL allocator,
1737 VmaPool VMA_NOT_NULL pool,
1738 VmaStatistics* VMA_NOT_NULL pPoolStats);
1739
1740/** \brief Retrieves detailed statistics of existing #VmaPool object.
1741
1742\param allocator Allocator object.
1743\param pool Pool object.
1744\param[out] pPoolStats Statistics of specified pool.
1745*/
1746VMA_CALL_PRE void VMA_CALL_POST vmaCalculatePoolStatistics(
1747 VmaAllocator VMA_NOT_NULL allocator,
1748 VmaPool VMA_NOT_NULL pool,
1749 VmaDetailedStatistics* VMA_NOT_NULL pPoolStats);
1750
1751/** @} */
1752
1753/**
1754\addtogroup group_alloc
1755@{
1756*/
1757
1758/** \brief Checks magic number in margins around all allocations in given memory pool in search for corruptions.
1759
1760Corruption detection is enabled only when `VMA_DEBUG_DETECT_CORRUPTION` macro is defined to nonzero,
1761`VMA_DEBUG_MARGIN` is defined to nonzero and the pool is created in memory type that is
1762`HOST_VISIBLE` and `HOST_COHERENT`. For more information, see [Corruption detection](@ref debugging_memory_usage_corruption_detection).
1763
1764Possible return values:
1765
1766- `VK_ERROR_FEATURE_NOT_PRESENT` - corruption detection is not enabled for specified pool.
1767- `VK_SUCCESS` - corruption detection has been performed and succeeded.
1768- `VK_ERROR_UNKNOWN` - corruption detection has been performed and found memory corruptions around one of the allocations.
1769 `VMA_ASSERT` is also fired in that case.
1770- Other value: Error returned by Vulkan, e.g. memory mapping failure.
1771*/
1772VMA_CALL_PRE VkResult VMA_CALL_POST vmaCheckPoolCorruption(
1773 VmaAllocator VMA_NOT_NULL allocator,
1774 VmaPool VMA_NOT_NULL pool);
1775
1776/** \brief Retrieves name of a custom pool.
1777
1778After the call `ppName` is either null or points to an internally-owned null-terminated string
1779containing name of the pool that was previously set. The pointer becomes invalid when the pool is
1780destroyed or its name is changed using vmaSetPoolName().
1781*/
1782VMA_CALL_PRE void VMA_CALL_POST vmaGetPoolName(
1783 VmaAllocator VMA_NOT_NULL allocator,
1784 VmaPool VMA_NOT_NULL pool,
1785 const char* VMA_NULLABLE* VMA_NOT_NULL ppName);
1786
1787/** \brief Sets name of a custom pool.
1788
1789`pName` can be either null or pointer to a null-terminated string with new name for the pool.
1790Function makes internal copy of the string, so it can be changed or freed immediately after this call.
1791*/
1792VMA_CALL_PRE void VMA_CALL_POST vmaSetPoolName(
1793 VmaAllocator VMA_NOT_NULL allocator,
1794 VmaPool VMA_NOT_NULL pool,
1795 const char* VMA_NULLABLE pName);
1796
1797/** \brief General purpose memory allocation.
1798
1799\param allocator
1800\param pVkMemoryRequirements
1801\param pCreateInfo
1802\param[out] pAllocation Handle to allocated memory.
1803\param[out] pAllocationInfo Optional. Information about allocated memory. It can be later fetched using function vmaGetAllocationInfo().
1804
1805You should free the memory using vmaFreeMemory() or vmaFreeMemoryPages().
1806
1807It is recommended to use vmaAllocateMemoryForBuffer(), vmaAllocateMemoryForImage(),
1808vmaCreateBuffer(), vmaCreateImage() instead whenever possible.
1809*/
1810VMA_CALL_PRE VkResult VMA_CALL_POST vmaAllocateMemory(
1811 VmaAllocator VMA_NOT_NULL allocator,
1812 const VkMemoryRequirements* VMA_NOT_NULL pVkMemoryRequirements,
1813 const VmaAllocationCreateInfo* VMA_NOT_NULL pCreateInfo,
1814 VmaAllocation VMA_NULLABLE* VMA_NOT_NULL pAllocation,
1815 VmaAllocationInfo* VMA_NULLABLE pAllocationInfo);
1816
1817/** \brief General purpose memory allocation for multiple allocation objects at once.
1818
1819\param allocator Allocator object.
1820\param pVkMemoryRequirements Memory requirements for each allocation.
1821\param pCreateInfo Creation parameters for each allocation.
1822\param allocationCount Number of allocations to make.
1823\param[out] pAllocations Pointer to array that will be filled with handles to created allocations.
1824\param[out] pAllocationInfo Optional. Pointer to array that will be filled with parameters of created allocations.
1825
1826You should free the memory using vmaFreeMemory() or vmaFreeMemoryPages().
1827
1828Word "pages" is just a suggestion to use this function to allocate pieces of memory needed for sparse binding.
1829It is just a general purpose allocation function able to make multiple allocations at once.
1830It may be internally optimized to be more efficient than calling vmaAllocateMemory() `allocationCount` times.
1831
1832All allocations are made using same parameters. All of them are created out of the same memory pool and type.
1833If any allocation fails, all allocations already made within this function call are also freed, so that when
1834returned result is not `VK_SUCCESS`, `pAllocation` array is always entirely filled with `VK_NULL_HANDLE`.
1835*/
1836VMA_CALL_PRE VkResult VMA_CALL_POST vmaAllocateMemoryPages(
1837 VmaAllocator VMA_NOT_NULL allocator,
1838 const VkMemoryRequirements* VMA_NOT_NULL VMA_LEN_IF_NOT_NULL(allocationCount) pVkMemoryRequirements,
1839 const VmaAllocationCreateInfo* VMA_NOT_NULL VMA_LEN_IF_NOT_NULL(allocationCount) pCreateInfo,
1840 size_t allocationCount,
1841 VmaAllocation VMA_NULLABLE* VMA_NOT_NULL VMA_LEN_IF_NOT_NULL(allocationCount) pAllocations,
1842 VmaAllocationInfo* VMA_NULLABLE VMA_LEN_IF_NOT_NULL(allocationCount) pAllocationInfo);
1843
1844/** \brief Allocates memory suitable for given `VkBuffer`.
1845
1846\param allocator
1847\param buffer
1848\param pCreateInfo
1849\param[out] pAllocation Handle to allocated memory.
1850\param[out] pAllocationInfo Optional. Information about allocated memory. It can be later fetched using function vmaGetAllocationInfo().
1851
1852It only creates #VmaAllocation. To bind the memory to the buffer, use vmaBindBufferMemory().
1853
1854This is a special-purpose function. In most cases you should use vmaCreateBuffer().
1855
1856You must free the allocation using vmaFreeMemory() when no longer needed.
1857*/
1858VMA_CALL_PRE VkResult VMA_CALL_POST vmaAllocateMemoryForBuffer(
1859 VmaAllocator VMA_NOT_NULL allocator,
1860 VkBuffer VMA_NOT_NULL_NON_DISPATCHABLE buffer,
1861 const VmaAllocationCreateInfo* VMA_NOT_NULL pCreateInfo,
1862 VmaAllocation VMA_NULLABLE* VMA_NOT_NULL pAllocation,
1863 VmaAllocationInfo* VMA_NULLABLE pAllocationInfo);
1864
1865/** \brief Allocates memory suitable for given `VkImage`.
1866
1867\param allocator
1868\param image
1869\param pCreateInfo
1870\param[out] pAllocation Handle to allocated memory.
1871\param[out] pAllocationInfo Optional. Information about allocated memory. It can be later fetched using function vmaGetAllocationInfo().
1872
1873It only creates #VmaAllocation. To bind the memory to the buffer, use vmaBindImageMemory().
1874
1875This is a special-purpose function. In most cases you should use vmaCreateImage().
1876
1877You must free the allocation using vmaFreeMemory() when no longer needed.
1878*/
1879VMA_CALL_PRE VkResult VMA_CALL_POST vmaAllocateMemoryForImage(
1880 VmaAllocator VMA_NOT_NULL allocator,
1881 VkImage VMA_NOT_NULL_NON_DISPATCHABLE image,
1882 const VmaAllocationCreateInfo* VMA_NOT_NULL pCreateInfo,
1883 VmaAllocation VMA_NULLABLE* VMA_NOT_NULL pAllocation,
1884 VmaAllocationInfo* VMA_NULLABLE pAllocationInfo);
1885
1886/** \brief Frees memory previously allocated using vmaAllocateMemory(), vmaAllocateMemoryForBuffer(), or vmaAllocateMemoryForImage().
1887
1888Passing `VK_NULL_HANDLE` as `allocation` is valid. Such function call is just skipped.
1889*/
1890VMA_CALL_PRE void VMA_CALL_POST vmaFreeMemory(
1891 VmaAllocator VMA_NOT_NULL allocator,
1892 const VmaAllocation VMA_NULLABLE allocation);
1893
1894/** \brief Frees memory and destroys multiple allocations.
1895
1896Word "pages" is just a suggestion to use this function to free pieces of memory used for sparse binding.
1897It is just a general purpose function to free memory and destroy allocations made using e.g. vmaAllocateMemory(),
1898vmaAllocateMemoryPages() and other functions.
1899It may be internally optimized to be more efficient than calling vmaFreeMemory() `allocationCount` times.
1900
1901Allocations in `pAllocations` array can come from any memory pools and types.
1902Passing `VK_NULL_HANDLE` as elements of `pAllocations` array is valid. Such entries are just skipped.
1903*/
1904VMA_CALL_PRE void VMA_CALL_POST vmaFreeMemoryPages(
1905 VmaAllocator VMA_NOT_NULL allocator,
1906 size_t allocationCount,
1907 const VmaAllocation VMA_NULLABLE* VMA_NOT_NULL VMA_LEN_IF_NOT_NULL(allocationCount) pAllocations);
1908
1909/** \brief Returns current information about specified allocation.
1910
1911Current paramteres of given allocation are returned in `pAllocationInfo`.
1912
1913Although this function doesn't lock any mutex, so it should be quite efficient,
1914you should avoid calling it too often.
1915You can retrieve same VmaAllocationInfo structure while creating your resource, from function
1916vmaCreateBuffer(), vmaCreateImage(). You can remember it if you are sure parameters don't change
1917(e.g. due to defragmentation).
1918*/
1919VMA_CALL_PRE void VMA_CALL_POST vmaGetAllocationInfo(
1920 VmaAllocator VMA_NOT_NULL allocator,
1921 VmaAllocation VMA_NOT_NULL allocation,
1922 VmaAllocationInfo* VMA_NOT_NULL pAllocationInfo);
1923
1924/** \brief Sets pUserData in given allocation to new value.
1925
1926The value of pointer `pUserData` is copied to allocation's `pUserData`.
1927It is opaque, so you can use it however you want - e.g.
1928as a pointer, ordinal number or some handle to you own data.
1929*/
1930VMA_CALL_PRE void VMA_CALL_POST vmaSetAllocationUserData(
1931 VmaAllocator VMA_NOT_NULL allocator,
1932 VmaAllocation VMA_NOT_NULL allocation,
1933 void* VMA_NULLABLE pUserData);
1934
1935/** \brief Sets pName in given allocation to new value.
1936
1937`pName` must be either null, or pointer to a null-terminated string. The function
1938makes local copy of the string and sets it as allocation's `pName`. String
1939passed as pName doesn't need to be valid for whole lifetime of the allocation -
1940you can free it after this call. String previously pointed by allocation's
1941`pName` is freed from memory.
1942*/
1943VMA_CALL_PRE void VMA_CALL_POST vmaSetAllocationName(
1944 VmaAllocator VMA_NOT_NULL allocator,
1945 VmaAllocation VMA_NOT_NULL allocation,
1946 const char* VMA_NULLABLE pName);
1947
1948/**
1949\brief Given an allocation, returns Property Flags of its memory type.
1950
1951This is just a convenience function. Same information can be obtained using
1952vmaGetAllocationInfo() + vmaGetMemoryProperties().
1953*/
1954VMA_CALL_PRE void VMA_CALL_POST vmaGetAllocationMemoryProperties(
1955 VmaAllocator VMA_NOT_NULL allocator,
1956 VmaAllocation VMA_NOT_NULL allocation,
1957 VkMemoryPropertyFlags* VMA_NOT_NULL pFlags);
1958
1959/** \brief Maps memory represented by given allocation and returns pointer to it.
1960
1961Maps memory represented by given allocation to make it accessible to CPU code.
1962When succeeded, `*ppData` contains pointer to first byte of this memory.
1963
1964\warning
1965If the allocation is part of a bigger `VkDeviceMemory` block, returned pointer is
1966correctly offsetted to the beginning of region assigned to this particular allocation.
1967Unlike the result of `vkMapMemory`, it points to the allocation, not to the beginning of the whole block.
1968You should not add VmaAllocationInfo::offset to it!
1969
1970Mapping is internally reference-counted and synchronized, so despite raw Vulkan
1971function `vkMapMemory()` cannot be used to map same block of `VkDeviceMemory`
1972multiple times simultaneously, it is safe to call this function on allocations
1973assigned to the same memory block. Actual Vulkan memory will be mapped on first
1974mapping and unmapped on last unmapping.
1975
1976If the function succeeded, you must call vmaUnmapMemory() to unmap the
1977allocation when mapping is no longer needed or before freeing the allocation, at
1978the latest.
1979
1980It also safe to call this function multiple times on the same allocation. You
1981must call vmaUnmapMemory() same number of times as you called vmaMapMemory().
1982
1983It is also safe to call this function on allocation created with
1984#VMA_ALLOCATION_CREATE_MAPPED_BIT flag. Its memory stays mapped all the time.
1985You must still call vmaUnmapMemory() same number of times as you called
1986vmaMapMemory(). You must not call vmaUnmapMemory() additional time to free the
1987"0-th" mapping made automatically due to #VMA_ALLOCATION_CREATE_MAPPED_BIT flag.
1988
1989This function fails when used on allocation made in memory type that is not
1990`HOST_VISIBLE`.
1991
1992This function doesn't automatically flush or invalidate caches.
1993If the allocation is made from a memory types that is not `HOST_COHERENT`,
1994you also need to use vmaInvalidateAllocation() / vmaFlushAllocation(), as required by Vulkan specification.
1995*/
1996VMA_CALL_PRE VkResult VMA_CALL_POST vmaMapMemory(
1997 VmaAllocator VMA_NOT_NULL allocator,
1998 VmaAllocation VMA_NOT_NULL allocation,
1999 void* VMA_NULLABLE* VMA_NOT_NULL ppData);
2000
2001/** \brief Unmaps memory represented by given allocation, mapped previously using vmaMapMemory().
2002
2003For details, see description of vmaMapMemory().
2004
2005This function doesn't automatically flush or invalidate caches.
2006If the allocation is made from a memory types that is not `HOST_COHERENT`,
2007you also need to use vmaInvalidateAllocation() / vmaFlushAllocation(), as required by Vulkan specification.
2008*/
2009VMA_CALL_PRE void VMA_CALL_POST vmaUnmapMemory(
2010 VmaAllocator VMA_NOT_NULL allocator,
2011 VmaAllocation VMA_NOT_NULL allocation);
2012
2013/** \brief Flushes memory of given allocation.
2014
2015Calls `vkFlushMappedMemoryRanges()` for memory associated with given range of given allocation.
2016It needs to be called after writing to a mapped memory for memory types that are not `HOST_COHERENT`.
2017Unmap operation doesn't do that automatically.
2018
2019- `offset` must be relative to the beginning of allocation.
2020- `size` can be `VK_WHOLE_SIZE`. It means all memory from `offset` the the end of given allocation.
2021- `offset` and `size` don't have to be aligned.
2022 They are internally rounded down/up to multiply of `nonCoherentAtomSize`.
2023- If `size` is 0, this call is ignored.
2024- If memory type that the `allocation` belongs to is not `HOST_VISIBLE` or it is `HOST_COHERENT`,
2025 this call is ignored.
2026
2027Warning! `offset` and `size` are relative to the contents of given `allocation`.
2028If you mean whole allocation, you can pass 0 and `VK_WHOLE_SIZE`, respectively.
2029Do not pass allocation's offset as `offset`!!!
2030
2031This function returns the `VkResult` from `vkFlushMappedMemoryRanges` if it is
2032called, otherwise `VK_SUCCESS`.
2033*/
2034VMA_CALL_PRE VkResult VMA_CALL_POST vmaFlushAllocation(
2035 VmaAllocator VMA_NOT_NULL allocator,
2036 VmaAllocation VMA_NOT_NULL allocation,
2037 VkDeviceSize offset,
2038 VkDeviceSize size);
2039
2040/** \brief Invalidates memory of given allocation.
2041
2042Calls `vkInvalidateMappedMemoryRanges()` for memory associated with given range of given allocation.
2043It needs to be called before reading from a mapped memory for memory types that are not `HOST_COHERENT`.
2044Map operation doesn't do that automatically.
2045
2046- `offset` must be relative to the beginning of allocation.
2047- `size` can be `VK_WHOLE_SIZE`. It means all memory from `offset` the the end of given allocation.
2048- `offset` and `size` don't have to be aligned.
2049 They are internally rounded down/up to multiply of `nonCoherentAtomSize`.
2050- If `size` is 0, this call is ignored.
2051- If memory type that the `allocation` belongs to is not `HOST_VISIBLE` or it is `HOST_COHERENT`,
2052 this call is ignored.
2053
2054Warning! `offset` and `size` are relative to the contents of given `allocation`.
2055If you mean whole allocation, you can pass 0 and `VK_WHOLE_SIZE`, respectively.
2056Do not pass allocation's offset as `offset`!!!
2057
2058This function returns the `VkResult` from `vkInvalidateMappedMemoryRanges` if
2059it is called, otherwise `VK_SUCCESS`.
2060*/
2061VMA_CALL_PRE VkResult VMA_CALL_POST vmaInvalidateAllocation(
2062 VmaAllocator VMA_NOT_NULL allocator,
2063 VmaAllocation VMA_NOT_NULL allocation,
2064 VkDeviceSize offset,
2065 VkDeviceSize size);
2066
2067/** \brief Flushes memory of given set of allocations.
2068
2069Calls `vkFlushMappedMemoryRanges()` for memory associated with given ranges of given allocations.
2070For more information, see documentation of vmaFlushAllocation().
2071
2072\param allocator
2073\param allocationCount
2074\param allocations
2075\param offsets If not null, it must point to an array of offsets of regions to flush, relative to the beginning of respective allocations. Null means all ofsets are zero.
2076\param sizes If not null, it must point to an array of sizes of regions to flush in respective allocations. Null means `VK_WHOLE_SIZE` for all allocations.
2077
2078This function returns the `VkResult` from `vkFlushMappedMemoryRanges` if it is
2079called, otherwise `VK_SUCCESS`.
2080*/
2081VMA_CALL_PRE VkResult VMA_CALL_POST vmaFlushAllocations(
2082 VmaAllocator VMA_NOT_NULL allocator,
2083 uint32_t allocationCount,
2084 const VmaAllocation VMA_NOT_NULL* VMA_NULLABLE VMA_LEN_IF_NOT_NULL(allocationCount) allocations,
2085 const VkDeviceSize* VMA_NULLABLE VMA_LEN_IF_NOT_NULL(allocationCount) offsets,
2086 const VkDeviceSize* VMA_NULLABLE VMA_LEN_IF_NOT_NULL(allocationCount) sizes);
2087
2088/** \brief Invalidates memory of given set of allocations.
2089
2090Calls `vkInvalidateMappedMemoryRanges()` for memory associated with given ranges of given allocations.
2091For more information, see documentation of vmaInvalidateAllocation().
2092
2093\param allocator
2094\param allocationCount
2095\param allocations
2096\param offsets If not null, it must point to an array of offsets of regions to flush, relative to the beginning of respective allocations. Null means all ofsets are zero.
2097\param sizes If not null, it must point to an array of sizes of regions to flush in respective allocations. Null means `VK_WHOLE_SIZE` for all allocations.
2098
2099This function returns the `VkResult` from `vkInvalidateMappedMemoryRanges` if it is
2100called, otherwise `VK_SUCCESS`.
2101*/
2102VMA_CALL_PRE VkResult VMA_CALL_POST vmaInvalidateAllocations(
2103 VmaAllocator VMA_NOT_NULL allocator,
2104 uint32_t allocationCount,
2105 const VmaAllocation VMA_NOT_NULL* VMA_NULLABLE VMA_LEN_IF_NOT_NULL(allocationCount) allocations,
2106 const VkDeviceSize* VMA_NULLABLE VMA_LEN_IF_NOT_NULL(allocationCount) offsets,
2107 const VkDeviceSize* VMA_NULLABLE VMA_LEN_IF_NOT_NULL(allocationCount) sizes);
2108
2109/** \brief Checks magic number in margins around all allocations in given memory types (in both default and custom pools) in search for corruptions.
2110
2111\param allocator
2112\param memoryTypeBits Bit mask, where each bit set means that a memory type with that index should be checked.
2113
2114Corruption detection is enabled only when `VMA_DEBUG_DETECT_CORRUPTION` macro is defined to nonzero,
2115`VMA_DEBUG_MARGIN` is defined to nonzero and only for memory types that are
2116`HOST_VISIBLE` and `HOST_COHERENT`. For more information, see [Corruption detection](@ref debugging_memory_usage_corruption_detection).
2117
2118Possible return values:
2119
2120- `VK_ERROR_FEATURE_NOT_PRESENT` - corruption detection is not enabled for any of specified memory types.
2121- `VK_SUCCESS` - corruption detection has been performed and succeeded.
2122- `VK_ERROR_UNKNOWN` - corruption detection has been performed and found memory corruptions around one of the allocations.
2123 `VMA_ASSERT` is also fired in that case.
2124- Other value: Error returned by Vulkan, e.g. memory mapping failure.
2125*/
2126VMA_CALL_PRE VkResult VMA_CALL_POST vmaCheckCorruption(
2127 VmaAllocator VMA_NOT_NULL allocator,
2128 uint32_t memoryTypeBits);
2129
2130/** \brief Begins defragmentation process.
2131
2132\param allocator Allocator object.
2133\param pInfo Structure filled with parameters of defragmentation.
2134\param[out] pContext Context object that must be passed to vmaEndDefragmentation() to finish defragmentation.
2135\returns
2136- `VK_SUCCESS` if defragmentation can begin.
2137- `VK_ERROR_FEATURE_NOT_PRESENT` if defragmentation is not supported.
2138
2139For more information about defragmentation, see documentation chapter:
2140[Defragmentation](@ref defragmentation).
2141*/
2142VMA_CALL_PRE VkResult VMA_CALL_POST vmaBeginDefragmentation(
2143 VmaAllocator VMA_NOT_NULL allocator,
2144 const VmaDefragmentationInfo* VMA_NOT_NULL pInfo,
2145 VmaDefragmentationContext VMA_NULLABLE* VMA_NOT_NULL pContext);
2146
2147/** \brief Ends defragmentation process.
2148
2149\param allocator Allocator object.
2150\param context Context object that has been created by vmaBeginDefragmentation().
2151\param[out] pStats Optional stats for the defragmentation. Can be null.
2152
2153Use this function to finish defragmentation started by vmaBeginDefragmentation().
2154*/
2155VMA_CALL_PRE void VMA_CALL_POST vmaEndDefragmentation(
2156 VmaAllocator VMA_NOT_NULL allocator,
2157 VmaDefragmentationContext VMA_NOT_NULL context,
2158 VmaDefragmentationStats* VMA_NULLABLE pStats);
2159
2160/** \brief Starts single defragmentation pass.
2161
2162\param allocator Allocator object.
2163\param context Context object that has been created by vmaBeginDefragmentation().
2164\param[out] pPassInfo Computed informations for current pass.
2165\returns
2166- `VK_SUCCESS` if no more moves are possible. Then you can omit call to vmaEndDefragmentationPass() and simply end whole defragmentation.
2167- `VK_INCOMPLETE` if there are pending moves returned in `pPassInfo`. You need to perform them, call vmaEndDefragmentationPass(),
2168 and then preferably try another pass with vmaBeginDefragmentationPass().
2169*/
2170VMA_CALL_PRE VkResult VMA_CALL_POST vmaBeginDefragmentationPass(
2171 VmaAllocator VMA_NOT_NULL allocator,
2172 VmaDefragmentationContext VMA_NOT_NULL context,
2173 VmaDefragmentationPassMoveInfo* VMA_NOT_NULL pPassInfo);
2174
2175/** \brief Ends single defragmentation pass.
2176
2177\param allocator Allocator object.
2178\param context Context object that has been created by vmaBeginDefragmentation().
2179\param pPassInfo Computed informations for current pass filled by vmaBeginDefragmentationPass() and possibly modified by you.
2180
2181Returns `VK_SUCCESS` if no more moves are possible or `VK_INCOMPLETE` if more defragmentations are possible.
2182
2183Ends incremental defragmentation pass and commits all defragmentation moves from `pPassInfo`.
2184After this call:
2185
2186- Allocations at `pPassInfo[i].srcAllocation` that had `pPassInfo[i].operation ==` #VMA_DEFRAGMENTATION_MOVE_OPERATION_COPY
2187 (which is the default) will be pointing to the new destination place.
2188- Allocation at `pPassInfo[i].srcAllocation` that had `pPassInfo[i].operation ==` #VMA_DEFRAGMENTATION_MOVE_OPERATION_DESTROY
2189 will be freed.
2190
2191If no more moves are possible you can end whole defragmentation.
2192*/
2193VMA_CALL_PRE VkResult VMA_CALL_POST vmaEndDefragmentationPass(
2194 VmaAllocator VMA_NOT_NULL allocator,
2195 VmaDefragmentationContext VMA_NOT_NULL context,
2196 VmaDefragmentationPassMoveInfo* VMA_NOT_NULL pPassInfo);
2197
2198/** \brief Binds buffer to allocation.
2199
2200Binds specified buffer to region of memory represented by specified allocation.
2201Gets `VkDeviceMemory` handle and offset from the allocation.
2202If you want to create a buffer, allocate memory for it and bind them together separately,
2203you should use this function for binding instead of standard `vkBindBufferMemory()`,
2204because it ensures proper synchronization so that when a `VkDeviceMemory` object is used by multiple
2205allocations, calls to `vkBind*Memory()` or `vkMapMemory()` won't happen from multiple threads simultaneously
2206(which is illegal in Vulkan).
2207
2208It is recommended to use function vmaCreateBuffer() instead of this one.
2209*/
2210VMA_CALL_PRE VkResult VMA_CALL_POST vmaBindBufferMemory(
2211 VmaAllocator VMA_NOT_NULL allocator,
2212 VmaAllocation VMA_NOT_NULL allocation,
2213 VkBuffer VMA_NOT_NULL_NON_DISPATCHABLE buffer);
2214
2215/** \brief Binds buffer to allocation with additional parameters.
2216
2217\param allocator
2218\param allocation
2219\param allocationLocalOffset Additional offset to be added while binding, relative to the beginning of the `allocation`. Normally it should be 0.
2220\param buffer
2221\param pNext A chain of structures to be attached to `VkBindBufferMemoryInfoKHR` structure used internally. Normally it should be null.
2222
2223This function is similar to vmaBindBufferMemory(), but it provides additional parameters.
2224
2225If `pNext` is not null, #VmaAllocator object must have been created with #VMA_ALLOCATOR_CREATE_KHR_BIND_MEMORY2_BIT flag
2226or with VmaAllocatorCreateInfo::vulkanApiVersion `>= VK_API_VERSION_1_1`. Otherwise the call fails.
2227*/
2228VMA_CALL_PRE VkResult VMA_CALL_POST vmaBindBufferMemory2(
2229 VmaAllocator VMA_NOT_NULL allocator,
2230 VmaAllocation VMA_NOT_NULL allocation,
2231 VkDeviceSize allocationLocalOffset,
2232 VkBuffer VMA_NOT_NULL_NON_DISPATCHABLE buffer,
2233 const void* VMA_NULLABLE pNext);
2234
2235/** \brief Binds image to allocation.
2236
2237Binds specified image to region of memory represented by specified allocation.
2238Gets `VkDeviceMemory` handle and offset from the allocation.
2239If you want to create an image, allocate memory for it and bind them together separately,
2240you should use this function for binding instead of standard `vkBindImageMemory()`,
2241because it ensures proper synchronization so that when a `VkDeviceMemory` object is used by multiple
2242allocations, calls to `vkBind*Memory()` or `vkMapMemory()` won't happen from multiple threads simultaneously
2243(which is illegal in Vulkan).
2244
2245It is recommended to use function vmaCreateImage() instead of this one.
2246*/
2247VMA_CALL_PRE VkResult VMA_CALL_POST vmaBindImageMemory(
2248 VmaAllocator VMA_NOT_NULL allocator,
2249 VmaAllocation VMA_NOT_NULL allocation,
2250 VkImage VMA_NOT_NULL_NON_DISPATCHABLE image);
2251
2252/** \brief Binds image to allocation with additional parameters.
2253
2254\param allocator
2255\param allocation
2256\param allocationLocalOffset Additional offset to be added while binding, relative to the beginning of the `allocation`. Normally it should be 0.
2257\param image
2258\param pNext A chain of structures to be attached to `VkBindImageMemoryInfoKHR` structure used internally. Normally it should be null.
2259
2260This function is similar to vmaBindImageMemory(), but it provides additional parameters.
2261
2262If `pNext` is not null, #VmaAllocator object must have been created with #VMA_ALLOCATOR_CREATE_KHR_BIND_MEMORY2_BIT flag
2263or with VmaAllocatorCreateInfo::vulkanApiVersion `>= VK_API_VERSION_1_1`. Otherwise the call fails.
2264*/
2265VMA_CALL_PRE VkResult VMA_CALL_POST vmaBindImageMemory2(
2266 VmaAllocator VMA_NOT_NULL allocator,
2267 VmaAllocation VMA_NOT_NULL allocation,
2268 VkDeviceSize allocationLocalOffset,
2269 VkImage VMA_NOT_NULL_NON_DISPATCHABLE image,
2270 const void* VMA_NULLABLE pNext);
2271
2272/** \brief Creates a new `VkBuffer`, allocates and binds memory for it.
2273
2274\param allocator
2275\param pBufferCreateInfo
2276\param pAllocationCreateInfo
2277\param[out] pBuffer Buffer that was created.
2278\param[out] pAllocation Allocation that was created.
2279\param[out] pAllocationInfo Optional. Information about allocated memory. It can be later fetched using function vmaGetAllocationInfo().
2280
2281This function automatically:
2282
2283-# Creates buffer.
2284-# Allocates appropriate memory for it.
2285-# Binds the buffer with the memory.
2286
2287If any of these operations fail, buffer and allocation are not created,
2288returned value is negative error code, `*pBuffer` and `*pAllocation` are null.
2289
2290If the function succeeded, you must destroy both buffer and allocation when you
2291no longer need them using either convenience function vmaDestroyBuffer() or
2292separately, using `vkDestroyBuffer()` and vmaFreeMemory().
2293
2294If #VMA_ALLOCATOR_CREATE_KHR_DEDICATED_ALLOCATION_BIT flag was used,
2295VK_KHR_dedicated_allocation extension is used internally to query driver whether
2296it requires or prefers the new buffer to have dedicated allocation. If yes,
2297and if dedicated allocation is possible
2298(#VMA_ALLOCATION_CREATE_NEVER_ALLOCATE_BIT is not used), it creates dedicated
2299allocation for this buffer, just like when using
2300#VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT.
2301
2302\note This function creates a new `VkBuffer`. Sub-allocation of parts of one large buffer,
2303although recommended as a good practice, is out of scope of this library and could be implemented
2304by the user as a higher-level logic on top of VMA.
2305*/
2306VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateBuffer(
2307 VmaAllocator VMA_NOT_NULL allocator,
2308 const VkBufferCreateInfo* VMA_NOT_NULL pBufferCreateInfo,
2309 const VmaAllocationCreateInfo* VMA_NOT_NULL pAllocationCreateInfo,
2310 VkBuffer VMA_NULLABLE_NON_DISPATCHABLE* VMA_NOT_NULL pBuffer,
2311 VmaAllocation VMA_NULLABLE* VMA_NOT_NULL pAllocation,
2312 VmaAllocationInfo* VMA_NULLABLE pAllocationInfo);
2313
2314/** \brief Creates a buffer with additional minimum alignment.
2315
2316Similar to vmaCreateBuffer() but provides additional parameter `minAlignment` which allows to specify custom,
2317minimum alignment to be used when placing the buffer inside a larger memory block, which may be needed e.g.
2318for interop with OpenGL.
2319*/
2320VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateBufferWithAlignment(
2321 VmaAllocator VMA_NOT_NULL allocator,
2322 const VkBufferCreateInfo* VMA_NOT_NULL pBufferCreateInfo,
2323 const VmaAllocationCreateInfo* VMA_NOT_NULL pAllocationCreateInfo,
2324 VkDeviceSize minAlignment,
2325 VkBuffer VMA_NULLABLE_NON_DISPATCHABLE* VMA_NOT_NULL pBuffer,
2326 VmaAllocation VMA_NULLABLE* VMA_NOT_NULL pAllocation,
2327 VmaAllocationInfo* VMA_NULLABLE pAllocationInfo);
2328
2329/** \brief Creates a new `VkBuffer`, binds already created memory for it.
2330
2331\param allocator
2332\param allocation Allocation that provides memory to be used for binding new buffer to it.
2333\param pBufferCreateInfo
2334\param[out] pBuffer Buffer that was created.
2335
2336This function automatically:
2337
2338-# Creates buffer.
2339-# Binds the buffer with the supplied memory.
2340
2341If any of these operations fail, buffer is not created,
2342returned value is negative error code and `*pBuffer` is null.
2343
2344If the function succeeded, you must destroy the buffer when you
2345no longer need it using `vkDestroyBuffer()`. If you want to also destroy the corresponding
2346allocation you can use convenience function vmaDestroyBuffer().
2347*/
2348VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateAliasingBuffer(
2349 VmaAllocator VMA_NOT_NULL allocator,
2350 VmaAllocation VMA_NOT_NULL allocation,
2351 const VkBufferCreateInfo* VMA_NOT_NULL pBufferCreateInfo,
2352 VkBuffer VMA_NULLABLE_NON_DISPATCHABLE* VMA_NOT_NULL pBuffer);
2353
2354/** \brief Destroys Vulkan buffer and frees allocated memory.
2355
2356This is just a convenience function equivalent to:
2357
2358\code
2359vkDestroyBuffer(device, buffer, allocationCallbacks);
2360vmaFreeMemory(allocator, allocation);
2361\endcode
2362
2363It it safe to pass null as buffer and/or allocation.
2364*/
2365VMA_CALL_PRE void VMA_CALL_POST vmaDestroyBuffer(
2366 VmaAllocator VMA_NOT_NULL allocator,
2367 VkBuffer VMA_NULLABLE_NON_DISPATCHABLE buffer,
2368 VmaAllocation VMA_NULLABLE allocation);
2369
2370/// Function similar to vmaCreateBuffer().
2371VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateImage(
2372 VmaAllocator VMA_NOT_NULL allocator,
2373 const VkImageCreateInfo* VMA_NOT_NULL pImageCreateInfo,
2374 const VmaAllocationCreateInfo* VMA_NOT_NULL pAllocationCreateInfo,
2375 VkImage VMA_NULLABLE_NON_DISPATCHABLE* VMA_NOT_NULL pImage,
2376 VmaAllocation VMA_NULLABLE* VMA_NOT_NULL pAllocation,
2377 VmaAllocationInfo* VMA_NULLABLE pAllocationInfo);
2378
2379/// Function similar to vmaCreateAliasingBuffer().
2380VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateAliasingImage(
2381 VmaAllocator VMA_NOT_NULL allocator,
2382 VmaAllocation VMA_NOT_NULL allocation,
2383 const VkImageCreateInfo* VMA_NOT_NULL pImageCreateInfo,
2384 VkImage VMA_NULLABLE_NON_DISPATCHABLE* VMA_NOT_NULL pImage);
2385
2386/** \brief Destroys Vulkan image and frees allocated memory.
2387
2388This is just a convenience function equivalent to:
2389
2390\code
2391vkDestroyImage(device, image, allocationCallbacks);
2392vmaFreeMemory(allocator, allocation);
2393\endcode
2394
2395It it safe to pass null as image and/or allocation.
2396*/
2397VMA_CALL_PRE void VMA_CALL_POST vmaDestroyImage(
2398 VmaAllocator VMA_NOT_NULL allocator,
2399 VkImage VMA_NULLABLE_NON_DISPATCHABLE image,
2400 VmaAllocation VMA_NULLABLE allocation);
2401
2402/** @} */
2403
2404/**
2405\addtogroup group_virtual
2406@{
2407*/
2408
2409/** \brief Creates new #VmaVirtualBlock object.
2410
2411\param pCreateInfo Parameters for creation.
2412\param[out] pVirtualBlock Returned virtual block object or `VMA_NULL` if creation failed.
2413*/
2414VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateVirtualBlock(
2415 const VmaVirtualBlockCreateInfo* VMA_NOT_NULL pCreateInfo,
2416 VmaVirtualBlock VMA_NULLABLE* VMA_NOT_NULL pVirtualBlock);
2417
2418/** \brief Destroys #VmaVirtualBlock object.
2419
2420Please note that you should consciously handle virtual allocations that could remain unfreed in the block.
2421You should either free them individually using vmaVirtualFree() or call vmaClearVirtualBlock()
2422if you are sure this is what you want. If you do neither, an assert is called.
2423
2424If you keep pointers to some additional metadata associated with your virtual allocations in their `pUserData`,
2425don't forget to free them.
2426*/
2427VMA_CALL_PRE void VMA_CALL_POST vmaDestroyVirtualBlock(
2428 VmaVirtualBlock VMA_NULLABLE virtualBlock);
2429
2430/** \brief Returns true of the #VmaVirtualBlock is empty - contains 0 virtual allocations and has all its space available for new allocations.
2431*/
2432VMA_CALL_PRE VkBool32 VMA_CALL_POST vmaIsVirtualBlockEmpty(
2433 VmaVirtualBlock VMA_NOT_NULL virtualBlock);
2434
2435/** \brief Returns information about a specific virtual allocation within a virtual block, like its size and `pUserData` pointer.
2436*/
2437VMA_CALL_PRE void VMA_CALL_POST vmaGetVirtualAllocationInfo(
2438 VmaVirtualBlock VMA_NOT_NULL virtualBlock,
2439 VmaVirtualAllocation VMA_NOT_NULL_NON_DISPATCHABLE allocation, VmaVirtualAllocationInfo* VMA_NOT_NULL pVirtualAllocInfo);
2440
2441/** \brief Allocates new virtual allocation inside given #VmaVirtualBlock.
2442
2443If the allocation fails due to not enough free space available, `VK_ERROR_OUT_OF_DEVICE_MEMORY` is returned
2444(despite the function doesn't ever allocate actual GPU memory).
2445`pAllocation` is then set to `VK_NULL_HANDLE` and `pOffset`, if not null, it set to `UINT64_MAX`.
2446
2447\param virtualBlock Virtual block
2448\param pCreateInfo Parameters for the allocation
2449\param[out] pAllocation Returned handle of the new allocation
2450\param[out] pOffset Returned offset of the new allocation. Optional, can be null.
2451*/
2452VMA_CALL_PRE VkResult VMA_CALL_POST vmaVirtualAllocate(
2453 VmaVirtualBlock VMA_NOT_NULL virtualBlock,
2454 const VmaVirtualAllocationCreateInfo* VMA_NOT_NULL pCreateInfo,
2455 VmaVirtualAllocation VMA_NULLABLE_NON_DISPATCHABLE* VMA_NOT_NULL pAllocation,
2456 VkDeviceSize* VMA_NULLABLE pOffset);
2457
2458/** \brief Frees virtual allocation inside given #VmaVirtualBlock.
2459
2460It is correct to call this function with `allocation == VK_NULL_HANDLE` - it does nothing.
2461*/
2462VMA_CALL_PRE void VMA_CALL_POST vmaVirtualFree(
2463 VmaVirtualBlock VMA_NOT_NULL virtualBlock,
2464 VmaVirtualAllocation VMA_NULLABLE_NON_DISPATCHABLE allocation);
2465
2466/** \brief Frees all virtual allocations inside given #VmaVirtualBlock.
2467
2468You must either call this function or free each virtual allocation individually with vmaVirtualFree()
2469before destroying a virtual block. Otherwise, an assert is called.
2470
2471If you keep pointer to some additional metadata associated with your virtual allocation in its `pUserData`,
2472don't forget to free it as well.
2473*/
2474VMA_CALL_PRE void VMA_CALL_POST vmaClearVirtualBlock(
2475 VmaVirtualBlock VMA_NOT_NULL virtualBlock);
2476
2477/** \brief Changes custom pointer associated with given virtual allocation.
2478*/
2479VMA_CALL_PRE void VMA_CALL_POST vmaSetVirtualAllocationUserData(
2480 VmaVirtualBlock VMA_NOT_NULL virtualBlock,
2481 VmaVirtualAllocation VMA_NOT_NULL_NON_DISPATCHABLE allocation,
2482 void* VMA_NULLABLE pUserData);
2483
2484/** \brief Calculates and returns statistics about virtual allocations and memory usage in given #VmaVirtualBlock.
2485
2486This function is fast to call. For more detailed statistics, see vmaCalculateVirtualBlockStatistics().
2487*/
2488VMA_CALL_PRE void VMA_CALL_POST vmaGetVirtualBlockStatistics(
2489 VmaVirtualBlock VMA_NOT_NULL virtualBlock,
2490 VmaStatistics* VMA_NOT_NULL pStats);
2491
2492/** \brief Calculates and returns detailed statistics about virtual allocations and memory usage in given #VmaVirtualBlock.
2493
2494This function is slow to call. Use for debugging purposes.
2495For less detailed statistics, see vmaGetVirtualBlockStatistics().
2496*/
2497VMA_CALL_PRE void VMA_CALL_POST vmaCalculateVirtualBlockStatistics(
2498 VmaVirtualBlock VMA_NOT_NULL virtualBlock,
2499 VmaDetailedStatistics* VMA_NOT_NULL pStats);
2500
2501/** @} */
2502
2503#if VMA_STATS_STRING_ENABLED
2504/**
2505\addtogroup group_stats
2506@{
2507*/
2508
2509/** \brief Builds and returns a null-terminated string in JSON format with information about given #VmaVirtualBlock.
2510\param virtualBlock Virtual block.
2511\param[out] ppStatsString Returned string.
2512\param detailedMap Pass `VK_FALSE` to only obtain statistics as returned by vmaCalculateVirtualBlockStatistics(). Pass `VK_TRUE` to also obtain full list of allocations and free spaces.
2513
2514Returned string must be freed using vmaFreeVirtualBlockStatsString().
2515*/
2516VMA_CALL_PRE void VMA_CALL_POST vmaBuildVirtualBlockStatsString(
2517 VmaVirtualBlock VMA_NOT_NULL virtualBlock,
2518 char* VMA_NULLABLE* VMA_NOT_NULL ppStatsString,
2519 VkBool32 detailedMap);
2520
2521/// Frees a string returned by vmaBuildVirtualBlockStatsString().
2522VMA_CALL_PRE void VMA_CALL_POST vmaFreeVirtualBlockStatsString(
2523 VmaVirtualBlock VMA_NOT_NULL virtualBlock,
2524 char* VMA_NULLABLE pStatsString);
2525
2526/** \brief Builds and returns statistics as a null-terminated string in JSON format.
2527\param allocator
2528\param[out] ppStatsString Must be freed using vmaFreeStatsString() function.
2529\param detailedMap
2530*/
2531VMA_CALL_PRE void VMA_CALL_POST vmaBuildStatsString(
2532 VmaAllocator VMA_NOT_NULL allocator,
2533 char* VMA_NULLABLE* VMA_NOT_NULL ppStatsString,
2534 VkBool32 detailedMap);
2535
2536VMA_CALL_PRE void VMA_CALL_POST vmaFreeStatsString(
2537 VmaAllocator VMA_NOT_NULL allocator,
2538 char* VMA_NULLABLE pStatsString);
2539
2540/** @} */
2541
2542#endif // VMA_STATS_STRING_ENABLED
2543
2544#endif // _VMA_FUNCTION_HEADERS
2545
2546#ifdef __cplusplus
2547}
2548#endif
2549
2550#endif // AMD_VULKAN_MEMORY_ALLOCATOR_H
2551
2552////////////////////////////////////////////////////////////////////////////////
2553////////////////////////////////////////////////////////////////////////////////
2554//
2555// IMPLEMENTATION
2556//
2557////////////////////////////////////////////////////////////////////////////////
2558////////////////////////////////////////////////////////////////////////////////
2559
2560// For Visual Studio IntelliSense.
2561#if defined(__cplusplus) && defined(__INTELLISENSE__)
2562#define VMA_IMPLEMENTATION
2563#endif
2564
2565#ifdef VMA_IMPLEMENTATION
2566#undef VMA_IMPLEMENTATION
2567
2568#if defined(__GNUC__) && !defined(__clang__)
2569#pragma GCC diagnostic push
2570#pragma GCC diagnostic ignored "-Wunused-variable"
2571#pragma GCC diagnostic ignored "-Wunused-parameter"
2572#pragma GCC diagnostic ignored "-Wmissing-field-initializers"
2573#pragma GCC diagnostic ignored "-Wparentheses"
2574#pragma GCC diagnostic ignored "-Wimplicit-fallthrough"
2575#elif defined(__clang__)
2576#pragma clang diagnostic push
2577#pragma clang diagnostic ignored "-Wunused-variable"
2578#pragma clang diagnostic ignored "-Wunused-parameter"
2579#pragma clang diagnostic ignored "-Wmissing-field-initializers"
2580#pragma clang diagnostic ignored "-Wparentheses"
2581#pragma clang diagnostic ignored "-Wimplicit-fallthrough"
2582#pragma clang diagnostic ignored "-Wnullability-completeness"
2583#endif
2584
2585#include <cstdint>
2586#include <cstdlib>
2587#include <cstring>
2588#include <utility>
2589#include <type_traits>
2590
2591#ifdef _MSC_VER
2592 #include <intrin.h> // For functions like __popcnt, _BitScanForward etc.
2593#endif
2594#if __cplusplus >= 202002L || _MSVC_LANG >= 202002L // C++20
2595 #include <bit> // For std::popcount
2596#endif
2597
2598/*******************************************************************************
2599CONFIGURATION SECTION
2600
2601Define some of these macros before each #include of this header or change them
2602here if you need other then default behavior depending on your environment.
2603*/
2604#ifndef _VMA_CONFIGURATION
2605
2606/*
2607Define this macro to 1 to make the library fetch pointers to Vulkan functions
2608internally, like:
2609
2610 vulkanFunctions.vkAllocateMemory = &vkAllocateMemory;
2611*/
2612#if !defined(VMA_STATIC_VULKAN_FUNCTIONS) && !defined(VK_NO_PROTOTYPES)
2613 #define VMA_STATIC_VULKAN_FUNCTIONS 1
2614#endif
2615
2616/*
2617Define this macro to 1 to make the library fetch pointers to Vulkan functions
2618internally, like:
2619
2620 vulkanFunctions.vkAllocateMemory = (PFN_vkAllocateMemory)vkGetDeviceProcAddr(device, "vkAllocateMemory");
2621
2622To use this feature in new versions of VMA you now have to pass
2623VmaVulkanFunctions::vkGetInstanceProcAddr and vkGetDeviceProcAddr as
2624VmaAllocatorCreateInfo::pVulkanFunctions. Other members can be null.
2625*/
2626#if !defined(VMA_DYNAMIC_VULKAN_FUNCTIONS)
2627 #define VMA_DYNAMIC_VULKAN_FUNCTIONS 1
2628#endif
2629
2630#ifndef VMA_USE_STL_SHARED_MUTEX
2631 // Compiler conforms to C++17.
2632 #if __cplusplus >= 201703L
2633 #define VMA_USE_STL_SHARED_MUTEX 1
2634 // Visual studio defines __cplusplus properly only when passed additional parameter: /Zc:__cplusplus
2635 // Otherwise it is always 199711L, despite shared_mutex works since Visual Studio 2015 Update 2.
2636 #elif defined(_MSC_FULL_VER) && _MSC_FULL_VER >= 190023918 && __cplusplus == 199711L && _MSVC_LANG >= 201703L
2637 #define VMA_USE_STL_SHARED_MUTEX 1
2638 #else
2639 #define VMA_USE_STL_SHARED_MUTEX 0
2640 #endif
2641#endif
2642
2643/*
2644Define this macro to include custom header files without having to edit this file directly, e.g.:
2645
2646 // Inside of "my_vma_configuration_user_includes.h":
2647
2648 #include "my_custom_assert.h" // for MY_CUSTOM_ASSERT
2649 #include "my_custom_min.h" // for my_custom_min
2650 #include <algorithm>
2651 #include <mutex>
2652
2653 // Inside a different file, which includes "vk_mem_alloc.h":
2654
2655 #define VMA_CONFIGURATION_USER_INCLUDES_H "my_vma_configuration_user_includes.h"
2656 #define VMA_ASSERT(expr) MY_CUSTOM_ASSERT(expr)
2657 #define VMA_MIN(v1, v2) (my_custom_min(v1, v2))
2658 #include "vk_mem_alloc.h"
2659 ...
2660
2661The following headers are used in this CONFIGURATION section only, so feel free to
2662remove them if not needed.
2663*/
2664#if !defined(VMA_CONFIGURATION_USER_INCLUDES_H)
2665 #include <cassert> // for assert
2666 #include <algorithm> // for min, max
2667 #include <mutex>
2668#else
2669 #include VMA_CONFIGURATION_USER_INCLUDES_H
2670#endif
2671
2672#ifndef VMA_NULL
2673 // Value used as null pointer. Define it to e.g.: nullptr, NULL, 0, (void*)0.
2674 #define VMA_NULL nullptr
2675#endif
2676
2677#if defined(__ANDROID_API__) && (__ANDROID_API__ < 16)
2678#include <cstdlib>
2679static void* vma_aligned_alloc(size_t alignment, size_t size)
2680{
2681 // alignment must be >= sizeof(void*)
2682 if(alignment < sizeof(void*))
2683 {
2684 alignment = sizeof(void*);
2685 }
2686
2687 return memalign(alignment, size);
2688}
2689#elif defined(__APPLE__) || defined(__ANDROID__) || (defined(__linux__) && defined(__GLIBCXX__) && !defined(_GLIBCXX_HAVE_ALIGNED_ALLOC))
2690#include <cstdlib>
2691
2692#if defined(__APPLE__)
2693#include <AvailabilityMacros.h>
2694#endif
2695
2696static void* vma_aligned_alloc(size_t alignment, size_t size)
2697{
2698 // Unfortunately, aligned_alloc causes VMA to crash due to it returning null pointers. (At least under 11.4)
2699 // Therefore, for now disable this specific exception until a proper solution is found.
2700 //#if defined(__APPLE__) && (defined(MAC_OS_X_VERSION_10_16) || defined(__IPHONE_14_0))
2701 //#if MAC_OS_X_VERSION_MAX_ALLOWED >= MAC_OS_X_VERSION_10_16 || __IPHONE_OS_VERSION_MAX_ALLOWED >= __IPHONE_14_0
2702 // // For C++14, usr/include/malloc/_malloc.h declares aligned_alloc()) only
2703 // // with the MacOSX11.0 SDK in Xcode 12 (which is what adds
2704 // // MAC_OS_X_VERSION_10_16), even though the function is marked
2705 // // availabe for 10.15. That is why the preprocessor checks for 10.16 but
2706 // // the __builtin_available checks for 10.15.
2707 // // People who use C++17 could call aligned_alloc with the 10.15 SDK already.
2708 // if (__builtin_available(macOS 10.15, iOS 13, *))
2709 // return aligned_alloc(alignment, size);
2710 //#endif
2711 //#endif
2712
2713 // alignment must be >= sizeof(void*)
2714 if(alignment < sizeof(void*))
2715 {
2716 alignment = sizeof(void*);
2717 }
2718
2719 void *pointer;
2720 if(posix_memalign(&pointer, alignment, size) == 0)
2721 return pointer;
2722 return VMA_NULL;
2723}
2724#elif defined(_WIN32)
2725static void* vma_aligned_alloc(size_t alignment, size_t size)
2726{
2727 return _aligned_malloc(size, alignment);
2728}
2729#else
2730static void* vma_aligned_alloc(size_t alignment, size_t size)
2731{
2732 return aligned_alloc(alignment: alignment, size: size);
2733}
2734#endif
2735
2736#if defined(_WIN32)
2737static void vma_aligned_free(void* ptr)
2738{
2739 _aligned_free(ptr);
2740}
2741#else
2742static void vma_aligned_free(void* VMA_NULLABLE ptr)
2743{
2744 free(ptr: ptr);
2745}
2746#endif
2747
2748// If your compiler is not compatible with C++11 and definition of
2749// aligned_alloc() function is missing, uncommeting following line may help:
2750
2751//#include <malloc.h>
2752
2753// Normal assert to check for programmer's errors, especially in Debug configuration.
2754#ifndef VMA_ASSERT
2755 #ifdef NDEBUG
2756 #define VMA_ASSERT(expr)
2757 #else
2758 #define VMA_ASSERT(expr) assert(expr)
2759 #endif
2760#endif
2761
2762// Assert that will be called very often, like inside data structures e.g. operator[].
2763// Making it non-empty can make program slow.
2764#ifndef VMA_HEAVY_ASSERT
2765 #ifdef NDEBUG
2766 #define VMA_HEAVY_ASSERT(expr)
2767 #else
2768 #define VMA_HEAVY_ASSERT(expr) //VMA_ASSERT(expr)
2769 #endif
2770#endif
2771
2772#ifndef VMA_ALIGN_OF
2773 #define VMA_ALIGN_OF(type) (__alignof(type))
2774#endif
2775
2776#ifndef VMA_SYSTEM_ALIGNED_MALLOC
2777 #define VMA_SYSTEM_ALIGNED_MALLOC(size, alignment) vma_aligned_alloc((alignment), (size))
2778#endif
2779
2780#ifndef VMA_SYSTEM_ALIGNED_FREE
2781 // VMA_SYSTEM_FREE is the old name, but might have been defined by the user
2782 #if defined(VMA_SYSTEM_FREE)
2783 #define VMA_SYSTEM_ALIGNED_FREE(ptr) VMA_SYSTEM_FREE(ptr)
2784 #else
2785 #define VMA_SYSTEM_ALIGNED_FREE(ptr) vma_aligned_free(ptr)
2786 #endif
2787#endif
2788
2789#ifndef VMA_COUNT_BITS_SET
2790 // Returns number of bits set to 1 in (v)
2791 #define VMA_COUNT_BITS_SET(v) VmaCountBitsSet(v)
2792#endif
2793
2794#ifndef VMA_BITSCAN_LSB
2795 // Scans integer for index of first nonzero value from the Least Significant Bit (LSB). If mask is 0 then returns UINT8_MAX
2796 #define VMA_BITSCAN_LSB(mask) VmaBitScanLSB(mask)
2797#endif
2798
2799#ifndef VMA_BITSCAN_MSB
2800 // Scans integer for index of first nonzero value from the Most Significant Bit (MSB). If mask is 0 then returns UINT8_MAX
2801 #define VMA_BITSCAN_MSB(mask) VmaBitScanMSB(mask)
2802#endif
2803
2804#ifndef VMA_MIN
2805 #define VMA_MIN(v1, v2) ((std::min)((v1), (v2)))
2806#endif
2807
2808#ifndef VMA_MAX
2809 #define VMA_MAX(v1, v2) ((std::max)((v1), (v2)))
2810#endif
2811
2812#ifndef VMA_SWAP
2813 #define VMA_SWAP(v1, v2) std::swap((v1), (v2))
2814#endif
2815
2816#ifndef VMA_SORT
2817 #define VMA_SORT(beg, end, cmp) std::sort(beg, end, cmp)
2818#endif
2819
2820#ifndef VMA_DEBUG_LOG
2821 #define VMA_DEBUG_LOG(format, ...)
2822 /*
2823 #define VMA_DEBUG_LOG(format, ...) do { \
2824 printf(format, __VA_ARGS__); \
2825 printf("\n"); \
2826 } while(false)
2827 */
2828#endif
2829
2830// Define this macro to 1 to enable functions: vmaBuildStatsString, vmaFreeStatsString.
2831#if VMA_STATS_STRING_ENABLED
2832 static inline void VmaUint32ToStr(char* VMA_NOT_NULL outStr, size_t strLen, uint32_t num)
2833 {
2834 snprintf(s: outStr, maxlen: strLen, format: "%u", static_cast<unsigned int>(num));
2835 }
2836 static inline void VmaUint64ToStr(char* VMA_NOT_NULL outStr, size_t strLen, uint64_t num)
2837 {
2838 snprintf(s: outStr, maxlen: strLen, format: "%llu", static_cast<unsigned long long>(num));
2839 }
2840 static inline void VmaPtrToStr(char* VMA_NOT_NULL outStr, size_t strLen, const void* ptr)
2841 {
2842 snprintf(s: outStr, maxlen: strLen, format: "%p", ptr);
2843 }
2844#endif
2845
2846#ifndef VMA_MUTEX
2847 class VmaMutex
2848 {
2849 public:
2850 void Lock() { m_Mutex.lock(); }
2851 void Unlock() { m_Mutex.unlock(); }
2852 bool TryLock() { return m_Mutex.try_lock(); }
2853 private:
2854 std::mutex m_Mutex;
2855 };
2856 #define VMA_MUTEX VmaMutex
2857#endif
2858
2859// Read-write mutex, where "read" is shared access, "write" is exclusive access.
2860#ifndef VMA_RW_MUTEX
2861 #if VMA_USE_STL_SHARED_MUTEX
2862 // Use std::shared_mutex from C++17.
2863 #include <shared_mutex>
2864 class VmaRWMutex
2865 {
2866 public:
2867 void LockRead() { m_Mutex.lock_shared(); }
2868 void UnlockRead() { m_Mutex.unlock_shared(); }
2869 bool TryLockRead() { return m_Mutex.try_lock_shared(); }
2870 void LockWrite() { m_Mutex.lock(); }
2871 void UnlockWrite() { m_Mutex.unlock(); }
2872 bool TryLockWrite() { return m_Mutex.try_lock(); }
2873 private:
2874 std::shared_mutex m_Mutex;
2875 };
2876 #define VMA_RW_MUTEX VmaRWMutex
2877 #elif defined(_WIN32) && defined(WINVER) && WINVER >= 0x0600 && !defined(__MINGW32__)
2878 // Use SRWLOCK from WinAPI.
2879 // Minimum supported client = Windows Vista, server = Windows Server 2008.
2880 class VmaRWMutex
2881 {
2882 public:
2883 VmaRWMutex() { InitializeSRWLock(&m_Lock); }
2884 void LockRead() { AcquireSRWLockShared(&m_Lock); }
2885 void UnlockRead() { ReleaseSRWLockShared(&m_Lock); }
2886 bool TryLockRead() { return TryAcquireSRWLockShared(&m_Lock) != FALSE; }
2887 void LockWrite() { AcquireSRWLockExclusive(&m_Lock); }
2888 void UnlockWrite() { ReleaseSRWLockExclusive(&m_Lock); }
2889 bool TryLockWrite() { return TryAcquireSRWLockExclusive(&m_Lock) != FALSE; }
2890 private:
2891 SRWLOCK m_Lock;
2892 };
2893 #define VMA_RW_MUTEX VmaRWMutex
2894 #else
2895 // Less efficient fallback: Use normal mutex.
2896 class VmaRWMutex
2897 {
2898 public:
2899 void LockRead() { m_Mutex.Lock(); }
2900 void UnlockRead() { m_Mutex.Unlock(); }
2901 bool TryLockRead() { return m_Mutex.TryLock(); }
2902 void LockWrite() { m_Mutex.Lock(); }
2903 void UnlockWrite() { m_Mutex.Unlock(); }
2904 bool TryLockWrite() { return m_Mutex.TryLock(); }
2905 private:
2906 VMA_MUTEX m_Mutex;
2907 };
2908 #define VMA_RW_MUTEX VmaRWMutex
2909 #endif // #if VMA_USE_STL_SHARED_MUTEX
2910#endif // #ifndef VMA_RW_MUTEX
2911
2912/*
2913If providing your own implementation, you need to implement a subset of std::atomic.
2914*/
2915#ifndef VMA_ATOMIC_UINT32
2916 #include <atomic>
2917 #define VMA_ATOMIC_UINT32 std::atomic<uint32_t>
2918#endif
2919
2920#ifndef VMA_ATOMIC_UINT64
2921 #include <atomic>
2922 #define VMA_ATOMIC_UINT64 std::atomic<uint64_t>
2923#endif
2924
2925#ifndef VMA_DEBUG_ALWAYS_DEDICATED_MEMORY
2926 /**
2927 Every allocation will have its own memory block.
2928 Define to 1 for debugging purposes only.
2929 */
2930 #define VMA_DEBUG_ALWAYS_DEDICATED_MEMORY (0)
2931#endif
2932
2933#ifndef VMA_MIN_ALIGNMENT
2934 /**
2935 Minimum alignment of all allocations, in bytes.
2936 Set to more than 1 for debugging purposes. Must be power of two.
2937 */
2938 #ifdef VMA_DEBUG_ALIGNMENT // Old name
2939 #define VMA_MIN_ALIGNMENT VMA_DEBUG_ALIGNMENT
2940 #else
2941 #define VMA_MIN_ALIGNMENT (1)
2942 #endif
2943#endif
2944
2945#ifndef VMA_DEBUG_MARGIN
2946 /**
2947 Minimum margin after every allocation, in bytes.
2948 Set nonzero for debugging purposes only.
2949 */
2950 #define VMA_DEBUG_MARGIN (0)
2951#endif
2952
2953#ifndef VMA_DEBUG_INITIALIZE_ALLOCATIONS
2954 /**
2955 Define this macro to 1 to automatically fill new allocations and destroyed
2956 allocations with some bit pattern.
2957 */
2958 #define VMA_DEBUG_INITIALIZE_ALLOCATIONS (0)
2959#endif
2960
2961#ifndef VMA_DEBUG_DETECT_CORRUPTION
2962 /**
2963 Define this macro to 1 together with non-zero value of VMA_DEBUG_MARGIN to
2964 enable writing magic value to the margin after every allocation and
2965 validating it, so that memory corruptions (out-of-bounds writes) are detected.
2966 */
2967 #define VMA_DEBUG_DETECT_CORRUPTION (0)
2968#endif
2969
2970#ifndef VMA_DEBUG_GLOBAL_MUTEX
2971 /**
2972 Set this to 1 for debugging purposes only, to enable single mutex protecting all
2973 entry calls to the library. Can be useful for debugging multithreading issues.
2974 */
2975 #define VMA_DEBUG_GLOBAL_MUTEX (0)
2976#endif
2977
2978#ifndef VMA_DEBUG_MIN_BUFFER_IMAGE_GRANULARITY
2979 /**
2980 Minimum value for VkPhysicalDeviceLimits::bufferImageGranularity.
2981 Set to more than 1 for debugging purposes only. Must be power of two.
2982 */
2983 #define VMA_DEBUG_MIN_BUFFER_IMAGE_GRANULARITY (1)
2984#endif
2985
2986#ifndef VMA_DEBUG_DONT_EXCEED_MAX_MEMORY_ALLOCATION_COUNT
2987 /*
2988 Set this to 1 to make VMA never exceed VkPhysicalDeviceLimits::maxMemoryAllocationCount
2989 and return error instead of leaving up to Vulkan implementation what to do in such cases.
2990 */
2991 #define VMA_DEBUG_DONT_EXCEED_MAX_MEMORY_ALLOCATION_COUNT (0)
2992#endif
2993
2994#ifndef VMA_SMALL_HEAP_MAX_SIZE
2995 /// Maximum size of a memory heap in Vulkan to consider it "small".
2996 #define VMA_SMALL_HEAP_MAX_SIZE (1024ull * 1024 * 1024)
2997#endif
2998
2999#ifndef VMA_DEFAULT_LARGE_HEAP_BLOCK_SIZE
3000 /// Default size of a block allocated as single VkDeviceMemory from a "large" heap.
3001 #define VMA_DEFAULT_LARGE_HEAP_BLOCK_SIZE (256ull * 1024 * 1024)
3002#endif
3003
3004/*
3005Mapping hysteresis is a logic that launches when vmaMapMemory/vmaUnmapMemory is called
3006or a persistently mapped allocation is created and destroyed several times in a row.
3007It keeps additional +1 mapping of a device memory block to prevent calling actual
3008vkMapMemory/vkUnmapMemory too many times, which may improve performance and help
3009tools like RenderDOc.
3010*/
3011#ifndef VMA_MAPPING_HYSTERESIS_ENABLED
3012 #define VMA_MAPPING_HYSTERESIS_ENABLED 1
3013#endif
3014
3015#ifndef VMA_CLASS_NO_COPY
3016 #define VMA_CLASS_NO_COPY(className) \
3017 private: \
3018 className(const className&) = delete; \
3019 className& operator=(const className&) = delete;
3020#endif
3021
3022#define VMA_VALIDATE(cond) do { if(!(cond)) { \
3023 VMA_ASSERT(0 && "Validation failed: " #cond); \
3024 return false; \
3025 } } while(false)
3026
3027/*******************************************************************************
3028END OF CONFIGURATION
3029*/
3030#endif // _VMA_CONFIGURATION
3031
3032
3033static const uint8_t VMA_ALLOCATION_FILL_PATTERN_CREATED = 0xDC;
3034static const uint8_t VMA_ALLOCATION_FILL_PATTERN_DESTROYED = 0xEF;
3035// Decimal 2139416166, float NaN, little-endian binary 66 E6 84 7F.
3036static const uint32_t VMA_CORRUPTION_DETECTION_MAGIC_VALUE = 0x7F84E666;
3037
3038// Copy of some Vulkan definitions so we don't need to check their existence just to handle few constants.
3039static const uint32_t VK_MEMORY_PROPERTY_DEVICE_COHERENT_BIT_AMD_COPY = 0x00000040;
3040static const uint32_t VK_MEMORY_PROPERTY_DEVICE_UNCACHED_BIT_AMD_COPY = 0x00000080;
3041static const uint32_t VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT_COPY = 0x00020000;
3042static const uint32_t VK_IMAGE_CREATE_DISJOINT_BIT_COPY = 0x00000200;
3043static const int32_t VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT_COPY = 1000158000;
3044static const uint32_t VMA_ALLOCATION_INTERNAL_STRATEGY_MIN_OFFSET = 0x10000000u;
3045static const uint32_t VMA_ALLOCATION_TRY_COUNT = 32;
3046static const uint32_t VMA_VENDOR_ID_AMD = 4098;
3047
3048// This one is tricky. Vulkan specification defines this code as available since
3049// Vulkan 1.0, but doesn't actually define it in Vulkan SDK earlier than 1.2.131.
3050// See pull request #207.
3051#define VK_ERROR_UNKNOWN_COPY ((VkResult)-13)
3052
3053
3054#if VMA_STATS_STRING_ENABLED
3055// Correspond to values of enum VmaSuballocationType.
3056static const char* VMA_SUBALLOCATION_TYPE_NAMES[] =
3057{
3058 "FREE",
3059 "UNKNOWN",
3060 "BUFFER",
3061 "IMAGE_UNKNOWN",
3062 "IMAGE_LINEAR",
3063 "IMAGE_OPTIMAL",
3064};
3065#endif
3066
3067static VkAllocationCallbacks VmaEmptyAllocationCallbacks =
3068 { VMA_NULL, VMA_NULL, VMA_NULL, VMA_NULL, VMA_NULL, VMA_NULL };
3069
3070
3071#ifndef _VMA_ENUM_DECLARATIONS
3072
3073enum VmaSuballocationType
3074{
3075 VMA_SUBALLOCATION_TYPE_FREE = 0,
3076 VMA_SUBALLOCATION_TYPE_UNKNOWN = 1,
3077 VMA_SUBALLOCATION_TYPE_BUFFER = 2,
3078 VMA_SUBALLOCATION_TYPE_IMAGE_UNKNOWN = 3,
3079 VMA_SUBALLOCATION_TYPE_IMAGE_LINEAR = 4,
3080 VMA_SUBALLOCATION_TYPE_IMAGE_OPTIMAL = 5,
3081 VMA_SUBALLOCATION_TYPE_MAX_ENUM = 0x7FFFFFFF
3082};
3083
3084enum VMA_CACHE_OPERATION
3085{
3086 VMA_CACHE_FLUSH,
3087 VMA_CACHE_INVALIDATE
3088};
3089
3090enum class VmaAllocationRequestType
3091{
3092 Normal,
3093 TLSF,
3094 // Used by "Linear" algorithm.
3095 UpperAddress,
3096 EndOf1st,
3097 EndOf2nd,
3098};
3099
3100#endif // _VMA_ENUM_DECLARATIONS
3101
3102#ifndef _VMA_FORWARD_DECLARATIONS
3103// Opaque handle used by allocation algorithms to identify single allocation in any conforming way.
3104VK_DEFINE_NON_DISPATCHABLE_HANDLE(VmaAllocHandle);
3105
3106struct VmaMutexLock;
3107struct VmaMutexLockRead;
3108struct VmaMutexLockWrite;
3109
3110template<typename T>
3111struct AtomicTransactionalIncrement;
3112
3113template<typename T>
3114struct VmaStlAllocator;
3115
3116template<typename T, typename AllocatorT>
3117class VmaVector;
3118
3119template<typename T, typename AllocatorT, size_t N>
3120class VmaSmallVector;
3121
3122template<typename T>
3123class VmaPoolAllocator;
3124
3125template<typename T>
3126struct VmaListItem;
3127
3128template<typename T>
3129class VmaRawList;
3130
3131template<typename T, typename AllocatorT>
3132class VmaList;
3133
3134template<typename ItemTypeTraits>
3135class VmaIntrusiveLinkedList;
3136
3137// Unused in this version
3138#if 0
3139template<typename T1, typename T2>
3140struct VmaPair;
3141template<typename FirstT, typename SecondT>
3142struct VmaPairFirstLess;
3143
3144template<typename KeyT, typename ValueT>
3145class VmaMap;
3146#endif
3147
3148#if VMA_STATS_STRING_ENABLED
3149class VmaStringBuilder;
3150class VmaJsonWriter;
3151#endif
3152
3153class VmaDeviceMemoryBlock;
3154
3155struct VmaDedicatedAllocationListItemTraits;
3156class VmaDedicatedAllocationList;
3157
3158struct VmaSuballocation;
3159struct VmaSuballocationOffsetLess;
3160struct VmaSuballocationOffsetGreater;
3161struct VmaSuballocationItemSizeLess;
3162
3163typedef VmaList<VmaSuballocation, VmaStlAllocator<VmaSuballocation>> VmaSuballocationList;
3164
3165struct VmaAllocationRequest;
3166
3167class VmaBlockMetadata;
3168class VmaBlockMetadata_Linear;
3169class VmaBlockMetadata_TLSF;
3170
3171class VmaBlockVector;
3172
3173struct VmaPoolListItemTraits;
3174
3175struct VmaCurrentBudgetData;
3176
3177class VmaAllocationObjectAllocator;
3178
3179#endif // _VMA_FORWARD_DECLARATIONS
3180
3181
3182#ifndef _VMA_FUNCTIONS
3183
3184/*
3185Returns number of bits set to 1 in (v).
3186
3187On specific platforms and compilers you can use instrinsics like:
3188
3189Visual Studio:
3190 return __popcnt(v);
3191GCC, Clang:
3192 return static_cast<uint32_t>(__builtin_popcount(v));
3193
3194Define macro VMA_COUNT_BITS_SET to provide your optimized implementation.
3195But you need to check in runtime whether user's CPU supports these, as some old processors don't.
3196*/
3197static inline uint32_t VmaCountBitsSet(uint32_t v)
3198{
3199#if __cplusplus >= 202002L || _MSVC_LANG >= 202002L // C++20
3200 return std::popcount(v);
3201#else
3202 uint32_t c = v - ((v >> 1) & 0x55555555);
3203 c = ((c >> 2) & 0x33333333) + (c & 0x33333333);
3204 c = ((c >> 4) + c) & 0x0F0F0F0F;
3205 c = ((c >> 8) + c) & 0x00FF00FF;
3206 c = ((c >> 16) + c) & 0x0000FFFF;
3207 return c;
3208#endif
3209}
3210
3211static inline uint8_t VmaBitScanLSB(uint64_t mask)
3212{
3213#if defined(_MSC_VER) && defined(_WIN64)
3214 unsigned long pos;
3215 if (_BitScanForward64(&pos, mask))
3216 return static_cast<uint8_t>(pos);
3217 return UINT8_MAX;
3218#elif defined __GNUC__ || defined __clang__
3219 return static_cast<uint8_t>(__builtin_ffsll(mask)) - 1U;
3220#else
3221 uint8_t pos = 0;
3222 uint64_t bit = 1;
3223 do
3224 {
3225 if (mask & bit)
3226 return pos;
3227 bit <<= 1;
3228 } while (pos++ < 63);
3229 return UINT8_MAX;
3230#endif
3231}
3232
3233static inline uint8_t VmaBitScanLSB(uint32_t mask)
3234{
3235#ifdef _MSC_VER
3236 unsigned long pos;
3237 if (_BitScanForward(&pos, mask))
3238 return static_cast<uint8_t>(pos);
3239 return UINT8_MAX;
3240#elif defined __GNUC__ || defined __clang__
3241 return static_cast<uint8_t>(__builtin_ffs(mask)) - 1U;
3242#else
3243 uint8_t pos = 0;
3244 uint32_t bit = 1;
3245 do
3246 {
3247 if (mask & bit)
3248 return pos;
3249 bit <<= 1;
3250 } while (pos++ < 31);
3251 return UINT8_MAX;
3252#endif
3253}
3254
3255static inline uint8_t VmaBitScanMSB(uint64_t mask)
3256{
3257#if defined(_MSC_VER) && defined(_WIN64)
3258 unsigned long pos;
3259 if (_BitScanReverse64(&pos, mask))
3260 return static_cast<uint8_t>(pos);
3261#elif defined __GNUC__ || defined __clang__
3262 if (mask)
3263 return 63 - static_cast<uint8_t>(__builtin_clzll(mask));
3264#else
3265 uint8_t pos = 63;
3266 uint64_t bit = 1ULL << 63;
3267 do
3268 {
3269 if (mask & bit)
3270 return pos;
3271 bit >>= 1;
3272 } while (pos-- > 0);
3273#endif
3274 return UINT8_MAX;
3275}
3276
3277static inline uint8_t VmaBitScanMSB(uint32_t mask)
3278{
3279#ifdef _MSC_VER
3280 unsigned long pos;
3281 if (_BitScanReverse(&pos, mask))
3282 return static_cast<uint8_t>(pos);
3283#elif defined __GNUC__ || defined __clang__
3284 if (mask)
3285 return 31 - static_cast<uint8_t>(__builtin_clz(mask));
3286#else
3287 uint8_t pos = 31;
3288 uint32_t bit = 1UL << 31;
3289 do
3290 {
3291 if (mask & bit)
3292 return pos;
3293 bit >>= 1;
3294 } while (pos-- > 0);
3295#endif
3296 return UINT8_MAX;
3297}
3298
3299/*
3300Returns true if given number is a power of two.
3301T must be unsigned integer number or signed integer but always nonnegative.
3302For 0 returns true.
3303*/
3304template <typename T>
3305inline bool VmaIsPow2(T x)
3306{
3307 return (x & (x - 1)) == 0;
3308}
3309
3310// Aligns given value up to nearest multiply of align value. For example: VmaAlignUp(11, 8) = 16.
3311// Use types like uint32_t, uint64_t as T.
3312template <typename T>
3313static inline T VmaAlignUp(T val, T alignment)
3314{
3315 VMA_HEAVY_ASSERT(VmaIsPow2(alignment));
3316 return (val + alignment - 1) & ~(alignment - 1);
3317}
3318
3319// Aligns given value down to nearest multiply of align value. For example: VmaAlignUp(11, 8) = 8.
3320// Use types like uint32_t, uint64_t as T.
3321template <typename T>
3322static inline T VmaAlignDown(T val, T alignment)
3323{
3324 VMA_HEAVY_ASSERT(VmaIsPow2(alignment));
3325 return val & ~(alignment - 1);
3326}
3327
3328// Division with mathematical rounding to nearest number.
3329template <typename T>
3330static inline T VmaRoundDiv(T x, T y)
3331{
3332 return (x + (y / (T)2)) / y;
3333}
3334
3335// Divide by 'y' and round up to nearest integer.
3336template <typename T>
3337static inline T VmaDivideRoundingUp(T x, T y)
3338{
3339 return (x + y - (T)1) / y;
3340}
3341
3342// Returns smallest power of 2 greater or equal to v.
3343static inline uint32_t VmaNextPow2(uint32_t v)
3344{
3345 v--;
3346 v |= v >> 1;
3347 v |= v >> 2;
3348 v |= v >> 4;
3349 v |= v >> 8;
3350 v |= v >> 16;
3351 v++;
3352 return v;
3353}
3354
3355static inline uint64_t VmaNextPow2(uint64_t v)
3356{
3357 v--;
3358 v |= v >> 1;
3359 v |= v >> 2;
3360 v |= v >> 4;
3361 v |= v >> 8;
3362 v |= v >> 16;
3363 v |= v >> 32;
3364 v++;
3365 return v;
3366}
3367
3368// Returns largest power of 2 less or equal to v.
3369static inline uint32_t VmaPrevPow2(uint32_t v)
3370{
3371 v |= v >> 1;
3372 v |= v >> 2;
3373 v |= v >> 4;
3374 v |= v >> 8;
3375 v |= v >> 16;
3376 v = v ^ (v >> 1);
3377 return v;
3378}
3379
3380static inline uint64_t VmaPrevPow2(uint64_t v)
3381{
3382 v |= v >> 1;
3383 v |= v >> 2;
3384 v |= v >> 4;
3385 v |= v >> 8;
3386 v |= v >> 16;
3387 v |= v >> 32;
3388 v = v ^ (v >> 1);
3389 return v;
3390}
3391
3392static inline bool VmaStrIsEmpty(const char* pStr)
3393{
3394 return pStr == VMA_NULL || *pStr == '\0';
3395}
3396
3397/*
3398Returns true if two memory blocks occupy overlapping pages.
3399ResourceA must be in less memory offset than ResourceB.
3400
3401Algorithm is based on "Vulkan 1.0.39 - A Specification (with all registered Vulkan extensions)"
3402chapter 11.6 "Resource Memory Association", paragraph "Buffer-Image Granularity".
3403*/
3404static inline bool VmaBlocksOnSamePage(
3405 VkDeviceSize resourceAOffset,
3406 VkDeviceSize resourceASize,
3407 VkDeviceSize resourceBOffset,
3408 VkDeviceSize pageSize)
3409{
3410 VMA_ASSERT(resourceAOffset + resourceASize <= resourceBOffset && resourceASize > 0 && pageSize > 0);
3411 VkDeviceSize resourceAEnd = resourceAOffset + resourceASize - 1;
3412 VkDeviceSize resourceAEndPage = resourceAEnd & ~(pageSize - 1);
3413 VkDeviceSize resourceBStart = resourceBOffset;
3414 VkDeviceSize resourceBStartPage = resourceBStart & ~(pageSize - 1);
3415 return resourceAEndPage == resourceBStartPage;
3416}
3417
3418/*
3419Returns true if given suballocation types could conflict and must respect
3420VkPhysicalDeviceLimits::bufferImageGranularity. They conflict if one is buffer
3421or linear image and another one is optimal image. If type is unknown, behave
3422conservatively.
3423*/
3424static inline bool VmaIsBufferImageGranularityConflict(
3425 VmaSuballocationType suballocType1,
3426 VmaSuballocationType suballocType2)
3427{
3428 if (suballocType1 > suballocType2)
3429 {
3430 VMA_SWAP(suballocType1, suballocType2);
3431 }
3432
3433 switch (suballocType1)
3434 {
3435 case VMA_SUBALLOCATION_TYPE_FREE:
3436 return false;
3437 case VMA_SUBALLOCATION_TYPE_UNKNOWN:
3438 return true;
3439 case VMA_SUBALLOCATION_TYPE_BUFFER:
3440 return
3441 suballocType2 == VMA_SUBALLOCATION_TYPE_IMAGE_UNKNOWN ||
3442 suballocType2 == VMA_SUBALLOCATION_TYPE_IMAGE_OPTIMAL;
3443 case VMA_SUBALLOCATION_TYPE_IMAGE_UNKNOWN:
3444 return
3445 suballocType2 == VMA_SUBALLOCATION_TYPE_IMAGE_UNKNOWN ||
3446 suballocType2 == VMA_SUBALLOCATION_TYPE_IMAGE_LINEAR ||
3447 suballocType2 == VMA_SUBALLOCATION_TYPE_IMAGE_OPTIMAL;
3448 case VMA_SUBALLOCATION_TYPE_IMAGE_LINEAR:
3449 return
3450 suballocType2 == VMA_SUBALLOCATION_TYPE_IMAGE_OPTIMAL;
3451 case VMA_SUBALLOCATION_TYPE_IMAGE_OPTIMAL:
3452 return false;
3453 default:
3454 VMA_ASSERT(0);
3455 return true;
3456 }
3457}
3458
3459static void VmaWriteMagicValue(void* pData, VkDeviceSize offset)
3460{
3461#if VMA_DEBUG_MARGIN > 0 && VMA_DEBUG_DETECT_CORRUPTION
3462 uint32_t* pDst = (uint32_t*)((char*)pData + offset);
3463 const size_t numberCount = VMA_DEBUG_MARGIN / sizeof(uint32_t);
3464 for (size_t i = 0; i < numberCount; ++i, ++pDst)
3465 {
3466 *pDst = VMA_CORRUPTION_DETECTION_MAGIC_VALUE;
3467 }
3468#else
3469 // no-op
3470#endif
3471}
3472
3473static bool VmaValidateMagicValue(const void* pData, VkDeviceSize offset)
3474{
3475#if VMA_DEBUG_MARGIN > 0 && VMA_DEBUG_DETECT_CORRUPTION
3476 const uint32_t* pSrc = (const uint32_t*)((const char*)pData + offset);
3477 const size_t numberCount = VMA_DEBUG_MARGIN / sizeof(uint32_t);
3478 for (size_t i = 0; i < numberCount; ++i, ++pSrc)
3479 {
3480 if (*pSrc != VMA_CORRUPTION_DETECTION_MAGIC_VALUE)
3481 {
3482 return false;
3483 }
3484 }
3485#endif
3486 return true;
3487}
3488
3489/*
3490Fills structure with parameters of an example buffer to be used for transfers
3491during GPU memory defragmentation.
3492*/
3493static void VmaFillGpuDefragmentationBufferCreateInfo(VkBufferCreateInfo& outBufCreateInfo)
3494{
3495 memset(s: &outBufCreateInfo, c: 0, n: sizeof(outBufCreateInfo));
3496 outBufCreateInfo.sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO;
3497 outBufCreateInfo.usage = VK_BUFFER_USAGE_TRANSFER_SRC_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT;
3498 outBufCreateInfo.size = (VkDeviceSize)VMA_DEFAULT_LARGE_HEAP_BLOCK_SIZE; // Example size.
3499}
3500
3501
3502/*
3503Performs binary search and returns iterator to first element that is greater or
3504equal to (key), according to comparison (cmp).
3505
3506Cmp should return true if first argument is less than second argument.
3507
3508Returned value is the found element, if present in the collection or place where
3509new element with value (key) should be inserted.
3510*/
3511template <typename CmpLess, typename IterT, typename KeyT>
3512static IterT VmaBinaryFindFirstNotLess(IterT beg, IterT end, const KeyT& key, const CmpLess& cmp)
3513{
3514 size_t down = 0, up = (end - beg);
3515 while (down < up)
3516 {
3517 const size_t mid = down + (up - down) / 2; // Overflow-safe midpoint calculation
3518 if (cmp(*(beg + mid), key))
3519 {
3520 down = mid + 1;
3521 }
3522 else
3523 {
3524 up = mid;
3525 }
3526 }
3527 return beg + down;
3528}
3529
3530template<typename CmpLess, typename IterT, typename KeyT>
3531IterT VmaBinaryFindSorted(const IterT& beg, const IterT& end, const KeyT& value, const CmpLess& cmp)
3532{
3533 IterT it = VmaBinaryFindFirstNotLess<CmpLess, IterT, KeyT>(
3534 beg, end, value, cmp);
3535 if (it == end ||
3536 (!cmp(*it, value) && !cmp(value, *it)))
3537 {
3538 return it;
3539 }
3540 return end;
3541}
3542
3543/*
3544Returns true if all pointers in the array are not-null and unique.
3545Warning! O(n^2) complexity. Use only inside VMA_HEAVY_ASSERT.
3546T must be pointer type, e.g. VmaAllocation, VmaPool.
3547*/
3548template<typename T>
3549static bool VmaValidatePointerArray(uint32_t count, const T* arr)
3550{
3551 for (uint32_t i = 0; i < count; ++i)
3552 {
3553 const T iPtr = arr[i];
3554 if (iPtr == VMA_NULL)
3555 {
3556 return false;
3557 }
3558 for (uint32_t j = i + 1; j < count; ++j)
3559 {
3560 if (iPtr == arr[j])
3561 {
3562 return false;
3563 }
3564 }
3565 }
3566 return true;
3567}
3568
3569template<typename MainT, typename NewT>
3570static inline void VmaPnextChainPushFront(MainT* mainStruct, NewT* newStruct)
3571{
3572 newStruct->pNext = mainStruct->pNext;
3573 mainStruct->pNext = newStruct;
3574}
3575
3576// This is the main algorithm that guides the selection of a memory type best for an allocation -
3577// converts usage to required/preferred/not preferred flags.
3578static bool FindMemoryPreferences(
3579 bool isIntegratedGPU,
3580 const VmaAllocationCreateInfo& allocCreateInfo,
3581 VkFlags bufImgUsage, // VkBufferCreateInfo::usage or VkImageCreateInfo::usage. UINT32_MAX if unknown.
3582 VkMemoryPropertyFlags& outRequiredFlags,
3583 VkMemoryPropertyFlags& outPreferredFlags,
3584 VkMemoryPropertyFlags& outNotPreferredFlags)
3585{
3586 outRequiredFlags = allocCreateInfo.requiredFlags;
3587 outPreferredFlags = allocCreateInfo.preferredFlags;
3588 outNotPreferredFlags = 0;
3589
3590 switch(allocCreateInfo.usage)
3591 {
3592 case VMA_MEMORY_USAGE_UNKNOWN:
3593 break;
3594 case VMA_MEMORY_USAGE_GPU_ONLY:
3595 if(!isIntegratedGPU || (outPreferredFlags & VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT) == 0)
3596 {
3597 outPreferredFlags |= VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT;
3598 }
3599 break;
3600 case VMA_MEMORY_USAGE_CPU_ONLY:
3601 outRequiredFlags |= VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | VK_MEMORY_PROPERTY_HOST_COHERENT_BIT;
3602 break;
3603 case VMA_MEMORY_USAGE_CPU_TO_GPU:
3604 outRequiredFlags |= VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT;
3605 if(!isIntegratedGPU || (outPreferredFlags & VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT) == 0)
3606 {
3607 outPreferredFlags |= VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT;
3608 }
3609 break;
3610 case VMA_MEMORY_USAGE_GPU_TO_CPU:
3611 outRequiredFlags |= VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT;
3612 outPreferredFlags |= VK_MEMORY_PROPERTY_HOST_CACHED_BIT;
3613 break;
3614 case VMA_MEMORY_USAGE_CPU_COPY:
3615 outNotPreferredFlags |= VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT;
3616 break;
3617 case VMA_MEMORY_USAGE_GPU_LAZILY_ALLOCATED:
3618 outRequiredFlags |= VK_MEMORY_PROPERTY_LAZILY_ALLOCATED_BIT;
3619 break;
3620 case VMA_MEMORY_USAGE_AUTO:
3621 case VMA_MEMORY_USAGE_AUTO_PREFER_DEVICE:
3622 case VMA_MEMORY_USAGE_AUTO_PREFER_HOST:
3623 {
3624 if(bufImgUsage == UINT32_MAX)
3625 {
3626 VMA_ASSERT(0 && "VMA_MEMORY_USAGE_AUTO* values can only be used with functions like vmaCreateBuffer, vmaCreateImage so that the details of the created resource are known.");
3627 return false;
3628 }
3629 // This relies on values of VK_IMAGE_USAGE_TRANSFER* being the same VK_BUFFER_IMAGE_TRANSFER*.
3630 const bool deviceAccess = (bufImgUsage & ~(VK_BUFFER_USAGE_TRANSFER_DST_BIT | VK_BUFFER_USAGE_TRANSFER_SRC_BIT)) != 0;
3631 const bool hostAccessSequentialWrite = (allocCreateInfo.flags & VMA_ALLOCATION_CREATE_HOST_ACCESS_SEQUENTIAL_WRITE_BIT) != 0;
3632 const bool hostAccessRandom = (allocCreateInfo.flags & VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT) != 0;
3633 const bool hostAccessAllowTransferInstead = (allocCreateInfo.flags & VMA_ALLOCATION_CREATE_HOST_ACCESS_ALLOW_TRANSFER_INSTEAD_BIT) != 0;
3634 const bool preferDevice = allocCreateInfo.usage == VMA_MEMORY_USAGE_AUTO_PREFER_DEVICE;
3635 const bool preferHost = allocCreateInfo.usage == VMA_MEMORY_USAGE_AUTO_PREFER_HOST;
3636
3637 // CPU random access - e.g. a buffer written to or transferred from GPU to read back on CPU.
3638 if(hostAccessRandom)
3639 {
3640 if(!isIntegratedGPU && deviceAccess && hostAccessAllowTransferInstead && !preferHost)
3641 {
3642 // Nice if it will end up in HOST_VISIBLE, but more importantly prefer DEVICE_LOCAL.
3643 // Omitting HOST_VISIBLE here is intentional.
3644 // In case there is DEVICE_LOCAL | HOST_VISIBLE | HOST_CACHED, it will pick that one.
3645 // Otherwise, this will give same weight to DEVICE_LOCAL as HOST_VISIBLE | HOST_CACHED and select the former if occurs first on the list.
3646 outPreferredFlags |= VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT | VK_MEMORY_PROPERTY_HOST_CACHED_BIT;
3647 }
3648 else
3649 {
3650 // Always CPU memory, cached.
3651 outRequiredFlags |= VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | VK_MEMORY_PROPERTY_HOST_CACHED_BIT;
3652 }
3653 }
3654 // CPU sequential write - may be CPU or host-visible GPU memory, uncached and write-combined.
3655 else if(hostAccessSequentialWrite)
3656 {
3657 // Want uncached and write-combined.
3658 outNotPreferredFlags |= VK_MEMORY_PROPERTY_HOST_CACHED_BIT;
3659
3660 if(!isIntegratedGPU && deviceAccess && hostAccessAllowTransferInstead && !preferHost)
3661 {
3662 outPreferredFlags |= VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT | VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT;
3663 }
3664 else
3665 {
3666 outRequiredFlags |= VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT;
3667 // Direct GPU access, CPU sequential write (e.g. a dynamic uniform buffer updated every frame)
3668 if(deviceAccess)
3669 {
3670 // Could go to CPU memory or GPU BAR/unified. Up to the user to decide. If no preference, choose GPU memory.
3671 if(preferHost)
3672 outNotPreferredFlags |= VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT;
3673 else
3674 outPreferredFlags |= VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT;
3675 }
3676 // GPU no direct access, CPU sequential write (e.g. an upload buffer to be transferred to the GPU)
3677 else
3678 {
3679 // Could go to CPU memory or GPU BAR/unified. Up to the user to decide. If no preference, choose CPU memory.
3680 if(preferDevice)
3681 outPreferredFlags |= VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT;
3682 else
3683 outNotPreferredFlags |= VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT;
3684 }
3685 }
3686 }
3687 // No CPU access
3688 else
3689 {
3690 // GPU access, no CPU access (e.g. a color attachment image) - prefer GPU memory
3691 if(deviceAccess)
3692 {
3693 // ...unless there is a clear preference from the user not to do so.
3694 if(preferHost)
3695 outNotPreferredFlags |= VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT;
3696 else
3697 outPreferredFlags |= VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT;
3698 }
3699 // No direct GPU access, no CPU access, just transfers.
3700 // It may be staging copy intended for e.g. preserving image for next frame (then better GPU memory) or
3701 // a "swap file" copy to free some GPU memory (then better CPU memory).
3702 // Up to the user to decide. If no preferece, assume the former and choose GPU memory.
3703 if(preferHost)
3704 outNotPreferredFlags |= VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT;
3705 else
3706 outPreferredFlags |= VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT;
3707 }
3708 break;
3709 }
3710 default:
3711 VMA_ASSERT(0);
3712 }
3713
3714 // Avoid DEVICE_COHERENT unless explicitly requested.
3715 if(((allocCreateInfo.requiredFlags | allocCreateInfo.preferredFlags) &
3716 (VK_MEMORY_PROPERTY_DEVICE_COHERENT_BIT_AMD_COPY | VK_MEMORY_PROPERTY_DEVICE_UNCACHED_BIT_AMD_COPY)) == 0)
3717 {
3718 outNotPreferredFlags |= VK_MEMORY_PROPERTY_DEVICE_UNCACHED_BIT_AMD_COPY;
3719 }
3720
3721 return true;
3722}
3723
3724////////////////////////////////////////////////////////////////////////////////
3725// Memory allocation
3726
3727static void* VmaMalloc(const VkAllocationCallbacks* pAllocationCallbacks, size_t size, size_t alignment)
3728{
3729 void* result = VMA_NULL;
3730 if ((pAllocationCallbacks != VMA_NULL) &&
3731 (pAllocationCallbacks->pfnAllocation != VMA_NULL))
3732 {
3733 result = (*pAllocationCallbacks->pfnAllocation)(
3734 pAllocationCallbacks->pUserData,
3735 size,
3736 alignment,
3737 VK_SYSTEM_ALLOCATION_SCOPE_OBJECT);
3738 }
3739 else
3740 {
3741 result = VMA_SYSTEM_ALIGNED_MALLOC(size, alignment);
3742 }
3743 VMA_ASSERT(result != VMA_NULL && "CPU memory allocation failed.");
3744 return result;
3745}
3746
3747static void VmaFree(const VkAllocationCallbacks* pAllocationCallbacks, void* ptr)
3748{
3749 if ((pAllocationCallbacks != VMA_NULL) &&
3750 (pAllocationCallbacks->pfnFree != VMA_NULL))
3751 {
3752 (*pAllocationCallbacks->pfnFree)(pAllocationCallbacks->pUserData, ptr);
3753 }
3754 else
3755 {
3756 VMA_SYSTEM_ALIGNED_FREE(ptr);
3757 }
3758}
3759
3760template<typename T>
3761static T* VmaAllocate(const VkAllocationCallbacks* pAllocationCallbacks)
3762{
3763 return (T*)VmaMalloc(pAllocationCallbacks, size: sizeof(T), VMA_ALIGN_OF(T));
3764}
3765
3766template<typename T>
3767static T* VmaAllocateArray(const VkAllocationCallbacks* pAllocationCallbacks, size_t count)
3768{
3769 return (T*)VmaMalloc(pAllocationCallbacks, size: sizeof(T) * count, VMA_ALIGN_OF(T));
3770}
3771
3772#define vma_new(allocator, type) new(VmaAllocate<type>(allocator))(type)
3773
3774#define vma_new_array(allocator, type, count) new(VmaAllocateArray<type>((allocator), (count)))(type)
3775
3776template<typename T>
3777static void vma_delete(const VkAllocationCallbacks* pAllocationCallbacks, T* ptr)
3778{
3779 ptr->~T();
3780 VmaFree(pAllocationCallbacks, ptr);
3781}
3782
3783template<typename T>
3784static void vma_delete_array(const VkAllocationCallbacks* pAllocationCallbacks, T* ptr, size_t count)
3785{
3786 if (ptr != VMA_NULL)
3787 {
3788 for (size_t i = count; i--; )
3789 {
3790 ptr[i].~T();
3791 }
3792 VmaFree(pAllocationCallbacks, ptr);
3793 }
3794}
3795
3796static char* VmaCreateStringCopy(const VkAllocationCallbacks* allocs, const char* srcStr)
3797{
3798 if (srcStr != VMA_NULL)
3799 {
3800 const size_t len = strlen(s: srcStr);
3801 char* const result = vma_new_array(allocs, char, len + 1);
3802 memcpy(dest: result, src: srcStr, n: len + 1);
3803 return result;
3804 }
3805 return VMA_NULL;
3806}
3807
3808#if VMA_STATS_STRING_ENABLED
3809static char* VmaCreateStringCopy(const VkAllocationCallbacks* allocs, const char* srcStr, size_t strLen)
3810{
3811 if (srcStr != VMA_NULL)
3812 {
3813 char* const result = vma_new_array(allocs, char, strLen + 1);
3814 memcpy(dest: result, src: srcStr, n: strLen);
3815 result[strLen] = '\0';
3816 return result;
3817 }
3818 return VMA_NULL;
3819}
3820#endif // VMA_STATS_STRING_ENABLED
3821
3822static void VmaFreeString(const VkAllocationCallbacks* allocs, char* str)
3823{
3824 if (str != VMA_NULL)
3825 {
3826 const size_t len = strlen(s: str);
3827 vma_delete_array(pAllocationCallbacks: allocs, ptr: str, count: len + 1);
3828 }
3829}
3830
3831template<typename CmpLess, typename VectorT>
3832size_t VmaVectorInsertSorted(VectorT& vector, const typename VectorT::value_type& value)
3833{
3834 const size_t indexToInsert = VmaBinaryFindFirstNotLess(
3835 vector.data(),
3836 vector.data() + vector.size(),
3837 value,
3838 CmpLess()) - vector.data();
3839 VmaVectorInsert(vector, indexToInsert, value);
3840 return indexToInsert;
3841}
3842
3843template<typename CmpLess, typename VectorT>
3844bool VmaVectorRemoveSorted(VectorT& vector, const typename VectorT::value_type& value)
3845{
3846 CmpLess comparator;
3847 typename VectorT::iterator it = VmaBinaryFindFirstNotLess(
3848 vector.begin(),
3849 vector.end(),
3850 value,
3851 comparator);
3852 if ((it != vector.end()) && !comparator(*it, value) && !comparator(value, *it))
3853 {
3854 size_t indexToRemove = it - vector.begin();
3855 VmaVectorRemove(vector, indexToRemove);
3856 return true;
3857 }
3858 return false;
3859}
3860#endif // _VMA_FUNCTIONS
3861
3862#ifndef _VMA_STATISTICS_FUNCTIONS
3863
3864static void VmaClearStatistics(VmaStatistics& outStats)
3865{
3866 outStats.blockCount = 0;
3867 outStats.allocationCount = 0;
3868 outStats.blockBytes = 0;
3869 outStats.allocationBytes = 0;
3870}
3871
3872static void VmaAddStatistics(VmaStatistics& inoutStats, const VmaStatistics& src)
3873{
3874 inoutStats.blockCount += src.blockCount;
3875 inoutStats.allocationCount += src.allocationCount;
3876 inoutStats.blockBytes += src.blockBytes;
3877 inoutStats.allocationBytes += src.allocationBytes;
3878}
3879
3880static void VmaClearDetailedStatistics(VmaDetailedStatistics& outStats)
3881{
3882 VmaClearStatistics(outStats&: outStats.statistics);
3883 outStats.unusedRangeCount = 0;
3884 outStats.allocationSizeMin = VK_WHOLE_SIZE;
3885 outStats.allocationSizeMax = 0;
3886 outStats.unusedRangeSizeMin = VK_WHOLE_SIZE;
3887 outStats.unusedRangeSizeMax = 0;
3888}
3889
3890static void VmaAddDetailedStatisticsAllocation(VmaDetailedStatistics& inoutStats, VkDeviceSize size)
3891{
3892 inoutStats.statistics.allocationCount++;
3893 inoutStats.statistics.allocationBytes += size;
3894 inoutStats.allocationSizeMin = VMA_MIN(inoutStats.allocationSizeMin, size);
3895 inoutStats.allocationSizeMax = VMA_MAX(inoutStats.allocationSizeMax, size);
3896}
3897
3898static void VmaAddDetailedStatisticsUnusedRange(VmaDetailedStatistics& inoutStats, VkDeviceSize size)
3899{
3900 inoutStats.unusedRangeCount++;
3901 inoutStats.unusedRangeSizeMin = VMA_MIN(inoutStats.unusedRangeSizeMin, size);
3902 inoutStats.unusedRangeSizeMax = VMA_MAX(inoutStats.unusedRangeSizeMax, size);
3903}
3904
3905static void VmaAddDetailedStatistics(VmaDetailedStatistics& inoutStats, const VmaDetailedStatistics& src)
3906{
3907 VmaAddStatistics(inoutStats&: inoutStats.statistics, src: src.statistics);
3908 inoutStats.unusedRangeCount += src.unusedRangeCount;
3909 inoutStats.allocationSizeMin = VMA_MIN(inoutStats.allocationSizeMin, src.allocationSizeMin);
3910 inoutStats.allocationSizeMax = VMA_MAX(inoutStats.allocationSizeMax, src.allocationSizeMax);
3911 inoutStats.unusedRangeSizeMin = VMA_MIN(inoutStats.unusedRangeSizeMin, src.unusedRangeSizeMin);
3912 inoutStats.unusedRangeSizeMax = VMA_MAX(inoutStats.unusedRangeSizeMax, src.unusedRangeSizeMax);
3913}
3914
3915#endif // _VMA_STATISTICS_FUNCTIONS
3916
3917#ifndef _VMA_MUTEX_LOCK
3918// Helper RAII class to lock a mutex in constructor and unlock it in destructor (at the end of scope).
3919struct VmaMutexLock
3920{
3921 VMA_CLASS_NO_COPY(VmaMutexLock)
3922public:
3923 VmaMutexLock(VMA_MUTEX& mutex, bool useMutex = true) :
3924 m_pMutex(useMutex ? &mutex : VMA_NULL)
3925 {
3926 if (m_pMutex) { m_pMutex->Lock(); }
3927 }
3928 ~VmaMutexLock() { if (m_pMutex) { m_pMutex->Unlock(); } }
3929
3930private:
3931 VMA_MUTEX* m_pMutex;
3932};
3933
3934// Helper RAII class to lock a RW mutex in constructor and unlock it in destructor (at the end of scope), for reading.
3935struct VmaMutexLockRead
3936{
3937 VMA_CLASS_NO_COPY(VmaMutexLockRead)
3938public:
3939 VmaMutexLockRead(VMA_RW_MUTEX& mutex, bool useMutex) :
3940 m_pMutex(useMutex ? &mutex : VMA_NULL)
3941 {
3942 if (m_pMutex) { m_pMutex->LockRead(); }
3943 }
3944 ~VmaMutexLockRead() { if (m_pMutex) { m_pMutex->UnlockRead(); } }
3945
3946private:
3947 VMA_RW_MUTEX* m_pMutex;
3948};
3949
3950// Helper RAII class to lock a RW mutex in constructor and unlock it in destructor (at the end of scope), for writing.
3951struct VmaMutexLockWrite
3952{
3953 VMA_CLASS_NO_COPY(VmaMutexLockWrite)
3954public:
3955 VmaMutexLockWrite(VMA_RW_MUTEX& mutex, bool useMutex)
3956 : m_pMutex(useMutex ? &mutex : VMA_NULL)
3957 {
3958 if (m_pMutex) { m_pMutex->LockWrite(); }
3959 }
3960 ~VmaMutexLockWrite() { if (m_pMutex) { m_pMutex->UnlockWrite(); } }
3961
3962private:
3963 VMA_RW_MUTEX* m_pMutex;
3964};
3965
3966#if VMA_DEBUG_GLOBAL_MUTEX
3967 static VMA_MUTEX gDebugGlobalMutex;
3968 #define VMA_DEBUG_GLOBAL_MUTEX_LOCK VmaMutexLock debugGlobalMutexLock(gDebugGlobalMutex, true);
3969#else
3970 #define VMA_DEBUG_GLOBAL_MUTEX_LOCK
3971#endif
3972#endif // _VMA_MUTEX_LOCK
3973
3974#ifndef _VMA_ATOMIC_TRANSACTIONAL_INCREMENT
3975// An object that increments given atomic but decrements it back in the destructor unless Commit() is called.
3976template<typename T>
3977struct AtomicTransactionalIncrement
3978{
3979public:
3980 typedef std::atomic<T> AtomicT;
3981
3982 ~AtomicTransactionalIncrement()
3983 {
3984 if(m_Atomic)
3985 --(*m_Atomic);
3986 }
3987
3988 void Commit() { m_Atomic = nullptr; }
3989 T Increment(AtomicT* atomic)
3990 {
3991 m_Atomic = atomic;
3992 return m_Atomic->fetch_add(1);
3993 }
3994
3995private:
3996 AtomicT* m_Atomic = nullptr;
3997};
3998#endif // _VMA_ATOMIC_TRANSACTIONAL_INCREMENT
3999
4000#ifndef _VMA_STL_ALLOCATOR
4001// STL-compatible allocator.
4002template<typename T>
4003struct VmaStlAllocator
4004{
4005 const VkAllocationCallbacks* const m_pCallbacks;
4006 typedef T value_type;
4007
4008 VmaStlAllocator(const VkAllocationCallbacks* pCallbacks) : m_pCallbacks(pCallbacks) {}
4009 template<typename U>
4010 VmaStlAllocator(const VmaStlAllocator<U>& src) : m_pCallbacks(src.m_pCallbacks) {}
4011 VmaStlAllocator(const VmaStlAllocator&) = default;
4012 VmaStlAllocator& operator=(const VmaStlAllocator&) = delete;
4013
4014 T* allocate(size_t n) { return VmaAllocateArray<T>(m_pCallbacks, n); }
4015 void deallocate(T* p, size_t n) { VmaFree(m_pCallbacks, p); }
4016
4017 template<typename U>
4018 bool operator==(const VmaStlAllocator<U>& rhs) const
4019 {
4020 return m_pCallbacks == rhs.m_pCallbacks;
4021 }
4022 template<typename U>
4023 bool operator!=(const VmaStlAllocator<U>& rhs) const
4024 {
4025 return m_pCallbacks != rhs.m_pCallbacks;
4026 }
4027};
4028#endif // _VMA_STL_ALLOCATOR
4029
4030#ifndef _VMA_VECTOR
4031/* Class with interface compatible with subset of std::vector.
4032T must be POD because constructors and destructors are not called and memcpy is
4033used for these objects. */
4034template<typename T, typename AllocatorT>
4035class VmaVector
4036{
4037public:
4038 typedef T value_type;
4039 typedef T* iterator;
4040 typedef const T* const_iterator;
4041
4042 VmaVector(const AllocatorT& allocator);
4043 VmaVector(size_t count, const AllocatorT& allocator);
4044 // This version of the constructor is here for compatibility with pre-C++14 std::vector.
4045 // value is unused.
4046 VmaVector(size_t count, const T& value, const AllocatorT& allocator) : VmaVector(count, allocator) {}
4047 VmaVector(const VmaVector<T, AllocatorT>& src);
4048 VmaVector& operator=(const VmaVector& rhs);
4049 ~VmaVector() { VmaFree(m_Allocator.m_pCallbacks, m_pArray); }
4050
4051 bool empty() const { return m_Count == 0; }
4052 size_t size() const { return m_Count; }
4053 T* data() { return m_pArray; }
4054 T& front() { VMA_HEAVY_ASSERT(m_Count > 0); return m_pArray[0]; }
4055 T& back() { VMA_HEAVY_ASSERT(m_Count > 0); return m_pArray[m_Count - 1]; }
4056 const T* data() const { return m_pArray; }
4057 const T& front() const { VMA_HEAVY_ASSERT(m_Count > 0); return m_pArray[0]; }
4058 const T& back() const { VMA_HEAVY_ASSERT(m_Count > 0); return m_pArray[m_Count - 1]; }
4059
4060 iterator begin() { return m_pArray; }
4061 iterator end() { return m_pArray + m_Count; }
4062 const_iterator cbegin() const { return m_pArray; }
4063 const_iterator cend() const { return m_pArray + m_Count; }
4064 const_iterator begin() const { return cbegin(); }
4065 const_iterator end() const { return cend(); }
4066
4067 void pop_front() { VMA_HEAVY_ASSERT(m_Count > 0); remove(index: 0); }
4068 void pop_back() { VMA_HEAVY_ASSERT(m_Count > 0); resize(newCount: size() - 1); }
4069 void push_front(const T& src) { insert(index: 0, src); }
4070
4071 void push_back(const T& src);
4072 void reserve(size_t newCapacity, bool freeMemory = false);
4073 void resize(size_t newCount);
4074 void clear() { resize(newCount: 0); }
4075 void shrink_to_fit();
4076 void insert(size_t index, const T& src);
4077 void remove(size_t index);
4078
4079 T& operator[](size_t index) { VMA_HEAVY_ASSERT(index < m_Count); return m_pArray[index]; }
4080 const T& operator[](size_t index) const { VMA_HEAVY_ASSERT(index < m_Count); return m_pArray[index]; }
4081
4082private:
4083 AllocatorT m_Allocator;
4084 T* m_pArray;
4085 size_t m_Count;
4086 size_t m_Capacity;
4087};
4088
4089#ifndef _VMA_VECTOR_FUNCTIONS
4090template<typename T, typename AllocatorT>
4091VmaVector<T, AllocatorT>::VmaVector(const AllocatorT& allocator)
4092 : m_Allocator(allocator),
4093 m_pArray(VMA_NULL),
4094 m_Count(0),
4095 m_Capacity(0) {}
4096
4097template<typename T, typename AllocatorT>
4098VmaVector<T, AllocatorT>::VmaVector(size_t count, const AllocatorT& allocator)
4099 : m_Allocator(allocator),
4100 m_pArray(count ? (T*)VmaAllocateArray<T>(allocator.m_pCallbacks, count) : VMA_NULL),
4101 m_Count(count),
4102 m_Capacity(count) {}
4103
4104template<typename T, typename AllocatorT>
4105VmaVector<T, AllocatorT>::VmaVector(const VmaVector& src)
4106 : m_Allocator(src.m_Allocator),
4107 m_pArray(src.m_Count ? (T*)VmaAllocateArray<T>(src.m_Allocator.m_pCallbacks, src.m_Count) : VMA_NULL),
4108 m_Count(src.m_Count),
4109 m_Capacity(src.m_Count)
4110{
4111 if (m_Count != 0)
4112 {
4113 memcpy(m_pArray, src.m_pArray, m_Count * sizeof(T));
4114 }
4115}
4116
4117template<typename T, typename AllocatorT>
4118VmaVector<T, AllocatorT>& VmaVector<T, AllocatorT>::operator=(const VmaVector& rhs)
4119{
4120 if (&rhs != this)
4121 {
4122 resize(newCount: rhs.m_Count);
4123 if (m_Count != 0)
4124 {
4125 memcpy(m_pArray, rhs.m_pArray, m_Count * sizeof(T));
4126 }
4127 }
4128 return *this;
4129}
4130
4131template<typename T, typename AllocatorT>
4132void VmaVector<T, AllocatorT>::push_back(const T& src)
4133{
4134 const size_t newIndex = size();
4135 resize(newCount: newIndex + 1);
4136 m_pArray[newIndex] = src;
4137}
4138
4139template<typename T, typename AllocatorT>
4140void VmaVector<T, AllocatorT>::reserve(size_t newCapacity, bool freeMemory)
4141{
4142 newCapacity = VMA_MAX(newCapacity, m_Count);
4143
4144 if ((newCapacity < m_Capacity) && !freeMemory)
4145 {
4146 newCapacity = m_Capacity;
4147 }
4148
4149 if (newCapacity != m_Capacity)
4150 {
4151 T* const newArray = newCapacity ? VmaAllocateArray<T>(m_Allocator, newCapacity) : VMA_NULL;
4152 if (m_Count != 0)
4153 {
4154 memcpy(newArray, m_pArray, m_Count * sizeof(T));
4155 }
4156 VmaFree(m_Allocator.m_pCallbacks, m_pArray);
4157 m_Capacity = newCapacity;
4158 m_pArray = newArray;
4159 }
4160}
4161
4162template<typename T, typename AllocatorT>
4163void VmaVector<T, AllocatorT>::resize(size_t newCount)
4164{
4165 size_t newCapacity = m_Capacity;
4166 if (newCount > m_Capacity)
4167 {
4168 newCapacity = VMA_MAX(newCount, VMA_MAX(m_Capacity * 3 / 2, (size_t)8));
4169 }
4170
4171 if (newCapacity != m_Capacity)
4172 {
4173 T* const newArray = newCapacity ? VmaAllocateArray<T>(m_Allocator.m_pCallbacks, newCapacity) : VMA_NULL;
4174 const size_t elementsToCopy = VMA_MIN(m_Count, newCount);
4175 if (elementsToCopy != 0)
4176 {
4177 memcpy(newArray, m_pArray, elementsToCopy * sizeof(T));
4178 }
4179 VmaFree(m_Allocator.m_pCallbacks, m_pArray);
4180 m_Capacity = newCapacity;
4181 m_pArray = newArray;
4182 }
4183
4184 m_Count = newCount;
4185}
4186
4187template<typename T, typename AllocatorT>
4188void VmaVector<T, AllocatorT>::shrink_to_fit()
4189{
4190 if (m_Capacity > m_Count)
4191 {
4192 T* newArray = VMA_NULL;
4193 if (m_Count > 0)
4194 {
4195 newArray = VmaAllocateArray<T>(m_Allocator.m_pCallbacks, m_Count);
4196 memcpy(newArray, m_pArray, m_Count * sizeof(T));
4197 }
4198 VmaFree(m_Allocator.m_pCallbacks, m_pArray);
4199 m_Capacity = m_Count;
4200 m_pArray = newArray;
4201 }
4202}
4203
4204template<typename T, typename AllocatorT>
4205void VmaVector<T, AllocatorT>::insert(size_t index, const T& src)
4206{
4207 VMA_HEAVY_ASSERT(index <= m_Count);
4208 const size_t oldCount = size();
4209 resize(newCount: oldCount + 1);
4210 if (index < oldCount)
4211 {
4212 memmove(m_pArray + (index + 1), m_pArray + index, (oldCount - index) * sizeof(T));
4213 }
4214 m_pArray[index] = src;
4215}
4216
4217template<typename T, typename AllocatorT>
4218void VmaVector<T, AllocatorT>::remove(size_t index)
4219{
4220 VMA_HEAVY_ASSERT(index < m_Count);
4221 const size_t oldCount = size();
4222 if (index < oldCount - 1)
4223 {
4224 memmove(m_pArray + index, m_pArray + (index + 1), (oldCount - index - 1) * sizeof(T));
4225 }
4226 resize(newCount: oldCount - 1);
4227}
4228#endif // _VMA_VECTOR_FUNCTIONS
4229
4230template<typename T, typename allocatorT>
4231static void VmaVectorInsert(VmaVector<T, allocatorT>& vec, size_t index, const T& item)
4232{
4233 vec.insert(index, item);
4234}
4235
4236template<typename T, typename allocatorT>
4237static void VmaVectorRemove(VmaVector<T, allocatorT>& vec, size_t index)
4238{
4239 vec.remove(index);
4240}
4241#endif // _VMA_VECTOR
4242
4243#ifndef _VMA_SMALL_VECTOR
4244/*
4245This is a vector (a variable-sized array), optimized for the case when the array is small.
4246
4247It contains some number of elements in-place, which allows it to avoid heap allocation
4248when the actual number of elements is below that threshold. This allows normal "small"
4249cases to be fast without losing generality for large inputs.
4250*/
4251template<typename T, typename AllocatorT, size_t N>
4252class VmaSmallVector
4253{
4254public:
4255 typedef T value_type;
4256 typedef T* iterator;
4257
4258 VmaSmallVector(const AllocatorT& allocator);
4259 VmaSmallVector(size_t count, const AllocatorT& allocator);
4260 template<typename SrcT, typename SrcAllocatorT, size_t SrcN>
4261 VmaSmallVector(const VmaSmallVector<SrcT, SrcAllocatorT, SrcN>&) = delete;
4262 template<typename SrcT, typename SrcAllocatorT, size_t SrcN>
4263 VmaSmallVector<T, AllocatorT, N>& operator=(const VmaSmallVector<SrcT, SrcAllocatorT, SrcN>&) = delete;
4264 ~VmaSmallVector() = default;
4265
4266 bool empty() const { return m_Count == 0; }
4267 size_t size() const { return m_Count; }
4268 T* data() { return m_Count > N ? m_DynamicArray.data() : m_StaticArray; }
4269 T& front() { VMA_HEAVY_ASSERT(m_Count > 0); return data()[0]; }
4270 T& back() { VMA_HEAVY_ASSERT(m_Count > 0); return data()[m_Count - 1]; }
4271 const T* data() const { return m_Count > N ? m_DynamicArray.data() : m_StaticArray; }
4272 const T& front() const { VMA_HEAVY_ASSERT(m_Count > 0); return data()[0]; }
4273 const T& back() const { VMA_HEAVY_ASSERT(m_Count > 0); return data()[m_Count - 1]; }
4274
4275 iterator begin() { return data(); }
4276 iterator end() { return data() + m_Count; }
4277
4278 void pop_front() { VMA_HEAVY_ASSERT(m_Count > 0); remove(index: 0); }
4279 void pop_back() { VMA_HEAVY_ASSERT(m_Count > 0); resize(newCount: size() - 1); }
4280 void push_front(const T& src) { insert(index: 0, src); }
4281
4282 void push_back(const T& src);
4283 void resize(size_t newCount, bool freeMemory = false);
4284 void clear(bool freeMemory = false);
4285 void insert(size_t index, const T& src);
4286 void remove(size_t index);
4287
4288 T& operator[](size_t index) { VMA_HEAVY_ASSERT(index < m_Count); return data()[index]; }
4289 const T& operator[](size_t index) const { VMA_HEAVY_ASSERT(index < m_Count); return data()[index]; }
4290
4291private:
4292 size_t m_Count;
4293 T m_StaticArray[N]; // Used when m_Size <= N
4294 VmaVector<T, AllocatorT> m_DynamicArray; // Used when m_Size > N
4295};
4296
4297#ifndef _VMA_SMALL_VECTOR_FUNCTIONS
4298template<typename T, typename AllocatorT, size_t N>
4299VmaSmallVector<T, AllocatorT, N>::VmaSmallVector(const AllocatorT& allocator)
4300 : m_Count(0),
4301 m_DynamicArray(allocator) {}
4302
4303template<typename T, typename AllocatorT, size_t N>
4304VmaSmallVector<T, AllocatorT, N>::VmaSmallVector(size_t count, const AllocatorT& allocator)
4305 : m_Count(count),
4306 m_DynamicArray(count > N ? count : 0, allocator) {}
4307
4308template<typename T, typename AllocatorT, size_t N>
4309void VmaSmallVector<T, AllocatorT, N>::push_back(const T& src)
4310{
4311 const size_t newIndex = size();
4312 resize(newCount: newIndex + 1);
4313 data()[newIndex] = src;
4314}
4315
4316template<typename T, typename AllocatorT, size_t N>
4317void VmaSmallVector<T, AllocatorT, N>::resize(size_t newCount, bool freeMemory)
4318{
4319 if (newCount > N && m_Count > N)
4320 {
4321 // Any direction, staying in m_DynamicArray
4322 m_DynamicArray.resize(newCount);
4323 if (freeMemory)
4324 {
4325 m_DynamicArray.shrink_to_fit();
4326 }
4327 }
4328 else if (newCount > N && m_Count <= N)
4329 {
4330 // Growing, moving from m_StaticArray to m_DynamicArray
4331 m_DynamicArray.resize(newCount);
4332 if (m_Count > 0)
4333 {
4334 memcpy(m_DynamicArray.data(), m_StaticArray, m_Count * sizeof(T));
4335 }
4336 }
4337 else if (newCount <= N && m_Count > N)
4338 {
4339 // Shrinking, moving from m_DynamicArray to m_StaticArray
4340 if (newCount > 0)
4341 {
4342 memcpy(m_StaticArray, m_DynamicArray.data(), newCount * sizeof(T));
4343 }
4344 m_DynamicArray.resize(0);
4345 if (freeMemory)
4346 {
4347 m_DynamicArray.shrink_to_fit();
4348 }
4349 }
4350 else
4351 {
4352 // Any direction, staying in m_StaticArray - nothing to do here
4353 }
4354 m_Count = newCount;
4355}
4356
4357template<typename T, typename AllocatorT, size_t N>
4358void VmaSmallVector<T, AllocatorT, N>::clear(bool freeMemory)
4359{
4360 m_DynamicArray.clear();
4361 if (freeMemory)
4362 {
4363 m_DynamicArray.shrink_to_fit();
4364 }
4365 m_Count = 0;
4366}
4367
4368template<typename T, typename AllocatorT, size_t N>
4369void VmaSmallVector<T, AllocatorT, N>::insert(size_t index, const T& src)
4370{
4371 VMA_HEAVY_ASSERT(index <= m_Count);
4372 const size_t oldCount = size();
4373 resize(newCount: oldCount + 1);
4374 T* const dataPtr = data();
4375 if (index < oldCount)
4376 {
4377 // I know, this could be more optimal for case where memmove can be memcpy directly from m_StaticArray to m_DynamicArray.
4378 memmove(dataPtr + (index + 1), dataPtr + index, (oldCount - index) * sizeof(T));
4379 }
4380 dataPtr[index] = src;
4381}
4382
4383template<typename T, typename AllocatorT, size_t N>
4384void VmaSmallVector<T, AllocatorT, N>::remove(size_t index)
4385{
4386 VMA_HEAVY_ASSERT(index < m_Count);
4387 const size_t oldCount = size();
4388 if (index < oldCount - 1)
4389 {
4390 // I know, this could be more optimal for case where memmove can be memcpy directly from m_DynamicArray to m_StaticArray.
4391 T* const dataPtr = data();
4392 memmove(dataPtr + index, dataPtr + (index + 1), (oldCount - index - 1) * sizeof(T));
4393 }
4394 resize(newCount: oldCount - 1);
4395}
4396#endif // _VMA_SMALL_VECTOR_FUNCTIONS
4397#endif // _VMA_SMALL_VECTOR
4398
4399#ifndef _VMA_POOL_ALLOCATOR
4400/*
4401Allocator for objects of type T using a list of arrays (pools) to speed up
4402allocation. Number of elements that can be allocated is not bounded because
4403allocator can create multiple blocks.
4404*/
4405template<typename T>
4406class VmaPoolAllocator
4407{
4408 VMA_CLASS_NO_COPY(VmaPoolAllocator)
4409public:
4410 VmaPoolAllocator(const VkAllocationCallbacks* pAllocationCallbacks, uint32_t firstBlockCapacity);
4411 ~VmaPoolAllocator();
4412 template<typename... Types> T* Alloc(Types&&... args);
4413 void Free(T* ptr);
4414
4415private:
4416 union Item
4417 {
4418 uint32_t NextFreeIndex;
4419 alignas(T) char Value[sizeof(T)];
4420 };
4421 struct ItemBlock
4422 {
4423 Item* pItems;
4424 uint32_t Capacity;
4425 uint32_t FirstFreeIndex;
4426 };
4427
4428 const VkAllocationCallbacks* m_pAllocationCallbacks;
4429 const uint32_t m_FirstBlockCapacity;
4430 VmaVector<ItemBlock, VmaStlAllocator<ItemBlock>> m_ItemBlocks;
4431
4432 ItemBlock& CreateNewBlock();
4433};
4434
4435#ifndef _VMA_POOL_ALLOCATOR_FUNCTIONS
4436template<typename T>
4437VmaPoolAllocator<T>::VmaPoolAllocator(const VkAllocationCallbacks* pAllocationCallbacks, uint32_t firstBlockCapacity)
4438 : m_pAllocationCallbacks(pAllocationCallbacks),
4439 m_FirstBlockCapacity(firstBlockCapacity),
4440 m_ItemBlocks(VmaStlAllocator<ItemBlock>(pAllocationCallbacks))
4441{
4442 VMA_ASSERT(m_FirstBlockCapacity > 1);
4443}
4444
4445template<typename T>
4446VmaPoolAllocator<T>::~VmaPoolAllocator()
4447{
4448 for (size_t i = m_ItemBlocks.size(); i--;)
4449 vma_delete_array(m_pAllocationCallbacks, m_ItemBlocks[i].pItems, m_ItemBlocks[i].Capacity);
4450 m_ItemBlocks.clear();
4451}
4452
4453template<typename T>
4454template<typename... Types> T* VmaPoolAllocator<T>::Alloc(Types&&... args)
4455{
4456 for (size_t i = m_ItemBlocks.size(); i--; )
4457 {
4458 ItemBlock& block = m_ItemBlocks[i];
4459 // This block has some free items: Use first one.
4460 if (block.FirstFreeIndex != UINT32_MAX)
4461 {
4462 Item* const pItem = &block.pItems[block.FirstFreeIndex];
4463 block.FirstFreeIndex = pItem->NextFreeIndex;
4464 T* result = (T*)&pItem->Value;
4465 new(result)T(std::forward<Types>(args)...); // Explicit constructor call.
4466 return result;
4467 }
4468 }
4469
4470 // No block has free item: Create new one and use it.
4471 ItemBlock& newBlock = CreateNewBlock();
4472 Item* const pItem = &newBlock.pItems[0];
4473 newBlock.FirstFreeIndex = pItem->NextFreeIndex;
4474 T* result = (T*)&pItem->Value;
4475 new(result) T(std::forward<Types>(args)...); // Explicit constructor call.
4476 return result;
4477}
4478
4479template<typename T>
4480void VmaPoolAllocator<T>::Free(T* ptr)
4481{
4482 // Search all memory blocks to find ptr.
4483 for (size_t i = m_ItemBlocks.size(); i--; )
4484 {
4485 ItemBlock& block = m_ItemBlocks[i];
4486
4487 // Casting to union.
4488 Item* pItemPtr;
4489 memcpy(&pItemPtr, &ptr, sizeof(pItemPtr));
4490
4491 // Check if pItemPtr is in address range of this block.
4492 if ((pItemPtr >= block.pItems) && (pItemPtr < block.pItems + block.Capacity))
4493 {
4494 ptr->~T(); // Explicit destructor call.
4495 const uint32_t index = static_cast<uint32_t>(pItemPtr - block.pItems);
4496 pItemPtr->NextFreeIndex = block.FirstFreeIndex;
4497 block.FirstFreeIndex = index;
4498 return;
4499 }
4500 }
4501 VMA_ASSERT(0 && "Pointer doesn't belong to this memory pool.");
4502}
4503
4504template<typename T>
4505typename VmaPoolAllocator<T>::ItemBlock& VmaPoolAllocator<T>::CreateNewBlock()
4506{
4507 const uint32_t newBlockCapacity = m_ItemBlocks.empty() ?
4508 m_FirstBlockCapacity : m_ItemBlocks.back().Capacity * 3 / 2;
4509
4510 const ItemBlock newBlock =
4511 {
4512 vma_new_array(m_pAllocationCallbacks, Item, newBlockCapacity),
4513 newBlockCapacity,
4514 0
4515 };
4516
4517 m_ItemBlocks.push_back(newBlock);
4518
4519 // Setup singly-linked list of all free items in this block.
4520 for (uint32_t i = 0; i < newBlockCapacity - 1; ++i)
4521 newBlock.pItems[i].NextFreeIndex = i + 1;
4522 newBlock.pItems[newBlockCapacity - 1].NextFreeIndex = UINT32_MAX;
4523 return m_ItemBlocks.back();
4524}
4525#endif // _VMA_POOL_ALLOCATOR_FUNCTIONS
4526#endif // _VMA_POOL_ALLOCATOR
4527
4528#ifndef _VMA_RAW_LIST
4529template<typename T>
4530struct VmaListItem
4531{
4532 VmaListItem* pPrev;
4533 VmaListItem* pNext;
4534 T Value;
4535};
4536
4537// Doubly linked list.
4538template<typename T>
4539class VmaRawList
4540{
4541 VMA_CLASS_NO_COPY(VmaRawList)
4542public:
4543 typedef VmaListItem<T> ItemType;
4544
4545 VmaRawList(const VkAllocationCallbacks* pAllocationCallbacks);
4546 // Intentionally not calling Clear, because that would be unnecessary
4547 // computations to return all items to m_ItemAllocator as free.
4548 ~VmaRawList() = default;
4549
4550 size_t GetCount() const { return m_Count; }
4551 bool IsEmpty() const { return m_Count == 0; }
4552
4553 ItemType* Front() { return m_pFront; }
4554 ItemType* Back() { return m_pBack; }
4555 const ItemType* Front() const { return m_pFront; }
4556 const ItemType* Back() const { return m_pBack; }
4557
4558 ItemType* PushFront();
4559 ItemType* PushBack();
4560 ItemType* PushFront(const T& value);
4561 ItemType* PushBack(const T& value);
4562 void PopFront();
4563 void PopBack();
4564
4565 // Item can be null - it means PushBack.
4566 ItemType* InsertBefore(ItemType* pItem);
4567 // Item can be null - it means PushFront.
4568 ItemType* InsertAfter(ItemType* pItem);
4569 ItemType* InsertBefore(ItemType* pItem, const T& value);
4570 ItemType* InsertAfter(ItemType* pItem, const T& value);
4571
4572 void Clear();
4573 void Remove(ItemType* pItem);
4574
4575private:
4576 const VkAllocationCallbacks* const m_pAllocationCallbacks;
4577 VmaPoolAllocator<ItemType> m_ItemAllocator;
4578 ItemType* m_pFront;
4579 ItemType* m_pBack;
4580 size_t m_Count;
4581};
4582
4583#ifndef _VMA_RAW_LIST_FUNCTIONS
4584template<typename T>
4585VmaRawList<T>::VmaRawList(const VkAllocationCallbacks* pAllocationCallbacks)
4586 : m_pAllocationCallbacks(pAllocationCallbacks),
4587 m_ItemAllocator(pAllocationCallbacks, 128),
4588 m_pFront(VMA_NULL),
4589 m_pBack(VMA_NULL),
4590 m_Count(0) {}
4591
4592template<typename T>
4593VmaListItem<T>* VmaRawList<T>::PushFront()
4594{
4595 ItemType* const pNewItem = m_ItemAllocator.Alloc();
4596 pNewItem->pPrev = VMA_NULL;
4597 if (IsEmpty())
4598 {
4599 pNewItem->pNext = VMA_NULL;
4600 m_pFront = pNewItem;
4601 m_pBack = pNewItem;
4602 m_Count = 1;
4603 }
4604 else
4605 {
4606 pNewItem->pNext = m_pFront;
4607 m_pFront->pPrev = pNewItem;
4608 m_pFront = pNewItem;
4609 ++m_Count;
4610 }
4611 return pNewItem;
4612}
4613
4614template<typename T>
4615VmaListItem<T>* VmaRawList<T>::PushBack()
4616{
4617 ItemType* const pNewItem = m_ItemAllocator.Alloc();
4618 pNewItem->pNext = VMA_NULL;
4619 if(IsEmpty())
4620 {
4621 pNewItem->pPrev = VMA_NULL;
4622 m_pFront = pNewItem;
4623 m_pBack = pNewItem;
4624 m_Count = 1;
4625 }
4626 else
4627 {
4628 pNewItem->pPrev = m_pBack;
4629 m_pBack->pNext = pNewItem;
4630 m_pBack = pNewItem;
4631 ++m_Count;
4632 }
4633 return pNewItem;
4634}
4635
4636template<typename T>
4637VmaListItem<T>* VmaRawList<T>::PushFront(const T& value)
4638{
4639 ItemType* const pNewItem = PushFront();
4640 pNewItem->Value = value;
4641 return pNewItem;
4642}
4643
4644template<typename T>
4645VmaListItem<T>* VmaRawList<T>::PushBack(const T& value)
4646{
4647 ItemType* const pNewItem = PushBack();
4648 pNewItem->Value = value;
4649 return pNewItem;
4650}
4651
4652template<typename T>
4653void VmaRawList<T>::PopFront()
4654{
4655 VMA_HEAVY_ASSERT(m_Count > 0);
4656 ItemType* const pFrontItem = m_pFront;
4657 ItemType* const pNextItem = pFrontItem->pNext;
4658 if (pNextItem != VMA_NULL)
4659 {
4660 pNextItem->pPrev = VMA_NULL;
4661 }
4662 m_pFront = pNextItem;
4663 m_ItemAllocator.Free(pFrontItem);
4664 --m_Count;
4665}
4666
4667template<typename T>
4668void VmaRawList<T>::PopBack()
4669{
4670 VMA_HEAVY_ASSERT(m_Count > 0);
4671 ItemType* const pBackItem = m_pBack;
4672 ItemType* const pPrevItem = pBackItem->pPrev;
4673 if(pPrevItem != VMA_NULL)
4674 {
4675 pPrevItem->pNext = VMA_NULL;
4676 }
4677 m_pBack = pPrevItem;
4678 m_ItemAllocator.Free(pBackItem);
4679 --m_Count;
4680}
4681
4682template<typename T>
4683void VmaRawList<T>::Clear()
4684{
4685 if (IsEmpty() == false)
4686 {
4687 ItemType* pItem = m_pBack;
4688 while (pItem != VMA_NULL)
4689 {
4690 ItemType* const pPrevItem = pItem->pPrev;
4691 m_ItemAllocator.Free(pItem);
4692 pItem = pPrevItem;
4693 }
4694 m_pFront = VMA_NULL;
4695 m_pBack = VMA_NULL;
4696 m_Count = 0;
4697 }
4698}
4699
4700template<typename T>
4701void VmaRawList<T>::Remove(ItemType* pItem)
4702{
4703 VMA_HEAVY_ASSERT(pItem != VMA_NULL);
4704 VMA_HEAVY_ASSERT(m_Count > 0);
4705
4706 if(pItem->pPrev != VMA_NULL)
4707 {
4708 pItem->pPrev->pNext = pItem->pNext;
4709 }
4710 else
4711 {
4712 VMA_HEAVY_ASSERT(m_pFront == pItem);
4713 m_pFront = pItem->pNext;
4714 }
4715
4716 if(pItem->pNext != VMA_NULL)
4717 {
4718 pItem->pNext->pPrev = pItem->pPrev;
4719 }
4720 else
4721 {
4722 VMA_HEAVY_ASSERT(m_pBack == pItem);
4723 m_pBack = pItem->pPrev;
4724 }
4725
4726 m_ItemAllocator.Free(pItem);
4727 --m_Count;
4728}
4729
4730template<typename T>
4731VmaListItem<T>* VmaRawList<T>::InsertBefore(ItemType* pItem)
4732{
4733 if(pItem != VMA_NULL)
4734 {
4735 ItemType* const prevItem = pItem->pPrev;
4736 ItemType* const newItem = m_ItemAllocator.Alloc();
4737 newItem->pPrev = prevItem;
4738 newItem->pNext = pItem;
4739 pItem->pPrev = newItem;
4740 if(prevItem != VMA_NULL)
4741 {
4742 prevItem->pNext = newItem;
4743 }
4744 else
4745 {
4746 VMA_HEAVY_ASSERT(m_pFront == pItem);
4747 m_pFront = newItem;
4748 }
4749 ++m_Count;
4750 return newItem;
4751 }
4752 else
4753 return PushBack();
4754}
4755
4756template<typename T>
4757VmaListItem<T>* VmaRawList<T>::InsertAfter(ItemType* pItem)
4758{
4759 if(pItem != VMA_NULL)
4760 {
4761 ItemType* const nextItem = pItem->pNext;
4762 ItemType* const newItem = m_ItemAllocator.Alloc();
4763 newItem->pNext = nextItem;
4764 newItem->pPrev = pItem;
4765 pItem->pNext = newItem;
4766 if(nextItem != VMA_NULL)
4767 {
4768 nextItem->pPrev = newItem;
4769 }
4770 else
4771 {
4772 VMA_HEAVY_ASSERT(m_pBack == pItem);
4773 m_pBack = newItem;
4774 }
4775 ++m_Count;
4776 return newItem;
4777 }
4778 else
4779 return PushFront();
4780}
4781
4782template<typename T>
4783VmaListItem<T>* VmaRawList<T>::InsertBefore(ItemType* pItem, const T& value)
4784{
4785 ItemType* const newItem = InsertBefore(pItem);
4786 newItem->Value = value;
4787 return newItem;
4788}
4789
4790template<typename T>
4791VmaListItem<T>* VmaRawList<T>::InsertAfter(ItemType* pItem, const T& value)
4792{
4793 ItemType* const newItem = InsertAfter(pItem);
4794 newItem->Value = value;
4795 return newItem;
4796}
4797#endif // _VMA_RAW_LIST_FUNCTIONS
4798#endif // _VMA_RAW_LIST
4799
4800#ifndef _VMA_LIST
4801template<typename T, typename AllocatorT>
4802class VmaList
4803{
4804 VMA_CLASS_NO_COPY(VmaList)
4805public:
4806 class reverse_iterator;
4807 class const_iterator;
4808 class const_reverse_iterator;
4809
4810 class iterator
4811 {
4812 friend class const_iterator;
4813 friend class VmaList<T, AllocatorT>;
4814 public:
4815 iterator() : m_pList(VMA_NULL), m_pItem(VMA_NULL) {}
4816 iterator(const reverse_iterator& src) : m_pList(src.m_pList), m_pItem(src.m_pItem) {}
4817
4818 T& operator*() const { VMA_HEAVY_ASSERT(m_pItem != VMA_NULL); return m_pItem->Value; }
4819 T* operator->() const { VMA_HEAVY_ASSERT(m_pItem != VMA_NULL); return &m_pItem->Value; }
4820
4821 bool operator==(const iterator& rhs) const { VMA_HEAVY_ASSERT(m_pList == rhs.m_pList); return m_pItem == rhs.m_pItem; }
4822 bool operator!=(const iterator& rhs) const { VMA_HEAVY_ASSERT(m_pList == rhs.m_pList); return m_pItem != rhs.m_pItem; }
4823
4824 iterator operator++(int) { iterator result = *this; ++*this; return result; }
4825 iterator operator--(int) { iterator result = *this; --*this; return result; }
4826
4827 iterator& operator++() { VMA_HEAVY_ASSERT(m_pItem != VMA_NULL); m_pItem = m_pItem->pNext; return *this; }
4828 iterator& operator--();
4829
4830 private:
4831 VmaRawList<T>* m_pList;
4832 VmaListItem<T>* m_pItem;
4833
4834 iterator(VmaRawList<T>* pList, VmaListItem<T>* pItem) : m_pList(pList), m_pItem(pItem) {}
4835 };
4836 class reverse_iterator
4837 {
4838 friend class const_reverse_iterator;
4839 friend class VmaList<T, AllocatorT>;
4840 public:
4841 reverse_iterator() : m_pList(VMA_NULL), m_pItem(VMA_NULL) {}
4842 reverse_iterator(const iterator& src) : m_pList(src.m_pList), m_pItem(src.m_pItem) {}
4843
4844 T& operator*() const { VMA_HEAVY_ASSERT(m_pItem != VMA_NULL); return m_pItem->Value; }
4845 T* operator->() const { VMA_HEAVY_ASSERT(m_pItem != VMA_NULL); return &m_pItem->Value; }
4846
4847 bool operator==(const reverse_iterator& rhs) const { VMA_HEAVY_ASSERT(m_pList == rhs.m_pList); return m_pItem == rhs.m_pItem; }
4848 bool operator!=(const reverse_iterator& rhs) const { VMA_HEAVY_ASSERT(m_pList == rhs.m_pList); return m_pItem != rhs.m_pItem; }
4849
4850 reverse_iterator operator++(int) { reverse_iterator result = *this; ++* this; return result; }
4851 reverse_iterator operator--(int) { reverse_iterator result = *this; --* this; return result; }
4852
4853 reverse_iterator& operator++() { VMA_HEAVY_ASSERT(m_pItem != VMA_NULL); m_pItem = m_pItem->pPrev; return *this; }
4854 reverse_iterator& operator--();
4855
4856 private:
4857 VmaRawList<T>* m_pList;
4858 VmaListItem<T>* m_pItem;
4859
4860 reverse_iterator(VmaRawList<T>* pList, VmaListItem<T>* pItem) : m_pList(pList), m_pItem(pItem) {}
4861 };
4862 class const_iterator
4863 {
4864 friend class VmaList<T, AllocatorT>;
4865 public:
4866 const_iterator() : m_pList(VMA_NULL), m_pItem(VMA_NULL) {}
4867 const_iterator(const iterator& src) : m_pList(src.m_pList), m_pItem(src.m_pItem) {}
4868 const_iterator(const reverse_iterator& src) : m_pList(src.m_pList), m_pItem(src.m_pItem) {}
4869
4870 iterator drop_const() { return { const_cast<VmaRawList<T>*>(m_pList), const_cast<VmaListItem<T>*>(m_pItem) }; }
4871
4872 const T& operator*() const { VMA_HEAVY_ASSERT(m_pItem != VMA_NULL); return m_pItem->Value; }
4873 const T* operator->() const { VMA_HEAVY_ASSERT(m_pItem != VMA_NULL); return &m_pItem->Value; }
4874
4875 bool operator==(const const_iterator& rhs) const { VMA_HEAVY_ASSERT(m_pList == rhs.m_pList); return m_pItem == rhs.m_pItem; }
4876 bool operator!=(const const_iterator& rhs) const { VMA_HEAVY_ASSERT(m_pList == rhs.m_pList); return m_pItem != rhs.m_pItem; }
4877
4878 const_iterator operator++(int) { const_iterator result = *this; ++* this; return result; }
4879 const_iterator operator--(int) { const_iterator result = *this; --* this; return result; }
4880
4881 const_iterator& operator++() { VMA_HEAVY_ASSERT(m_pItem != VMA_NULL); m_pItem = m_pItem->pNext; return *this; }
4882 const_iterator& operator--();
4883
4884 private:
4885 const VmaRawList<T>* m_pList;
4886 const VmaListItem<T>* m_pItem;
4887
4888 const_iterator(const VmaRawList<T>* pList, const VmaListItem<T>* pItem) : m_pList(pList), m_pItem(pItem) {}
4889 };
4890 class const_reverse_iterator
4891 {
4892 friend class VmaList<T, AllocatorT>;
4893 public:
4894 const_reverse_iterator() : m_pList(VMA_NULL), m_pItem(VMA_NULL) {}
4895 const_reverse_iterator(const reverse_iterator& src) : m_pList(src.m_pList), m_pItem(src.m_pItem) {}
4896 const_reverse_iterator(const iterator& src) : m_pList(src.m_pList), m_pItem(src.m_pItem) {}
4897
4898 reverse_iterator drop_const() { return { const_cast<VmaRawList<T>*>(m_pList), const_cast<VmaListItem<T>*>(m_pItem) }; }
4899
4900 const T& operator*() const { VMA_HEAVY_ASSERT(m_pItem != VMA_NULL); return m_pItem->Value; }
4901 const T* operator->() const { VMA_HEAVY_ASSERT(m_pItem != VMA_NULL); return &m_pItem->Value; }
4902
4903 bool operator==(const const_reverse_iterator& rhs) const { VMA_HEAVY_ASSERT(m_pList == rhs.m_pList); return m_pItem == rhs.m_pItem; }
4904 bool operator!=(const const_reverse_iterator& rhs) const { VMA_HEAVY_ASSERT(m_pList == rhs.m_pList); return m_pItem != rhs.m_pItem; }
4905
4906 const_reverse_iterator operator++(int) { const_reverse_iterator result = *this; ++* this; return result; }
4907 const_reverse_iterator operator--(int) { const_reverse_iterator result = *this; --* this; return result; }
4908
4909 const_reverse_iterator& operator++() { VMA_HEAVY_ASSERT(m_pItem != VMA_NULL); m_pItem = m_pItem->pPrev; return *this; }
4910 const_reverse_iterator& operator--();
4911
4912 private:
4913 const VmaRawList<T>* m_pList;
4914 const VmaListItem<T>* m_pItem;
4915
4916 const_reverse_iterator(const VmaRawList<T>* pList, const VmaListItem<T>* pItem) : m_pList(pList), m_pItem(pItem) {}
4917 };
4918
4919 VmaList(const AllocatorT& allocator) : m_RawList(allocator.m_pCallbacks) {}
4920
4921 bool empty() const { return m_RawList.IsEmpty(); }
4922 size_t size() const { return m_RawList.GetCount(); }
4923
4924 iterator begin() { return iterator(&m_RawList, m_RawList.Front()); }
4925 iterator end() { return iterator(&m_RawList, VMA_NULL); }
4926
4927 const_iterator cbegin() const { return const_iterator(&m_RawList, m_RawList.Front()); }
4928 const_iterator cend() const { return const_iterator(&m_RawList, VMA_NULL); }
4929
4930 const_iterator begin() const { return cbegin(); }
4931 const_iterator end() const { return cend(); }
4932
4933 reverse_iterator rbegin() { return reverse_iterator(&m_RawList, m_RawList.Back()); }
4934 reverse_iterator rend() { return reverse_iterator(&m_RawList, VMA_NULL); }
4935
4936 const_reverse_iterator crbegin() const { return const_reverse_iterator(&m_RawList, m_RawList.Back()); }
4937 const_reverse_iterator crend() const { return const_reverse_iterator(&m_RawList, VMA_NULL); }
4938
4939 const_reverse_iterator rbegin() const { return crbegin(); }
4940 const_reverse_iterator rend() const { return crend(); }
4941
4942 void push_back(const T& value) { m_RawList.PushBack(value); }
4943 iterator insert(iterator it, const T& value) { return iterator(&m_RawList, m_RawList.InsertBefore(it.m_pItem, value)); }
4944
4945 void clear() { m_RawList.Clear(); }
4946 void erase(iterator it) { m_RawList.Remove(it.m_pItem); }
4947
4948private:
4949 VmaRawList<T> m_RawList;
4950};
4951
4952#ifndef _VMA_LIST_FUNCTIONS
4953template<typename T, typename AllocatorT>
4954typename VmaList<T, AllocatorT>::iterator& VmaList<T, AllocatorT>::iterator::operator--()
4955{
4956 if (m_pItem != VMA_NULL)
4957 {
4958 m_pItem = m_pItem->pPrev;
4959 }
4960 else
4961 {
4962 VMA_HEAVY_ASSERT(!m_pList->IsEmpty());
4963 m_pItem = m_pList->Back();
4964 }
4965 return *this;
4966}
4967
4968template<typename T, typename AllocatorT>
4969typename VmaList<T, AllocatorT>::reverse_iterator& VmaList<T, AllocatorT>::reverse_iterator::operator--()
4970{
4971 if (m_pItem != VMA_NULL)
4972 {
4973 m_pItem = m_pItem->pNext;
4974 }
4975 else
4976 {
4977 VMA_HEAVY_ASSERT(!m_pList->IsEmpty());
4978 m_pItem = m_pList->Front();
4979 }
4980 return *this;
4981}
4982
4983template<typename T, typename AllocatorT>
4984typename VmaList<T, AllocatorT>::const_iterator& VmaList<T, AllocatorT>::const_iterator::operator--()
4985{
4986 if (m_pItem != VMA_NULL)
4987 {
4988 m_pItem = m_pItem->pPrev;
4989 }
4990 else
4991 {
4992 VMA_HEAVY_ASSERT(!m_pList->IsEmpty());
4993 m_pItem = m_pList->Back();
4994 }
4995 return *this;
4996}
4997
4998template<typename T, typename AllocatorT>
4999typename VmaList<T, AllocatorT>::const_reverse_iterator& VmaList<T, AllocatorT>::const_reverse_iterator::operator--()
5000{
5001 if (m_pItem != VMA_NULL)
5002 {
5003 m_pItem = m_pItem->pNext;
5004 }
5005 else
5006 {
5007 VMA_HEAVY_ASSERT(!m_pList->IsEmpty());
5008 m_pItem = m_pList->Back();
5009 }
5010 return *this;
5011}
5012#endif // _VMA_LIST_FUNCTIONS
5013#endif // _VMA_LIST
5014
5015#ifndef _VMA_INTRUSIVE_LINKED_LIST
5016/*
5017Expected interface of ItemTypeTraits:
5018struct MyItemTypeTraits
5019{
5020 typedef MyItem ItemType;
5021 static ItemType* GetPrev(const ItemType* item) { return item->myPrevPtr; }
5022 static ItemType* GetNext(const ItemType* item) { return item->myNextPtr; }
5023 static ItemType*& AccessPrev(ItemType* item) { return item->myPrevPtr; }
5024 static ItemType*& AccessNext(ItemType* item) { return item->myNextPtr; }
5025};
5026*/
5027template<typename ItemTypeTraits>
5028class VmaIntrusiveLinkedList
5029{
5030public:
5031 typedef typename ItemTypeTraits::ItemType ItemType;
5032 static ItemType* GetPrev(const ItemType* item) { return ItemTypeTraits::GetPrev(item); }
5033 static ItemType* GetNext(const ItemType* item) { return ItemTypeTraits::GetNext(item); }
5034
5035 // Movable, not copyable.
5036 VmaIntrusiveLinkedList() = default;
5037 VmaIntrusiveLinkedList(VmaIntrusiveLinkedList && src);
5038 VmaIntrusiveLinkedList(const VmaIntrusiveLinkedList&) = delete;
5039 VmaIntrusiveLinkedList& operator=(VmaIntrusiveLinkedList&& src);
5040 VmaIntrusiveLinkedList& operator=(const VmaIntrusiveLinkedList&) = delete;
5041 ~VmaIntrusiveLinkedList() { VMA_HEAVY_ASSERT(IsEmpty()); }
5042
5043 size_t GetCount() const { return m_Count; }
5044 bool IsEmpty() const { return m_Count == 0; }
5045 ItemType* Front() { return m_Front; }
5046 ItemType* Back() { return m_Back; }
5047 const ItemType* Front() const { return m_Front; }
5048 const ItemType* Back() const { return m_Back; }
5049
5050 void PushBack(ItemType* item);
5051 void PushFront(ItemType* item);
5052 ItemType* PopBack();
5053 ItemType* PopFront();
5054
5055 // MyItem can be null - it means PushBack.
5056 void InsertBefore(ItemType* existingItem, ItemType* newItem);
5057 // MyItem can be null - it means PushFront.
5058 void InsertAfter(ItemType* existingItem, ItemType* newItem);
5059 void Remove(ItemType* item);
5060 void RemoveAll();
5061
5062private:
5063 ItemType* m_Front = VMA_NULL;
5064 ItemType* m_Back = VMA_NULL;
5065 size_t m_Count = 0;
5066};
5067
5068#ifndef _VMA_INTRUSIVE_LINKED_LIST_FUNCTIONS
5069template<typename ItemTypeTraits>
5070VmaIntrusiveLinkedList<ItemTypeTraits>::VmaIntrusiveLinkedList(VmaIntrusiveLinkedList&& src)
5071 : m_Front(src.m_Front), m_Back(src.m_Back), m_Count(src.m_Count)
5072{
5073 src.m_Front = src.m_Back = VMA_NULL;
5074 src.m_Count = 0;
5075}
5076
5077template<typename ItemTypeTraits>
5078VmaIntrusiveLinkedList<ItemTypeTraits>& VmaIntrusiveLinkedList<ItemTypeTraits>::operator=(VmaIntrusiveLinkedList&& src)
5079{
5080 if (&src != this)
5081 {
5082 VMA_HEAVY_ASSERT(IsEmpty());
5083 m_Front = src.m_Front;
5084 m_Back = src.m_Back;
5085 m_Count = src.m_Count;
5086 src.m_Front = src.m_Back = VMA_NULL;
5087 src.m_Count = 0;
5088 }
5089 return *this;
5090}
5091
5092template<typename ItemTypeTraits>
5093void VmaIntrusiveLinkedList<ItemTypeTraits>::PushBack(ItemType* item)
5094{
5095 VMA_HEAVY_ASSERT(ItemTypeTraits::GetPrev(item) == VMA_NULL && ItemTypeTraits::GetNext(item) == VMA_NULL);
5096 if (IsEmpty())
5097 {
5098 m_Front = item;
5099 m_Back = item;
5100 m_Count = 1;
5101 }
5102 else
5103 {
5104 ItemTypeTraits::AccessPrev(item) = m_Back;
5105 ItemTypeTraits::AccessNext(m_Back) = item;
5106 m_Back = item;
5107 ++m_Count;
5108 }
5109}
5110
5111template<typename ItemTypeTraits>
5112void VmaIntrusiveLinkedList<ItemTypeTraits>::PushFront(ItemType* item)
5113{
5114 VMA_HEAVY_ASSERT(ItemTypeTraits::GetPrev(item) == VMA_NULL && ItemTypeTraits::GetNext(item) == VMA_NULL);
5115 if (IsEmpty())
5116 {
5117 m_Front = item;
5118 m_Back = item;
5119 m_Count = 1;
5120 }
5121 else
5122 {
5123 ItemTypeTraits::AccessNext(item) = m_Front;
5124 ItemTypeTraits::AccessPrev(m_Front) = item;
5125 m_Front = item;
5126 ++m_Count;
5127 }
5128}
5129
5130template<typename ItemTypeTraits>
5131typename VmaIntrusiveLinkedList<ItemTypeTraits>::ItemType* VmaIntrusiveLinkedList<ItemTypeTraits>::PopBack()
5132{
5133 VMA_HEAVY_ASSERT(m_Count > 0);
5134 ItemType* const backItem = m_Back;
5135 ItemType* const prevItem = ItemTypeTraits::GetPrev(backItem);
5136 if (prevItem != VMA_NULL)
5137 {
5138 ItemTypeTraits::AccessNext(prevItem) = VMA_NULL;
5139 }
5140 m_Back = prevItem;
5141 --m_Count;
5142 ItemTypeTraits::AccessPrev(backItem) = VMA_NULL;
5143 ItemTypeTraits::AccessNext(backItem) = VMA_NULL;
5144 return backItem;
5145}
5146
5147template<typename ItemTypeTraits>
5148typename VmaIntrusiveLinkedList<ItemTypeTraits>::ItemType* VmaIntrusiveLinkedList<ItemTypeTraits>::PopFront()
5149{
5150 VMA_HEAVY_ASSERT(m_Count > 0);
5151 ItemType* const frontItem = m_Front;
5152 ItemType* const nextItem = ItemTypeTraits::GetNext(frontItem);
5153 if (nextItem != VMA_NULL)
5154 {
5155 ItemTypeTraits::AccessPrev(nextItem) = VMA_NULL;
5156 }
5157 m_Front = nextItem;
5158 --m_Count;
5159 ItemTypeTraits::AccessPrev(frontItem) = VMA_NULL;
5160 ItemTypeTraits::AccessNext(frontItem) = VMA_NULL;
5161 return frontItem;
5162}
5163
5164template<typename ItemTypeTraits>
5165void VmaIntrusiveLinkedList<ItemTypeTraits>::InsertBefore(ItemType* existingItem, ItemType* newItem)
5166{
5167 VMA_HEAVY_ASSERT(newItem != VMA_NULL && ItemTypeTraits::GetPrev(newItem) == VMA_NULL && ItemTypeTraits::GetNext(newItem) == VMA_NULL);
5168 if (existingItem != VMA_NULL)
5169 {
5170 ItemType* const prevItem = ItemTypeTraits::GetPrev(existingItem);
5171 ItemTypeTraits::AccessPrev(newItem) = prevItem;
5172 ItemTypeTraits::AccessNext(newItem) = existingItem;
5173 ItemTypeTraits::AccessPrev(existingItem) = newItem;
5174 if (prevItem != VMA_NULL)
5175 {
5176 ItemTypeTraits::AccessNext(prevItem) = newItem;
5177 }
5178 else
5179 {
5180 VMA_HEAVY_ASSERT(m_Front == existingItem);
5181 m_Front = newItem;
5182 }
5183 ++m_Count;
5184 }
5185 else
5186 PushBack(item: newItem);
5187}
5188
5189template<typename ItemTypeTraits>
5190void VmaIntrusiveLinkedList<ItemTypeTraits>::InsertAfter(ItemType* existingItem, ItemType* newItem)
5191{
5192 VMA_HEAVY_ASSERT(newItem != VMA_NULL && ItemTypeTraits::GetPrev(newItem) == VMA_NULL && ItemTypeTraits::GetNext(newItem) == VMA_NULL);
5193 if (existingItem != VMA_NULL)
5194 {
5195 ItemType* const nextItem = ItemTypeTraits::GetNext(existingItem);
5196 ItemTypeTraits::AccessNext(newItem) = nextItem;
5197 ItemTypeTraits::AccessPrev(newItem) = existingItem;
5198 ItemTypeTraits::AccessNext(existingItem) = newItem;
5199 if (nextItem != VMA_NULL)
5200 {
5201 ItemTypeTraits::AccessPrev(nextItem) = newItem;
5202 }
5203 else
5204 {
5205 VMA_HEAVY_ASSERT(m_Back == existingItem);
5206 m_Back = newItem;
5207 }
5208 ++m_Count;
5209 }
5210 else
5211 return PushFront(item: newItem);
5212}
5213
5214template<typename ItemTypeTraits>
5215void VmaIntrusiveLinkedList<ItemTypeTraits>::Remove(ItemType* item)
5216{
5217 VMA_HEAVY_ASSERT(item != VMA_NULL && m_Count > 0);
5218 if (ItemTypeTraits::GetPrev(item) != VMA_NULL)
5219 {
5220 ItemTypeTraits::AccessNext(ItemTypeTraits::AccessPrev(item)) = ItemTypeTraits::GetNext(item);
5221 }
5222 else
5223 {
5224 VMA_HEAVY_ASSERT(m_Front == item);
5225 m_Front = ItemTypeTraits::GetNext(item);
5226 }
5227
5228 if (ItemTypeTraits::GetNext(item) != VMA_NULL)
5229 {
5230 ItemTypeTraits::AccessPrev(ItemTypeTraits::AccessNext(item)) = ItemTypeTraits::GetPrev(item);
5231 }
5232 else
5233 {
5234 VMA_HEAVY_ASSERT(m_Back == item);
5235 m_Back = ItemTypeTraits::GetPrev(item);
5236 }
5237 ItemTypeTraits::AccessPrev(item) = VMA_NULL;
5238 ItemTypeTraits::AccessNext(item) = VMA_NULL;
5239 --m_Count;
5240}
5241
5242template<typename ItemTypeTraits>
5243void VmaIntrusiveLinkedList<ItemTypeTraits>::RemoveAll()
5244{
5245 if (!IsEmpty())
5246 {
5247 ItemType* item = m_Back;
5248 while (item != VMA_NULL)
5249 {
5250 ItemType* const prevItem = ItemTypeTraits::AccessPrev(item);
5251 ItemTypeTraits::AccessPrev(item) = VMA_NULL;
5252 ItemTypeTraits::AccessNext(item) = VMA_NULL;
5253 item = prevItem;
5254 }
5255 m_Front = VMA_NULL;
5256 m_Back = VMA_NULL;
5257 m_Count = 0;
5258 }
5259}
5260#endif // _VMA_INTRUSIVE_LINKED_LIST_FUNCTIONS
5261#endif // _VMA_INTRUSIVE_LINKED_LIST
5262
5263// Unused in this version.
5264#if 0
5265
5266#ifndef _VMA_PAIR
5267template<typename T1, typename T2>
5268struct VmaPair
5269{
5270 T1 first;
5271 T2 second;
5272
5273 VmaPair() : first(), second() {}
5274 VmaPair(const T1& firstSrc, const T2& secondSrc) : first(firstSrc), second(secondSrc) {}
5275};
5276
5277template<typename FirstT, typename SecondT>
5278struct VmaPairFirstLess
5279{
5280 bool operator()(const VmaPair<FirstT, SecondT>& lhs, const VmaPair<FirstT, SecondT>& rhs) const
5281 {
5282 return lhs.first < rhs.first;
5283 }
5284 bool operator()(const VmaPair<FirstT, SecondT>& lhs, const FirstT& rhsFirst) const
5285 {
5286 return lhs.first < rhsFirst;
5287 }
5288};
5289#endif // _VMA_PAIR
5290
5291#ifndef _VMA_MAP
5292/* Class compatible with subset of interface of std::unordered_map.
5293KeyT, ValueT must be POD because they will be stored in VmaVector.
5294*/
5295template<typename KeyT, typename ValueT>
5296class VmaMap
5297{
5298public:
5299 typedef VmaPair<KeyT, ValueT> PairType;
5300 typedef PairType* iterator;
5301
5302 VmaMap(const VmaStlAllocator<PairType>& allocator) : m_Vector(allocator) {}
5303
5304 iterator begin() { return m_Vector.begin(); }
5305 iterator end() { return m_Vector.end(); }
5306 size_t size() { return m_Vector.size(); }
5307
5308 void insert(const PairType& pair);
5309 iterator find(const KeyT& key);
5310 void erase(iterator it);
5311
5312private:
5313 VmaVector< PairType, VmaStlAllocator<PairType>> m_Vector;
5314};
5315
5316#ifndef _VMA_MAP_FUNCTIONS
5317template<typename KeyT, typename ValueT>
5318void VmaMap<KeyT, ValueT>::insert(const PairType& pair)
5319{
5320 const size_t indexToInsert = VmaBinaryFindFirstNotLess(
5321 m_Vector.data(),
5322 m_Vector.data() + m_Vector.size(),
5323 pair,
5324 VmaPairFirstLess<KeyT, ValueT>()) - m_Vector.data();
5325 VmaVectorInsert(m_Vector, indexToInsert, pair);
5326}
5327
5328template<typename KeyT, typename ValueT>
5329VmaPair<KeyT, ValueT>* VmaMap<KeyT, ValueT>::find(const KeyT& key)
5330{
5331 PairType* it = VmaBinaryFindFirstNotLess(
5332 m_Vector.data(),
5333 m_Vector.data() + m_Vector.size(),
5334 key,
5335 VmaPairFirstLess<KeyT, ValueT>());
5336 if ((it != m_Vector.end()) && (it->first == key))
5337 {
5338 return it;
5339 }
5340 else
5341 {
5342 return m_Vector.end();
5343 }
5344}
5345
5346template<typename KeyT, typename ValueT>
5347void VmaMap<KeyT, ValueT>::erase(iterator it)
5348{
5349 VmaVectorRemove(m_Vector, it - m_Vector.begin());
5350}
5351#endif // _VMA_MAP_FUNCTIONS
5352#endif // _VMA_MAP
5353
5354#endif // #if 0
5355
5356#if !defined(_VMA_STRING_BUILDER) && VMA_STATS_STRING_ENABLED
5357class VmaStringBuilder
5358{
5359public:
5360 VmaStringBuilder(const VkAllocationCallbacks* allocationCallbacks) : m_Data(VmaStlAllocator<char>(allocationCallbacks)) {}
5361 ~VmaStringBuilder() = default;
5362
5363 size_t GetLength() const { return m_Data.size(); }
5364 const char* GetData() const { return m_Data.data(); }
5365 void AddNewLine() { Add(ch: '\n'); }
5366 void Add(char ch) { m_Data.push_back(src: ch); }
5367
5368 void Add(const char* pStr);
5369 void AddNumber(uint32_t num);
5370 void AddNumber(uint64_t num);
5371 void AddPointer(const void* ptr);
5372
5373private:
5374 VmaVector<char, VmaStlAllocator<char>> m_Data;
5375};
5376
5377#ifndef _VMA_STRING_BUILDER_FUNCTIONS
5378void VmaStringBuilder::Add(const char* pStr)
5379{
5380 const size_t strLen = strlen(s: pStr);
5381 if (strLen > 0)
5382 {
5383 const size_t oldCount = m_Data.size();
5384 m_Data.resize(newCount: oldCount + strLen);
5385 memcpy(dest: m_Data.data() + oldCount, src: pStr, n: strLen);
5386 }
5387}
5388
5389void VmaStringBuilder::AddNumber(uint32_t num)
5390{
5391 char buf[11];
5392 buf[10] = '\0';
5393 char* p = &buf[10];
5394 do
5395 {
5396 *--p = '0' + (num % 10);
5397 num /= 10;
5398 } while (num);
5399 Add(pStr: p);
5400}
5401
5402void VmaStringBuilder::AddNumber(uint64_t num)
5403{
5404 char buf[21];
5405 buf[20] = '\0';
5406 char* p = &buf[20];
5407 do
5408 {
5409 *--p = '0' + (num % 10);
5410 num /= 10;
5411 } while (num);
5412 Add(pStr: p);
5413}
5414
5415void VmaStringBuilder::AddPointer(const void* ptr)
5416{
5417 char buf[21];
5418 VmaPtrToStr(outStr: buf, strLen: sizeof(buf), ptr);
5419 Add(pStr: buf);
5420}
5421#endif //_VMA_STRING_BUILDER_FUNCTIONS
5422#endif // _VMA_STRING_BUILDER
5423
5424#if !defined(_VMA_JSON_WRITER) && VMA_STATS_STRING_ENABLED
5425/*
5426Allows to conveniently build a correct JSON document to be written to the
5427VmaStringBuilder passed to the constructor.
5428*/
5429class VmaJsonWriter
5430{
5431 VMA_CLASS_NO_COPY(VmaJsonWriter)
5432public:
5433 // sb - string builder to write the document to. Must remain alive for the whole lifetime of this object.
5434 VmaJsonWriter(const VkAllocationCallbacks* pAllocationCallbacks, VmaStringBuilder& sb);
5435 ~VmaJsonWriter();
5436
5437 // Begins object by writing "{".
5438 // Inside an object, you must call pairs of WriteString and a value, e.g.:
5439 // j.BeginObject(true); j.WriteString("A"); j.WriteNumber(1); j.WriteString("B"); j.WriteNumber(2); j.EndObject();
5440 // Will write: { "A": 1, "B": 2 }
5441 void BeginObject(bool singleLine = false);
5442 // Ends object by writing "}".
5443 void EndObject();
5444
5445 // Begins array by writing "[".
5446 // Inside an array, you can write a sequence of any values.
5447 void BeginArray(bool singleLine = false);
5448 // Ends array by writing "[".
5449 void EndArray();
5450
5451 // Writes a string value inside "".
5452 // pStr can contain any ANSI characters, including '"', new line etc. - they will be properly escaped.
5453 void WriteString(const char* pStr);
5454
5455 // Begins writing a string value.
5456 // Call BeginString, ContinueString, ContinueString, ..., EndString instead of
5457 // WriteString to conveniently build the string content incrementally, made of
5458 // parts including numbers.
5459 void BeginString(const char* pStr = VMA_NULL);
5460 // Posts next part of an open string.
5461 void ContinueString(const char* pStr);
5462 // Posts next part of an open string. The number is converted to decimal characters.
5463 void ContinueString(uint32_t n);
5464 void ContinueString(uint64_t n);
5465 void ContinueString_Size(size_t n);
5466 // Posts next part of an open string. Pointer value is converted to characters
5467 // using "%p" formatting - shown as hexadecimal number, e.g.: 000000081276Ad00
5468 void ContinueString_Pointer(const void* ptr);
5469 // Ends writing a string value by writing '"'.
5470 void EndString(const char* pStr = VMA_NULL);
5471
5472 // Writes a number value.
5473 void WriteNumber(uint32_t n);
5474 void WriteNumber(uint64_t n);
5475 void WriteSize(size_t n);
5476 // Writes a boolean value - false or true.
5477 void WriteBool(bool b);
5478 // Writes a null value.
5479 void WriteNull();
5480
5481private:
5482 enum COLLECTION_TYPE
5483 {
5484 COLLECTION_TYPE_OBJECT,
5485 COLLECTION_TYPE_ARRAY,
5486 };
5487 struct StackItem
5488 {
5489 COLLECTION_TYPE type;
5490 uint32_t valueCount;
5491 bool singleLineMode;
5492 };
5493
5494 static const char* const INDENT;
5495
5496 VmaStringBuilder& m_SB;
5497 VmaVector< StackItem, VmaStlAllocator<StackItem> > m_Stack;
5498 bool m_InsideString;
5499
5500 // Write size_t for less than 64bits
5501 void WriteSize(size_t n, std::integral_constant<bool, false>) { m_SB.AddNumber(num: static_cast<uint32_t>(n)); }
5502 // Write size_t for 64bits
5503 void WriteSize(size_t n, std::integral_constant<bool, true>) { m_SB.AddNumber(num: static_cast<uint64_t>(n)); }
5504
5505 void BeginValue(bool isString);
5506 void WriteIndent(bool oneLess = false);
5507};
5508const char* const VmaJsonWriter::INDENT = " ";
5509
5510#ifndef _VMA_JSON_WRITER_FUNCTIONS
5511VmaJsonWriter::VmaJsonWriter(const VkAllocationCallbacks* pAllocationCallbacks, VmaStringBuilder& sb)
5512 : m_SB(sb),
5513 m_Stack(VmaStlAllocator<StackItem>(pAllocationCallbacks)),
5514 m_InsideString(false) {}
5515
5516VmaJsonWriter::~VmaJsonWriter()
5517{
5518 VMA_ASSERT(!m_InsideString);
5519 VMA_ASSERT(m_Stack.empty());
5520}
5521
5522void VmaJsonWriter::BeginObject(bool singleLine)
5523{
5524 VMA_ASSERT(!m_InsideString);
5525
5526 BeginValue(isString: false);
5527 m_SB.Add(ch: '{');
5528
5529 StackItem item;
5530 item.type = COLLECTION_TYPE_OBJECT;
5531 item.valueCount = 0;
5532 item.singleLineMode = singleLine;
5533 m_Stack.push_back(src: item);
5534}
5535
5536void VmaJsonWriter::EndObject()
5537{
5538 VMA_ASSERT(!m_InsideString);
5539
5540 WriteIndent(oneLess: true);
5541 m_SB.Add(ch: '}');
5542
5543 VMA_ASSERT(!m_Stack.empty() && m_Stack.back().type == COLLECTION_TYPE_OBJECT);
5544 m_Stack.pop_back();
5545}
5546
5547void VmaJsonWriter::BeginArray(bool singleLine)
5548{
5549 VMA_ASSERT(!m_InsideString);
5550
5551 BeginValue(isString: false);
5552 m_SB.Add(ch: '[');
5553
5554 StackItem item;
5555 item.type = COLLECTION_TYPE_ARRAY;
5556 item.valueCount = 0;
5557 item.singleLineMode = singleLine;
5558 m_Stack.push_back(src: item);
5559}
5560
5561void VmaJsonWriter::EndArray()
5562{
5563 VMA_ASSERT(!m_InsideString);
5564
5565 WriteIndent(oneLess: true);
5566 m_SB.Add(ch: ']');
5567
5568 VMA_ASSERT(!m_Stack.empty() && m_Stack.back().type == COLLECTION_TYPE_ARRAY);
5569 m_Stack.pop_back();
5570}
5571
5572void VmaJsonWriter::WriteString(const char* pStr)
5573{
5574 BeginString(pStr);
5575 EndString();
5576}
5577
5578void VmaJsonWriter::BeginString(const char* pStr)
5579{
5580 VMA_ASSERT(!m_InsideString);
5581
5582 BeginValue(isString: true);
5583 m_SB.Add(ch: '"');
5584 m_InsideString = true;
5585 if (pStr != VMA_NULL && pStr[0] != '\0')
5586 {
5587 ContinueString(pStr);
5588 }
5589}
5590
5591void VmaJsonWriter::ContinueString(const char* pStr)
5592{
5593 VMA_ASSERT(m_InsideString);
5594
5595 const size_t strLen = strlen(s: pStr);
5596 for (size_t i = 0; i < strLen; ++i)
5597 {
5598 char ch = pStr[i];
5599 if (ch == '\\')
5600 {
5601 m_SB.Add(pStr: "\\\\");
5602 }
5603 else if (ch == '"')
5604 {
5605 m_SB.Add(pStr: "\\\"");
5606 }
5607 else if (ch >= 32)
5608 {
5609 m_SB.Add(ch);
5610 }
5611 else switch (ch)
5612 {
5613 case '\b':
5614 m_SB.Add(pStr: "\\b");
5615 break;
5616 case '\f':
5617 m_SB.Add(pStr: "\\f");
5618 break;
5619 case '\n':
5620 m_SB.Add(pStr: "\\n");
5621 break;
5622 case '\r':
5623 m_SB.Add(pStr: "\\r");
5624 break;
5625 case '\t':
5626 m_SB.Add(pStr: "\\t");
5627 break;
5628 default:
5629 VMA_ASSERT(0 && "Character not currently supported.");
5630 break;
5631 }
5632 }
5633}
5634
5635void VmaJsonWriter::ContinueString(uint32_t n)
5636{
5637 VMA_ASSERT(m_InsideString);
5638 m_SB.AddNumber(num: n);
5639}
5640
5641void VmaJsonWriter::ContinueString(uint64_t n)
5642{
5643 VMA_ASSERT(m_InsideString);
5644 m_SB.AddNumber(num: n);
5645}
5646
5647void VmaJsonWriter::ContinueString_Size(size_t n)
5648{
5649 VMA_ASSERT(m_InsideString);
5650 // Fix for AppleClang incorrect type casting
5651 // TODO: Change to if constexpr when C++17 used as minimal standard
5652 WriteSize(n, std::is_same<size_t, uint64_t>{});
5653}
5654
5655void VmaJsonWriter::ContinueString_Pointer(const void* ptr)
5656{
5657 VMA_ASSERT(m_InsideString);
5658 m_SB.AddPointer(ptr);
5659}
5660
5661void VmaJsonWriter::EndString(const char* pStr)
5662{
5663 VMA_ASSERT(m_InsideString);
5664 if (pStr != VMA_NULL && pStr[0] != '\0')
5665 {
5666 ContinueString(pStr);
5667 }
5668 m_SB.Add(ch: '"');
5669 m_InsideString = false;
5670}
5671
5672void VmaJsonWriter::WriteNumber(uint32_t n)
5673{
5674 VMA_ASSERT(!m_InsideString);
5675 BeginValue(isString: false);
5676 m_SB.AddNumber(num: n);
5677}
5678
5679void VmaJsonWriter::WriteNumber(uint64_t n)
5680{
5681 VMA_ASSERT(!m_InsideString);
5682 BeginValue(isString: false);
5683 m_SB.AddNumber(num: n);
5684}
5685
5686void VmaJsonWriter::WriteSize(size_t n)
5687{
5688 VMA_ASSERT(!m_InsideString);
5689 BeginValue(isString: false);
5690 // Fix for AppleClang incorrect type casting
5691 // TODO: Change to if constexpr when C++17 used as minimal standard
5692 WriteSize(n, std::is_same<size_t, uint64_t>{});
5693}
5694
5695void VmaJsonWriter::WriteBool(bool b)
5696{
5697 VMA_ASSERT(!m_InsideString);
5698 BeginValue(isString: false);
5699 m_SB.Add(pStr: b ? "true" : "false");
5700}
5701
5702void VmaJsonWriter::WriteNull()
5703{
5704 VMA_ASSERT(!m_InsideString);
5705 BeginValue(isString: false);
5706 m_SB.Add(pStr: "null");
5707}
5708
5709void VmaJsonWriter::BeginValue(bool isString)
5710{
5711 if (!m_Stack.empty())
5712 {
5713 StackItem& currItem = m_Stack.back();
5714 if (currItem.type == COLLECTION_TYPE_OBJECT &&
5715 currItem.valueCount % 2 == 0)
5716 {
5717 VMA_ASSERT(isString);
5718 }
5719
5720 if (currItem.type == COLLECTION_TYPE_OBJECT &&
5721 currItem.valueCount % 2 != 0)
5722 {
5723 m_SB.Add(pStr: ": ");
5724 }
5725 else if (currItem.valueCount > 0)
5726 {
5727 m_SB.Add(pStr: ", ");
5728 WriteIndent();
5729 }
5730 else
5731 {
5732 WriteIndent();
5733 }
5734 ++currItem.valueCount;
5735 }
5736}
5737
5738void VmaJsonWriter::WriteIndent(bool oneLess)
5739{
5740 if (!m_Stack.empty() && !m_Stack.back().singleLineMode)
5741 {
5742 m_SB.AddNewLine();
5743
5744 size_t count = m_Stack.size();
5745 if (count > 0 && oneLess)
5746 {
5747 --count;
5748 }
5749 for (size_t i = 0; i < count; ++i)
5750 {
5751 m_SB.Add(pStr: INDENT);
5752 }
5753 }
5754}
5755#endif // _VMA_JSON_WRITER_FUNCTIONS
5756
5757static void VmaPrintDetailedStatistics(VmaJsonWriter& json, const VmaDetailedStatistics& stat)
5758{
5759 json.BeginObject();
5760
5761 json.WriteString(pStr: "BlockCount");
5762 json.WriteNumber(n: stat.statistics.blockCount);
5763 json.WriteString(pStr: "BlockBytes");
5764 json.WriteNumber(n: stat.statistics.blockBytes);
5765 json.WriteString(pStr: "AllocationCount");
5766 json.WriteNumber(n: stat.statistics.allocationCount);
5767 json.WriteString(pStr: "AllocationBytes");
5768 json.WriteNumber(n: stat.statistics.allocationBytes);
5769 json.WriteString(pStr: "UnusedRangeCount");
5770 json.WriteNumber(n: stat.unusedRangeCount);
5771
5772 if (stat.statistics.allocationCount > 1)
5773 {
5774 json.WriteString(pStr: "AllocationSizeMin");
5775 json.WriteNumber(n: stat.allocationSizeMin);
5776 json.WriteString(pStr: "AllocationSizeMax");
5777 json.WriteNumber(n: stat.allocationSizeMax);
5778 }
5779 if (stat.unusedRangeCount > 1)
5780 {
5781 json.WriteString(pStr: "UnusedRangeSizeMin");
5782 json.WriteNumber(n: stat.unusedRangeSizeMin);
5783 json.WriteString(pStr: "UnusedRangeSizeMax");
5784 json.WriteNumber(n: stat.unusedRangeSizeMax);
5785 }
5786 json.EndObject();
5787}
5788#endif // _VMA_JSON_WRITER
5789
5790#ifndef _VMA_MAPPING_HYSTERESIS
5791
5792class VmaMappingHysteresis
5793{
5794 VMA_CLASS_NO_COPY(VmaMappingHysteresis)
5795public:
5796 VmaMappingHysteresis() = default;
5797
5798 uint32_t GetExtraMapping() const { return m_ExtraMapping; }
5799
5800 // Call when Map was called.
5801 // Returns true if switched to extra +1 mapping reference count.
5802 bool PostMap()
5803 {
5804#if VMA_MAPPING_HYSTERESIS_ENABLED
5805 if(m_ExtraMapping == 0)
5806 {
5807 ++m_MajorCounter;
5808 if(m_MajorCounter >= COUNTER_MIN_EXTRA_MAPPING)
5809 {
5810 m_ExtraMapping = 1;
5811 m_MajorCounter = 0;
5812 m_MinorCounter = 0;
5813 return true;
5814 }
5815 }
5816 else // m_ExtraMapping == 1
5817 PostMinorCounter();
5818#endif // #if VMA_MAPPING_HYSTERESIS_ENABLED
5819 return false;
5820 }
5821
5822 // Call when Unmap was called.
5823 void PostUnmap()
5824 {
5825#if VMA_MAPPING_HYSTERESIS_ENABLED
5826 if(m_ExtraMapping == 0)
5827 ++m_MajorCounter;
5828 else // m_ExtraMapping == 1
5829 PostMinorCounter();
5830#endif // #if VMA_MAPPING_HYSTERESIS_ENABLED
5831 }
5832
5833 // Call when allocation was made from the memory block.
5834 void PostAlloc()
5835 {
5836#if VMA_MAPPING_HYSTERESIS_ENABLED
5837 if(m_ExtraMapping == 1)
5838 ++m_MajorCounter;
5839 else // m_ExtraMapping == 0
5840 PostMinorCounter();
5841#endif // #if VMA_MAPPING_HYSTERESIS_ENABLED
5842 }
5843
5844 // Call when allocation was freed from the memory block.
5845 // Returns true if switched to extra -1 mapping reference count.
5846 bool PostFree()
5847 {
5848#if VMA_MAPPING_HYSTERESIS_ENABLED
5849 if(m_ExtraMapping == 1)
5850 {
5851 ++m_MajorCounter;
5852 if(m_MajorCounter >= COUNTER_MIN_EXTRA_MAPPING &&
5853 m_MajorCounter > m_MinorCounter + 1)
5854 {
5855 m_ExtraMapping = 0;
5856 m_MajorCounter = 0;
5857 m_MinorCounter = 0;
5858 return true;
5859 }
5860 }
5861 else // m_ExtraMapping == 0
5862 PostMinorCounter();
5863#endif // #if VMA_MAPPING_HYSTERESIS_ENABLED
5864 return false;
5865 }
5866
5867private:
5868 static const int32_t COUNTER_MIN_EXTRA_MAPPING = 7;
5869
5870 uint32_t m_MinorCounter = 0;
5871 uint32_t m_MajorCounter = 0;
5872 uint32_t m_ExtraMapping = 0; // 0 or 1.
5873
5874 void PostMinorCounter()
5875 {
5876 if(m_MinorCounter < m_MajorCounter)
5877 {
5878 ++m_MinorCounter;
5879 }
5880 else if(m_MajorCounter > 0)
5881 {
5882 --m_MajorCounter;
5883 --m_MinorCounter;
5884 }
5885 }
5886};
5887
5888#endif // _VMA_MAPPING_HYSTERESIS
5889
5890#ifndef _VMA_DEVICE_MEMORY_BLOCK
5891/*
5892Represents a single block of device memory (`VkDeviceMemory`) with all the
5893data about its regions (aka suballocations, #VmaAllocation), assigned and free.
5894
5895Thread-safety:
5896- Access to m_pMetadata must be externally synchronized.
5897- Map, Unmap, Bind* are synchronized internally.
5898*/
5899class VmaDeviceMemoryBlock
5900{
5901 VMA_CLASS_NO_COPY(VmaDeviceMemoryBlock)
5902public:
5903 VmaBlockMetadata* m_pMetadata;
5904
5905 VmaDeviceMemoryBlock(VmaAllocator hAllocator);
5906 ~VmaDeviceMemoryBlock();
5907
5908 // Always call after construction.
5909 void Init(
5910 VmaAllocator hAllocator,
5911 VmaPool hParentPool,
5912 uint32_t newMemoryTypeIndex,
5913 VkDeviceMemory newMemory,
5914 VkDeviceSize newSize,
5915 uint32_t id,
5916 uint32_t algorithm,
5917 VkDeviceSize bufferImageGranularity);
5918 // Always call before destruction.
5919 void Destroy(VmaAllocator allocator);
5920
5921 VmaPool GetParentPool() const { return m_hParentPool; }
5922 VkDeviceMemory GetDeviceMemory() const { return m_hMemory; }
5923 uint32_t GetMemoryTypeIndex() const { return m_MemoryTypeIndex; }
5924 uint32_t GetId() const { return m_Id; }
5925 void* GetMappedData() const { return m_pMappedData; }
5926 uint32_t GetMapRefCount() const { return m_MapCount; }
5927
5928 // Call when allocation/free was made from m_pMetadata.
5929 // Used for m_MappingHysteresis.
5930 void PostAlloc() { m_MappingHysteresis.PostAlloc(); }
5931 void PostFree(VmaAllocator hAllocator);
5932
5933 // Validates all data structures inside this object. If not valid, returns false.
5934 bool Validate() const;
5935 VkResult CheckCorruption(VmaAllocator hAllocator);
5936
5937 // ppData can be null.
5938 VkResult Map(VmaAllocator hAllocator, uint32_t count, void** ppData);
5939 void Unmap(VmaAllocator hAllocator, uint32_t count);
5940
5941 VkResult WriteMagicValueAfterAllocation(VmaAllocator hAllocator, VkDeviceSize allocOffset, VkDeviceSize allocSize);
5942 VkResult ValidateMagicValueAfterAllocation(VmaAllocator hAllocator, VkDeviceSize allocOffset, VkDeviceSize allocSize);
5943
5944 VkResult BindBufferMemory(
5945 const VmaAllocator hAllocator,
5946 const VmaAllocation hAllocation,
5947 VkDeviceSize allocationLocalOffset,
5948 VkBuffer hBuffer,
5949 const void* pNext);
5950 VkResult BindImageMemory(
5951 const VmaAllocator hAllocator,
5952 const VmaAllocation hAllocation,
5953 VkDeviceSize allocationLocalOffset,
5954 VkImage hImage,
5955 const void* pNext);
5956
5957private:
5958 VmaPool m_hParentPool; // VK_NULL_HANDLE if not belongs to custom pool.
5959 uint32_t m_MemoryTypeIndex;
5960 uint32_t m_Id;
5961 VkDeviceMemory m_hMemory;
5962
5963 /*
5964 Protects access to m_hMemory so it is not used by multiple threads simultaneously, e.g. vkMapMemory, vkBindBufferMemory.
5965 Also protects m_MapCount, m_pMappedData.
5966 Allocations, deallocations, any change in m_pMetadata is protected by parent's VmaBlockVector::m_Mutex.
5967 */
5968 VMA_MUTEX m_MapAndBindMutex;
5969 VmaMappingHysteresis m_MappingHysteresis;
5970 uint32_t m_MapCount;
5971 void* m_pMappedData;
5972};
5973#endif // _VMA_DEVICE_MEMORY_BLOCK
5974
5975#ifndef _VMA_ALLOCATION_T
5976struct VmaAllocation_T
5977{
5978 friend struct VmaDedicatedAllocationListItemTraits;
5979
5980 enum FLAGS
5981 {
5982 FLAG_PERSISTENT_MAP = 0x01,
5983 FLAG_MAPPING_ALLOWED = 0x02,
5984 };
5985
5986public:
5987 enum ALLOCATION_TYPE
5988 {
5989 ALLOCATION_TYPE_NONE,
5990 ALLOCATION_TYPE_BLOCK,
5991 ALLOCATION_TYPE_DEDICATED,
5992 };
5993
5994 // This struct is allocated using VmaPoolAllocator.
5995 VmaAllocation_T(bool mappingAllowed);
5996 ~VmaAllocation_T();
5997
5998 void InitBlockAllocation(
5999 VmaDeviceMemoryBlock* block,
6000 VmaAllocHandle allocHandle,
6001 VkDeviceSize alignment,
6002 VkDeviceSize size,
6003 uint32_t memoryTypeIndex,
6004 VmaSuballocationType suballocationType,
6005 bool mapped);
6006 // pMappedData not null means allocation is created with MAPPED flag.
6007 void InitDedicatedAllocation(
6008 VmaPool hParentPool,
6009 uint32_t memoryTypeIndex,
6010 VkDeviceMemory hMemory,
6011 VmaSuballocationType suballocationType,
6012 void* pMappedData,
6013 VkDeviceSize size);
6014
6015 ALLOCATION_TYPE GetType() const { return (ALLOCATION_TYPE)m_Type; }
6016 VkDeviceSize GetAlignment() const { return m_Alignment; }
6017 VkDeviceSize GetSize() const { return m_Size; }
6018 void* GetUserData() const { return m_pUserData; }
6019 const char* GetName() const { return m_pName; }
6020 VmaSuballocationType GetSuballocationType() const { return (VmaSuballocationType)m_SuballocationType; }
6021
6022 VmaDeviceMemoryBlock* GetBlock() const { VMA_ASSERT(m_Type == ALLOCATION_TYPE_BLOCK); return m_BlockAllocation.m_Block; }
6023 uint32_t GetMemoryTypeIndex() const { return m_MemoryTypeIndex; }
6024 bool IsPersistentMap() const { return (m_Flags & FLAG_PERSISTENT_MAP) != 0; }
6025 bool IsMappingAllowed() const { return (m_Flags & FLAG_MAPPING_ALLOWED) != 0; }
6026
6027 void SetUserData(VmaAllocator hAllocator, void* pUserData) { m_pUserData = pUserData; }
6028 void SetName(VmaAllocator hAllocator, const char* pName);
6029 void FreeName(VmaAllocator hAllocator);
6030 uint8_t SwapBlockAllocation(VmaAllocator hAllocator, VmaAllocation allocation);
6031 VmaAllocHandle GetAllocHandle() const;
6032 VkDeviceSize GetOffset() const;
6033 VmaPool GetParentPool() const;
6034 VkDeviceMemory GetMemory() const;
6035 void* GetMappedData() const;
6036
6037 void BlockAllocMap();
6038 void BlockAllocUnmap();
6039 VkResult DedicatedAllocMap(VmaAllocator hAllocator, void** ppData);
6040 void DedicatedAllocUnmap(VmaAllocator hAllocator);
6041
6042#if VMA_STATS_STRING_ENABLED
6043 uint32_t GetBufferImageUsage() const { return m_BufferImageUsage; }
6044
6045 void InitBufferImageUsage(uint32_t bufferImageUsage);
6046 void PrintParameters(class VmaJsonWriter& json) const;
6047#endif
6048
6049private:
6050 // Allocation out of VmaDeviceMemoryBlock.
6051 struct BlockAllocation
6052 {
6053 VmaDeviceMemoryBlock* m_Block;
6054 VmaAllocHandle m_AllocHandle;
6055 };
6056 // Allocation for an object that has its own private VkDeviceMemory.
6057 struct DedicatedAllocation
6058 {
6059 VmaPool m_hParentPool; // VK_NULL_HANDLE if not belongs to custom pool.
6060 VkDeviceMemory m_hMemory;
6061 void* m_pMappedData; // Not null means memory is mapped.
6062 VmaAllocation_T* m_Prev;
6063 VmaAllocation_T* m_Next;
6064 };
6065 union
6066 {
6067 // Allocation out of VmaDeviceMemoryBlock.
6068 BlockAllocation m_BlockAllocation;
6069 // Allocation for an object that has its own private VkDeviceMemory.
6070 DedicatedAllocation m_DedicatedAllocation;
6071 };
6072
6073 VkDeviceSize m_Alignment;
6074 VkDeviceSize m_Size;
6075 void* m_pUserData;
6076 char* m_pName;
6077 uint32_t m_MemoryTypeIndex;
6078 uint8_t m_Type; // ALLOCATION_TYPE
6079 uint8_t m_SuballocationType; // VmaSuballocationType
6080 // Reference counter for vmaMapMemory()/vmaUnmapMemory().
6081 uint8_t m_MapCount;
6082 uint8_t m_Flags; // enum FLAGS
6083#if VMA_STATS_STRING_ENABLED
6084 uint32_t m_BufferImageUsage; // 0 if unknown.
6085#endif
6086};
6087#endif // _VMA_ALLOCATION_T
6088
6089#ifndef _VMA_DEDICATED_ALLOCATION_LIST_ITEM_TRAITS
6090struct VmaDedicatedAllocationListItemTraits
6091{
6092 typedef VmaAllocation_T ItemType;
6093
6094 static ItemType* GetPrev(const ItemType* item)
6095 {
6096 VMA_HEAVY_ASSERT(item->GetType() == VmaAllocation_T::ALLOCATION_TYPE_DEDICATED);
6097 return item->m_DedicatedAllocation.m_Prev;
6098 }
6099 static ItemType* GetNext(const ItemType* item)
6100 {
6101 VMA_HEAVY_ASSERT(item->GetType() == VmaAllocation_T::ALLOCATION_TYPE_DEDICATED);
6102 return item->m_DedicatedAllocation.m_Next;
6103 }
6104 static ItemType*& AccessPrev(ItemType* item)
6105 {
6106 VMA_HEAVY_ASSERT(item->GetType() == VmaAllocation_T::ALLOCATION_TYPE_DEDICATED);
6107 return item->m_DedicatedAllocation.m_Prev;
6108 }
6109 static ItemType*& AccessNext(ItemType* item)
6110 {
6111 VMA_HEAVY_ASSERT(item->GetType() == VmaAllocation_T::ALLOCATION_TYPE_DEDICATED);
6112 return item->m_DedicatedAllocation.m_Next;
6113 }
6114};
6115#endif // _VMA_DEDICATED_ALLOCATION_LIST_ITEM_TRAITS
6116
6117#ifndef _VMA_DEDICATED_ALLOCATION_LIST
6118/*
6119Stores linked list of VmaAllocation_T objects.
6120Thread-safe, synchronized internally.
6121*/
6122class VmaDedicatedAllocationList
6123{
6124public:
6125 VmaDedicatedAllocationList() {}
6126 ~VmaDedicatedAllocationList();
6127
6128 void Init(bool useMutex) { m_UseMutex = useMutex; }
6129 bool Validate();
6130
6131 void AddDetailedStatistics(VmaDetailedStatistics& inoutStats);
6132 void AddStatistics(VmaStatistics& inoutStats);
6133#if VMA_STATS_STRING_ENABLED
6134 // Writes JSON array with the list of allocations.
6135 void BuildStatsString(VmaJsonWriter& json);
6136#endif
6137
6138 bool IsEmpty();
6139 void Register(VmaAllocation alloc);
6140 void Unregister(VmaAllocation alloc);
6141
6142private:
6143 typedef VmaIntrusiveLinkedList<VmaDedicatedAllocationListItemTraits> DedicatedAllocationLinkedList;
6144
6145 bool m_UseMutex = true;
6146 VMA_RW_MUTEX m_Mutex;
6147 DedicatedAllocationLinkedList m_AllocationList;
6148};
6149
6150#ifndef _VMA_DEDICATED_ALLOCATION_LIST_FUNCTIONS
6151
6152VmaDedicatedAllocationList::~VmaDedicatedAllocationList()
6153{
6154 VMA_HEAVY_ASSERT(Validate());
6155
6156 if (!m_AllocationList.IsEmpty())
6157 {
6158 VMA_ASSERT(false && "Unfreed dedicated allocations found!");
6159 }
6160}
6161
6162bool VmaDedicatedAllocationList::Validate()
6163{
6164 const size_t declaredCount = m_AllocationList.GetCount();
6165 size_t actualCount = 0;
6166 VmaMutexLockRead lock(m_Mutex, m_UseMutex);
6167 for (VmaAllocation alloc = m_AllocationList.Front();
6168 alloc != VMA_NULL; alloc = m_AllocationList.GetNext(item: alloc))
6169 {
6170 ++actualCount;
6171 }
6172 VMA_VALIDATE(actualCount == declaredCount);
6173
6174 return true;
6175}
6176
6177void VmaDedicatedAllocationList::AddDetailedStatistics(VmaDetailedStatistics& inoutStats)
6178{
6179 for(auto* item = m_AllocationList.Front(); item != nullptr; item = DedicatedAllocationLinkedList::GetNext(item))
6180 {
6181 const VkDeviceSize size = item->GetSize();
6182 inoutStats.statistics.blockCount++;
6183 inoutStats.statistics.blockBytes += size;
6184 VmaAddDetailedStatisticsAllocation(inoutStats, size: item->GetSize());
6185 }
6186}
6187
6188void VmaDedicatedAllocationList::AddStatistics(VmaStatistics& inoutStats)
6189{
6190 VmaMutexLockRead lock(m_Mutex, m_UseMutex);
6191
6192 const uint32_t allocCount = (uint32_t)m_AllocationList.GetCount();
6193 inoutStats.blockCount += allocCount;
6194 inoutStats.allocationCount += allocCount;
6195
6196 for(auto* item = m_AllocationList.Front(); item != nullptr; item = DedicatedAllocationLinkedList::GetNext(item))
6197 {
6198 const VkDeviceSize size = item->GetSize();
6199 inoutStats.blockBytes += size;
6200 inoutStats.allocationBytes += size;
6201 }
6202}
6203
6204#if VMA_STATS_STRING_ENABLED
6205void VmaDedicatedAllocationList::BuildStatsString(VmaJsonWriter& json)
6206{
6207 VmaMutexLockRead lock(m_Mutex, m_UseMutex);
6208 json.BeginArray();
6209 for (VmaAllocation alloc = m_AllocationList.Front();
6210 alloc != VMA_NULL; alloc = m_AllocationList.GetNext(item: alloc))
6211 {
6212 json.BeginObject(singleLine: true);
6213 alloc->PrintParameters(json);
6214 json.EndObject();
6215 }
6216 json.EndArray();
6217}
6218#endif // VMA_STATS_STRING_ENABLED
6219
6220bool VmaDedicatedAllocationList::IsEmpty()
6221{
6222 VmaMutexLockRead lock(m_Mutex, m_UseMutex);
6223 return m_AllocationList.IsEmpty();
6224}
6225
6226void VmaDedicatedAllocationList::Register(VmaAllocation alloc)
6227{
6228 VmaMutexLockWrite lock(m_Mutex, m_UseMutex);
6229 m_AllocationList.PushBack(item: alloc);
6230}
6231
6232void VmaDedicatedAllocationList::Unregister(VmaAllocation alloc)
6233{
6234 VmaMutexLockWrite lock(m_Mutex, m_UseMutex);
6235 m_AllocationList.Remove(item: alloc);
6236}
6237#endif // _VMA_DEDICATED_ALLOCATION_LIST_FUNCTIONS
6238#endif // _VMA_DEDICATED_ALLOCATION_LIST
6239
6240#ifndef _VMA_SUBALLOCATION
6241/*
6242Represents a region of VmaDeviceMemoryBlock that is either assigned and returned as
6243allocated memory block or free.
6244*/
6245struct VmaSuballocation
6246{
6247 VkDeviceSize offset;
6248 VkDeviceSize size;
6249 void* userData;
6250 VmaSuballocationType type;
6251};
6252
6253// Comparator for offsets.
6254struct VmaSuballocationOffsetLess
6255{
6256 bool operator()(const VmaSuballocation& lhs, const VmaSuballocation& rhs) const
6257 {
6258 return lhs.offset < rhs.offset;
6259 }
6260};
6261
6262struct VmaSuballocationOffsetGreater
6263{
6264 bool operator()(const VmaSuballocation& lhs, const VmaSuballocation& rhs) const
6265 {
6266 return lhs.offset > rhs.offset;
6267 }
6268};
6269
6270struct VmaSuballocationItemSizeLess
6271{
6272 bool operator()(const VmaSuballocationList::iterator lhs,
6273 const VmaSuballocationList::iterator rhs) const
6274 {
6275 return lhs->size < rhs->size;
6276 }
6277
6278 bool operator()(const VmaSuballocationList::iterator lhs,
6279 VkDeviceSize rhsSize) const
6280 {
6281 return lhs->size < rhsSize;
6282 }
6283};
6284#endif // _VMA_SUBALLOCATION
6285
6286#ifndef _VMA_ALLOCATION_REQUEST
6287/*
6288Parameters of planned allocation inside a VmaDeviceMemoryBlock.
6289item points to a FREE suballocation.
6290*/
6291struct VmaAllocationRequest
6292{
6293 VmaAllocHandle allocHandle;
6294 VkDeviceSize size;
6295 VmaSuballocationList::iterator item;
6296 void* customData;
6297 uint64_t algorithmData;
6298 VmaAllocationRequestType type;
6299};
6300#endif // _VMA_ALLOCATION_REQUEST
6301
6302#ifndef _VMA_BLOCK_METADATA
6303/*
6304Data structure used for bookkeeping of allocations and unused ranges of memory
6305in a single VkDeviceMemory block.
6306*/
6307class VmaBlockMetadata
6308{
6309public:
6310 // pAllocationCallbacks, if not null, must be owned externally - alive and unchanged for the whole lifetime of this object.
6311 VmaBlockMetadata(const VkAllocationCallbacks* pAllocationCallbacks,
6312 VkDeviceSize bufferImageGranularity, bool isVirtual);
6313 virtual ~VmaBlockMetadata() = default;
6314
6315 virtual void Init(VkDeviceSize size) { m_Size = size; }
6316 bool IsVirtual() const { return m_IsVirtual; }
6317 VkDeviceSize GetSize() const { return m_Size; }
6318
6319 // Validates all data structures inside this object. If not valid, returns false.
6320 virtual bool Validate() const = 0;
6321 virtual size_t GetAllocationCount() const = 0;
6322 virtual size_t GetFreeRegionsCount() const = 0;
6323 virtual VkDeviceSize GetSumFreeSize() const = 0;
6324 // Returns true if this block is empty - contains only single free suballocation.
6325 virtual bool IsEmpty() const = 0;
6326 virtual void GetAllocationInfo(VmaAllocHandle allocHandle, VmaVirtualAllocationInfo& outInfo) = 0;
6327 virtual VkDeviceSize GetAllocationOffset(VmaAllocHandle allocHandle) const = 0;
6328 virtual void* GetAllocationUserData(VmaAllocHandle allocHandle) const = 0;
6329
6330 virtual VmaAllocHandle GetAllocationListBegin() const = 0;
6331 virtual VmaAllocHandle GetNextAllocation(VmaAllocHandle prevAlloc) const = 0;
6332 virtual VkDeviceSize GetNextFreeRegionSize(VmaAllocHandle alloc) const = 0;
6333
6334 // Shouldn't modify blockCount.
6335 virtual void AddDetailedStatistics(VmaDetailedStatistics& inoutStats) const = 0;
6336 virtual void AddStatistics(VmaStatistics& inoutStats) const = 0;
6337
6338#if VMA_STATS_STRING_ENABLED
6339 virtual void PrintDetailedMap(class VmaJsonWriter& json) const = 0;
6340#endif
6341
6342 // Tries to find a place for suballocation with given parameters inside this block.
6343 // If succeeded, fills pAllocationRequest and returns true.
6344 // If failed, returns false.
6345 virtual bool CreateAllocationRequest(
6346 VkDeviceSize allocSize,
6347 VkDeviceSize allocAlignment,
6348 bool upperAddress,
6349 VmaSuballocationType allocType,
6350 // Always one of VMA_ALLOCATION_CREATE_STRATEGY_* or VMA_ALLOCATION_INTERNAL_STRATEGY_* flags.
6351 uint32_t strategy,
6352 VmaAllocationRequest* pAllocationRequest) = 0;
6353
6354 virtual VkResult CheckCorruption(const void* pBlockData) = 0;
6355
6356 // Makes actual allocation based on request. Request must already be checked and valid.
6357 virtual void Alloc(
6358 const VmaAllocationRequest& request,
6359 VmaSuballocationType type,
6360 void* userData) = 0;
6361
6362 // Frees suballocation assigned to given memory region.
6363 virtual void Free(VmaAllocHandle allocHandle) = 0;
6364
6365 // Frees all allocations.
6366 // Careful! Don't call it if there are VmaAllocation objects owned by userData of cleared allocations!
6367 virtual void Clear() = 0;
6368
6369 virtual void SetAllocationUserData(VmaAllocHandle allocHandle, void* userData) = 0;
6370 virtual void DebugLogAllAllocations() const = 0;
6371
6372protected:
6373 const VkAllocationCallbacks* GetAllocationCallbacks() const { return m_pAllocationCallbacks; }
6374 VkDeviceSize GetBufferImageGranularity() const { return m_BufferImageGranularity; }
6375 VkDeviceSize GetDebugMargin() const { return IsVirtual() ? 0 : VMA_DEBUG_MARGIN; }
6376
6377 void DebugLogAllocation(VkDeviceSize offset, VkDeviceSize size, void* userData) const;
6378#if VMA_STATS_STRING_ENABLED
6379 // mapRefCount == UINT32_MAX means unspecified.
6380 void PrintDetailedMap_Begin(class VmaJsonWriter& json,
6381 VkDeviceSize unusedBytes,
6382 size_t allocationCount,
6383 size_t unusedRangeCount) const;
6384 void PrintDetailedMap_Allocation(class VmaJsonWriter& json,
6385 VkDeviceSize offset, VkDeviceSize size, void* userData) const;
6386 void PrintDetailedMap_UnusedRange(class VmaJsonWriter& json,
6387 VkDeviceSize offset,
6388 VkDeviceSize size) const;
6389 void PrintDetailedMap_End(class VmaJsonWriter& json) const;
6390#endif
6391
6392private:
6393 VkDeviceSize m_Size;
6394 const VkAllocationCallbacks* m_pAllocationCallbacks;
6395 const VkDeviceSize m_BufferImageGranularity;
6396 const bool m_IsVirtual;
6397};
6398
6399#ifndef _VMA_BLOCK_METADATA_FUNCTIONS
6400VmaBlockMetadata::VmaBlockMetadata(const VkAllocationCallbacks* pAllocationCallbacks,
6401 VkDeviceSize bufferImageGranularity, bool isVirtual)
6402 : m_Size(0),
6403 m_pAllocationCallbacks(pAllocationCallbacks),
6404 m_BufferImageGranularity(bufferImageGranularity),
6405 m_IsVirtual(isVirtual) {}
6406
6407void VmaBlockMetadata::DebugLogAllocation(VkDeviceSize offset, VkDeviceSize size, void* userData) const
6408{
6409 if (IsVirtual())
6410 {
6411 VMA_DEBUG_LOG("UNFREED VIRTUAL ALLOCATION; Offset: %llu; Size: %llu; UserData: %p", offset, size, userData);
6412 }
6413 else
6414 {
6415 VMA_ASSERT(userData != VMA_NULL);
6416 VmaAllocation allocation = reinterpret_cast<VmaAllocation>(userData);
6417
6418 userData = allocation->GetUserData();
6419 const char* name = allocation->GetName();
6420
6421#if VMA_STATS_STRING_ENABLED
6422 VMA_DEBUG_LOG("UNFREED ALLOCATION; Offset: %llu; Size: %llu; UserData: %p; Name: %s; Type: %s; Usage: %u",
6423 offset, size, userData, name ? name : "vma_empty",
6424 VMA_SUBALLOCATION_TYPE_NAMES[allocation->GetSuballocationType()],
6425 allocation->GetBufferImageUsage());
6426#else
6427 VMA_DEBUG_LOG("UNFREED ALLOCATION; Offset: %llu; Size: %llu; UserData: %p; Name: %s; Type: %u",
6428 offset, size, userData, name ? name : "vma_empty",
6429 (uint32_t)allocation->GetSuballocationType());
6430#endif // VMA_STATS_STRING_ENABLED
6431 }
6432
6433}
6434
6435#if VMA_STATS_STRING_ENABLED
6436void VmaBlockMetadata::PrintDetailedMap_Begin(class VmaJsonWriter& json,
6437 VkDeviceSize unusedBytes, size_t allocationCount, size_t unusedRangeCount) const
6438{
6439 json.WriteString(pStr: "TotalBytes");
6440 json.WriteNumber(n: GetSize());
6441
6442 json.WriteString(pStr: "UnusedBytes");
6443 json.WriteSize(n: unusedBytes);
6444
6445 json.WriteString(pStr: "Allocations");
6446 json.WriteSize(n: allocationCount);
6447
6448 json.WriteString(pStr: "UnusedRanges");
6449 json.WriteSize(n: unusedRangeCount);
6450
6451 json.WriteString(pStr: "Suballocations");
6452 json.BeginArray();
6453}
6454
6455void VmaBlockMetadata::PrintDetailedMap_Allocation(class VmaJsonWriter& json,
6456 VkDeviceSize offset, VkDeviceSize size, void* userData) const
6457{
6458 json.BeginObject(singleLine: true);
6459
6460 json.WriteString(pStr: "Offset");
6461 json.WriteNumber(n: offset);
6462
6463 if (IsVirtual())
6464 {
6465 json.WriteString(pStr: "Size");
6466 json.WriteNumber(n: size);
6467 if (userData)
6468 {
6469 json.WriteString(pStr: "CustomData");
6470 json.BeginString();
6471 json.ContinueString_Pointer(ptr: userData);
6472 json.EndString();
6473 }
6474 }
6475 else
6476 {
6477 ((VmaAllocation)userData)->PrintParameters(json);
6478 }
6479
6480 json.EndObject();
6481}
6482
6483void VmaBlockMetadata::PrintDetailedMap_UnusedRange(class VmaJsonWriter& json,
6484 VkDeviceSize offset, VkDeviceSize size) const
6485{
6486 json.BeginObject(singleLine: true);
6487
6488 json.WriteString(pStr: "Offset");
6489 json.WriteNumber(n: offset);
6490
6491 json.WriteString(pStr: "Type");
6492 json.WriteString(pStr: VMA_SUBALLOCATION_TYPE_NAMES[VMA_SUBALLOCATION_TYPE_FREE]);
6493
6494 json.WriteString(pStr: "Size");
6495 json.WriteNumber(n: size);
6496
6497 json.EndObject();
6498}
6499
6500void VmaBlockMetadata::PrintDetailedMap_End(class VmaJsonWriter& json) const
6501{
6502 json.EndArray();
6503}
6504#endif // VMA_STATS_STRING_ENABLED
6505#endif // _VMA_BLOCK_METADATA_FUNCTIONS
6506#endif // _VMA_BLOCK_METADATA
6507
6508#ifndef _VMA_BLOCK_BUFFER_IMAGE_GRANULARITY
6509// Before deleting object of this class remember to call 'Destroy()'
6510class VmaBlockBufferImageGranularity final
6511{
6512public:
6513 struct ValidationContext
6514 {
6515 const VkAllocationCallbacks* allocCallbacks;
6516 uint16_t* pageAllocs;
6517 };
6518
6519 VmaBlockBufferImageGranularity(VkDeviceSize bufferImageGranularity);
6520 ~VmaBlockBufferImageGranularity();
6521
6522 bool IsEnabled() const { return m_BufferImageGranularity > MAX_LOW_BUFFER_IMAGE_GRANULARITY; }
6523
6524 void Init(const VkAllocationCallbacks* pAllocationCallbacks, VkDeviceSize size);
6525 // Before destroying object you must call free it's memory
6526 void Destroy(const VkAllocationCallbacks* pAllocationCallbacks);
6527
6528 void RoundupAllocRequest(VmaSuballocationType allocType,
6529 VkDeviceSize& inOutAllocSize,
6530 VkDeviceSize& inOutAllocAlignment) const;
6531
6532 bool CheckConflictAndAlignUp(VkDeviceSize& inOutAllocOffset,
6533 VkDeviceSize allocSize,
6534 VkDeviceSize blockOffset,
6535 VkDeviceSize blockSize,
6536 VmaSuballocationType allocType) const;
6537
6538 void AllocPages(uint8_t allocType, VkDeviceSize offset, VkDeviceSize size);
6539 void FreePages(VkDeviceSize offset, VkDeviceSize size);
6540 void Clear();
6541
6542 ValidationContext StartValidation(const VkAllocationCallbacks* pAllocationCallbacks,
6543 bool isVirutal) const;
6544 bool Validate(ValidationContext& ctx, VkDeviceSize offset, VkDeviceSize size) const;
6545 bool FinishValidation(ValidationContext& ctx) const;
6546
6547private:
6548 static const uint16_t MAX_LOW_BUFFER_IMAGE_GRANULARITY = 256;
6549
6550 struct RegionInfo
6551 {
6552 uint8_t allocType;
6553 uint16_t allocCount;
6554 };
6555
6556 VkDeviceSize m_BufferImageGranularity;
6557 uint32_t m_RegionCount;
6558 RegionInfo* m_RegionInfo;
6559
6560 uint32_t GetStartPage(VkDeviceSize offset) const { return OffsetToPageIndex(offset: offset & ~(m_BufferImageGranularity - 1)); }
6561 uint32_t GetEndPage(VkDeviceSize offset, VkDeviceSize size) const { return OffsetToPageIndex(offset: (offset + size - 1) & ~(m_BufferImageGranularity - 1)); }
6562
6563 uint32_t OffsetToPageIndex(VkDeviceSize offset) const;
6564 void AllocPage(RegionInfo& page, uint8_t allocType);
6565};
6566
6567#ifndef _VMA_BLOCK_BUFFER_IMAGE_GRANULARITY_FUNCTIONS
6568VmaBlockBufferImageGranularity::VmaBlockBufferImageGranularity(VkDeviceSize bufferImageGranularity)
6569 : m_BufferImageGranularity(bufferImageGranularity),
6570 m_RegionCount(0),
6571 m_RegionInfo(VMA_NULL) {}
6572
6573VmaBlockBufferImageGranularity::~VmaBlockBufferImageGranularity()
6574{
6575 VMA_ASSERT(m_RegionInfo == VMA_NULL && "Free not called before destroying object!");
6576}
6577
6578void VmaBlockBufferImageGranularity::Init(const VkAllocationCallbacks* pAllocationCallbacks, VkDeviceSize size)
6579{
6580 if (IsEnabled())
6581 {
6582 m_RegionCount = static_cast<uint32_t>(VmaDivideRoundingUp(x: size, y: m_BufferImageGranularity));
6583 m_RegionInfo = vma_new_array(pAllocationCallbacks, RegionInfo, m_RegionCount);
6584 memset(s: m_RegionInfo, c: 0, n: m_RegionCount * sizeof(RegionInfo));
6585 }
6586}
6587
6588void VmaBlockBufferImageGranularity::Destroy(const VkAllocationCallbacks* pAllocationCallbacks)
6589{
6590 if (m_RegionInfo)
6591 {
6592 vma_delete_array(pAllocationCallbacks, ptr: m_RegionInfo, count: m_RegionCount);
6593 m_RegionInfo = VMA_NULL;
6594 }
6595}
6596
6597void VmaBlockBufferImageGranularity::RoundupAllocRequest(VmaSuballocationType allocType,
6598 VkDeviceSize& inOutAllocSize,
6599 VkDeviceSize& inOutAllocAlignment) const
6600{
6601 if (m_BufferImageGranularity > 1 &&
6602 m_BufferImageGranularity <= MAX_LOW_BUFFER_IMAGE_GRANULARITY)
6603 {
6604 if (allocType == VMA_SUBALLOCATION_TYPE_UNKNOWN ||
6605 allocType == VMA_SUBALLOCATION_TYPE_IMAGE_UNKNOWN ||
6606 allocType == VMA_SUBALLOCATION_TYPE_IMAGE_OPTIMAL)
6607 {
6608 inOutAllocAlignment = VMA_MAX(inOutAllocAlignment, m_BufferImageGranularity);
6609 inOutAllocSize = VmaAlignUp(val: inOutAllocSize, alignment: m_BufferImageGranularity);
6610 }
6611 }
6612}
6613
6614bool VmaBlockBufferImageGranularity::CheckConflictAndAlignUp(VkDeviceSize& inOutAllocOffset,
6615 VkDeviceSize allocSize,
6616 VkDeviceSize blockOffset,
6617 VkDeviceSize blockSize,
6618 VmaSuballocationType allocType) const
6619{
6620 if (IsEnabled())
6621 {
6622 uint32_t startPage = GetStartPage(offset: inOutAllocOffset);
6623 if (m_RegionInfo[startPage].allocCount > 0 &&
6624 VmaIsBufferImageGranularityConflict(suballocType1: static_cast<VmaSuballocationType>(m_RegionInfo[startPage].allocType), suballocType2: allocType))
6625 {
6626 inOutAllocOffset = VmaAlignUp(val: inOutAllocOffset, alignment: m_BufferImageGranularity);
6627 if (blockSize < allocSize + inOutAllocOffset - blockOffset)
6628 return true;
6629 ++startPage;
6630 }
6631 uint32_t endPage = GetEndPage(offset: inOutAllocOffset, size: allocSize);
6632 if (endPage != startPage &&
6633 m_RegionInfo[endPage].allocCount > 0 &&
6634 VmaIsBufferImageGranularityConflict(suballocType1: static_cast<VmaSuballocationType>(m_RegionInfo[endPage].allocType), suballocType2: allocType))
6635 {
6636 return true;
6637 }
6638 }
6639 return false;
6640}
6641
6642void VmaBlockBufferImageGranularity::AllocPages(uint8_t allocType, VkDeviceSize offset, VkDeviceSize size)
6643{
6644 if (IsEnabled())
6645 {
6646 uint32_t startPage = GetStartPage(offset);
6647 AllocPage(page&: m_RegionInfo[startPage], allocType);
6648
6649 uint32_t endPage = GetEndPage(offset, size);
6650 if (startPage != endPage)
6651 AllocPage(page&: m_RegionInfo[endPage], allocType);
6652 }
6653}
6654
6655void VmaBlockBufferImageGranularity::FreePages(VkDeviceSize offset, VkDeviceSize size)
6656{
6657 if (IsEnabled())
6658 {
6659 uint32_t startPage = GetStartPage(offset);
6660 --m_RegionInfo[startPage].allocCount;
6661 if (m_RegionInfo[startPage].allocCount == 0)
6662 m_RegionInfo[startPage].allocType = VMA_SUBALLOCATION_TYPE_FREE;
6663 uint32_t endPage = GetEndPage(offset, size);
6664 if (startPage != endPage)
6665 {
6666 --m_RegionInfo[endPage].allocCount;
6667 if (m_RegionInfo[endPage].allocCount == 0)
6668 m_RegionInfo[endPage].allocType = VMA_SUBALLOCATION_TYPE_FREE;
6669 }
6670 }
6671}
6672
6673void VmaBlockBufferImageGranularity::Clear()
6674{
6675 if (m_RegionInfo)
6676 memset(s: m_RegionInfo, c: 0, n: m_RegionCount * sizeof(RegionInfo));
6677}
6678
6679VmaBlockBufferImageGranularity::ValidationContext VmaBlockBufferImageGranularity::StartValidation(
6680 const VkAllocationCallbacks* pAllocationCallbacks, bool isVirutal) const
6681{
6682 ValidationContext ctx{ .allocCallbacks: pAllocationCallbacks, VMA_NULL };
6683 if (!isVirutal && IsEnabled())
6684 {
6685 ctx.pageAllocs = vma_new_array(pAllocationCallbacks, uint16_t, m_RegionCount);
6686 memset(s: ctx.pageAllocs, c: 0, n: m_RegionCount * sizeof(uint16_t));
6687 }
6688 return ctx;
6689}
6690
6691bool VmaBlockBufferImageGranularity::Validate(ValidationContext& ctx,
6692 VkDeviceSize offset, VkDeviceSize size) const
6693{
6694 if (IsEnabled())
6695 {
6696 uint32_t start = GetStartPage(offset);
6697 ++ctx.pageAllocs[start];
6698 VMA_VALIDATE(m_RegionInfo[start].allocCount > 0);
6699
6700 uint32_t end = GetEndPage(offset, size);
6701 if (start != end)
6702 {
6703 ++ctx.pageAllocs[end];
6704 VMA_VALIDATE(m_RegionInfo[end].allocCount > 0);
6705 }
6706 }
6707 return true;
6708}
6709
6710bool VmaBlockBufferImageGranularity::FinishValidation(ValidationContext& ctx) const
6711{
6712 // Check proper page structure
6713 if (IsEnabled())
6714 {
6715 VMA_ASSERT(ctx.pageAllocs != VMA_NULL && "Validation context not initialized!");
6716
6717 for (uint32_t page = 0; page < m_RegionCount; ++page)
6718 {
6719 VMA_VALIDATE(ctx.pageAllocs[page] == m_RegionInfo[page].allocCount);
6720 }
6721 vma_delete_array(pAllocationCallbacks: ctx.allocCallbacks, ptr: ctx.pageAllocs, count: m_RegionCount);
6722 ctx.pageAllocs = VMA_NULL;
6723 }
6724 return true;
6725}
6726
6727uint32_t VmaBlockBufferImageGranularity::OffsetToPageIndex(VkDeviceSize offset) const
6728{
6729 return static_cast<uint32_t>(offset >> VMA_BITSCAN_MSB(m_BufferImageGranularity));
6730}
6731
6732void VmaBlockBufferImageGranularity::AllocPage(RegionInfo& page, uint8_t allocType)
6733{
6734 // When current alloc type is free then it can be overriden by new type
6735 if (page.allocCount == 0 || (page.allocCount > 0 && page.allocType == VMA_SUBALLOCATION_TYPE_FREE))
6736 page.allocType = allocType;
6737
6738 ++page.allocCount;
6739}
6740#endif // _VMA_BLOCK_BUFFER_IMAGE_GRANULARITY_FUNCTIONS
6741#endif // _VMA_BLOCK_BUFFER_IMAGE_GRANULARITY
6742
6743#if 0
6744#ifndef _VMA_BLOCK_METADATA_GENERIC
6745class VmaBlockMetadata_Generic : public VmaBlockMetadata
6746{
6747 friend class VmaDefragmentationAlgorithm_Generic;
6748 friend class VmaDefragmentationAlgorithm_Fast;
6749 VMA_CLASS_NO_COPY(VmaBlockMetadata_Generic)
6750public:
6751 VmaBlockMetadata_Generic(const VkAllocationCallbacks* pAllocationCallbacks,
6752 VkDeviceSize bufferImageGranularity, bool isVirtual);
6753 virtual ~VmaBlockMetadata_Generic() = default;
6754
6755 size_t GetAllocationCount() const override { return m_Suballocations.size() - m_FreeCount; }
6756 VkDeviceSize GetSumFreeSize() const override { return m_SumFreeSize; }
6757 bool IsEmpty() const override { return (m_Suballocations.size() == 1) && (m_FreeCount == 1); }
6758 void Free(VmaAllocHandle allocHandle) override { FreeSuballocation(FindAtOffset((VkDeviceSize)allocHandle - 1)); }
6759 VkDeviceSize GetAllocationOffset(VmaAllocHandle allocHandle) const override { return (VkDeviceSize)allocHandle - 1; };
6760
6761 void Init(VkDeviceSize size) override;
6762 bool Validate() const override;
6763
6764 void AddDetailedStatistics(VmaDetailedStatistics& inoutStats) const override;
6765 void AddStatistics(VmaStatistics& inoutStats) const override;
6766
6767#if VMA_STATS_STRING_ENABLED
6768 void PrintDetailedMap(class VmaJsonWriter& json, uint32_t mapRefCount) const override;
6769#endif
6770
6771 bool CreateAllocationRequest(
6772 VkDeviceSize allocSize,
6773 VkDeviceSize allocAlignment,
6774 bool upperAddress,
6775 VmaSuballocationType allocType,
6776 uint32_t strategy,
6777 VmaAllocationRequest* pAllocationRequest) override;
6778
6779 VkResult CheckCorruption(const void* pBlockData) override;
6780
6781 void Alloc(
6782 const VmaAllocationRequest& request,
6783 VmaSuballocationType type,
6784 void* userData) override;
6785
6786 void GetAllocationInfo(VmaAllocHandle allocHandle, VmaVirtualAllocationInfo& outInfo) override;
6787 void* GetAllocationUserData(VmaAllocHandle allocHandle) const override;
6788 VmaAllocHandle GetAllocationListBegin() const override;
6789 VmaAllocHandle GetNextAllocation(VmaAllocHandle prevAlloc) const override;
6790 void Clear() override;
6791 void SetAllocationUserData(VmaAllocHandle allocHandle, void* userData) override;
6792 void DebugLogAllAllocations() const override;
6793
6794private:
6795 uint32_t m_FreeCount;
6796 VkDeviceSize m_SumFreeSize;
6797 VmaSuballocationList m_Suballocations;
6798 // Suballocations that are free. Sorted by size, ascending.
6799 VmaVector<VmaSuballocationList::iterator, VmaStlAllocator<VmaSuballocationList::iterator>> m_FreeSuballocationsBySize;
6800
6801 VkDeviceSize AlignAllocationSize(VkDeviceSize size) const { return IsVirtual() ? size : VmaAlignUp(size, (VkDeviceSize)16); }
6802
6803 VmaSuballocationList::iterator FindAtOffset(VkDeviceSize offset) const;
6804 bool ValidateFreeSuballocationList() const;
6805
6806 // Checks if requested suballocation with given parameters can be placed in given pFreeSuballocItem.
6807 // If yes, fills pOffset and returns true. If no, returns false.
6808 bool CheckAllocation(
6809 VkDeviceSize allocSize,
6810 VkDeviceSize allocAlignment,
6811 VmaSuballocationType allocType,
6812 VmaSuballocationList::const_iterator suballocItem,
6813 VmaAllocHandle* pAllocHandle) const;
6814
6815 // Given free suballocation, it merges it with following one, which must also be free.
6816 void MergeFreeWithNext(VmaSuballocationList::iterator item);
6817 // Releases given suballocation, making it free.
6818 // Merges it with adjacent free suballocations if applicable.
6819 // Returns iterator to new free suballocation at this place.
6820 VmaSuballocationList::iterator FreeSuballocation(VmaSuballocationList::iterator suballocItem);
6821 // Given free suballocation, it inserts it into sorted list of
6822 // m_FreeSuballocationsBySize if it is suitable.
6823 void RegisterFreeSuballocation(VmaSuballocationList::iterator item);
6824 // Given free suballocation, it removes it from sorted list of
6825 // m_FreeSuballocationsBySize if it is suitable.
6826 void UnregisterFreeSuballocation(VmaSuballocationList::iterator item);
6827};
6828
6829#ifndef _VMA_BLOCK_METADATA_GENERIC_FUNCTIONS
6830VmaBlockMetadata_Generic::VmaBlockMetadata_Generic(const VkAllocationCallbacks* pAllocationCallbacks,
6831 VkDeviceSize bufferImageGranularity, bool isVirtual)
6832 : VmaBlockMetadata(pAllocationCallbacks, bufferImageGranularity, isVirtual),
6833 m_FreeCount(0),
6834 m_SumFreeSize(0),
6835 m_Suballocations(VmaStlAllocator<VmaSuballocation>(pAllocationCallbacks)),
6836 m_FreeSuballocationsBySize(VmaStlAllocator<VmaSuballocationList::iterator>(pAllocationCallbacks)) {}
6837
6838void VmaBlockMetadata_Generic::Init(VkDeviceSize size)
6839{
6840 VmaBlockMetadata::Init(size);
6841
6842 m_FreeCount = 1;
6843 m_SumFreeSize = size;
6844
6845 VmaSuballocation suballoc = {};
6846 suballoc.offset = 0;
6847 suballoc.size = size;
6848 suballoc.type = VMA_SUBALLOCATION_TYPE_FREE;
6849
6850 m_Suballocations.push_back(suballoc);
6851 m_FreeSuballocationsBySize.push_back(m_Suballocations.begin());
6852}
6853
6854bool VmaBlockMetadata_Generic::Validate() const
6855{
6856 VMA_VALIDATE(!m_Suballocations.empty());
6857
6858 // Expected offset of new suballocation as calculated from previous ones.
6859 VkDeviceSize calculatedOffset = 0;
6860 // Expected number of free suballocations as calculated from traversing their list.
6861 uint32_t calculatedFreeCount = 0;
6862 // Expected sum size of free suballocations as calculated from traversing their list.
6863 VkDeviceSize calculatedSumFreeSize = 0;
6864 // Expected number of free suballocations that should be registered in
6865 // m_FreeSuballocationsBySize calculated from traversing their list.
6866 size_t freeSuballocationsToRegister = 0;
6867 // True if previous visited suballocation was free.
6868 bool prevFree = false;
6869
6870 const VkDeviceSize debugMargin = GetDebugMargin();
6871
6872 for (const auto& subAlloc : m_Suballocations)
6873 {
6874 // Actual offset of this suballocation doesn't match expected one.
6875 VMA_VALIDATE(subAlloc.offset == calculatedOffset);
6876
6877 const bool currFree = (subAlloc.type == VMA_SUBALLOCATION_TYPE_FREE);
6878 // Two adjacent free suballocations are invalid. They should be merged.
6879 VMA_VALIDATE(!prevFree || !currFree);
6880
6881 VmaAllocation alloc = (VmaAllocation)subAlloc.userData;
6882 if (!IsVirtual())
6883 {
6884 VMA_VALIDATE(currFree == (alloc == VK_NULL_HANDLE));
6885 }
6886
6887 if (currFree)
6888 {
6889 calculatedSumFreeSize += subAlloc.size;
6890 ++calculatedFreeCount;
6891 ++freeSuballocationsToRegister;
6892
6893 // Margin required between allocations - every free space must be at least that large.
6894 VMA_VALIDATE(subAlloc.size >= debugMargin);
6895 }
6896 else
6897 {
6898 if (!IsVirtual())
6899 {
6900 VMA_VALIDATE((VkDeviceSize)alloc->GetAllocHandle() == subAlloc.offset + 1);
6901 VMA_VALIDATE(alloc->GetSize() == subAlloc.size);
6902 }
6903
6904 // Margin required between allocations - previous allocation must be free.
6905 VMA_VALIDATE(debugMargin == 0 || prevFree);
6906 }
6907
6908 calculatedOffset += subAlloc.size;
6909 prevFree = currFree;
6910 }
6911
6912 // Number of free suballocations registered in m_FreeSuballocationsBySize doesn't
6913 // match expected one.
6914 VMA_VALIDATE(m_FreeSuballocationsBySize.size() == freeSuballocationsToRegister);
6915
6916 VkDeviceSize lastSize = 0;
6917 for (size_t i = 0; i < m_FreeSuballocationsBySize.size(); ++i)
6918 {
6919 VmaSuballocationList::iterator suballocItem = m_FreeSuballocationsBySize[i];
6920
6921 // Only free suballocations can be registered in m_FreeSuballocationsBySize.
6922 VMA_VALIDATE(suballocItem->type == VMA_SUBALLOCATION_TYPE_FREE);
6923 // They must be sorted by size ascending.
6924 VMA_VALIDATE(suballocItem->size >= lastSize);
6925
6926 lastSize = suballocItem->size;
6927 }
6928
6929 // Check if totals match calculated values.
6930 VMA_VALIDATE(ValidateFreeSuballocationList());
6931 VMA_VALIDATE(calculatedOffset == GetSize());
6932 VMA_VALIDATE(calculatedSumFreeSize == m_SumFreeSize);
6933 VMA_VALIDATE(calculatedFreeCount == m_FreeCount);
6934
6935 return true;
6936}
6937
6938void VmaBlockMetadata_Generic::AddDetailedStatistics(VmaDetailedStatistics& inoutStats) const
6939{
6940 const uint32_t rangeCount = (uint32_t)m_Suballocations.size();
6941 inoutStats.statistics.blockCount++;
6942 inoutStats.statistics.blockBytes += GetSize();
6943
6944 for (const auto& suballoc : m_Suballocations)
6945 {
6946 if (suballoc.type != VMA_SUBALLOCATION_TYPE_FREE)
6947 VmaAddDetailedStatisticsAllocation(inoutStats, suballoc.size);
6948 else
6949 VmaAddDetailedStatisticsUnusedRange(inoutStats, suballoc.size);
6950 }
6951}
6952
6953void VmaBlockMetadata_Generic::AddStatistics(VmaStatistics& inoutStats) const
6954{
6955 inoutStats.blockCount++;
6956 inoutStats.allocationCount += (uint32_t)m_Suballocations.size() - m_FreeCount;
6957 inoutStats.blockBytes += GetSize();
6958 inoutStats.allocationBytes += GetSize() - m_SumFreeSize;
6959}
6960
6961#if VMA_STATS_STRING_ENABLED
6962void VmaBlockMetadata_Generic::PrintDetailedMap(class VmaJsonWriter& json, uint32_t mapRefCount) const
6963{
6964 PrintDetailedMap_Begin(json,
6965 m_SumFreeSize, // unusedBytes
6966 m_Suballocations.size() - (size_t)m_FreeCount, // allocationCount
6967 m_FreeCount, // unusedRangeCount
6968 mapRefCount);
6969
6970 for (const auto& suballoc : m_Suballocations)
6971 {
6972 if (suballoc.type == VMA_SUBALLOCATION_TYPE_FREE)
6973 {
6974 PrintDetailedMap_UnusedRange(json, suballoc.offset, suballoc.size);
6975 }
6976 else
6977 {
6978 PrintDetailedMap_Allocation(json, suballoc.offset, suballoc.size, suballoc.userData);
6979 }
6980 }
6981
6982 PrintDetailedMap_End(json);
6983}
6984#endif // VMA_STATS_STRING_ENABLED
6985
6986bool VmaBlockMetadata_Generic::CreateAllocationRequest(
6987 VkDeviceSize allocSize,
6988 VkDeviceSize allocAlignment,
6989 bool upperAddress,
6990 VmaSuballocationType allocType,
6991 uint32_t strategy,
6992 VmaAllocationRequest* pAllocationRequest)
6993{
6994 VMA_ASSERT(allocSize > 0);
6995 VMA_ASSERT(!upperAddress);
6996 VMA_ASSERT(allocType != VMA_SUBALLOCATION_TYPE_FREE);
6997 VMA_ASSERT(pAllocationRequest != VMA_NULL);
6998 VMA_HEAVY_ASSERT(Validate());
6999
7000 allocSize = AlignAllocationSize(allocSize);
7001
7002 pAllocationRequest->type = VmaAllocationRequestType::Normal;
7003 pAllocationRequest->size = allocSize;
7004
7005 const VkDeviceSize debugMargin = GetDebugMargin();
7006
7007 // There is not enough total free space in this block to fulfill the request: Early return.
7008 if (m_SumFreeSize < allocSize + debugMargin)
7009 {
7010 return false;
7011 }
7012
7013 // New algorithm, efficiently searching freeSuballocationsBySize.
7014 const size_t freeSuballocCount = m_FreeSuballocationsBySize.size();
7015 if (freeSuballocCount > 0)
7016 {
7017 if (strategy == 0 ||
7018 strategy == VMA_ALLOCATION_CREATE_STRATEGY_MIN_MEMORY_BIT)
7019 {
7020 // Find first free suballocation with size not less than allocSize + debugMargin.
7021 VmaSuballocationList::iterator* const it = VmaBinaryFindFirstNotLess(
7022 m_FreeSuballocationsBySize.data(),
7023 m_FreeSuballocationsBySize.data() + freeSuballocCount,
7024 allocSize + debugMargin,
7025 VmaSuballocationItemSizeLess());
7026 size_t index = it - m_FreeSuballocationsBySize.data();
7027 for (; index < freeSuballocCount; ++index)
7028 {
7029 if (CheckAllocation(
7030 allocSize,
7031 allocAlignment,
7032 allocType,
7033 m_FreeSuballocationsBySize[index],
7034 &pAllocationRequest->allocHandle))
7035 {
7036 pAllocationRequest->item = m_FreeSuballocationsBySize[index];
7037 return true;
7038 }
7039 }
7040 }
7041 else if (strategy == VMA_ALLOCATION_INTERNAL_STRATEGY_MIN_OFFSET)
7042 {
7043 for (VmaSuballocationList::iterator it = m_Suballocations.begin();
7044 it != m_Suballocations.end();
7045 ++it)
7046 {
7047 if (it->type == VMA_SUBALLOCATION_TYPE_FREE && CheckAllocation(
7048 allocSize,
7049 allocAlignment,
7050 allocType,
7051 it,
7052 &pAllocationRequest->allocHandle))
7053 {
7054 pAllocationRequest->item = it;
7055 return true;
7056 }
7057 }
7058 }
7059 else
7060 {
7061 VMA_ASSERT(strategy & (VMA_ALLOCATION_CREATE_STRATEGY_MIN_TIME_BIT | VMA_ALLOCATION_CREATE_STRATEGY_MIN_OFFSET_BIT ));
7062 // Search staring from biggest suballocations.
7063 for (size_t index = freeSuballocCount; index--; )
7064 {
7065 if (CheckAllocation(
7066 allocSize,
7067 allocAlignment,
7068 allocType,
7069 m_FreeSuballocationsBySize[index],
7070 &pAllocationRequest->allocHandle))
7071 {
7072 pAllocationRequest->item = m_FreeSuballocationsBySize[index];
7073 return true;
7074 }
7075 }
7076 }
7077 }
7078
7079 return false;
7080}
7081
7082VkResult VmaBlockMetadata_Generic::CheckCorruption(const void* pBlockData)
7083{
7084 for (auto& suballoc : m_Suballocations)
7085 {
7086 if (suballoc.type != VMA_SUBALLOCATION_TYPE_FREE)
7087 {
7088 if (!VmaValidateMagicValue(pBlockData, suballoc.offset + suballoc.size))
7089 {
7090 VMA_ASSERT(0 && "MEMORY CORRUPTION DETECTED AFTER VALIDATED ALLOCATION!");
7091 return VK_ERROR_UNKNOWN_COPY;
7092 }
7093 }
7094 }
7095
7096 return VK_SUCCESS;
7097}
7098
7099void VmaBlockMetadata_Generic::Alloc(
7100 const VmaAllocationRequest& request,
7101 VmaSuballocationType type,
7102 void* userData)
7103{
7104 VMA_ASSERT(request.type == VmaAllocationRequestType::Normal);
7105 VMA_ASSERT(request.item != m_Suballocations.end());
7106 VmaSuballocation& suballoc = *request.item;
7107 // Given suballocation is a free block.
7108 VMA_ASSERT(suballoc.type == VMA_SUBALLOCATION_TYPE_FREE);
7109
7110 // Given offset is inside this suballocation.
7111 VMA_ASSERT((VkDeviceSize)request.allocHandle - 1 >= suballoc.offset);
7112 const VkDeviceSize paddingBegin = (VkDeviceSize)request.allocHandle - suballoc.offset - 1;
7113 VMA_ASSERT(suballoc.size >= paddingBegin + request.size);
7114 const VkDeviceSize paddingEnd = suballoc.size - paddingBegin - request.size;
7115
7116 // Unregister this free suballocation from m_FreeSuballocationsBySize and update
7117 // it to become used.
7118 UnregisterFreeSuballocation(request.item);
7119
7120 suballoc.offset = (VkDeviceSize)request.allocHandle - 1;
7121 suballoc.size = request.size;
7122 suballoc.type = type;
7123 suballoc.userData = userData;
7124
7125 // If there are any free bytes remaining at the end, insert new free suballocation after current one.
7126 if (paddingEnd)
7127 {
7128 VmaSuballocation paddingSuballoc = {};
7129 paddingSuballoc.offset = suballoc.offset + suballoc.size;
7130 paddingSuballoc.size = paddingEnd;
7131 paddingSuballoc.type = VMA_SUBALLOCATION_TYPE_FREE;
7132 VmaSuballocationList::iterator next = request.item;
7133 ++next;
7134 const VmaSuballocationList::iterator paddingEndItem =
7135 m_Suballocations.insert(next, paddingSuballoc);
7136 RegisterFreeSuballocation(paddingEndItem);
7137 }
7138
7139 // If there are any free bytes remaining at the beginning, insert new free suballocation before current one.
7140 if (paddingBegin)
7141 {
7142 VmaSuballocation paddingSuballoc = {};
7143 paddingSuballoc.offset = suballoc.offset - paddingBegin;
7144 paddingSuballoc.size = paddingBegin;
7145 paddingSuballoc.type = VMA_SUBALLOCATION_TYPE_FREE;
7146 const VmaSuballocationList::iterator paddingBeginItem =
7147 m_Suballocations.insert(request.item, paddingSuballoc);
7148 RegisterFreeSuballocation(paddingBeginItem);
7149 }
7150
7151 // Update totals.
7152 m_FreeCount = m_FreeCount - 1;
7153 if (paddingBegin > 0)
7154 {
7155 ++m_FreeCount;
7156 }
7157 if (paddingEnd > 0)
7158 {
7159 ++m_FreeCount;
7160 }
7161 m_SumFreeSize -= request.size;
7162}
7163
7164void VmaBlockMetadata_Generic::GetAllocationInfo(VmaAllocHandle allocHandle, VmaVirtualAllocationInfo& outInfo)
7165{
7166 outInfo.offset = (VkDeviceSize)allocHandle - 1;
7167 const VmaSuballocation& suballoc = *FindAtOffset(outInfo.offset);
7168 outInfo.size = suballoc.size;
7169 outInfo.pUserData = suballoc.userData;
7170}
7171
7172void* VmaBlockMetadata_Generic::GetAllocationUserData(VmaAllocHandle allocHandle) const
7173{
7174 return FindAtOffset((VkDeviceSize)allocHandle - 1)->userData;
7175}
7176
7177VmaAllocHandle VmaBlockMetadata_Generic::GetAllocationListBegin() const
7178{
7179 if (IsEmpty())
7180 return VK_NULL_HANDLE;
7181
7182 for (const auto& suballoc : m_Suballocations)
7183 {
7184 if (suballoc.type != VMA_SUBALLOCATION_TYPE_FREE)
7185 return (VmaAllocHandle)(suballoc.offset + 1);
7186 }
7187 VMA_ASSERT(false && "Should contain at least 1 allocation!");
7188 return VK_NULL_HANDLE;
7189}
7190
7191VmaAllocHandle VmaBlockMetadata_Generic::GetNextAllocation(VmaAllocHandle prevAlloc) const
7192{
7193 VmaSuballocationList::const_iterator prev = FindAtOffset((VkDeviceSize)prevAlloc - 1);
7194
7195 for (VmaSuballocationList::const_iterator it = ++prev; it != m_Suballocations.end(); ++it)
7196 {
7197 if (it->type != VMA_SUBALLOCATION_TYPE_FREE)
7198 return (VmaAllocHandle)(it->offset + 1);
7199 }
7200 return VK_NULL_HANDLE;
7201}
7202
7203void VmaBlockMetadata_Generic::Clear()
7204{
7205 const VkDeviceSize size = GetSize();
7206
7207 VMA_ASSERT(IsVirtual());
7208 m_FreeCount = 1;
7209 m_SumFreeSize = size;
7210 m_Suballocations.clear();
7211 m_FreeSuballocationsBySize.clear();
7212
7213 VmaSuballocation suballoc = {};
7214 suballoc.offset = 0;
7215 suballoc.size = size;
7216 suballoc.type = VMA_SUBALLOCATION_TYPE_FREE;
7217 m_Suballocations.push_back(suballoc);
7218
7219 m_FreeSuballocationsBySize.push_back(m_Suballocations.begin());
7220}
7221
7222void VmaBlockMetadata_Generic::SetAllocationUserData(VmaAllocHandle allocHandle, void* userData)
7223{
7224 VmaSuballocation& suballoc = *FindAtOffset((VkDeviceSize)allocHandle - 1);
7225 suballoc.userData = userData;
7226}
7227
7228void VmaBlockMetadata_Generic::DebugLogAllAllocations() const
7229{
7230 for (const auto& suballoc : m_Suballocations)
7231 {
7232 if (suballoc.type != VMA_SUBALLOCATION_TYPE_FREE)
7233 DebugLogAllocation(suballoc.offset, suballoc.size, suballoc.userData);
7234 }
7235}
7236
7237VmaSuballocationList::iterator VmaBlockMetadata_Generic::FindAtOffset(VkDeviceSize offset) const
7238{
7239 VMA_HEAVY_ASSERT(!m_Suballocations.empty());
7240 const VkDeviceSize last = m_Suballocations.rbegin()->offset;
7241 if (last == offset)
7242 return m_Suballocations.rbegin().drop_const();
7243 const VkDeviceSize first = m_Suballocations.begin()->offset;
7244 if (first == offset)
7245 return m_Suballocations.begin().drop_const();
7246
7247 const size_t suballocCount = m_Suballocations.size();
7248 const VkDeviceSize step = (last - first + m_Suballocations.begin()->size) / suballocCount;
7249 auto findSuballocation = [&](auto begin, auto end) -> VmaSuballocationList::iterator
7250 {
7251 for (auto suballocItem = begin;
7252 suballocItem != end;
7253 ++suballocItem)
7254 {
7255 if (suballocItem->offset == offset)
7256 return suballocItem.drop_const();
7257 }
7258 VMA_ASSERT(false && "Not found!");
7259 return m_Suballocations.end().drop_const();
7260 };
7261 // If requested offset is closer to the end of range, search from the end
7262 if (offset - first > suballocCount * step / 2)
7263 {
7264 return findSuballocation(m_Suballocations.rbegin(), m_Suballocations.rend());
7265 }
7266 return findSuballocation(m_Suballocations.begin(), m_Suballocations.end());
7267}
7268
7269bool VmaBlockMetadata_Generic::ValidateFreeSuballocationList() const
7270{
7271 VkDeviceSize lastSize = 0;
7272 for (size_t i = 0, count = m_FreeSuballocationsBySize.size(); i < count; ++i)
7273 {
7274 const VmaSuballocationList::iterator it = m_FreeSuballocationsBySize[i];
7275
7276 VMA_VALIDATE(it->type == VMA_SUBALLOCATION_TYPE_FREE);
7277 VMA_VALIDATE(it->size >= lastSize);
7278 lastSize = it->size;
7279 }
7280 return true;
7281}
7282
7283bool VmaBlockMetadata_Generic::CheckAllocation(
7284 VkDeviceSize allocSize,
7285 VkDeviceSize allocAlignment,
7286 VmaSuballocationType allocType,
7287 VmaSuballocationList::const_iterator suballocItem,
7288 VmaAllocHandle* pAllocHandle) const
7289{
7290 VMA_ASSERT(allocSize > 0);
7291 VMA_ASSERT(allocType != VMA_SUBALLOCATION_TYPE_FREE);
7292 VMA_ASSERT(suballocItem != m_Suballocations.cend());
7293 VMA_ASSERT(pAllocHandle != VMA_NULL);
7294
7295 const VkDeviceSize debugMargin = GetDebugMargin();
7296 const VkDeviceSize bufferImageGranularity = GetBufferImageGranularity();
7297
7298 const VmaSuballocation& suballoc = *suballocItem;
7299 VMA_ASSERT(suballoc.type == VMA_SUBALLOCATION_TYPE_FREE);
7300
7301 // Size of this suballocation is too small for this request: Early return.
7302 if (suballoc.size < allocSize)
7303 {
7304 return false;
7305 }
7306
7307 // Start from offset equal to beginning of this suballocation.
7308 VkDeviceSize offset = suballoc.offset + (suballocItem == m_Suballocations.cbegin() ? 0 : GetDebugMargin());
7309
7310 // Apply debugMargin from the end of previous alloc.
7311 if (debugMargin > 0)
7312 {
7313 offset += debugMargin;
7314 }
7315
7316 // Apply alignment.
7317 offset = VmaAlignUp(offset, allocAlignment);
7318
7319 // Check previous suballocations for BufferImageGranularity conflicts.
7320 // Make bigger alignment if necessary.
7321 if (bufferImageGranularity > 1 && bufferImageGranularity != allocAlignment)
7322 {
7323 bool bufferImageGranularityConflict = false;
7324 VmaSuballocationList::const_iterator prevSuballocItem = suballocItem;
7325 while (prevSuballocItem != m_Suballocations.cbegin())
7326 {
7327 --prevSuballocItem;
7328 const VmaSuballocation& prevSuballoc = *prevSuballocItem;
7329 if (VmaBlocksOnSamePage(prevSuballoc.offset, prevSuballoc.size, offset, bufferImageGranularity))
7330 {
7331 if (VmaIsBufferImageGranularityConflict(prevSuballoc.type, allocType))
7332 {
7333 bufferImageGranularityConflict = true;
7334 break;
7335 }
7336 }
7337 else
7338 // Already on previous page.
7339 break;
7340 }
7341 if (bufferImageGranularityConflict)
7342 {
7343 offset = VmaAlignUp(offset, bufferImageGranularity);
7344 }
7345 }
7346
7347 // Calculate padding at the beginning based on current offset.
7348 const VkDeviceSize paddingBegin = offset - suballoc.offset;
7349
7350 // Fail if requested size plus margin after is bigger than size of this suballocation.
7351 if (paddingBegin + allocSize + debugMargin > suballoc.size)
7352 {
7353 return false;
7354 }
7355
7356 // Check next suballocations for BufferImageGranularity conflicts.
7357 // If conflict exists, allocation cannot be made here.
7358 if (allocSize % bufferImageGranularity || offset % bufferImageGranularity)
7359 {
7360 VmaSuballocationList::const_iterator nextSuballocItem = suballocItem;
7361 ++nextSuballocItem;
7362 while (nextSuballocItem != m_Suballocations.cend())
7363 {
7364 const VmaSuballocation& nextSuballoc = *nextSuballocItem;
7365 if (VmaBlocksOnSamePage(offset, allocSize, nextSuballoc.offset, bufferImageGranularity))
7366 {
7367 if (VmaIsBufferImageGranularityConflict(allocType, nextSuballoc.type))
7368 {
7369 return false;
7370 }
7371 }
7372 else
7373 {
7374 // Already on next page.
7375 break;
7376 }
7377 ++nextSuballocItem;
7378 }
7379 }
7380
7381 *pAllocHandle = (VmaAllocHandle)(offset + 1);
7382 // All tests passed: Success. pAllocHandle is already filled.
7383 return true;
7384}
7385
7386void VmaBlockMetadata_Generic::MergeFreeWithNext(VmaSuballocationList::iterator item)
7387{
7388 VMA_ASSERT(item != m_Suballocations.end());
7389 VMA_ASSERT(item->type == VMA_SUBALLOCATION_TYPE_FREE);
7390
7391 VmaSuballocationList::iterator nextItem = item;
7392 ++nextItem;
7393 VMA_ASSERT(nextItem != m_Suballocations.end());
7394 VMA_ASSERT(nextItem->type == VMA_SUBALLOCATION_TYPE_FREE);
7395
7396 item->size += nextItem->size;
7397 --m_FreeCount;
7398 m_Suballocations.erase(nextItem);
7399}
7400
7401VmaSuballocationList::iterator VmaBlockMetadata_Generic::FreeSuballocation(VmaSuballocationList::iterator suballocItem)
7402{
7403 // Change this suballocation to be marked as free.
7404 VmaSuballocation& suballoc = *suballocItem;
7405 suballoc.type = VMA_SUBALLOCATION_TYPE_FREE;
7406 suballoc.userData = VMA_NULL;
7407
7408 // Update totals.
7409 ++m_FreeCount;
7410 m_SumFreeSize += suballoc.size;
7411
7412 // Merge with previous and/or next suballocation if it's also free.
7413 bool mergeWithNext = false;
7414 bool mergeWithPrev = false;
7415
7416 VmaSuballocationList::iterator nextItem = suballocItem;
7417 ++nextItem;
7418 if ((nextItem != m_Suballocations.end()) && (nextItem->type == VMA_SUBALLOCATION_TYPE_FREE))
7419 {
7420 mergeWithNext = true;
7421 }
7422
7423 VmaSuballocationList::iterator prevItem = suballocItem;
7424 if (suballocItem != m_Suballocations.begin())
7425 {
7426 --prevItem;
7427 if (prevItem->type == VMA_SUBALLOCATION_TYPE_FREE)
7428 {
7429 mergeWithPrev = true;
7430 }
7431 }
7432
7433 if (mergeWithNext)
7434 {
7435 UnregisterFreeSuballocation(nextItem);
7436 MergeFreeWithNext(suballocItem);
7437 }
7438
7439 if (mergeWithPrev)
7440 {
7441 UnregisterFreeSuballocation(prevItem);
7442 MergeFreeWithNext(prevItem);
7443 RegisterFreeSuballocation(prevItem);
7444 return prevItem;
7445 }
7446 else
7447 {
7448 RegisterFreeSuballocation(suballocItem);
7449 return suballocItem;
7450 }
7451}
7452
7453void VmaBlockMetadata_Generic::RegisterFreeSuballocation(VmaSuballocationList::iterator item)
7454{
7455 VMA_ASSERT(item->type == VMA_SUBALLOCATION_TYPE_FREE);
7456 VMA_ASSERT(item->size > 0);
7457
7458 // You may want to enable this validation at the beginning or at the end of
7459 // this function, depending on what do you want to check.
7460 VMA_HEAVY_ASSERT(ValidateFreeSuballocationList());
7461
7462 if (m_FreeSuballocationsBySize.empty())
7463 {
7464 m_FreeSuballocationsBySize.push_back(item);
7465 }
7466 else
7467 {
7468 VmaVectorInsertSorted<VmaSuballocationItemSizeLess>(m_FreeSuballocationsBySize, item);
7469 }
7470
7471 //VMA_HEAVY_ASSERT(ValidateFreeSuballocationList());
7472}
7473
7474void VmaBlockMetadata_Generic::UnregisterFreeSuballocation(VmaSuballocationList::iterator item)
7475{
7476 VMA_ASSERT(item->type == VMA_SUBALLOCATION_TYPE_FREE);
7477 VMA_ASSERT(item->size > 0);
7478
7479 // You may want to enable this validation at the beginning or at the end of
7480 // this function, depending on what do you want to check.
7481 VMA_HEAVY_ASSERT(ValidateFreeSuballocationList());
7482
7483 VmaSuballocationList::iterator* const it = VmaBinaryFindFirstNotLess(
7484 m_FreeSuballocationsBySize.data(),
7485 m_FreeSuballocationsBySize.data() + m_FreeSuballocationsBySize.size(),
7486 item,
7487 VmaSuballocationItemSizeLess());
7488 for (size_t index = it - m_FreeSuballocationsBySize.data();
7489 index < m_FreeSuballocationsBySize.size();
7490 ++index)
7491 {
7492 if (m_FreeSuballocationsBySize[index] == item)
7493 {
7494 VmaVectorRemove(m_FreeSuballocationsBySize, index);
7495 return;
7496 }
7497 VMA_ASSERT((m_FreeSuballocationsBySize[index]->size == item->size) && "Not found.");
7498 }
7499 VMA_ASSERT(0 && "Not found.");
7500
7501 //VMA_HEAVY_ASSERT(ValidateFreeSuballocationList());
7502}
7503#endif // _VMA_BLOCK_METADATA_GENERIC_FUNCTIONS
7504#endif // _VMA_BLOCK_METADATA_GENERIC
7505#endif // #if 0
7506
7507#ifndef _VMA_BLOCK_METADATA_LINEAR
7508/*
7509Allocations and their references in internal data structure look like this:
7510
7511if(m_2ndVectorMode == SECOND_VECTOR_EMPTY):
7512
7513 0 +-------+
7514 | |
7515 | |
7516 | |
7517 +-------+
7518 | Alloc | 1st[m_1stNullItemsBeginCount]
7519 +-------+
7520 | Alloc | 1st[m_1stNullItemsBeginCount + 1]
7521 +-------+
7522 | ... |
7523 +-------+
7524 | Alloc | 1st[1st.size() - 1]
7525 +-------+
7526 | |
7527 | |
7528 | |
7529GetSize() +-------+
7530
7531if(m_2ndVectorMode == SECOND_VECTOR_RING_BUFFER):
7532
7533 0 +-------+
7534 | Alloc | 2nd[0]
7535 +-------+
7536 | Alloc | 2nd[1]
7537 +-------+
7538 | ... |
7539 +-------+
7540 | Alloc | 2nd[2nd.size() - 1]
7541 +-------+
7542 | |
7543 | |
7544 | |
7545 +-------+
7546 | Alloc | 1st[m_1stNullItemsBeginCount]
7547 +-------+
7548 | Alloc | 1st[m_1stNullItemsBeginCount + 1]
7549 +-------+
7550 | ... |
7551 +-------+
7552 | Alloc | 1st[1st.size() - 1]
7553 +-------+
7554 | |
7555GetSize() +-------+
7556
7557if(m_2ndVectorMode == SECOND_VECTOR_DOUBLE_STACK):
7558
7559 0 +-------+
7560 | |
7561 | |
7562 | |
7563 +-------+
7564 | Alloc | 1st[m_1stNullItemsBeginCount]
7565 +-------+
7566 | Alloc | 1st[m_1stNullItemsBeginCount + 1]
7567 +-------+
7568 | ... |
7569 +-------+
7570 | Alloc | 1st[1st.size() - 1]
7571 +-------+
7572 | |
7573 | |
7574 | |
7575 +-------+
7576 | Alloc | 2nd[2nd.size() - 1]
7577 +-------+
7578 | ... |
7579 +-------+
7580 | Alloc | 2nd[1]
7581 +-------+
7582 | Alloc | 2nd[0]
7583GetSize() +-------+
7584
7585*/
7586class VmaBlockMetadata_Linear : public VmaBlockMetadata
7587{
7588 VMA_CLASS_NO_COPY(VmaBlockMetadata_Linear)
7589public:
7590 VmaBlockMetadata_Linear(const VkAllocationCallbacks* pAllocationCallbacks,
7591 VkDeviceSize bufferImageGranularity, bool isVirtual);
7592 virtual ~VmaBlockMetadata_Linear() = default;
7593
7594 VkDeviceSize GetSumFreeSize() const override { return m_SumFreeSize; }
7595 bool IsEmpty() const override { return GetAllocationCount() == 0; }
7596 VkDeviceSize GetAllocationOffset(VmaAllocHandle allocHandle) const override { return (VkDeviceSize)allocHandle - 1; };
7597
7598 void Init(VkDeviceSize size) override;
7599 bool Validate() const override;
7600 size_t GetAllocationCount() const override;
7601 size_t GetFreeRegionsCount() const override;
7602
7603 void AddDetailedStatistics(VmaDetailedStatistics& inoutStats) const override;
7604 void AddStatistics(VmaStatistics& inoutStats) const override;
7605
7606#if VMA_STATS_STRING_ENABLED
7607 void PrintDetailedMap(class VmaJsonWriter& json) const override;
7608#endif
7609
7610 bool CreateAllocationRequest(
7611 VkDeviceSize allocSize,
7612 VkDeviceSize allocAlignment,
7613 bool upperAddress,
7614 VmaSuballocationType allocType,
7615 uint32_t strategy,
7616 VmaAllocationRequest* pAllocationRequest) override;
7617
7618 VkResult CheckCorruption(const void* pBlockData) override;
7619
7620 void Alloc(
7621 const VmaAllocationRequest& request,
7622 VmaSuballocationType type,
7623 void* userData) override;
7624
7625 void Free(VmaAllocHandle allocHandle) override;
7626 void GetAllocationInfo(VmaAllocHandle allocHandle, VmaVirtualAllocationInfo& outInfo) override;
7627 void* GetAllocationUserData(VmaAllocHandle allocHandle) const override;
7628 VmaAllocHandle GetAllocationListBegin() const override;
7629 VmaAllocHandle GetNextAllocation(VmaAllocHandle prevAlloc) const override;
7630 VkDeviceSize GetNextFreeRegionSize(VmaAllocHandle alloc) const override;
7631 void Clear() override;
7632 void SetAllocationUserData(VmaAllocHandle allocHandle, void* userData) override;
7633 void DebugLogAllAllocations() const override;
7634
7635private:
7636 /*
7637 There are two suballocation vectors, used in ping-pong way.
7638 The one with index m_1stVectorIndex is called 1st.
7639 The one with index (m_1stVectorIndex ^ 1) is called 2nd.
7640 2nd can be non-empty only when 1st is not empty.
7641 When 2nd is not empty, m_2ndVectorMode indicates its mode of operation.
7642 */
7643 typedef VmaVector<VmaSuballocation, VmaStlAllocator<VmaSuballocation>> SuballocationVectorType;
7644
7645 enum SECOND_VECTOR_MODE
7646 {
7647 SECOND_VECTOR_EMPTY,
7648 /*
7649 Suballocations in 2nd vector are created later than the ones in 1st, but they
7650 all have smaller offset.
7651 */
7652 SECOND_VECTOR_RING_BUFFER,
7653 /*
7654 Suballocations in 2nd vector are upper side of double stack.
7655 They all have offsets higher than those in 1st vector.
7656 Top of this stack means smaller offsets, but higher indices in this vector.
7657 */
7658 SECOND_VECTOR_DOUBLE_STACK,
7659 };
7660
7661 VkDeviceSize m_SumFreeSize;
7662 SuballocationVectorType m_Suballocations0, m_Suballocations1;
7663 uint32_t m_1stVectorIndex;
7664 SECOND_VECTOR_MODE m_2ndVectorMode;
7665 // Number of items in 1st vector with hAllocation = null at the beginning.
7666 size_t m_1stNullItemsBeginCount;
7667 // Number of other items in 1st vector with hAllocation = null somewhere in the middle.
7668 size_t m_1stNullItemsMiddleCount;
7669 // Number of items in 2nd vector with hAllocation = null.
7670 size_t m_2ndNullItemsCount;
7671
7672 SuballocationVectorType& AccessSuballocations1st() { return m_1stVectorIndex ? m_Suballocations1 : m_Suballocations0; }
7673 SuballocationVectorType& AccessSuballocations2nd() { return m_1stVectorIndex ? m_Suballocations0 : m_Suballocations1; }
7674 const SuballocationVectorType& AccessSuballocations1st() const { return m_1stVectorIndex ? m_Suballocations1 : m_Suballocations0; }
7675 const SuballocationVectorType& AccessSuballocations2nd() const { return m_1stVectorIndex ? m_Suballocations0 : m_Suballocations1; }
7676
7677 VmaSuballocation& FindSuballocation(VkDeviceSize offset) const;
7678 bool ShouldCompact1st() const;
7679 void CleanupAfterFree();
7680
7681 bool CreateAllocationRequest_LowerAddress(
7682 VkDeviceSize allocSize,
7683 VkDeviceSize allocAlignment,
7684 VmaSuballocationType allocType,
7685 uint32_t strategy,
7686 VmaAllocationRequest* pAllocationRequest);
7687 bool CreateAllocationRequest_UpperAddress(
7688 VkDeviceSize allocSize,
7689 VkDeviceSize allocAlignment,
7690 VmaSuballocationType allocType,
7691 uint32_t strategy,
7692 VmaAllocationRequest* pAllocationRequest);
7693};
7694
7695#ifndef _VMA_BLOCK_METADATA_LINEAR_FUNCTIONS
7696VmaBlockMetadata_Linear::VmaBlockMetadata_Linear(const VkAllocationCallbacks* pAllocationCallbacks,
7697 VkDeviceSize bufferImageGranularity, bool isVirtual)
7698 : VmaBlockMetadata(pAllocationCallbacks, bufferImageGranularity, isVirtual),
7699 m_SumFreeSize(0),
7700 m_Suballocations0(VmaStlAllocator<VmaSuballocation>(pAllocationCallbacks)),
7701 m_Suballocations1(VmaStlAllocator<VmaSuballocation>(pAllocationCallbacks)),
7702 m_1stVectorIndex(0),
7703 m_2ndVectorMode(SECOND_VECTOR_EMPTY),
7704 m_1stNullItemsBeginCount(0),
7705 m_1stNullItemsMiddleCount(0),
7706 m_2ndNullItemsCount(0) {}
7707
7708void VmaBlockMetadata_Linear::Init(VkDeviceSize size)
7709{
7710 VmaBlockMetadata::Init(size);
7711 m_SumFreeSize = size;
7712}
7713
7714bool VmaBlockMetadata_Linear::Validate() const
7715{
7716 const SuballocationVectorType& suballocations1st = AccessSuballocations1st();
7717 const SuballocationVectorType& suballocations2nd = AccessSuballocations2nd();
7718
7719 VMA_VALIDATE(suballocations2nd.empty() == (m_2ndVectorMode == SECOND_VECTOR_EMPTY));
7720 VMA_VALIDATE(!suballocations1st.empty() ||
7721 suballocations2nd.empty() ||
7722 m_2ndVectorMode != SECOND_VECTOR_RING_BUFFER);
7723
7724 if (!suballocations1st.empty())
7725 {
7726 // Null item at the beginning should be accounted into m_1stNullItemsBeginCount.
7727 VMA_VALIDATE(suballocations1st[m_1stNullItemsBeginCount].type != VMA_SUBALLOCATION_TYPE_FREE);
7728 // Null item at the end should be just pop_back().
7729 VMA_VALIDATE(suballocations1st.back().type != VMA_SUBALLOCATION_TYPE_FREE);
7730 }
7731 if (!suballocations2nd.empty())
7732 {
7733 // Null item at the end should be just pop_back().
7734 VMA_VALIDATE(suballocations2nd.back().type != VMA_SUBALLOCATION_TYPE_FREE);
7735 }
7736
7737 VMA_VALIDATE(m_1stNullItemsBeginCount + m_1stNullItemsMiddleCount <= suballocations1st.size());
7738 VMA_VALIDATE(m_2ndNullItemsCount <= suballocations2nd.size());
7739
7740 VkDeviceSize sumUsedSize = 0;
7741 const size_t suballoc1stCount = suballocations1st.size();
7742 const VkDeviceSize debugMargin = GetDebugMargin();
7743 VkDeviceSize offset = 0;
7744
7745 if (m_2ndVectorMode == SECOND_VECTOR_RING_BUFFER)
7746 {
7747 const size_t suballoc2ndCount = suballocations2nd.size();
7748 size_t nullItem2ndCount = 0;
7749 for (size_t i = 0; i < suballoc2ndCount; ++i)
7750 {
7751 const VmaSuballocation& suballoc = suballocations2nd[i];
7752 const bool currFree = (suballoc.type == VMA_SUBALLOCATION_TYPE_FREE);
7753
7754 VmaAllocation const alloc = (VmaAllocation)suballoc.userData;
7755 if (!IsVirtual())
7756 {
7757 VMA_VALIDATE(currFree == (alloc == VK_NULL_HANDLE));
7758 }
7759 VMA_VALIDATE(suballoc.offset >= offset);
7760
7761 if (!currFree)
7762 {
7763 if (!IsVirtual())
7764 {
7765 VMA_VALIDATE((VkDeviceSize)alloc->GetAllocHandle() == suballoc.offset + 1);
7766 VMA_VALIDATE(alloc->GetSize() == suballoc.size);
7767 }
7768 sumUsedSize += suballoc.size;
7769 }
7770 else
7771 {
7772 ++nullItem2ndCount;
7773 }
7774
7775 offset = suballoc.offset + suballoc.size + debugMargin;
7776 }
7777
7778 VMA_VALIDATE(nullItem2ndCount == m_2ndNullItemsCount);
7779 }
7780
7781 for (size_t i = 0; i < m_1stNullItemsBeginCount; ++i)
7782 {
7783 const VmaSuballocation& suballoc = suballocations1st[i];
7784 VMA_VALIDATE(suballoc.type == VMA_SUBALLOCATION_TYPE_FREE &&
7785 suballoc.userData == VMA_NULL);
7786 }
7787
7788 size_t nullItem1stCount = m_1stNullItemsBeginCount;
7789
7790 for (size_t i = m_1stNullItemsBeginCount; i < suballoc1stCount; ++i)
7791 {
7792 const VmaSuballocation& suballoc = suballocations1st[i];
7793 const bool currFree = (suballoc.type == VMA_SUBALLOCATION_TYPE_FREE);
7794
7795 VmaAllocation const alloc = (VmaAllocation)suballoc.userData;
7796 if (!IsVirtual())
7797 {
7798 VMA_VALIDATE(currFree == (alloc == VK_NULL_HANDLE));
7799 }
7800 VMA_VALIDATE(suballoc.offset >= offset);
7801 VMA_VALIDATE(i >= m_1stNullItemsBeginCount || currFree);
7802
7803 if (!currFree)
7804 {
7805 if (!IsVirtual())
7806 {
7807 VMA_VALIDATE((VkDeviceSize)alloc->GetAllocHandle() == suballoc.offset + 1);
7808 VMA_VALIDATE(alloc->GetSize() == suballoc.size);
7809 }
7810 sumUsedSize += suballoc.size;
7811 }
7812 else
7813 {
7814 ++nullItem1stCount;
7815 }
7816
7817 offset = suballoc.offset + suballoc.size + debugMargin;
7818 }
7819 VMA_VALIDATE(nullItem1stCount == m_1stNullItemsBeginCount + m_1stNullItemsMiddleCount);
7820
7821 if (m_2ndVectorMode == SECOND_VECTOR_DOUBLE_STACK)
7822 {
7823 const size_t suballoc2ndCount = suballocations2nd.size();
7824 size_t nullItem2ndCount = 0;
7825 for (size_t i = suballoc2ndCount; i--; )
7826 {
7827 const VmaSuballocation& suballoc = suballocations2nd[i];
7828 const bool currFree = (suballoc.type == VMA_SUBALLOCATION_TYPE_FREE);
7829
7830 VmaAllocation const alloc = (VmaAllocation)suballoc.userData;
7831 if (!IsVirtual())
7832 {
7833 VMA_VALIDATE(currFree == (alloc == VK_NULL_HANDLE));
7834 }
7835 VMA_VALIDATE(suballoc.offset >= offset);
7836
7837 if (!currFree)
7838 {
7839 if (!IsVirtual())
7840 {
7841 VMA_VALIDATE((VkDeviceSize)alloc->GetAllocHandle() == suballoc.offset + 1);
7842 VMA_VALIDATE(alloc->GetSize() == suballoc.size);
7843 }
7844 sumUsedSize += suballoc.size;
7845 }
7846 else
7847 {
7848 ++nullItem2ndCount;
7849 }
7850
7851 offset = suballoc.offset + suballoc.size + debugMargin;
7852 }
7853
7854 VMA_VALIDATE(nullItem2ndCount == m_2ndNullItemsCount);
7855 }
7856
7857 VMA_VALIDATE(offset <= GetSize());
7858 VMA_VALIDATE(m_SumFreeSize == GetSize() - sumUsedSize);
7859
7860 return true;
7861}
7862
7863size_t VmaBlockMetadata_Linear::GetAllocationCount() const
7864{
7865 return AccessSuballocations1st().size() - m_1stNullItemsBeginCount - m_1stNullItemsMiddleCount +
7866 AccessSuballocations2nd().size() - m_2ndNullItemsCount;
7867}
7868
7869size_t VmaBlockMetadata_Linear::GetFreeRegionsCount() const
7870{
7871 // Function only used for defragmentation, which is disabled for this algorithm
7872 VMA_ASSERT(0);
7873 return SIZE_MAX;
7874}
7875
7876void VmaBlockMetadata_Linear::AddDetailedStatistics(VmaDetailedStatistics& inoutStats) const
7877{
7878 const VkDeviceSize size = GetSize();
7879 const SuballocationVectorType& suballocations1st = AccessSuballocations1st();
7880 const SuballocationVectorType& suballocations2nd = AccessSuballocations2nd();
7881 const size_t suballoc1stCount = suballocations1st.size();
7882 const size_t suballoc2ndCount = suballocations2nd.size();
7883
7884 inoutStats.statistics.blockCount++;
7885 inoutStats.statistics.blockBytes += size;
7886
7887 VkDeviceSize lastOffset = 0;
7888
7889 if (m_2ndVectorMode == SECOND_VECTOR_RING_BUFFER)
7890 {
7891 const VkDeviceSize freeSpace2ndTo1stEnd = suballocations1st[m_1stNullItemsBeginCount].offset;
7892 size_t nextAlloc2ndIndex = 0;
7893 while (lastOffset < freeSpace2ndTo1stEnd)
7894 {
7895 // Find next non-null allocation or move nextAllocIndex to the end.
7896 while (nextAlloc2ndIndex < suballoc2ndCount &&
7897 suballocations2nd[nextAlloc2ndIndex].userData == VMA_NULL)
7898 {
7899 ++nextAlloc2ndIndex;
7900 }
7901
7902 // Found non-null allocation.
7903 if (nextAlloc2ndIndex < suballoc2ndCount)
7904 {
7905 const VmaSuballocation& suballoc = suballocations2nd[nextAlloc2ndIndex];
7906
7907 // 1. Process free space before this allocation.
7908 if (lastOffset < suballoc.offset)
7909 {
7910 // There is free space from lastOffset to suballoc.offset.
7911 const VkDeviceSize unusedRangeSize = suballoc.offset - lastOffset;
7912 VmaAddDetailedStatisticsUnusedRange(inoutStats, size: unusedRangeSize);
7913 }
7914
7915 // 2. Process this allocation.
7916 // There is allocation with suballoc.offset, suballoc.size.
7917 VmaAddDetailedStatisticsAllocation(inoutStats, size: suballoc.size);
7918
7919 // 3. Prepare for next iteration.
7920 lastOffset = suballoc.offset + suballoc.size;
7921 ++nextAlloc2ndIndex;
7922 }
7923 // We are at the end.
7924 else
7925 {
7926 // There is free space from lastOffset to freeSpace2ndTo1stEnd.
7927 if (lastOffset < freeSpace2ndTo1stEnd)
7928 {
7929 const VkDeviceSize unusedRangeSize = freeSpace2ndTo1stEnd - lastOffset;
7930 VmaAddDetailedStatisticsUnusedRange(inoutStats, size: unusedRangeSize);
7931 }
7932
7933 // End of loop.
7934 lastOffset = freeSpace2ndTo1stEnd;
7935 }
7936 }
7937 }
7938
7939 size_t nextAlloc1stIndex = m_1stNullItemsBeginCount;
7940 const VkDeviceSize freeSpace1stTo2ndEnd =
7941 m_2ndVectorMode == SECOND_VECTOR_DOUBLE_STACK ? suballocations2nd.back().offset : size;
7942 while (lastOffset < freeSpace1stTo2ndEnd)
7943 {
7944 // Find next non-null allocation or move nextAllocIndex to the end.
7945 while (nextAlloc1stIndex < suballoc1stCount &&
7946 suballocations1st[nextAlloc1stIndex].userData == VMA_NULL)
7947 {
7948 ++nextAlloc1stIndex;
7949 }
7950
7951 // Found non-null allocation.
7952 if (nextAlloc1stIndex < suballoc1stCount)
7953 {
7954 const VmaSuballocation& suballoc = suballocations1st[nextAlloc1stIndex];
7955
7956 // 1. Process free space before this allocation.
7957 if (lastOffset < suballoc.offset)
7958 {
7959 // There is free space from lastOffset to suballoc.offset.
7960 const VkDeviceSize unusedRangeSize = suballoc.offset - lastOffset;
7961 VmaAddDetailedStatisticsUnusedRange(inoutStats, size: unusedRangeSize);
7962 }
7963
7964 // 2. Process this allocation.
7965 // There is allocation with suballoc.offset, suballoc.size.
7966 VmaAddDetailedStatisticsAllocation(inoutStats, size: suballoc.size);
7967
7968 // 3. Prepare for next iteration.
7969 lastOffset = suballoc.offset + suballoc.size;
7970 ++nextAlloc1stIndex;
7971 }
7972 // We are at the end.
7973 else
7974 {
7975 // There is free space from lastOffset to freeSpace1stTo2ndEnd.
7976 if (lastOffset < freeSpace1stTo2ndEnd)
7977 {
7978 const VkDeviceSize unusedRangeSize = freeSpace1stTo2ndEnd - lastOffset;
7979 VmaAddDetailedStatisticsUnusedRange(inoutStats, size: unusedRangeSize);
7980 }
7981
7982 // End of loop.
7983 lastOffset = freeSpace1stTo2ndEnd;
7984 }
7985 }
7986
7987 if (m_2ndVectorMode == SECOND_VECTOR_DOUBLE_STACK)
7988 {
7989 size_t nextAlloc2ndIndex = suballocations2nd.size() - 1;
7990 while (lastOffset < size)
7991 {
7992 // Find next non-null allocation or move nextAllocIndex to the end.
7993 while (nextAlloc2ndIndex != SIZE_MAX &&
7994 suballocations2nd[nextAlloc2ndIndex].userData == VMA_NULL)
7995 {
7996 --nextAlloc2ndIndex;
7997 }
7998
7999 // Found non-null allocation.
8000 if (nextAlloc2ndIndex != SIZE_MAX)
8001 {
8002 const VmaSuballocation& suballoc = suballocations2nd[nextAlloc2ndIndex];
8003
8004 // 1. Process free space before this allocation.
8005 if (lastOffset < suballoc.offset)
8006 {
8007 // There is free space from lastOffset to suballoc.offset.
8008 const VkDeviceSize unusedRangeSize = suballoc.offset - lastOffset;
8009 VmaAddDetailedStatisticsUnusedRange(inoutStats, size: unusedRangeSize);
8010 }
8011
8012 // 2. Process this allocation.
8013 // There is allocation with suballoc.offset, suballoc.size.
8014 VmaAddDetailedStatisticsAllocation(inoutStats, size: suballoc.size);
8015
8016 // 3. Prepare for next iteration.
8017 lastOffset = suballoc.offset + suballoc.size;
8018 --nextAlloc2ndIndex;
8019 }
8020 // We are at the end.
8021 else
8022 {
8023 // There is free space from lastOffset to size.
8024 if (lastOffset < size)
8025 {
8026 const VkDeviceSize unusedRangeSize = size - lastOffset;
8027 VmaAddDetailedStatisticsUnusedRange(inoutStats, size: unusedRangeSize);
8028 }
8029
8030 // End of loop.
8031 lastOffset = size;
8032 }
8033 }
8034 }
8035}
8036
8037void VmaBlockMetadata_Linear::AddStatistics(VmaStatistics& inoutStats) const
8038{
8039 const SuballocationVectorType& suballocations1st = AccessSuballocations1st();
8040 const SuballocationVectorType& suballocations2nd = AccessSuballocations2nd();
8041 const VkDeviceSize size = GetSize();
8042 const size_t suballoc1stCount = suballocations1st.size();
8043 const size_t suballoc2ndCount = suballocations2nd.size();
8044
8045 inoutStats.blockCount++;
8046 inoutStats.blockBytes += size;
8047 inoutStats.allocationBytes += size - m_SumFreeSize;
8048
8049 VkDeviceSize lastOffset = 0;
8050
8051 if (m_2ndVectorMode == SECOND_VECTOR_RING_BUFFER)
8052 {
8053 const VkDeviceSize freeSpace2ndTo1stEnd = suballocations1st[m_1stNullItemsBeginCount].offset;
8054 size_t nextAlloc2ndIndex = m_1stNullItemsBeginCount;
8055 while (lastOffset < freeSpace2ndTo1stEnd)
8056 {
8057 // Find next non-null allocation or move nextAlloc2ndIndex to the end.
8058 while (nextAlloc2ndIndex < suballoc2ndCount &&
8059 suballocations2nd[nextAlloc2ndIndex].userData == VMA_NULL)
8060 {
8061 ++nextAlloc2ndIndex;
8062 }
8063
8064 // Found non-null allocation.
8065 if (nextAlloc2ndIndex < suballoc2ndCount)
8066 {
8067 const VmaSuballocation& suballoc = suballocations2nd[nextAlloc2ndIndex];
8068
8069 // 1. Process free space before this allocation.
8070 if (lastOffset < suballoc.offset)
8071 {
8072 // There is free space from lastOffset to suballoc.offset.
8073 const VkDeviceSize unusedRangeSize = suballoc.offset - lastOffset;
8074 }
8075
8076 // 2. Process this allocation.
8077 // There is allocation with suballoc.offset, suballoc.size.
8078 ++inoutStats.allocationCount;
8079
8080 // 3. Prepare for next iteration.
8081 lastOffset = suballoc.offset + suballoc.size;
8082 ++nextAlloc2ndIndex;
8083 }
8084 // We are at the end.
8085 else
8086 {
8087 if (lastOffset < freeSpace2ndTo1stEnd)
8088 {
8089 // There is free space from lastOffset to freeSpace2ndTo1stEnd.
8090 const VkDeviceSize unusedRangeSize = freeSpace2ndTo1stEnd - lastOffset;
8091 }
8092
8093 // End of loop.
8094 lastOffset = freeSpace2ndTo1stEnd;
8095 }
8096 }
8097 }
8098
8099 size_t nextAlloc1stIndex = m_1stNullItemsBeginCount;
8100 const VkDeviceSize freeSpace1stTo2ndEnd =
8101 m_2ndVectorMode == SECOND_VECTOR_DOUBLE_STACK ? suballocations2nd.back().offset : size;
8102 while (lastOffset < freeSpace1stTo2ndEnd)
8103 {
8104 // Find next non-null allocation or move nextAllocIndex to the end.
8105 while (nextAlloc1stIndex < suballoc1stCount &&
8106 suballocations1st[nextAlloc1stIndex].userData == VMA_NULL)
8107 {
8108 ++nextAlloc1stIndex;
8109 }
8110
8111 // Found non-null allocation.
8112 if (nextAlloc1stIndex < suballoc1stCount)
8113 {
8114 const VmaSuballocation& suballoc = suballocations1st[nextAlloc1stIndex];
8115
8116 // 1. Process free space before this allocation.
8117 if (lastOffset < suballoc.offset)
8118 {
8119 // There is free space from lastOffset to suballoc.offset.
8120 const VkDeviceSize unusedRangeSize = suballoc.offset - lastOffset;
8121 }
8122
8123 // 2. Process this allocation.
8124 // There is allocation with suballoc.offset, suballoc.size.
8125 ++inoutStats.allocationCount;
8126
8127 // 3. Prepare for next iteration.
8128 lastOffset = suballoc.offset + suballoc.size;
8129 ++nextAlloc1stIndex;
8130 }
8131 // We are at the end.
8132 else
8133 {
8134 if (lastOffset < freeSpace1stTo2ndEnd)
8135 {
8136 // There is free space from lastOffset to freeSpace1stTo2ndEnd.
8137 const VkDeviceSize unusedRangeSize = freeSpace1stTo2ndEnd - lastOffset;
8138 }
8139
8140 // End of loop.
8141 lastOffset = freeSpace1stTo2ndEnd;
8142 }
8143 }
8144
8145 if (m_2ndVectorMode == SECOND_VECTOR_DOUBLE_STACK)
8146 {
8147 size_t nextAlloc2ndIndex = suballocations2nd.size() - 1;
8148 while (lastOffset < size)
8149 {
8150 // Find next non-null allocation or move nextAlloc2ndIndex to the end.
8151 while (nextAlloc2ndIndex != SIZE_MAX &&
8152 suballocations2nd[nextAlloc2ndIndex].userData == VMA_NULL)
8153 {
8154 --nextAlloc2ndIndex;
8155 }
8156
8157 // Found non-null allocation.
8158 if (nextAlloc2ndIndex != SIZE_MAX)
8159 {
8160 const VmaSuballocation& suballoc = suballocations2nd[nextAlloc2ndIndex];
8161
8162 // 1. Process free space before this allocation.
8163 if (lastOffset < suballoc.offset)
8164 {
8165 // There is free space from lastOffset to suballoc.offset.
8166 const VkDeviceSize unusedRangeSize = suballoc.offset - lastOffset;
8167 }
8168
8169 // 2. Process this allocation.
8170 // There is allocation with suballoc.offset, suballoc.size.
8171 ++inoutStats.allocationCount;
8172
8173 // 3. Prepare for next iteration.
8174 lastOffset = suballoc.offset + suballoc.size;
8175 --nextAlloc2ndIndex;
8176 }
8177 // We are at the end.
8178 else
8179 {
8180 if (lastOffset < size)
8181 {
8182 // There is free space from lastOffset to size.
8183 const VkDeviceSize unusedRangeSize = size - lastOffset;
8184 }
8185
8186 // End of loop.
8187 lastOffset = size;
8188 }
8189 }
8190 }
8191}
8192
8193#if VMA_STATS_STRING_ENABLED
8194void VmaBlockMetadata_Linear::PrintDetailedMap(class VmaJsonWriter& json) const
8195{
8196 const VkDeviceSize size = GetSize();
8197 const SuballocationVectorType& suballocations1st = AccessSuballocations1st();
8198 const SuballocationVectorType& suballocations2nd = AccessSuballocations2nd();
8199 const size_t suballoc1stCount = suballocations1st.size();
8200 const size_t suballoc2ndCount = suballocations2nd.size();
8201
8202 // FIRST PASS
8203
8204 size_t unusedRangeCount = 0;
8205 VkDeviceSize usedBytes = 0;
8206
8207 VkDeviceSize lastOffset = 0;
8208
8209 size_t alloc2ndCount = 0;
8210 if (m_2ndVectorMode == SECOND_VECTOR_RING_BUFFER)
8211 {
8212 const VkDeviceSize freeSpace2ndTo1stEnd = suballocations1st[m_1stNullItemsBeginCount].offset;
8213 size_t nextAlloc2ndIndex = 0;
8214 while (lastOffset < freeSpace2ndTo1stEnd)
8215 {
8216 // Find next non-null allocation or move nextAlloc2ndIndex to the end.
8217 while (nextAlloc2ndIndex < suballoc2ndCount &&
8218 suballocations2nd[nextAlloc2ndIndex].userData == VMA_NULL)
8219 {
8220 ++nextAlloc2ndIndex;
8221 }
8222
8223 // Found non-null allocation.
8224 if (nextAlloc2ndIndex < suballoc2ndCount)
8225 {
8226 const VmaSuballocation& suballoc = suballocations2nd[nextAlloc2ndIndex];
8227
8228 // 1. Process free space before this allocation.
8229 if (lastOffset < suballoc.offset)
8230 {
8231 // There is free space from lastOffset to suballoc.offset.
8232 ++unusedRangeCount;
8233 }
8234
8235 // 2. Process this allocation.
8236 // There is allocation with suballoc.offset, suballoc.size.
8237 ++alloc2ndCount;
8238 usedBytes += suballoc.size;
8239
8240 // 3. Prepare for next iteration.
8241 lastOffset = suballoc.offset + suballoc.size;
8242 ++nextAlloc2ndIndex;
8243 }
8244 // We are at the end.
8245 else
8246 {
8247 if (lastOffset < freeSpace2ndTo1stEnd)
8248 {
8249 // There is free space from lastOffset to freeSpace2ndTo1stEnd.
8250 ++unusedRangeCount;
8251 }
8252
8253 // End of loop.
8254 lastOffset = freeSpace2ndTo1stEnd;
8255 }
8256 }
8257 }
8258
8259 size_t nextAlloc1stIndex = m_1stNullItemsBeginCount;
8260 size_t alloc1stCount = 0;
8261 const VkDeviceSize freeSpace1stTo2ndEnd =
8262 m_2ndVectorMode == SECOND_VECTOR_DOUBLE_STACK ? suballocations2nd.back().offset : size;
8263 while (lastOffset < freeSpace1stTo2ndEnd)
8264 {
8265 // Find next non-null allocation or move nextAllocIndex to the end.
8266 while (nextAlloc1stIndex < suballoc1stCount &&
8267 suballocations1st[nextAlloc1stIndex].userData == VMA_NULL)
8268 {
8269 ++nextAlloc1stIndex;
8270 }
8271
8272 // Found non-null allocation.
8273 if (nextAlloc1stIndex < suballoc1stCount)
8274 {
8275 const VmaSuballocation& suballoc = suballocations1st[nextAlloc1stIndex];
8276
8277 // 1. Process free space before this allocation.
8278 if (lastOffset < suballoc.offset)
8279 {
8280 // There is free space from lastOffset to suballoc.offset.
8281 ++unusedRangeCount;
8282 }
8283
8284 // 2. Process this allocation.
8285 // There is allocation with suballoc.offset, suballoc.size.
8286 ++alloc1stCount;
8287 usedBytes += suballoc.size;
8288
8289 // 3. Prepare for next iteration.
8290 lastOffset = suballoc.offset + suballoc.size;
8291 ++nextAlloc1stIndex;
8292 }
8293 // We are at the end.
8294 else
8295 {
8296 if (lastOffset < size)
8297 {
8298 // There is free space from lastOffset to freeSpace1stTo2ndEnd.
8299 ++unusedRangeCount;
8300 }
8301
8302 // End of loop.
8303 lastOffset = freeSpace1stTo2ndEnd;
8304 }
8305 }
8306
8307 if (m_2ndVectorMode == SECOND_VECTOR_DOUBLE_STACK)
8308 {
8309 size_t nextAlloc2ndIndex = suballocations2nd.size() - 1;
8310 while (lastOffset < size)
8311 {
8312 // Find next non-null allocation or move nextAlloc2ndIndex to the end.
8313 while (nextAlloc2ndIndex != SIZE_MAX &&
8314 suballocations2nd[nextAlloc2ndIndex].userData == VMA_NULL)
8315 {
8316 --nextAlloc2ndIndex;
8317 }
8318
8319 // Found non-null allocation.
8320 if (nextAlloc2ndIndex != SIZE_MAX)
8321 {
8322 const VmaSuballocation& suballoc = suballocations2nd[nextAlloc2ndIndex];
8323
8324 // 1. Process free space before this allocation.
8325 if (lastOffset < suballoc.offset)
8326 {
8327 // There is free space from lastOffset to suballoc.offset.
8328 ++unusedRangeCount;
8329 }
8330
8331 // 2. Process this allocation.
8332 // There is allocation with suballoc.offset, suballoc.size.
8333 ++alloc2ndCount;
8334 usedBytes += suballoc.size;
8335
8336 // 3. Prepare for next iteration.
8337 lastOffset = suballoc.offset + suballoc.size;
8338 --nextAlloc2ndIndex;
8339 }
8340 // We are at the end.
8341 else
8342 {
8343 if (lastOffset < size)
8344 {
8345 // There is free space from lastOffset to size.
8346 ++unusedRangeCount;
8347 }
8348
8349 // End of loop.
8350 lastOffset = size;
8351 }
8352 }
8353 }
8354
8355 const VkDeviceSize unusedBytes = size - usedBytes;
8356 PrintDetailedMap_Begin(json, unusedBytes, allocationCount: alloc1stCount + alloc2ndCount, unusedRangeCount);
8357
8358 // SECOND PASS
8359 lastOffset = 0;
8360
8361 if (m_2ndVectorMode == SECOND_VECTOR_RING_BUFFER)
8362 {
8363 const VkDeviceSize freeSpace2ndTo1stEnd = suballocations1st[m_1stNullItemsBeginCount].offset;
8364 size_t nextAlloc2ndIndex = 0;
8365 while (lastOffset < freeSpace2ndTo1stEnd)
8366 {
8367 // Find next non-null allocation or move nextAlloc2ndIndex to the end.
8368 while (nextAlloc2ndIndex < suballoc2ndCount &&
8369 suballocations2nd[nextAlloc2ndIndex].userData == VMA_NULL)
8370 {
8371 ++nextAlloc2ndIndex;
8372 }
8373
8374 // Found non-null allocation.
8375 if (nextAlloc2ndIndex < suballoc2ndCount)
8376 {
8377 const VmaSuballocation& suballoc = suballocations2nd[nextAlloc2ndIndex];
8378
8379 // 1. Process free space before this allocation.
8380 if (lastOffset < suballoc.offset)
8381 {
8382 // There is free space from lastOffset to suballoc.offset.
8383 const VkDeviceSize unusedRangeSize = suballoc.offset - lastOffset;
8384 PrintDetailedMap_UnusedRange(json, offset: lastOffset, size: unusedRangeSize);
8385 }
8386
8387 // 2. Process this allocation.
8388 // There is allocation with suballoc.offset, suballoc.size.
8389 PrintDetailedMap_Allocation(json, offset: suballoc.offset, size: suballoc.size, userData: suballoc.userData);
8390
8391 // 3. Prepare for next iteration.
8392 lastOffset = suballoc.offset + suballoc.size;
8393 ++nextAlloc2ndIndex;
8394 }
8395 // We are at the end.
8396 else
8397 {
8398 if (lastOffset < freeSpace2ndTo1stEnd)
8399 {
8400 // There is free space from lastOffset to freeSpace2ndTo1stEnd.
8401 const VkDeviceSize unusedRangeSize = freeSpace2ndTo1stEnd - lastOffset;
8402 PrintDetailedMap_UnusedRange(json, offset: lastOffset, size: unusedRangeSize);
8403 }
8404
8405 // End of loop.
8406 lastOffset = freeSpace2ndTo1stEnd;
8407 }
8408 }
8409 }
8410
8411 nextAlloc1stIndex = m_1stNullItemsBeginCount;
8412 while (lastOffset < freeSpace1stTo2ndEnd)
8413 {
8414 // Find next non-null allocation or move nextAllocIndex to the end.
8415 while (nextAlloc1stIndex < suballoc1stCount &&
8416 suballocations1st[nextAlloc1stIndex].userData == VMA_NULL)
8417 {
8418 ++nextAlloc1stIndex;
8419 }
8420
8421 // Found non-null allocation.
8422 if (nextAlloc1stIndex < suballoc1stCount)
8423 {
8424 const VmaSuballocation& suballoc = suballocations1st[nextAlloc1stIndex];
8425
8426 // 1. Process free space before this allocation.
8427 if (lastOffset < suballoc.offset)
8428 {
8429 // There is free space from lastOffset to suballoc.offset.
8430 const VkDeviceSize unusedRangeSize = suballoc.offset - lastOffset;
8431 PrintDetailedMap_UnusedRange(json, offset: lastOffset, size: unusedRangeSize);
8432 }
8433
8434 // 2. Process this allocation.
8435 // There is allocation with suballoc.offset, suballoc.size.
8436 PrintDetailedMap_Allocation(json, offset: suballoc.offset, size: suballoc.size, userData: suballoc.userData);
8437
8438 // 3. Prepare for next iteration.
8439 lastOffset = suballoc.offset + suballoc.size;
8440 ++nextAlloc1stIndex;
8441 }
8442 // We are at the end.
8443 else
8444 {
8445 if (lastOffset < freeSpace1stTo2ndEnd)
8446 {
8447 // There is free space from lastOffset to freeSpace1stTo2ndEnd.
8448 const VkDeviceSize unusedRangeSize = freeSpace1stTo2ndEnd - lastOffset;
8449 PrintDetailedMap_UnusedRange(json, offset: lastOffset, size: unusedRangeSize);
8450 }
8451
8452 // End of loop.
8453 lastOffset = freeSpace1stTo2ndEnd;
8454 }
8455 }
8456
8457 if (m_2ndVectorMode == SECOND_VECTOR_DOUBLE_STACK)
8458 {
8459 size_t nextAlloc2ndIndex = suballocations2nd.size() - 1;
8460 while (lastOffset < size)
8461 {
8462 // Find next non-null allocation or move nextAlloc2ndIndex to the end.
8463 while (nextAlloc2ndIndex != SIZE_MAX &&
8464 suballocations2nd[nextAlloc2ndIndex].userData == VMA_NULL)
8465 {
8466 --nextAlloc2ndIndex;
8467 }
8468
8469 // Found non-null allocation.
8470 if (nextAlloc2ndIndex != SIZE_MAX)
8471 {
8472 const VmaSuballocation& suballoc = suballocations2nd[nextAlloc2ndIndex];
8473
8474 // 1. Process free space before this allocation.
8475 if (lastOffset < suballoc.offset)
8476 {
8477 // There is free space from lastOffset to suballoc.offset.
8478 const VkDeviceSize unusedRangeSize = suballoc.offset - lastOffset;
8479 PrintDetailedMap_UnusedRange(json, offset: lastOffset, size: unusedRangeSize);
8480 }
8481
8482 // 2. Process this allocation.
8483 // There is allocation with suballoc.offset, suballoc.size.
8484 PrintDetailedMap_Allocation(json, offset: suballoc.offset, size: suballoc.size, userData: suballoc.userData);
8485
8486 // 3. Prepare for next iteration.
8487 lastOffset = suballoc.offset + suballoc.size;
8488 --nextAlloc2ndIndex;
8489 }
8490 // We are at the end.
8491 else
8492 {
8493 if (lastOffset < size)
8494 {
8495 // There is free space from lastOffset to size.
8496 const VkDeviceSize unusedRangeSize = size - lastOffset;
8497 PrintDetailedMap_UnusedRange(json, offset: lastOffset, size: unusedRangeSize);
8498 }
8499
8500 // End of loop.
8501 lastOffset = size;
8502 }
8503 }
8504 }
8505
8506 PrintDetailedMap_End(json);
8507}
8508#endif // VMA_STATS_STRING_ENABLED
8509
8510bool VmaBlockMetadata_Linear::CreateAllocationRequest(
8511 VkDeviceSize allocSize,
8512 VkDeviceSize allocAlignment,
8513 bool upperAddress,
8514 VmaSuballocationType allocType,
8515 uint32_t strategy,
8516 VmaAllocationRequest* pAllocationRequest)
8517{
8518 VMA_ASSERT(allocSize > 0);
8519 VMA_ASSERT(allocType != VMA_SUBALLOCATION_TYPE_FREE);
8520 VMA_ASSERT(pAllocationRequest != VMA_NULL);
8521 VMA_HEAVY_ASSERT(Validate());
8522 pAllocationRequest->size = allocSize;
8523 return upperAddress ?
8524 CreateAllocationRequest_UpperAddress(
8525 allocSize, allocAlignment, allocType, strategy, pAllocationRequest) :
8526 CreateAllocationRequest_LowerAddress(
8527 allocSize, allocAlignment, allocType, strategy, pAllocationRequest);
8528}
8529
8530VkResult VmaBlockMetadata_Linear::CheckCorruption(const void* pBlockData)
8531{
8532 VMA_ASSERT(!IsVirtual());
8533 SuballocationVectorType& suballocations1st = AccessSuballocations1st();
8534 for (size_t i = m_1stNullItemsBeginCount, count = suballocations1st.size(); i < count; ++i)
8535 {
8536 const VmaSuballocation& suballoc = suballocations1st[i];
8537 if (suballoc.type != VMA_SUBALLOCATION_TYPE_FREE)
8538 {
8539 if (!VmaValidateMagicValue(pData: pBlockData, offset: suballoc.offset + suballoc.size))
8540 {
8541 VMA_ASSERT(0 && "MEMORY CORRUPTION DETECTED AFTER VALIDATED ALLOCATION!");
8542 return VK_ERROR_UNKNOWN_COPY;
8543 }
8544 }
8545 }
8546
8547 SuballocationVectorType& suballocations2nd = AccessSuballocations2nd();
8548 for (size_t i = 0, count = suballocations2nd.size(); i < count; ++i)
8549 {
8550 const VmaSuballocation& suballoc = suballocations2nd[i];
8551 if (suballoc.type != VMA_SUBALLOCATION_TYPE_FREE)
8552 {
8553 if (!VmaValidateMagicValue(pData: pBlockData, offset: suballoc.offset + suballoc.size))
8554 {
8555 VMA_ASSERT(0 && "MEMORY CORRUPTION DETECTED AFTER VALIDATED ALLOCATION!");
8556 return VK_ERROR_UNKNOWN_COPY;
8557 }
8558 }
8559 }
8560
8561 return VK_SUCCESS;
8562}
8563
8564void VmaBlockMetadata_Linear::Alloc(
8565 const VmaAllocationRequest& request,
8566 VmaSuballocationType type,
8567 void* userData)
8568{
8569 const VkDeviceSize offset = (VkDeviceSize)request.allocHandle - 1;
8570 const VmaSuballocation newSuballoc = { .offset: offset, .size: request.size, .userData: userData, .type: type };
8571
8572 switch (request.type)
8573 {
8574 case VmaAllocationRequestType::UpperAddress:
8575 {
8576 VMA_ASSERT(m_2ndVectorMode != SECOND_VECTOR_RING_BUFFER &&
8577 "CRITICAL ERROR: Trying to use linear allocator as double stack while it was already used as ring buffer.");
8578 SuballocationVectorType& suballocations2nd = AccessSuballocations2nd();
8579 suballocations2nd.push_back(src: newSuballoc);
8580 m_2ndVectorMode = SECOND_VECTOR_DOUBLE_STACK;
8581 }
8582 break;
8583 case VmaAllocationRequestType::EndOf1st:
8584 {
8585 SuballocationVectorType& suballocations1st = AccessSuballocations1st();
8586
8587 VMA_ASSERT(suballocations1st.empty() ||
8588 offset >= suballocations1st.back().offset + suballocations1st.back().size);
8589 // Check if it fits before the end of the block.
8590 VMA_ASSERT(offset + request.size <= GetSize());
8591
8592 suballocations1st.push_back(src: newSuballoc);
8593 }
8594 break;
8595 case VmaAllocationRequestType::EndOf2nd:
8596 {
8597 SuballocationVectorType& suballocations1st = AccessSuballocations1st();
8598 // New allocation at the end of 2-part ring buffer, so before first allocation from 1st vector.
8599 VMA_ASSERT(!suballocations1st.empty() &&
8600 offset + request.size <= suballocations1st[m_1stNullItemsBeginCount].offset);
8601 SuballocationVectorType& suballocations2nd = AccessSuballocations2nd();
8602
8603 switch (m_2ndVectorMode)
8604 {
8605 case SECOND_VECTOR_EMPTY:
8606 // First allocation from second part ring buffer.
8607 VMA_ASSERT(suballocations2nd.empty());
8608 m_2ndVectorMode = SECOND_VECTOR_RING_BUFFER;
8609 break;
8610 case SECOND_VECTOR_RING_BUFFER:
8611 // 2-part ring buffer is already started.
8612 VMA_ASSERT(!suballocations2nd.empty());
8613 break;
8614 case SECOND_VECTOR_DOUBLE_STACK:
8615 VMA_ASSERT(0 && "CRITICAL ERROR: Trying to use linear allocator as ring buffer while it was already used as double stack.");
8616 break;
8617 default:
8618 VMA_ASSERT(0);
8619 }
8620
8621 suballocations2nd.push_back(src: newSuballoc);
8622 }
8623 break;
8624 default:
8625 VMA_ASSERT(0 && "CRITICAL INTERNAL ERROR.");
8626 }
8627
8628 m_SumFreeSize -= newSuballoc.size;
8629}
8630
8631void VmaBlockMetadata_Linear::Free(VmaAllocHandle allocHandle)
8632{
8633 SuballocationVectorType& suballocations1st = AccessSuballocations1st();
8634 SuballocationVectorType& suballocations2nd = AccessSuballocations2nd();
8635 VkDeviceSize offset = (VkDeviceSize)allocHandle - 1;
8636
8637 if (!suballocations1st.empty())
8638 {
8639 // First allocation: Mark it as next empty at the beginning.
8640 VmaSuballocation& firstSuballoc = suballocations1st[m_1stNullItemsBeginCount];
8641 if (firstSuballoc.offset == offset)
8642 {
8643 firstSuballoc.type = VMA_SUBALLOCATION_TYPE_FREE;
8644 firstSuballoc.userData = VMA_NULL;
8645 m_SumFreeSize += firstSuballoc.size;
8646 ++m_1stNullItemsBeginCount;
8647 CleanupAfterFree();
8648 return;
8649 }
8650 }
8651
8652 // Last allocation in 2-part ring buffer or top of upper stack (same logic).
8653 if (m_2ndVectorMode == SECOND_VECTOR_RING_BUFFER ||
8654 m_2ndVectorMode == SECOND_VECTOR_DOUBLE_STACK)
8655 {
8656 VmaSuballocation& lastSuballoc = suballocations2nd.back();
8657 if (lastSuballoc.offset == offset)
8658 {
8659 m_SumFreeSize += lastSuballoc.size;
8660 suballocations2nd.pop_back();
8661 CleanupAfterFree();
8662 return;
8663 }
8664 }
8665 // Last allocation in 1st vector.
8666 else if (m_2ndVectorMode == SECOND_VECTOR_EMPTY)
8667 {
8668 VmaSuballocation& lastSuballoc = suballocations1st.back();
8669 if (lastSuballoc.offset == offset)
8670 {
8671 m_SumFreeSize += lastSuballoc.size;
8672 suballocations1st.pop_back();
8673 CleanupAfterFree();
8674 return;
8675 }
8676 }
8677
8678 VmaSuballocation refSuballoc;
8679 refSuballoc.offset = offset;
8680 // Rest of members stays uninitialized intentionally for better performance.
8681
8682 // Item from the middle of 1st vector.
8683 {
8684 const SuballocationVectorType::iterator it = VmaBinaryFindSorted(
8685 beg: suballocations1st.begin() + m_1stNullItemsBeginCount,
8686 end: suballocations1st.end(),
8687 value: refSuballoc,
8688 cmp: VmaSuballocationOffsetLess());
8689 if (it != suballocations1st.end())
8690 {
8691 it->type = VMA_SUBALLOCATION_TYPE_FREE;
8692 it->userData = VMA_NULL;
8693 ++m_1stNullItemsMiddleCount;
8694 m_SumFreeSize += it->size;
8695 CleanupAfterFree();
8696 return;
8697 }
8698 }
8699
8700 if (m_2ndVectorMode != SECOND_VECTOR_EMPTY)
8701 {
8702 // Item from the middle of 2nd vector.
8703 const SuballocationVectorType::iterator it = m_2ndVectorMode == SECOND_VECTOR_RING_BUFFER ?
8704 VmaBinaryFindSorted(beg: suballocations2nd.begin(), end: suballocations2nd.end(), value: refSuballoc, cmp: VmaSuballocationOffsetLess()) :
8705 VmaBinaryFindSorted(beg: suballocations2nd.begin(), end: suballocations2nd.end(), value: refSuballoc, cmp: VmaSuballocationOffsetGreater());
8706 if (it != suballocations2nd.end())
8707 {
8708 it->type = VMA_SUBALLOCATION_TYPE_FREE;
8709 it->userData = VMA_NULL;
8710 ++m_2ndNullItemsCount;
8711 m_SumFreeSize += it->size;
8712 CleanupAfterFree();
8713 return;
8714 }
8715 }
8716
8717 VMA_ASSERT(0 && "Allocation to free not found in linear allocator!");
8718}
8719
8720void VmaBlockMetadata_Linear::GetAllocationInfo(VmaAllocHandle allocHandle, VmaVirtualAllocationInfo& outInfo)
8721{
8722 outInfo.offset = (VkDeviceSize)allocHandle - 1;
8723 VmaSuballocation& suballoc = FindSuballocation(offset: outInfo.offset);
8724 outInfo.size = suballoc.size;
8725 outInfo.pUserData = suballoc.userData;
8726}
8727
8728void* VmaBlockMetadata_Linear::GetAllocationUserData(VmaAllocHandle allocHandle) const
8729{
8730 return FindSuballocation(offset: (VkDeviceSize)allocHandle - 1).userData;
8731}
8732
8733VmaAllocHandle VmaBlockMetadata_Linear::GetAllocationListBegin() const
8734{
8735 // Function only used for defragmentation, which is disabled for this algorithm
8736 VMA_ASSERT(0);
8737 return VK_NULL_HANDLE;
8738}
8739
8740VmaAllocHandle VmaBlockMetadata_Linear::GetNextAllocation(VmaAllocHandle prevAlloc) const
8741{
8742 // Function only used for defragmentation, which is disabled for this algorithm
8743 VMA_ASSERT(0);
8744 return VK_NULL_HANDLE;
8745}
8746
8747VkDeviceSize VmaBlockMetadata_Linear::GetNextFreeRegionSize(VmaAllocHandle alloc) const
8748{
8749 // Function only used for defragmentation, which is disabled for this algorithm
8750 VMA_ASSERT(0);
8751 return 0;
8752}
8753
8754void VmaBlockMetadata_Linear::Clear()
8755{
8756 m_SumFreeSize = GetSize();
8757 m_Suballocations0.clear();
8758 m_Suballocations1.clear();
8759 // Leaving m_1stVectorIndex unchanged - it doesn't matter.
8760 m_2ndVectorMode = SECOND_VECTOR_EMPTY;
8761 m_1stNullItemsBeginCount = 0;
8762 m_1stNullItemsMiddleCount = 0;
8763 m_2ndNullItemsCount = 0;
8764}
8765
8766void VmaBlockMetadata_Linear::SetAllocationUserData(VmaAllocHandle allocHandle, void* userData)
8767{
8768 VmaSuballocation& suballoc = FindSuballocation(offset: (VkDeviceSize)allocHandle - 1);
8769 suballoc.userData = userData;
8770}
8771
8772void VmaBlockMetadata_Linear::DebugLogAllAllocations() const
8773{
8774 const SuballocationVectorType& suballocations1st = AccessSuballocations1st();
8775 for (auto it = suballocations1st.begin() + m_1stNullItemsBeginCount; it != suballocations1st.end(); ++it)
8776 if (it->type != VMA_SUBALLOCATION_TYPE_FREE)
8777 DebugLogAllocation(offset: it->offset, size: it->size, userData: it->userData);
8778
8779 const SuballocationVectorType& suballocations2nd = AccessSuballocations2nd();
8780 for (auto it = suballocations2nd.begin(); it != suballocations2nd.end(); ++it)
8781 if (it->type != VMA_SUBALLOCATION_TYPE_FREE)
8782 DebugLogAllocation(offset: it->offset, size: it->size, userData: it->userData);
8783}
8784
8785VmaSuballocation& VmaBlockMetadata_Linear::FindSuballocation(VkDeviceSize offset) const
8786{
8787 const SuballocationVectorType& suballocations1st = AccessSuballocations1st();
8788 const SuballocationVectorType& suballocations2nd = AccessSuballocations2nd();
8789
8790 VmaSuballocation refSuballoc;
8791 refSuballoc.offset = offset;
8792 // Rest of members stays uninitialized intentionally for better performance.
8793
8794 // Item from the 1st vector.
8795 {
8796 SuballocationVectorType::const_iterator it = VmaBinaryFindSorted(
8797 beg: suballocations1st.begin() + m_1stNullItemsBeginCount,
8798 end: suballocations1st.end(),
8799 value: refSuballoc,
8800 cmp: VmaSuballocationOffsetLess());
8801 if (it != suballocations1st.end())
8802 {
8803 return const_cast<VmaSuballocation&>(*it);
8804 }
8805 }
8806
8807 if (m_2ndVectorMode != SECOND_VECTOR_EMPTY)
8808 {
8809 // Rest of members stays uninitialized intentionally for better performance.
8810 SuballocationVectorType::const_iterator it = m_2ndVectorMode == SECOND_VECTOR_RING_BUFFER ?
8811 VmaBinaryFindSorted(beg: suballocations2nd.begin(), end: suballocations2nd.end(), value: refSuballoc, cmp: VmaSuballocationOffsetLess()) :
8812 VmaBinaryFindSorted(beg: suballocations2nd.begin(), end: suballocations2nd.end(), value: refSuballoc, cmp: VmaSuballocationOffsetGreater());
8813 if (it != suballocations2nd.end())
8814 {
8815 return const_cast<VmaSuballocation&>(*it);
8816 }
8817 }
8818
8819 VMA_ASSERT(0 && "Allocation not found in linear allocator!");
8820 return const_cast<VmaSuballocation&>(suballocations1st.back()); // Should never occur.
8821}
8822
8823bool VmaBlockMetadata_Linear::ShouldCompact1st() const
8824{
8825 const size_t nullItemCount = m_1stNullItemsBeginCount + m_1stNullItemsMiddleCount;
8826 const size_t suballocCount = AccessSuballocations1st().size();
8827 return suballocCount > 32 && nullItemCount * 2 >= (suballocCount - nullItemCount) * 3;
8828}
8829
8830void VmaBlockMetadata_Linear::CleanupAfterFree()
8831{
8832 SuballocationVectorType& suballocations1st = AccessSuballocations1st();
8833 SuballocationVectorType& suballocations2nd = AccessSuballocations2nd();
8834
8835 if (IsEmpty())
8836 {
8837 suballocations1st.clear();
8838 suballocations2nd.clear();
8839 m_1stNullItemsBeginCount = 0;
8840 m_1stNullItemsMiddleCount = 0;
8841 m_2ndNullItemsCount = 0;
8842 m_2ndVectorMode = SECOND_VECTOR_EMPTY;
8843 }
8844 else
8845 {
8846 const size_t suballoc1stCount = suballocations1st.size();
8847 const size_t nullItem1stCount = m_1stNullItemsBeginCount + m_1stNullItemsMiddleCount;
8848 VMA_ASSERT(nullItem1stCount <= suballoc1stCount);
8849
8850 // Find more null items at the beginning of 1st vector.
8851 while (m_1stNullItemsBeginCount < suballoc1stCount &&
8852 suballocations1st[m_1stNullItemsBeginCount].type == VMA_SUBALLOCATION_TYPE_FREE)
8853 {
8854 ++m_1stNullItemsBeginCount;
8855 --m_1stNullItemsMiddleCount;
8856 }
8857
8858 // Find more null items at the end of 1st vector.
8859 while (m_1stNullItemsMiddleCount > 0 &&
8860 suballocations1st.back().type == VMA_SUBALLOCATION_TYPE_FREE)
8861 {
8862 --m_1stNullItemsMiddleCount;
8863 suballocations1st.pop_back();
8864 }
8865
8866 // Find more null items at the end of 2nd vector.
8867 while (m_2ndNullItemsCount > 0 &&
8868 suballocations2nd.back().type == VMA_SUBALLOCATION_TYPE_FREE)
8869 {
8870 --m_2ndNullItemsCount;
8871 suballocations2nd.pop_back();
8872 }
8873
8874 // Find more null items at the beginning of 2nd vector.
8875 while (m_2ndNullItemsCount > 0 &&
8876 suballocations2nd[0].type == VMA_SUBALLOCATION_TYPE_FREE)
8877 {
8878 --m_2ndNullItemsCount;
8879 VmaVectorRemove(vec&: suballocations2nd, index: 0);
8880 }
8881
8882 if (ShouldCompact1st())
8883 {
8884 const size_t nonNullItemCount = suballoc1stCount - nullItem1stCount;
8885 size_t srcIndex = m_1stNullItemsBeginCount;
8886 for (size_t dstIndex = 0; dstIndex < nonNullItemCount; ++dstIndex)
8887 {
8888 while (suballocations1st[srcIndex].type == VMA_SUBALLOCATION_TYPE_FREE)
8889 {
8890 ++srcIndex;
8891 }
8892 if (dstIndex != srcIndex)
8893 {
8894 suballocations1st[dstIndex] = suballocations1st[srcIndex];
8895 }
8896 ++srcIndex;
8897 }
8898 suballocations1st.resize(newCount: nonNullItemCount);
8899 m_1stNullItemsBeginCount = 0;
8900 m_1stNullItemsMiddleCount = 0;
8901 }
8902
8903 // 2nd vector became empty.
8904 if (suballocations2nd.empty())
8905 {
8906 m_2ndVectorMode = SECOND_VECTOR_EMPTY;
8907 }
8908
8909 // 1st vector became empty.
8910 if (suballocations1st.size() - m_1stNullItemsBeginCount == 0)
8911 {
8912 suballocations1st.clear();
8913 m_1stNullItemsBeginCount = 0;
8914
8915 if (!suballocations2nd.empty() && m_2ndVectorMode == SECOND_VECTOR_RING_BUFFER)
8916 {
8917 // Swap 1st with 2nd. Now 2nd is empty.
8918 m_2ndVectorMode = SECOND_VECTOR_EMPTY;
8919 m_1stNullItemsMiddleCount = m_2ndNullItemsCount;
8920 while (m_1stNullItemsBeginCount < suballocations2nd.size() &&
8921 suballocations2nd[m_1stNullItemsBeginCount].type == VMA_SUBALLOCATION_TYPE_FREE)
8922 {
8923 ++m_1stNullItemsBeginCount;
8924 --m_1stNullItemsMiddleCount;
8925 }
8926 m_2ndNullItemsCount = 0;
8927 m_1stVectorIndex ^= 1;
8928 }
8929 }
8930 }
8931
8932 VMA_HEAVY_ASSERT(Validate());
8933}
8934
8935bool VmaBlockMetadata_Linear::CreateAllocationRequest_LowerAddress(
8936 VkDeviceSize allocSize,
8937 VkDeviceSize allocAlignment,
8938 VmaSuballocationType allocType,
8939 uint32_t strategy,
8940 VmaAllocationRequest* pAllocationRequest)
8941{
8942 const VkDeviceSize blockSize = GetSize();
8943 const VkDeviceSize debugMargin = GetDebugMargin();
8944 const VkDeviceSize bufferImageGranularity = GetBufferImageGranularity();
8945 SuballocationVectorType& suballocations1st = AccessSuballocations1st();
8946 SuballocationVectorType& suballocations2nd = AccessSuballocations2nd();
8947
8948 if (m_2ndVectorMode == SECOND_VECTOR_EMPTY || m_2ndVectorMode == SECOND_VECTOR_DOUBLE_STACK)
8949 {
8950 // Try to allocate at the end of 1st vector.
8951
8952 VkDeviceSize resultBaseOffset = 0;
8953 if (!suballocations1st.empty())
8954 {
8955 const VmaSuballocation& lastSuballoc = suballocations1st.back();
8956 resultBaseOffset = lastSuballoc.offset + lastSuballoc.size + debugMargin;
8957 }
8958
8959 // Start from offset equal to beginning of free space.
8960 VkDeviceSize resultOffset = resultBaseOffset;
8961
8962 // Apply alignment.
8963 resultOffset = VmaAlignUp(val: resultOffset, alignment: allocAlignment);
8964
8965 // Check previous suballocations for BufferImageGranularity conflicts.
8966 // Make bigger alignment if necessary.
8967 if (bufferImageGranularity > 1 && bufferImageGranularity != allocAlignment && !suballocations1st.empty())
8968 {
8969 bool bufferImageGranularityConflict = false;
8970 for (size_t prevSuballocIndex = suballocations1st.size(); prevSuballocIndex--; )
8971 {
8972 const VmaSuballocation& prevSuballoc = suballocations1st[prevSuballocIndex];
8973 if (VmaBlocksOnSamePage(resourceAOffset: prevSuballoc.offset, resourceASize: prevSuballoc.size, resourceBOffset: resultOffset, pageSize: bufferImageGranularity))
8974 {
8975 if (VmaIsBufferImageGranularityConflict(suballocType1: prevSuballoc.type, suballocType2: allocType))
8976 {
8977 bufferImageGranularityConflict = true;
8978 break;
8979 }
8980 }
8981 else
8982 // Already on previous page.
8983 break;
8984 }
8985 if (bufferImageGranularityConflict)
8986 {
8987 resultOffset = VmaAlignUp(val: resultOffset, alignment: bufferImageGranularity);
8988 }
8989 }
8990
8991 const VkDeviceSize freeSpaceEnd = m_2ndVectorMode == SECOND_VECTOR_DOUBLE_STACK ?
8992 suballocations2nd.back().offset : blockSize;
8993
8994 // There is enough free space at the end after alignment.
8995 if (resultOffset + allocSize + debugMargin <= freeSpaceEnd)
8996 {
8997 // Check next suballocations for BufferImageGranularity conflicts.
8998 // If conflict exists, allocation cannot be made here.
8999 if ((allocSize % bufferImageGranularity || resultOffset % bufferImageGranularity) && m_2ndVectorMode == SECOND_VECTOR_DOUBLE_STACK)
9000 {
9001 for (size_t nextSuballocIndex = suballocations2nd.size(); nextSuballocIndex--; )
9002 {
9003 const VmaSuballocation& nextSuballoc = suballocations2nd[nextSuballocIndex];
9004 if (VmaBlocksOnSamePage(resourceAOffset: resultOffset, resourceASize: allocSize, resourceBOffset: nextSuballoc.offset, pageSize: bufferImageGranularity))
9005 {
9006 if (VmaIsBufferImageGranularityConflict(suballocType1: allocType, suballocType2: nextSuballoc.type))
9007 {
9008 return false;
9009 }
9010 }
9011 else
9012 {
9013 // Already on previous page.
9014 break;
9015 }
9016 }
9017 }
9018
9019 // All tests passed: Success.
9020 pAllocationRequest->allocHandle = (VmaAllocHandle)(resultOffset + 1);
9021 // pAllocationRequest->item, customData unused.
9022 pAllocationRequest->type = VmaAllocationRequestType::EndOf1st;
9023 return true;
9024 }
9025 }
9026
9027 // Wrap-around to end of 2nd vector. Try to allocate there, watching for the
9028 // beginning of 1st vector as the end of free space.
9029 if (m_2ndVectorMode == SECOND_VECTOR_EMPTY || m_2ndVectorMode == SECOND_VECTOR_RING_BUFFER)
9030 {
9031 VMA_ASSERT(!suballocations1st.empty());
9032
9033 VkDeviceSize resultBaseOffset = 0;
9034 if (!suballocations2nd.empty())
9035 {
9036 const VmaSuballocation& lastSuballoc = suballocations2nd.back();
9037 resultBaseOffset = lastSuballoc.offset + lastSuballoc.size + debugMargin;
9038 }
9039
9040 // Start from offset equal to beginning of free space.
9041 VkDeviceSize resultOffset = resultBaseOffset;
9042
9043 // Apply alignment.
9044 resultOffset = VmaAlignUp(val: resultOffset, alignment: allocAlignment);
9045
9046 // Check previous suballocations for BufferImageGranularity conflicts.
9047 // Make bigger alignment if necessary.
9048 if (bufferImageGranularity > 1 && bufferImageGranularity != allocAlignment && !suballocations2nd.empty())
9049 {
9050 bool bufferImageGranularityConflict = false;
9051 for (size_t prevSuballocIndex = suballocations2nd.size(); prevSuballocIndex--; )
9052 {
9053 const VmaSuballocation& prevSuballoc = suballocations2nd[prevSuballocIndex];
9054 if (VmaBlocksOnSamePage(resourceAOffset: prevSuballoc.offset, resourceASize: prevSuballoc.size, resourceBOffset: resultOffset, pageSize: bufferImageGranularity))
9055 {
9056 if (VmaIsBufferImageGranularityConflict(suballocType1: prevSuballoc.type, suballocType2: allocType))
9057 {
9058 bufferImageGranularityConflict = true;
9059 break;
9060 }
9061 }
9062 else
9063 // Already on previous page.
9064 break;
9065 }
9066 if (bufferImageGranularityConflict)
9067 {
9068 resultOffset = VmaAlignUp(val: resultOffset, alignment: bufferImageGranularity);
9069 }
9070 }
9071
9072 size_t index1st = m_1stNullItemsBeginCount;
9073
9074 // There is enough free space at the end after alignment.
9075 if ((index1st == suballocations1st.size() && resultOffset + allocSize + debugMargin <= blockSize) ||
9076 (index1st < suballocations1st.size() && resultOffset + allocSize + debugMargin <= suballocations1st[index1st].offset))
9077 {
9078 // Check next suballocations for BufferImageGranularity conflicts.
9079 // If conflict exists, allocation cannot be made here.
9080 if (allocSize % bufferImageGranularity || resultOffset % bufferImageGranularity)
9081 {
9082 for (size_t nextSuballocIndex = index1st;
9083 nextSuballocIndex < suballocations1st.size();
9084 nextSuballocIndex++)
9085 {
9086 const VmaSuballocation& nextSuballoc = suballocations1st[nextSuballocIndex];
9087 if (VmaBlocksOnSamePage(resourceAOffset: resultOffset, resourceASize: allocSize, resourceBOffset: nextSuballoc.offset, pageSize: bufferImageGranularity))
9088 {
9089 if (VmaIsBufferImageGranularityConflict(suballocType1: allocType, suballocType2: nextSuballoc.type))
9090 {
9091 return false;
9092 }
9093 }
9094 else
9095 {
9096 // Already on next page.
9097 break;
9098 }
9099 }
9100 }
9101
9102 // All tests passed: Success.
9103 pAllocationRequest->allocHandle = (VmaAllocHandle)(resultOffset + 1);
9104 pAllocationRequest->type = VmaAllocationRequestType::EndOf2nd;
9105 // pAllocationRequest->item, customData unused.
9106 return true;
9107 }
9108 }
9109
9110 return false;
9111}
9112
9113bool VmaBlockMetadata_Linear::CreateAllocationRequest_UpperAddress(
9114 VkDeviceSize allocSize,
9115 VkDeviceSize allocAlignment,
9116 VmaSuballocationType allocType,
9117 uint32_t strategy,
9118 VmaAllocationRequest* pAllocationRequest)
9119{
9120 const VkDeviceSize blockSize = GetSize();
9121 const VkDeviceSize bufferImageGranularity = GetBufferImageGranularity();
9122 SuballocationVectorType& suballocations1st = AccessSuballocations1st();
9123 SuballocationVectorType& suballocations2nd = AccessSuballocations2nd();
9124
9125 if (m_2ndVectorMode == SECOND_VECTOR_RING_BUFFER)
9126 {
9127 VMA_ASSERT(0 && "Trying to use pool with linear algorithm as double stack, while it is already being used as ring buffer.");
9128 return false;
9129 }
9130
9131 // Try to allocate before 2nd.back(), or end of block if 2nd.empty().
9132 if (allocSize > blockSize)
9133 {
9134 return false;
9135 }
9136 VkDeviceSize resultBaseOffset = blockSize - allocSize;
9137 if (!suballocations2nd.empty())
9138 {
9139 const VmaSuballocation& lastSuballoc = suballocations2nd.back();
9140 resultBaseOffset = lastSuballoc.offset - allocSize;
9141 if (allocSize > lastSuballoc.offset)
9142 {
9143 return false;
9144 }
9145 }
9146
9147 // Start from offset equal to end of free space.
9148 VkDeviceSize resultOffset = resultBaseOffset;
9149
9150 const VkDeviceSize debugMargin = GetDebugMargin();
9151
9152 // Apply debugMargin at the end.
9153 if (debugMargin > 0)
9154 {
9155 if (resultOffset < debugMargin)
9156 {
9157 return false;
9158 }
9159 resultOffset -= debugMargin;
9160 }
9161
9162 // Apply alignment.
9163 resultOffset = VmaAlignDown(val: resultOffset, alignment: allocAlignment);
9164
9165 // Check next suballocations from 2nd for BufferImageGranularity conflicts.
9166 // Make bigger alignment if necessary.
9167 if (bufferImageGranularity > 1 && bufferImageGranularity != allocAlignment && !suballocations2nd.empty())
9168 {
9169 bool bufferImageGranularityConflict = false;
9170 for (size_t nextSuballocIndex = suballocations2nd.size(); nextSuballocIndex--; )
9171 {
9172 const VmaSuballocation& nextSuballoc = suballocations2nd[nextSuballocIndex];
9173 if (VmaBlocksOnSamePage(resourceAOffset: resultOffset, resourceASize: allocSize, resourceBOffset: nextSuballoc.offset, pageSize: bufferImageGranularity))
9174 {
9175 if (VmaIsBufferImageGranularityConflict(suballocType1: nextSuballoc.type, suballocType2: allocType))
9176 {
9177 bufferImageGranularityConflict = true;
9178 break;
9179 }
9180 }
9181 else
9182 // Already on previous page.
9183 break;
9184 }
9185 if (bufferImageGranularityConflict)
9186 {
9187 resultOffset = VmaAlignDown(val: resultOffset, alignment: bufferImageGranularity);
9188 }
9189 }
9190
9191 // There is enough free space.
9192 const VkDeviceSize endOf1st = !suballocations1st.empty() ?
9193 suballocations1st.back().offset + suballocations1st.back().size :
9194 0;
9195 if (endOf1st + debugMargin <= resultOffset)
9196 {
9197 // Check previous suballocations for BufferImageGranularity conflicts.
9198 // If conflict exists, allocation cannot be made here.
9199 if (bufferImageGranularity > 1)
9200 {
9201 for (size_t prevSuballocIndex = suballocations1st.size(); prevSuballocIndex--; )
9202 {
9203 const VmaSuballocation& prevSuballoc = suballocations1st[prevSuballocIndex];
9204 if (VmaBlocksOnSamePage(resourceAOffset: prevSuballoc.offset, resourceASize: prevSuballoc.size, resourceBOffset: resultOffset, pageSize: bufferImageGranularity))
9205 {
9206 if (VmaIsBufferImageGranularityConflict(suballocType1: allocType, suballocType2: prevSuballoc.type))
9207 {
9208 return false;
9209 }
9210 }
9211 else
9212 {
9213 // Already on next page.
9214 break;
9215 }
9216 }
9217 }
9218
9219 // All tests passed: Success.
9220 pAllocationRequest->allocHandle = (VmaAllocHandle)(resultOffset + 1);
9221 // pAllocationRequest->item unused.
9222 pAllocationRequest->type = VmaAllocationRequestType::UpperAddress;
9223 return true;
9224 }
9225
9226 return false;
9227}
9228#endif // _VMA_BLOCK_METADATA_LINEAR_FUNCTIONS
9229#endif // _VMA_BLOCK_METADATA_LINEAR
9230
9231#if 0
9232#ifndef _VMA_BLOCK_METADATA_BUDDY
9233/*
9234- GetSize() is the original size of allocated memory block.
9235- m_UsableSize is this size aligned down to a power of two.
9236 All allocations and calculations happen relative to m_UsableSize.
9237- GetUnusableSize() is the difference between them.
9238 It is reported as separate, unused range, not available for allocations.
9239
9240Node at level 0 has size = m_UsableSize.
9241Each next level contains nodes with size 2 times smaller than current level.
9242m_LevelCount is the maximum number of levels to use in the current object.
9243*/
9244class VmaBlockMetadata_Buddy : public VmaBlockMetadata
9245{
9246 VMA_CLASS_NO_COPY(VmaBlockMetadata_Buddy)
9247public:
9248 VmaBlockMetadata_Buddy(const VkAllocationCallbacks* pAllocationCallbacks,
9249 VkDeviceSize bufferImageGranularity, bool isVirtual);
9250 virtual ~VmaBlockMetadata_Buddy();
9251
9252 size_t GetAllocationCount() const override { return m_AllocationCount; }
9253 VkDeviceSize GetSumFreeSize() const override { return m_SumFreeSize + GetUnusableSize(); }
9254 bool IsEmpty() const override { return m_Root->type == Node::TYPE_FREE; }
9255 VkResult CheckCorruption(const void* pBlockData) override { return VK_ERROR_FEATURE_NOT_PRESENT; }
9256 VkDeviceSize GetAllocationOffset(VmaAllocHandle allocHandle) const override { return (VkDeviceSize)allocHandle - 1; };
9257 void DebugLogAllAllocations() const override { DebugLogAllAllocationNode(m_Root, 0); }
9258
9259 void Init(VkDeviceSize size) override;
9260 bool Validate() const override;
9261
9262 void AddDetailedStatistics(VmaDetailedStatistics& inoutStats) const override;
9263 void AddStatistics(VmaStatistics& inoutStats) const override;
9264
9265#if VMA_STATS_STRING_ENABLED
9266 void PrintDetailedMap(class VmaJsonWriter& json, uint32_t mapRefCount) const override;
9267#endif
9268
9269 bool CreateAllocationRequest(
9270 VkDeviceSize allocSize,
9271 VkDeviceSize allocAlignment,
9272 bool upperAddress,
9273 VmaSuballocationType allocType,
9274 uint32_t strategy,
9275 VmaAllocationRequest* pAllocationRequest) override;
9276
9277 void Alloc(
9278 const VmaAllocationRequest& request,
9279 VmaSuballocationType type,
9280 void* userData) override;
9281
9282 void Free(VmaAllocHandle allocHandle) override;
9283 void GetAllocationInfo(VmaAllocHandle allocHandle, VmaVirtualAllocationInfo& outInfo) override;
9284 void* GetAllocationUserData(VmaAllocHandle allocHandle) const override;
9285 VmaAllocHandle GetAllocationListBegin() const override;
9286 VmaAllocHandle GetNextAllocation(VmaAllocHandle prevAlloc) const override;
9287 void Clear() override;
9288 void SetAllocationUserData(VmaAllocHandle allocHandle, void* userData) override;
9289
9290private:
9291 static const size_t MAX_LEVELS = 48;
9292
9293 struct ValidationContext
9294 {
9295 size_t calculatedAllocationCount = 0;
9296 size_t calculatedFreeCount = 0;
9297 VkDeviceSize calculatedSumFreeSize = 0;
9298 };
9299 struct Node
9300 {
9301 VkDeviceSize offset;
9302 enum TYPE
9303 {
9304 TYPE_FREE,
9305 TYPE_ALLOCATION,
9306 TYPE_SPLIT,
9307 TYPE_COUNT
9308 } type;
9309 Node* parent;
9310 Node* buddy;
9311
9312 union
9313 {
9314 struct
9315 {
9316 Node* prev;
9317 Node* next;
9318 } free;
9319 struct
9320 {
9321 void* userData;
9322 } allocation;
9323 struct
9324 {
9325 Node* leftChild;
9326 } split;
9327 };
9328 };
9329
9330 // Size of the memory block aligned down to a power of two.
9331 VkDeviceSize m_UsableSize;
9332 uint32_t m_LevelCount;
9333 VmaPoolAllocator<Node> m_NodeAllocator;
9334 Node* m_Root;
9335 struct
9336 {
9337 Node* front;
9338 Node* back;
9339 } m_FreeList[MAX_LEVELS];
9340
9341 // Number of nodes in the tree with type == TYPE_ALLOCATION.
9342 size_t m_AllocationCount;
9343 // Number of nodes in the tree with type == TYPE_FREE.
9344 size_t m_FreeCount;
9345 // Doesn't include space wasted due to internal fragmentation - allocation sizes are just aligned up to node sizes.
9346 // Doesn't include unusable size.
9347 VkDeviceSize m_SumFreeSize;
9348
9349 VkDeviceSize GetUnusableSize() const { return GetSize() - m_UsableSize; }
9350 VkDeviceSize LevelToNodeSize(uint32_t level) const { return m_UsableSize >> level; }
9351
9352 VkDeviceSize AlignAllocationSize(VkDeviceSize size) const
9353 {
9354 if (!IsVirtual())
9355 {
9356 size = VmaAlignUp(size, (VkDeviceSize)16);
9357 }
9358 return VmaNextPow2(size);
9359 }
9360 Node* FindAllocationNode(VkDeviceSize offset, uint32_t& outLevel) const;
9361 void DeleteNodeChildren(Node* node);
9362 bool ValidateNode(ValidationContext& ctx, const Node* parent, const Node* curr, uint32_t level, VkDeviceSize levelNodeSize) const;
9363 uint32_t AllocSizeToLevel(VkDeviceSize allocSize) const;
9364 void AddNodeToDetailedStatistics(VmaDetailedStatistics& inoutStats, const Node* node, VkDeviceSize levelNodeSize) const;
9365 // Adds node to the front of FreeList at given level.
9366 // node->type must be FREE.
9367 // node->free.prev, next can be undefined.
9368 void AddToFreeListFront(uint32_t level, Node* node);
9369 // Removes node from FreeList at given level.
9370 // node->type must be FREE.
9371 // node->free.prev, next stay untouched.
9372 void RemoveFromFreeList(uint32_t level, Node* node);
9373 void DebugLogAllAllocationNode(Node* node, uint32_t level) const;
9374
9375#if VMA_STATS_STRING_ENABLED
9376 void PrintDetailedMapNode(class VmaJsonWriter& json, const Node* node, VkDeviceSize levelNodeSize) const;
9377#endif
9378};
9379
9380#ifndef _VMA_BLOCK_METADATA_BUDDY_FUNCTIONS
9381VmaBlockMetadata_Buddy::VmaBlockMetadata_Buddy(const VkAllocationCallbacks* pAllocationCallbacks,
9382 VkDeviceSize bufferImageGranularity, bool isVirtual)
9383 : VmaBlockMetadata(pAllocationCallbacks, bufferImageGranularity, isVirtual),
9384 m_NodeAllocator(pAllocationCallbacks, 32), // firstBlockCapacity
9385 m_Root(VMA_NULL),
9386 m_AllocationCount(0),
9387 m_FreeCount(1),
9388 m_SumFreeSize(0)
9389{
9390 memset(m_FreeList, 0, sizeof(m_FreeList));
9391}
9392
9393VmaBlockMetadata_Buddy::~VmaBlockMetadata_Buddy()
9394{
9395 DeleteNodeChildren(m_Root);
9396 m_NodeAllocator.Free(m_Root);
9397}
9398
9399void VmaBlockMetadata_Buddy::Init(VkDeviceSize size)
9400{
9401 VmaBlockMetadata::Init(size);
9402
9403 m_UsableSize = VmaPrevPow2(size);
9404 m_SumFreeSize = m_UsableSize;
9405
9406 // Calculate m_LevelCount.
9407 const VkDeviceSize minNodeSize = IsVirtual() ? 1 : 16;
9408 m_LevelCount = 1;
9409 while (m_LevelCount < MAX_LEVELS &&
9410 LevelToNodeSize(m_LevelCount) >= minNodeSize)
9411 {
9412 ++m_LevelCount;
9413 }
9414
9415 Node* rootNode = m_NodeAllocator.Alloc();
9416 rootNode->offset = 0;
9417 rootNode->type = Node::TYPE_FREE;
9418 rootNode->parent = VMA_NULL;
9419 rootNode->buddy = VMA_NULL;
9420
9421 m_Root = rootNode;
9422 AddToFreeListFront(0, rootNode);
9423}
9424
9425bool VmaBlockMetadata_Buddy::Validate() const
9426{
9427 // Validate tree.
9428 ValidationContext ctx;
9429 if (!ValidateNode(ctx, VMA_NULL, m_Root, 0, LevelToNodeSize(0)))
9430 {
9431 VMA_VALIDATE(false && "ValidateNode failed.");
9432 }
9433 VMA_VALIDATE(m_AllocationCount == ctx.calculatedAllocationCount);
9434 VMA_VALIDATE(m_SumFreeSize == ctx.calculatedSumFreeSize);
9435
9436 // Validate free node lists.
9437 for (uint32_t level = 0; level < m_LevelCount; ++level)
9438 {
9439 VMA_VALIDATE(m_FreeList[level].front == VMA_NULL ||
9440 m_FreeList[level].front->free.prev == VMA_NULL);
9441
9442 for (Node* node = m_FreeList[level].front;
9443 node != VMA_NULL;
9444 node = node->free.next)
9445 {
9446 VMA_VALIDATE(node->type == Node::TYPE_FREE);
9447
9448 if (node->free.next == VMA_NULL)
9449 {
9450 VMA_VALIDATE(m_FreeList[level].back == node);
9451 }
9452 else
9453 {
9454 VMA_VALIDATE(node->free.next->free.prev == node);
9455 }
9456 }
9457 }
9458
9459 // Validate that free lists ar higher levels are empty.
9460 for (uint32_t level = m_LevelCount; level < MAX_LEVELS; ++level)
9461 {
9462 VMA_VALIDATE(m_FreeList[level].front == VMA_NULL && m_FreeList[level].back == VMA_NULL);
9463 }
9464
9465 return true;
9466}
9467
9468void VmaBlockMetadata_Buddy::AddDetailedStatistics(VmaDetailedStatistics& inoutStats) const
9469{
9470 inoutStats.statistics.blockCount++;
9471 inoutStats.statistics.blockBytes += GetSize();
9472
9473 AddNodeToDetailedStatistics(inoutStats, m_Root, LevelToNodeSize(0));
9474
9475 const VkDeviceSize unusableSize = GetUnusableSize();
9476 if (unusableSize > 0)
9477 VmaAddDetailedStatisticsUnusedRange(inoutStats, unusableSize);
9478}
9479
9480void VmaBlockMetadata_Buddy::AddStatistics(VmaStatistics& inoutStats) const
9481{
9482 inoutStats.blockCount++;
9483 inoutStats.allocationCount += (uint32_t)m_AllocationCount;
9484 inoutStats.blockBytes += GetSize();
9485 inoutStats.allocationBytes += GetSize() - m_SumFreeSize;
9486}
9487
9488#if VMA_STATS_STRING_ENABLED
9489void VmaBlockMetadata_Buddy::PrintDetailedMap(class VmaJsonWriter& json, uint32_t mapRefCount) const
9490{
9491 VmaDetailedStatistics stats;
9492 VmaClearDetailedStatistics(stats);
9493 AddDetailedStatistics(stats);
9494
9495 PrintDetailedMap_Begin(
9496 json,
9497 stats.statistics.blockBytes - stats.statistics.allocationBytes,
9498 stats.statistics.allocationCount,
9499 stats.unusedRangeCount,
9500 mapRefCount);
9501
9502 PrintDetailedMapNode(json, m_Root, LevelToNodeSize(0));
9503
9504 const VkDeviceSize unusableSize = GetUnusableSize();
9505 if (unusableSize > 0)
9506 {
9507 PrintDetailedMap_UnusedRange(json,
9508 m_UsableSize, // offset
9509 unusableSize); // size
9510 }
9511
9512 PrintDetailedMap_End(json);
9513}
9514#endif // VMA_STATS_STRING_ENABLED
9515
9516bool VmaBlockMetadata_Buddy::CreateAllocationRequest(
9517 VkDeviceSize allocSize,
9518 VkDeviceSize allocAlignment,
9519 bool upperAddress,
9520 VmaSuballocationType allocType,
9521 uint32_t strategy,
9522 VmaAllocationRequest* pAllocationRequest)
9523{
9524 VMA_ASSERT(!upperAddress && "VMA_ALLOCATION_CREATE_UPPER_ADDRESS_BIT can be used only with linear algorithm.");
9525
9526 allocSize = AlignAllocationSize(allocSize);
9527
9528 // Simple way to respect bufferImageGranularity. May be optimized some day.
9529 // Whenever it might be an OPTIMAL image...
9530 if (allocType == VMA_SUBALLOCATION_TYPE_UNKNOWN ||
9531 allocType == VMA_SUBALLOCATION_TYPE_IMAGE_UNKNOWN ||
9532 allocType == VMA_SUBALLOCATION_TYPE_IMAGE_OPTIMAL)
9533 {
9534 allocAlignment = VMA_MAX(allocAlignment, GetBufferImageGranularity());
9535 allocSize = VmaAlignUp(allocSize, GetBufferImageGranularity());
9536 }
9537
9538 if (allocSize > m_UsableSize)
9539 {
9540 return false;
9541 }
9542
9543 const uint32_t targetLevel = AllocSizeToLevel(allocSize);
9544 for (uint32_t level = targetLevel; level--; )
9545 {
9546 for (Node* freeNode = m_FreeList[level].front;
9547 freeNode != VMA_NULL;
9548 freeNode = freeNode->free.next)
9549 {
9550 if (freeNode->offset % allocAlignment == 0)
9551 {
9552 pAllocationRequest->type = VmaAllocationRequestType::Normal;
9553 pAllocationRequest->allocHandle = (VmaAllocHandle)(freeNode->offset + 1);
9554 pAllocationRequest->size = allocSize;
9555 pAllocationRequest->customData = (void*)(uintptr_t)level;
9556 return true;
9557 }
9558 }
9559 }
9560
9561 return false;
9562}
9563
9564void VmaBlockMetadata_Buddy::Alloc(
9565 const VmaAllocationRequest& request,
9566 VmaSuballocationType type,
9567 void* userData)
9568{
9569 VMA_ASSERT(request.type == VmaAllocationRequestType::Normal);
9570
9571 const uint32_t targetLevel = AllocSizeToLevel(request.size);
9572 uint32_t currLevel = (uint32_t)(uintptr_t)request.customData;
9573
9574 Node* currNode = m_FreeList[currLevel].front;
9575 VMA_ASSERT(currNode != VMA_NULL && currNode->type == Node::TYPE_FREE);
9576 const VkDeviceSize offset = (VkDeviceSize)request.allocHandle - 1;
9577 while (currNode->offset != offset)
9578 {
9579 currNode = currNode->free.next;
9580 VMA_ASSERT(currNode != VMA_NULL && currNode->type == Node::TYPE_FREE);
9581 }
9582
9583 // Go down, splitting free nodes.
9584 while (currLevel < targetLevel)
9585 {
9586 // currNode is already first free node at currLevel.
9587 // Remove it from list of free nodes at this currLevel.
9588 RemoveFromFreeList(currLevel, currNode);
9589
9590 const uint32_t childrenLevel = currLevel + 1;
9591
9592 // Create two free sub-nodes.
9593 Node* leftChild = m_NodeAllocator.Alloc();
9594 Node* rightChild = m_NodeAllocator.Alloc();
9595
9596 leftChild->offset = currNode->offset;
9597 leftChild->type = Node::TYPE_FREE;
9598 leftChild->parent = currNode;
9599 leftChild->buddy = rightChild;
9600
9601 rightChild->offset = currNode->offset + LevelToNodeSize(childrenLevel);
9602 rightChild->type = Node::TYPE_FREE;
9603 rightChild->parent = currNode;
9604 rightChild->buddy = leftChild;
9605
9606 // Convert current currNode to split type.
9607 currNode->type = Node::TYPE_SPLIT;
9608 currNode->split.leftChild = leftChild;
9609
9610 // Add child nodes to free list. Order is important!
9611 AddToFreeListFront(childrenLevel, rightChild);
9612 AddToFreeListFront(childrenLevel, leftChild);
9613
9614 ++m_FreeCount;
9615 ++currLevel;
9616 currNode = m_FreeList[currLevel].front;
9617
9618 /*
9619 We can be sure that currNode, as left child of node previously split,
9620 also fulfills the alignment requirement.
9621 */
9622 }
9623
9624 // Remove from free list.
9625 VMA_ASSERT(currLevel == targetLevel &&
9626 currNode != VMA_NULL &&
9627 currNode->type == Node::TYPE_FREE);
9628 RemoveFromFreeList(currLevel, currNode);
9629
9630 // Convert to allocation node.
9631 currNode->type = Node::TYPE_ALLOCATION;
9632 currNode->allocation.userData = userData;
9633
9634 ++m_AllocationCount;
9635 --m_FreeCount;
9636 m_SumFreeSize -= request.size;
9637}
9638
9639void VmaBlockMetadata_Buddy::GetAllocationInfo(VmaAllocHandle allocHandle, VmaVirtualAllocationInfo& outInfo)
9640{
9641 uint32_t level = 0;
9642 outInfo.offset = (VkDeviceSize)allocHandle - 1;
9643 const Node* const node = FindAllocationNode(outInfo.offset, level);
9644 outInfo.size = LevelToNodeSize(level);
9645 outInfo.pUserData = node->allocation.userData;
9646}
9647
9648void* VmaBlockMetadata_Buddy::GetAllocationUserData(VmaAllocHandle allocHandle) const
9649{
9650 uint32_t level = 0;
9651 const Node* const node = FindAllocationNode((VkDeviceSize)allocHandle - 1, level);
9652 return node->allocation.userData;
9653}
9654
9655VmaAllocHandle VmaBlockMetadata_Buddy::GetAllocationListBegin() const
9656{
9657 // Function only used for defragmentation, which is disabled for this algorithm
9658 return VK_NULL_HANDLE;
9659}
9660
9661VmaAllocHandle VmaBlockMetadata_Buddy::GetNextAllocation(VmaAllocHandle prevAlloc) const
9662{
9663 // Function only used for defragmentation, which is disabled for this algorithm
9664 return VK_NULL_HANDLE;
9665}
9666
9667void VmaBlockMetadata_Buddy::DeleteNodeChildren(Node* node)
9668{
9669 if (node->type == Node::TYPE_SPLIT)
9670 {
9671 DeleteNodeChildren(node->split.leftChild->buddy);
9672 DeleteNodeChildren(node->split.leftChild);
9673 const VkAllocationCallbacks* allocationCallbacks = GetAllocationCallbacks();
9674 m_NodeAllocator.Free(node->split.leftChild->buddy);
9675 m_NodeAllocator.Free(node->split.leftChild);
9676 }
9677}
9678
9679void VmaBlockMetadata_Buddy::Clear()
9680{
9681 DeleteNodeChildren(m_Root);
9682 m_Root->type = Node::TYPE_FREE;
9683 m_AllocationCount = 0;
9684 m_FreeCount = 1;
9685 m_SumFreeSize = m_UsableSize;
9686}
9687
9688void VmaBlockMetadata_Buddy::SetAllocationUserData(VmaAllocHandle allocHandle, void* userData)
9689{
9690 uint32_t level = 0;
9691 Node* const node = FindAllocationNode((VkDeviceSize)allocHandle - 1, level);
9692 node->allocation.userData = userData;
9693}
9694
9695VmaBlockMetadata_Buddy::Node* VmaBlockMetadata_Buddy::FindAllocationNode(VkDeviceSize offset, uint32_t& outLevel) const
9696{
9697 Node* node = m_Root;
9698 VkDeviceSize nodeOffset = 0;
9699 outLevel = 0;
9700 VkDeviceSize levelNodeSize = LevelToNodeSize(0);
9701 while (node->type == Node::TYPE_SPLIT)
9702 {
9703 const VkDeviceSize nextLevelNodeSize = levelNodeSize >> 1;
9704 if (offset < nodeOffset + nextLevelNodeSize)
9705 {
9706 node = node->split.leftChild;
9707 }
9708 else
9709 {
9710 node = node->split.leftChild->buddy;
9711 nodeOffset += nextLevelNodeSize;
9712 }
9713 ++outLevel;
9714 levelNodeSize = nextLevelNodeSize;
9715 }
9716
9717 VMA_ASSERT(node != VMA_NULL && node->type == Node::TYPE_ALLOCATION);
9718 return node;
9719}
9720
9721bool VmaBlockMetadata_Buddy::ValidateNode(ValidationContext& ctx, const Node* parent, const Node* curr, uint32_t level, VkDeviceSize levelNodeSize) const
9722{
9723 VMA_VALIDATE(level < m_LevelCount);
9724 VMA_VALIDATE(curr->parent == parent);
9725 VMA_VALIDATE((curr->buddy == VMA_NULL) == (parent == VMA_NULL));
9726 VMA_VALIDATE(curr->buddy == VMA_NULL || curr->buddy->buddy == curr);
9727 switch (curr->type)
9728 {
9729 case Node::TYPE_FREE:
9730 // curr->free.prev, next are validated separately.
9731 ctx.calculatedSumFreeSize += levelNodeSize;
9732 ++ctx.calculatedFreeCount;
9733 break;
9734 case Node::TYPE_ALLOCATION:
9735 ++ctx.calculatedAllocationCount;
9736 if (!IsVirtual())
9737 {
9738 VMA_VALIDATE(curr->allocation.userData != VMA_NULL);
9739 }
9740 break;
9741 case Node::TYPE_SPLIT:
9742 {
9743 const uint32_t childrenLevel = level + 1;
9744 const VkDeviceSize childrenLevelNodeSize = levelNodeSize >> 1;
9745 const Node* const leftChild = curr->split.leftChild;
9746 VMA_VALIDATE(leftChild != VMA_NULL);
9747 VMA_VALIDATE(leftChild->offset == curr->offset);
9748 if (!ValidateNode(ctx, curr, leftChild, childrenLevel, childrenLevelNodeSize))
9749 {
9750 VMA_VALIDATE(false && "ValidateNode for left child failed.");
9751 }
9752 const Node* const rightChild = leftChild->buddy;
9753 VMA_VALIDATE(rightChild->offset == curr->offset + childrenLevelNodeSize);
9754 if (!ValidateNode(ctx, curr, rightChild, childrenLevel, childrenLevelNodeSize))
9755 {
9756 VMA_VALIDATE(false && "ValidateNode for right child failed.");
9757 }
9758 }
9759 break;
9760 default:
9761 return false;
9762 }
9763
9764 return true;
9765}
9766
9767uint32_t VmaBlockMetadata_Buddy::AllocSizeToLevel(VkDeviceSize allocSize) const
9768{
9769 // I know this could be optimized somehow e.g. by using std::log2p1 from C++20.
9770 uint32_t level = 0;
9771 VkDeviceSize currLevelNodeSize = m_UsableSize;
9772 VkDeviceSize nextLevelNodeSize = currLevelNodeSize >> 1;
9773 while (allocSize <= nextLevelNodeSize && level + 1 < m_LevelCount)
9774 {
9775 ++level;
9776 currLevelNodeSize >>= 1;
9777 nextLevelNodeSize >>= 1;
9778 }
9779 return level;
9780}
9781
9782void VmaBlockMetadata_Buddy::Free(VmaAllocHandle allocHandle)
9783{
9784 uint32_t level = 0;
9785 Node* node = FindAllocationNode((VkDeviceSize)allocHandle - 1, level);
9786
9787 ++m_FreeCount;
9788 --m_AllocationCount;
9789 m_SumFreeSize += LevelToNodeSize(level);
9790
9791 node->type = Node::TYPE_FREE;
9792
9793 // Join free nodes if possible.
9794 while (level > 0 && node->buddy->type == Node::TYPE_FREE)
9795 {
9796 RemoveFromFreeList(level, node->buddy);
9797 Node* const parent = node->parent;
9798
9799 m_NodeAllocator.Free(node->buddy);
9800 m_NodeAllocator.Free(node);
9801 parent->type = Node::TYPE_FREE;
9802
9803 node = parent;
9804 --level;
9805 --m_FreeCount;
9806 }
9807
9808 AddToFreeListFront(level, node);
9809}
9810
9811void VmaBlockMetadata_Buddy::AddNodeToDetailedStatistics(VmaDetailedStatistics& inoutStats, const Node* node, VkDeviceSize levelNodeSize) const
9812{
9813 switch (node->type)
9814 {
9815 case Node::TYPE_FREE:
9816 VmaAddDetailedStatisticsUnusedRange(inoutStats, levelNodeSize);
9817 break;
9818 case Node::TYPE_ALLOCATION:
9819 VmaAddDetailedStatisticsAllocation(inoutStats, levelNodeSize);
9820 break;
9821 case Node::TYPE_SPLIT:
9822 {
9823 const VkDeviceSize childrenNodeSize = levelNodeSize / 2;
9824 const Node* const leftChild = node->split.leftChild;
9825 AddNodeToDetailedStatistics(inoutStats, leftChild, childrenNodeSize);
9826 const Node* const rightChild = leftChild->buddy;
9827 AddNodeToDetailedStatistics(inoutStats, rightChild, childrenNodeSize);
9828 }
9829 break;
9830 default:
9831 VMA_ASSERT(0);
9832 }
9833}
9834
9835void VmaBlockMetadata_Buddy::AddToFreeListFront(uint32_t level, Node* node)
9836{
9837 VMA_ASSERT(node->type == Node::TYPE_FREE);
9838
9839 // List is empty.
9840 Node* const frontNode = m_FreeList[level].front;
9841 if (frontNode == VMA_NULL)
9842 {
9843 VMA_ASSERT(m_FreeList[level].back == VMA_NULL);
9844 node->free.prev = node->free.next = VMA_NULL;
9845 m_FreeList[level].front = m_FreeList[level].back = node;
9846 }
9847 else
9848 {
9849 VMA_ASSERT(frontNode->free.prev == VMA_NULL);
9850 node->free.prev = VMA_NULL;
9851 node->free.next = frontNode;
9852 frontNode->free.prev = node;
9853 m_FreeList[level].front = node;
9854 }
9855}
9856
9857void VmaBlockMetadata_Buddy::RemoveFromFreeList(uint32_t level, Node* node)
9858{
9859 VMA_ASSERT(m_FreeList[level].front != VMA_NULL);
9860
9861 // It is at the front.
9862 if (node->free.prev == VMA_NULL)
9863 {
9864 VMA_ASSERT(m_FreeList[level].front == node);
9865 m_FreeList[level].front = node->free.next;
9866 }
9867 else
9868 {
9869 Node* const prevFreeNode = node->free.prev;
9870 VMA_ASSERT(prevFreeNode->free.next == node);
9871 prevFreeNode->free.next = node->free.next;
9872 }
9873
9874 // It is at the back.
9875 if (node->free.next == VMA_NULL)
9876 {
9877 VMA_ASSERT(m_FreeList[level].back == node);
9878 m_FreeList[level].back = node->free.prev;
9879 }
9880 else
9881 {
9882 Node* const nextFreeNode = node->free.next;
9883 VMA_ASSERT(nextFreeNode->free.prev == node);
9884 nextFreeNode->free.prev = node->free.prev;
9885 }
9886}
9887
9888void VmaBlockMetadata_Buddy::DebugLogAllAllocationNode(Node* node, uint32_t level) const
9889{
9890 switch (node->type)
9891 {
9892 case Node::TYPE_FREE:
9893 break;
9894 case Node::TYPE_ALLOCATION:
9895 DebugLogAllocation(node->offset, LevelToNodeSize(level), node->allocation.userData);
9896 break;
9897 case Node::TYPE_SPLIT:
9898 {
9899 ++level;
9900 DebugLogAllAllocationNode(node->split.leftChild, level);
9901 DebugLogAllAllocationNode(node->split.leftChild->buddy, level);
9902 }
9903 break;
9904 default:
9905 VMA_ASSERT(0);
9906 }
9907}
9908
9909#if VMA_STATS_STRING_ENABLED
9910void VmaBlockMetadata_Buddy::PrintDetailedMapNode(class VmaJsonWriter& json, const Node* node, VkDeviceSize levelNodeSize) const
9911{
9912 switch (node->type)
9913 {
9914 case Node::TYPE_FREE:
9915 PrintDetailedMap_UnusedRange(json, node->offset, levelNodeSize);
9916 break;
9917 case Node::TYPE_ALLOCATION:
9918 PrintDetailedMap_Allocation(json, node->offset, levelNodeSize, node->allocation.userData);
9919 break;
9920 case Node::TYPE_SPLIT:
9921 {
9922 const VkDeviceSize childrenNodeSize = levelNodeSize / 2;
9923 const Node* const leftChild = node->split.leftChild;
9924 PrintDetailedMapNode(json, leftChild, childrenNodeSize);
9925 const Node* const rightChild = leftChild->buddy;
9926 PrintDetailedMapNode(json, rightChild, childrenNodeSize);
9927 }
9928 break;
9929 default:
9930 VMA_ASSERT(0);
9931 }
9932}
9933#endif // VMA_STATS_STRING_ENABLED
9934#endif // _VMA_BLOCK_METADATA_BUDDY_FUNCTIONS
9935#endif // _VMA_BLOCK_METADATA_BUDDY
9936#endif // #if 0
9937
9938#ifndef _VMA_BLOCK_METADATA_TLSF
9939// To not search current larger region if first allocation won't succeed and skip to smaller range
9940// use with VMA_ALLOCATION_CREATE_STRATEGY_MIN_MEMORY_BIT as strategy in CreateAllocationRequest().
9941// When fragmentation and reusal of previous blocks doesn't matter then use with
9942// VMA_ALLOCATION_CREATE_STRATEGY_MIN_TIME_BIT for fastest alloc time possible.
9943class VmaBlockMetadata_TLSF : public VmaBlockMetadata
9944{
9945 VMA_CLASS_NO_COPY(VmaBlockMetadata_TLSF)
9946public:
9947 VmaBlockMetadata_TLSF(const VkAllocationCallbacks* pAllocationCallbacks,
9948 VkDeviceSize bufferImageGranularity, bool isVirtual);
9949 virtual ~VmaBlockMetadata_TLSF();
9950
9951 size_t GetAllocationCount() const override { return m_AllocCount; }
9952 size_t GetFreeRegionsCount() const override { return m_BlocksFreeCount + 1; }
9953 VkDeviceSize GetSumFreeSize() const override { return m_BlocksFreeSize + m_NullBlock->size; }
9954 bool IsEmpty() const override { return m_NullBlock->offset == 0; }
9955 VkDeviceSize GetAllocationOffset(VmaAllocHandle allocHandle) const override { return ((Block*)allocHandle)->offset; };
9956
9957 void Init(VkDeviceSize size) override;
9958 bool Validate() const override;
9959
9960 void AddDetailedStatistics(VmaDetailedStatistics& inoutStats) const override;
9961 void AddStatistics(VmaStatistics& inoutStats) const override;
9962
9963#if VMA_STATS_STRING_ENABLED
9964 void PrintDetailedMap(class VmaJsonWriter& json) const override;
9965#endif
9966
9967 bool CreateAllocationRequest(
9968 VkDeviceSize allocSize,
9969 VkDeviceSize allocAlignment,
9970 bool upperAddress,
9971 VmaSuballocationType allocType,
9972 uint32_t strategy,
9973 VmaAllocationRequest* pAllocationRequest) override;
9974
9975 VkResult CheckCorruption(const void* pBlockData) override;
9976 void Alloc(
9977 const VmaAllocationRequest& request,
9978 VmaSuballocationType type,
9979 void* userData) override;
9980
9981 void Free(VmaAllocHandle allocHandle) override;
9982 void GetAllocationInfo(VmaAllocHandle allocHandle, VmaVirtualAllocationInfo& outInfo) override;
9983 void* GetAllocationUserData(VmaAllocHandle allocHandle) const override;
9984 VmaAllocHandle GetAllocationListBegin() const override;
9985 VmaAllocHandle GetNextAllocation(VmaAllocHandle prevAlloc) const override;
9986 VkDeviceSize GetNextFreeRegionSize(VmaAllocHandle alloc) const override;
9987 void Clear() override;
9988 void SetAllocationUserData(VmaAllocHandle allocHandle, void* userData) override;
9989 void DebugLogAllAllocations() const override;
9990
9991private:
9992 // According to original paper it should be preferable 4 or 5:
9993 // M. Masmano, I. Ripoll, A. Crespo, and J. Real "TLSF: a New Dynamic Memory Allocator for Real-Time Systems"
9994 // http://www.gii.upv.es/tlsf/files/ecrts04_tlsf.pdf
9995 static const uint8_t SECOND_LEVEL_INDEX = 5;
9996 static const uint16_t SMALL_BUFFER_SIZE = 256;
9997 static const uint32_t INITIAL_BLOCK_ALLOC_COUNT = 16;
9998 static const uint8_t MEMORY_CLASS_SHIFT = 7;
9999 static const uint8_t MAX_MEMORY_CLASSES = 65 - MEMORY_CLASS_SHIFT;
10000
10001 class Block
10002 {
10003 public:
10004 VkDeviceSize offset;
10005 VkDeviceSize size;
10006 Block* prevPhysical;
10007 Block* nextPhysical;
10008
10009 void MarkFree() { prevFree = VMA_NULL; }
10010 void MarkTaken() { prevFree = this; }
10011 bool IsFree() const { return prevFree != this; }
10012 void*& UserData() { VMA_HEAVY_ASSERT(!IsFree()); return userData; }
10013 Block*& PrevFree() { return prevFree; }
10014 Block*& NextFree() { VMA_HEAVY_ASSERT(IsFree()); return nextFree; }
10015
10016 private:
10017 Block* prevFree; // Address of the same block here indicates that block is taken
10018 union
10019 {
10020 Block* nextFree;
10021 void* userData;
10022 };
10023 };
10024
10025 size_t m_AllocCount;
10026 // Total number of free blocks besides null block
10027 size_t m_BlocksFreeCount;
10028 // Total size of free blocks excluding null block
10029 VkDeviceSize m_BlocksFreeSize;
10030 uint32_t m_IsFreeBitmap;
10031 uint8_t m_MemoryClasses;
10032 uint32_t m_InnerIsFreeBitmap[MAX_MEMORY_CLASSES];
10033 uint32_t m_ListsCount;
10034 /*
10035 * 0: 0-3 lists for small buffers
10036 * 1+: 0-(2^SLI-1) lists for normal buffers
10037 */
10038 Block** m_FreeList;
10039 VmaPoolAllocator<Block> m_BlockAllocator;
10040 Block* m_NullBlock;
10041 VmaBlockBufferImageGranularity m_GranularityHandler;
10042
10043 uint8_t SizeToMemoryClass(VkDeviceSize size) const;
10044 uint16_t SizeToSecondIndex(VkDeviceSize size, uint8_t memoryClass) const;
10045 uint32_t GetListIndex(uint8_t memoryClass, uint16_t secondIndex) const;
10046 uint32_t GetListIndex(VkDeviceSize size) const;
10047
10048 void RemoveFreeBlock(Block* block);
10049 void InsertFreeBlock(Block* block);
10050 void MergeBlock(Block* block, Block* prev);
10051
10052 Block* FindFreeBlock(VkDeviceSize size, uint32_t& listIndex) const;
10053 bool CheckBlock(
10054 Block& block,
10055 uint32_t listIndex,
10056 VkDeviceSize allocSize,
10057 VkDeviceSize allocAlignment,
10058 VmaSuballocationType allocType,
10059 VmaAllocationRequest* pAllocationRequest);
10060};
10061
10062#ifndef _VMA_BLOCK_METADATA_TLSF_FUNCTIONS
10063VmaBlockMetadata_TLSF::VmaBlockMetadata_TLSF(const VkAllocationCallbacks* pAllocationCallbacks,
10064 VkDeviceSize bufferImageGranularity, bool isVirtual)
10065 : VmaBlockMetadata(pAllocationCallbacks, bufferImageGranularity, isVirtual),
10066 m_AllocCount(0),
10067 m_BlocksFreeCount(0),
10068 m_BlocksFreeSize(0),
10069 m_IsFreeBitmap(0),
10070 m_MemoryClasses(0),
10071 m_ListsCount(0),
10072 m_FreeList(VMA_NULL),
10073 m_BlockAllocator(pAllocationCallbacks, INITIAL_BLOCK_ALLOC_COUNT),
10074 m_NullBlock(VMA_NULL),
10075 m_GranularityHandler(bufferImageGranularity) {}
10076
10077VmaBlockMetadata_TLSF::~VmaBlockMetadata_TLSF()
10078{
10079 if (m_FreeList)
10080 vma_delete_array(pAllocationCallbacks: GetAllocationCallbacks(), ptr: m_FreeList, count: m_ListsCount);
10081 m_GranularityHandler.Destroy(pAllocationCallbacks: GetAllocationCallbacks());
10082}
10083
10084void VmaBlockMetadata_TLSF::Init(VkDeviceSize size)
10085{
10086 VmaBlockMetadata::Init(size);
10087
10088 if (!IsVirtual())
10089 m_GranularityHandler.Init(pAllocationCallbacks: GetAllocationCallbacks(), size);
10090
10091 m_NullBlock = m_BlockAllocator.Alloc();
10092 m_NullBlock->size = size;
10093 m_NullBlock->offset = 0;
10094 m_NullBlock->prevPhysical = VMA_NULL;
10095 m_NullBlock->nextPhysical = VMA_NULL;
10096 m_NullBlock->MarkFree();
10097 m_NullBlock->NextFree() = VMA_NULL;
10098 m_NullBlock->PrevFree() = VMA_NULL;
10099 uint8_t memoryClass = SizeToMemoryClass(size);
10100 uint16_t sli = SizeToSecondIndex(size, memoryClass);
10101 m_ListsCount = (memoryClass == 0 ? 0 : (memoryClass - 1) * (1UL << SECOND_LEVEL_INDEX) + sli) + 1;
10102 if (IsVirtual())
10103 m_ListsCount += 1UL << SECOND_LEVEL_INDEX;
10104 else
10105 m_ListsCount += 4;
10106
10107 m_MemoryClasses = memoryClass + 2;
10108 memset(s: m_InnerIsFreeBitmap, c: 0, n: MAX_MEMORY_CLASSES * sizeof(uint32_t));
10109
10110 m_FreeList = vma_new_array(GetAllocationCallbacks(), Block*, m_ListsCount);
10111 memset(s: m_FreeList, c: 0, n: m_ListsCount * sizeof(Block*));
10112}
10113
10114bool VmaBlockMetadata_TLSF::Validate() const
10115{
10116 VMA_VALIDATE(GetSumFreeSize() <= GetSize());
10117
10118 VkDeviceSize calculatedSize = m_NullBlock->size;
10119 VkDeviceSize calculatedFreeSize = m_NullBlock->size;
10120 size_t allocCount = 0;
10121 size_t freeCount = 0;
10122
10123 // Check integrity of free lists
10124 for (uint32_t list = 0; list < m_ListsCount; ++list)
10125 {
10126 Block* block = m_FreeList[list];
10127 if (block != VMA_NULL)
10128 {
10129 VMA_VALIDATE(block->IsFree());
10130 VMA_VALIDATE(block->PrevFree() == VMA_NULL);
10131 while (block->NextFree())
10132 {
10133 VMA_VALIDATE(block->NextFree()->IsFree());
10134 VMA_VALIDATE(block->NextFree()->PrevFree() == block);
10135 block = block->NextFree();
10136 }
10137 }
10138 }
10139
10140 VkDeviceSize nextOffset = m_NullBlock->offset;
10141 auto validateCtx = m_GranularityHandler.StartValidation(pAllocationCallbacks: GetAllocationCallbacks(), isVirutal: IsVirtual());
10142
10143 VMA_VALIDATE(m_NullBlock->nextPhysical == VMA_NULL);
10144 if (m_NullBlock->prevPhysical)
10145 {
10146 VMA_VALIDATE(m_NullBlock->prevPhysical->nextPhysical == m_NullBlock);
10147 }
10148 // Check all blocks
10149 for (Block* prev = m_NullBlock->prevPhysical; prev != VMA_NULL; prev = prev->prevPhysical)
10150 {
10151 VMA_VALIDATE(prev->offset + prev->size == nextOffset);
10152 nextOffset = prev->offset;
10153 calculatedSize += prev->size;
10154
10155 uint32_t listIndex = GetListIndex(size: prev->size);
10156 if (prev->IsFree())
10157 {
10158 ++freeCount;
10159 // Check if free block belongs to free list
10160 Block* freeBlock = m_FreeList[listIndex];
10161 VMA_VALIDATE(freeBlock != VMA_NULL);
10162
10163 bool found = false;
10164 do
10165 {
10166 if (freeBlock == prev)
10167 found = true;
10168
10169 freeBlock = freeBlock->NextFree();
10170 } while (!found && freeBlock != VMA_NULL);
10171
10172 VMA_VALIDATE(found);
10173 calculatedFreeSize += prev->size;
10174 }
10175 else
10176 {
10177 ++allocCount;
10178 // Check if taken block is not on a free list
10179 Block* freeBlock = m_FreeList[listIndex];
10180 while (freeBlock)
10181 {
10182 VMA_VALIDATE(freeBlock != prev);
10183 freeBlock = freeBlock->NextFree();
10184 }
10185
10186 if (!IsVirtual())
10187 {
10188 VMA_VALIDATE(m_GranularityHandler.Validate(validateCtx, prev->offset, prev->size));
10189 }
10190 }
10191
10192 if (prev->prevPhysical)
10193 {
10194 VMA_VALIDATE(prev->prevPhysical->nextPhysical == prev);
10195 }
10196 }
10197
10198 if (!IsVirtual())
10199 {
10200 VMA_VALIDATE(m_GranularityHandler.FinishValidation(validateCtx));
10201 }
10202
10203 VMA_VALIDATE(nextOffset == 0);
10204 VMA_VALIDATE(calculatedSize == GetSize());
10205 VMA_VALIDATE(calculatedFreeSize == GetSumFreeSize());
10206 VMA_VALIDATE(allocCount == m_AllocCount);
10207 VMA_VALIDATE(freeCount == m_BlocksFreeCount);
10208
10209 return true;
10210}
10211
10212void VmaBlockMetadata_TLSF::AddDetailedStatistics(VmaDetailedStatistics& inoutStats) const
10213{
10214 inoutStats.statistics.blockCount++;
10215 inoutStats.statistics.blockBytes += GetSize();
10216 if (m_NullBlock->size > 0)
10217 VmaAddDetailedStatisticsUnusedRange(inoutStats, size: m_NullBlock->size);
10218
10219 for (Block* block = m_NullBlock->prevPhysical; block != VMA_NULL; block = block->prevPhysical)
10220 {
10221 if (block->IsFree())
10222 VmaAddDetailedStatisticsUnusedRange(inoutStats, size: block->size);
10223 else
10224 VmaAddDetailedStatisticsAllocation(inoutStats, size: block->size);
10225 }
10226}
10227
10228void VmaBlockMetadata_TLSF::AddStatistics(VmaStatistics& inoutStats) const
10229{
10230 inoutStats.blockCount++;
10231 inoutStats.allocationCount += (uint32_t)m_AllocCount;
10232 inoutStats.blockBytes += GetSize();
10233 inoutStats.allocationBytes += GetSize() - GetSumFreeSize();
10234}
10235
10236#if VMA_STATS_STRING_ENABLED
10237void VmaBlockMetadata_TLSF::PrintDetailedMap(class VmaJsonWriter& json) const
10238{
10239 size_t blockCount = m_AllocCount + m_BlocksFreeCount;
10240 VmaStlAllocator<Block*> allocator(GetAllocationCallbacks());
10241 VmaVector<Block*, VmaStlAllocator<Block*>> blockList(blockCount, allocator);
10242
10243 size_t i = blockCount;
10244 for (Block* block = m_NullBlock->prevPhysical; block != VMA_NULL; block = block->prevPhysical)
10245 {
10246 blockList[--i] = block;
10247 }
10248 VMA_ASSERT(i == 0);
10249
10250 VmaDetailedStatistics stats;
10251 VmaClearDetailedStatistics(outStats&: stats);
10252 AddDetailedStatistics(inoutStats&: stats);
10253
10254 PrintDetailedMap_Begin(json,
10255 unusedBytes: stats.statistics.blockBytes - stats.statistics.allocationBytes,
10256 allocationCount: stats.statistics.allocationCount,
10257 unusedRangeCount: stats.unusedRangeCount);
10258
10259 for (; i < blockCount; ++i)
10260 {
10261 Block* block = blockList[i];
10262 if (block->IsFree())
10263 PrintDetailedMap_UnusedRange(json, offset: block->offset, size: block->size);
10264 else
10265 PrintDetailedMap_Allocation(json, offset: block->offset, size: block->size, userData: block->UserData());
10266 }
10267 if (m_NullBlock->size > 0)
10268 PrintDetailedMap_UnusedRange(json, offset: m_NullBlock->offset, size: m_NullBlock->size);
10269
10270 PrintDetailedMap_End(json);
10271}
10272#endif
10273
10274bool VmaBlockMetadata_TLSF::CreateAllocationRequest(
10275 VkDeviceSize allocSize,
10276 VkDeviceSize allocAlignment,
10277 bool upperAddress,
10278 VmaSuballocationType allocType,
10279 uint32_t strategy,
10280 VmaAllocationRequest* pAllocationRequest)
10281{
10282 VMA_ASSERT(allocSize > 0 && "Cannot allocate empty block!");
10283 VMA_ASSERT(!upperAddress && "VMA_ALLOCATION_CREATE_UPPER_ADDRESS_BIT can be used only with linear algorithm.");
10284
10285 // For small granularity round up
10286 if (!IsVirtual())
10287 m_GranularityHandler.RoundupAllocRequest(allocType, inOutAllocSize&: allocSize, inOutAllocAlignment&: allocAlignment);
10288
10289 allocSize += GetDebugMargin();
10290 // Quick check for too small pool
10291 if (allocSize > GetSumFreeSize())
10292 return false;
10293
10294 // If no free blocks in pool then check only null block
10295 if (m_BlocksFreeCount == 0)
10296 return CheckBlock(block&: *m_NullBlock, listIndex: m_ListsCount, allocSize, allocAlignment, allocType, pAllocationRequest);
10297
10298 // Round up to the next block
10299 VkDeviceSize sizeForNextList = allocSize;
10300 VkDeviceSize smallSizeStep = SMALL_BUFFER_SIZE / (IsVirtual() ? 1 << SECOND_LEVEL_INDEX : 4);
10301 if (allocSize > SMALL_BUFFER_SIZE)
10302 {
10303 sizeForNextList += (1ULL << (VMA_BITSCAN_MSB(allocSize) - SECOND_LEVEL_INDEX));
10304 }
10305 else if (allocSize > SMALL_BUFFER_SIZE - smallSizeStep)
10306 sizeForNextList = SMALL_BUFFER_SIZE + 1;
10307 else
10308 sizeForNextList += smallSizeStep;
10309
10310 uint32_t nextListIndex = 0;
10311 uint32_t prevListIndex = 0;
10312 Block* nextListBlock = VMA_NULL;
10313 Block* prevListBlock = VMA_NULL;
10314
10315 // Check blocks according to strategies
10316 if (strategy & VMA_ALLOCATION_CREATE_STRATEGY_MIN_TIME_BIT)
10317 {
10318 // Quick check for larger block first
10319 nextListBlock = FindFreeBlock(size: sizeForNextList, listIndex&: nextListIndex);
10320 if (nextListBlock != VMA_NULL && CheckBlock(block&: *nextListBlock, listIndex: nextListIndex, allocSize, allocAlignment, allocType, pAllocationRequest))
10321 return true;
10322
10323 // If not fitted then null block
10324 if (CheckBlock(block&: *m_NullBlock, listIndex: m_ListsCount, allocSize, allocAlignment, allocType, pAllocationRequest))
10325 return true;
10326
10327 // Null block failed, search larger bucket
10328 while (nextListBlock)
10329 {
10330 if (CheckBlock(block&: *nextListBlock, listIndex: nextListIndex, allocSize, allocAlignment, allocType, pAllocationRequest))
10331 return true;
10332 nextListBlock = nextListBlock->NextFree();
10333 }
10334
10335 // Failed again, check best fit bucket
10336 prevListBlock = FindFreeBlock(size: allocSize, listIndex&: prevListIndex);
10337 while (prevListBlock)
10338 {
10339 if (CheckBlock(block&: *prevListBlock, listIndex: prevListIndex, allocSize, allocAlignment, allocType, pAllocationRequest))
10340 return true;
10341 prevListBlock = prevListBlock->NextFree();
10342 }
10343 }
10344 else if (strategy & VMA_ALLOCATION_CREATE_STRATEGY_MIN_MEMORY_BIT)
10345 {
10346 // Check best fit bucket
10347 prevListBlock = FindFreeBlock(size: allocSize, listIndex&: prevListIndex);
10348 while (prevListBlock)
10349 {
10350 if (CheckBlock(block&: *prevListBlock, listIndex: prevListIndex, allocSize, allocAlignment, allocType, pAllocationRequest))
10351 return true;
10352 prevListBlock = prevListBlock->NextFree();
10353 }
10354
10355 // If failed check null block
10356 if (CheckBlock(block&: *m_NullBlock, listIndex: m_ListsCount, allocSize, allocAlignment, allocType, pAllocationRequest))
10357 return true;
10358
10359 // Check larger bucket
10360 nextListBlock = FindFreeBlock(size: sizeForNextList, listIndex&: nextListIndex);
10361 while (nextListBlock)
10362 {
10363 if (CheckBlock(block&: *nextListBlock, listIndex: nextListIndex, allocSize, allocAlignment, allocType, pAllocationRequest))
10364 return true;
10365 nextListBlock = nextListBlock->NextFree();
10366 }
10367 }
10368 else if (strategy & VMA_ALLOCATION_CREATE_STRATEGY_MIN_OFFSET_BIT )
10369 {
10370 // Perform search from the start
10371 VmaStlAllocator<Block*> allocator(GetAllocationCallbacks());
10372 VmaVector<Block*, VmaStlAllocator<Block*>> blockList(m_BlocksFreeCount, allocator);
10373
10374 size_t i = m_BlocksFreeCount;
10375 for (Block* block = m_NullBlock->prevPhysical; block != VMA_NULL; block = block->prevPhysical)
10376 {
10377 if (block->IsFree() && block->size >= allocSize)
10378 blockList[--i] = block;
10379 }
10380
10381 for (; i < m_BlocksFreeCount; ++i)
10382 {
10383 Block& block = *blockList[i];
10384 if (CheckBlock(block, listIndex: GetListIndex(size: block.size), allocSize, allocAlignment, allocType, pAllocationRequest))
10385 return true;
10386 }
10387
10388 // If failed check null block
10389 if (CheckBlock(block&: *m_NullBlock, listIndex: m_ListsCount, allocSize, allocAlignment, allocType, pAllocationRequest))
10390 return true;
10391
10392 // Whole range searched, no more memory
10393 return false;
10394 }
10395 else
10396 {
10397 // Check larger bucket
10398 nextListBlock = FindFreeBlock(size: sizeForNextList, listIndex&: nextListIndex);
10399 while (nextListBlock)
10400 {
10401 if (CheckBlock(block&: *nextListBlock, listIndex: nextListIndex, allocSize, allocAlignment, allocType, pAllocationRequest))
10402 return true;
10403 nextListBlock = nextListBlock->NextFree();
10404 }
10405
10406 // If failed check null block
10407 if (CheckBlock(block&: *m_NullBlock, listIndex: m_ListsCount, allocSize, allocAlignment, allocType, pAllocationRequest))
10408 return true;
10409
10410 // Check best fit bucket
10411 prevListBlock = FindFreeBlock(size: allocSize, listIndex&: prevListIndex);
10412 while (prevListBlock)
10413 {
10414 if (CheckBlock(block&: *prevListBlock, listIndex: prevListIndex, allocSize, allocAlignment, allocType, pAllocationRequest))
10415 return true;
10416 prevListBlock = prevListBlock->NextFree();
10417 }
10418 }
10419
10420 // Worst case, full search has to be done
10421 while (++nextListIndex < m_ListsCount)
10422 {
10423 nextListBlock = m_FreeList[nextListIndex];
10424 while (nextListBlock)
10425 {
10426 if (CheckBlock(block&: *nextListBlock, listIndex: nextListIndex, allocSize, allocAlignment, allocType, pAllocationRequest))
10427 return true;
10428 nextListBlock = nextListBlock->NextFree();
10429 }
10430 }
10431
10432 // No more memory sadly
10433 return false;
10434}
10435
10436VkResult VmaBlockMetadata_TLSF::CheckCorruption(const void* pBlockData)
10437{
10438 for (Block* block = m_NullBlock->prevPhysical; block != VMA_NULL; block = block->prevPhysical)
10439 {
10440 if (!block->IsFree())
10441 {
10442 if (!VmaValidateMagicValue(pData: pBlockData, offset: block->offset + block->size))
10443 {
10444 VMA_ASSERT(0 && "MEMORY CORRUPTION DETECTED AFTER VALIDATED ALLOCATION!");
10445 return VK_ERROR_UNKNOWN_COPY;
10446 }
10447 }
10448 }
10449
10450 return VK_SUCCESS;
10451}
10452
10453void VmaBlockMetadata_TLSF::Alloc(
10454 const VmaAllocationRequest& request,
10455 VmaSuballocationType type,
10456 void* userData)
10457{
10458 VMA_ASSERT(request.type == VmaAllocationRequestType::TLSF);
10459
10460 // Get block and pop it from the free list
10461 Block* currentBlock = (Block*)request.allocHandle;
10462 VkDeviceSize offset = request.algorithmData;
10463 VMA_ASSERT(currentBlock != VMA_NULL);
10464 VMA_ASSERT(currentBlock->offset <= offset);
10465
10466 if (currentBlock != m_NullBlock)
10467 RemoveFreeBlock(block: currentBlock);
10468
10469 VkDeviceSize debugMargin = GetDebugMargin();
10470 VkDeviceSize misssingAlignment = offset - currentBlock->offset;
10471
10472 // Append missing alignment to prev block or create new one
10473 if (misssingAlignment)
10474 {
10475 Block* prevBlock = currentBlock->prevPhysical;
10476 VMA_ASSERT(prevBlock != VMA_NULL && "There should be no missing alignment at offset 0!");
10477
10478 if (prevBlock->IsFree() && prevBlock->size != debugMargin)
10479 {
10480 uint32_t oldList = GetListIndex(size: prevBlock->size);
10481 prevBlock->size += misssingAlignment;
10482 // Check if new size crosses list bucket
10483 if (oldList != GetListIndex(size: prevBlock->size))
10484 {
10485 prevBlock->size -= misssingAlignment;
10486 RemoveFreeBlock(block: prevBlock);
10487 prevBlock->size += misssingAlignment;
10488 InsertFreeBlock(block: prevBlock);
10489 }
10490 else
10491 m_BlocksFreeSize += misssingAlignment;
10492 }
10493 else
10494 {
10495 Block* newBlock = m_BlockAllocator.Alloc();
10496 currentBlock->prevPhysical = newBlock;
10497 prevBlock->nextPhysical = newBlock;
10498 newBlock->prevPhysical = prevBlock;
10499 newBlock->nextPhysical = currentBlock;
10500 newBlock->size = misssingAlignment;
10501 newBlock->offset = currentBlock->offset;
10502 newBlock->MarkTaken();
10503
10504 InsertFreeBlock(block: newBlock);
10505 }
10506
10507 currentBlock->size -= misssingAlignment;
10508 currentBlock->offset += misssingAlignment;
10509 }
10510
10511 VkDeviceSize size = request.size + debugMargin;
10512 if (currentBlock->size == size)
10513 {
10514 if (currentBlock == m_NullBlock)
10515 {
10516 // Setup new null block
10517 m_NullBlock = m_BlockAllocator.Alloc();
10518 m_NullBlock->size = 0;
10519 m_NullBlock->offset = currentBlock->offset + size;
10520 m_NullBlock->prevPhysical = currentBlock;
10521 m_NullBlock->nextPhysical = VMA_NULL;
10522 m_NullBlock->MarkFree();
10523 m_NullBlock->PrevFree() = VMA_NULL;
10524 m_NullBlock->NextFree() = VMA_NULL;
10525 currentBlock->nextPhysical = m_NullBlock;
10526 currentBlock->MarkTaken();
10527 }
10528 }
10529 else
10530 {
10531 VMA_ASSERT(currentBlock->size > size && "Proper block already found, shouldn't find smaller one!");
10532
10533 // Create new free block
10534 Block* newBlock = m_BlockAllocator.Alloc();
10535 newBlock->size = currentBlock->size - size;
10536 newBlock->offset = currentBlock->offset + size;
10537 newBlock->prevPhysical = currentBlock;
10538 newBlock->nextPhysical = currentBlock->nextPhysical;
10539 currentBlock->nextPhysical = newBlock;
10540 currentBlock->size = size;
10541
10542 if (currentBlock == m_NullBlock)
10543 {
10544 m_NullBlock = newBlock;
10545 m_NullBlock->MarkFree();
10546 m_NullBlock->NextFree() = VMA_NULL;
10547 m_NullBlock->PrevFree() = VMA_NULL;
10548 currentBlock->MarkTaken();
10549 }
10550 else
10551 {
10552 newBlock->nextPhysical->prevPhysical = newBlock;
10553 newBlock->MarkTaken();
10554 InsertFreeBlock(block: newBlock);
10555 }
10556 }
10557 currentBlock->UserData() = userData;
10558
10559 if (debugMargin > 0)
10560 {
10561 currentBlock->size -= debugMargin;
10562 Block* newBlock = m_BlockAllocator.Alloc();
10563 newBlock->size = debugMargin;
10564 newBlock->offset = currentBlock->offset + currentBlock->size;
10565 newBlock->prevPhysical = currentBlock;
10566 newBlock->nextPhysical = currentBlock->nextPhysical;
10567 newBlock->MarkTaken();
10568 currentBlock->nextPhysical->prevPhysical = newBlock;
10569 currentBlock->nextPhysical = newBlock;
10570 InsertFreeBlock(block: newBlock);
10571 }
10572
10573 if (!IsVirtual())
10574 m_GranularityHandler.AllocPages(allocType: (uint8_t)(uintptr_t)request.customData,
10575 offset: currentBlock->offset, size: currentBlock->size);
10576 ++m_AllocCount;
10577}
10578
10579void VmaBlockMetadata_TLSF::Free(VmaAllocHandle allocHandle)
10580{
10581 Block* block = (Block*)allocHandle;
10582 Block* next = block->nextPhysical;
10583 VMA_ASSERT(!block->IsFree() && "Block is already free!");
10584
10585 if (!IsVirtual())
10586 m_GranularityHandler.FreePages(offset: block->offset, size: block->size);
10587 --m_AllocCount;
10588
10589 VkDeviceSize debugMargin = GetDebugMargin();
10590 if (debugMargin > 0)
10591 {
10592 RemoveFreeBlock(block: next);
10593 MergeBlock(block: next, prev: block);
10594 block = next;
10595 next = next->nextPhysical;
10596 }
10597
10598 // Try merging
10599 Block* prev = block->prevPhysical;
10600 if (prev != VMA_NULL && prev->IsFree() && prev->size != debugMargin)
10601 {
10602 RemoveFreeBlock(block: prev);
10603 MergeBlock(block, prev);
10604 }
10605
10606 if (!next->IsFree())
10607 InsertFreeBlock(block);
10608 else if (next == m_NullBlock)
10609 MergeBlock(block: m_NullBlock, prev: block);
10610 else
10611 {
10612 RemoveFreeBlock(block: next);
10613 MergeBlock(block: next, prev: block);
10614 InsertFreeBlock(block: next);
10615 }
10616}
10617
10618void VmaBlockMetadata_TLSF::GetAllocationInfo(VmaAllocHandle allocHandle, VmaVirtualAllocationInfo& outInfo)
10619{
10620 Block* block = (Block*)allocHandle;
10621 VMA_ASSERT(!block->IsFree() && "Cannot get allocation info for free block!");
10622 outInfo.offset = block->offset;
10623 outInfo.size = block->size;
10624 outInfo.pUserData = block->UserData();
10625}
10626
10627void* VmaBlockMetadata_TLSF::GetAllocationUserData(VmaAllocHandle allocHandle) const
10628{
10629 Block* block = (Block*)allocHandle;
10630 VMA_ASSERT(!block->IsFree() && "Cannot get user data for free block!");
10631 return block->UserData();
10632}
10633
10634VmaAllocHandle VmaBlockMetadata_TLSF::GetAllocationListBegin() const
10635{
10636 if (m_AllocCount == 0)
10637 return VK_NULL_HANDLE;
10638
10639 for (Block* block = m_NullBlock->prevPhysical; block; block = block->prevPhysical)
10640 {
10641 if (!block->IsFree())
10642 return (VmaAllocHandle)block;
10643 }
10644 VMA_ASSERT(false && "If m_AllocCount > 0 then should find any allocation!");
10645 return VK_NULL_HANDLE;
10646}
10647
10648VmaAllocHandle VmaBlockMetadata_TLSF::GetNextAllocation(VmaAllocHandle prevAlloc) const
10649{
10650 Block* startBlock = (Block*)prevAlloc;
10651 VMA_ASSERT(!startBlock->IsFree() && "Incorrect block!");
10652
10653 for (Block* block = startBlock->prevPhysical; block; block = block->prevPhysical)
10654 {
10655 if (!block->IsFree())
10656 return (VmaAllocHandle)block;
10657 }
10658 return VK_NULL_HANDLE;
10659}
10660
10661VkDeviceSize VmaBlockMetadata_TLSF::GetNextFreeRegionSize(VmaAllocHandle alloc) const
10662{
10663 Block* block = (Block*)alloc;
10664 VMA_ASSERT(!block->IsFree() && "Incorrect block!");
10665
10666 if (block->prevPhysical)
10667 return block->prevPhysical->IsFree() ? block->prevPhysical->size : 0;
10668 return 0;
10669}
10670
10671void VmaBlockMetadata_TLSF::Clear()
10672{
10673 m_AllocCount = 0;
10674 m_BlocksFreeCount = 0;
10675 m_BlocksFreeSize = 0;
10676 m_IsFreeBitmap = 0;
10677 m_NullBlock->offset = 0;
10678 m_NullBlock->size = GetSize();
10679 Block* block = m_NullBlock->prevPhysical;
10680 m_NullBlock->prevPhysical = VMA_NULL;
10681 while (block)
10682 {
10683 Block* prev = block->prevPhysical;
10684 m_BlockAllocator.Free(ptr: block);
10685 block = prev;
10686 }
10687 memset(s: m_FreeList, c: 0, n: m_ListsCount * sizeof(Block*));
10688 memset(s: m_InnerIsFreeBitmap, c: 0, n: m_MemoryClasses * sizeof(uint32_t));
10689 m_GranularityHandler.Clear();
10690}
10691
10692void VmaBlockMetadata_TLSF::SetAllocationUserData(VmaAllocHandle allocHandle, void* userData)
10693{
10694 Block* block = (Block*)allocHandle;
10695 VMA_ASSERT(!block->IsFree() && "Trying to set user data for not allocated block!");
10696 block->UserData() = userData;
10697}
10698
10699void VmaBlockMetadata_TLSF::DebugLogAllAllocations() const
10700{
10701 for (Block* block = m_NullBlock->prevPhysical; block != VMA_NULL; block = block->prevPhysical)
10702 if (!block->IsFree())
10703 DebugLogAllocation(offset: block->offset, size: block->size, userData: block->UserData());
10704}
10705
10706uint8_t VmaBlockMetadata_TLSF::SizeToMemoryClass(VkDeviceSize size) const
10707{
10708 if (size > SMALL_BUFFER_SIZE)
10709 return VMA_BITSCAN_MSB(size) - MEMORY_CLASS_SHIFT;
10710 return 0;
10711}
10712
10713uint16_t VmaBlockMetadata_TLSF::SizeToSecondIndex(VkDeviceSize size, uint8_t memoryClass) const
10714{
10715 if (memoryClass == 0)
10716 {
10717 if (IsVirtual())
10718 return static_cast<uint16_t>((size - 1) / 8);
10719 else
10720 return static_cast<uint16_t>((size - 1) / 64);
10721 }
10722 return static_cast<uint16_t>((size >> (memoryClass + MEMORY_CLASS_SHIFT - SECOND_LEVEL_INDEX)) ^ (1U << SECOND_LEVEL_INDEX));
10723}
10724
10725uint32_t VmaBlockMetadata_TLSF::GetListIndex(uint8_t memoryClass, uint16_t secondIndex) const
10726{
10727 if (memoryClass == 0)
10728 return secondIndex;
10729
10730 const uint32_t index = static_cast<uint32_t>(memoryClass - 1) * (1 << SECOND_LEVEL_INDEX) + secondIndex;
10731 if (IsVirtual())
10732 return index + (1 << SECOND_LEVEL_INDEX);
10733 else
10734 return index + 4;
10735}
10736
10737uint32_t VmaBlockMetadata_TLSF::GetListIndex(VkDeviceSize size) const
10738{
10739 uint8_t memoryClass = SizeToMemoryClass(size);
10740 return GetListIndex(memoryClass, secondIndex: SizeToSecondIndex(size, memoryClass));
10741}
10742
10743void VmaBlockMetadata_TLSF::RemoveFreeBlock(Block* block)
10744{
10745 VMA_ASSERT(block != m_NullBlock);
10746 VMA_ASSERT(block->IsFree());
10747
10748 if (block->NextFree() != VMA_NULL)
10749 block->NextFree()->PrevFree() = block->PrevFree();
10750 if (block->PrevFree() != VMA_NULL)
10751 block->PrevFree()->NextFree() = block->NextFree();
10752 else
10753 {
10754 uint8_t memClass = SizeToMemoryClass(size: block->size);
10755 uint16_t secondIndex = SizeToSecondIndex(size: block->size, memoryClass: memClass);
10756 uint32_t index = GetListIndex(memoryClass: memClass, secondIndex);
10757 VMA_ASSERT(m_FreeList[index] == block);
10758 m_FreeList[index] = block->NextFree();
10759 if (block->NextFree() == VMA_NULL)
10760 {
10761 m_InnerIsFreeBitmap[memClass] &= ~(1U << secondIndex);
10762 if (m_InnerIsFreeBitmap[memClass] == 0)
10763 m_IsFreeBitmap &= ~(1UL << memClass);
10764 }
10765 }
10766 block->MarkTaken();
10767 block->UserData() = VMA_NULL;
10768 --m_BlocksFreeCount;
10769 m_BlocksFreeSize -= block->size;
10770}
10771
10772void VmaBlockMetadata_TLSF::InsertFreeBlock(Block* block)
10773{
10774 VMA_ASSERT(block != m_NullBlock);
10775 VMA_ASSERT(!block->IsFree() && "Cannot insert block twice!");
10776
10777 uint8_t memClass = SizeToMemoryClass(size: block->size);
10778 uint16_t secondIndex = SizeToSecondIndex(size: block->size, memoryClass: memClass);
10779 uint32_t index = GetListIndex(memoryClass: memClass, secondIndex);
10780 VMA_ASSERT(index < m_ListsCount);
10781 block->PrevFree() = VMA_NULL;
10782 block->NextFree() = m_FreeList[index];
10783 m_FreeList[index] = block;
10784 if (block->NextFree() != VMA_NULL)
10785 block->NextFree()->PrevFree() = block;
10786 else
10787 {
10788 m_InnerIsFreeBitmap[memClass] |= 1U << secondIndex;
10789 m_IsFreeBitmap |= 1UL << memClass;
10790 }
10791 ++m_BlocksFreeCount;
10792 m_BlocksFreeSize += block->size;
10793}
10794
10795void VmaBlockMetadata_TLSF::MergeBlock(Block* block, Block* prev)
10796{
10797 VMA_ASSERT(block->prevPhysical == prev && "Cannot merge seperate physical regions!");
10798 VMA_ASSERT(!prev->IsFree() && "Cannot merge block that belongs to free list!");
10799
10800 block->offset = prev->offset;
10801 block->size += prev->size;
10802 block->prevPhysical = prev->prevPhysical;
10803 if (block->prevPhysical)
10804 block->prevPhysical->nextPhysical = block;
10805 m_BlockAllocator.Free(ptr: prev);
10806}
10807
10808VmaBlockMetadata_TLSF::Block* VmaBlockMetadata_TLSF::FindFreeBlock(VkDeviceSize size, uint32_t& listIndex) const
10809{
10810 uint8_t memoryClass = SizeToMemoryClass(size);
10811 uint32_t innerFreeMap = m_InnerIsFreeBitmap[memoryClass] & (~0U << SizeToSecondIndex(size, memoryClass));
10812 if (!innerFreeMap)
10813 {
10814 // Check higher levels for avaiable blocks
10815 uint32_t freeMap = m_IsFreeBitmap & (~0UL << (memoryClass + 1));
10816 if (!freeMap)
10817 return VMA_NULL; // No more memory avaible
10818
10819 // Find lowest free region
10820 memoryClass = VMA_BITSCAN_LSB(freeMap);
10821 innerFreeMap = m_InnerIsFreeBitmap[memoryClass];
10822 VMA_ASSERT(innerFreeMap != 0);
10823 }
10824 // Find lowest free subregion
10825 listIndex = GetListIndex(memoryClass, VMA_BITSCAN_LSB(innerFreeMap));
10826 VMA_ASSERT(m_FreeList[listIndex]);
10827 return m_FreeList[listIndex];
10828}
10829
10830bool VmaBlockMetadata_TLSF::CheckBlock(
10831 Block& block,
10832 uint32_t listIndex,
10833 VkDeviceSize allocSize,
10834 VkDeviceSize allocAlignment,
10835 VmaSuballocationType allocType,
10836 VmaAllocationRequest* pAllocationRequest)
10837{
10838 VMA_ASSERT(block.IsFree() && "Block is already taken!");
10839
10840 VkDeviceSize alignedOffset = VmaAlignUp(val: block.offset, alignment: allocAlignment);
10841 if (block.size < allocSize + alignedOffset - block.offset)
10842 return false;
10843
10844 // Check for granularity conflicts
10845 if (!IsVirtual() &&
10846 m_GranularityHandler.CheckConflictAndAlignUp(inOutAllocOffset&: alignedOffset, allocSize, blockOffset: block.offset, blockSize: block.size, allocType))
10847 return false;
10848
10849 // Alloc successful
10850 pAllocationRequest->type = VmaAllocationRequestType::TLSF;
10851 pAllocationRequest->allocHandle = (VmaAllocHandle)&block;
10852 pAllocationRequest->size = allocSize - GetDebugMargin();
10853 pAllocationRequest->customData = (void*)allocType;
10854 pAllocationRequest->algorithmData = alignedOffset;
10855
10856 // Place block at the start of list if it's normal block
10857 if (listIndex != m_ListsCount && block.PrevFree())
10858 {
10859 block.PrevFree()->NextFree() = block.NextFree();
10860 if (block.NextFree())
10861 block.NextFree()->PrevFree() = block.PrevFree();
10862 block.PrevFree() = VMA_NULL;
10863 block.NextFree() = m_FreeList[listIndex];
10864 m_FreeList[listIndex] = &block;
10865 if (block.NextFree())
10866 block.NextFree()->PrevFree() = &block;
10867 }
10868
10869 return true;
10870}
10871#endif // _VMA_BLOCK_METADATA_TLSF_FUNCTIONS
10872#endif // _VMA_BLOCK_METADATA_TLSF
10873
10874#ifndef _VMA_BLOCK_VECTOR
10875/*
10876Sequence of VmaDeviceMemoryBlock. Represents memory blocks allocated for a specific
10877Vulkan memory type.
10878
10879Synchronized internally with a mutex.
10880*/
10881class VmaBlockVector
10882{
10883 friend struct VmaDefragmentationContext_T;
10884 VMA_CLASS_NO_COPY(VmaBlockVector)
10885public:
10886 VmaBlockVector(
10887 VmaAllocator hAllocator,
10888 VmaPool hParentPool,
10889 uint32_t memoryTypeIndex,
10890 VkDeviceSize preferredBlockSize,
10891 size_t minBlockCount,
10892 size_t maxBlockCount,
10893 VkDeviceSize bufferImageGranularity,
10894 bool explicitBlockSize,
10895 uint32_t algorithm,
10896 float priority,
10897 VkDeviceSize minAllocationAlignment,
10898 void* pMemoryAllocateNext);
10899 ~VmaBlockVector();
10900
10901 VmaAllocator GetAllocator() const { return m_hAllocator; }
10902 VmaPool GetParentPool() const { return m_hParentPool; }
10903 bool IsCustomPool() const { return m_hParentPool != VMA_NULL; }
10904 uint32_t GetMemoryTypeIndex() const { return m_MemoryTypeIndex; }
10905 VkDeviceSize GetPreferredBlockSize() const { return m_PreferredBlockSize; }
10906 VkDeviceSize GetBufferImageGranularity() const { return m_BufferImageGranularity; }
10907 uint32_t GetAlgorithm() const { return m_Algorithm; }
10908 bool HasExplicitBlockSize() const { return m_ExplicitBlockSize; }
10909 float GetPriority() const { return m_Priority; }
10910 const void* GetAllocationNextPtr() const { return m_pMemoryAllocateNext; }
10911 // To be used only while the m_Mutex is locked. Used during defragmentation.
10912 size_t GetBlockCount() const { return m_Blocks.size(); }
10913 // To be used only while the m_Mutex is locked. Used during defragmentation.
10914 VmaDeviceMemoryBlock* GetBlock(size_t index) const { return m_Blocks[index]; }
10915 VMA_RW_MUTEX &GetMutex() { return m_Mutex; }
10916
10917 VkResult CreateMinBlocks();
10918 void AddStatistics(VmaStatistics& inoutStats);
10919 void AddDetailedStatistics(VmaDetailedStatistics& inoutStats);
10920 bool IsEmpty();
10921 bool IsCorruptionDetectionEnabled() const;
10922
10923 VkResult Allocate(
10924 VkDeviceSize size,
10925 VkDeviceSize alignment,
10926 const VmaAllocationCreateInfo& createInfo,
10927 VmaSuballocationType suballocType,
10928 size_t allocationCount,
10929 VmaAllocation* pAllocations);
10930
10931 void Free(const VmaAllocation hAllocation);
10932
10933#if VMA_STATS_STRING_ENABLED
10934 void PrintDetailedMap(class VmaJsonWriter& json);
10935#endif
10936
10937 VkResult CheckCorruption();
10938
10939private:
10940 const VmaAllocator m_hAllocator;
10941 const VmaPool m_hParentPool;
10942 const uint32_t m_MemoryTypeIndex;
10943 const VkDeviceSize m_PreferredBlockSize;
10944 const size_t m_MinBlockCount;
10945 const size_t m_MaxBlockCount;
10946 const VkDeviceSize m_BufferImageGranularity;
10947 const bool m_ExplicitBlockSize;
10948 const uint32_t m_Algorithm;
10949 const float m_Priority;
10950 const VkDeviceSize m_MinAllocationAlignment;
10951
10952 void* const m_pMemoryAllocateNext;
10953 VMA_RW_MUTEX m_Mutex;
10954 // Incrementally sorted by sumFreeSize, ascending.
10955 VmaVector<VmaDeviceMemoryBlock*, VmaStlAllocator<VmaDeviceMemoryBlock*>> m_Blocks;
10956 uint32_t m_NextBlockId;
10957 bool m_IncrementalSort = true;
10958
10959 void SetIncrementalSort(bool val) { m_IncrementalSort = val; }
10960
10961 VkDeviceSize CalcMaxBlockSize() const;
10962 // Finds and removes given block from vector.
10963 void Remove(VmaDeviceMemoryBlock* pBlock);
10964 // Performs single step in sorting m_Blocks. They may not be fully sorted
10965 // after this call.
10966 void IncrementallySortBlocks();
10967 void SortByFreeSize();
10968
10969 VkResult AllocatePage(
10970 VkDeviceSize size,
10971 VkDeviceSize alignment,
10972 const VmaAllocationCreateInfo& createInfo,
10973 VmaSuballocationType suballocType,
10974 VmaAllocation* pAllocation);
10975
10976 VkResult AllocateFromBlock(
10977 VmaDeviceMemoryBlock* pBlock,
10978 VkDeviceSize size,
10979 VkDeviceSize alignment,
10980 VmaAllocationCreateFlags allocFlags,
10981 void* pUserData,
10982 VmaSuballocationType suballocType,
10983 uint32_t strategy,
10984 VmaAllocation* pAllocation);
10985
10986 VkResult CommitAllocationRequest(
10987 VmaAllocationRequest& allocRequest,
10988 VmaDeviceMemoryBlock* pBlock,
10989 VkDeviceSize alignment,
10990 VmaAllocationCreateFlags allocFlags,
10991 void* pUserData,
10992 VmaSuballocationType suballocType,
10993 VmaAllocation* pAllocation);
10994
10995 VkResult CreateBlock(VkDeviceSize blockSize, size_t* pNewBlockIndex);
10996 bool HasEmptyBlock();
10997};
10998#endif // _VMA_BLOCK_VECTOR
10999
11000#ifndef _VMA_DEFRAGMENTATION_CONTEXT
11001struct VmaDefragmentationContext_T
11002{
11003 VMA_CLASS_NO_COPY(VmaDefragmentationContext_T)
11004public:
11005 VmaDefragmentationContext_T(
11006 VmaAllocator hAllocator,
11007 const VmaDefragmentationInfo& info);
11008 ~VmaDefragmentationContext_T();
11009
11010 void GetStats(VmaDefragmentationStats& outStats) { outStats = m_GlobalStats; }
11011
11012 VkResult DefragmentPassBegin(VmaDefragmentationPassMoveInfo& moveInfo);
11013 VkResult DefragmentPassEnd(VmaDefragmentationPassMoveInfo& moveInfo);
11014
11015private:
11016 // Max number of allocations to ignore due to size constraints before ending single pass
11017 static const uint8_t MAX_ALLOCS_TO_IGNORE = 16;
11018 enum class CounterStatus { Pass, Ignore, End };
11019
11020 struct FragmentedBlock
11021 {
11022 uint32_t data;
11023 VmaDeviceMemoryBlock* block;
11024 };
11025 struct StateBalanced
11026 {
11027 VkDeviceSize avgFreeSize = 0;
11028 VkDeviceSize avgAllocSize = UINT64_MAX;
11029 };
11030 struct StateExtensive
11031 {
11032 enum class Operation : uint8_t
11033 {
11034 FindFreeBlockBuffer, FindFreeBlockTexture, FindFreeBlockAll,
11035 MoveBuffers, MoveTextures, MoveAll,
11036 Cleanup, Done
11037 };
11038
11039 Operation operation = Operation::FindFreeBlockTexture;
11040 size_t firstFreeBlock = SIZE_MAX;
11041 };
11042 struct MoveAllocationData
11043 {
11044 VkDeviceSize size;
11045 VkDeviceSize alignment;
11046 VmaSuballocationType type;
11047 VmaAllocationCreateFlags flags;
11048 VmaDefragmentationMove move = {};
11049 };
11050
11051 const VkDeviceSize m_MaxPassBytes;
11052 const uint32_t m_MaxPassAllocations;
11053
11054 VmaStlAllocator<VmaDefragmentationMove> m_MoveAllocator;
11055 VmaVector<VmaDefragmentationMove, VmaStlAllocator<VmaDefragmentationMove>> m_Moves;
11056
11057 uint8_t m_IgnoredAllocs = 0;
11058 uint32_t m_Algorithm;
11059 uint32_t m_BlockVectorCount;
11060 VmaBlockVector* m_PoolBlockVector;
11061 VmaBlockVector** m_pBlockVectors;
11062 size_t m_ImmovableBlockCount = 0;
11063 VmaDefragmentationStats m_GlobalStats = { .bytesMoved: 0 };
11064 VmaDefragmentationStats m_PassStats = { .bytesMoved: 0 };
11065 void* m_AlgorithmState = VMA_NULL;
11066
11067 static MoveAllocationData GetMoveData(VmaAllocHandle handle, VmaBlockMetadata* metadata);
11068 CounterStatus CheckCounters(VkDeviceSize bytes);
11069 bool IncrementCounters(VkDeviceSize bytes);
11070 bool ReallocWithinBlock(VmaBlockVector& vector, VmaDeviceMemoryBlock* block);
11071 bool AllocInOtherBlock(size_t start, size_t end, MoveAllocationData& data, VmaBlockVector& vector);
11072
11073 bool ComputeDefragmentation(VmaBlockVector& vector, size_t index);
11074 bool ComputeDefragmentation_Fast(VmaBlockVector& vector);
11075 bool ComputeDefragmentation_Balanced(VmaBlockVector& vector, size_t index, bool update);
11076 bool ComputeDefragmentation_Full(VmaBlockVector& vector);
11077 bool ComputeDefragmentation_Extensive(VmaBlockVector& vector, size_t index);
11078
11079 void UpdateVectorStatistics(VmaBlockVector& vector, StateBalanced& state);
11080 bool MoveDataToFreeBlocks(VmaSuballocationType currentType,
11081 VmaBlockVector& vector, size_t firstFreeBlock,
11082 bool& texturePresent, bool& bufferPresent, bool& otherPresent);
11083};
11084#endif // _VMA_DEFRAGMENTATION_CONTEXT
11085
11086#ifndef _VMA_POOL_T
11087struct VmaPool_T
11088{
11089 friend struct VmaPoolListItemTraits;
11090 VMA_CLASS_NO_COPY(VmaPool_T)
11091public:
11092 VmaBlockVector m_BlockVector;
11093 VmaDedicatedAllocationList m_DedicatedAllocations;
11094
11095 VmaPool_T(
11096 VmaAllocator hAllocator,
11097 const VmaPoolCreateInfo& createInfo,
11098 VkDeviceSize preferredBlockSize);
11099 ~VmaPool_T();
11100
11101 uint32_t GetId() const { return m_Id; }
11102 void SetId(uint32_t id) { VMA_ASSERT(m_Id == 0); m_Id = id; }
11103
11104 const char* GetName() const { return m_Name; }
11105 void SetName(const char* pName);
11106
11107#if VMA_STATS_STRING_ENABLED
11108 //void PrintDetailedMap(class VmaStringBuilder& sb);
11109#endif
11110
11111private:
11112 uint32_t m_Id;
11113 char* m_Name;
11114 VmaPool_T* m_PrevPool = VMA_NULL;
11115 VmaPool_T* m_NextPool = VMA_NULL;
11116};
11117
11118struct VmaPoolListItemTraits
11119{
11120 typedef VmaPool_T ItemType;
11121
11122 static ItemType* GetPrev(const ItemType* item) { return item->m_PrevPool; }
11123 static ItemType* GetNext(const ItemType* item) { return item->m_NextPool; }
11124 static ItemType*& AccessPrev(ItemType* item) { return item->m_PrevPool; }
11125 static ItemType*& AccessNext(ItemType* item) { return item->m_NextPool; }
11126};
11127#endif // _VMA_POOL_T
11128
11129#ifndef _VMA_CURRENT_BUDGET_DATA
11130struct VmaCurrentBudgetData
11131{
11132 VMA_ATOMIC_UINT32 m_BlockCount[VK_MAX_MEMORY_HEAPS];
11133 VMA_ATOMIC_UINT32 m_AllocationCount[VK_MAX_MEMORY_HEAPS];
11134 VMA_ATOMIC_UINT64 m_BlockBytes[VK_MAX_MEMORY_HEAPS];
11135 VMA_ATOMIC_UINT64 m_AllocationBytes[VK_MAX_MEMORY_HEAPS];
11136
11137#if VMA_MEMORY_BUDGET
11138 VMA_ATOMIC_UINT32 m_OperationsSinceBudgetFetch;
11139 VMA_RW_MUTEX m_BudgetMutex;
11140 uint64_t m_VulkanUsage[VK_MAX_MEMORY_HEAPS];
11141 uint64_t m_VulkanBudget[VK_MAX_MEMORY_HEAPS];
11142 uint64_t m_BlockBytesAtBudgetFetch[VK_MAX_MEMORY_HEAPS];
11143#endif // VMA_MEMORY_BUDGET
11144
11145 VmaCurrentBudgetData();
11146
11147 void AddAllocation(uint32_t heapIndex, VkDeviceSize allocationSize);
11148 void RemoveAllocation(uint32_t heapIndex, VkDeviceSize allocationSize);
11149};
11150
11151#ifndef _VMA_CURRENT_BUDGET_DATA_FUNCTIONS
11152VmaCurrentBudgetData::VmaCurrentBudgetData()
11153{
11154 for (uint32_t heapIndex = 0; heapIndex < VK_MAX_MEMORY_HEAPS; ++heapIndex)
11155 {
11156 m_BlockCount[heapIndex] = 0;
11157 m_AllocationCount[heapIndex] = 0;
11158 m_BlockBytes[heapIndex] = 0;
11159 m_AllocationBytes[heapIndex] = 0;
11160#if VMA_MEMORY_BUDGET
11161 m_VulkanUsage[heapIndex] = 0;
11162 m_VulkanBudget[heapIndex] = 0;
11163 m_BlockBytesAtBudgetFetch[heapIndex] = 0;
11164#endif
11165 }
11166
11167#if VMA_MEMORY_BUDGET
11168 m_OperationsSinceBudgetFetch = 0;
11169#endif
11170}
11171
11172void VmaCurrentBudgetData::AddAllocation(uint32_t heapIndex, VkDeviceSize allocationSize)
11173{
11174 m_AllocationBytes[heapIndex] += allocationSize;
11175 ++m_AllocationCount[heapIndex];
11176#if VMA_MEMORY_BUDGET
11177 ++m_OperationsSinceBudgetFetch;
11178#endif
11179}
11180
11181void VmaCurrentBudgetData::RemoveAllocation(uint32_t heapIndex, VkDeviceSize allocationSize)
11182{
11183 VMA_ASSERT(m_AllocationBytes[heapIndex] >= allocationSize);
11184 m_AllocationBytes[heapIndex] -= allocationSize;
11185 VMA_ASSERT(m_AllocationCount[heapIndex] > 0);
11186 --m_AllocationCount[heapIndex];
11187#if VMA_MEMORY_BUDGET
11188 ++m_OperationsSinceBudgetFetch;
11189#endif
11190}
11191#endif // _VMA_CURRENT_BUDGET_DATA_FUNCTIONS
11192#endif // _VMA_CURRENT_BUDGET_DATA
11193
11194#ifndef _VMA_ALLOCATION_OBJECT_ALLOCATOR
11195/*
11196Thread-safe wrapper over VmaPoolAllocator free list, for allocation of VmaAllocation_T objects.
11197*/
11198class VmaAllocationObjectAllocator
11199{
11200 VMA_CLASS_NO_COPY(VmaAllocationObjectAllocator)
11201public:
11202 VmaAllocationObjectAllocator(const VkAllocationCallbacks* pAllocationCallbacks)
11203 : m_Allocator(pAllocationCallbacks, 1024) {}
11204
11205 template<typename... Types> VmaAllocation Allocate(Types&&... args);
11206 void Free(VmaAllocation hAlloc);
11207
11208private:
11209 VMA_MUTEX m_Mutex;
11210 VmaPoolAllocator<VmaAllocation_T> m_Allocator;
11211};
11212
11213template<typename... Types>
11214VmaAllocation VmaAllocationObjectAllocator::Allocate(Types&&... args)
11215{
11216 VmaMutexLock mutexLock(m_Mutex);
11217 return m_Allocator.Alloc<Types...>(std::forward<Types>(args)...);
11218}
11219
11220void VmaAllocationObjectAllocator::Free(VmaAllocation hAlloc)
11221{
11222 VmaMutexLock mutexLock(m_Mutex);
11223 m_Allocator.Free(ptr: hAlloc);
11224}
11225#endif // _VMA_ALLOCATION_OBJECT_ALLOCATOR
11226
11227#ifndef _VMA_VIRTUAL_BLOCK_T
11228struct VmaVirtualBlock_T
11229{
11230 VMA_CLASS_NO_COPY(VmaVirtualBlock_T)
11231public:
11232 const bool m_AllocationCallbacksSpecified;
11233 const VkAllocationCallbacks m_AllocationCallbacks;
11234
11235 VmaVirtualBlock_T(const VmaVirtualBlockCreateInfo& createInfo);
11236 ~VmaVirtualBlock_T();
11237
11238 VkResult Init() { return VK_SUCCESS; }
11239 bool IsEmpty() const { return m_Metadata->IsEmpty(); }
11240 void Free(VmaVirtualAllocation allocation) { m_Metadata->Free(allocHandle: (VmaAllocHandle)allocation); }
11241 void SetAllocationUserData(VmaVirtualAllocation allocation, void* userData) { m_Metadata->SetAllocationUserData(allocHandle: (VmaAllocHandle)allocation, userData); }
11242 void Clear() { m_Metadata->Clear(); }
11243
11244 const VkAllocationCallbacks* GetAllocationCallbacks() const;
11245 void GetAllocationInfo(VmaVirtualAllocation allocation, VmaVirtualAllocationInfo& outInfo);
11246 VkResult Allocate(const VmaVirtualAllocationCreateInfo& createInfo, VmaVirtualAllocation& outAllocation,
11247 VkDeviceSize* outOffset);
11248 void GetStatistics(VmaStatistics& outStats) const;
11249 void CalculateDetailedStatistics(VmaDetailedStatistics& outStats) const;
11250#if VMA_STATS_STRING_ENABLED
11251 void BuildStatsString(bool detailedMap, VmaStringBuilder& sb) const;
11252#endif
11253
11254private:
11255 VmaBlockMetadata* m_Metadata;
11256};
11257
11258#ifndef _VMA_VIRTUAL_BLOCK_T_FUNCTIONS
11259VmaVirtualBlock_T::VmaVirtualBlock_T(const VmaVirtualBlockCreateInfo& createInfo)
11260 : m_AllocationCallbacksSpecified(createInfo.pAllocationCallbacks != VMA_NULL),
11261 m_AllocationCallbacks(createInfo.pAllocationCallbacks != VMA_NULL ? *createInfo.pAllocationCallbacks : VmaEmptyAllocationCallbacks)
11262{
11263 const uint32_t algorithm = createInfo.flags & VMA_VIRTUAL_BLOCK_CREATE_ALGORITHM_MASK;
11264 switch (algorithm)
11265 {
11266 default:
11267 VMA_ASSERT(0);
11268 case 0:
11269 m_Metadata = vma_new(GetAllocationCallbacks(), VmaBlockMetadata_TLSF)(VK_NULL_HANDLE, 1, true);
11270 break;
11271 case VMA_VIRTUAL_BLOCK_CREATE_LINEAR_ALGORITHM_BIT:
11272 m_Metadata = vma_new(GetAllocationCallbacks(), VmaBlockMetadata_Linear)(VK_NULL_HANDLE, 1, true);
11273 break;
11274 }
11275
11276 m_Metadata->Init(size: createInfo.size);
11277}
11278
11279VmaVirtualBlock_T::~VmaVirtualBlock_T()
11280{
11281 // Define macro VMA_DEBUG_LOG to receive the list of the unfreed allocations
11282 if (!m_Metadata->IsEmpty())
11283 m_Metadata->DebugLogAllAllocations();
11284 // This is the most important assert in the entire library.
11285 // Hitting it means you have some memory leak - unreleased virtual allocations.
11286 VMA_ASSERT(m_Metadata->IsEmpty() && "Some virtual allocations were not freed before destruction of this virtual block!");
11287
11288 vma_delete(pAllocationCallbacks: GetAllocationCallbacks(), ptr: m_Metadata);
11289}
11290
11291const VkAllocationCallbacks* VmaVirtualBlock_T::GetAllocationCallbacks() const
11292{
11293 return m_AllocationCallbacksSpecified ? &m_AllocationCallbacks : VMA_NULL;
11294}
11295
11296void VmaVirtualBlock_T::GetAllocationInfo(VmaVirtualAllocation allocation, VmaVirtualAllocationInfo& outInfo)
11297{
11298 m_Metadata->GetAllocationInfo(allocHandle: (VmaAllocHandle)allocation, outInfo);
11299}
11300
11301VkResult VmaVirtualBlock_T::Allocate(const VmaVirtualAllocationCreateInfo& createInfo, VmaVirtualAllocation& outAllocation,
11302 VkDeviceSize* outOffset)
11303{
11304 VmaAllocationRequest request = {};
11305 if (m_Metadata->CreateAllocationRequest(
11306 allocSize: createInfo.size, // allocSize
11307 VMA_MAX(createInfo.alignment, (VkDeviceSize)1), // allocAlignment
11308 upperAddress: (createInfo.flags & VMA_VIRTUAL_ALLOCATION_CREATE_UPPER_ADDRESS_BIT) != 0, // upperAddress
11309 allocType: VMA_SUBALLOCATION_TYPE_UNKNOWN, // allocType - unimportant
11310 strategy: createInfo.flags & VMA_VIRTUAL_ALLOCATION_CREATE_STRATEGY_MASK, // strategy
11311 pAllocationRequest: &request))
11312 {
11313 m_Metadata->Alloc(request,
11314 type: VMA_SUBALLOCATION_TYPE_UNKNOWN, // type - unimportant
11315 userData: createInfo.pUserData);
11316 outAllocation = (VmaVirtualAllocation)request.allocHandle;
11317 if(outOffset)
11318 *outOffset = m_Metadata->GetAllocationOffset(allocHandle: request.allocHandle);
11319 return VK_SUCCESS;
11320 }
11321 outAllocation = (VmaVirtualAllocation)VK_NULL_HANDLE;
11322 if (outOffset)
11323 *outOffset = UINT64_MAX;
11324 return VK_ERROR_OUT_OF_DEVICE_MEMORY;
11325}
11326
11327void VmaVirtualBlock_T::GetStatistics(VmaStatistics& outStats) const
11328{
11329 VmaClearStatistics(outStats);
11330 m_Metadata->AddStatistics(inoutStats&: outStats);
11331}
11332
11333void VmaVirtualBlock_T::CalculateDetailedStatistics(VmaDetailedStatistics& outStats) const
11334{
11335 VmaClearDetailedStatistics(outStats);
11336 m_Metadata->AddDetailedStatistics(inoutStats&: outStats);
11337}
11338
11339#if VMA_STATS_STRING_ENABLED
11340void VmaVirtualBlock_T::BuildStatsString(bool detailedMap, VmaStringBuilder& sb) const
11341{
11342 VmaJsonWriter json(GetAllocationCallbacks(), sb);
11343 json.BeginObject();
11344
11345 VmaDetailedStatistics stats;
11346 CalculateDetailedStatistics(outStats&: stats);
11347
11348 json.WriteString(pStr: "Stats");
11349 VmaPrintDetailedStatistics(json, stat: stats);
11350
11351 if (detailedMap)
11352 {
11353 json.WriteString(pStr: "Details");
11354 json.BeginObject();
11355 m_Metadata->PrintDetailedMap(json);
11356 json.EndObject();
11357 }
11358
11359 json.EndObject();
11360}
11361#endif // VMA_STATS_STRING_ENABLED
11362#endif // _VMA_VIRTUAL_BLOCK_T_FUNCTIONS
11363#endif // _VMA_VIRTUAL_BLOCK_T
11364
11365
11366// Main allocator object.
11367struct VmaAllocator_T
11368{
11369 VMA_CLASS_NO_COPY(VmaAllocator_T)
11370public:
11371 bool m_UseMutex;
11372 uint32_t m_VulkanApiVersion;
11373 bool m_UseKhrDedicatedAllocation; // Can be set only if m_VulkanApiVersion < VK_MAKE_VERSION(1, 1, 0).
11374 bool m_UseKhrBindMemory2; // Can be set only if m_VulkanApiVersion < VK_MAKE_VERSION(1, 1, 0).
11375 bool m_UseExtMemoryBudget;
11376 bool m_UseAmdDeviceCoherentMemory;
11377 bool m_UseKhrBufferDeviceAddress;
11378 bool m_UseExtMemoryPriority;
11379 VkDevice m_hDevice;
11380 VkInstance m_hInstance;
11381 bool m_AllocationCallbacksSpecified;
11382 VkAllocationCallbacks m_AllocationCallbacks;
11383 VmaDeviceMemoryCallbacks m_DeviceMemoryCallbacks;
11384 VmaAllocationObjectAllocator m_AllocationObjectAllocator;
11385
11386 // Each bit (1 << i) is set if HeapSizeLimit is enabled for that heap, so cannot allocate more than the heap size.
11387 uint32_t m_HeapSizeLimitMask;
11388
11389 VkPhysicalDeviceProperties m_PhysicalDeviceProperties;
11390 VkPhysicalDeviceMemoryProperties m_MemProps;
11391
11392 // Default pools.
11393 VmaBlockVector* m_pBlockVectors[VK_MAX_MEMORY_TYPES];
11394 VmaDedicatedAllocationList m_DedicatedAllocations[VK_MAX_MEMORY_TYPES];
11395
11396 VmaCurrentBudgetData m_Budget;
11397 VMA_ATOMIC_UINT32 m_DeviceMemoryCount; // Total number of VkDeviceMemory objects.
11398
11399 VmaAllocator_T(const VmaAllocatorCreateInfo* pCreateInfo);
11400 VkResult Init(const VmaAllocatorCreateInfo* pCreateInfo);
11401 ~VmaAllocator_T();
11402
11403 const VkAllocationCallbacks* GetAllocationCallbacks() const
11404 {
11405 return m_AllocationCallbacksSpecified ? &m_AllocationCallbacks : VMA_NULL;
11406 }
11407 const VmaVulkanFunctions& GetVulkanFunctions() const
11408 {
11409 return m_VulkanFunctions;
11410 }
11411
11412 VkPhysicalDevice GetPhysicalDevice() const { return m_PhysicalDevice; }
11413
11414 VkDeviceSize GetBufferImageGranularity() const
11415 {
11416 return VMA_MAX(
11417 static_cast<VkDeviceSize>(VMA_DEBUG_MIN_BUFFER_IMAGE_GRANULARITY),
11418 m_PhysicalDeviceProperties.limits.bufferImageGranularity);
11419 }
11420
11421 uint32_t GetMemoryHeapCount() const { return m_MemProps.memoryHeapCount; }
11422 uint32_t GetMemoryTypeCount() const { return m_MemProps.memoryTypeCount; }
11423
11424 uint32_t MemoryTypeIndexToHeapIndex(uint32_t memTypeIndex) const
11425 {
11426 VMA_ASSERT(memTypeIndex < m_MemProps.memoryTypeCount);
11427 return m_MemProps.memoryTypes[memTypeIndex].heapIndex;
11428 }
11429 // True when specific memory type is HOST_VISIBLE but not HOST_COHERENT.
11430 bool IsMemoryTypeNonCoherent(uint32_t memTypeIndex) const
11431 {
11432 return (m_MemProps.memoryTypes[memTypeIndex].propertyFlags & (VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | VK_MEMORY_PROPERTY_HOST_COHERENT_BIT)) ==
11433 VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT;
11434 }
11435 // Minimum alignment for all allocations in specific memory type.
11436 VkDeviceSize GetMemoryTypeMinAlignment(uint32_t memTypeIndex) const
11437 {
11438 return IsMemoryTypeNonCoherent(memTypeIndex) ?
11439 VMA_MAX((VkDeviceSize)VMA_MIN_ALIGNMENT, m_PhysicalDeviceProperties.limits.nonCoherentAtomSize) :
11440 (VkDeviceSize)VMA_MIN_ALIGNMENT;
11441 }
11442
11443 bool IsIntegratedGpu() const
11444 {
11445 return m_PhysicalDeviceProperties.deviceType == VK_PHYSICAL_DEVICE_TYPE_INTEGRATED_GPU;
11446 }
11447
11448 uint32_t GetGlobalMemoryTypeBits() const { return m_GlobalMemoryTypeBits; }
11449
11450 void GetBufferMemoryRequirements(
11451 VkBuffer hBuffer,
11452 VkMemoryRequirements& memReq,
11453 bool& requiresDedicatedAllocation,
11454 bool& prefersDedicatedAllocation) const;
11455 void GetImageMemoryRequirements(
11456 VkImage hImage,
11457 VkMemoryRequirements& memReq,
11458 bool& requiresDedicatedAllocation,
11459 bool& prefersDedicatedAllocation) const;
11460 VkResult FindMemoryTypeIndex(
11461 uint32_t memoryTypeBits,
11462 const VmaAllocationCreateInfo* pAllocationCreateInfo,
11463 VkFlags bufImgUsage, // VkBufferCreateInfo::usage or VkImageCreateInfo::usage. UINT32_MAX if unknown.
11464 uint32_t* pMemoryTypeIndex) const;
11465
11466 // Main allocation function.
11467 VkResult AllocateMemory(
11468 const VkMemoryRequirements& vkMemReq,
11469 bool requiresDedicatedAllocation,
11470 bool prefersDedicatedAllocation,
11471 VkBuffer dedicatedBuffer,
11472 VkImage dedicatedImage,
11473 VkFlags dedicatedBufferImageUsage, // UINT32_MAX if unknown.
11474 const VmaAllocationCreateInfo& createInfo,
11475 VmaSuballocationType suballocType,
11476 size_t allocationCount,
11477 VmaAllocation* pAllocations);
11478
11479 // Main deallocation function.
11480 void FreeMemory(
11481 size_t allocationCount,
11482 const VmaAllocation* pAllocations);
11483
11484 void CalculateStatistics(VmaTotalStatistics* pStats);
11485
11486 void GetHeapBudgets(
11487 VmaBudget* outBudgets, uint32_t firstHeap, uint32_t heapCount);
11488
11489#if VMA_STATS_STRING_ENABLED
11490 void PrintDetailedMap(class VmaJsonWriter& json);
11491#endif
11492
11493 void GetAllocationInfo(VmaAllocation hAllocation, VmaAllocationInfo* pAllocationInfo);
11494
11495 VkResult CreatePool(const VmaPoolCreateInfo* pCreateInfo, VmaPool* pPool);
11496 void DestroyPool(VmaPool pool);
11497 void GetPoolStatistics(VmaPool pool, VmaStatistics* pPoolStats);
11498 void CalculatePoolStatistics(VmaPool pool, VmaDetailedStatistics* pPoolStats);
11499
11500 void SetCurrentFrameIndex(uint32_t frameIndex);
11501 uint32_t GetCurrentFrameIndex() const { return m_CurrentFrameIndex.load(); }
11502
11503 VkResult CheckPoolCorruption(VmaPool hPool);
11504 VkResult CheckCorruption(uint32_t memoryTypeBits);
11505
11506 // Call to Vulkan function vkAllocateMemory with accompanying bookkeeping.
11507 VkResult AllocateVulkanMemory(const VkMemoryAllocateInfo* pAllocateInfo, VkDeviceMemory* pMemory);
11508 // Call to Vulkan function vkFreeMemory with accompanying bookkeeping.
11509 void FreeVulkanMemory(uint32_t memoryType, VkDeviceSize size, VkDeviceMemory hMemory);
11510 // Call to Vulkan function vkBindBufferMemory or vkBindBufferMemory2KHR.
11511 VkResult BindVulkanBuffer(
11512 VkDeviceMemory memory,
11513 VkDeviceSize memoryOffset,
11514 VkBuffer buffer,
11515 const void* pNext);
11516 // Call to Vulkan function vkBindImageMemory or vkBindImageMemory2KHR.
11517 VkResult BindVulkanImage(
11518 VkDeviceMemory memory,
11519 VkDeviceSize memoryOffset,
11520 VkImage image,
11521 const void* pNext);
11522
11523 VkResult Map(VmaAllocation hAllocation, void** ppData);
11524 void Unmap(VmaAllocation hAllocation);
11525
11526 VkResult BindBufferMemory(
11527 VmaAllocation hAllocation,
11528 VkDeviceSize allocationLocalOffset,
11529 VkBuffer hBuffer,
11530 const void* pNext);
11531 VkResult BindImageMemory(
11532 VmaAllocation hAllocation,
11533 VkDeviceSize allocationLocalOffset,
11534 VkImage hImage,
11535 const void* pNext);
11536
11537 VkResult FlushOrInvalidateAllocation(
11538 VmaAllocation hAllocation,
11539 VkDeviceSize offset, VkDeviceSize size,
11540 VMA_CACHE_OPERATION op);
11541 VkResult FlushOrInvalidateAllocations(
11542 uint32_t allocationCount,
11543 const VmaAllocation* allocations,
11544 const VkDeviceSize* offsets, const VkDeviceSize* sizes,
11545 VMA_CACHE_OPERATION op);
11546
11547 void FillAllocation(const VmaAllocation hAllocation, uint8_t pattern);
11548
11549 /*
11550 Returns bit mask of memory types that can support defragmentation on GPU as
11551 they support creation of required buffer for copy operations.
11552 */
11553 uint32_t GetGpuDefragmentationMemoryTypeBits();
11554
11555#if VMA_EXTERNAL_MEMORY
11556 VkExternalMemoryHandleTypeFlagsKHR GetExternalMemoryHandleTypeFlags(uint32_t memTypeIndex) const
11557 {
11558 return m_TypeExternalMemoryHandleTypes[memTypeIndex];
11559 }
11560#endif // #if VMA_EXTERNAL_MEMORY
11561
11562private:
11563 VkDeviceSize m_PreferredLargeHeapBlockSize;
11564
11565 VkPhysicalDevice m_PhysicalDevice;
11566 VMA_ATOMIC_UINT32 m_CurrentFrameIndex;
11567 VMA_ATOMIC_UINT32 m_GpuDefragmentationMemoryTypeBits; // UINT32_MAX means uninitialized.
11568#if VMA_EXTERNAL_MEMORY
11569 VkExternalMemoryHandleTypeFlagsKHR m_TypeExternalMemoryHandleTypes[VK_MAX_MEMORY_TYPES];
11570#endif // #if VMA_EXTERNAL_MEMORY
11571
11572 VMA_RW_MUTEX m_PoolsMutex;
11573 typedef VmaIntrusiveLinkedList<VmaPoolListItemTraits> PoolList;
11574 // Protected by m_PoolsMutex.
11575 PoolList m_Pools;
11576 uint32_t m_NextPoolId;
11577
11578 VmaVulkanFunctions m_VulkanFunctions;
11579
11580 // Global bit mask AND-ed with any memoryTypeBits to disallow certain memory types.
11581 uint32_t m_GlobalMemoryTypeBits;
11582
11583 void ImportVulkanFunctions(const VmaVulkanFunctions* pVulkanFunctions);
11584
11585#if VMA_STATIC_VULKAN_FUNCTIONS == 1
11586 void ImportVulkanFunctions_Static();
11587#endif
11588
11589 void ImportVulkanFunctions_Custom(const VmaVulkanFunctions* pVulkanFunctions);
11590
11591#if VMA_DYNAMIC_VULKAN_FUNCTIONS == 1
11592 void ImportVulkanFunctions_Dynamic();
11593#endif
11594
11595 void ValidateVulkanFunctions();
11596
11597 VkDeviceSize CalcPreferredBlockSize(uint32_t memTypeIndex);
11598
11599 VkResult AllocateMemoryOfType(
11600 VmaPool pool,
11601 VkDeviceSize size,
11602 VkDeviceSize alignment,
11603 bool dedicatedPreferred,
11604 VkBuffer dedicatedBuffer,
11605 VkImage dedicatedImage,
11606 VkFlags dedicatedBufferImageUsage,
11607 const VmaAllocationCreateInfo& createInfo,
11608 uint32_t memTypeIndex,
11609 VmaSuballocationType suballocType,
11610 VmaDedicatedAllocationList& dedicatedAllocations,
11611 VmaBlockVector& blockVector,
11612 size_t allocationCount,
11613 VmaAllocation* pAllocations);
11614
11615 // Helper function only to be used inside AllocateDedicatedMemory.
11616 VkResult AllocateDedicatedMemoryPage(
11617 VmaPool pool,
11618 VkDeviceSize size,
11619 VmaSuballocationType suballocType,
11620 uint32_t memTypeIndex,
11621 const VkMemoryAllocateInfo& allocInfo,
11622 bool map,
11623 bool isUserDataString,
11624 bool isMappingAllowed,
11625 void* pUserData,
11626 VmaAllocation* pAllocation);
11627
11628 // Allocates and registers new VkDeviceMemory specifically for dedicated allocations.
11629 VkResult AllocateDedicatedMemory(
11630 VmaPool pool,
11631 VkDeviceSize size,
11632 VmaSuballocationType suballocType,
11633 VmaDedicatedAllocationList& dedicatedAllocations,
11634 uint32_t memTypeIndex,
11635 bool map,
11636 bool isUserDataString,
11637 bool isMappingAllowed,
11638 bool canAliasMemory,
11639 void* pUserData,
11640 float priority,
11641 VkBuffer dedicatedBuffer,
11642 VkImage dedicatedImage,
11643 VkFlags dedicatedBufferImageUsage,
11644 size_t allocationCount,
11645 VmaAllocation* pAllocations,
11646 const void* pNextChain = nullptr);
11647
11648 void FreeDedicatedMemory(const VmaAllocation allocation);
11649
11650 VkResult CalcMemTypeParams(
11651 VmaAllocationCreateInfo& outCreateInfo,
11652 uint32_t memTypeIndex,
11653 VkDeviceSize size,
11654 size_t allocationCount);
11655 VkResult CalcAllocationParams(
11656 VmaAllocationCreateInfo& outCreateInfo,
11657 bool dedicatedRequired,
11658 bool dedicatedPreferred);
11659
11660 /*
11661 Calculates and returns bit mask of memory types that can support defragmentation
11662 on GPU as they support creation of required buffer for copy operations.
11663 */
11664 uint32_t CalculateGpuDefragmentationMemoryTypeBits() const;
11665 uint32_t CalculateGlobalMemoryTypeBits() const;
11666
11667 bool GetFlushOrInvalidateRange(
11668 VmaAllocation allocation,
11669 VkDeviceSize offset, VkDeviceSize size,
11670 VkMappedMemoryRange& outRange) const;
11671
11672#if VMA_MEMORY_BUDGET
11673 void UpdateVulkanBudget();
11674#endif // #if VMA_MEMORY_BUDGET
11675};
11676
11677
11678#ifndef _VMA_MEMORY_FUNCTIONS
11679static void* VmaMalloc(VmaAllocator hAllocator, size_t size, size_t alignment)
11680{
11681 return VmaMalloc(pAllocationCallbacks: &hAllocator->m_AllocationCallbacks, size, alignment);
11682}
11683
11684static void VmaFree(VmaAllocator hAllocator, void* ptr)
11685{
11686 VmaFree(pAllocationCallbacks: &hAllocator->m_AllocationCallbacks, ptr);
11687}
11688
11689template<typename T>
11690static T* VmaAllocate(VmaAllocator hAllocator)
11691{
11692 return (T*)VmaMalloc(hAllocator, size: sizeof(T), VMA_ALIGN_OF(T));
11693}
11694
11695template<typename T>
11696static T* VmaAllocateArray(VmaAllocator hAllocator, size_t count)
11697{
11698 return (T*)VmaMalloc(hAllocator, size: sizeof(T) * count, VMA_ALIGN_OF(T));
11699}
11700
11701template<typename T>
11702static void vma_delete(VmaAllocator hAllocator, T* ptr)
11703{
11704 if(ptr != VMA_NULL)
11705 {
11706 ptr->~T();
11707 VmaFree(hAllocator, ptr);
11708 }
11709}
11710
11711template<typename T>
11712static void vma_delete_array(VmaAllocator hAllocator, T* ptr, size_t count)
11713{
11714 if(ptr != VMA_NULL)
11715 {
11716 for(size_t i = count; i--; )
11717 ptr[i].~T();
11718 VmaFree(hAllocator, ptr);
11719 }
11720}
11721#endif // _VMA_MEMORY_FUNCTIONS
11722
11723#ifndef _VMA_DEVICE_MEMORY_BLOCK_FUNCTIONS
11724VmaDeviceMemoryBlock::VmaDeviceMemoryBlock(VmaAllocator hAllocator)
11725 : m_pMetadata(VMA_NULL),
11726 m_MemoryTypeIndex(UINT32_MAX),
11727 m_Id(0),
11728 m_hMemory(VK_NULL_HANDLE),
11729 m_MapCount(0),
11730 m_pMappedData(VMA_NULL) {}
11731
11732VmaDeviceMemoryBlock::~VmaDeviceMemoryBlock()
11733{
11734 VMA_ASSERT(m_MapCount == 0 && "VkDeviceMemory block is being destroyed while it is still mapped.");
11735 VMA_ASSERT(m_hMemory == VK_NULL_HANDLE);
11736}
11737
11738void VmaDeviceMemoryBlock::Init(
11739 VmaAllocator hAllocator,
11740 VmaPool hParentPool,
11741 uint32_t newMemoryTypeIndex,
11742 VkDeviceMemory newMemory,
11743 VkDeviceSize newSize,
11744 uint32_t id,
11745 uint32_t algorithm,
11746 VkDeviceSize bufferImageGranularity)
11747{
11748 VMA_ASSERT(m_hMemory == VK_NULL_HANDLE);
11749
11750 m_hParentPool = hParentPool;
11751 m_MemoryTypeIndex = newMemoryTypeIndex;
11752 m_Id = id;
11753 m_hMemory = newMemory;
11754
11755 switch (algorithm)
11756 {
11757 case VMA_POOL_CREATE_LINEAR_ALGORITHM_BIT:
11758 m_pMetadata = vma_new(hAllocator, VmaBlockMetadata_Linear)(hAllocator->GetAllocationCallbacks(),
11759 bufferImageGranularity, false); // isVirtual
11760 break;
11761 default:
11762 VMA_ASSERT(0);
11763 // Fall-through.
11764 case 0:
11765 m_pMetadata = vma_new(hAllocator, VmaBlockMetadata_TLSF)(hAllocator->GetAllocationCallbacks(),
11766 bufferImageGranularity, false); // isVirtual
11767 }
11768 m_pMetadata->Init(size: newSize);
11769}
11770
11771void VmaDeviceMemoryBlock::Destroy(VmaAllocator allocator)
11772{
11773 // Define macro VMA_DEBUG_LOG to receive the list of the unfreed allocations
11774 if (!m_pMetadata->IsEmpty())
11775 m_pMetadata->DebugLogAllAllocations();
11776 // This is the most important assert in the entire library.
11777 // Hitting it means you have some memory leak - unreleased VmaAllocation objects.
11778 VMA_ASSERT(m_pMetadata->IsEmpty() && "Some allocations were not freed before destruction of this memory block!");
11779
11780 VMA_ASSERT(m_hMemory != VK_NULL_HANDLE);
11781 allocator->FreeVulkanMemory(memoryType: m_MemoryTypeIndex, size: m_pMetadata->GetSize(), hMemory: m_hMemory);
11782 m_hMemory = VK_NULL_HANDLE;
11783
11784 vma_delete(hAllocator: allocator, ptr: m_pMetadata);
11785 m_pMetadata = VMA_NULL;
11786}
11787
11788void VmaDeviceMemoryBlock::PostFree(VmaAllocator hAllocator)
11789{
11790 if(m_MappingHysteresis.PostFree())
11791 {
11792 VMA_ASSERT(m_MappingHysteresis.GetExtraMapping() == 0);
11793 if (m_MapCount == 0)
11794 {
11795 m_pMappedData = VMA_NULL;
11796 (*hAllocator->GetVulkanFunctions().vkUnmapMemory)(hAllocator->m_hDevice, m_hMemory);
11797 }
11798 }
11799}
11800
11801bool VmaDeviceMemoryBlock::Validate() const
11802{
11803 VMA_VALIDATE((m_hMemory != VK_NULL_HANDLE) &&
11804 (m_pMetadata->GetSize() != 0));
11805
11806 return m_pMetadata->Validate();
11807}
11808
11809VkResult VmaDeviceMemoryBlock::CheckCorruption(VmaAllocator hAllocator)
11810{
11811 void* pData = nullptr;
11812 VkResult res = Map(hAllocator, count: 1, ppData: &pData);
11813 if (res != VK_SUCCESS)
11814 {
11815 return res;
11816 }
11817
11818 res = m_pMetadata->CheckCorruption(pBlockData: pData);
11819
11820 Unmap(hAllocator, count: 1);
11821
11822 return res;
11823}
11824
11825VkResult VmaDeviceMemoryBlock::Map(VmaAllocator hAllocator, uint32_t count, void** ppData)
11826{
11827 if (count == 0)
11828 {
11829 return VK_SUCCESS;
11830 }
11831
11832 VmaMutexLock lock(m_MapAndBindMutex, hAllocator->m_UseMutex);
11833 const uint32_t oldTotalMapCount = m_MapCount + m_MappingHysteresis.GetExtraMapping();
11834 m_MappingHysteresis.PostMap();
11835 if (oldTotalMapCount != 0)
11836 {
11837 m_MapCount += count;
11838 VMA_ASSERT(m_pMappedData != VMA_NULL);
11839 if (ppData != VMA_NULL)
11840 {
11841 *ppData = m_pMappedData;
11842 }
11843 return VK_SUCCESS;
11844 }
11845 else
11846 {
11847 VkResult result = (*hAllocator->GetVulkanFunctions().vkMapMemory)(
11848 hAllocator->m_hDevice,
11849 m_hMemory,
11850 0, // offset
11851 VK_WHOLE_SIZE,
11852 0, // flags
11853 &m_pMappedData);
11854 if (result == VK_SUCCESS)
11855 {
11856 if (ppData != VMA_NULL)
11857 {
11858 *ppData = m_pMappedData;
11859 }
11860 m_MapCount = count;
11861 }
11862 return result;
11863 }
11864}
11865
11866void VmaDeviceMemoryBlock::Unmap(VmaAllocator hAllocator, uint32_t count)
11867{
11868 if (count == 0)
11869 {
11870 return;
11871 }
11872
11873 VmaMutexLock lock(m_MapAndBindMutex, hAllocator->m_UseMutex);
11874 if (m_MapCount >= count)
11875 {
11876 m_MapCount -= count;
11877 const uint32_t totalMapCount = m_MapCount + m_MappingHysteresis.GetExtraMapping();
11878 if (totalMapCount == 0)
11879 {
11880 m_pMappedData = VMA_NULL;
11881 (*hAllocator->GetVulkanFunctions().vkUnmapMemory)(hAllocator->m_hDevice, m_hMemory);
11882 }
11883 m_MappingHysteresis.PostUnmap();
11884 }
11885 else
11886 {
11887 VMA_ASSERT(0 && "VkDeviceMemory block is being unmapped while it was not previously mapped.");
11888 }
11889}
11890
11891VkResult VmaDeviceMemoryBlock::WriteMagicValueAfterAllocation(VmaAllocator hAllocator, VkDeviceSize allocOffset, VkDeviceSize allocSize)
11892{
11893 VMA_ASSERT(VMA_DEBUG_MARGIN > 0 && VMA_DEBUG_MARGIN % 4 == 0 && VMA_DEBUG_DETECT_CORRUPTION);
11894
11895 void* pData;
11896 VkResult res = Map(hAllocator, count: 1, ppData: &pData);
11897 if (res != VK_SUCCESS)
11898 {
11899 return res;
11900 }
11901
11902 VmaWriteMagicValue(pData, offset: allocOffset + allocSize);
11903
11904 Unmap(hAllocator, count: 1);
11905 return VK_SUCCESS;
11906}
11907
11908VkResult VmaDeviceMemoryBlock::ValidateMagicValueAfterAllocation(VmaAllocator hAllocator, VkDeviceSize allocOffset, VkDeviceSize allocSize)
11909{
11910 VMA_ASSERT(VMA_DEBUG_MARGIN > 0 && VMA_DEBUG_MARGIN % 4 == 0 && VMA_DEBUG_DETECT_CORRUPTION);
11911
11912 void* pData;
11913 VkResult res = Map(hAllocator, count: 1, ppData: &pData);
11914 if (res != VK_SUCCESS)
11915 {
11916 return res;
11917 }
11918
11919 if (!VmaValidateMagicValue(pData, offset: allocOffset + allocSize))
11920 {
11921 VMA_ASSERT(0 && "MEMORY CORRUPTION DETECTED AFTER FREED ALLOCATION!");
11922 }
11923
11924 Unmap(hAllocator, count: 1);
11925 return VK_SUCCESS;
11926}
11927
11928VkResult VmaDeviceMemoryBlock::BindBufferMemory(
11929 const VmaAllocator hAllocator,
11930 const VmaAllocation hAllocation,
11931 VkDeviceSize allocationLocalOffset,
11932 VkBuffer hBuffer,
11933 const void* pNext)
11934{
11935 VMA_ASSERT(hAllocation->GetType() == VmaAllocation_T::ALLOCATION_TYPE_BLOCK &&
11936 hAllocation->GetBlock() == this);
11937 VMA_ASSERT(allocationLocalOffset < hAllocation->GetSize() &&
11938 "Invalid allocationLocalOffset. Did you forget that this offset is relative to the beginning of the allocation, not the whole memory block?");
11939 const VkDeviceSize memoryOffset = hAllocation->GetOffset() + allocationLocalOffset;
11940 // This lock is important so that we don't call vkBind... and/or vkMap... simultaneously on the same VkDeviceMemory from multiple threads.
11941 VmaMutexLock lock(m_MapAndBindMutex, hAllocator->m_UseMutex);
11942 return hAllocator->BindVulkanBuffer(memory: m_hMemory, memoryOffset, buffer: hBuffer, pNext);
11943}
11944
11945VkResult VmaDeviceMemoryBlock::BindImageMemory(
11946 const VmaAllocator hAllocator,
11947 const VmaAllocation hAllocation,
11948 VkDeviceSize allocationLocalOffset,
11949 VkImage hImage,
11950 const void* pNext)
11951{
11952 VMA_ASSERT(hAllocation->GetType() == VmaAllocation_T::ALLOCATION_TYPE_BLOCK &&
11953 hAllocation->GetBlock() == this);
11954 VMA_ASSERT(allocationLocalOffset < hAllocation->GetSize() &&
11955 "Invalid allocationLocalOffset. Did you forget that this offset is relative to the beginning of the allocation, not the whole memory block?");
11956 const VkDeviceSize memoryOffset = hAllocation->GetOffset() + allocationLocalOffset;
11957 // This lock is important so that we don't call vkBind... and/or vkMap... simultaneously on the same VkDeviceMemory from multiple threads.
11958 VmaMutexLock lock(m_MapAndBindMutex, hAllocator->m_UseMutex);
11959 return hAllocator->BindVulkanImage(memory: m_hMemory, memoryOffset, image: hImage, pNext);
11960}
11961#endif // _VMA_DEVICE_MEMORY_BLOCK_FUNCTIONS
11962
11963#ifndef _VMA_ALLOCATION_T_FUNCTIONS
11964VmaAllocation_T::VmaAllocation_T(bool mappingAllowed)
11965 : m_Alignment{ 1 },
11966 m_Size{ 0 },
11967 m_pUserData{ VMA_NULL },
11968 m_pName{ VMA_NULL },
11969 m_MemoryTypeIndex{ 0 },
11970 m_Type{ (uint8_t)ALLOCATION_TYPE_NONE },
11971 m_SuballocationType{ (uint8_t)VMA_SUBALLOCATION_TYPE_UNKNOWN },
11972 m_MapCount{ 0 },
11973 m_Flags{ 0 }
11974{
11975 if(mappingAllowed)
11976 m_Flags |= (uint8_t)FLAG_MAPPING_ALLOWED;
11977
11978#if VMA_STATS_STRING_ENABLED
11979 m_BufferImageUsage = 0;
11980#endif
11981}
11982
11983VmaAllocation_T::~VmaAllocation_T()
11984{
11985 VMA_ASSERT(m_MapCount == 0 && "Allocation was not unmapped before destruction.");
11986
11987 // Check if owned string was freed.
11988 VMA_ASSERT(m_pName == VMA_NULL);
11989}
11990
11991void VmaAllocation_T::InitBlockAllocation(
11992 VmaDeviceMemoryBlock* block,
11993 VmaAllocHandle allocHandle,
11994 VkDeviceSize alignment,
11995 VkDeviceSize size,
11996 uint32_t memoryTypeIndex,
11997 VmaSuballocationType suballocationType,
11998 bool mapped)
11999{
12000 VMA_ASSERT(m_Type == ALLOCATION_TYPE_NONE);
12001 VMA_ASSERT(block != VMA_NULL);
12002 m_Type = (uint8_t)ALLOCATION_TYPE_BLOCK;
12003 m_Alignment = alignment;
12004 m_Size = size;
12005 m_MemoryTypeIndex = memoryTypeIndex;
12006 if(mapped)
12007 {
12008 VMA_ASSERT(IsMappingAllowed() && "Mapping is not allowed on this allocation! Please use one of the new VMA_ALLOCATION_CREATE_HOST_ACCESS_* flags when creating it.");
12009 m_Flags |= (uint8_t)FLAG_PERSISTENT_MAP;
12010 }
12011 m_SuballocationType = (uint8_t)suballocationType;
12012 m_BlockAllocation.m_Block = block;
12013 m_BlockAllocation.m_AllocHandle = allocHandle;
12014}
12015
12016void VmaAllocation_T::InitDedicatedAllocation(
12017 VmaPool hParentPool,
12018 uint32_t memoryTypeIndex,
12019 VkDeviceMemory hMemory,
12020 VmaSuballocationType suballocationType,
12021 void* pMappedData,
12022 VkDeviceSize size)
12023{
12024 VMA_ASSERT(m_Type == ALLOCATION_TYPE_NONE);
12025 VMA_ASSERT(hMemory != VK_NULL_HANDLE);
12026 m_Type = (uint8_t)ALLOCATION_TYPE_DEDICATED;
12027 m_Alignment = 0;
12028 m_Size = size;
12029 m_MemoryTypeIndex = memoryTypeIndex;
12030 m_SuballocationType = (uint8_t)suballocationType;
12031 if(pMappedData != VMA_NULL)
12032 {
12033 VMA_ASSERT(IsMappingAllowed() && "Mapping is not allowed on this allocation! Please use one of the new VMA_ALLOCATION_CREATE_HOST_ACCESS_* flags when creating it.");
12034 m_Flags |= (uint8_t)FLAG_PERSISTENT_MAP;
12035 }
12036 m_DedicatedAllocation.m_hParentPool = hParentPool;
12037 m_DedicatedAllocation.m_hMemory = hMemory;
12038 m_DedicatedAllocation.m_pMappedData = pMappedData;
12039 m_DedicatedAllocation.m_Prev = VMA_NULL;
12040 m_DedicatedAllocation.m_Next = VMA_NULL;
12041}
12042
12043void VmaAllocation_T::SetName(VmaAllocator hAllocator, const char* pName)
12044{
12045 VMA_ASSERT(pName == VMA_NULL || pName != m_pName);
12046
12047 FreeName(hAllocator);
12048
12049 if (pName != VMA_NULL)
12050 m_pName = VmaCreateStringCopy(allocs: hAllocator->GetAllocationCallbacks(), srcStr: pName);
12051}
12052
12053uint8_t VmaAllocation_T::SwapBlockAllocation(VmaAllocator hAllocator, VmaAllocation allocation)
12054{
12055 VMA_ASSERT(allocation != VMA_NULL);
12056 VMA_ASSERT(m_Type == ALLOCATION_TYPE_BLOCK);
12057 VMA_ASSERT(allocation->m_Type == ALLOCATION_TYPE_BLOCK);
12058
12059 if (m_MapCount != 0)
12060 m_BlockAllocation.m_Block->Unmap(hAllocator, count: m_MapCount);
12061
12062 m_BlockAllocation.m_Block->m_pMetadata->SetAllocationUserData(allocHandle: m_BlockAllocation.m_AllocHandle, userData: allocation);
12063 VMA_SWAP(m_BlockAllocation, allocation->m_BlockAllocation);
12064 m_BlockAllocation.m_Block->m_pMetadata->SetAllocationUserData(allocHandle: m_BlockAllocation.m_AllocHandle, userData: this);
12065
12066#if VMA_STATS_STRING_ENABLED
12067 VMA_SWAP(m_BufferImageUsage, allocation->m_BufferImageUsage);
12068#endif
12069 return m_MapCount;
12070}
12071
12072VmaAllocHandle VmaAllocation_T::GetAllocHandle() const
12073{
12074 switch (m_Type)
12075 {
12076 case ALLOCATION_TYPE_BLOCK:
12077 return m_BlockAllocation.m_AllocHandle;
12078 case ALLOCATION_TYPE_DEDICATED:
12079 return VK_NULL_HANDLE;
12080 default:
12081 VMA_ASSERT(0);
12082 return VK_NULL_HANDLE;
12083 }
12084}
12085
12086VkDeviceSize VmaAllocation_T::GetOffset() const
12087{
12088 switch (m_Type)
12089 {
12090 case ALLOCATION_TYPE_BLOCK:
12091 return m_BlockAllocation.m_Block->m_pMetadata->GetAllocationOffset(allocHandle: m_BlockAllocation.m_AllocHandle);
12092 case ALLOCATION_TYPE_DEDICATED:
12093 return 0;
12094 default:
12095 VMA_ASSERT(0);
12096 return 0;
12097 }
12098}
12099
12100VmaPool VmaAllocation_T::GetParentPool() const
12101{
12102 switch (m_Type)
12103 {
12104 case ALLOCATION_TYPE_BLOCK:
12105 return m_BlockAllocation.m_Block->GetParentPool();
12106 case ALLOCATION_TYPE_DEDICATED:
12107 return m_DedicatedAllocation.m_hParentPool;
12108 default:
12109 VMA_ASSERT(0);
12110 return VK_NULL_HANDLE;
12111 }
12112}
12113
12114VkDeviceMemory VmaAllocation_T::GetMemory() const
12115{
12116 switch (m_Type)
12117 {
12118 case ALLOCATION_TYPE_BLOCK:
12119 return m_BlockAllocation.m_Block->GetDeviceMemory();
12120 case ALLOCATION_TYPE_DEDICATED:
12121 return m_DedicatedAllocation.m_hMemory;
12122 default:
12123 VMA_ASSERT(0);
12124 return VK_NULL_HANDLE;
12125 }
12126}
12127
12128void* VmaAllocation_T::GetMappedData() const
12129{
12130 switch (m_Type)
12131 {
12132 case ALLOCATION_TYPE_BLOCK:
12133 if (m_MapCount != 0 || IsPersistentMap())
12134 {
12135 void* pBlockData = m_BlockAllocation.m_Block->GetMappedData();
12136 VMA_ASSERT(pBlockData != VMA_NULL);
12137 return (char*)pBlockData + GetOffset();
12138 }
12139 else
12140 {
12141 return VMA_NULL;
12142 }
12143 break;
12144 case ALLOCATION_TYPE_DEDICATED:
12145 VMA_ASSERT((m_DedicatedAllocation.m_pMappedData != VMA_NULL) == (m_MapCount != 0 || IsPersistentMap()));
12146 return m_DedicatedAllocation.m_pMappedData;
12147 default:
12148 VMA_ASSERT(0);
12149 return VMA_NULL;
12150 }
12151}
12152
12153void VmaAllocation_T::BlockAllocMap()
12154{
12155 VMA_ASSERT(GetType() == ALLOCATION_TYPE_BLOCK);
12156 VMA_ASSERT(IsMappingAllowed() && "Mapping is not allowed on this allocation! Please use one of the new VMA_ALLOCATION_CREATE_HOST_ACCESS_* flags when creating it.");
12157
12158 if (m_MapCount < 0xFF)
12159 {
12160 ++m_MapCount;
12161 }
12162 else
12163 {
12164 VMA_ASSERT(0 && "Allocation mapped too many times simultaneously.");
12165 }
12166}
12167
12168void VmaAllocation_T::BlockAllocUnmap()
12169{
12170 VMA_ASSERT(GetType() == ALLOCATION_TYPE_BLOCK);
12171
12172 if (m_MapCount > 0)
12173 {
12174 --m_MapCount;
12175 }
12176 else
12177 {
12178 VMA_ASSERT(0 && "Unmapping allocation not previously mapped.");
12179 }
12180}
12181
12182VkResult VmaAllocation_T::DedicatedAllocMap(VmaAllocator hAllocator, void** ppData)
12183{
12184 VMA_ASSERT(GetType() == ALLOCATION_TYPE_DEDICATED);
12185 VMA_ASSERT(IsMappingAllowed() && "Mapping is not allowed on this allocation! Please use one of the new VMA_ALLOCATION_CREATE_HOST_ACCESS_* flags when creating it.");
12186
12187 if (m_MapCount != 0 || IsPersistentMap())
12188 {
12189 if (m_MapCount < 0xFF)
12190 {
12191 VMA_ASSERT(m_DedicatedAllocation.m_pMappedData != VMA_NULL);
12192 *ppData = m_DedicatedAllocation.m_pMappedData;
12193 ++m_MapCount;
12194 return VK_SUCCESS;
12195 }
12196 else
12197 {
12198 VMA_ASSERT(0 && "Dedicated allocation mapped too many times simultaneously.");
12199 return VK_ERROR_MEMORY_MAP_FAILED;
12200 }
12201 }
12202 else
12203 {
12204 VkResult result = (*hAllocator->GetVulkanFunctions().vkMapMemory)(
12205 hAllocator->m_hDevice,
12206 m_DedicatedAllocation.m_hMemory,
12207 0, // offset
12208 VK_WHOLE_SIZE,
12209 0, // flags
12210 ppData);
12211 if (result == VK_SUCCESS)
12212 {
12213 m_DedicatedAllocation.m_pMappedData = *ppData;
12214 m_MapCount = 1;
12215 }
12216 return result;
12217 }
12218}
12219
12220void VmaAllocation_T::DedicatedAllocUnmap(VmaAllocator hAllocator)
12221{
12222 VMA_ASSERT(GetType() == ALLOCATION_TYPE_DEDICATED);
12223
12224 if (m_MapCount > 0)
12225 {
12226 --m_MapCount;
12227 if (m_MapCount == 0 && !IsPersistentMap())
12228 {
12229 m_DedicatedAllocation.m_pMappedData = VMA_NULL;
12230 (*hAllocator->GetVulkanFunctions().vkUnmapMemory)(
12231 hAllocator->m_hDevice,
12232 m_DedicatedAllocation.m_hMemory);
12233 }
12234 }
12235 else
12236 {
12237 VMA_ASSERT(0 && "Unmapping dedicated allocation not previously mapped.");
12238 }
12239}
12240
12241#if VMA_STATS_STRING_ENABLED
12242void VmaAllocation_T::InitBufferImageUsage(uint32_t bufferImageUsage)
12243{
12244 VMA_ASSERT(m_BufferImageUsage == 0);
12245 m_BufferImageUsage = bufferImageUsage;
12246}
12247
12248void VmaAllocation_T::PrintParameters(class VmaJsonWriter& json) const
12249{
12250 json.WriteString(pStr: "Type");
12251 json.WriteString(pStr: VMA_SUBALLOCATION_TYPE_NAMES[m_SuballocationType]);
12252
12253 json.WriteString(pStr: "Size");
12254 json.WriteNumber(n: m_Size);
12255 json.WriteString(pStr: "Usage");
12256 json.WriteNumber(n: m_BufferImageUsage);
12257
12258 if (m_pUserData != VMA_NULL)
12259 {
12260 json.WriteString(pStr: "CustomData");
12261 json.BeginString();
12262 json.ContinueString_Pointer(ptr: m_pUserData);
12263 json.EndString();
12264 }
12265 if (m_pName != VMA_NULL)
12266 {
12267 json.WriteString(pStr: "Name");
12268 json.WriteString(pStr: m_pName);
12269 }
12270}
12271#endif // VMA_STATS_STRING_ENABLED
12272
12273void VmaAllocation_T::FreeName(VmaAllocator hAllocator)
12274{
12275 if(m_pName)
12276 {
12277 VmaFreeString(allocs: hAllocator->GetAllocationCallbacks(), str: m_pName);
12278 m_pName = VMA_NULL;
12279 }
12280}
12281#endif // _VMA_ALLOCATION_T_FUNCTIONS
12282
12283#ifndef _VMA_BLOCK_VECTOR_FUNCTIONS
12284VmaBlockVector::VmaBlockVector(
12285 VmaAllocator hAllocator,
12286 VmaPool hParentPool,
12287 uint32_t memoryTypeIndex,
12288 VkDeviceSize preferredBlockSize,
12289 size_t minBlockCount,
12290 size_t maxBlockCount,
12291 VkDeviceSize bufferImageGranularity,
12292 bool explicitBlockSize,
12293 uint32_t algorithm,
12294 float priority,
12295 VkDeviceSize minAllocationAlignment,
12296 void* pMemoryAllocateNext)
12297 : m_hAllocator(hAllocator),
12298 m_hParentPool(hParentPool),
12299 m_MemoryTypeIndex(memoryTypeIndex),
12300 m_PreferredBlockSize(preferredBlockSize),
12301 m_MinBlockCount(minBlockCount),
12302 m_MaxBlockCount(maxBlockCount),
12303 m_BufferImageGranularity(bufferImageGranularity),
12304 m_ExplicitBlockSize(explicitBlockSize),
12305 m_Algorithm(algorithm),
12306 m_Priority(priority),
12307 m_MinAllocationAlignment(minAllocationAlignment),
12308 m_pMemoryAllocateNext(pMemoryAllocateNext),
12309 m_Blocks(VmaStlAllocator<VmaDeviceMemoryBlock*>(hAllocator->GetAllocationCallbacks())),
12310 m_NextBlockId(0) {}
12311
12312VmaBlockVector::~VmaBlockVector()
12313{
12314 for (size_t i = m_Blocks.size(); i--; )
12315 {
12316 m_Blocks[i]->Destroy(allocator: m_hAllocator);
12317 vma_delete(hAllocator: m_hAllocator, ptr: m_Blocks[i]);
12318 }
12319}
12320
12321VkResult VmaBlockVector::CreateMinBlocks()
12322{
12323 for (size_t i = 0; i < m_MinBlockCount; ++i)
12324 {
12325 VkResult res = CreateBlock(blockSize: m_PreferredBlockSize, VMA_NULL);
12326 if (res != VK_SUCCESS)
12327 {
12328 return res;
12329 }
12330 }
12331 return VK_SUCCESS;
12332}
12333
12334void VmaBlockVector::AddStatistics(VmaStatistics& inoutStats)
12335{
12336 VmaMutexLockRead lock(m_Mutex, m_hAllocator->m_UseMutex);
12337
12338 const size_t blockCount = m_Blocks.size();
12339 for (uint32_t blockIndex = 0; blockIndex < blockCount; ++blockIndex)
12340 {
12341 const VmaDeviceMemoryBlock* const pBlock = m_Blocks[blockIndex];
12342 VMA_ASSERT(pBlock);
12343 VMA_HEAVY_ASSERT(pBlock->Validate());
12344 pBlock->m_pMetadata->AddStatistics(inoutStats);
12345 }
12346}
12347
12348void VmaBlockVector::AddDetailedStatistics(VmaDetailedStatistics& inoutStats)
12349{
12350 VmaMutexLockRead lock(m_Mutex, m_hAllocator->m_UseMutex);
12351
12352 const size_t blockCount = m_Blocks.size();
12353 for (uint32_t blockIndex = 0; blockIndex < blockCount; ++blockIndex)
12354 {
12355 const VmaDeviceMemoryBlock* const pBlock = m_Blocks[blockIndex];
12356 VMA_ASSERT(pBlock);
12357 VMA_HEAVY_ASSERT(pBlock->Validate());
12358 pBlock->m_pMetadata->AddDetailedStatistics(inoutStats);
12359 }
12360}
12361
12362bool VmaBlockVector::IsEmpty()
12363{
12364 VmaMutexLockRead lock(m_Mutex, m_hAllocator->m_UseMutex);
12365 return m_Blocks.empty();
12366}
12367
12368bool VmaBlockVector::IsCorruptionDetectionEnabled() const
12369{
12370 const uint32_t requiredMemFlags = VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | VK_MEMORY_PROPERTY_HOST_COHERENT_BIT;
12371 return (VMA_DEBUG_DETECT_CORRUPTION != 0) &&
12372 (VMA_DEBUG_MARGIN > 0) &&
12373 (m_Algorithm == 0 || m_Algorithm == VMA_POOL_CREATE_LINEAR_ALGORITHM_BIT) &&
12374 (m_hAllocator->m_MemProps.memoryTypes[m_MemoryTypeIndex].propertyFlags & requiredMemFlags) == requiredMemFlags;
12375}
12376
12377VkResult VmaBlockVector::Allocate(
12378 VkDeviceSize size,
12379 VkDeviceSize alignment,
12380 const VmaAllocationCreateInfo& createInfo,
12381 VmaSuballocationType suballocType,
12382 size_t allocationCount,
12383 VmaAllocation* pAllocations)
12384{
12385 size_t allocIndex;
12386 VkResult res = VK_SUCCESS;
12387
12388 alignment = VMA_MAX(alignment, m_MinAllocationAlignment);
12389
12390 if (IsCorruptionDetectionEnabled())
12391 {
12392 size = VmaAlignUp<VkDeviceSize>(val: size, alignment: sizeof(VMA_CORRUPTION_DETECTION_MAGIC_VALUE));
12393 alignment = VmaAlignUp<VkDeviceSize>(val: alignment, alignment: sizeof(VMA_CORRUPTION_DETECTION_MAGIC_VALUE));
12394 }
12395
12396 {
12397 VmaMutexLockWrite lock(m_Mutex, m_hAllocator->m_UseMutex);
12398 for (allocIndex = 0; allocIndex < allocationCount; ++allocIndex)
12399 {
12400 res = AllocatePage(
12401 size,
12402 alignment,
12403 createInfo,
12404 suballocType,
12405 pAllocation: pAllocations + allocIndex);
12406 if (res != VK_SUCCESS)
12407 {
12408 break;
12409 }
12410 }
12411 }
12412
12413 if (res != VK_SUCCESS)
12414 {
12415 // Free all already created allocations.
12416 while (allocIndex--)
12417 Free(hAllocation: pAllocations[allocIndex]);
12418 memset(s: pAllocations, c: 0, n: sizeof(VmaAllocation) * allocationCount);
12419 }
12420
12421 return res;
12422}
12423
12424VkResult VmaBlockVector::AllocatePage(
12425 VkDeviceSize size,
12426 VkDeviceSize alignment,
12427 const VmaAllocationCreateInfo& createInfo,
12428 VmaSuballocationType suballocType,
12429 VmaAllocation* pAllocation)
12430{
12431 const bool isUpperAddress = (createInfo.flags & VMA_ALLOCATION_CREATE_UPPER_ADDRESS_BIT) != 0;
12432
12433 VkDeviceSize freeMemory;
12434 {
12435 const uint32_t heapIndex = m_hAllocator->MemoryTypeIndexToHeapIndex(memTypeIndex: m_MemoryTypeIndex);
12436 VmaBudget heapBudget = {};
12437 m_hAllocator->GetHeapBudgets(outBudgets: &heapBudget, firstHeap: heapIndex, heapCount: 1);
12438 freeMemory = (heapBudget.usage < heapBudget.budget) ? (heapBudget.budget - heapBudget.usage) : 0;
12439 }
12440
12441 const bool canFallbackToDedicated = !HasExplicitBlockSize() &&
12442 (createInfo.flags & VMA_ALLOCATION_CREATE_NEVER_ALLOCATE_BIT) == 0;
12443 const bool canCreateNewBlock =
12444 ((createInfo.flags & VMA_ALLOCATION_CREATE_NEVER_ALLOCATE_BIT) == 0) &&
12445 (m_Blocks.size() < m_MaxBlockCount) &&
12446 (freeMemory >= size || !canFallbackToDedicated);
12447 uint32_t strategy = createInfo.flags & VMA_ALLOCATION_CREATE_STRATEGY_MASK;
12448
12449 // Upper address can only be used with linear allocator and within single memory block.
12450 if (isUpperAddress &&
12451 (m_Algorithm != VMA_POOL_CREATE_LINEAR_ALGORITHM_BIT || m_MaxBlockCount > 1))
12452 {
12453 return VK_ERROR_FEATURE_NOT_PRESENT;
12454 }
12455
12456 // Early reject: requested allocation size is larger that maximum block size for this block vector.
12457 if (size + VMA_DEBUG_MARGIN > m_PreferredBlockSize)
12458 {
12459 return VK_ERROR_OUT_OF_DEVICE_MEMORY;
12460 }
12461
12462 // 1. Search existing allocations. Try to allocate.
12463 if (m_Algorithm == VMA_POOL_CREATE_LINEAR_ALGORITHM_BIT)
12464 {
12465 // Use only last block.
12466 if (!m_Blocks.empty())
12467 {
12468 VmaDeviceMemoryBlock* const pCurrBlock = m_Blocks.back();
12469 VMA_ASSERT(pCurrBlock);
12470 VkResult res = AllocateFromBlock(
12471 pBlock: pCurrBlock, size, alignment, allocFlags: createInfo.flags, pUserData: createInfo.pUserData, suballocType, strategy, pAllocation);
12472 if (res == VK_SUCCESS)
12473 {
12474 VMA_DEBUG_LOG(" Returned from last block #%u", pCurrBlock->GetId());
12475 IncrementallySortBlocks();
12476 return VK_SUCCESS;
12477 }
12478 }
12479 }
12480 else
12481 {
12482 if (strategy != VMA_ALLOCATION_CREATE_STRATEGY_MIN_TIME_BIT) // MIN_MEMORY or default
12483 {
12484 const bool isHostVisible =
12485 (m_hAllocator->m_MemProps.memoryTypes[m_MemoryTypeIndex].propertyFlags & VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT) != 0;
12486 if(isHostVisible)
12487 {
12488 const bool isMappingAllowed = (createInfo.flags &
12489 (VMA_ALLOCATION_CREATE_HOST_ACCESS_SEQUENTIAL_WRITE_BIT | VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT)) != 0;
12490 /*
12491 For non-mappable allocations, check blocks that are not mapped first.
12492 For mappable allocations, check blocks that are already mapped first.
12493 This way, having many blocks, we will separate mappable and non-mappable allocations,
12494 hopefully limiting the number of blocks that are mapped, which will help tools like RenderDoc.
12495 */
12496 for(size_t mappingI = 0; mappingI < 2; ++mappingI)
12497 {
12498 // Forward order in m_Blocks - prefer blocks with smallest amount of free space.
12499 for (size_t blockIndex = 0; blockIndex < m_Blocks.size(); ++blockIndex)
12500 {
12501 VmaDeviceMemoryBlock* const pCurrBlock = m_Blocks[blockIndex];
12502 VMA_ASSERT(pCurrBlock);
12503 const bool isBlockMapped = pCurrBlock->GetMappedData() != VMA_NULL;
12504 if((mappingI == 0) == (isMappingAllowed == isBlockMapped))
12505 {
12506 VkResult res = AllocateFromBlock(
12507 pBlock: pCurrBlock, size, alignment, allocFlags: createInfo.flags, pUserData: createInfo.pUserData, suballocType, strategy, pAllocation);
12508 if (res == VK_SUCCESS)
12509 {
12510 VMA_DEBUG_LOG(" Returned from existing block #%u", pCurrBlock->GetId());
12511 IncrementallySortBlocks();
12512 return VK_SUCCESS;
12513 }
12514 }
12515 }
12516 }
12517 }
12518 else
12519 {
12520 // Forward order in m_Blocks - prefer blocks with smallest amount of free space.
12521 for (size_t blockIndex = 0; blockIndex < m_Blocks.size(); ++blockIndex)
12522 {
12523 VmaDeviceMemoryBlock* const pCurrBlock = m_Blocks[blockIndex];
12524 VMA_ASSERT(pCurrBlock);
12525 VkResult res = AllocateFromBlock(
12526 pBlock: pCurrBlock, size, alignment, allocFlags: createInfo.flags, pUserData: createInfo.pUserData, suballocType, strategy, pAllocation);
12527 if (res == VK_SUCCESS)
12528 {
12529 VMA_DEBUG_LOG(" Returned from existing block #%u", pCurrBlock->GetId());
12530 IncrementallySortBlocks();
12531 return VK_SUCCESS;
12532 }
12533 }
12534 }
12535 }
12536 else // VMA_ALLOCATION_CREATE_STRATEGY_MIN_TIME_BIT
12537 {
12538 // Backward order in m_Blocks - prefer blocks with largest amount of free space.
12539 for (size_t blockIndex = m_Blocks.size(); blockIndex--; )
12540 {
12541 VmaDeviceMemoryBlock* const pCurrBlock = m_Blocks[blockIndex];
12542 VMA_ASSERT(pCurrBlock);
12543 VkResult res = AllocateFromBlock(pBlock: pCurrBlock, size, alignment, allocFlags: createInfo.flags, pUserData: createInfo.pUserData, suballocType, strategy, pAllocation);
12544 if (res == VK_SUCCESS)
12545 {
12546 VMA_DEBUG_LOG(" Returned from existing block #%u", pCurrBlock->GetId());
12547 IncrementallySortBlocks();
12548 return VK_SUCCESS;
12549 }
12550 }
12551 }
12552 }
12553
12554 // 2. Try to create new block.
12555 if (canCreateNewBlock)
12556 {
12557 // Calculate optimal size for new block.
12558 VkDeviceSize newBlockSize = m_PreferredBlockSize;
12559 uint32_t newBlockSizeShift = 0;
12560 const uint32_t NEW_BLOCK_SIZE_SHIFT_MAX = 3;
12561
12562 if (!m_ExplicitBlockSize)
12563 {
12564 // Allocate 1/8, 1/4, 1/2 as first blocks.
12565 const VkDeviceSize maxExistingBlockSize = CalcMaxBlockSize();
12566 for (uint32_t i = 0; i < NEW_BLOCK_SIZE_SHIFT_MAX; ++i)
12567 {
12568 const VkDeviceSize smallerNewBlockSize = newBlockSize / 2;
12569 if (smallerNewBlockSize > maxExistingBlockSize && smallerNewBlockSize >= size * 2)
12570 {
12571 newBlockSize = smallerNewBlockSize;
12572 ++newBlockSizeShift;
12573 }
12574 else
12575 {
12576 break;
12577 }
12578 }
12579 }
12580
12581 size_t newBlockIndex = 0;
12582 VkResult res = (newBlockSize <= freeMemory || !canFallbackToDedicated) ?
12583 CreateBlock(blockSize: newBlockSize, pNewBlockIndex: &newBlockIndex) : VK_ERROR_OUT_OF_DEVICE_MEMORY;
12584 // Allocation of this size failed? Try 1/2, 1/4, 1/8 of m_PreferredBlockSize.
12585 if (!m_ExplicitBlockSize)
12586 {
12587 while (res < 0 && newBlockSizeShift < NEW_BLOCK_SIZE_SHIFT_MAX)
12588 {
12589 const VkDeviceSize smallerNewBlockSize = newBlockSize / 2;
12590 if (smallerNewBlockSize >= size)
12591 {
12592 newBlockSize = smallerNewBlockSize;
12593 ++newBlockSizeShift;
12594 res = (newBlockSize <= freeMemory || !canFallbackToDedicated) ?
12595 CreateBlock(blockSize: newBlockSize, pNewBlockIndex: &newBlockIndex) : VK_ERROR_OUT_OF_DEVICE_MEMORY;
12596 }
12597 else
12598 {
12599 break;
12600 }
12601 }
12602 }
12603
12604 if (res == VK_SUCCESS)
12605 {
12606 VmaDeviceMemoryBlock* const pBlock = m_Blocks[newBlockIndex];
12607 VMA_ASSERT(pBlock->m_pMetadata->GetSize() >= size);
12608
12609 res = AllocateFromBlock(
12610 pBlock, size, alignment, allocFlags: createInfo.flags, pUserData: createInfo.pUserData, suballocType, strategy, pAllocation);
12611 if (res == VK_SUCCESS)
12612 {
12613 VMA_DEBUG_LOG(" Created new block #%u Size=%llu", pBlock->GetId(), newBlockSize);
12614 IncrementallySortBlocks();
12615 return VK_SUCCESS;
12616 }
12617 else
12618 {
12619 // Allocation from new block failed, possibly due to VMA_DEBUG_MARGIN or alignment.
12620 return VK_ERROR_OUT_OF_DEVICE_MEMORY;
12621 }
12622 }
12623 }
12624
12625 return VK_ERROR_OUT_OF_DEVICE_MEMORY;
12626}
12627
12628void VmaBlockVector::Free(const VmaAllocation hAllocation)
12629{
12630 VmaDeviceMemoryBlock* pBlockToDelete = VMA_NULL;
12631
12632 bool budgetExceeded = false;
12633 {
12634 const uint32_t heapIndex = m_hAllocator->MemoryTypeIndexToHeapIndex(memTypeIndex: m_MemoryTypeIndex);
12635 VmaBudget heapBudget = {};
12636 m_hAllocator->GetHeapBudgets(outBudgets: &heapBudget, firstHeap: heapIndex, heapCount: 1);
12637 budgetExceeded = heapBudget.usage >= heapBudget.budget;
12638 }
12639
12640 // Scope for lock.
12641 {
12642 VmaMutexLockWrite lock(m_Mutex, m_hAllocator->m_UseMutex);
12643
12644 VmaDeviceMemoryBlock* pBlock = hAllocation->GetBlock();
12645
12646 if (IsCorruptionDetectionEnabled())
12647 {
12648 VkResult res = pBlock->ValidateMagicValueAfterAllocation(hAllocator: m_hAllocator, allocOffset: hAllocation->GetOffset(), allocSize: hAllocation->GetSize());
12649 VMA_ASSERT(res == VK_SUCCESS && "Couldn't map block memory to validate magic value.");
12650 }
12651
12652 if (hAllocation->IsPersistentMap())
12653 {
12654 pBlock->Unmap(hAllocator: m_hAllocator, count: 1);
12655 }
12656
12657 const bool hadEmptyBlockBeforeFree = HasEmptyBlock();
12658 pBlock->m_pMetadata->Free(allocHandle: hAllocation->GetAllocHandle());
12659 pBlock->PostFree(hAllocator: m_hAllocator);
12660 VMA_HEAVY_ASSERT(pBlock->Validate());
12661
12662 VMA_DEBUG_LOG(" Freed from MemoryTypeIndex=%u", m_MemoryTypeIndex);
12663
12664 const bool canDeleteBlock = m_Blocks.size() > m_MinBlockCount;
12665 // pBlock became empty after this deallocation.
12666 if (pBlock->m_pMetadata->IsEmpty())
12667 {
12668 // Already had empty block. We don't want to have two, so delete this one.
12669 if ((hadEmptyBlockBeforeFree || budgetExceeded) && canDeleteBlock)
12670 {
12671 pBlockToDelete = pBlock;
12672 Remove(pBlock);
12673 }
12674 // else: We now have one empty block - leave it. A hysteresis to avoid allocating whole block back and forth.
12675 }
12676 // pBlock didn't become empty, but we have another empty block - find and free that one.
12677 // (This is optional, heuristics.)
12678 else if (hadEmptyBlockBeforeFree && canDeleteBlock)
12679 {
12680 VmaDeviceMemoryBlock* pLastBlock = m_Blocks.back();
12681 if (pLastBlock->m_pMetadata->IsEmpty())
12682 {
12683 pBlockToDelete = pLastBlock;
12684 m_Blocks.pop_back();
12685 }
12686 }
12687
12688 IncrementallySortBlocks();
12689 }
12690
12691 // Destruction of a free block. Deferred until this point, outside of mutex
12692 // lock, for performance reason.
12693 if (pBlockToDelete != VMA_NULL)
12694 {
12695 VMA_DEBUG_LOG(" Deleted empty block #%u", pBlockToDelete->GetId());
12696 pBlockToDelete->Destroy(allocator: m_hAllocator);
12697 vma_delete(hAllocator: m_hAllocator, ptr: pBlockToDelete);
12698 }
12699
12700 m_hAllocator->m_Budget.RemoveAllocation(heapIndex: m_hAllocator->MemoryTypeIndexToHeapIndex(memTypeIndex: m_MemoryTypeIndex), allocationSize: hAllocation->GetSize());
12701 m_hAllocator->m_AllocationObjectAllocator.Free(hAlloc: hAllocation);
12702}
12703
12704VkDeviceSize VmaBlockVector::CalcMaxBlockSize() const
12705{
12706 VkDeviceSize result = 0;
12707 for (size_t i = m_Blocks.size(); i--; )
12708 {
12709 result = VMA_MAX(result, m_Blocks[i]->m_pMetadata->GetSize());
12710 if (result >= m_PreferredBlockSize)
12711 {
12712 break;
12713 }
12714 }
12715 return result;
12716}
12717
12718void VmaBlockVector::Remove(VmaDeviceMemoryBlock* pBlock)
12719{
12720 for (uint32_t blockIndex = 0; blockIndex < m_Blocks.size(); ++blockIndex)
12721 {
12722 if (m_Blocks[blockIndex] == pBlock)
12723 {
12724 VmaVectorRemove(vec&: m_Blocks, index: blockIndex);
12725 return;
12726 }
12727 }
12728 VMA_ASSERT(0);
12729}
12730
12731void VmaBlockVector::IncrementallySortBlocks()
12732{
12733 if (!m_IncrementalSort)
12734 return;
12735 if (m_Algorithm != VMA_POOL_CREATE_LINEAR_ALGORITHM_BIT)
12736 {
12737 // Bubble sort only until first swap.
12738 for (size_t i = 1; i < m_Blocks.size(); ++i)
12739 {
12740 if (m_Blocks[i - 1]->m_pMetadata->GetSumFreeSize() > m_Blocks[i]->m_pMetadata->GetSumFreeSize())
12741 {
12742 VMA_SWAP(m_Blocks[i - 1], m_Blocks[i]);
12743 return;
12744 }
12745 }
12746 }
12747}
12748
12749void VmaBlockVector::SortByFreeSize()
12750{
12751 VMA_SORT(m_Blocks.begin(), m_Blocks.end(),
12752 [](VmaDeviceMemoryBlock* b1, VmaDeviceMemoryBlock* b2) -> bool
12753 {
12754 return b1->m_pMetadata->GetSumFreeSize() < b2->m_pMetadata->GetSumFreeSize();
12755 });
12756}
12757
12758VkResult VmaBlockVector::AllocateFromBlock(
12759 VmaDeviceMemoryBlock* pBlock,
12760 VkDeviceSize size,
12761 VkDeviceSize alignment,
12762 VmaAllocationCreateFlags allocFlags,
12763 void* pUserData,
12764 VmaSuballocationType suballocType,
12765 uint32_t strategy,
12766 VmaAllocation* pAllocation)
12767{
12768 const bool isUpperAddress = (allocFlags & VMA_ALLOCATION_CREATE_UPPER_ADDRESS_BIT) != 0;
12769
12770 VmaAllocationRequest currRequest = {};
12771 if (pBlock->m_pMetadata->CreateAllocationRequest(
12772 allocSize: size,
12773 allocAlignment: alignment,
12774 upperAddress: isUpperAddress,
12775 allocType: suballocType,
12776 strategy,
12777 pAllocationRequest: &currRequest))
12778 {
12779 return CommitAllocationRequest(allocRequest&: currRequest, pBlock, alignment, allocFlags, pUserData, suballocType, pAllocation);
12780 }
12781 return VK_ERROR_OUT_OF_DEVICE_MEMORY;
12782}
12783
12784VkResult VmaBlockVector::CommitAllocationRequest(
12785 VmaAllocationRequest& allocRequest,
12786 VmaDeviceMemoryBlock* pBlock,
12787 VkDeviceSize alignment,
12788 VmaAllocationCreateFlags allocFlags,
12789 void* pUserData,
12790 VmaSuballocationType suballocType,
12791 VmaAllocation* pAllocation)
12792{
12793 const bool mapped = (allocFlags & VMA_ALLOCATION_CREATE_MAPPED_BIT) != 0;
12794 const bool isUserDataString = (allocFlags & VMA_ALLOCATION_CREATE_USER_DATA_COPY_STRING_BIT) != 0;
12795 const bool isMappingAllowed = (allocFlags &
12796 (VMA_ALLOCATION_CREATE_HOST_ACCESS_SEQUENTIAL_WRITE_BIT | VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT)) != 0;
12797
12798 pBlock->PostAlloc();
12799 // Allocate from pCurrBlock.
12800 if (mapped)
12801 {
12802 VkResult res = pBlock->Map(hAllocator: m_hAllocator, count: 1, VMA_NULL);
12803 if (res != VK_SUCCESS)
12804 {
12805 return res;
12806 }
12807 }
12808
12809 *pAllocation = m_hAllocator->m_AllocationObjectAllocator.Allocate(args: isMappingAllowed);
12810 pBlock->m_pMetadata->Alloc(request: allocRequest, type: suballocType, userData: *pAllocation);
12811 (*pAllocation)->InitBlockAllocation(
12812 block: pBlock,
12813 allocHandle: allocRequest.allocHandle,
12814 alignment,
12815 size: allocRequest.size, // Not size, as actual allocation size may be larger than requested!
12816 memoryTypeIndex: m_MemoryTypeIndex,
12817 suballocationType: suballocType,
12818 mapped);
12819 VMA_HEAVY_ASSERT(pBlock->Validate());
12820 if (isUserDataString)
12821 (*pAllocation)->SetName(hAllocator: m_hAllocator, pName: (const char*)pUserData);
12822 else
12823 (*pAllocation)->SetUserData(hAllocator: m_hAllocator, pUserData);
12824 m_hAllocator->m_Budget.AddAllocation(heapIndex: m_hAllocator->MemoryTypeIndexToHeapIndex(memTypeIndex: m_MemoryTypeIndex), allocationSize: allocRequest.size);
12825 if (VMA_DEBUG_INITIALIZE_ALLOCATIONS)
12826 {
12827 m_hAllocator->FillAllocation(hAllocation: *pAllocation, pattern: VMA_ALLOCATION_FILL_PATTERN_CREATED);
12828 }
12829 if (IsCorruptionDetectionEnabled())
12830 {
12831 VkResult res = pBlock->WriteMagicValueAfterAllocation(hAllocator: m_hAllocator, allocOffset: (*pAllocation)->GetOffset(), allocSize: allocRequest.size);
12832 VMA_ASSERT(res == VK_SUCCESS && "Couldn't map block memory to write magic value.");
12833 }
12834 return VK_SUCCESS;
12835}
12836
12837VkResult VmaBlockVector::CreateBlock(VkDeviceSize blockSize, size_t* pNewBlockIndex)
12838{
12839 VkMemoryAllocateInfo allocInfo = { .sType: VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO };
12840 allocInfo.pNext = m_pMemoryAllocateNext;
12841 allocInfo.memoryTypeIndex = m_MemoryTypeIndex;
12842 allocInfo.allocationSize = blockSize;
12843
12844#if VMA_BUFFER_DEVICE_ADDRESS
12845 // Every standalone block can potentially contain a buffer with VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT - always enable the feature.
12846 VkMemoryAllocateFlagsInfoKHR allocFlagsInfo = { .sType: VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_FLAGS_INFO_KHR };
12847 if (m_hAllocator->m_UseKhrBufferDeviceAddress)
12848 {
12849 allocFlagsInfo.flags = VK_MEMORY_ALLOCATE_DEVICE_ADDRESS_BIT_KHR;
12850 VmaPnextChainPushFront(mainStruct: &allocInfo, newStruct: &allocFlagsInfo);
12851 }
12852#endif // VMA_BUFFER_DEVICE_ADDRESS
12853
12854#if VMA_MEMORY_PRIORITY
12855 VkMemoryPriorityAllocateInfoEXT priorityInfo = { .sType: VK_STRUCTURE_TYPE_MEMORY_PRIORITY_ALLOCATE_INFO_EXT };
12856 if (m_hAllocator->m_UseExtMemoryPriority)
12857 {
12858 VMA_ASSERT(m_Priority >= 0.f && m_Priority <= 1.f);
12859 priorityInfo.priority = m_Priority;
12860 VmaPnextChainPushFront(mainStruct: &allocInfo, newStruct: &priorityInfo);
12861 }
12862#endif // VMA_MEMORY_PRIORITY
12863
12864#if VMA_EXTERNAL_MEMORY
12865 // Attach VkExportMemoryAllocateInfoKHR if necessary.
12866 VkExportMemoryAllocateInfoKHR exportMemoryAllocInfo = { .sType: VK_STRUCTURE_TYPE_EXPORT_MEMORY_ALLOCATE_INFO_KHR };
12867 exportMemoryAllocInfo.handleTypes = m_hAllocator->GetExternalMemoryHandleTypeFlags(memTypeIndex: m_MemoryTypeIndex);
12868 if (exportMemoryAllocInfo.handleTypes != 0)
12869 {
12870 VmaPnextChainPushFront(mainStruct: &allocInfo, newStruct: &exportMemoryAllocInfo);
12871 }
12872#endif // VMA_EXTERNAL_MEMORY
12873
12874 VkDeviceMemory mem = VK_NULL_HANDLE;
12875 VkResult res = m_hAllocator->AllocateVulkanMemory(pAllocateInfo: &allocInfo, pMemory: &mem);
12876 if (res < 0)
12877 {
12878 return res;
12879 }
12880
12881 // New VkDeviceMemory successfully created.
12882
12883 // Create new Allocation for it.
12884 VmaDeviceMemoryBlock* const pBlock = vma_new(m_hAllocator, VmaDeviceMemoryBlock)(m_hAllocator);
12885 pBlock->Init(
12886 hAllocator: m_hAllocator,
12887 hParentPool: m_hParentPool,
12888 newMemoryTypeIndex: m_MemoryTypeIndex,
12889 newMemory: mem,
12890 newSize: allocInfo.allocationSize,
12891 id: m_NextBlockId++,
12892 algorithm: m_Algorithm,
12893 bufferImageGranularity: m_BufferImageGranularity);
12894
12895 m_Blocks.push_back(src: pBlock);
12896 if (pNewBlockIndex != VMA_NULL)
12897 {
12898 *pNewBlockIndex = m_Blocks.size() - 1;
12899 }
12900
12901 return VK_SUCCESS;
12902}
12903
12904bool VmaBlockVector::HasEmptyBlock()
12905{
12906 for (size_t index = 0, count = m_Blocks.size(); index < count; ++index)
12907 {
12908 VmaDeviceMemoryBlock* const pBlock = m_Blocks[index];
12909 if (pBlock->m_pMetadata->IsEmpty())
12910 {
12911 return true;
12912 }
12913 }
12914 return false;
12915}
12916
12917#if VMA_STATS_STRING_ENABLED
12918void VmaBlockVector::PrintDetailedMap(class VmaJsonWriter& json)
12919{
12920 VmaMutexLockRead lock(m_Mutex, m_hAllocator->m_UseMutex);
12921
12922
12923 json.BeginObject();
12924 for (size_t i = 0; i < m_Blocks.size(); ++i)
12925 {
12926 json.BeginString();
12927 json.ContinueString(n: m_Blocks[i]->GetId());
12928 json.EndString();
12929
12930 json.BeginObject();
12931 json.WriteString(pStr: "MapRefCount");
12932 json.WriteNumber(n: m_Blocks[i]->GetMapRefCount());
12933
12934 m_Blocks[i]->m_pMetadata->PrintDetailedMap(json);
12935 json.EndObject();
12936 }
12937 json.EndObject();
12938}
12939#endif // VMA_STATS_STRING_ENABLED
12940
12941VkResult VmaBlockVector::CheckCorruption()
12942{
12943 if (!IsCorruptionDetectionEnabled())
12944 {
12945 return VK_ERROR_FEATURE_NOT_PRESENT;
12946 }
12947
12948 VmaMutexLockRead lock(m_Mutex, m_hAllocator->m_UseMutex);
12949 for (uint32_t blockIndex = 0; blockIndex < m_Blocks.size(); ++blockIndex)
12950 {
12951 VmaDeviceMemoryBlock* const pBlock = m_Blocks[blockIndex];
12952 VMA_ASSERT(pBlock);
12953 VkResult res = pBlock->CheckCorruption(hAllocator: m_hAllocator);
12954 if (res != VK_SUCCESS)
12955 {
12956 return res;
12957 }
12958 }
12959 return VK_SUCCESS;
12960}
12961
12962#endif // _VMA_BLOCK_VECTOR_FUNCTIONS
12963
12964#ifndef _VMA_DEFRAGMENTATION_CONTEXT_FUNCTIONS
12965VmaDefragmentationContext_T::VmaDefragmentationContext_T(
12966 VmaAllocator hAllocator,
12967 const VmaDefragmentationInfo& info)
12968 : m_MaxPassBytes(info.maxBytesPerPass == 0 ? VK_WHOLE_SIZE : info.maxBytesPerPass),
12969 m_MaxPassAllocations(info.maxAllocationsPerPass == 0 ? UINT32_MAX : info.maxAllocationsPerPass),
12970 m_MoveAllocator(hAllocator->GetAllocationCallbacks()),
12971 m_Moves(m_MoveAllocator)
12972{
12973 m_Algorithm = info.flags & VMA_DEFRAGMENTATION_FLAG_ALGORITHM_MASK;
12974
12975 if (info.pool != VMA_NULL)
12976 {
12977 m_BlockVectorCount = 1;
12978 m_PoolBlockVector = &info.pool->m_BlockVector;
12979 m_pBlockVectors = &m_PoolBlockVector;
12980 m_PoolBlockVector->SetIncrementalSort(false);
12981 m_PoolBlockVector->SortByFreeSize();
12982 }
12983 else
12984 {
12985 m_BlockVectorCount = hAllocator->GetMemoryTypeCount();
12986 m_PoolBlockVector = VMA_NULL;
12987 m_pBlockVectors = hAllocator->m_pBlockVectors;
12988 for (uint32_t i = 0; i < m_BlockVectorCount; ++i)
12989 {
12990 VmaBlockVector* vector = m_pBlockVectors[i];
12991 if (vector != VMA_NULL)
12992 {
12993 vector->SetIncrementalSort(false);
12994 vector->SortByFreeSize();
12995 }
12996 }
12997 }
12998
12999 switch (m_Algorithm)
13000 {
13001 case 0: // Default algorithm
13002 m_Algorithm = VMA_DEFRAGMENTATION_FLAG_ALGORITHM_BALANCED_BIT;
13003 case VMA_DEFRAGMENTATION_FLAG_ALGORITHM_BALANCED_BIT:
13004 {
13005 m_AlgorithmState = vma_new_array(hAllocator, StateBalanced, m_BlockVectorCount);
13006 break;
13007 }
13008 case VMA_DEFRAGMENTATION_FLAG_ALGORITHM_EXTENSIVE_BIT:
13009 {
13010 if (hAllocator->GetBufferImageGranularity() > 1)
13011 {
13012 m_AlgorithmState = vma_new_array(hAllocator, StateExtensive, m_BlockVectorCount);
13013 }
13014 break;
13015 }
13016 }
13017}
13018
13019VmaDefragmentationContext_T::~VmaDefragmentationContext_T()
13020{
13021 if (m_PoolBlockVector != VMA_NULL)
13022 {
13023 m_PoolBlockVector->SetIncrementalSort(true);
13024 }
13025 else
13026 {
13027 for (uint32_t i = 0; i < m_BlockVectorCount; ++i)
13028 {
13029 VmaBlockVector* vector = m_pBlockVectors[i];
13030 if (vector != VMA_NULL)
13031 vector->SetIncrementalSort(true);
13032 }
13033 }
13034
13035 if (m_AlgorithmState)
13036 {
13037 switch (m_Algorithm)
13038 {
13039 case VMA_DEFRAGMENTATION_FLAG_ALGORITHM_BALANCED_BIT:
13040 vma_delete_array(pAllocationCallbacks: m_MoveAllocator.m_pCallbacks, ptr: reinterpret_cast<StateBalanced*>(m_AlgorithmState), count: m_BlockVectorCount);
13041 break;
13042 case VMA_DEFRAGMENTATION_FLAG_ALGORITHM_EXTENSIVE_BIT:
13043 vma_delete_array(pAllocationCallbacks: m_MoveAllocator.m_pCallbacks, ptr: reinterpret_cast<StateExtensive*>(m_AlgorithmState), count: m_BlockVectorCount);
13044 break;
13045 default:
13046 VMA_ASSERT(0);
13047 }
13048 }
13049}
13050
13051VkResult VmaDefragmentationContext_T::DefragmentPassBegin(VmaDefragmentationPassMoveInfo& moveInfo)
13052{
13053 if (m_PoolBlockVector != VMA_NULL)
13054 {
13055 VmaMutexLockWrite lock(m_PoolBlockVector->GetMutex(), m_PoolBlockVector->GetAllocator()->m_UseMutex);
13056
13057 if (m_PoolBlockVector->GetBlockCount() > 1)
13058 ComputeDefragmentation(vector&: *m_PoolBlockVector, index: 0);
13059 else if (m_PoolBlockVector->GetBlockCount() == 1)
13060 ReallocWithinBlock(vector&: *m_PoolBlockVector, block: m_PoolBlockVector->GetBlock(index: 0));
13061 }
13062 else
13063 {
13064 for (uint32_t i = 0; i < m_BlockVectorCount; ++i)
13065 {
13066 if (m_pBlockVectors[i] != VMA_NULL)
13067 {
13068 VmaMutexLockWrite lock(m_pBlockVectors[i]->GetMutex(), m_pBlockVectors[i]->GetAllocator()->m_UseMutex);
13069
13070 if (m_pBlockVectors[i]->GetBlockCount() > 1)
13071 {
13072 if (ComputeDefragmentation(vector&: *m_pBlockVectors[i], index: i))
13073 break;
13074 }
13075 else if (m_pBlockVectors[i]->GetBlockCount() == 1)
13076 {
13077 if (ReallocWithinBlock(vector&: *m_pBlockVectors[i], block: m_pBlockVectors[i]->GetBlock(index: 0)))
13078 break;
13079 }
13080 }
13081 }
13082 }
13083
13084 moveInfo.moveCount = static_cast<uint32_t>(m_Moves.size());
13085 if (moveInfo.moveCount > 0)
13086 {
13087 moveInfo.pMoves = m_Moves.data();
13088 return VK_INCOMPLETE;
13089 }
13090
13091 moveInfo.pMoves = VMA_NULL;
13092 return VK_SUCCESS;
13093}
13094
13095VkResult VmaDefragmentationContext_T::DefragmentPassEnd(VmaDefragmentationPassMoveInfo& moveInfo)
13096{
13097 VMA_ASSERT(moveInfo.moveCount > 0 ? moveInfo.pMoves != VMA_NULL : true);
13098
13099 VkResult result = VK_SUCCESS;
13100 VmaStlAllocator<FragmentedBlock> blockAllocator(m_MoveAllocator.m_pCallbacks);
13101 VmaVector<FragmentedBlock, VmaStlAllocator<FragmentedBlock>> immovableBlocks(blockAllocator);
13102 VmaVector<FragmentedBlock, VmaStlAllocator<FragmentedBlock>> mappedBlocks(blockAllocator);
13103
13104 VmaAllocator allocator = VMA_NULL;
13105 for (uint32_t i = 0; i < moveInfo.moveCount; ++i)
13106 {
13107 VmaDefragmentationMove& move = moveInfo.pMoves[i];
13108 size_t prevCount = 0, currentCount = 0;
13109 VkDeviceSize freedBlockSize = 0;
13110
13111 uint32_t vectorIndex;
13112 VmaBlockVector* vector;
13113 if (m_PoolBlockVector != VMA_NULL)
13114 {
13115 vectorIndex = 0;
13116 vector = m_PoolBlockVector;
13117 }
13118 else
13119 {
13120 vectorIndex = move.srcAllocation->GetMemoryTypeIndex();
13121 vector = m_pBlockVectors[vectorIndex];
13122 VMA_ASSERT(vector != VMA_NULL);
13123 }
13124
13125 switch (move.operation)
13126 {
13127 case VMA_DEFRAGMENTATION_MOVE_OPERATION_COPY:
13128 {
13129 uint8_t mapCount = move.srcAllocation->SwapBlockAllocation(hAllocator: vector->m_hAllocator, allocation: move.dstTmpAllocation);
13130 if (mapCount > 0)
13131 {
13132 allocator = vector->m_hAllocator;
13133 VmaDeviceMemoryBlock* newMapBlock = move.srcAllocation->GetBlock();
13134 bool notPresent = true;
13135 for (FragmentedBlock& block : mappedBlocks)
13136 {
13137 if (block.block == newMapBlock)
13138 {
13139 notPresent = false;
13140 block.data += mapCount;
13141 break;
13142 }
13143 }
13144 if (notPresent)
13145 mappedBlocks.push_back(src: { .data: mapCount, .block: newMapBlock });
13146 }
13147
13148 // Scope for locks, Free have it's own lock
13149 {
13150 VmaMutexLockRead lock(vector->GetMutex(), vector->GetAllocator()->m_UseMutex);
13151 prevCount = vector->GetBlockCount();
13152 freedBlockSize = move.dstTmpAllocation->GetBlock()->m_pMetadata->GetSize();
13153 }
13154 vector->Free(hAllocation: move.dstTmpAllocation);
13155 {
13156 VmaMutexLockRead lock(vector->GetMutex(), vector->GetAllocator()->m_UseMutex);
13157 currentCount = vector->GetBlockCount();
13158 }
13159
13160 result = VK_INCOMPLETE;
13161 break;
13162 }
13163 case VMA_DEFRAGMENTATION_MOVE_OPERATION_IGNORE:
13164 {
13165 m_PassStats.bytesMoved -= move.srcAllocation->GetSize();
13166 --m_PassStats.allocationsMoved;
13167 vector->Free(hAllocation: move.dstTmpAllocation);
13168
13169 VmaDeviceMemoryBlock* newBlock = move.srcAllocation->GetBlock();
13170 bool notPresent = true;
13171 for (const FragmentedBlock& block : immovableBlocks)
13172 {
13173 if (block.block == newBlock)
13174 {
13175 notPresent = false;
13176 break;
13177 }
13178 }
13179 if (notPresent)
13180 immovableBlocks.push_back(src: { .data: vectorIndex, .block: newBlock });
13181 break;
13182 }
13183 case VMA_DEFRAGMENTATION_MOVE_OPERATION_DESTROY:
13184 {
13185 m_PassStats.bytesMoved -= move.srcAllocation->GetSize();
13186 --m_PassStats.allocationsMoved;
13187 // Scope for locks, Free have it's own lock
13188 {
13189 VmaMutexLockRead lock(vector->GetMutex(), vector->GetAllocator()->m_UseMutex);
13190 prevCount = vector->GetBlockCount();
13191 freedBlockSize = move.srcAllocation->GetBlock()->m_pMetadata->GetSize();
13192 }
13193 vector->Free(hAllocation: move.srcAllocation);
13194 {
13195 VmaMutexLockRead lock(vector->GetMutex(), vector->GetAllocator()->m_UseMutex);
13196 currentCount = vector->GetBlockCount();
13197 }
13198 freedBlockSize *= prevCount - currentCount;
13199
13200 VkDeviceSize dstBlockSize;
13201 {
13202 VmaMutexLockRead lock(vector->GetMutex(), vector->GetAllocator()->m_UseMutex);
13203 dstBlockSize = move.dstTmpAllocation->GetBlock()->m_pMetadata->GetSize();
13204 }
13205 vector->Free(hAllocation: move.dstTmpAllocation);
13206 {
13207 VmaMutexLockRead lock(vector->GetMutex(), vector->GetAllocator()->m_UseMutex);
13208 freedBlockSize += dstBlockSize * (currentCount - vector->GetBlockCount());
13209 currentCount = vector->GetBlockCount();
13210 }
13211
13212 result = VK_INCOMPLETE;
13213 break;
13214 }
13215 default:
13216 VMA_ASSERT(0);
13217 }
13218
13219 if (prevCount > currentCount)
13220 {
13221 size_t freedBlocks = prevCount - currentCount;
13222 m_PassStats.deviceMemoryBlocksFreed += static_cast<uint32_t>(freedBlocks);
13223 m_PassStats.bytesFreed += freedBlockSize;
13224 }
13225
13226 switch (m_Algorithm)
13227 {
13228 case VMA_DEFRAGMENTATION_FLAG_ALGORITHM_EXTENSIVE_BIT:
13229 {
13230 if (m_AlgorithmState != VMA_NULL)
13231 {
13232 // Avoid unnecessary tries to allocate when new free block is avaiable
13233 StateExtensive& state = reinterpret_cast<StateExtensive*>(m_AlgorithmState)[vectorIndex];
13234 if (state.firstFreeBlock != SIZE_MAX)
13235 {
13236 const size_t diff = prevCount - currentCount;
13237 if (state.firstFreeBlock >= diff)
13238 {
13239 state.firstFreeBlock -= diff;
13240 if (state.firstFreeBlock != 0)
13241 state.firstFreeBlock -= vector->GetBlock(index: state.firstFreeBlock - 1)->m_pMetadata->IsEmpty();
13242 }
13243 else
13244 state.firstFreeBlock = 0;
13245 }
13246 }
13247 }
13248 }
13249 }
13250 moveInfo.moveCount = 0;
13251 moveInfo.pMoves = VMA_NULL;
13252 m_Moves.clear();
13253
13254 // Update stats
13255 m_GlobalStats.allocationsMoved += m_PassStats.allocationsMoved;
13256 m_GlobalStats.bytesFreed += m_PassStats.bytesFreed;
13257 m_GlobalStats.bytesMoved += m_PassStats.bytesMoved;
13258 m_GlobalStats.deviceMemoryBlocksFreed += m_PassStats.deviceMemoryBlocksFreed;
13259 m_PassStats = { .bytesMoved: 0 };
13260
13261 // Move blocks with immovable allocations according to algorithm
13262 if (immovableBlocks.size() > 0)
13263 {
13264 switch (m_Algorithm)
13265 {
13266 case VMA_DEFRAGMENTATION_FLAG_ALGORITHM_EXTENSIVE_BIT:
13267 {
13268 if (m_AlgorithmState != VMA_NULL)
13269 {
13270 bool swapped = false;
13271 // Move to the start of free blocks range
13272 for (const FragmentedBlock& block : immovableBlocks)
13273 {
13274 StateExtensive& state = reinterpret_cast<StateExtensive*>(m_AlgorithmState)[block.data];
13275 if (state.operation != StateExtensive::Operation::Cleanup)
13276 {
13277 VmaBlockVector* vector = m_pBlockVectors[block.data];
13278 VmaMutexLockWrite lock(vector->GetMutex(), vector->GetAllocator()->m_UseMutex);
13279
13280 for (size_t i = 0, count = vector->GetBlockCount() - m_ImmovableBlockCount; i < count; ++i)
13281 {
13282 if (vector->GetBlock(index: i) == block.block)
13283 {
13284 VMA_SWAP(vector->m_Blocks[i], vector->m_Blocks[vector->GetBlockCount() - ++m_ImmovableBlockCount]);
13285 if (state.firstFreeBlock != SIZE_MAX)
13286 {
13287 if (i + 1 < state.firstFreeBlock)
13288 {
13289 if (state.firstFreeBlock > 1)
13290 VMA_SWAP(vector->m_Blocks[i], vector->m_Blocks[--state.firstFreeBlock]);
13291 else
13292 --state.firstFreeBlock;
13293 }
13294 }
13295 swapped = true;
13296 break;
13297 }
13298 }
13299 }
13300 }
13301 if (swapped)
13302 result = VK_INCOMPLETE;
13303 break;
13304 }
13305 }
13306 default:
13307 {
13308 // Move to the begining
13309 for (const FragmentedBlock& block : immovableBlocks)
13310 {
13311 VmaBlockVector* vector = m_pBlockVectors[block.data];
13312 VmaMutexLockWrite lock(vector->GetMutex(), vector->GetAllocator()->m_UseMutex);
13313
13314 for (size_t i = m_ImmovableBlockCount; i < vector->GetBlockCount(); ++i)
13315 {
13316 if (vector->GetBlock(index: i) == block.block)
13317 {
13318 VMA_SWAP(vector->m_Blocks[i], vector->m_Blocks[m_ImmovableBlockCount++]);
13319 break;
13320 }
13321 }
13322 }
13323 break;
13324 }
13325 }
13326 }
13327
13328 // Bulk-map destination blocks
13329 for (const FragmentedBlock& block : mappedBlocks)
13330 {
13331 VkResult res = block.block->Map(hAllocator: allocator, count: block.data, VMA_NULL);
13332 VMA_ASSERT(res == VK_SUCCESS);
13333 }
13334 return result;
13335}
13336
13337bool VmaDefragmentationContext_T::ComputeDefragmentation(VmaBlockVector& vector, size_t index)
13338{
13339 switch (m_Algorithm)
13340 {
13341 case VMA_DEFRAGMENTATION_FLAG_ALGORITHM_FAST_BIT:
13342 return ComputeDefragmentation_Fast(vector);
13343 default:
13344 VMA_ASSERT(0);
13345 case VMA_DEFRAGMENTATION_FLAG_ALGORITHM_BALANCED_BIT:
13346 return ComputeDefragmentation_Balanced(vector, index, update: true);
13347 case VMA_DEFRAGMENTATION_FLAG_ALGORITHM_FULL_BIT:
13348 return ComputeDefragmentation_Full(vector);
13349 case VMA_DEFRAGMENTATION_FLAG_ALGORITHM_EXTENSIVE_BIT:
13350 return ComputeDefragmentation_Extensive(vector, index);
13351 }
13352}
13353
13354VmaDefragmentationContext_T::MoveAllocationData VmaDefragmentationContext_T::GetMoveData(
13355 VmaAllocHandle handle, VmaBlockMetadata* metadata)
13356{
13357 MoveAllocationData moveData;
13358 moveData.move.srcAllocation = (VmaAllocation)metadata->GetAllocationUserData(allocHandle: handle);
13359 moveData.size = moveData.move.srcAllocation->GetSize();
13360 moveData.alignment = moveData.move.srcAllocation->GetAlignment();
13361 moveData.type = moveData.move.srcAllocation->GetSuballocationType();
13362 moveData.flags = 0;
13363
13364 if (moveData.move.srcAllocation->IsPersistentMap())
13365 moveData.flags |= VMA_ALLOCATION_CREATE_MAPPED_BIT;
13366 if (moveData.move.srcAllocation->IsMappingAllowed())
13367 moveData.flags |= VMA_ALLOCATION_CREATE_HOST_ACCESS_SEQUENTIAL_WRITE_BIT | VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT;
13368
13369 return moveData;
13370}
13371
13372VmaDefragmentationContext_T::CounterStatus VmaDefragmentationContext_T::CheckCounters(VkDeviceSize bytes)
13373{
13374 // Ignore allocation if will exceed max size for copy
13375 if (m_PassStats.bytesMoved + bytes > m_MaxPassBytes)
13376 {
13377 if (++m_IgnoredAllocs < MAX_ALLOCS_TO_IGNORE)
13378 return CounterStatus::Ignore;
13379 else
13380 return CounterStatus::End;
13381 }
13382 return CounterStatus::Pass;
13383}
13384
13385bool VmaDefragmentationContext_T::IncrementCounters(VkDeviceSize bytes)
13386{
13387 m_PassStats.bytesMoved += bytes;
13388 // Early return when max found
13389 if (++m_PassStats.allocationsMoved >= m_MaxPassAllocations || m_PassStats.bytesMoved >= m_MaxPassBytes)
13390 {
13391 VMA_ASSERT(m_PassStats.allocationsMoved == m_MaxPassAllocations ||
13392 m_PassStats.bytesMoved == m_MaxPassBytes && "Exceeded maximal pass threshold!");
13393 return true;
13394 }
13395 return false;
13396}
13397
13398bool VmaDefragmentationContext_T::ReallocWithinBlock(VmaBlockVector& vector, VmaDeviceMemoryBlock* block)
13399{
13400 VmaBlockMetadata* metadata = block->m_pMetadata;
13401
13402 for (VmaAllocHandle handle = metadata->GetAllocationListBegin();
13403 handle != VK_NULL_HANDLE;
13404 handle = metadata->GetNextAllocation(prevAlloc: handle))
13405 {
13406 MoveAllocationData moveData = GetMoveData(handle, metadata);
13407 // Ignore newly created allocations by defragmentation algorithm
13408 if (moveData.move.srcAllocation->GetUserData() == this)
13409 continue;
13410 switch (CheckCounters(bytes: moveData.move.srcAllocation->GetSize()))
13411 {
13412 case CounterStatus::Ignore:
13413 continue;
13414 case CounterStatus::End:
13415 return true;
13416 default:
13417 VMA_ASSERT(0);
13418 case CounterStatus::Pass:
13419 break;
13420 }
13421
13422 VkDeviceSize offset = moveData.move.srcAllocation->GetOffset();
13423 if (offset != 0 && metadata->GetSumFreeSize() >= moveData.size)
13424 {
13425 VmaAllocationRequest request = {};
13426 if (metadata->CreateAllocationRequest(
13427 allocSize: moveData.size,
13428 allocAlignment: moveData.alignment,
13429 upperAddress: false,
13430 allocType: moveData.type,
13431 strategy: VMA_ALLOCATION_CREATE_STRATEGY_MIN_OFFSET_BIT,
13432 pAllocationRequest: &request))
13433 {
13434 if (metadata->GetAllocationOffset(allocHandle: request.allocHandle) < offset)
13435 {
13436 if (vector.CommitAllocationRequest(
13437 allocRequest&: request,
13438 pBlock: block,
13439 alignment: moveData.alignment,
13440 allocFlags: moveData.flags,
13441 pUserData: this,
13442 suballocType: moveData.type,
13443 pAllocation: &moveData.move.dstTmpAllocation) == VK_SUCCESS)
13444 {
13445 m_Moves.push_back(src: moveData.move);
13446 if (IncrementCounters(bytes: moveData.size))
13447 return true;
13448 }
13449 }
13450 }
13451 }
13452 }
13453 return false;
13454}
13455
13456bool VmaDefragmentationContext_T::AllocInOtherBlock(size_t start, size_t end, MoveAllocationData& data, VmaBlockVector& vector)
13457{
13458 for (; start < end; ++start)
13459 {
13460 VmaDeviceMemoryBlock* dstBlock = vector.GetBlock(index: start);
13461 if (dstBlock->m_pMetadata->GetSumFreeSize() >= data.size)
13462 {
13463 if (vector.AllocateFromBlock(pBlock: dstBlock,
13464 size: data.size,
13465 alignment: data.alignment,
13466 allocFlags: data.flags,
13467 pUserData: this,
13468 suballocType: data.type,
13469 strategy: 0,
13470 pAllocation: &data.move.dstTmpAllocation) == VK_SUCCESS)
13471 {
13472 m_Moves.push_back(src: data.move);
13473 if (IncrementCounters(bytes: data.size))
13474 return true;
13475 break;
13476 }
13477 }
13478 }
13479 return false;
13480}
13481
13482bool VmaDefragmentationContext_T::ComputeDefragmentation_Fast(VmaBlockVector& vector)
13483{
13484 // Move only between blocks
13485
13486 // Go through allocations in last blocks and try to fit them inside first ones
13487 for (size_t i = vector.GetBlockCount() - 1; i > m_ImmovableBlockCount; --i)
13488 {
13489 VmaBlockMetadata* metadata = vector.GetBlock(index: i)->m_pMetadata;
13490
13491 for (VmaAllocHandle handle = metadata->GetAllocationListBegin();
13492 handle != VK_NULL_HANDLE;
13493 handle = metadata->GetNextAllocation(prevAlloc: handle))
13494 {
13495 MoveAllocationData moveData = GetMoveData(handle, metadata);
13496 // Ignore newly created allocations by defragmentation algorithm
13497 if (moveData.move.srcAllocation->GetUserData() == this)
13498 continue;
13499 switch (CheckCounters(bytes: moveData.move.srcAllocation->GetSize()))
13500 {
13501 case CounterStatus::Ignore:
13502 continue;
13503 case CounterStatus::End:
13504 return true;
13505 default:
13506 VMA_ASSERT(0);
13507 case CounterStatus::Pass:
13508 break;
13509 }
13510
13511 // Check all previous blocks for free space
13512 if (AllocInOtherBlock(start: 0, end: i, data&: moveData, vector))
13513 return true;
13514 }
13515 }
13516 return false;
13517}
13518
13519bool VmaDefragmentationContext_T::ComputeDefragmentation_Balanced(VmaBlockVector& vector, size_t index, bool update)
13520{
13521 // Go over every allocation and try to fit it in previous blocks at lowest offsets,
13522 // if not possible: realloc within single block to minimize offset (exclude offset == 0),
13523 // but only if there are noticable gaps between them (some heuristic, ex. average size of allocation in block)
13524 VMA_ASSERT(m_AlgorithmState != VMA_NULL);
13525
13526 StateBalanced& vectorState = reinterpret_cast<StateBalanced*>(m_AlgorithmState)[index];
13527 if (update && vectorState.avgAllocSize == UINT64_MAX)
13528 UpdateVectorStatistics(vector, state&: vectorState);
13529
13530 const size_t startMoveCount = m_Moves.size();
13531 VkDeviceSize minimalFreeRegion = vectorState.avgFreeSize / 2;
13532 for (size_t i = vector.GetBlockCount() - 1; i > m_ImmovableBlockCount; --i)
13533 {
13534 VmaDeviceMemoryBlock* block = vector.GetBlock(index: i);
13535 VmaBlockMetadata* metadata = block->m_pMetadata;
13536 VkDeviceSize prevFreeRegionSize = 0;
13537
13538 for (VmaAllocHandle handle = metadata->GetAllocationListBegin();
13539 handle != VK_NULL_HANDLE;
13540 handle = metadata->GetNextAllocation(prevAlloc: handle))
13541 {
13542 MoveAllocationData moveData = GetMoveData(handle, metadata);
13543 // Ignore newly created allocations by defragmentation algorithm
13544 if (moveData.move.srcAllocation->GetUserData() == this)
13545 continue;
13546 switch (CheckCounters(bytes: moveData.move.srcAllocation->GetSize()))
13547 {
13548 case CounterStatus::Ignore:
13549 continue;
13550 case CounterStatus::End:
13551 return true;
13552 default:
13553 VMA_ASSERT(0);
13554 case CounterStatus::Pass:
13555 break;
13556 }
13557
13558 // Check all previous blocks for free space
13559 const size_t prevMoveCount = m_Moves.size();
13560 if (AllocInOtherBlock(start: 0, end: i, data&: moveData, vector))
13561 return true;
13562
13563 VkDeviceSize nextFreeRegionSize = metadata->GetNextFreeRegionSize(alloc: handle);
13564 // If no room found then realloc within block for lower offset
13565 VkDeviceSize offset = moveData.move.srcAllocation->GetOffset();
13566 if (prevMoveCount == m_Moves.size() && offset != 0 && metadata->GetSumFreeSize() >= moveData.size)
13567 {
13568 // Check if realloc will make sense
13569 if (prevFreeRegionSize >= minimalFreeRegion ||
13570 nextFreeRegionSize >= minimalFreeRegion ||
13571 moveData.size <= vectorState.avgFreeSize ||
13572 moveData.size <= vectorState.avgAllocSize)
13573 {
13574 VmaAllocationRequest request = {};
13575 if (metadata->CreateAllocationRequest(
13576 allocSize: moveData.size,
13577 allocAlignment: moveData.alignment,
13578 upperAddress: false,
13579 allocType: moveData.type,
13580 strategy: VMA_ALLOCATION_CREATE_STRATEGY_MIN_OFFSET_BIT,
13581 pAllocationRequest: &request))
13582 {
13583 if (metadata->GetAllocationOffset(allocHandle: request.allocHandle) < offset)
13584 {
13585 if (vector.CommitAllocationRequest(
13586 allocRequest&: request,
13587 pBlock: block,
13588 alignment: moveData.alignment,
13589 allocFlags: moveData.flags,
13590 pUserData: this,
13591 suballocType: moveData.type,
13592 pAllocation: &moveData.move.dstTmpAllocation) == VK_SUCCESS)
13593 {
13594 m_Moves.push_back(src: moveData.move);
13595 if (IncrementCounters(bytes: moveData.size))
13596 return true;
13597 }
13598 }
13599 }
13600 }
13601 }
13602 prevFreeRegionSize = nextFreeRegionSize;
13603 }
13604 }
13605
13606 // No moves perfomed, update statistics to current vector state
13607 if (startMoveCount == m_Moves.size() && !update)
13608 {
13609 vectorState.avgAllocSize = UINT64_MAX;
13610 return ComputeDefragmentation_Balanced(vector, index, update: false);
13611 }
13612 return false;
13613}
13614
13615bool VmaDefragmentationContext_T::ComputeDefragmentation_Full(VmaBlockVector& vector)
13616{
13617 // Go over every allocation and try to fit it in previous blocks at lowest offsets,
13618 // if not possible: realloc within single block to minimize offset (exclude offset == 0)
13619
13620 for (size_t i = vector.GetBlockCount() - 1; i > m_ImmovableBlockCount; --i)
13621 {
13622 VmaDeviceMemoryBlock* block = vector.GetBlock(index: i);
13623 VmaBlockMetadata* metadata = block->m_pMetadata;
13624
13625 for (VmaAllocHandle handle = metadata->GetAllocationListBegin();
13626 handle != VK_NULL_HANDLE;
13627 handle = metadata->GetNextAllocation(prevAlloc: handle))
13628 {
13629 MoveAllocationData moveData = GetMoveData(handle, metadata);
13630 // Ignore newly created allocations by defragmentation algorithm
13631 if (moveData.move.srcAllocation->GetUserData() == this)
13632 continue;
13633 switch (CheckCounters(bytes: moveData.move.srcAllocation->GetSize()))
13634 {
13635 case CounterStatus::Ignore:
13636 continue;
13637 case CounterStatus::End:
13638 return true;
13639 default:
13640 VMA_ASSERT(0);
13641 case CounterStatus::Pass:
13642 break;
13643 }
13644
13645 // Check all previous blocks for free space
13646 const size_t prevMoveCount = m_Moves.size();
13647 if (AllocInOtherBlock(start: 0, end: i, data&: moveData, vector))
13648 return true;
13649
13650 // If no room found then realloc within block for lower offset
13651 VkDeviceSize offset = moveData.move.srcAllocation->GetOffset();
13652 if (prevMoveCount == m_Moves.size() && offset != 0 && metadata->GetSumFreeSize() >= moveData.size)
13653 {
13654 VmaAllocationRequest request = {};
13655 if (metadata->CreateAllocationRequest(
13656 allocSize: moveData.size,
13657 allocAlignment: moveData.alignment,
13658 upperAddress: false,
13659 allocType: moveData.type,
13660 strategy: VMA_ALLOCATION_CREATE_STRATEGY_MIN_OFFSET_BIT,
13661 pAllocationRequest: &request))
13662 {
13663 if (metadata->GetAllocationOffset(allocHandle: request.allocHandle) < offset)
13664 {
13665 if (vector.CommitAllocationRequest(
13666 allocRequest&: request,
13667 pBlock: block,
13668 alignment: moveData.alignment,
13669 allocFlags: moveData.flags,
13670 pUserData: this,
13671 suballocType: moveData.type,
13672 pAllocation: &moveData.move.dstTmpAllocation) == VK_SUCCESS)
13673 {
13674 m_Moves.push_back(src: moveData.move);
13675 if (IncrementCounters(bytes: moveData.size))
13676 return true;
13677 }
13678 }
13679 }
13680 }
13681 }
13682 }
13683 return false;
13684}
13685
13686bool VmaDefragmentationContext_T::ComputeDefragmentation_Extensive(VmaBlockVector& vector, size_t index)
13687{
13688 // First free single block, then populate it to the brim, then free another block, and so on
13689
13690 // Fallback to previous algorithm since without granularity conflicts it can achieve max packing
13691 if (vector.m_BufferImageGranularity == 1)
13692 return ComputeDefragmentation_Full(vector);
13693
13694 VMA_ASSERT(m_AlgorithmState != VMA_NULL);
13695
13696 StateExtensive& vectorState = reinterpret_cast<StateExtensive*>(m_AlgorithmState)[index];
13697
13698 bool texturePresent = false, bufferPresent = false, otherPresent = false;
13699 switch (vectorState.operation)
13700 {
13701 case StateExtensive::Operation::Done: // Vector defragmented
13702 return false;
13703 case StateExtensive::Operation::FindFreeBlockBuffer:
13704 case StateExtensive::Operation::FindFreeBlockTexture:
13705 case StateExtensive::Operation::FindFreeBlockAll:
13706 {
13707 // No more blocks to free, just perform fast realloc and move to cleanup
13708 if (vectorState.firstFreeBlock == 0)
13709 {
13710 vectorState.operation = StateExtensive::Operation::Cleanup;
13711 return ComputeDefragmentation_Fast(vector);
13712 }
13713
13714 // No free blocks, have to clear last one
13715 size_t last = (vectorState.firstFreeBlock == SIZE_MAX ? vector.GetBlockCount() : vectorState.firstFreeBlock) - 1;
13716 VmaBlockMetadata* freeMetadata = vector.GetBlock(index: last)->m_pMetadata;
13717
13718 const size_t prevMoveCount = m_Moves.size();
13719 for (VmaAllocHandle handle = freeMetadata->GetAllocationListBegin();
13720 handle != VK_NULL_HANDLE;
13721 handle = freeMetadata->GetNextAllocation(prevAlloc: handle))
13722 {
13723 MoveAllocationData moveData = GetMoveData(handle, metadata: freeMetadata);
13724 switch (CheckCounters(bytes: moveData.move.srcAllocation->GetSize()))
13725 {
13726 case CounterStatus::Ignore:
13727 continue;
13728 case CounterStatus::End:
13729 return true;
13730 default:
13731 VMA_ASSERT(0);
13732 case CounterStatus::Pass:
13733 break;
13734 }
13735
13736 // Check all previous blocks for free space
13737 if (AllocInOtherBlock(start: 0, end: last, data&: moveData, vector))
13738 {
13739 // Full clear performed already
13740 if (prevMoveCount != m_Moves.size() && freeMetadata->GetNextAllocation(prevAlloc: handle) == VK_NULL_HANDLE)
13741 reinterpret_cast<size_t*>(m_AlgorithmState)[index] = last;
13742 return true;
13743 }
13744 }
13745
13746 if (prevMoveCount == m_Moves.size())
13747 {
13748 // Cannot perform full clear, have to move data in other blocks around
13749 if (last != 0)
13750 {
13751 for (size_t i = last - 1; i; --i)
13752 {
13753 if (ReallocWithinBlock(vector, block: vector.GetBlock(index: i)))
13754 return true;
13755 }
13756 }
13757
13758 if (prevMoveCount == m_Moves.size())
13759 {
13760 // No possible reallocs within blocks, try to move them around fast
13761 return ComputeDefragmentation_Fast(vector);
13762 }
13763 }
13764 else
13765 {
13766 switch (vectorState.operation)
13767 {
13768 case StateExtensive::Operation::FindFreeBlockBuffer:
13769 vectorState.operation = StateExtensive::Operation::MoveBuffers;
13770 break;
13771 default:
13772 VMA_ASSERT(0);
13773 case StateExtensive::Operation::FindFreeBlockTexture:
13774 vectorState.operation = StateExtensive::Operation::MoveTextures;
13775 break;
13776 case StateExtensive::Operation::FindFreeBlockAll:
13777 vectorState.operation = StateExtensive::Operation::MoveAll;
13778 break;
13779 }
13780 vectorState.firstFreeBlock = last;
13781 // Nothing done, block found without reallocations, can perform another reallocs in same pass
13782 return ComputeDefragmentation_Extensive(vector, index);
13783 }
13784 break;
13785 }
13786 case StateExtensive::Operation::MoveTextures:
13787 {
13788 if (MoveDataToFreeBlocks(currentType: VMA_SUBALLOCATION_TYPE_IMAGE_OPTIMAL, vector,
13789 firstFreeBlock: vectorState.firstFreeBlock, texturePresent, bufferPresent, otherPresent))
13790 {
13791 if (texturePresent)
13792 {
13793 vectorState.operation = StateExtensive::Operation::FindFreeBlockTexture;
13794 return ComputeDefragmentation_Extensive(vector, index);
13795 }
13796
13797 if (!bufferPresent && !otherPresent)
13798 {
13799 vectorState.operation = StateExtensive::Operation::Cleanup;
13800 break;
13801 }
13802
13803 // No more textures to move, check buffers
13804 vectorState.operation = StateExtensive::Operation::MoveBuffers;
13805 bufferPresent = false;
13806 otherPresent = false;
13807 }
13808 else
13809 break;
13810 }
13811 case StateExtensive::Operation::MoveBuffers:
13812 {
13813 if (MoveDataToFreeBlocks(currentType: VMA_SUBALLOCATION_TYPE_BUFFER, vector,
13814 firstFreeBlock: vectorState.firstFreeBlock, texturePresent, bufferPresent, otherPresent))
13815 {
13816 if (bufferPresent)
13817 {
13818 vectorState.operation = StateExtensive::Operation::FindFreeBlockBuffer;
13819 return ComputeDefragmentation_Extensive(vector, index);
13820 }
13821
13822 if (!otherPresent)
13823 {
13824 vectorState.operation = StateExtensive::Operation::Cleanup;
13825 break;
13826 }
13827
13828 // No more buffers to move, check all others
13829 vectorState.operation = StateExtensive::Operation::MoveAll;
13830 otherPresent = false;
13831 }
13832 else
13833 break;
13834 }
13835 case StateExtensive::Operation::MoveAll:
13836 {
13837 if (MoveDataToFreeBlocks(currentType: VMA_SUBALLOCATION_TYPE_FREE, vector,
13838 firstFreeBlock: vectorState.firstFreeBlock, texturePresent, bufferPresent, otherPresent))
13839 {
13840 if (otherPresent)
13841 {
13842 vectorState.operation = StateExtensive::Operation::FindFreeBlockBuffer;
13843 return ComputeDefragmentation_Extensive(vector, index);
13844 }
13845 // Everything moved
13846 vectorState.operation = StateExtensive::Operation::Cleanup;
13847 }
13848 break;
13849 }
13850 case StateExtensive::Operation::Cleanup:
13851 // Cleanup is handled below so that other operations may reuse the cleanup code. This case is here to prevent the unhandled enum value warning (C4062).
13852 break;
13853 }
13854
13855 if (vectorState.operation == StateExtensive::Operation::Cleanup)
13856 {
13857 // All other work done, pack data in blocks even tighter if possible
13858 const size_t prevMoveCount = m_Moves.size();
13859 for (size_t i = 0; i < vector.GetBlockCount(); ++i)
13860 {
13861 if (ReallocWithinBlock(vector, block: vector.GetBlock(index: i)))
13862 return true;
13863 }
13864
13865 if (prevMoveCount == m_Moves.size())
13866 vectorState.operation = StateExtensive::Operation::Done;
13867 }
13868 return false;
13869}
13870
13871void VmaDefragmentationContext_T::UpdateVectorStatistics(VmaBlockVector& vector, StateBalanced& state)
13872{
13873 size_t allocCount = 0;
13874 size_t freeCount = 0;
13875 state.avgFreeSize = 0;
13876 state.avgAllocSize = 0;
13877
13878 for (size_t i = 0; i < vector.GetBlockCount(); ++i)
13879 {
13880 VmaBlockMetadata* metadata = vector.GetBlock(index: i)->m_pMetadata;
13881
13882 allocCount += metadata->GetAllocationCount();
13883 freeCount += metadata->GetFreeRegionsCount();
13884 state.avgFreeSize += metadata->GetSumFreeSize();
13885 state.avgAllocSize += metadata->GetSize();
13886 }
13887
13888 state.avgAllocSize = (state.avgAllocSize - state.avgFreeSize) / allocCount;
13889 state.avgFreeSize /= freeCount;
13890}
13891
13892bool VmaDefragmentationContext_T::MoveDataToFreeBlocks(VmaSuballocationType currentType,
13893 VmaBlockVector& vector, size_t firstFreeBlock,
13894 bool& texturePresent, bool& bufferPresent, bool& otherPresent)
13895{
13896 const size_t prevMoveCount = m_Moves.size();
13897 for (size_t i = firstFreeBlock ; i;)
13898 {
13899 VmaDeviceMemoryBlock* block = vector.GetBlock(index: --i);
13900 VmaBlockMetadata* metadata = block->m_pMetadata;
13901
13902 for (VmaAllocHandle handle = metadata->GetAllocationListBegin();
13903 handle != VK_NULL_HANDLE;
13904 handle = metadata->GetNextAllocation(prevAlloc: handle))
13905 {
13906 MoveAllocationData moveData = GetMoveData(handle, metadata);
13907 // Ignore newly created allocations by defragmentation algorithm
13908 if (moveData.move.srcAllocation->GetUserData() == this)
13909 continue;
13910 switch (CheckCounters(bytes: moveData.move.srcAllocation->GetSize()))
13911 {
13912 case CounterStatus::Ignore:
13913 continue;
13914 case CounterStatus::End:
13915 return true;
13916 default:
13917 VMA_ASSERT(0);
13918 case CounterStatus::Pass:
13919 break;
13920 }
13921
13922 // Move only single type of resources at once
13923 if (!VmaIsBufferImageGranularityConflict(suballocType1: moveData.type, suballocType2: currentType))
13924 {
13925 // Try to fit allocation into free blocks
13926 if (AllocInOtherBlock(start: firstFreeBlock, end: vector.GetBlockCount(), data&: moveData, vector))
13927 return false;
13928 }
13929
13930 if (!VmaIsBufferImageGranularityConflict(suballocType1: moveData.type, suballocType2: VMA_SUBALLOCATION_TYPE_IMAGE_OPTIMAL))
13931 texturePresent = true;
13932 else if (!VmaIsBufferImageGranularityConflict(suballocType1: moveData.type, suballocType2: VMA_SUBALLOCATION_TYPE_BUFFER))
13933 bufferPresent = true;
13934 else
13935 otherPresent = true;
13936 }
13937 }
13938 return prevMoveCount == m_Moves.size();
13939}
13940#endif // _VMA_DEFRAGMENTATION_CONTEXT_FUNCTIONS
13941
13942#ifndef _VMA_POOL_T_FUNCTIONS
13943VmaPool_T::VmaPool_T(
13944 VmaAllocator hAllocator,
13945 const VmaPoolCreateInfo& createInfo,
13946 VkDeviceSize preferredBlockSize)
13947 : m_BlockVector(
13948 hAllocator,
13949 this, // hParentPool
13950 createInfo.memoryTypeIndex,
13951 createInfo.blockSize != 0 ? createInfo.blockSize : preferredBlockSize,
13952 createInfo.minBlockCount,
13953 createInfo.maxBlockCount,
13954 (createInfo.flags& VMA_POOL_CREATE_IGNORE_BUFFER_IMAGE_GRANULARITY_BIT) != 0 ? 1 : hAllocator->GetBufferImageGranularity(),
13955 createInfo.blockSize != 0, // explicitBlockSize
13956 createInfo.flags & VMA_POOL_CREATE_ALGORITHM_MASK, // algorithm
13957 createInfo.priority,
13958 VMA_MAX(hAllocator->GetMemoryTypeMinAlignment(createInfo.memoryTypeIndex), createInfo.minAllocationAlignment),
13959 createInfo.pMemoryAllocateNext),
13960 m_Id(0),
13961 m_Name(VMA_NULL) {}
13962
13963VmaPool_T::~VmaPool_T()
13964{
13965 VMA_ASSERT(m_PrevPool == VMA_NULL && m_NextPool == VMA_NULL);
13966}
13967
13968void VmaPool_T::SetName(const char* pName)
13969{
13970 const VkAllocationCallbacks* allocs = m_BlockVector.GetAllocator()->GetAllocationCallbacks();
13971 VmaFreeString(allocs, str: m_Name);
13972
13973 if (pName != VMA_NULL)
13974 {
13975 m_Name = VmaCreateStringCopy(allocs, srcStr: pName);
13976 }
13977 else
13978 {
13979 m_Name = VMA_NULL;
13980 }
13981}
13982#endif // _VMA_POOL_T_FUNCTIONS
13983
13984#ifndef _VMA_ALLOCATOR_T_FUNCTIONS
13985VmaAllocator_T::VmaAllocator_T(const VmaAllocatorCreateInfo* pCreateInfo) :
13986 m_UseMutex((pCreateInfo->flags & VMA_ALLOCATOR_CREATE_EXTERNALLY_SYNCHRONIZED_BIT) == 0),
13987 m_VulkanApiVersion(pCreateInfo->vulkanApiVersion != 0 ? pCreateInfo->vulkanApiVersion : VK_API_VERSION_1_0),
13988 m_UseKhrDedicatedAllocation((pCreateInfo->flags & VMA_ALLOCATOR_CREATE_KHR_DEDICATED_ALLOCATION_BIT) != 0),
13989 m_UseKhrBindMemory2((pCreateInfo->flags & VMA_ALLOCATOR_CREATE_KHR_BIND_MEMORY2_BIT) != 0),
13990 m_UseExtMemoryBudget((pCreateInfo->flags & VMA_ALLOCATOR_CREATE_EXT_MEMORY_BUDGET_BIT) != 0),
13991 m_UseAmdDeviceCoherentMemory((pCreateInfo->flags & VMA_ALLOCATOR_CREATE_AMD_DEVICE_COHERENT_MEMORY_BIT) != 0),
13992 m_UseKhrBufferDeviceAddress((pCreateInfo->flags & VMA_ALLOCATOR_CREATE_BUFFER_DEVICE_ADDRESS_BIT) != 0),
13993 m_UseExtMemoryPriority((pCreateInfo->flags & VMA_ALLOCATOR_CREATE_EXT_MEMORY_PRIORITY_BIT) != 0),
13994 m_hDevice(pCreateInfo->device),
13995 m_hInstance(pCreateInfo->instance),
13996 m_AllocationCallbacksSpecified(pCreateInfo->pAllocationCallbacks != VMA_NULL),
13997 m_AllocationCallbacks(pCreateInfo->pAllocationCallbacks ?
13998 *pCreateInfo->pAllocationCallbacks : VmaEmptyAllocationCallbacks),
13999 m_AllocationObjectAllocator(&m_AllocationCallbacks),
14000 m_HeapSizeLimitMask(0),
14001 m_DeviceMemoryCount(0),
14002 m_PreferredLargeHeapBlockSize(0),
14003 m_PhysicalDevice(pCreateInfo->physicalDevice),
14004 m_GpuDefragmentationMemoryTypeBits(UINT32_MAX),
14005 m_NextPoolId(0),
14006 m_GlobalMemoryTypeBits(UINT32_MAX)
14007{
14008 if(m_VulkanApiVersion >= VK_MAKE_VERSION(1, 1, 0))
14009 {
14010 m_UseKhrDedicatedAllocation = false;
14011 m_UseKhrBindMemory2 = false;
14012 }
14013
14014 if(VMA_DEBUG_DETECT_CORRUPTION)
14015 {
14016 // Needs to be multiply of uint32_t size because we are going to write VMA_CORRUPTION_DETECTION_MAGIC_VALUE to it.
14017 VMA_ASSERT(VMA_DEBUG_MARGIN % sizeof(uint32_t) == 0);
14018 }
14019
14020 VMA_ASSERT(pCreateInfo->physicalDevice && pCreateInfo->device && pCreateInfo->instance);
14021
14022 if(m_VulkanApiVersion < VK_MAKE_VERSION(1, 1, 0))
14023 {
14024#if !(VMA_DEDICATED_ALLOCATION)
14025 if((pCreateInfo->flags & VMA_ALLOCATOR_CREATE_KHR_DEDICATED_ALLOCATION_BIT) != 0)
14026 {
14027 VMA_ASSERT(0 && "VMA_ALLOCATOR_CREATE_KHR_DEDICATED_ALLOCATION_BIT set but required extensions are disabled by preprocessor macros.");
14028 }
14029#endif
14030#if !(VMA_BIND_MEMORY2)
14031 if((pCreateInfo->flags & VMA_ALLOCATOR_CREATE_KHR_BIND_MEMORY2_BIT) != 0)
14032 {
14033 VMA_ASSERT(0 && "VMA_ALLOCATOR_CREATE_KHR_BIND_MEMORY2_BIT set but required extension is disabled by preprocessor macros.");
14034 }
14035#endif
14036 }
14037#if !(VMA_MEMORY_BUDGET)
14038 if((pCreateInfo->flags & VMA_ALLOCATOR_CREATE_EXT_MEMORY_BUDGET_BIT) != 0)
14039 {
14040 VMA_ASSERT(0 && "VMA_ALLOCATOR_CREATE_EXT_MEMORY_BUDGET_BIT set but required extension is disabled by preprocessor macros.");
14041 }
14042#endif
14043#if !(VMA_BUFFER_DEVICE_ADDRESS)
14044 if(m_UseKhrBufferDeviceAddress)
14045 {
14046 VMA_ASSERT(0 && "VMA_ALLOCATOR_CREATE_BUFFER_DEVICE_ADDRESS_BIT is set but required extension or Vulkan 1.2 is not available in your Vulkan header or its support in VMA has been disabled by a preprocessor macro.");
14047 }
14048#endif
14049#if VMA_VULKAN_VERSION < 1002000
14050 if(m_VulkanApiVersion >= VK_MAKE_VERSION(1, 2, 0))
14051 {
14052 VMA_ASSERT(0 && "vulkanApiVersion >= VK_API_VERSION_1_2 but required Vulkan version is disabled by preprocessor macros.");
14053 }
14054#endif
14055#if VMA_VULKAN_VERSION < 1001000
14056 if(m_VulkanApiVersion >= VK_MAKE_VERSION(1, 1, 0))
14057 {
14058 VMA_ASSERT(0 && "vulkanApiVersion >= VK_API_VERSION_1_1 but required Vulkan version is disabled by preprocessor macros.");
14059 }
14060#endif
14061#if !(VMA_MEMORY_PRIORITY)
14062 if(m_UseExtMemoryPriority)
14063 {
14064 VMA_ASSERT(0 && "VMA_ALLOCATOR_CREATE_EXT_MEMORY_PRIORITY_BIT is set but required extension is not available in your Vulkan header or its support in VMA has been disabled by a preprocessor macro.");
14065 }
14066#endif
14067
14068 memset(s: &m_DeviceMemoryCallbacks, c: 0 ,n: sizeof(m_DeviceMemoryCallbacks));
14069 memset(s: &m_PhysicalDeviceProperties, c: 0, n: sizeof(m_PhysicalDeviceProperties));
14070 memset(s: &m_MemProps, c: 0, n: sizeof(m_MemProps));
14071
14072 memset(s: &m_pBlockVectors, c: 0, n: sizeof(m_pBlockVectors));
14073 memset(s: &m_VulkanFunctions, c: 0, n: sizeof(m_VulkanFunctions));
14074
14075#if VMA_EXTERNAL_MEMORY
14076 memset(s: &m_TypeExternalMemoryHandleTypes, c: 0, n: sizeof(m_TypeExternalMemoryHandleTypes));
14077#endif // #if VMA_EXTERNAL_MEMORY
14078
14079 if(pCreateInfo->pDeviceMemoryCallbacks != VMA_NULL)
14080 {
14081 m_DeviceMemoryCallbacks.pUserData = pCreateInfo->pDeviceMemoryCallbacks->pUserData;
14082 m_DeviceMemoryCallbacks.pfnAllocate = pCreateInfo->pDeviceMemoryCallbacks->pfnAllocate;
14083 m_DeviceMemoryCallbacks.pfnFree = pCreateInfo->pDeviceMemoryCallbacks->pfnFree;
14084 }
14085
14086 ImportVulkanFunctions(pVulkanFunctions: pCreateInfo->pVulkanFunctions);
14087
14088 (*m_VulkanFunctions.vkGetPhysicalDeviceProperties)(m_PhysicalDevice, &m_PhysicalDeviceProperties);
14089 (*m_VulkanFunctions.vkGetPhysicalDeviceMemoryProperties)(m_PhysicalDevice, &m_MemProps);
14090
14091 VMA_ASSERT(VmaIsPow2(VMA_MIN_ALIGNMENT));
14092 VMA_ASSERT(VmaIsPow2(VMA_DEBUG_MIN_BUFFER_IMAGE_GRANULARITY));
14093 VMA_ASSERT(VmaIsPow2(m_PhysicalDeviceProperties.limits.bufferImageGranularity));
14094 VMA_ASSERT(VmaIsPow2(m_PhysicalDeviceProperties.limits.nonCoherentAtomSize));
14095
14096 m_PreferredLargeHeapBlockSize = (pCreateInfo->preferredLargeHeapBlockSize != 0) ?
14097 pCreateInfo->preferredLargeHeapBlockSize : static_cast<VkDeviceSize>(VMA_DEFAULT_LARGE_HEAP_BLOCK_SIZE);
14098
14099 m_GlobalMemoryTypeBits = CalculateGlobalMemoryTypeBits();
14100
14101#if VMA_EXTERNAL_MEMORY
14102 if(pCreateInfo->pTypeExternalMemoryHandleTypes != VMA_NULL)
14103 {
14104 memcpy(dest: m_TypeExternalMemoryHandleTypes, src: pCreateInfo->pTypeExternalMemoryHandleTypes,
14105 n: sizeof(VkExternalMemoryHandleTypeFlagsKHR) * GetMemoryTypeCount());
14106 }
14107#endif // #if VMA_EXTERNAL_MEMORY
14108
14109 if(pCreateInfo->pHeapSizeLimit != VMA_NULL)
14110 {
14111 for(uint32_t heapIndex = 0; heapIndex < GetMemoryHeapCount(); ++heapIndex)
14112 {
14113 const VkDeviceSize limit = pCreateInfo->pHeapSizeLimit[heapIndex];
14114 if(limit != VK_WHOLE_SIZE)
14115 {
14116 m_HeapSizeLimitMask |= 1u << heapIndex;
14117 if(limit < m_MemProps.memoryHeaps[heapIndex].size)
14118 {
14119 m_MemProps.memoryHeaps[heapIndex].size = limit;
14120 }
14121 }
14122 }
14123 }
14124
14125 for(uint32_t memTypeIndex = 0; memTypeIndex < GetMemoryTypeCount(); ++memTypeIndex)
14126 {
14127 // Create only supported types
14128 if((m_GlobalMemoryTypeBits & (1u << memTypeIndex)) != 0)
14129 {
14130 const VkDeviceSize preferredBlockSize = CalcPreferredBlockSize(memTypeIndex);
14131 m_pBlockVectors[memTypeIndex] = vma_new(this, VmaBlockVector)(
14132 this,
14133 VK_NULL_HANDLE, // hParentPool
14134 memTypeIndex,
14135 preferredBlockSize,
14136 0,
14137 SIZE_MAX,
14138 GetBufferImageGranularity(),
14139 false, // explicitBlockSize
14140 0, // algorithm
14141 0.5f, // priority (0.5 is the default per Vulkan spec)
14142 GetMemoryTypeMinAlignment(memTypeIndex), // minAllocationAlignment
14143 VMA_NULL); // // pMemoryAllocateNext
14144 // No need to call m_pBlockVectors[memTypeIndex][blockVectorTypeIndex]->CreateMinBlocks here,
14145 // becase minBlockCount is 0.
14146 }
14147 }
14148}
14149
14150VkResult VmaAllocator_T::Init(const VmaAllocatorCreateInfo* pCreateInfo)
14151{
14152 VkResult res = VK_SUCCESS;
14153
14154#if VMA_MEMORY_BUDGET
14155 if(m_UseExtMemoryBudget)
14156 {
14157 UpdateVulkanBudget();
14158 }
14159#endif // #if VMA_MEMORY_BUDGET
14160
14161 return res;
14162}
14163
14164VmaAllocator_T::~VmaAllocator_T()
14165{
14166 VMA_ASSERT(m_Pools.IsEmpty());
14167
14168 for(size_t memTypeIndex = GetMemoryTypeCount(); memTypeIndex--; )
14169 {
14170 vma_delete(hAllocator: this, ptr: m_pBlockVectors[memTypeIndex]);
14171 }
14172}
14173
14174void VmaAllocator_T::ImportVulkanFunctions(const VmaVulkanFunctions* pVulkanFunctions)
14175{
14176#if VMA_STATIC_VULKAN_FUNCTIONS == 1
14177 ImportVulkanFunctions_Static();
14178#endif
14179
14180 if(pVulkanFunctions != VMA_NULL)
14181 {
14182 ImportVulkanFunctions_Custom(pVulkanFunctions);
14183 }
14184
14185#if VMA_DYNAMIC_VULKAN_FUNCTIONS == 1
14186 ImportVulkanFunctions_Dynamic();
14187#endif
14188
14189 ValidateVulkanFunctions();
14190}
14191
14192#if VMA_STATIC_VULKAN_FUNCTIONS == 1
14193
14194void VmaAllocator_T::ImportVulkanFunctions_Static()
14195{
14196 // Vulkan 1.0
14197 m_VulkanFunctions.vkGetInstanceProcAddr = (PFN_vkGetInstanceProcAddr)vkGetInstanceProcAddr;
14198 m_VulkanFunctions.vkGetDeviceProcAddr = (PFN_vkGetDeviceProcAddr)vkGetDeviceProcAddr;
14199 m_VulkanFunctions.vkGetPhysicalDeviceProperties = (PFN_vkGetPhysicalDeviceProperties)vkGetPhysicalDeviceProperties;
14200 m_VulkanFunctions.vkGetPhysicalDeviceMemoryProperties = (PFN_vkGetPhysicalDeviceMemoryProperties)vkGetPhysicalDeviceMemoryProperties;
14201 m_VulkanFunctions.vkAllocateMemory = (PFN_vkAllocateMemory)vkAllocateMemory;
14202 m_VulkanFunctions.vkFreeMemory = (PFN_vkFreeMemory)vkFreeMemory;
14203 m_VulkanFunctions.vkMapMemory = (PFN_vkMapMemory)vkMapMemory;
14204 m_VulkanFunctions.vkUnmapMemory = (PFN_vkUnmapMemory)vkUnmapMemory;
14205 m_VulkanFunctions.vkFlushMappedMemoryRanges = (PFN_vkFlushMappedMemoryRanges)vkFlushMappedMemoryRanges;
14206 m_VulkanFunctions.vkInvalidateMappedMemoryRanges = (PFN_vkInvalidateMappedMemoryRanges)vkInvalidateMappedMemoryRanges;
14207 m_VulkanFunctions.vkBindBufferMemory = (PFN_vkBindBufferMemory)vkBindBufferMemory;
14208 m_VulkanFunctions.vkBindImageMemory = (PFN_vkBindImageMemory)vkBindImageMemory;
14209 m_VulkanFunctions.vkGetBufferMemoryRequirements = (PFN_vkGetBufferMemoryRequirements)vkGetBufferMemoryRequirements;
14210 m_VulkanFunctions.vkGetImageMemoryRequirements = (PFN_vkGetImageMemoryRequirements)vkGetImageMemoryRequirements;
14211 m_VulkanFunctions.vkCreateBuffer = (PFN_vkCreateBuffer)vkCreateBuffer;
14212 m_VulkanFunctions.vkDestroyBuffer = (PFN_vkDestroyBuffer)vkDestroyBuffer;
14213 m_VulkanFunctions.vkCreateImage = (PFN_vkCreateImage)vkCreateImage;
14214 m_VulkanFunctions.vkDestroyImage = (PFN_vkDestroyImage)vkDestroyImage;
14215 m_VulkanFunctions.vkCmdCopyBuffer = (PFN_vkCmdCopyBuffer)vkCmdCopyBuffer;
14216
14217 // Vulkan 1.1
14218#if VMA_VULKAN_VERSION >= 1001000
14219 if(m_VulkanApiVersion >= VK_MAKE_VERSION(1, 1, 0))
14220 {
14221 m_VulkanFunctions.vkGetBufferMemoryRequirements2KHR = (PFN_vkGetBufferMemoryRequirements2)vkGetBufferMemoryRequirements2;
14222 m_VulkanFunctions.vkGetImageMemoryRequirements2KHR = (PFN_vkGetImageMemoryRequirements2)vkGetImageMemoryRequirements2;
14223 m_VulkanFunctions.vkBindBufferMemory2KHR = (PFN_vkBindBufferMemory2)vkBindBufferMemory2;
14224 m_VulkanFunctions.vkBindImageMemory2KHR = (PFN_vkBindImageMemory2)vkBindImageMemory2;
14225 m_VulkanFunctions.vkGetPhysicalDeviceMemoryProperties2KHR = (PFN_vkGetPhysicalDeviceMemoryProperties2)vkGetPhysicalDeviceMemoryProperties2;
14226 }
14227#endif
14228
14229#if VMA_VULKAN_VERSION >= 1003000
14230 if(m_VulkanApiVersion >= VK_MAKE_VERSION(1, 3, 0))
14231 {
14232 m_VulkanFunctions.vkGetDeviceBufferMemoryRequirements = (PFN_vkGetDeviceBufferMemoryRequirements)vkGetDeviceBufferMemoryRequirements;
14233 m_VulkanFunctions.vkGetDeviceImageMemoryRequirements = (PFN_vkGetDeviceImageMemoryRequirements)vkGetDeviceImageMemoryRequirements;
14234 }
14235#endif
14236}
14237
14238#endif // VMA_STATIC_VULKAN_FUNCTIONS == 1
14239
14240void VmaAllocator_T::ImportVulkanFunctions_Custom(const VmaVulkanFunctions* pVulkanFunctions)
14241{
14242 VMA_ASSERT(pVulkanFunctions != VMA_NULL);
14243
14244#define VMA_COPY_IF_NOT_NULL(funcName) \
14245 if(pVulkanFunctions->funcName != VMA_NULL) m_VulkanFunctions.funcName = pVulkanFunctions->funcName;
14246
14247 VMA_COPY_IF_NOT_NULL(vkGetInstanceProcAddr);
14248 VMA_COPY_IF_NOT_NULL(vkGetDeviceProcAddr);
14249 VMA_COPY_IF_NOT_NULL(vkGetPhysicalDeviceProperties);
14250 VMA_COPY_IF_NOT_NULL(vkGetPhysicalDeviceMemoryProperties);
14251 VMA_COPY_IF_NOT_NULL(vkAllocateMemory);
14252 VMA_COPY_IF_NOT_NULL(vkFreeMemory);
14253 VMA_COPY_IF_NOT_NULL(vkMapMemory);
14254 VMA_COPY_IF_NOT_NULL(vkUnmapMemory);
14255 VMA_COPY_IF_NOT_NULL(vkFlushMappedMemoryRanges);
14256 VMA_COPY_IF_NOT_NULL(vkInvalidateMappedMemoryRanges);
14257 VMA_COPY_IF_NOT_NULL(vkBindBufferMemory);
14258 VMA_COPY_IF_NOT_NULL(vkBindImageMemory);
14259 VMA_COPY_IF_NOT_NULL(vkGetBufferMemoryRequirements);
14260 VMA_COPY_IF_NOT_NULL(vkGetImageMemoryRequirements);
14261 VMA_COPY_IF_NOT_NULL(vkCreateBuffer);
14262 VMA_COPY_IF_NOT_NULL(vkDestroyBuffer);
14263 VMA_COPY_IF_NOT_NULL(vkCreateImage);
14264 VMA_COPY_IF_NOT_NULL(vkDestroyImage);
14265 VMA_COPY_IF_NOT_NULL(vkCmdCopyBuffer);
14266
14267#if VMA_DEDICATED_ALLOCATION || VMA_VULKAN_VERSION >= 1001000
14268 VMA_COPY_IF_NOT_NULL(vkGetBufferMemoryRequirements2KHR);
14269 VMA_COPY_IF_NOT_NULL(vkGetImageMemoryRequirements2KHR);
14270#endif
14271
14272#if VMA_BIND_MEMORY2 || VMA_VULKAN_VERSION >= 1001000
14273 VMA_COPY_IF_NOT_NULL(vkBindBufferMemory2KHR);
14274 VMA_COPY_IF_NOT_NULL(vkBindImageMemory2KHR);
14275#endif
14276
14277#if VMA_MEMORY_BUDGET
14278 VMA_COPY_IF_NOT_NULL(vkGetPhysicalDeviceMemoryProperties2KHR);
14279#endif
14280
14281#if VMA_VULKAN_VERSION >= 1003000
14282 VMA_COPY_IF_NOT_NULL(vkGetDeviceBufferMemoryRequirements);
14283 VMA_COPY_IF_NOT_NULL(vkGetDeviceImageMemoryRequirements);
14284#endif
14285
14286#undef VMA_COPY_IF_NOT_NULL
14287}
14288
14289#if VMA_DYNAMIC_VULKAN_FUNCTIONS == 1
14290
14291void VmaAllocator_T::ImportVulkanFunctions_Dynamic()
14292{
14293 VMA_ASSERT(m_VulkanFunctions.vkGetInstanceProcAddr && m_VulkanFunctions.vkGetDeviceProcAddr &&
14294 "To use VMA_DYNAMIC_VULKAN_FUNCTIONS in new versions of VMA you now have to pass "
14295 "VmaVulkanFunctions::vkGetInstanceProcAddr and vkGetDeviceProcAddr as VmaAllocatorCreateInfo::pVulkanFunctions. "
14296 "Other members can be null.");
14297
14298#define VMA_FETCH_INSTANCE_FUNC(memberName, functionPointerType, functionNameString) \
14299 if(m_VulkanFunctions.memberName == VMA_NULL) \
14300 m_VulkanFunctions.memberName = \
14301 (functionPointerType)m_VulkanFunctions.vkGetInstanceProcAddr(m_hInstance, functionNameString);
14302#define VMA_FETCH_DEVICE_FUNC(memberName, functionPointerType, functionNameString) \
14303 if(m_VulkanFunctions.memberName == VMA_NULL) \
14304 m_VulkanFunctions.memberName = \
14305 (functionPointerType)m_VulkanFunctions.vkGetDeviceProcAddr(m_hDevice, functionNameString);
14306
14307 VMA_FETCH_INSTANCE_FUNC(vkGetPhysicalDeviceProperties, PFN_vkGetPhysicalDeviceProperties, "vkGetPhysicalDeviceProperties");
14308 VMA_FETCH_INSTANCE_FUNC(vkGetPhysicalDeviceMemoryProperties, PFN_vkGetPhysicalDeviceMemoryProperties, "vkGetPhysicalDeviceMemoryProperties");
14309 VMA_FETCH_DEVICE_FUNC(vkAllocateMemory, PFN_vkAllocateMemory, "vkAllocateMemory");
14310 VMA_FETCH_DEVICE_FUNC(vkFreeMemory, PFN_vkFreeMemory, "vkFreeMemory");
14311 VMA_FETCH_DEVICE_FUNC(vkMapMemory, PFN_vkMapMemory, "vkMapMemory");
14312 VMA_FETCH_DEVICE_FUNC(vkUnmapMemory, PFN_vkUnmapMemory, "vkUnmapMemory");
14313 VMA_FETCH_DEVICE_FUNC(vkFlushMappedMemoryRanges, PFN_vkFlushMappedMemoryRanges, "vkFlushMappedMemoryRanges");
14314 VMA_FETCH_DEVICE_FUNC(vkInvalidateMappedMemoryRanges, PFN_vkInvalidateMappedMemoryRanges, "vkInvalidateMappedMemoryRanges");
14315 VMA_FETCH_DEVICE_FUNC(vkBindBufferMemory, PFN_vkBindBufferMemory, "vkBindBufferMemory");
14316 VMA_FETCH_DEVICE_FUNC(vkBindImageMemory, PFN_vkBindImageMemory, "vkBindImageMemory");
14317 VMA_FETCH_DEVICE_FUNC(vkGetBufferMemoryRequirements, PFN_vkGetBufferMemoryRequirements, "vkGetBufferMemoryRequirements");
14318 VMA_FETCH_DEVICE_FUNC(vkGetImageMemoryRequirements, PFN_vkGetImageMemoryRequirements, "vkGetImageMemoryRequirements");
14319 VMA_FETCH_DEVICE_FUNC(vkCreateBuffer, PFN_vkCreateBuffer, "vkCreateBuffer");
14320 VMA_FETCH_DEVICE_FUNC(vkDestroyBuffer, PFN_vkDestroyBuffer, "vkDestroyBuffer");
14321 VMA_FETCH_DEVICE_FUNC(vkCreateImage, PFN_vkCreateImage, "vkCreateImage");
14322 VMA_FETCH_DEVICE_FUNC(vkDestroyImage, PFN_vkDestroyImage, "vkDestroyImage");
14323 VMA_FETCH_DEVICE_FUNC(vkCmdCopyBuffer, PFN_vkCmdCopyBuffer, "vkCmdCopyBuffer");
14324
14325#if VMA_VULKAN_VERSION >= 1001000
14326 if(m_VulkanApiVersion >= VK_MAKE_VERSION(1, 1, 0))
14327 {
14328 VMA_FETCH_DEVICE_FUNC(vkGetBufferMemoryRequirements2KHR, PFN_vkGetBufferMemoryRequirements2, "vkGetBufferMemoryRequirements2");
14329 VMA_FETCH_DEVICE_FUNC(vkGetImageMemoryRequirements2KHR, PFN_vkGetImageMemoryRequirements2, "vkGetImageMemoryRequirements2");
14330 VMA_FETCH_DEVICE_FUNC(vkBindBufferMemory2KHR, PFN_vkBindBufferMemory2, "vkBindBufferMemory2");
14331 VMA_FETCH_DEVICE_FUNC(vkBindImageMemory2KHR, PFN_vkBindImageMemory2, "vkBindImageMemory2");
14332 VMA_FETCH_INSTANCE_FUNC(vkGetPhysicalDeviceMemoryProperties2KHR, PFN_vkGetPhysicalDeviceMemoryProperties2, "vkGetPhysicalDeviceMemoryProperties2");
14333 }
14334#endif
14335
14336#if VMA_DEDICATED_ALLOCATION
14337 if(m_UseKhrDedicatedAllocation)
14338 {
14339 VMA_FETCH_DEVICE_FUNC(vkGetBufferMemoryRequirements2KHR, PFN_vkGetBufferMemoryRequirements2KHR, "vkGetBufferMemoryRequirements2KHR");
14340 VMA_FETCH_DEVICE_FUNC(vkGetImageMemoryRequirements2KHR, PFN_vkGetImageMemoryRequirements2KHR, "vkGetImageMemoryRequirements2KHR");
14341 }
14342#endif
14343
14344#if VMA_BIND_MEMORY2
14345 if(m_UseKhrBindMemory2)
14346 {
14347 VMA_FETCH_DEVICE_FUNC(vkBindBufferMemory2KHR, PFN_vkBindBufferMemory2KHR, "vkBindBufferMemory2KHR");
14348 VMA_FETCH_DEVICE_FUNC(vkBindImageMemory2KHR, PFN_vkBindImageMemory2KHR, "vkBindImageMemory2KHR");
14349 }
14350#endif // #if VMA_BIND_MEMORY2
14351
14352#if VMA_MEMORY_BUDGET
14353 if(m_UseExtMemoryBudget)
14354 {
14355 VMA_FETCH_INSTANCE_FUNC(vkGetPhysicalDeviceMemoryProperties2KHR, PFN_vkGetPhysicalDeviceMemoryProperties2KHR, "vkGetPhysicalDeviceMemoryProperties2KHR");
14356 }
14357#endif // #if VMA_MEMORY_BUDGET
14358
14359#if VMA_VULKAN_VERSION >= 1003000
14360 if(m_VulkanApiVersion >= VK_MAKE_VERSION(1, 3, 0))
14361 {
14362 VMA_FETCH_DEVICE_FUNC(vkGetDeviceBufferMemoryRequirements, PFN_vkGetDeviceBufferMemoryRequirements, "vkGetDeviceBufferMemoryRequirements");
14363 VMA_FETCH_DEVICE_FUNC(vkGetDeviceImageMemoryRequirements, PFN_vkGetDeviceImageMemoryRequirements, "vkGetDeviceImageMemoryRequirements");
14364 }
14365#endif
14366
14367#undef VMA_FETCH_DEVICE_FUNC
14368#undef VMA_FETCH_INSTANCE_FUNC
14369}
14370
14371#endif // VMA_DYNAMIC_VULKAN_FUNCTIONS == 1
14372
14373void VmaAllocator_T::ValidateVulkanFunctions()
14374{
14375 VMA_ASSERT(m_VulkanFunctions.vkGetPhysicalDeviceProperties != VMA_NULL);
14376 VMA_ASSERT(m_VulkanFunctions.vkGetPhysicalDeviceMemoryProperties != VMA_NULL);
14377 VMA_ASSERT(m_VulkanFunctions.vkAllocateMemory != VMA_NULL);
14378 VMA_ASSERT(m_VulkanFunctions.vkFreeMemory != VMA_NULL);
14379 VMA_ASSERT(m_VulkanFunctions.vkMapMemory != VMA_NULL);
14380 VMA_ASSERT(m_VulkanFunctions.vkUnmapMemory != VMA_NULL);
14381 VMA_ASSERT(m_VulkanFunctions.vkFlushMappedMemoryRanges != VMA_NULL);
14382 VMA_ASSERT(m_VulkanFunctions.vkInvalidateMappedMemoryRanges != VMA_NULL);
14383 VMA_ASSERT(m_VulkanFunctions.vkBindBufferMemory != VMA_NULL);
14384 VMA_ASSERT(m_VulkanFunctions.vkBindImageMemory != VMA_NULL);
14385 VMA_ASSERT(m_VulkanFunctions.vkGetBufferMemoryRequirements != VMA_NULL);
14386 VMA_ASSERT(m_VulkanFunctions.vkGetImageMemoryRequirements != VMA_NULL);
14387 VMA_ASSERT(m_VulkanFunctions.vkCreateBuffer != VMA_NULL);
14388 VMA_ASSERT(m_VulkanFunctions.vkDestroyBuffer != VMA_NULL);
14389 VMA_ASSERT(m_VulkanFunctions.vkCreateImage != VMA_NULL);
14390 VMA_ASSERT(m_VulkanFunctions.vkDestroyImage != VMA_NULL);
14391 VMA_ASSERT(m_VulkanFunctions.vkCmdCopyBuffer != VMA_NULL);
14392
14393#if VMA_DEDICATED_ALLOCATION || VMA_VULKAN_VERSION >= 1001000
14394 if(m_VulkanApiVersion >= VK_MAKE_VERSION(1, 1, 0) || m_UseKhrDedicatedAllocation)
14395 {
14396 VMA_ASSERT(m_VulkanFunctions.vkGetBufferMemoryRequirements2KHR != VMA_NULL);
14397 VMA_ASSERT(m_VulkanFunctions.vkGetImageMemoryRequirements2KHR != VMA_NULL);
14398 }
14399#endif
14400
14401#if VMA_BIND_MEMORY2 || VMA_VULKAN_VERSION >= 1001000
14402 if(m_VulkanApiVersion >= VK_MAKE_VERSION(1, 1, 0) || m_UseKhrBindMemory2)
14403 {
14404 VMA_ASSERT(m_VulkanFunctions.vkBindBufferMemory2KHR != VMA_NULL);
14405 VMA_ASSERT(m_VulkanFunctions.vkBindImageMemory2KHR != VMA_NULL);
14406 }
14407#endif
14408
14409#if VMA_MEMORY_BUDGET || VMA_VULKAN_VERSION >= 1001000
14410 if(m_UseExtMemoryBudget || m_VulkanApiVersion >= VK_MAKE_VERSION(1, 1, 0))
14411 {
14412 VMA_ASSERT(m_VulkanFunctions.vkGetPhysicalDeviceMemoryProperties2KHR != VMA_NULL);
14413 }
14414#endif
14415
14416#if VMA_VULKAN_VERSION >= 1003000
14417 if(m_VulkanApiVersion >= VK_MAKE_VERSION(1, 3, 0))
14418 {
14419 VMA_ASSERT(m_VulkanFunctions.vkGetDeviceBufferMemoryRequirements != VMA_NULL);
14420 VMA_ASSERT(m_VulkanFunctions.vkGetDeviceImageMemoryRequirements != VMA_NULL);
14421 }
14422#endif
14423}
14424
14425VkDeviceSize VmaAllocator_T::CalcPreferredBlockSize(uint32_t memTypeIndex)
14426{
14427 const uint32_t heapIndex = MemoryTypeIndexToHeapIndex(memTypeIndex);
14428 const VkDeviceSize heapSize = m_MemProps.memoryHeaps[heapIndex].size;
14429 const bool isSmallHeap = heapSize <= VMA_SMALL_HEAP_MAX_SIZE;
14430 return VmaAlignUp(val: isSmallHeap ? (heapSize / 8) : m_PreferredLargeHeapBlockSize, alignment: (VkDeviceSize)32);
14431}
14432
14433VkResult VmaAllocator_T::AllocateMemoryOfType(
14434 VmaPool pool,
14435 VkDeviceSize size,
14436 VkDeviceSize alignment,
14437 bool dedicatedPreferred,
14438 VkBuffer dedicatedBuffer,
14439 VkImage dedicatedImage,
14440 VkFlags dedicatedBufferImageUsage,
14441 const VmaAllocationCreateInfo& createInfo,
14442 uint32_t memTypeIndex,
14443 VmaSuballocationType suballocType,
14444 VmaDedicatedAllocationList& dedicatedAllocations,
14445 VmaBlockVector& blockVector,
14446 size_t allocationCount,
14447 VmaAllocation* pAllocations)
14448{
14449 VMA_ASSERT(pAllocations != VMA_NULL);
14450 VMA_DEBUG_LOG(" AllocateMemory: MemoryTypeIndex=%u, AllocationCount=%zu, Size=%llu", memTypeIndex, allocationCount, size);
14451
14452 VmaAllocationCreateInfo finalCreateInfo = createInfo;
14453 VkResult res = CalcMemTypeParams(
14454 outCreateInfo&: finalCreateInfo,
14455 memTypeIndex,
14456 size,
14457 allocationCount);
14458 if(res != VK_SUCCESS)
14459 return res;
14460
14461 if((finalCreateInfo.flags & VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT) != 0)
14462 {
14463 return AllocateDedicatedMemory(
14464 pool,
14465 size,
14466 suballocType,
14467 dedicatedAllocations,
14468 memTypeIndex,
14469 map: (finalCreateInfo.flags & VMA_ALLOCATION_CREATE_MAPPED_BIT) != 0,
14470 isUserDataString: (finalCreateInfo.flags & VMA_ALLOCATION_CREATE_USER_DATA_COPY_STRING_BIT) != 0,
14471 isMappingAllowed: (finalCreateInfo.flags &
14472 (VMA_ALLOCATION_CREATE_HOST_ACCESS_SEQUENTIAL_WRITE_BIT | VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT)) != 0,
14473 canAliasMemory: (finalCreateInfo.flags & VMA_ALLOCATION_CREATE_CAN_ALIAS_BIT) != 0,
14474 pUserData: finalCreateInfo.pUserData,
14475 priority: finalCreateInfo.priority,
14476 dedicatedBuffer,
14477 dedicatedImage,
14478 dedicatedBufferImageUsage,
14479 allocationCount,
14480 pAllocations,
14481 pNextChain: blockVector.GetAllocationNextPtr());
14482 }
14483 else
14484 {
14485 const bool canAllocateDedicated =
14486 (finalCreateInfo.flags & VMA_ALLOCATION_CREATE_NEVER_ALLOCATE_BIT) == 0 &&
14487 (pool == VK_NULL_HANDLE || !blockVector.HasExplicitBlockSize());
14488
14489 if(canAllocateDedicated)
14490 {
14491 // Heuristics: Allocate dedicated memory if requested size if greater than half of preferred block size.
14492 if(size > blockVector.GetPreferredBlockSize() / 2)
14493 {
14494 dedicatedPreferred = true;
14495 }
14496 // Protection against creating each allocation as dedicated when we reach or exceed heap size/budget,
14497 // which can quickly deplete maxMemoryAllocationCount: Don't prefer dedicated allocations when above
14498 // 3/4 of the maximum allocation count.
14499 if(m_DeviceMemoryCount.load() > m_PhysicalDeviceProperties.limits.maxMemoryAllocationCount * 3 / 4)
14500 {
14501 dedicatedPreferred = false;
14502 }
14503
14504 if(dedicatedPreferred)
14505 {
14506 res = AllocateDedicatedMemory(
14507 pool,
14508 size,
14509 suballocType,
14510 dedicatedAllocations,
14511 memTypeIndex,
14512 map: (finalCreateInfo.flags & VMA_ALLOCATION_CREATE_MAPPED_BIT) != 0,
14513 isUserDataString: (finalCreateInfo.flags & VMA_ALLOCATION_CREATE_USER_DATA_COPY_STRING_BIT) != 0,
14514 isMappingAllowed: (finalCreateInfo.flags &
14515 (VMA_ALLOCATION_CREATE_HOST_ACCESS_SEQUENTIAL_WRITE_BIT | VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT)) != 0,
14516 canAliasMemory: (finalCreateInfo.flags & VMA_ALLOCATION_CREATE_CAN_ALIAS_BIT) != 0,
14517 pUserData: finalCreateInfo.pUserData,
14518 priority: finalCreateInfo.priority,
14519 dedicatedBuffer,
14520 dedicatedImage,
14521 dedicatedBufferImageUsage,
14522 allocationCount,
14523 pAllocations,
14524 pNextChain: blockVector.GetAllocationNextPtr());
14525 if(res == VK_SUCCESS)
14526 {
14527 // Succeeded: AllocateDedicatedMemory function already filld pMemory, nothing more to do here.
14528 VMA_DEBUG_LOG(" Allocated as DedicatedMemory");
14529 return VK_SUCCESS;
14530 }
14531 }
14532 }
14533
14534 res = blockVector.Allocate(
14535 size,
14536 alignment,
14537 createInfo: finalCreateInfo,
14538 suballocType,
14539 allocationCount,
14540 pAllocations);
14541 if(res == VK_SUCCESS)
14542 return VK_SUCCESS;
14543
14544 // Try dedicated memory.
14545 if(canAllocateDedicated && !dedicatedPreferred)
14546 {
14547 res = AllocateDedicatedMemory(
14548 pool,
14549 size,
14550 suballocType,
14551 dedicatedAllocations,
14552 memTypeIndex,
14553 map: (finalCreateInfo.flags & VMA_ALLOCATION_CREATE_MAPPED_BIT) != 0,
14554 isUserDataString: (finalCreateInfo.flags & VMA_ALLOCATION_CREATE_USER_DATA_COPY_STRING_BIT) != 0,
14555 isMappingAllowed: (finalCreateInfo.flags &
14556 (VMA_ALLOCATION_CREATE_HOST_ACCESS_SEQUENTIAL_WRITE_BIT | VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT)) != 0,
14557 canAliasMemory: (finalCreateInfo.flags & VMA_ALLOCATION_CREATE_CAN_ALIAS_BIT) != 0,
14558 pUserData: finalCreateInfo.pUserData,
14559 priority: finalCreateInfo.priority,
14560 dedicatedBuffer,
14561 dedicatedImage,
14562 dedicatedBufferImageUsage,
14563 allocationCount,
14564 pAllocations,
14565 pNextChain: blockVector.GetAllocationNextPtr());
14566 if(res == VK_SUCCESS)
14567 {
14568 // Succeeded: AllocateDedicatedMemory function already filld pMemory, nothing more to do here.
14569 VMA_DEBUG_LOG(" Allocated as DedicatedMemory");
14570 return VK_SUCCESS;
14571 }
14572 }
14573 // Everything failed: Return error code.
14574 VMA_DEBUG_LOG(" vkAllocateMemory FAILED");
14575 return res;
14576 }
14577}
14578
14579VkResult VmaAllocator_T::AllocateDedicatedMemory(
14580 VmaPool pool,
14581 VkDeviceSize size,
14582 VmaSuballocationType suballocType,
14583 VmaDedicatedAllocationList& dedicatedAllocations,
14584 uint32_t memTypeIndex,
14585 bool map,
14586 bool isUserDataString,
14587 bool isMappingAllowed,
14588 bool canAliasMemory,
14589 void* pUserData,
14590 float priority,
14591 VkBuffer dedicatedBuffer,
14592 VkImage dedicatedImage,
14593 VkFlags dedicatedBufferImageUsage,
14594 size_t allocationCount,
14595 VmaAllocation* pAllocations,
14596 const void* pNextChain)
14597{
14598 VMA_ASSERT(allocationCount > 0 && pAllocations);
14599
14600 VkMemoryAllocateInfo allocInfo = { .sType: VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO };
14601 allocInfo.memoryTypeIndex = memTypeIndex;
14602 allocInfo.allocationSize = size;
14603 allocInfo.pNext = pNextChain;
14604
14605#if VMA_DEDICATED_ALLOCATION || VMA_VULKAN_VERSION >= 1001000
14606 VkMemoryDedicatedAllocateInfoKHR dedicatedAllocInfo = { .sType: VK_STRUCTURE_TYPE_MEMORY_DEDICATED_ALLOCATE_INFO_KHR };
14607 if(!canAliasMemory)
14608 {
14609 if(m_UseKhrDedicatedAllocation || m_VulkanApiVersion >= VK_MAKE_VERSION(1, 1, 0))
14610 {
14611 if(dedicatedBuffer != VK_NULL_HANDLE)
14612 {
14613 VMA_ASSERT(dedicatedImage == VK_NULL_HANDLE);
14614 dedicatedAllocInfo.buffer = dedicatedBuffer;
14615 VmaPnextChainPushFront(mainStruct: &allocInfo, newStruct: &dedicatedAllocInfo);
14616 }
14617 else if(dedicatedImage != VK_NULL_HANDLE)
14618 {
14619 dedicatedAllocInfo.image = dedicatedImage;
14620 VmaPnextChainPushFront(mainStruct: &allocInfo, newStruct: &dedicatedAllocInfo);
14621 }
14622 }
14623 }
14624#endif // #if VMA_DEDICATED_ALLOCATION || VMA_VULKAN_VERSION >= 1001000
14625
14626#if VMA_BUFFER_DEVICE_ADDRESS
14627 VkMemoryAllocateFlagsInfoKHR allocFlagsInfo = { .sType: VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_FLAGS_INFO_KHR };
14628 if(m_UseKhrBufferDeviceAddress)
14629 {
14630 bool canContainBufferWithDeviceAddress = true;
14631 if(dedicatedBuffer != VK_NULL_HANDLE)
14632 {
14633 canContainBufferWithDeviceAddress = dedicatedBufferImageUsage == UINT32_MAX || // Usage flags unknown
14634 (dedicatedBufferImageUsage & VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT_EXT) != 0;
14635 }
14636 else if(dedicatedImage != VK_NULL_HANDLE)
14637 {
14638 canContainBufferWithDeviceAddress = false;
14639 }
14640 if(canContainBufferWithDeviceAddress)
14641 {
14642 allocFlagsInfo.flags = VK_MEMORY_ALLOCATE_DEVICE_ADDRESS_BIT_KHR;
14643 VmaPnextChainPushFront(mainStruct: &allocInfo, newStruct: &allocFlagsInfo);
14644 }
14645 }
14646#endif // #if VMA_BUFFER_DEVICE_ADDRESS
14647
14648#if VMA_MEMORY_PRIORITY
14649 VkMemoryPriorityAllocateInfoEXT priorityInfo = { .sType: VK_STRUCTURE_TYPE_MEMORY_PRIORITY_ALLOCATE_INFO_EXT };
14650 if(m_UseExtMemoryPriority)
14651 {
14652 VMA_ASSERT(priority >= 0.f && priority <= 1.f);
14653 priorityInfo.priority = priority;
14654 VmaPnextChainPushFront(mainStruct: &allocInfo, newStruct: &priorityInfo);
14655 }
14656#endif // #if VMA_MEMORY_PRIORITY
14657
14658#if VMA_EXTERNAL_MEMORY
14659 // Attach VkExportMemoryAllocateInfoKHR if necessary.
14660 VkExportMemoryAllocateInfoKHR exportMemoryAllocInfo = { .sType: VK_STRUCTURE_TYPE_EXPORT_MEMORY_ALLOCATE_INFO_KHR };
14661 exportMemoryAllocInfo.handleTypes = GetExternalMemoryHandleTypeFlags(memTypeIndex);
14662 if(exportMemoryAllocInfo.handleTypes != 0)
14663 {
14664 VmaPnextChainPushFront(mainStruct: &allocInfo, newStruct: &exportMemoryAllocInfo);
14665 }
14666#endif // #if VMA_EXTERNAL_MEMORY
14667
14668 size_t allocIndex;
14669 VkResult res = VK_SUCCESS;
14670 for(allocIndex = 0; allocIndex < allocationCount; ++allocIndex)
14671 {
14672 res = AllocateDedicatedMemoryPage(
14673 pool,
14674 size,
14675 suballocType,
14676 memTypeIndex,
14677 allocInfo,
14678 map,
14679 isUserDataString,
14680 isMappingAllowed,
14681 pUserData,
14682 pAllocation: pAllocations + allocIndex);
14683 if(res != VK_SUCCESS)
14684 {
14685 break;
14686 }
14687 }
14688
14689 if(res == VK_SUCCESS)
14690 {
14691 for (allocIndex = 0; allocIndex < allocationCount; ++allocIndex)
14692 {
14693 dedicatedAllocations.Register(alloc: pAllocations[allocIndex]);
14694 }
14695 VMA_DEBUG_LOG(" Allocated DedicatedMemory Count=%zu, MemoryTypeIndex=#%u", allocationCount, memTypeIndex);
14696 }
14697 else
14698 {
14699 // Free all already created allocations.
14700 while(allocIndex--)
14701 {
14702 VmaAllocation currAlloc = pAllocations[allocIndex];
14703 VkDeviceMemory hMemory = currAlloc->GetMemory();
14704
14705 /*
14706 There is no need to call this, because Vulkan spec allows to skip vkUnmapMemory
14707 before vkFreeMemory.
14708
14709 if(currAlloc->GetMappedData() != VMA_NULL)
14710 {
14711 (*m_VulkanFunctions.vkUnmapMemory)(m_hDevice, hMemory);
14712 }
14713 */
14714
14715 FreeVulkanMemory(memoryType: memTypeIndex, size: currAlloc->GetSize(), hMemory);
14716 m_Budget.RemoveAllocation(heapIndex: MemoryTypeIndexToHeapIndex(memTypeIndex), allocationSize: currAlloc->GetSize());
14717 m_AllocationObjectAllocator.Free(hAlloc: currAlloc);
14718 }
14719
14720 memset(s: pAllocations, c: 0, n: sizeof(VmaAllocation) * allocationCount);
14721 }
14722
14723 return res;
14724}
14725
14726VkResult VmaAllocator_T::AllocateDedicatedMemoryPage(
14727 VmaPool pool,
14728 VkDeviceSize size,
14729 VmaSuballocationType suballocType,
14730 uint32_t memTypeIndex,
14731 const VkMemoryAllocateInfo& allocInfo,
14732 bool map,
14733 bool isUserDataString,
14734 bool isMappingAllowed,
14735 void* pUserData,
14736 VmaAllocation* pAllocation)
14737{
14738 VkDeviceMemory hMemory = VK_NULL_HANDLE;
14739 VkResult res = AllocateVulkanMemory(pAllocateInfo: &allocInfo, pMemory: &hMemory);
14740 if(res < 0)
14741 {
14742 VMA_DEBUG_LOG(" vkAllocateMemory FAILED");
14743 return res;
14744 }
14745
14746 void* pMappedData = VMA_NULL;
14747 if(map)
14748 {
14749 res = (*m_VulkanFunctions.vkMapMemory)(
14750 m_hDevice,
14751 hMemory,
14752 0,
14753 VK_WHOLE_SIZE,
14754 0,
14755 &pMappedData);
14756 if(res < 0)
14757 {
14758 VMA_DEBUG_LOG(" vkMapMemory FAILED");
14759 FreeVulkanMemory(memoryType: memTypeIndex, size, hMemory);
14760 return res;
14761 }
14762 }
14763
14764 *pAllocation = m_AllocationObjectAllocator.Allocate(args&: isMappingAllowed);
14765 (*pAllocation)->InitDedicatedAllocation(hParentPool: pool, memoryTypeIndex: memTypeIndex, hMemory, suballocationType: suballocType, pMappedData, size);
14766 if (isUserDataString)
14767 (*pAllocation)->SetName(hAllocator: this, pName: (const char*)pUserData);
14768 else
14769 (*pAllocation)->SetUserData(hAllocator: this, pUserData);
14770 m_Budget.AddAllocation(heapIndex: MemoryTypeIndexToHeapIndex(memTypeIndex), allocationSize: size);
14771 if(VMA_DEBUG_INITIALIZE_ALLOCATIONS)
14772 {
14773 FillAllocation(hAllocation: *pAllocation, pattern: VMA_ALLOCATION_FILL_PATTERN_CREATED);
14774 }
14775
14776 return VK_SUCCESS;
14777}
14778
14779void VmaAllocator_T::GetBufferMemoryRequirements(
14780 VkBuffer hBuffer,
14781 VkMemoryRequirements& memReq,
14782 bool& requiresDedicatedAllocation,
14783 bool& prefersDedicatedAllocation) const
14784{
14785#if VMA_DEDICATED_ALLOCATION || VMA_VULKAN_VERSION >= 1001000
14786 if(m_UseKhrDedicatedAllocation || m_VulkanApiVersion >= VK_MAKE_VERSION(1, 1, 0))
14787 {
14788 VkBufferMemoryRequirementsInfo2KHR memReqInfo = { .sType: VK_STRUCTURE_TYPE_BUFFER_MEMORY_REQUIREMENTS_INFO_2_KHR };
14789 memReqInfo.buffer = hBuffer;
14790
14791 VkMemoryDedicatedRequirementsKHR memDedicatedReq = { .sType: VK_STRUCTURE_TYPE_MEMORY_DEDICATED_REQUIREMENTS_KHR };
14792
14793 VkMemoryRequirements2KHR memReq2 = { .sType: VK_STRUCTURE_TYPE_MEMORY_REQUIREMENTS_2_KHR };
14794 VmaPnextChainPushFront(mainStruct: &memReq2, newStruct: &memDedicatedReq);
14795
14796 (*m_VulkanFunctions.vkGetBufferMemoryRequirements2KHR)(m_hDevice, &memReqInfo, &memReq2);
14797
14798 memReq = memReq2.memoryRequirements;
14799 requiresDedicatedAllocation = (memDedicatedReq.requiresDedicatedAllocation != VK_FALSE);
14800 prefersDedicatedAllocation = (memDedicatedReq.prefersDedicatedAllocation != VK_FALSE);
14801 }
14802 else
14803#endif // #if VMA_DEDICATED_ALLOCATION || VMA_VULKAN_VERSION >= 1001000
14804 {
14805 (*m_VulkanFunctions.vkGetBufferMemoryRequirements)(m_hDevice, hBuffer, &memReq);
14806 requiresDedicatedAllocation = false;
14807 prefersDedicatedAllocation = false;
14808 }
14809}
14810
14811void VmaAllocator_T::GetImageMemoryRequirements(
14812 VkImage hImage,
14813 VkMemoryRequirements& memReq,
14814 bool& requiresDedicatedAllocation,
14815 bool& prefersDedicatedAllocation) const
14816{
14817#if VMA_DEDICATED_ALLOCATION || VMA_VULKAN_VERSION >= 1001000
14818 if(m_UseKhrDedicatedAllocation || m_VulkanApiVersion >= VK_MAKE_VERSION(1, 1, 0))
14819 {
14820 VkImageMemoryRequirementsInfo2KHR memReqInfo = { .sType: VK_STRUCTURE_TYPE_IMAGE_MEMORY_REQUIREMENTS_INFO_2_KHR };
14821 memReqInfo.image = hImage;
14822
14823 VkMemoryDedicatedRequirementsKHR memDedicatedReq = { .sType: VK_STRUCTURE_TYPE_MEMORY_DEDICATED_REQUIREMENTS_KHR };
14824
14825 VkMemoryRequirements2KHR memReq2 = { .sType: VK_STRUCTURE_TYPE_MEMORY_REQUIREMENTS_2_KHR };
14826 VmaPnextChainPushFront(mainStruct: &memReq2, newStruct: &memDedicatedReq);
14827
14828 (*m_VulkanFunctions.vkGetImageMemoryRequirements2KHR)(m_hDevice, &memReqInfo, &memReq2);
14829
14830 memReq = memReq2.memoryRequirements;
14831 requiresDedicatedAllocation = (memDedicatedReq.requiresDedicatedAllocation != VK_FALSE);
14832 prefersDedicatedAllocation = (memDedicatedReq.prefersDedicatedAllocation != VK_FALSE);
14833 }
14834 else
14835#endif // #if VMA_DEDICATED_ALLOCATION || VMA_VULKAN_VERSION >= 1001000
14836 {
14837 (*m_VulkanFunctions.vkGetImageMemoryRequirements)(m_hDevice, hImage, &memReq);
14838 requiresDedicatedAllocation = false;
14839 prefersDedicatedAllocation = false;
14840 }
14841}
14842
14843VkResult VmaAllocator_T::FindMemoryTypeIndex(
14844 uint32_t memoryTypeBits,
14845 const VmaAllocationCreateInfo* pAllocationCreateInfo,
14846 VkFlags bufImgUsage,
14847 uint32_t* pMemoryTypeIndex) const
14848{
14849 memoryTypeBits &= GetGlobalMemoryTypeBits();
14850
14851 if(pAllocationCreateInfo->memoryTypeBits != 0)
14852 {
14853 memoryTypeBits &= pAllocationCreateInfo->memoryTypeBits;
14854 }
14855
14856 VkMemoryPropertyFlags requiredFlags = 0, preferredFlags = 0, notPreferredFlags = 0;
14857 if(!FindMemoryPreferences(
14858 isIntegratedGPU: IsIntegratedGpu(),
14859 allocCreateInfo: *pAllocationCreateInfo,
14860 bufImgUsage,
14861 outRequiredFlags&: requiredFlags, outPreferredFlags&: preferredFlags, outNotPreferredFlags&: notPreferredFlags))
14862 {
14863 return VK_ERROR_FEATURE_NOT_PRESENT;
14864 }
14865
14866 *pMemoryTypeIndex = UINT32_MAX;
14867 uint32_t minCost = UINT32_MAX;
14868 for(uint32_t memTypeIndex = 0, memTypeBit = 1;
14869 memTypeIndex < GetMemoryTypeCount();
14870 ++memTypeIndex, memTypeBit <<= 1)
14871 {
14872 // This memory type is acceptable according to memoryTypeBits bitmask.
14873 if((memTypeBit & memoryTypeBits) != 0)
14874 {
14875 const VkMemoryPropertyFlags currFlags =
14876 m_MemProps.memoryTypes[memTypeIndex].propertyFlags;
14877 // This memory type contains requiredFlags.
14878 if((requiredFlags & ~currFlags) == 0)
14879 {
14880 // Calculate cost as number of bits from preferredFlags not present in this memory type.
14881 uint32_t currCost = VMA_COUNT_BITS_SET(preferredFlags & ~currFlags) +
14882 VMA_COUNT_BITS_SET(currFlags & notPreferredFlags);
14883 // Remember memory type with lowest cost.
14884 if(currCost < minCost)
14885 {
14886 *pMemoryTypeIndex = memTypeIndex;
14887 if(currCost == 0)
14888 {
14889 return VK_SUCCESS;
14890 }
14891 minCost = currCost;
14892 }
14893 }
14894 }
14895 }
14896 return (*pMemoryTypeIndex != UINT32_MAX) ? VK_SUCCESS : VK_ERROR_FEATURE_NOT_PRESENT;
14897}
14898
14899VkResult VmaAllocator_T::CalcMemTypeParams(
14900 VmaAllocationCreateInfo& inoutCreateInfo,
14901 uint32_t memTypeIndex,
14902 VkDeviceSize size,
14903 size_t allocationCount)
14904{
14905 // If memory type is not HOST_VISIBLE, disable MAPPED.
14906 if((inoutCreateInfo.flags & VMA_ALLOCATION_CREATE_MAPPED_BIT) != 0 &&
14907 (m_MemProps.memoryTypes[memTypeIndex].propertyFlags & VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT) == 0)
14908 {
14909 inoutCreateInfo.flags &= ~VMA_ALLOCATION_CREATE_MAPPED_BIT;
14910 }
14911
14912 if((inoutCreateInfo.flags & VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT) != 0 &&
14913 (inoutCreateInfo.flags & VMA_ALLOCATION_CREATE_WITHIN_BUDGET_BIT) != 0)
14914 {
14915 const uint32_t heapIndex = MemoryTypeIndexToHeapIndex(memTypeIndex);
14916 VmaBudget heapBudget = {};
14917 GetHeapBudgets(outBudgets: &heapBudget, firstHeap: heapIndex, heapCount: 1);
14918 if(heapBudget.usage + size * allocationCount > heapBudget.budget)
14919 {
14920 return VK_ERROR_OUT_OF_DEVICE_MEMORY;
14921 }
14922 }
14923 return VK_SUCCESS;
14924}
14925
14926VkResult VmaAllocator_T::CalcAllocationParams(
14927 VmaAllocationCreateInfo& inoutCreateInfo,
14928 bool dedicatedRequired,
14929 bool dedicatedPreferred)
14930{
14931 VMA_ASSERT((inoutCreateInfo.flags &
14932 (VMA_ALLOCATION_CREATE_HOST_ACCESS_SEQUENTIAL_WRITE_BIT | VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT)) !=
14933 (VMA_ALLOCATION_CREATE_HOST_ACCESS_SEQUENTIAL_WRITE_BIT | VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT) &&
14934 "Specifying both flags VMA_ALLOCATION_CREATE_HOST_ACCESS_SEQUENTIAL_WRITE_BIT and VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT is incorrect.");
14935 VMA_ASSERT((((inoutCreateInfo.flags & VMA_ALLOCATION_CREATE_HOST_ACCESS_ALLOW_TRANSFER_INSTEAD_BIT) == 0 ||
14936 (inoutCreateInfo.flags & (VMA_ALLOCATION_CREATE_HOST_ACCESS_SEQUENTIAL_WRITE_BIT | VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT)) != 0)) &&
14937 "Specifying VMA_ALLOCATION_CREATE_HOST_ACCESS_ALLOW_TRANSFER_INSTEAD_BIT requires also VMA_ALLOCATION_CREATE_HOST_ACCESS_SEQUENTIAL_WRITE_BIT or VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT.");
14938 if(inoutCreateInfo.usage == VMA_MEMORY_USAGE_AUTO || inoutCreateInfo.usage == VMA_MEMORY_USAGE_AUTO_PREFER_DEVICE || inoutCreateInfo.usage == VMA_MEMORY_USAGE_AUTO_PREFER_HOST)
14939 {
14940 if((inoutCreateInfo.flags & VMA_ALLOCATION_CREATE_MAPPED_BIT) != 0)
14941 {
14942 VMA_ASSERT((inoutCreateInfo.flags & (VMA_ALLOCATION_CREATE_HOST_ACCESS_SEQUENTIAL_WRITE_BIT | VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT)) != 0 &&
14943 "When using VMA_ALLOCATION_CREATE_MAPPED_BIT and usage = VMA_MEMORY_USAGE_AUTO*, you must also specify VMA_ALLOCATION_CREATE_HOST_ACCESS_SEQUENTIAL_WRITE_BIT or VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT.");
14944 }
14945 }
14946
14947 // If memory is lazily allocated, it should be always dedicated.
14948 if(dedicatedRequired ||
14949 inoutCreateInfo.usage == VMA_MEMORY_USAGE_GPU_LAZILY_ALLOCATED)
14950 {
14951 inoutCreateInfo.flags |= VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT;
14952 }
14953
14954 if(inoutCreateInfo.pool != VK_NULL_HANDLE)
14955 {
14956 if(inoutCreateInfo.pool->m_BlockVector.HasExplicitBlockSize() &&
14957 (inoutCreateInfo.flags & VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT) != 0)
14958 {
14959 VMA_ASSERT(0 && "Specifying VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT while current custom pool doesn't support dedicated allocations.");
14960 return VK_ERROR_FEATURE_NOT_PRESENT;
14961 }
14962 inoutCreateInfo.priority = inoutCreateInfo.pool->m_BlockVector.GetPriority();
14963 }
14964
14965 if((inoutCreateInfo.flags & VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT) != 0 &&
14966 (inoutCreateInfo.flags & VMA_ALLOCATION_CREATE_NEVER_ALLOCATE_BIT) != 0)
14967 {
14968 VMA_ASSERT(0 && "Specifying VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT together with VMA_ALLOCATION_CREATE_NEVER_ALLOCATE_BIT makes no sense.");
14969 return VK_ERROR_FEATURE_NOT_PRESENT;
14970 }
14971
14972 if(VMA_DEBUG_ALWAYS_DEDICATED_MEMORY &&
14973 (inoutCreateInfo.flags & VMA_ALLOCATION_CREATE_NEVER_ALLOCATE_BIT) != 0)
14974 {
14975 inoutCreateInfo.flags |= VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT;
14976 }
14977
14978 // Non-auto USAGE values imply HOST_ACCESS flags.
14979 // And so does VMA_MEMORY_USAGE_UNKNOWN because it is used with custom pools.
14980 // Which specific flag is used doesn't matter. They change things only when used with VMA_MEMORY_USAGE_AUTO*.
14981 // Otherwise they just protect from assert on mapping.
14982 if(inoutCreateInfo.usage != VMA_MEMORY_USAGE_AUTO &&
14983 inoutCreateInfo.usage != VMA_MEMORY_USAGE_AUTO_PREFER_DEVICE &&
14984 inoutCreateInfo.usage != VMA_MEMORY_USAGE_AUTO_PREFER_HOST)
14985 {
14986 if((inoutCreateInfo.flags & (VMA_ALLOCATION_CREATE_HOST_ACCESS_SEQUENTIAL_WRITE_BIT | VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT)) == 0)
14987 {
14988 inoutCreateInfo.flags |= VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT;
14989 }
14990 }
14991
14992 return VK_SUCCESS;
14993}
14994
14995VkResult VmaAllocator_T::AllocateMemory(
14996 const VkMemoryRequirements& vkMemReq,
14997 bool requiresDedicatedAllocation,
14998 bool prefersDedicatedAllocation,
14999 VkBuffer dedicatedBuffer,
15000 VkImage dedicatedImage,
15001 VkFlags dedicatedBufferImageUsage,
15002 const VmaAllocationCreateInfo& createInfo,
15003 VmaSuballocationType suballocType,
15004 size_t allocationCount,
15005 VmaAllocation* pAllocations)
15006{
15007 memset(s: pAllocations, c: 0, n: sizeof(VmaAllocation) * allocationCount);
15008
15009 VMA_ASSERT(VmaIsPow2(vkMemReq.alignment));
15010
15011 if(vkMemReq.size == 0)
15012 {
15013 return VK_ERROR_INITIALIZATION_FAILED;
15014 }
15015
15016 VmaAllocationCreateInfo createInfoFinal = createInfo;
15017 VkResult res = CalcAllocationParams(inoutCreateInfo&: createInfoFinal, dedicatedRequired: requiresDedicatedAllocation, dedicatedPreferred: prefersDedicatedAllocation);
15018 if(res != VK_SUCCESS)
15019 return res;
15020
15021 if(createInfoFinal.pool != VK_NULL_HANDLE)
15022 {
15023 VmaBlockVector& blockVector = createInfoFinal.pool->m_BlockVector;
15024 return AllocateMemoryOfType(
15025 pool: createInfoFinal.pool,
15026 size: vkMemReq.size,
15027 alignment: vkMemReq.alignment,
15028 dedicatedPreferred: prefersDedicatedAllocation,
15029 dedicatedBuffer,
15030 dedicatedImage,
15031 dedicatedBufferImageUsage,
15032 createInfo: createInfoFinal,
15033 memTypeIndex: blockVector.GetMemoryTypeIndex(),
15034 suballocType,
15035 dedicatedAllocations&: createInfoFinal.pool->m_DedicatedAllocations,
15036 blockVector,
15037 allocationCount,
15038 pAllocations);
15039 }
15040 else
15041 {
15042 // Bit mask of memory Vulkan types acceptable for this allocation.
15043 uint32_t memoryTypeBits = vkMemReq.memoryTypeBits;
15044 uint32_t memTypeIndex = UINT32_MAX;
15045 res = FindMemoryTypeIndex(memoryTypeBits, pAllocationCreateInfo: &createInfoFinal, bufImgUsage: dedicatedBufferImageUsage, pMemoryTypeIndex: &memTypeIndex);
15046 // Can't find any single memory type matching requirements. res is VK_ERROR_FEATURE_NOT_PRESENT.
15047 if(res != VK_SUCCESS)
15048 return res;
15049 do
15050 {
15051 VmaBlockVector* blockVector = m_pBlockVectors[memTypeIndex];
15052 VMA_ASSERT(blockVector && "Trying to use unsupported memory type!");
15053 res = AllocateMemoryOfType(
15054 VK_NULL_HANDLE,
15055 size: vkMemReq.size,
15056 alignment: vkMemReq.alignment,
15057 dedicatedPreferred: requiresDedicatedAllocation || prefersDedicatedAllocation,
15058 dedicatedBuffer,
15059 dedicatedImage,
15060 dedicatedBufferImageUsage,
15061 createInfo: createInfoFinal,
15062 memTypeIndex,
15063 suballocType,
15064 dedicatedAllocations&: m_DedicatedAllocations[memTypeIndex],
15065 blockVector&: *blockVector,
15066 allocationCount,
15067 pAllocations);
15068 // Allocation succeeded
15069 if(res == VK_SUCCESS)
15070 return VK_SUCCESS;
15071
15072 // Remove old memTypeIndex from list of possibilities.
15073 memoryTypeBits &= ~(1u << memTypeIndex);
15074 // Find alternative memTypeIndex.
15075 res = FindMemoryTypeIndex(memoryTypeBits, pAllocationCreateInfo: &createInfoFinal, bufImgUsage: dedicatedBufferImageUsage, pMemoryTypeIndex: &memTypeIndex);
15076 } while(res == VK_SUCCESS);
15077
15078 // No other matching memory type index could be found.
15079 // Not returning res, which is VK_ERROR_FEATURE_NOT_PRESENT, because we already failed to allocate once.
15080 return VK_ERROR_OUT_OF_DEVICE_MEMORY;
15081 }
15082}
15083
15084void VmaAllocator_T::FreeMemory(
15085 size_t allocationCount,
15086 const VmaAllocation* pAllocations)
15087{
15088 VMA_ASSERT(pAllocations);
15089
15090 for(size_t allocIndex = allocationCount; allocIndex--; )
15091 {
15092 VmaAllocation allocation = pAllocations[allocIndex];
15093
15094 if(allocation != VK_NULL_HANDLE)
15095 {
15096 if(VMA_DEBUG_INITIALIZE_ALLOCATIONS)
15097 {
15098 FillAllocation(hAllocation: allocation, pattern: VMA_ALLOCATION_FILL_PATTERN_DESTROYED);
15099 }
15100
15101 allocation->FreeName(hAllocator: this);
15102
15103 switch(allocation->GetType())
15104 {
15105 case VmaAllocation_T::ALLOCATION_TYPE_BLOCK:
15106 {
15107 VmaBlockVector* pBlockVector = VMA_NULL;
15108 VmaPool hPool = allocation->GetParentPool();
15109 if(hPool != VK_NULL_HANDLE)
15110 {
15111 pBlockVector = &hPool->m_BlockVector;
15112 }
15113 else
15114 {
15115 const uint32_t memTypeIndex = allocation->GetMemoryTypeIndex();
15116 pBlockVector = m_pBlockVectors[memTypeIndex];
15117 VMA_ASSERT(pBlockVector && "Trying to free memory of unsupported type!");
15118 }
15119 pBlockVector->Free(hAllocation: allocation);
15120 }
15121 break;
15122 case VmaAllocation_T::ALLOCATION_TYPE_DEDICATED:
15123 FreeDedicatedMemory(allocation);
15124 break;
15125 default:
15126 VMA_ASSERT(0);
15127 }
15128 }
15129 }
15130}
15131
15132void VmaAllocator_T::CalculateStatistics(VmaTotalStatistics* pStats)
15133{
15134 // Initialize.
15135 VmaClearDetailedStatistics(outStats&: pStats->total);
15136 for(uint32_t i = 0; i < VK_MAX_MEMORY_TYPES; ++i)
15137 VmaClearDetailedStatistics(outStats&: pStats->memoryType[i]);
15138 for(uint32_t i = 0; i < VK_MAX_MEMORY_HEAPS; ++i)
15139 VmaClearDetailedStatistics(outStats&: pStats->memoryHeap[i]);
15140
15141 // Process default pools.
15142 for(uint32_t memTypeIndex = 0; memTypeIndex < GetMemoryTypeCount(); ++memTypeIndex)
15143 {
15144 VmaBlockVector* const pBlockVector = m_pBlockVectors[memTypeIndex];
15145 if (pBlockVector != VMA_NULL)
15146 pBlockVector->AddDetailedStatistics(inoutStats&: pStats->memoryType[memTypeIndex]);
15147 }
15148
15149 // Process custom pools.
15150 {
15151 VmaMutexLockRead lock(m_PoolsMutex, m_UseMutex);
15152 for(VmaPool pool = m_Pools.Front(); pool != VMA_NULL; pool = m_Pools.GetNext(item: pool))
15153 {
15154 VmaBlockVector& blockVector = pool->m_BlockVector;
15155 const uint32_t memTypeIndex = blockVector.GetMemoryTypeIndex();
15156 blockVector.AddDetailedStatistics(inoutStats&: pStats->memoryType[memTypeIndex]);
15157 pool->m_DedicatedAllocations.AddDetailedStatistics(inoutStats&: pStats->memoryType[memTypeIndex]);
15158 }
15159 }
15160
15161 // Process dedicated allocations.
15162 for(uint32_t memTypeIndex = 0; memTypeIndex < GetMemoryTypeCount(); ++memTypeIndex)
15163 {
15164 m_DedicatedAllocations[memTypeIndex].AddDetailedStatistics(inoutStats&: pStats->memoryType[memTypeIndex]);
15165 }
15166
15167 // Sum from memory types to memory heaps.
15168 for(uint32_t memTypeIndex = 0; memTypeIndex < GetMemoryTypeCount(); ++memTypeIndex)
15169 {
15170 const uint32_t memHeapIndex = m_MemProps.memoryTypes[memTypeIndex].heapIndex;
15171 VmaAddDetailedStatistics(inoutStats&: pStats->memoryHeap[memHeapIndex], src: pStats->memoryType[memTypeIndex]);
15172 }
15173
15174 // Sum from memory heaps to total.
15175 for(uint32_t memHeapIndex = 0; memHeapIndex < GetMemoryHeapCount(); ++memHeapIndex)
15176 VmaAddDetailedStatistics(inoutStats&: pStats->total, src: pStats->memoryHeap[memHeapIndex]);
15177
15178 VMA_ASSERT(pStats->total.statistics.allocationCount == 0 ||
15179 pStats->total.allocationSizeMax >= pStats->total.allocationSizeMin);
15180 VMA_ASSERT(pStats->total.unusedRangeCount == 0 ||
15181 pStats->total.unusedRangeSizeMax >= pStats->total.unusedRangeSizeMin);
15182}
15183
15184void VmaAllocator_T::GetHeapBudgets(VmaBudget* outBudgets, uint32_t firstHeap, uint32_t heapCount)
15185{
15186#if VMA_MEMORY_BUDGET
15187 if(m_UseExtMemoryBudget)
15188 {
15189 if(m_Budget.m_OperationsSinceBudgetFetch < 30)
15190 {
15191 VmaMutexLockRead lockRead(m_Budget.m_BudgetMutex, m_UseMutex);
15192 for(uint32_t i = 0; i < heapCount; ++i, ++outBudgets)
15193 {
15194 const uint32_t heapIndex = firstHeap + i;
15195
15196 outBudgets->statistics.blockCount = m_Budget.m_BlockCount[heapIndex];
15197 outBudgets->statistics.allocationCount = m_Budget.m_AllocationCount[heapIndex];
15198 outBudgets->statistics.blockBytes = m_Budget.m_BlockBytes[heapIndex];
15199 outBudgets->statistics.allocationBytes = m_Budget.m_AllocationBytes[heapIndex];
15200
15201 if(m_Budget.m_VulkanUsage[heapIndex] + outBudgets->statistics.blockBytes > m_Budget.m_BlockBytesAtBudgetFetch[heapIndex])
15202 {
15203 outBudgets->usage = m_Budget.m_VulkanUsage[heapIndex] +
15204 outBudgets->statistics.blockBytes - m_Budget.m_BlockBytesAtBudgetFetch[heapIndex];
15205 }
15206 else
15207 {
15208 outBudgets->usage = 0;
15209 }
15210
15211 // Have to take MIN with heap size because explicit HeapSizeLimit is included in it.
15212 outBudgets->budget = VMA_MIN(
15213 m_Budget.m_VulkanBudget[heapIndex], m_MemProps.memoryHeaps[heapIndex].size);
15214 }
15215 }
15216 else
15217 {
15218 UpdateVulkanBudget(); // Outside of mutex lock
15219 GetHeapBudgets(outBudgets, firstHeap, heapCount); // Recursion
15220 }
15221 }
15222 else
15223#endif
15224 {
15225 for(uint32_t i = 0; i < heapCount; ++i, ++outBudgets)
15226 {
15227 const uint32_t heapIndex = firstHeap + i;
15228
15229 outBudgets->statistics.blockCount = m_Budget.m_BlockCount[heapIndex];
15230 outBudgets->statistics.allocationCount = m_Budget.m_AllocationCount[heapIndex];
15231 outBudgets->statistics.blockBytes = m_Budget.m_BlockBytes[heapIndex];
15232 outBudgets->statistics.allocationBytes = m_Budget.m_AllocationBytes[heapIndex];
15233
15234 outBudgets->usage = outBudgets->statistics.blockBytes;
15235 outBudgets->budget = m_MemProps.memoryHeaps[heapIndex].size * 8 / 10; // 80% heuristics.
15236 }
15237 }
15238}
15239
15240void VmaAllocator_T::GetAllocationInfo(VmaAllocation hAllocation, VmaAllocationInfo* pAllocationInfo)
15241{
15242 pAllocationInfo->memoryType = hAllocation->GetMemoryTypeIndex();
15243 pAllocationInfo->deviceMemory = hAllocation->GetMemory();
15244 pAllocationInfo->offset = hAllocation->GetOffset();
15245 pAllocationInfo->size = hAllocation->GetSize();
15246 pAllocationInfo->pMappedData = hAllocation->GetMappedData();
15247 pAllocationInfo->pUserData = hAllocation->GetUserData();
15248 pAllocationInfo->pName = hAllocation->GetName();
15249}
15250
15251VkResult VmaAllocator_T::CreatePool(const VmaPoolCreateInfo* pCreateInfo, VmaPool* pPool)
15252{
15253 VMA_DEBUG_LOG(" CreatePool: MemoryTypeIndex=%u, flags=%u", pCreateInfo->memoryTypeIndex, pCreateInfo->flags);
15254
15255 VmaPoolCreateInfo newCreateInfo = *pCreateInfo;
15256
15257 // Protection against uninitialized new structure member. If garbage data are left there, this pointer dereference would crash.
15258 if(pCreateInfo->pMemoryAllocateNext)
15259 {
15260 VMA_ASSERT(((const VkBaseInStructure*)pCreateInfo->pMemoryAllocateNext)->sType != 0);
15261 }
15262
15263 if(newCreateInfo.maxBlockCount == 0)
15264 {
15265 newCreateInfo.maxBlockCount = SIZE_MAX;
15266 }
15267 if(newCreateInfo.minBlockCount > newCreateInfo.maxBlockCount)
15268 {
15269 return VK_ERROR_INITIALIZATION_FAILED;
15270 }
15271 // Memory type index out of range or forbidden.
15272 if(pCreateInfo->memoryTypeIndex >= GetMemoryTypeCount() ||
15273 ((1u << pCreateInfo->memoryTypeIndex) & m_GlobalMemoryTypeBits) == 0)
15274 {
15275 return VK_ERROR_FEATURE_NOT_PRESENT;
15276 }
15277 if(newCreateInfo.minAllocationAlignment > 0)
15278 {
15279 VMA_ASSERT(VmaIsPow2(newCreateInfo.minAllocationAlignment));
15280 }
15281
15282 const VkDeviceSize preferredBlockSize = CalcPreferredBlockSize(memTypeIndex: newCreateInfo.memoryTypeIndex);
15283
15284 *pPool = vma_new(this, VmaPool_T)(this, newCreateInfo, preferredBlockSize);
15285
15286 VkResult res = (*pPool)->m_BlockVector.CreateMinBlocks();
15287 if(res != VK_SUCCESS)
15288 {
15289 vma_delete(hAllocator: this, ptr: *pPool);
15290 *pPool = VMA_NULL;
15291 return res;
15292 }
15293
15294 // Add to m_Pools.
15295 {
15296 VmaMutexLockWrite lock(m_PoolsMutex, m_UseMutex);
15297 (*pPool)->SetId(m_NextPoolId++);
15298 m_Pools.PushBack(item: *pPool);
15299 }
15300
15301 return VK_SUCCESS;
15302}
15303
15304void VmaAllocator_T::DestroyPool(VmaPool pool)
15305{
15306 // Remove from m_Pools.
15307 {
15308 VmaMutexLockWrite lock(m_PoolsMutex, m_UseMutex);
15309 m_Pools.Remove(item: pool);
15310 }
15311
15312 vma_delete(hAllocator: this, ptr: pool);
15313}
15314
15315void VmaAllocator_T::GetPoolStatistics(VmaPool pool, VmaStatistics* pPoolStats)
15316{
15317 VmaClearStatistics(outStats&: *pPoolStats);
15318 pool->m_BlockVector.AddStatistics(inoutStats&: *pPoolStats);
15319 pool->m_DedicatedAllocations.AddStatistics(inoutStats&: *pPoolStats);
15320}
15321
15322void VmaAllocator_T::CalculatePoolStatistics(VmaPool pool, VmaDetailedStatistics* pPoolStats)
15323{
15324 VmaClearDetailedStatistics(outStats&: *pPoolStats);
15325 pool->m_BlockVector.AddDetailedStatistics(inoutStats&: *pPoolStats);
15326 pool->m_DedicatedAllocations.AddDetailedStatistics(inoutStats&: *pPoolStats);
15327}
15328
15329void VmaAllocator_T::SetCurrentFrameIndex(uint32_t frameIndex)
15330{
15331 m_CurrentFrameIndex.store(i: frameIndex);
15332
15333#if VMA_MEMORY_BUDGET
15334 if(m_UseExtMemoryBudget)
15335 {
15336 UpdateVulkanBudget();
15337 }
15338#endif // #if VMA_MEMORY_BUDGET
15339}
15340
15341VkResult VmaAllocator_T::CheckPoolCorruption(VmaPool hPool)
15342{
15343 return hPool->m_BlockVector.CheckCorruption();
15344}
15345
15346VkResult VmaAllocator_T::CheckCorruption(uint32_t memoryTypeBits)
15347{
15348 VkResult finalRes = VK_ERROR_FEATURE_NOT_PRESENT;
15349
15350 // Process default pools.
15351 for(uint32_t memTypeIndex = 0; memTypeIndex < GetMemoryTypeCount(); ++memTypeIndex)
15352 {
15353 VmaBlockVector* const pBlockVector = m_pBlockVectors[memTypeIndex];
15354 if(pBlockVector != VMA_NULL)
15355 {
15356 VkResult localRes = pBlockVector->CheckCorruption();
15357 switch(localRes)
15358 {
15359 case VK_ERROR_FEATURE_NOT_PRESENT:
15360 break;
15361 case VK_SUCCESS:
15362 finalRes = VK_SUCCESS;
15363 break;
15364 default:
15365 return localRes;
15366 }
15367 }
15368 }
15369
15370 // Process custom pools.
15371 {
15372 VmaMutexLockRead lock(m_PoolsMutex, m_UseMutex);
15373 for(VmaPool pool = m_Pools.Front(); pool != VMA_NULL; pool = m_Pools.GetNext(item: pool))
15374 {
15375 if(((1u << pool->m_BlockVector.GetMemoryTypeIndex()) & memoryTypeBits) != 0)
15376 {
15377 VkResult localRes = pool->m_BlockVector.CheckCorruption();
15378 switch(localRes)
15379 {
15380 case VK_ERROR_FEATURE_NOT_PRESENT:
15381 break;
15382 case VK_SUCCESS:
15383 finalRes = VK_SUCCESS;
15384 break;
15385 default:
15386 return localRes;
15387 }
15388 }
15389 }
15390 }
15391
15392 return finalRes;
15393}
15394
15395VkResult VmaAllocator_T::AllocateVulkanMemory(const VkMemoryAllocateInfo* pAllocateInfo, VkDeviceMemory* pMemory)
15396{
15397 AtomicTransactionalIncrement<uint32_t> deviceMemoryCountIncrement;
15398 const uint64_t prevDeviceMemoryCount = deviceMemoryCountIncrement.Increment(atomic: &m_DeviceMemoryCount);
15399#if VMA_DEBUG_DONT_EXCEED_MAX_MEMORY_ALLOCATION_COUNT
15400 if(prevDeviceMemoryCount >= m_PhysicalDeviceProperties.limits.maxMemoryAllocationCount)
15401 {
15402 return VK_ERROR_TOO_MANY_OBJECTS;
15403 }
15404#endif
15405
15406 const uint32_t heapIndex = MemoryTypeIndexToHeapIndex(memTypeIndex: pAllocateInfo->memoryTypeIndex);
15407
15408 // HeapSizeLimit is in effect for this heap.
15409 if((m_HeapSizeLimitMask & (1u << heapIndex)) != 0)
15410 {
15411 const VkDeviceSize heapSize = m_MemProps.memoryHeaps[heapIndex].size;
15412 VkDeviceSize blockBytes = m_Budget.m_BlockBytes[heapIndex];
15413 for(;;)
15414 {
15415 const VkDeviceSize blockBytesAfterAllocation = blockBytes + pAllocateInfo->allocationSize;
15416 if(blockBytesAfterAllocation > heapSize)
15417 {
15418 return VK_ERROR_OUT_OF_DEVICE_MEMORY;
15419 }
15420 if(m_Budget.m_BlockBytes[heapIndex].compare_exchange_strong(i1&: blockBytes, i2: blockBytesAfterAllocation))
15421 {
15422 break;
15423 }
15424 }
15425 }
15426 else
15427 {
15428 m_Budget.m_BlockBytes[heapIndex] += pAllocateInfo->allocationSize;
15429 }
15430 ++m_Budget.m_BlockCount[heapIndex];
15431
15432 // VULKAN CALL vkAllocateMemory.
15433 VkResult res = (*m_VulkanFunctions.vkAllocateMemory)(m_hDevice, pAllocateInfo, GetAllocationCallbacks(), pMemory);
15434
15435 if(res == VK_SUCCESS)
15436 {
15437#if VMA_MEMORY_BUDGET
15438 ++m_Budget.m_OperationsSinceBudgetFetch;
15439#endif
15440
15441 // Informative callback.
15442 if(m_DeviceMemoryCallbacks.pfnAllocate != VMA_NULL)
15443 {
15444 (*m_DeviceMemoryCallbacks.pfnAllocate)(this, pAllocateInfo->memoryTypeIndex, *pMemory, pAllocateInfo->allocationSize, m_DeviceMemoryCallbacks.pUserData);
15445 }
15446
15447 deviceMemoryCountIncrement.Commit();
15448 }
15449 else
15450 {
15451 --m_Budget.m_BlockCount[heapIndex];
15452 m_Budget.m_BlockBytes[heapIndex] -= pAllocateInfo->allocationSize;
15453 }
15454
15455 return res;
15456}
15457
15458void VmaAllocator_T::FreeVulkanMemory(uint32_t memoryType, VkDeviceSize size, VkDeviceMemory hMemory)
15459{
15460 // Informative callback.
15461 if(m_DeviceMemoryCallbacks.pfnFree != VMA_NULL)
15462 {
15463 (*m_DeviceMemoryCallbacks.pfnFree)(this, memoryType, hMemory, size, m_DeviceMemoryCallbacks.pUserData);
15464 }
15465
15466 // VULKAN CALL vkFreeMemory.
15467 (*m_VulkanFunctions.vkFreeMemory)(m_hDevice, hMemory, GetAllocationCallbacks());
15468
15469 const uint32_t heapIndex = MemoryTypeIndexToHeapIndex(memTypeIndex: memoryType);
15470 --m_Budget.m_BlockCount[heapIndex];
15471 m_Budget.m_BlockBytes[heapIndex] -= size;
15472
15473 --m_DeviceMemoryCount;
15474}
15475
15476VkResult VmaAllocator_T::BindVulkanBuffer(
15477 VkDeviceMemory memory,
15478 VkDeviceSize memoryOffset,
15479 VkBuffer buffer,
15480 const void* pNext)
15481{
15482 if(pNext != VMA_NULL)
15483 {
15484#if VMA_VULKAN_VERSION >= 1001000 || VMA_BIND_MEMORY2
15485 if((m_UseKhrBindMemory2 || m_VulkanApiVersion >= VK_MAKE_VERSION(1, 1, 0)) &&
15486 m_VulkanFunctions.vkBindBufferMemory2KHR != VMA_NULL)
15487 {
15488 VkBindBufferMemoryInfoKHR bindBufferMemoryInfo = { .sType: VK_STRUCTURE_TYPE_BIND_BUFFER_MEMORY_INFO_KHR };
15489 bindBufferMemoryInfo.pNext = pNext;
15490 bindBufferMemoryInfo.buffer = buffer;
15491 bindBufferMemoryInfo.memory = memory;
15492 bindBufferMemoryInfo.memoryOffset = memoryOffset;
15493 return (*m_VulkanFunctions.vkBindBufferMemory2KHR)(m_hDevice, 1, &bindBufferMemoryInfo);
15494 }
15495 else
15496#endif // #if VMA_VULKAN_VERSION >= 1001000 || VMA_BIND_MEMORY2
15497 {
15498 return VK_ERROR_EXTENSION_NOT_PRESENT;
15499 }
15500 }
15501 else
15502 {
15503 return (*m_VulkanFunctions.vkBindBufferMemory)(m_hDevice, buffer, memory, memoryOffset);
15504 }
15505}
15506
15507VkResult VmaAllocator_T::BindVulkanImage(
15508 VkDeviceMemory memory,
15509 VkDeviceSize memoryOffset,
15510 VkImage image,
15511 const void* pNext)
15512{
15513 if(pNext != VMA_NULL)
15514 {
15515#if VMA_VULKAN_VERSION >= 1001000 || VMA_BIND_MEMORY2
15516 if((m_UseKhrBindMemory2 || m_VulkanApiVersion >= VK_MAKE_VERSION(1, 1, 0)) &&
15517 m_VulkanFunctions.vkBindImageMemory2KHR != VMA_NULL)
15518 {
15519 VkBindImageMemoryInfoKHR bindBufferMemoryInfo = { .sType: VK_STRUCTURE_TYPE_BIND_IMAGE_MEMORY_INFO_KHR };
15520 bindBufferMemoryInfo.pNext = pNext;
15521 bindBufferMemoryInfo.image = image;
15522 bindBufferMemoryInfo.memory = memory;
15523 bindBufferMemoryInfo.memoryOffset = memoryOffset;
15524 return (*m_VulkanFunctions.vkBindImageMemory2KHR)(m_hDevice, 1, &bindBufferMemoryInfo);
15525 }
15526 else
15527#endif // #if VMA_BIND_MEMORY2
15528 {
15529 return VK_ERROR_EXTENSION_NOT_PRESENT;
15530 }
15531 }
15532 else
15533 {
15534 return (*m_VulkanFunctions.vkBindImageMemory)(m_hDevice, image, memory, memoryOffset);
15535 }
15536}
15537
15538VkResult VmaAllocator_T::Map(VmaAllocation hAllocation, void** ppData)
15539{
15540 switch(hAllocation->GetType())
15541 {
15542 case VmaAllocation_T::ALLOCATION_TYPE_BLOCK:
15543 {
15544 VmaDeviceMemoryBlock* const pBlock = hAllocation->GetBlock();
15545 char *pBytes = VMA_NULL;
15546 VkResult res = pBlock->Map(hAllocator: this, count: 1, ppData: (void**)&pBytes);
15547 if(res == VK_SUCCESS)
15548 {
15549 *ppData = pBytes + (ptrdiff_t)hAllocation->GetOffset();
15550 hAllocation->BlockAllocMap();
15551 }
15552 return res;
15553 }
15554 case VmaAllocation_T::ALLOCATION_TYPE_DEDICATED:
15555 return hAllocation->DedicatedAllocMap(hAllocator: this, ppData);
15556 default:
15557 VMA_ASSERT(0);
15558 return VK_ERROR_MEMORY_MAP_FAILED;
15559 }
15560}
15561
15562void VmaAllocator_T::Unmap(VmaAllocation hAllocation)
15563{
15564 switch(hAllocation->GetType())
15565 {
15566 case VmaAllocation_T::ALLOCATION_TYPE_BLOCK:
15567 {
15568 VmaDeviceMemoryBlock* const pBlock = hAllocation->GetBlock();
15569 hAllocation->BlockAllocUnmap();
15570 pBlock->Unmap(hAllocator: this, count: 1);
15571 }
15572 break;
15573 case VmaAllocation_T::ALLOCATION_TYPE_DEDICATED:
15574 hAllocation->DedicatedAllocUnmap(hAllocator: this);
15575 break;
15576 default:
15577 VMA_ASSERT(0);
15578 }
15579}
15580
15581VkResult VmaAllocator_T::BindBufferMemory(
15582 VmaAllocation hAllocation,
15583 VkDeviceSize allocationLocalOffset,
15584 VkBuffer hBuffer,
15585 const void* pNext)
15586{
15587 VkResult res = VK_SUCCESS;
15588 switch(hAllocation->GetType())
15589 {
15590 case VmaAllocation_T::ALLOCATION_TYPE_DEDICATED:
15591 res = BindVulkanBuffer(memory: hAllocation->GetMemory(), memoryOffset: allocationLocalOffset, buffer: hBuffer, pNext);
15592 break;
15593 case VmaAllocation_T::ALLOCATION_TYPE_BLOCK:
15594 {
15595 VmaDeviceMemoryBlock* const pBlock = hAllocation->GetBlock();
15596 VMA_ASSERT(pBlock && "Binding buffer to allocation that doesn't belong to any block.");
15597 res = pBlock->BindBufferMemory(hAllocator: this, hAllocation, allocationLocalOffset, hBuffer, pNext);
15598 break;
15599 }
15600 default:
15601 VMA_ASSERT(0);
15602 }
15603 return res;
15604}
15605
15606VkResult VmaAllocator_T::BindImageMemory(
15607 VmaAllocation hAllocation,
15608 VkDeviceSize allocationLocalOffset,
15609 VkImage hImage,
15610 const void* pNext)
15611{
15612 VkResult res = VK_SUCCESS;
15613 switch(hAllocation->GetType())
15614 {
15615 case VmaAllocation_T::ALLOCATION_TYPE_DEDICATED:
15616 res = BindVulkanImage(memory: hAllocation->GetMemory(), memoryOffset: allocationLocalOffset, image: hImage, pNext);
15617 break;
15618 case VmaAllocation_T::ALLOCATION_TYPE_BLOCK:
15619 {
15620 VmaDeviceMemoryBlock* pBlock = hAllocation->GetBlock();
15621 VMA_ASSERT(pBlock && "Binding image to allocation that doesn't belong to any block.");
15622 res = pBlock->BindImageMemory(hAllocator: this, hAllocation, allocationLocalOffset, hImage, pNext);
15623 break;
15624 }
15625 default:
15626 VMA_ASSERT(0);
15627 }
15628 return res;
15629}
15630
15631VkResult VmaAllocator_T::FlushOrInvalidateAllocation(
15632 VmaAllocation hAllocation,
15633 VkDeviceSize offset, VkDeviceSize size,
15634 VMA_CACHE_OPERATION op)
15635{
15636 VkResult res = VK_SUCCESS;
15637
15638 VkMappedMemoryRange memRange = {};
15639 if(GetFlushOrInvalidateRange(allocation: hAllocation, offset, size, outRange&: memRange))
15640 {
15641 switch(op)
15642 {
15643 case VMA_CACHE_FLUSH:
15644 res = (*GetVulkanFunctions().vkFlushMappedMemoryRanges)(m_hDevice, 1, &memRange);
15645 break;
15646 case VMA_CACHE_INVALIDATE:
15647 res = (*GetVulkanFunctions().vkInvalidateMappedMemoryRanges)(m_hDevice, 1, &memRange);
15648 break;
15649 default:
15650 VMA_ASSERT(0);
15651 }
15652 }
15653 // else: Just ignore this call.
15654 return res;
15655}
15656
15657VkResult VmaAllocator_T::FlushOrInvalidateAllocations(
15658 uint32_t allocationCount,
15659 const VmaAllocation* allocations,
15660 const VkDeviceSize* offsets, const VkDeviceSize* sizes,
15661 VMA_CACHE_OPERATION op)
15662{
15663 typedef VmaStlAllocator<VkMappedMemoryRange> RangeAllocator;
15664 typedef VmaSmallVector<VkMappedMemoryRange, RangeAllocator, 16> RangeVector;
15665 RangeVector ranges = RangeVector(RangeAllocator(GetAllocationCallbacks()));
15666
15667 for(uint32_t allocIndex = 0; allocIndex < allocationCount; ++allocIndex)
15668 {
15669 const VmaAllocation alloc = allocations[allocIndex];
15670 const VkDeviceSize offset = offsets != VMA_NULL ? offsets[allocIndex] : 0;
15671 const VkDeviceSize size = sizes != VMA_NULL ? sizes[allocIndex] : VK_WHOLE_SIZE;
15672 VkMappedMemoryRange newRange;
15673 if(GetFlushOrInvalidateRange(allocation: alloc, offset, size, outRange&: newRange))
15674 {
15675 ranges.push_back(src: newRange);
15676 }
15677 }
15678
15679 VkResult res = VK_SUCCESS;
15680 if(!ranges.empty())
15681 {
15682 switch(op)
15683 {
15684 case VMA_CACHE_FLUSH:
15685 res = (*GetVulkanFunctions().vkFlushMappedMemoryRanges)(m_hDevice, (uint32_t)ranges.size(), ranges.data());
15686 break;
15687 case VMA_CACHE_INVALIDATE:
15688 res = (*GetVulkanFunctions().vkInvalidateMappedMemoryRanges)(m_hDevice, (uint32_t)ranges.size(), ranges.data());
15689 break;
15690 default:
15691 VMA_ASSERT(0);
15692 }
15693 }
15694 // else: Just ignore this call.
15695 return res;
15696}
15697
15698void VmaAllocator_T::FreeDedicatedMemory(const VmaAllocation allocation)
15699{
15700 VMA_ASSERT(allocation && allocation->GetType() == VmaAllocation_T::ALLOCATION_TYPE_DEDICATED);
15701
15702 const uint32_t memTypeIndex = allocation->GetMemoryTypeIndex();
15703 VmaPool parentPool = allocation->GetParentPool();
15704 if(parentPool == VK_NULL_HANDLE)
15705 {
15706 // Default pool
15707 m_DedicatedAllocations[memTypeIndex].Unregister(alloc: allocation);
15708 }
15709 else
15710 {
15711 // Custom pool
15712 parentPool->m_DedicatedAllocations.Unregister(alloc: allocation);
15713 }
15714
15715 VkDeviceMemory hMemory = allocation->GetMemory();
15716
15717 /*
15718 There is no need to call this, because Vulkan spec allows to skip vkUnmapMemory
15719 before vkFreeMemory.
15720
15721 if(allocation->GetMappedData() != VMA_NULL)
15722 {
15723 (*m_VulkanFunctions.vkUnmapMemory)(m_hDevice, hMemory);
15724 }
15725 */
15726
15727 FreeVulkanMemory(memoryType: memTypeIndex, size: allocation->GetSize(), hMemory);
15728
15729 m_Budget.RemoveAllocation(heapIndex: MemoryTypeIndexToHeapIndex(memTypeIndex: allocation->GetMemoryTypeIndex()), allocationSize: allocation->GetSize());
15730 m_AllocationObjectAllocator.Free(hAlloc: allocation);
15731
15732 VMA_DEBUG_LOG(" Freed DedicatedMemory MemoryTypeIndex=%u", memTypeIndex);
15733}
15734
15735uint32_t VmaAllocator_T::CalculateGpuDefragmentationMemoryTypeBits() const
15736{
15737 VkBufferCreateInfo dummyBufCreateInfo;
15738 VmaFillGpuDefragmentationBufferCreateInfo(outBufCreateInfo&: dummyBufCreateInfo);
15739
15740 uint32_t memoryTypeBits = 0;
15741
15742 // Create buffer.
15743 VkBuffer buf = VK_NULL_HANDLE;
15744 VkResult res = (*GetVulkanFunctions().vkCreateBuffer)(
15745 m_hDevice, &dummyBufCreateInfo, GetAllocationCallbacks(), &buf);
15746 if(res == VK_SUCCESS)
15747 {
15748 // Query for supported memory types.
15749 VkMemoryRequirements memReq;
15750 (*GetVulkanFunctions().vkGetBufferMemoryRequirements)(m_hDevice, buf, &memReq);
15751 memoryTypeBits = memReq.memoryTypeBits;
15752
15753 // Destroy buffer.
15754 (*GetVulkanFunctions().vkDestroyBuffer)(m_hDevice, buf, GetAllocationCallbacks());
15755 }
15756
15757 return memoryTypeBits;
15758}
15759
15760uint32_t VmaAllocator_T::CalculateGlobalMemoryTypeBits() const
15761{
15762 // Make sure memory information is already fetched.
15763 VMA_ASSERT(GetMemoryTypeCount() > 0);
15764
15765 uint32_t memoryTypeBits = UINT32_MAX;
15766
15767 if(!m_UseAmdDeviceCoherentMemory)
15768 {
15769 // Exclude memory types that have VK_MEMORY_PROPERTY_DEVICE_COHERENT_BIT_AMD.
15770 for(uint32_t memTypeIndex = 0; memTypeIndex < GetMemoryTypeCount(); ++memTypeIndex)
15771 {
15772 if((m_MemProps.memoryTypes[memTypeIndex].propertyFlags & VK_MEMORY_PROPERTY_DEVICE_COHERENT_BIT_AMD_COPY) != 0)
15773 {
15774 memoryTypeBits &= ~(1u << memTypeIndex);
15775 }
15776 }
15777 }
15778
15779 return memoryTypeBits;
15780}
15781
15782bool VmaAllocator_T::GetFlushOrInvalidateRange(
15783 VmaAllocation allocation,
15784 VkDeviceSize offset, VkDeviceSize size,
15785 VkMappedMemoryRange& outRange) const
15786{
15787 const uint32_t memTypeIndex = allocation->GetMemoryTypeIndex();
15788 if(size > 0 && IsMemoryTypeNonCoherent(memTypeIndex))
15789 {
15790 const VkDeviceSize nonCoherentAtomSize = m_PhysicalDeviceProperties.limits.nonCoherentAtomSize;
15791 const VkDeviceSize allocationSize = allocation->GetSize();
15792 VMA_ASSERT(offset <= allocationSize);
15793
15794 outRange.sType = VK_STRUCTURE_TYPE_MAPPED_MEMORY_RANGE;
15795 outRange.pNext = VMA_NULL;
15796 outRange.memory = allocation->GetMemory();
15797
15798 switch(allocation->GetType())
15799 {
15800 case VmaAllocation_T::ALLOCATION_TYPE_DEDICATED:
15801 outRange.offset = VmaAlignDown(val: offset, alignment: nonCoherentAtomSize);
15802 if(size == VK_WHOLE_SIZE)
15803 {
15804 outRange.size = allocationSize - outRange.offset;
15805 }
15806 else
15807 {
15808 VMA_ASSERT(offset + size <= allocationSize);
15809 outRange.size = VMA_MIN(
15810 VmaAlignUp(size + (offset - outRange.offset), nonCoherentAtomSize),
15811 allocationSize - outRange.offset);
15812 }
15813 break;
15814 case VmaAllocation_T::ALLOCATION_TYPE_BLOCK:
15815 {
15816 // 1. Still within this allocation.
15817 outRange.offset = VmaAlignDown(val: offset, alignment: nonCoherentAtomSize);
15818 if(size == VK_WHOLE_SIZE)
15819 {
15820 size = allocationSize - offset;
15821 }
15822 else
15823 {
15824 VMA_ASSERT(offset + size <= allocationSize);
15825 }
15826 outRange.size = VmaAlignUp(val: size + (offset - outRange.offset), alignment: nonCoherentAtomSize);
15827
15828 // 2. Adjust to whole block.
15829 const VkDeviceSize allocationOffset = allocation->GetOffset();
15830 VMA_ASSERT(allocationOffset % nonCoherentAtomSize == 0);
15831 const VkDeviceSize blockSize = allocation->GetBlock()->m_pMetadata->GetSize();
15832 outRange.offset += allocationOffset;
15833 outRange.size = VMA_MIN(outRange.size, blockSize - outRange.offset);
15834
15835 break;
15836 }
15837 default:
15838 VMA_ASSERT(0);
15839 }
15840 return true;
15841 }
15842 return false;
15843}
15844
15845#if VMA_MEMORY_BUDGET
15846void VmaAllocator_T::UpdateVulkanBudget()
15847{
15848 VMA_ASSERT(m_UseExtMemoryBudget);
15849
15850 VkPhysicalDeviceMemoryProperties2KHR memProps = { .sType: VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_MEMORY_PROPERTIES_2_KHR };
15851
15852 VkPhysicalDeviceMemoryBudgetPropertiesEXT budgetProps = { .sType: VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_MEMORY_BUDGET_PROPERTIES_EXT };
15853 VmaPnextChainPushFront(mainStruct: &memProps, newStruct: &budgetProps);
15854
15855 GetVulkanFunctions().vkGetPhysicalDeviceMemoryProperties2KHR(m_PhysicalDevice, &memProps);
15856
15857 {
15858 VmaMutexLockWrite lockWrite(m_Budget.m_BudgetMutex, m_UseMutex);
15859
15860 for(uint32_t heapIndex = 0; heapIndex < GetMemoryHeapCount(); ++heapIndex)
15861 {
15862 m_Budget.m_VulkanUsage[heapIndex] = budgetProps.heapUsage[heapIndex];
15863 m_Budget.m_VulkanBudget[heapIndex] = budgetProps.heapBudget[heapIndex];
15864 m_Budget.m_BlockBytesAtBudgetFetch[heapIndex] = m_Budget.m_BlockBytes[heapIndex].load();
15865
15866 // Some bugged drivers return the budget incorrectly, e.g. 0 or much bigger than heap size.
15867 if(m_Budget.m_VulkanBudget[heapIndex] == 0)
15868 {
15869 m_Budget.m_VulkanBudget[heapIndex] = m_MemProps.memoryHeaps[heapIndex].size * 8 / 10; // 80% heuristics.
15870 }
15871 else if(m_Budget.m_VulkanBudget[heapIndex] > m_MemProps.memoryHeaps[heapIndex].size)
15872 {
15873 m_Budget.m_VulkanBudget[heapIndex] = m_MemProps.memoryHeaps[heapIndex].size;
15874 }
15875 if(m_Budget.m_VulkanUsage[heapIndex] == 0 && m_Budget.m_BlockBytesAtBudgetFetch[heapIndex] > 0)
15876 {
15877 m_Budget.m_VulkanUsage[heapIndex] = m_Budget.m_BlockBytesAtBudgetFetch[heapIndex];
15878 }
15879 }
15880 m_Budget.m_OperationsSinceBudgetFetch = 0;
15881 }
15882}
15883#endif // VMA_MEMORY_BUDGET
15884
15885void VmaAllocator_T::FillAllocation(const VmaAllocation hAllocation, uint8_t pattern)
15886{
15887 if(VMA_DEBUG_INITIALIZE_ALLOCATIONS &&
15888 hAllocation->IsMappingAllowed() &&
15889 (m_MemProps.memoryTypes[hAllocation->GetMemoryTypeIndex()].propertyFlags & VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT) != 0)
15890 {
15891 void* pData = VMA_NULL;
15892 VkResult res = Map(hAllocation, ppData: &pData);
15893 if(res == VK_SUCCESS)
15894 {
15895 memset(s: pData, c: (int)pattern, n: (size_t)hAllocation->GetSize());
15896 FlushOrInvalidateAllocation(hAllocation, offset: 0, VK_WHOLE_SIZE, op: VMA_CACHE_FLUSH);
15897 Unmap(hAllocation);
15898 }
15899 else
15900 {
15901 VMA_ASSERT(0 && "VMA_DEBUG_INITIALIZE_ALLOCATIONS is enabled, but couldn't map memory to fill allocation.");
15902 }
15903 }
15904}
15905
15906uint32_t VmaAllocator_T::GetGpuDefragmentationMemoryTypeBits()
15907{
15908 uint32_t memoryTypeBits = m_GpuDefragmentationMemoryTypeBits.load();
15909 if(memoryTypeBits == UINT32_MAX)
15910 {
15911 memoryTypeBits = CalculateGpuDefragmentationMemoryTypeBits();
15912 m_GpuDefragmentationMemoryTypeBits.store(i: memoryTypeBits);
15913 }
15914 return memoryTypeBits;
15915}
15916
15917#if VMA_STATS_STRING_ENABLED
15918void VmaAllocator_T::PrintDetailedMap(VmaJsonWriter& json)
15919{
15920 json.WriteString(pStr: "DefaultPools");
15921 json.BeginObject();
15922 {
15923 for (uint32_t memTypeIndex = 0; memTypeIndex < GetMemoryTypeCount(); ++memTypeIndex)
15924 {
15925 VmaBlockVector* pBlockVector = m_pBlockVectors[memTypeIndex];
15926 VmaDedicatedAllocationList& dedicatedAllocList = m_DedicatedAllocations[memTypeIndex];
15927 if (pBlockVector != VMA_NULL)
15928 {
15929 json.BeginString(pStr: "Type ");
15930 json.ContinueString(n: memTypeIndex);
15931 json.EndString();
15932 json.BeginObject();
15933 {
15934 json.WriteString(pStr: "PreferredBlockSize");
15935 json.WriteNumber(n: pBlockVector->GetPreferredBlockSize());
15936
15937 json.WriteString(pStr: "Blocks");
15938 pBlockVector->PrintDetailedMap(json);
15939
15940 json.WriteString(pStr: "DedicatedAllocations");
15941 dedicatedAllocList.BuildStatsString(json);
15942 }
15943 json.EndObject();
15944 }
15945 }
15946 }
15947 json.EndObject();
15948
15949 json.WriteString(pStr: "CustomPools");
15950 json.BeginObject();
15951 {
15952 VmaMutexLockRead lock(m_PoolsMutex, m_UseMutex);
15953 if (!m_Pools.IsEmpty())
15954 {
15955 for (uint32_t memTypeIndex = 0; memTypeIndex < GetMemoryTypeCount(); ++memTypeIndex)
15956 {
15957 bool displayType = true;
15958 size_t index = 0;
15959 for (VmaPool pool = m_Pools.Front(); pool != VMA_NULL; pool = m_Pools.GetNext(item: pool))
15960 {
15961 VmaBlockVector& blockVector = pool->m_BlockVector;
15962 if (blockVector.GetMemoryTypeIndex() == memTypeIndex)
15963 {
15964 if (displayType)
15965 {
15966 json.BeginString(pStr: "Type ");
15967 json.ContinueString(n: memTypeIndex);
15968 json.EndString();
15969 json.BeginArray();
15970 displayType = false;
15971 }
15972
15973 json.BeginObject();
15974 {
15975 json.WriteString(pStr: "Name");
15976 json.BeginString();
15977 json.ContinueString_Size(n: index++);
15978 if (pool->GetName())
15979 {
15980 json.ContinueString(pStr: " - ");
15981 json.ContinueString(pStr: pool->GetName());
15982 }
15983 json.EndString();
15984
15985 json.WriteString(pStr: "PreferredBlockSize");
15986 json.WriteNumber(n: blockVector.GetPreferredBlockSize());
15987
15988 json.WriteString(pStr: "Blocks");
15989 blockVector.PrintDetailedMap(json);
15990
15991 json.WriteString(pStr: "DedicatedAllocations");
15992 pool->m_DedicatedAllocations.BuildStatsString(json);
15993 }
15994 json.EndObject();
15995 }
15996 }
15997
15998 if (!displayType)
15999 json.EndArray();
16000 }
16001 }
16002 }
16003 json.EndObject();
16004}
16005#endif // VMA_STATS_STRING_ENABLED
16006#endif // _VMA_ALLOCATOR_T_FUNCTIONS
16007
16008
16009#ifndef _VMA_PUBLIC_INTERFACE
16010VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateAllocator(
16011 const VmaAllocatorCreateInfo* pCreateInfo,
16012 VmaAllocator* pAllocator)
16013{
16014 VMA_ASSERT(pCreateInfo && pAllocator);
16015 VMA_ASSERT(pCreateInfo->vulkanApiVersion == 0 ||
16016 (VK_VERSION_MAJOR(pCreateInfo->vulkanApiVersion) == 1 && VK_VERSION_MINOR(pCreateInfo->vulkanApiVersion) <= 3));
16017 VMA_DEBUG_LOG("vmaCreateAllocator");
16018 *pAllocator = vma_new(pCreateInfo->pAllocationCallbacks, VmaAllocator_T)(pCreateInfo);
16019 VkResult result = (*pAllocator)->Init(pCreateInfo);
16020 if(result < 0)
16021 {
16022 vma_delete(pAllocationCallbacks: pCreateInfo->pAllocationCallbacks, ptr: *pAllocator);
16023 *pAllocator = VK_NULL_HANDLE;
16024 }
16025 return result;
16026}
16027
16028VMA_CALL_PRE void VMA_CALL_POST vmaDestroyAllocator(
16029 VmaAllocator allocator)
16030{
16031 if(allocator != VK_NULL_HANDLE)
16032 {
16033 VMA_DEBUG_LOG("vmaDestroyAllocator");
16034 VkAllocationCallbacks allocationCallbacks = allocator->m_AllocationCallbacks; // Have to copy the callbacks when destroying.
16035 vma_delete(pAllocationCallbacks: &allocationCallbacks, ptr: allocator);
16036 }
16037}
16038
16039VMA_CALL_PRE void VMA_CALL_POST vmaGetAllocatorInfo(VmaAllocator allocator, VmaAllocatorInfo* pAllocatorInfo)
16040{
16041 VMA_ASSERT(allocator && pAllocatorInfo);
16042 pAllocatorInfo->instance = allocator->m_hInstance;
16043 pAllocatorInfo->physicalDevice = allocator->GetPhysicalDevice();
16044 pAllocatorInfo->device = allocator->m_hDevice;
16045}
16046
16047VMA_CALL_PRE void VMA_CALL_POST vmaGetPhysicalDeviceProperties(
16048 VmaAllocator allocator,
16049 const VkPhysicalDeviceProperties **ppPhysicalDeviceProperties)
16050{
16051 VMA_ASSERT(allocator && ppPhysicalDeviceProperties);
16052 *ppPhysicalDeviceProperties = &allocator->m_PhysicalDeviceProperties;
16053}
16054
16055VMA_CALL_PRE void VMA_CALL_POST vmaGetMemoryProperties(
16056 VmaAllocator allocator,
16057 const VkPhysicalDeviceMemoryProperties** ppPhysicalDeviceMemoryProperties)
16058{
16059 VMA_ASSERT(allocator && ppPhysicalDeviceMemoryProperties);
16060 *ppPhysicalDeviceMemoryProperties = &allocator->m_MemProps;
16061}
16062
16063VMA_CALL_PRE void VMA_CALL_POST vmaGetMemoryTypeProperties(
16064 VmaAllocator allocator,
16065 uint32_t memoryTypeIndex,
16066 VkMemoryPropertyFlags* pFlags)
16067{
16068 VMA_ASSERT(allocator && pFlags);
16069 VMA_ASSERT(memoryTypeIndex < allocator->GetMemoryTypeCount());
16070 *pFlags = allocator->m_MemProps.memoryTypes[memoryTypeIndex].propertyFlags;
16071}
16072
16073VMA_CALL_PRE void VMA_CALL_POST vmaSetCurrentFrameIndex(
16074 VmaAllocator allocator,
16075 uint32_t frameIndex)
16076{
16077 VMA_ASSERT(allocator);
16078
16079 VMA_DEBUG_GLOBAL_MUTEX_LOCK
16080
16081 allocator->SetCurrentFrameIndex(frameIndex);
16082}
16083
16084VMA_CALL_PRE void VMA_CALL_POST vmaCalculateStatistics(
16085 VmaAllocator allocator,
16086 VmaTotalStatistics* pStats)
16087{
16088 VMA_ASSERT(allocator && pStats);
16089 VMA_DEBUG_GLOBAL_MUTEX_LOCK
16090 allocator->CalculateStatistics(pStats);
16091}
16092
16093VMA_CALL_PRE void VMA_CALL_POST vmaGetHeapBudgets(
16094 VmaAllocator allocator,
16095 VmaBudget* pBudgets)
16096{
16097 VMA_ASSERT(allocator && pBudgets);
16098 VMA_DEBUG_GLOBAL_MUTEX_LOCK
16099 allocator->GetHeapBudgets(outBudgets: pBudgets, firstHeap: 0, heapCount: allocator->GetMemoryHeapCount());
16100}
16101
16102#if VMA_STATS_STRING_ENABLED
16103
16104VMA_CALL_PRE void VMA_CALL_POST vmaBuildStatsString(
16105 VmaAllocator allocator,
16106 char** ppStatsString,
16107 VkBool32 detailedMap)
16108{
16109 VMA_ASSERT(allocator && ppStatsString);
16110 VMA_DEBUG_GLOBAL_MUTEX_LOCK
16111
16112 VmaStringBuilder sb(allocator->GetAllocationCallbacks());
16113 {
16114 VmaBudget budgets[VK_MAX_MEMORY_HEAPS];
16115 allocator->GetHeapBudgets(outBudgets: budgets, firstHeap: 0, heapCount: allocator->GetMemoryHeapCount());
16116
16117 VmaTotalStatistics stats;
16118 allocator->CalculateStatistics(pStats: &stats);
16119
16120 VmaJsonWriter json(allocator->GetAllocationCallbacks(), sb);
16121 json.BeginObject();
16122 {
16123 json.WriteString(pStr: "General");
16124 json.BeginObject();
16125 {
16126 const VkPhysicalDeviceProperties& deviceProperties = allocator->m_PhysicalDeviceProperties;
16127 const VkPhysicalDeviceMemoryProperties& memoryProperties = allocator->m_MemProps;
16128
16129 json.WriteString(pStr: "API");
16130 json.WriteString(pStr: "Vulkan");
16131
16132 json.WriteString(pStr: "apiVersion");
16133 json.BeginString();
16134 json.ContinueString(VK_VERSION_MAJOR(deviceProperties.apiVersion));
16135 json.ContinueString(pStr: ".");
16136 json.ContinueString(VK_VERSION_MINOR(deviceProperties.apiVersion));
16137 json.ContinueString(pStr: ".");
16138 json.ContinueString(VK_VERSION_PATCH(deviceProperties.apiVersion));
16139 json.EndString();
16140
16141 json.WriteString(pStr: "GPU");
16142 json.WriteString(pStr: deviceProperties.deviceName);
16143 json.WriteString(pStr: "deviceType");
16144 json.WriteNumber(n: static_cast<uint32_t>(deviceProperties.deviceType));
16145
16146 json.WriteString(pStr: "maxMemoryAllocationCount");
16147 json.WriteNumber(n: deviceProperties.limits.maxMemoryAllocationCount);
16148 json.WriteString(pStr: "bufferImageGranularity");
16149 json.WriteNumber(n: deviceProperties.limits.bufferImageGranularity);
16150 json.WriteString(pStr: "nonCoherentAtomSize");
16151 json.WriteNumber(n: deviceProperties.limits.nonCoherentAtomSize);
16152
16153 json.WriteString(pStr: "memoryHeapCount");
16154 json.WriteNumber(n: memoryProperties.memoryHeapCount);
16155 json.WriteString(pStr: "memoryTypeCount");
16156 json.WriteNumber(n: memoryProperties.memoryTypeCount);
16157 }
16158 json.EndObject();
16159 }
16160 {
16161 json.WriteString(pStr: "Total");
16162 VmaPrintDetailedStatistics(json, stat: stats.total);
16163 }
16164 {
16165 json.WriteString(pStr: "MemoryInfo");
16166 json.BeginObject();
16167 {
16168 for (uint32_t heapIndex = 0; heapIndex < allocator->GetMemoryHeapCount(); ++heapIndex)
16169 {
16170 json.BeginString(pStr: "Heap ");
16171 json.ContinueString(n: heapIndex);
16172 json.EndString();
16173 json.BeginObject();
16174 {
16175 const VkMemoryHeap& heapInfo = allocator->m_MemProps.memoryHeaps[heapIndex];
16176 json.WriteString(pStr: "Flags");
16177 json.BeginArray(singleLine: true);
16178 {
16179 if (heapInfo.flags & VK_MEMORY_HEAP_DEVICE_LOCAL_BIT)
16180 json.WriteString(pStr: "DEVICE_LOCAL");
16181 #if VMA_VULKAN_VERSION >= 1001000
16182 if (heapInfo.flags & VK_MEMORY_HEAP_MULTI_INSTANCE_BIT)
16183 json.WriteString(pStr: "MULTI_INSTANCE");
16184 #endif
16185
16186 VkMemoryHeapFlags flags = heapInfo.flags &
16187 ~(VK_MEMORY_HEAP_DEVICE_LOCAL_BIT
16188 #if VMA_VULKAN_VERSION >= 1001000
16189 | VK_MEMORY_HEAP_MULTI_INSTANCE_BIT
16190 #endif
16191 );
16192 if (flags != 0)
16193 json.WriteNumber(n: flags);
16194 }
16195 json.EndArray();
16196
16197 json.WriteString(pStr: "Size");
16198 json.WriteNumber(n: heapInfo.size);
16199
16200 json.WriteString(pStr: "Budget");
16201 json.BeginObject();
16202 {
16203 json.WriteString(pStr: "BudgetBytes");
16204 json.WriteNumber(n: budgets[heapIndex].budget);
16205 json.WriteString(pStr: "UsageBytes");
16206 json.WriteNumber(n: budgets[heapIndex].usage);
16207 }
16208 json.EndObject();
16209
16210 json.WriteString(pStr: "Stats");
16211 VmaPrintDetailedStatistics(json, stat: stats.memoryHeap[heapIndex]);
16212
16213 json.WriteString(pStr: "MemoryPools");
16214 json.BeginObject();
16215 {
16216 for (uint32_t typeIndex = 0; typeIndex < allocator->GetMemoryTypeCount(); ++typeIndex)
16217 {
16218 if (allocator->MemoryTypeIndexToHeapIndex(memTypeIndex: typeIndex) == heapIndex)
16219 {
16220 json.BeginString(pStr: "Type ");
16221 json.ContinueString(n: typeIndex);
16222 json.EndString();
16223 json.BeginObject();
16224 {
16225 json.WriteString(pStr: "Flags");
16226 json.BeginArray(singleLine: true);
16227 {
16228 VkMemoryPropertyFlags flags = allocator->m_MemProps.memoryTypes[typeIndex].propertyFlags;
16229 if (flags & VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT)
16230 json.WriteString(pStr: "DEVICE_LOCAL");
16231 if (flags & VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT)
16232 json.WriteString(pStr: "HOST_VISIBLE");
16233 if (flags & VK_MEMORY_PROPERTY_HOST_COHERENT_BIT)
16234 json.WriteString(pStr: "HOST_COHERENT");
16235 if (flags & VK_MEMORY_PROPERTY_HOST_CACHED_BIT)
16236 json.WriteString(pStr: "HOST_CACHED");
16237 if (flags & VK_MEMORY_PROPERTY_LAZILY_ALLOCATED_BIT)
16238 json.WriteString(pStr: "LAZILY_ALLOCATED");
16239 #if VMA_VULKAN_VERSION >= 1001000
16240 if (flags & VK_MEMORY_PROPERTY_PROTECTED_BIT)
16241 json.WriteString(pStr: "PROTECTED");
16242 #endif
16243 #if VK_AMD_device_coherent_memory
16244 if (flags & VK_MEMORY_PROPERTY_DEVICE_COHERENT_BIT_AMD_COPY)
16245 json.WriteString(pStr: "DEVICE_COHERENT_AMD");
16246 if (flags & VK_MEMORY_PROPERTY_DEVICE_UNCACHED_BIT_AMD_COPY)
16247 json.WriteString(pStr: "DEVICE_UNCACHED_AMD");
16248 #endif
16249
16250 flags &= ~(VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT
16251 #if VMA_VULKAN_VERSION >= 1001000
16252 | VK_MEMORY_PROPERTY_LAZILY_ALLOCATED_BIT
16253 #endif
16254 #if VK_AMD_device_coherent_memory
16255 | VK_MEMORY_PROPERTY_DEVICE_COHERENT_BIT_AMD_COPY
16256 | VK_MEMORY_PROPERTY_DEVICE_UNCACHED_BIT_AMD_COPY
16257 #endif
16258 | VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT
16259 | VK_MEMORY_PROPERTY_HOST_COHERENT_BIT
16260 | VK_MEMORY_PROPERTY_HOST_CACHED_BIT);
16261 if (flags != 0)
16262 json.WriteNumber(n: flags);
16263 }
16264 json.EndArray();
16265
16266 json.WriteString(pStr: "Stats");
16267 VmaPrintDetailedStatistics(json, stat: stats.memoryType[typeIndex]);
16268 }
16269 json.EndObject();
16270 }
16271 }
16272
16273 }
16274 json.EndObject();
16275 }
16276 json.EndObject();
16277 }
16278 }
16279 json.EndObject();
16280 }
16281
16282 if (detailedMap == VK_TRUE)
16283 allocator->PrintDetailedMap(json);
16284
16285 json.EndObject();
16286 }
16287
16288 *ppStatsString = VmaCreateStringCopy(allocs: allocator->GetAllocationCallbacks(), srcStr: sb.GetData(), strLen: sb.GetLength());
16289}
16290
16291VMA_CALL_PRE void VMA_CALL_POST vmaFreeStatsString(
16292 VmaAllocator allocator,
16293 char* pStatsString)
16294{
16295 if(pStatsString != VMA_NULL)
16296 {
16297 VMA_ASSERT(allocator);
16298 VmaFreeString(allocs: allocator->GetAllocationCallbacks(), str: pStatsString);
16299 }
16300}
16301
16302#endif // VMA_STATS_STRING_ENABLED
16303
16304/*
16305This function is not protected by any mutex because it just reads immutable data.
16306*/
16307VMA_CALL_PRE VkResult VMA_CALL_POST vmaFindMemoryTypeIndex(
16308 VmaAllocator allocator,
16309 uint32_t memoryTypeBits,
16310 const VmaAllocationCreateInfo* pAllocationCreateInfo,
16311 uint32_t* pMemoryTypeIndex)
16312{
16313 VMA_ASSERT(allocator != VK_NULL_HANDLE);
16314 VMA_ASSERT(pAllocationCreateInfo != VMA_NULL);
16315 VMA_ASSERT(pMemoryTypeIndex != VMA_NULL);
16316
16317 return allocator->FindMemoryTypeIndex(memoryTypeBits, pAllocationCreateInfo, UINT32_MAX, pMemoryTypeIndex);
16318}
16319
16320VMA_CALL_PRE VkResult VMA_CALL_POST vmaFindMemoryTypeIndexForBufferInfo(
16321 VmaAllocator allocator,
16322 const VkBufferCreateInfo* pBufferCreateInfo,
16323 const VmaAllocationCreateInfo* pAllocationCreateInfo,
16324 uint32_t* pMemoryTypeIndex)
16325{
16326 VMA_ASSERT(allocator != VK_NULL_HANDLE);
16327 VMA_ASSERT(pBufferCreateInfo != VMA_NULL);
16328 VMA_ASSERT(pAllocationCreateInfo != VMA_NULL);
16329 VMA_ASSERT(pMemoryTypeIndex != VMA_NULL);
16330
16331 const VkDevice hDev = allocator->m_hDevice;
16332 const VmaVulkanFunctions* funcs = &allocator->GetVulkanFunctions();
16333 VkResult res;
16334
16335#if VMA_VULKAN_VERSION >= 1003000
16336 if(funcs->vkGetDeviceBufferMemoryRequirements)
16337 {
16338 // Can query straight from VkBufferCreateInfo :)
16339 VkDeviceBufferMemoryRequirements devBufMemReq = {.sType: VK_STRUCTURE_TYPE_DEVICE_BUFFER_MEMORY_REQUIREMENTS};
16340 devBufMemReq.pCreateInfo = pBufferCreateInfo;
16341
16342 VkMemoryRequirements2 memReq = {.sType: VK_STRUCTURE_TYPE_MEMORY_REQUIREMENTS_2};
16343 (*funcs->vkGetDeviceBufferMemoryRequirements)(hDev, &devBufMemReq, &memReq);
16344
16345 res = allocator->FindMemoryTypeIndex(
16346 memoryTypeBits: memReq.memoryRequirements.memoryTypeBits, pAllocationCreateInfo, bufImgUsage: pBufferCreateInfo->usage, pMemoryTypeIndex);
16347 }
16348 else
16349#endif // #if VMA_VULKAN_VERSION >= 1003000
16350 {
16351 // Must create a dummy buffer to query :(
16352 VkBuffer hBuffer = VK_NULL_HANDLE;
16353 res = funcs->vkCreateBuffer(
16354 hDev, pBufferCreateInfo, allocator->GetAllocationCallbacks(), &hBuffer);
16355 if(res == VK_SUCCESS)
16356 {
16357 VkMemoryRequirements memReq = {};
16358 funcs->vkGetBufferMemoryRequirements(hDev, hBuffer, &memReq);
16359
16360 res = allocator->FindMemoryTypeIndex(
16361 memoryTypeBits: memReq.memoryTypeBits, pAllocationCreateInfo, bufImgUsage: pBufferCreateInfo->usage, pMemoryTypeIndex);
16362
16363 funcs->vkDestroyBuffer(
16364 hDev, hBuffer, allocator->GetAllocationCallbacks());
16365 }
16366 }
16367 return res;
16368}
16369
16370VMA_CALL_PRE VkResult VMA_CALL_POST vmaFindMemoryTypeIndexForImageInfo(
16371 VmaAllocator allocator,
16372 const VkImageCreateInfo* pImageCreateInfo,
16373 const VmaAllocationCreateInfo* pAllocationCreateInfo,
16374 uint32_t* pMemoryTypeIndex)
16375{
16376 VMA_ASSERT(allocator != VK_NULL_HANDLE);
16377 VMA_ASSERT(pImageCreateInfo != VMA_NULL);
16378 VMA_ASSERT(pAllocationCreateInfo != VMA_NULL);
16379 VMA_ASSERT(pMemoryTypeIndex != VMA_NULL);
16380
16381 const VkDevice hDev = allocator->m_hDevice;
16382 const VmaVulkanFunctions* funcs = &allocator->GetVulkanFunctions();
16383 VkResult res;
16384
16385#if VMA_VULKAN_VERSION >= 1003000
16386 if(funcs->vkGetDeviceImageMemoryRequirements)
16387 {
16388 // Can query straight from VkImageCreateInfo :)
16389 VkDeviceImageMemoryRequirements devImgMemReq = {.sType: VK_STRUCTURE_TYPE_DEVICE_IMAGE_MEMORY_REQUIREMENTS};
16390 devImgMemReq.pCreateInfo = pImageCreateInfo;
16391 VMA_ASSERT(pImageCreateInfo->tiling != VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT_COPY && (pImageCreateInfo->flags & VK_IMAGE_CREATE_DISJOINT_BIT_COPY) == 0 &&
16392 "Cannot use this VkImageCreateInfo with vmaFindMemoryTypeIndexForImageInfo as I don't know what to pass as VkDeviceImageMemoryRequirements::planeAspect.");
16393
16394 VkMemoryRequirements2 memReq = {.sType: VK_STRUCTURE_TYPE_MEMORY_REQUIREMENTS_2};
16395 (*funcs->vkGetDeviceImageMemoryRequirements)(hDev, &devImgMemReq, &memReq);
16396
16397 res = allocator->FindMemoryTypeIndex(
16398 memoryTypeBits: memReq.memoryRequirements.memoryTypeBits, pAllocationCreateInfo, bufImgUsage: pImageCreateInfo->usage, pMemoryTypeIndex);
16399 }
16400 else
16401#endif // #if VMA_VULKAN_VERSION >= 1003000
16402 {
16403 // Must create a dummy image to query :(
16404 VkImage hImage = VK_NULL_HANDLE;
16405 res = funcs->vkCreateImage(
16406 hDev, pImageCreateInfo, allocator->GetAllocationCallbacks(), &hImage);
16407 if(res == VK_SUCCESS)
16408 {
16409 VkMemoryRequirements memReq = {};
16410 funcs->vkGetImageMemoryRequirements(hDev, hImage, &memReq);
16411
16412 res = allocator->FindMemoryTypeIndex(
16413 memoryTypeBits: memReq.memoryTypeBits, pAllocationCreateInfo, bufImgUsage: pImageCreateInfo->usage, pMemoryTypeIndex);
16414
16415 funcs->vkDestroyImage(
16416 hDev, hImage, allocator->GetAllocationCallbacks());
16417 }
16418 }
16419 return res;
16420}
16421
16422VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreatePool(
16423 VmaAllocator allocator,
16424 const VmaPoolCreateInfo* pCreateInfo,
16425 VmaPool* pPool)
16426{
16427 VMA_ASSERT(allocator && pCreateInfo && pPool);
16428
16429 VMA_DEBUG_LOG("vmaCreatePool");
16430
16431 VMA_DEBUG_GLOBAL_MUTEX_LOCK
16432
16433 return allocator->CreatePool(pCreateInfo, pPool);
16434}
16435
16436VMA_CALL_PRE void VMA_CALL_POST vmaDestroyPool(
16437 VmaAllocator allocator,
16438 VmaPool pool)
16439{
16440 VMA_ASSERT(allocator);
16441
16442 if(pool == VK_NULL_HANDLE)
16443 {
16444 return;
16445 }
16446
16447 VMA_DEBUG_LOG("vmaDestroyPool");
16448
16449 VMA_DEBUG_GLOBAL_MUTEX_LOCK
16450
16451 allocator->DestroyPool(pool);
16452}
16453
16454VMA_CALL_PRE void VMA_CALL_POST vmaGetPoolStatistics(
16455 VmaAllocator allocator,
16456 VmaPool pool,
16457 VmaStatistics* pPoolStats)
16458{
16459 VMA_ASSERT(allocator && pool && pPoolStats);
16460
16461 VMA_DEBUG_GLOBAL_MUTEX_LOCK
16462
16463 allocator->GetPoolStatistics(pool, pPoolStats);
16464}
16465
16466VMA_CALL_PRE void VMA_CALL_POST vmaCalculatePoolStatistics(
16467 VmaAllocator allocator,
16468 VmaPool pool,
16469 VmaDetailedStatistics* pPoolStats)
16470{
16471 VMA_ASSERT(allocator && pool && pPoolStats);
16472
16473 VMA_DEBUG_GLOBAL_MUTEX_LOCK
16474
16475 allocator->CalculatePoolStatistics(pool, pPoolStats);
16476}
16477
16478VMA_CALL_PRE VkResult VMA_CALL_POST vmaCheckPoolCorruption(VmaAllocator allocator, VmaPool pool)
16479{
16480 VMA_ASSERT(allocator && pool);
16481
16482 VMA_DEBUG_GLOBAL_MUTEX_LOCK
16483
16484 VMA_DEBUG_LOG("vmaCheckPoolCorruption");
16485
16486 return allocator->CheckPoolCorruption(hPool: pool);
16487}
16488
16489VMA_CALL_PRE void VMA_CALL_POST vmaGetPoolName(
16490 VmaAllocator allocator,
16491 VmaPool pool,
16492 const char** ppName)
16493{
16494 VMA_ASSERT(allocator && pool && ppName);
16495
16496 VMA_DEBUG_LOG("vmaGetPoolName");
16497
16498 VMA_DEBUG_GLOBAL_MUTEX_LOCK
16499
16500 *ppName = pool->GetName();
16501}
16502
16503VMA_CALL_PRE void VMA_CALL_POST vmaSetPoolName(
16504 VmaAllocator allocator,
16505 VmaPool pool,
16506 const char* pName)
16507{
16508 VMA_ASSERT(allocator && pool);
16509
16510 VMA_DEBUG_LOG("vmaSetPoolName");
16511
16512 VMA_DEBUG_GLOBAL_MUTEX_LOCK
16513
16514 pool->SetName(pName);
16515}
16516
16517VMA_CALL_PRE VkResult VMA_CALL_POST vmaAllocateMemory(
16518 VmaAllocator allocator,
16519 const VkMemoryRequirements* pVkMemoryRequirements,
16520 const VmaAllocationCreateInfo* pCreateInfo,
16521 VmaAllocation* pAllocation,
16522 VmaAllocationInfo* pAllocationInfo)
16523{
16524 VMA_ASSERT(allocator && pVkMemoryRequirements && pCreateInfo && pAllocation);
16525
16526 VMA_DEBUG_LOG("vmaAllocateMemory");
16527
16528 VMA_DEBUG_GLOBAL_MUTEX_LOCK
16529
16530 VkResult result = allocator->AllocateMemory(
16531 vkMemReq: *pVkMemoryRequirements,
16532 requiresDedicatedAllocation: false, // requiresDedicatedAllocation
16533 prefersDedicatedAllocation: false, // prefersDedicatedAllocation
16534 VK_NULL_HANDLE, // dedicatedBuffer
16535 VK_NULL_HANDLE, // dedicatedImage
16536 UINT32_MAX, // dedicatedBufferImageUsage
16537 createInfo: *pCreateInfo,
16538 suballocType: VMA_SUBALLOCATION_TYPE_UNKNOWN,
16539 allocationCount: 1, // allocationCount
16540 pAllocations: pAllocation);
16541
16542 if(pAllocationInfo != VMA_NULL && result == VK_SUCCESS)
16543 {
16544 allocator->GetAllocationInfo(hAllocation: *pAllocation, pAllocationInfo);
16545 }
16546
16547 return result;
16548}
16549
16550VMA_CALL_PRE VkResult VMA_CALL_POST vmaAllocateMemoryPages(
16551 VmaAllocator allocator,
16552 const VkMemoryRequirements* pVkMemoryRequirements,
16553 const VmaAllocationCreateInfo* pCreateInfo,
16554 size_t allocationCount,
16555 VmaAllocation* pAllocations,
16556 VmaAllocationInfo* pAllocationInfo)
16557{
16558 if(allocationCount == 0)
16559 {
16560 return VK_SUCCESS;
16561 }
16562
16563 VMA_ASSERT(allocator && pVkMemoryRequirements && pCreateInfo && pAllocations);
16564
16565 VMA_DEBUG_LOG("vmaAllocateMemoryPages");
16566
16567 VMA_DEBUG_GLOBAL_MUTEX_LOCK
16568
16569 VkResult result = allocator->AllocateMemory(
16570 vkMemReq: *pVkMemoryRequirements,
16571 requiresDedicatedAllocation: false, // requiresDedicatedAllocation
16572 prefersDedicatedAllocation: false, // prefersDedicatedAllocation
16573 VK_NULL_HANDLE, // dedicatedBuffer
16574 VK_NULL_HANDLE, // dedicatedImage
16575 UINT32_MAX, // dedicatedBufferImageUsage
16576 createInfo: *pCreateInfo,
16577 suballocType: VMA_SUBALLOCATION_TYPE_UNKNOWN,
16578 allocationCount,
16579 pAllocations);
16580
16581 if(pAllocationInfo != VMA_NULL && result == VK_SUCCESS)
16582 {
16583 for(size_t i = 0; i < allocationCount; ++i)
16584 {
16585 allocator->GetAllocationInfo(hAllocation: pAllocations[i], pAllocationInfo: pAllocationInfo + i);
16586 }
16587 }
16588
16589 return result;
16590}
16591
16592VMA_CALL_PRE VkResult VMA_CALL_POST vmaAllocateMemoryForBuffer(
16593 VmaAllocator allocator,
16594 VkBuffer buffer,
16595 const VmaAllocationCreateInfo* pCreateInfo,
16596 VmaAllocation* pAllocation,
16597 VmaAllocationInfo* pAllocationInfo)
16598{
16599 VMA_ASSERT(allocator && buffer != VK_NULL_HANDLE && pCreateInfo && pAllocation);
16600
16601 VMA_DEBUG_LOG("vmaAllocateMemoryForBuffer");
16602
16603 VMA_DEBUG_GLOBAL_MUTEX_LOCK
16604
16605 VkMemoryRequirements vkMemReq = {};
16606 bool requiresDedicatedAllocation = false;
16607 bool prefersDedicatedAllocation = false;
16608 allocator->GetBufferMemoryRequirements(hBuffer: buffer, memReq&: vkMemReq,
16609 requiresDedicatedAllocation,
16610 prefersDedicatedAllocation);
16611
16612 VkResult result = allocator->AllocateMemory(
16613 vkMemReq,
16614 requiresDedicatedAllocation,
16615 prefersDedicatedAllocation,
16616 dedicatedBuffer: buffer, // dedicatedBuffer
16617 VK_NULL_HANDLE, // dedicatedImage
16618 UINT32_MAX, // dedicatedBufferImageUsage
16619 createInfo: *pCreateInfo,
16620 suballocType: VMA_SUBALLOCATION_TYPE_BUFFER,
16621 allocationCount: 1, // allocationCount
16622 pAllocations: pAllocation);
16623
16624 if(pAllocationInfo && result == VK_SUCCESS)
16625 {
16626 allocator->GetAllocationInfo(hAllocation: *pAllocation, pAllocationInfo);
16627 }
16628
16629 return result;
16630}
16631
16632VMA_CALL_PRE VkResult VMA_CALL_POST vmaAllocateMemoryForImage(
16633 VmaAllocator allocator,
16634 VkImage image,
16635 const VmaAllocationCreateInfo* pCreateInfo,
16636 VmaAllocation* pAllocation,
16637 VmaAllocationInfo* pAllocationInfo)
16638{
16639 VMA_ASSERT(allocator && image != VK_NULL_HANDLE && pCreateInfo && pAllocation);
16640
16641 VMA_DEBUG_LOG("vmaAllocateMemoryForImage");
16642
16643 VMA_DEBUG_GLOBAL_MUTEX_LOCK
16644
16645 VkMemoryRequirements vkMemReq = {};
16646 bool requiresDedicatedAllocation = false;
16647 bool prefersDedicatedAllocation = false;
16648 allocator->GetImageMemoryRequirements(hImage: image, memReq&: vkMemReq,
16649 requiresDedicatedAllocation, prefersDedicatedAllocation);
16650
16651 VkResult result = allocator->AllocateMemory(
16652 vkMemReq,
16653 requiresDedicatedAllocation,
16654 prefersDedicatedAllocation,
16655 VK_NULL_HANDLE, // dedicatedBuffer
16656 dedicatedImage: image, // dedicatedImage
16657 UINT32_MAX, // dedicatedBufferImageUsage
16658 createInfo: *pCreateInfo,
16659 suballocType: VMA_SUBALLOCATION_TYPE_IMAGE_UNKNOWN,
16660 allocationCount: 1, // allocationCount
16661 pAllocations: pAllocation);
16662
16663 if(pAllocationInfo && result == VK_SUCCESS)
16664 {
16665 allocator->GetAllocationInfo(hAllocation: *pAllocation, pAllocationInfo);
16666 }
16667
16668 return result;
16669}
16670
16671VMA_CALL_PRE void VMA_CALL_POST vmaFreeMemory(
16672 VmaAllocator allocator,
16673 VmaAllocation allocation)
16674{
16675 VMA_ASSERT(allocator);
16676
16677 if(allocation == VK_NULL_HANDLE)
16678 {
16679 return;
16680 }
16681
16682 VMA_DEBUG_LOG("vmaFreeMemory");
16683
16684 VMA_DEBUG_GLOBAL_MUTEX_LOCK
16685
16686 allocator->FreeMemory(
16687 allocationCount: 1, // allocationCount
16688 pAllocations: &allocation);
16689}
16690
16691VMA_CALL_PRE void VMA_CALL_POST vmaFreeMemoryPages(
16692 VmaAllocator allocator,
16693 size_t allocationCount,
16694 const VmaAllocation* pAllocations)
16695{
16696 if(allocationCount == 0)
16697 {
16698 return;
16699 }
16700
16701 VMA_ASSERT(allocator);
16702
16703 VMA_DEBUG_LOG("vmaFreeMemoryPages");
16704
16705 VMA_DEBUG_GLOBAL_MUTEX_LOCK
16706
16707 allocator->FreeMemory(allocationCount, pAllocations);
16708}
16709
16710VMA_CALL_PRE void VMA_CALL_POST vmaGetAllocationInfo(
16711 VmaAllocator allocator,
16712 VmaAllocation allocation,
16713 VmaAllocationInfo* pAllocationInfo)
16714{
16715 VMA_ASSERT(allocator && allocation && pAllocationInfo);
16716
16717 VMA_DEBUG_GLOBAL_MUTEX_LOCK
16718
16719 allocator->GetAllocationInfo(hAllocation: allocation, pAllocationInfo);
16720}
16721
16722VMA_CALL_PRE void VMA_CALL_POST vmaSetAllocationUserData(
16723 VmaAllocator allocator,
16724 VmaAllocation allocation,
16725 void* pUserData)
16726{
16727 VMA_ASSERT(allocator && allocation);
16728
16729 VMA_DEBUG_GLOBAL_MUTEX_LOCK
16730
16731 allocation->SetUserData(hAllocator: allocator, pUserData);
16732}
16733
16734VMA_CALL_PRE void VMA_CALL_POST vmaSetAllocationName(
16735 VmaAllocator VMA_NOT_NULL allocator,
16736 VmaAllocation VMA_NOT_NULL allocation,
16737 const char* VMA_NULLABLE pName)
16738{
16739 allocation->SetName(hAllocator: allocator, pName);
16740}
16741
16742VMA_CALL_PRE void VMA_CALL_POST vmaGetAllocationMemoryProperties(
16743 VmaAllocator VMA_NOT_NULL allocator,
16744 VmaAllocation VMA_NOT_NULL allocation,
16745 VkMemoryPropertyFlags* VMA_NOT_NULL pFlags)
16746{
16747 VMA_ASSERT(allocator && allocation && pFlags);
16748 const uint32_t memTypeIndex = allocation->GetMemoryTypeIndex();
16749 *pFlags = allocator->m_MemProps.memoryTypes[memTypeIndex].propertyFlags;
16750}
16751
16752VMA_CALL_PRE VkResult VMA_CALL_POST vmaMapMemory(
16753 VmaAllocator allocator,
16754 VmaAllocation allocation,
16755 void** ppData)
16756{
16757 VMA_ASSERT(allocator && allocation && ppData);
16758
16759 VMA_DEBUG_GLOBAL_MUTEX_LOCK
16760
16761 return allocator->Map(hAllocation: allocation, ppData);
16762}
16763
16764VMA_CALL_PRE void VMA_CALL_POST vmaUnmapMemory(
16765 VmaAllocator allocator,
16766 VmaAllocation allocation)
16767{
16768 VMA_ASSERT(allocator && allocation);
16769
16770 VMA_DEBUG_GLOBAL_MUTEX_LOCK
16771
16772 allocator->Unmap(hAllocation: allocation);
16773}
16774
16775VMA_CALL_PRE VkResult VMA_CALL_POST vmaFlushAllocation(
16776 VmaAllocator allocator,
16777 VmaAllocation allocation,
16778 VkDeviceSize offset,
16779 VkDeviceSize size)
16780{
16781 VMA_ASSERT(allocator && allocation);
16782
16783 VMA_DEBUG_LOG("vmaFlushAllocation");
16784
16785 VMA_DEBUG_GLOBAL_MUTEX_LOCK
16786
16787 const VkResult res = allocator->FlushOrInvalidateAllocation(hAllocation: allocation, offset, size, op: VMA_CACHE_FLUSH);
16788
16789 return res;
16790}
16791
16792VMA_CALL_PRE VkResult VMA_CALL_POST vmaInvalidateAllocation(
16793 VmaAllocator allocator,
16794 VmaAllocation allocation,
16795 VkDeviceSize offset,
16796 VkDeviceSize size)
16797{
16798 VMA_ASSERT(allocator && allocation);
16799
16800 VMA_DEBUG_LOG("vmaInvalidateAllocation");
16801
16802 VMA_DEBUG_GLOBAL_MUTEX_LOCK
16803
16804 const VkResult res = allocator->FlushOrInvalidateAllocation(hAllocation: allocation, offset, size, op: VMA_CACHE_INVALIDATE);
16805
16806 return res;
16807}
16808
16809VMA_CALL_PRE VkResult VMA_CALL_POST vmaFlushAllocations(
16810 VmaAllocator allocator,
16811 uint32_t allocationCount,
16812 const VmaAllocation* allocations,
16813 const VkDeviceSize* offsets,
16814 const VkDeviceSize* sizes)
16815{
16816 VMA_ASSERT(allocator);
16817
16818 if(allocationCount == 0)
16819 {
16820 return VK_SUCCESS;
16821 }
16822
16823 VMA_ASSERT(allocations);
16824
16825 VMA_DEBUG_LOG("vmaFlushAllocations");
16826
16827 VMA_DEBUG_GLOBAL_MUTEX_LOCK
16828
16829 const VkResult res = allocator->FlushOrInvalidateAllocations(allocationCount, allocations, offsets, sizes, op: VMA_CACHE_FLUSH);
16830
16831 return res;
16832}
16833
16834VMA_CALL_PRE VkResult VMA_CALL_POST vmaInvalidateAllocations(
16835 VmaAllocator allocator,
16836 uint32_t allocationCount,
16837 const VmaAllocation* allocations,
16838 const VkDeviceSize* offsets,
16839 const VkDeviceSize* sizes)
16840{
16841 VMA_ASSERT(allocator);
16842
16843 if(allocationCount == 0)
16844 {
16845 return VK_SUCCESS;
16846 }
16847
16848 VMA_ASSERT(allocations);
16849
16850 VMA_DEBUG_LOG("vmaInvalidateAllocations");
16851
16852 VMA_DEBUG_GLOBAL_MUTEX_LOCK
16853
16854 const VkResult res = allocator->FlushOrInvalidateAllocations(allocationCount, allocations, offsets, sizes, op: VMA_CACHE_INVALIDATE);
16855
16856 return res;
16857}
16858
16859VMA_CALL_PRE VkResult VMA_CALL_POST vmaCheckCorruption(
16860 VmaAllocator allocator,
16861 uint32_t memoryTypeBits)
16862{
16863 VMA_ASSERT(allocator);
16864
16865 VMA_DEBUG_LOG("vmaCheckCorruption");
16866
16867 VMA_DEBUG_GLOBAL_MUTEX_LOCK
16868
16869 return allocator->CheckCorruption(memoryTypeBits);
16870}
16871
16872VMA_CALL_PRE VkResult VMA_CALL_POST vmaBeginDefragmentation(
16873 VmaAllocator allocator,
16874 const VmaDefragmentationInfo* pInfo,
16875 VmaDefragmentationContext* pContext)
16876{
16877 VMA_ASSERT(allocator && pInfo && pContext);
16878
16879 VMA_DEBUG_LOG("vmaBeginDefragmentation");
16880
16881 if (pInfo->pool != VMA_NULL)
16882 {
16883 // Check if run on supported algorithms
16884 if (pInfo->pool->m_BlockVector.GetAlgorithm() & VMA_POOL_CREATE_LINEAR_ALGORITHM_BIT)
16885 return VK_ERROR_FEATURE_NOT_PRESENT;
16886 }
16887
16888 VMA_DEBUG_GLOBAL_MUTEX_LOCK
16889
16890 *pContext = vma_new(allocator, VmaDefragmentationContext_T)(allocator, *pInfo);
16891 return VK_SUCCESS;
16892}
16893
16894VMA_CALL_PRE void VMA_CALL_POST vmaEndDefragmentation(
16895 VmaAllocator allocator,
16896 VmaDefragmentationContext context,
16897 VmaDefragmentationStats* pStats)
16898{
16899 VMA_ASSERT(allocator && context);
16900
16901 VMA_DEBUG_LOG("vmaEndDefragmentation");
16902
16903 VMA_DEBUG_GLOBAL_MUTEX_LOCK
16904
16905 if (pStats)
16906 context->GetStats(outStats&: *pStats);
16907 vma_delete(hAllocator: allocator, ptr: context);
16908}
16909
16910VMA_CALL_PRE VkResult VMA_CALL_POST vmaBeginDefragmentationPass(
16911 VmaAllocator VMA_NOT_NULL allocator,
16912 VmaDefragmentationContext VMA_NOT_NULL context,
16913 VmaDefragmentationPassMoveInfo* VMA_NOT_NULL pPassInfo)
16914{
16915 VMA_ASSERT(context && pPassInfo);
16916
16917 VMA_DEBUG_LOG("vmaBeginDefragmentationPass");
16918
16919 VMA_DEBUG_GLOBAL_MUTEX_LOCK
16920
16921 return context->DefragmentPassBegin(moveInfo&: *pPassInfo);
16922}
16923
16924VMA_CALL_PRE VkResult VMA_CALL_POST vmaEndDefragmentationPass(
16925 VmaAllocator VMA_NOT_NULL allocator,
16926 VmaDefragmentationContext VMA_NOT_NULL context,
16927 VmaDefragmentationPassMoveInfo* VMA_NOT_NULL pPassInfo)
16928{
16929 VMA_ASSERT(context && pPassInfo);
16930
16931 VMA_DEBUG_LOG("vmaEndDefragmentationPass");
16932
16933 VMA_DEBUG_GLOBAL_MUTEX_LOCK
16934
16935 return context->DefragmentPassEnd(moveInfo&: *pPassInfo);
16936}
16937
16938VMA_CALL_PRE VkResult VMA_CALL_POST vmaBindBufferMemory(
16939 VmaAllocator allocator,
16940 VmaAllocation allocation,
16941 VkBuffer buffer)
16942{
16943 VMA_ASSERT(allocator && allocation && buffer);
16944
16945 VMA_DEBUG_LOG("vmaBindBufferMemory");
16946
16947 VMA_DEBUG_GLOBAL_MUTEX_LOCK
16948
16949 return allocator->BindBufferMemory(hAllocation: allocation, allocationLocalOffset: 0, hBuffer: buffer, VMA_NULL);
16950}
16951
16952VMA_CALL_PRE VkResult VMA_CALL_POST vmaBindBufferMemory2(
16953 VmaAllocator allocator,
16954 VmaAllocation allocation,
16955 VkDeviceSize allocationLocalOffset,
16956 VkBuffer buffer,
16957 const void* pNext)
16958{
16959 VMA_ASSERT(allocator && allocation && buffer);
16960
16961 VMA_DEBUG_LOG("vmaBindBufferMemory2");
16962
16963 VMA_DEBUG_GLOBAL_MUTEX_LOCK
16964
16965 return allocator->BindBufferMemory(hAllocation: allocation, allocationLocalOffset, hBuffer: buffer, pNext);
16966}
16967
16968VMA_CALL_PRE VkResult VMA_CALL_POST vmaBindImageMemory(
16969 VmaAllocator allocator,
16970 VmaAllocation allocation,
16971 VkImage image)
16972{
16973 VMA_ASSERT(allocator && allocation && image);
16974
16975 VMA_DEBUG_LOG("vmaBindImageMemory");
16976
16977 VMA_DEBUG_GLOBAL_MUTEX_LOCK
16978
16979 return allocator->BindImageMemory(hAllocation: allocation, allocationLocalOffset: 0, hImage: image, VMA_NULL);
16980}
16981
16982VMA_CALL_PRE VkResult VMA_CALL_POST vmaBindImageMemory2(
16983 VmaAllocator allocator,
16984 VmaAllocation allocation,
16985 VkDeviceSize allocationLocalOffset,
16986 VkImage image,
16987 const void* pNext)
16988{
16989 VMA_ASSERT(allocator && allocation && image);
16990
16991 VMA_DEBUG_LOG("vmaBindImageMemory2");
16992
16993 VMA_DEBUG_GLOBAL_MUTEX_LOCK
16994
16995 return allocator->BindImageMemory(hAllocation: allocation, allocationLocalOffset, hImage: image, pNext);
16996}
16997
16998VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateBuffer(
16999 VmaAllocator allocator,
17000 const VkBufferCreateInfo* pBufferCreateInfo,
17001 const VmaAllocationCreateInfo* pAllocationCreateInfo,
17002 VkBuffer* pBuffer,
17003 VmaAllocation* pAllocation,
17004 VmaAllocationInfo* pAllocationInfo)
17005{
17006 VMA_ASSERT(allocator && pBufferCreateInfo && pAllocationCreateInfo && pBuffer && pAllocation);
17007
17008 if(pBufferCreateInfo->size == 0)
17009 {
17010 return VK_ERROR_INITIALIZATION_FAILED;
17011 }
17012 if((pBufferCreateInfo->usage & VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT_COPY) != 0 &&
17013 !allocator->m_UseKhrBufferDeviceAddress)
17014 {
17015 VMA_ASSERT(0 && "Creating a buffer with VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT is not valid if VMA_ALLOCATOR_CREATE_BUFFER_DEVICE_ADDRESS_BIT was not used.");
17016 return VK_ERROR_INITIALIZATION_FAILED;
17017 }
17018
17019 VMA_DEBUG_LOG("vmaCreateBuffer");
17020
17021 VMA_DEBUG_GLOBAL_MUTEX_LOCK
17022
17023 *pBuffer = VK_NULL_HANDLE;
17024 *pAllocation = VK_NULL_HANDLE;
17025
17026 // 1. Create VkBuffer.
17027 VkResult res = (*allocator->GetVulkanFunctions().vkCreateBuffer)(
17028 allocator->m_hDevice,
17029 pBufferCreateInfo,
17030 allocator->GetAllocationCallbacks(),
17031 pBuffer);
17032 if(res >= 0)
17033 {
17034 // 2. vkGetBufferMemoryRequirements.
17035 VkMemoryRequirements vkMemReq = {};
17036 bool requiresDedicatedAllocation = false;
17037 bool prefersDedicatedAllocation = false;
17038 allocator->GetBufferMemoryRequirements(hBuffer: *pBuffer, memReq&: vkMemReq,
17039 requiresDedicatedAllocation, prefersDedicatedAllocation);
17040
17041 // 3. Allocate memory using allocator.
17042 res = allocator->AllocateMemory(
17043 vkMemReq,
17044 requiresDedicatedAllocation,
17045 prefersDedicatedAllocation,
17046 dedicatedBuffer: *pBuffer, // dedicatedBuffer
17047 VK_NULL_HANDLE, // dedicatedImage
17048 dedicatedBufferImageUsage: pBufferCreateInfo->usage, // dedicatedBufferImageUsage
17049 createInfo: *pAllocationCreateInfo,
17050 suballocType: VMA_SUBALLOCATION_TYPE_BUFFER,
17051 allocationCount: 1, // allocationCount
17052 pAllocations: pAllocation);
17053
17054 if(res >= 0)
17055 {
17056 // 3. Bind buffer with memory.
17057 if((pAllocationCreateInfo->flags & VMA_ALLOCATION_CREATE_DONT_BIND_BIT) == 0)
17058 {
17059 res = allocator->BindBufferMemory(hAllocation: *pAllocation, allocationLocalOffset: 0, hBuffer: *pBuffer, VMA_NULL);
17060 }
17061 if(res >= 0)
17062 {
17063 // All steps succeeded.
17064 #if VMA_STATS_STRING_ENABLED
17065 (*pAllocation)->InitBufferImageUsage(bufferImageUsage: pBufferCreateInfo->usage);
17066 #endif
17067 if(pAllocationInfo != VMA_NULL)
17068 {
17069 allocator->GetAllocationInfo(hAllocation: *pAllocation, pAllocationInfo);
17070 }
17071
17072 return VK_SUCCESS;
17073 }
17074 allocator->FreeMemory(
17075 allocationCount: 1, // allocationCount
17076 pAllocations: pAllocation);
17077 *pAllocation = VK_NULL_HANDLE;
17078 (*allocator->GetVulkanFunctions().vkDestroyBuffer)(allocator->m_hDevice, *pBuffer, allocator->GetAllocationCallbacks());
17079 *pBuffer = VK_NULL_HANDLE;
17080 return res;
17081 }
17082 (*allocator->GetVulkanFunctions().vkDestroyBuffer)(allocator->m_hDevice, *pBuffer, allocator->GetAllocationCallbacks());
17083 *pBuffer = VK_NULL_HANDLE;
17084 return res;
17085 }
17086 return res;
17087}
17088
17089VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateBufferWithAlignment(
17090 VmaAllocator allocator,
17091 const VkBufferCreateInfo* pBufferCreateInfo,
17092 const VmaAllocationCreateInfo* pAllocationCreateInfo,
17093 VkDeviceSize minAlignment,
17094 VkBuffer* pBuffer,
17095 VmaAllocation* pAllocation,
17096 VmaAllocationInfo* pAllocationInfo)
17097{
17098 VMA_ASSERT(allocator && pBufferCreateInfo && pAllocationCreateInfo && VmaIsPow2(minAlignment) && pBuffer && pAllocation);
17099
17100 if(pBufferCreateInfo->size == 0)
17101 {
17102 return VK_ERROR_INITIALIZATION_FAILED;
17103 }
17104 if((pBufferCreateInfo->usage & VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT_COPY) != 0 &&
17105 !allocator->m_UseKhrBufferDeviceAddress)
17106 {
17107 VMA_ASSERT(0 && "Creating a buffer with VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT is not valid if VMA_ALLOCATOR_CREATE_BUFFER_DEVICE_ADDRESS_BIT was not used.");
17108 return VK_ERROR_INITIALIZATION_FAILED;
17109 }
17110
17111 VMA_DEBUG_LOG("vmaCreateBufferWithAlignment");
17112
17113 VMA_DEBUG_GLOBAL_MUTEX_LOCK
17114
17115 *pBuffer = VK_NULL_HANDLE;
17116 *pAllocation = VK_NULL_HANDLE;
17117
17118 // 1. Create VkBuffer.
17119 VkResult res = (*allocator->GetVulkanFunctions().vkCreateBuffer)(
17120 allocator->m_hDevice,
17121 pBufferCreateInfo,
17122 allocator->GetAllocationCallbacks(),
17123 pBuffer);
17124 if(res >= 0)
17125 {
17126 // 2. vkGetBufferMemoryRequirements.
17127 VkMemoryRequirements vkMemReq = {};
17128 bool requiresDedicatedAllocation = false;
17129 bool prefersDedicatedAllocation = false;
17130 allocator->GetBufferMemoryRequirements(hBuffer: *pBuffer, memReq&: vkMemReq,
17131 requiresDedicatedAllocation, prefersDedicatedAllocation);
17132
17133 // 2a. Include minAlignment
17134 vkMemReq.alignment = VMA_MAX(vkMemReq.alignment, minAlignment);
17135
17136 // 3. Allocate memory using allocator.
17137 res = allocator->AllocateMemory(
17138 vkMemReq,
17139 requiresDedicatedAllocation,
17140 prefersDedicatedAllocation,
17141 dedicatedBuffer: *pBuffer, // dedicatedBuffer
17142 VK_NULL_HANDLE, // dedicatedImage
17143 dedicatedBufferImageUsage: pBufferCreateInfo->usage, // dedicatedBufferImageUsage
17144 createInfo: *pAllocationCreateInfo,
17145 suballocType: VMA_SUBALLOCATION_TYPE_BUFFER,
17146 allocationCount: 1, // allocationCount
17147 pAllocations: pAllocation);
17148
17149 if(res >= 0)
17150 {
17151 // 3. Bind buffer with memory.
17152 if((pAllocationCreateInfo->flags & VMA_ALLOCATION_CREATE_DONT_BIND_BIT) == 0)
17153 {
17154 res = allocator->BindBufferMemory(hAllocation: *pAllocation, allocationLocalOffset: 0, hBuffer: *pBuffer, VMA_NULL);
17155 }
17156 if(res >= 0)
17157 {
17158 // All steps succeeded.
17159 #if VMA_STATS_STRING_ENABLED
17160 (*pAllocation)->InitBufferImageUsage(bufferImageUsage: pBufferCreateInfo->usage);
17161 #endif
17162 if(pAllocationInfo != VMA_NULL)
17163 {
17164 allocator->GetAllocationInfo(hAllocation: *pAllocation, pAllocationInfo);
17165 }
17166
17167 return VK_SUCCESS;
17168 }
17169 allocator->FreeMemory(
17170 allocationCount: 1, // allocationCount
17171 pAllocations: pAllocation);
17172 *pAllocation = VK_NULL_HANDLE;
17173 (*allocator->GetVulkanFunctions().vkDestroyBuffer)(allocator->m_hDevice, *pBuffer, allocator->GetAllocationCallbacks());
17174 *pBuffer = VK_NULL_HANDLE;
17175 return res;
17176 }
17177 (*allocator->GetVulkanFunctions().vkDestroyBuffer)(allocator->m_hDevice, *pBuffer, allocator->GetAllocationCallbacks());
17178 *pBuffer = VK_NULL_HANDLE;
17179 return res;
17180 }
17181 return res;
17182}
17183
17184VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateAliasingBuffer(
17185 VmaAllocator VMA_NOT_NULL allocator,
17186 VmaAllocation VMA_NOT_NULL allocation,
17187 const VkBufferCreateInfo* VMA_NOT_NULL pBufferCreateInfo,
17188 VkBuffer VMA_NULLABLE_NON_DISPATCHABLE* VMA_NOT_NULL pBuffer)
17189{
17190 VMA_ASSERT(allocator && pBufferCreateInfo && pBuffer && allocation);
17191
17192 VMA_DEBUG_LOG("vmaCreateAliasingBuffer");
17193
17194 *pBuffer = VK_NULL_HANDLE;
17195
17196 if (pBufferCreateInfo->size == 0)
17197 {
17198 return VK_ERROR_INITIALIZATION_FAILED;
17199 }
17200 if ((pBufferCreateInfo->usage & VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT_COPY) != 0 &&
17201 !allocator->m_UseKhrBufferDeviceAddress)
17202 {
17203 VMA_ASSERT(0 && "Creating a buffer with VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT is not valid if VMA_ALLOCATOR_CREATE_BUFFER_DEVICE_ADDRESS_BIT was not used.");
17204 return VK_ERROR_INITIALIZATION_FAILED;
17205 }
17206
17207 VMA_DEBUG_GLOBAL_MUTEX_LOCK
17208
17209 // 1. Create VkBuffer.
17210 VkResult res = (*allocator->GetVulkanFunctions().vkCreateBuffer)(
17211 allocator->m_hDevice,
17212 pBufferCreateInfo,
17213 allocator->GetAllocationCallbacks(),
17214 pBuffer);
17215 if (res >= 0)
17216 {
17217 // 2. Bind buffer with memory.
17218 res = allocator->BindBufferMemory(hAllocation: allocation, allocationLocalOffset: 0, hBuffer: *pBuffer, VMA_NULL);
17219 if (res >= 0)
17220 {
17221 return VK_SUCCESS;
17222 }
17223 (*allocator->GetVulkanFunctions().vkDestroyBuffer)(allocator->m_hDevice, *pBuffer, allocator->GetAllocationCallbacks());
17224 }
17225 return res;
17226}
17227
17228VMA_CALL_PRE void VMA_CALL_POST vmaDestroyBuffer(
17229 VmaAllocator allocator,
17230 VkBuffer buffer,
17231 VmaAllocation allocation)
17232{
17233 VMA_ASSERT(allocator);
17234
17235 if(buffer == VK_NULL_HANDLE && allocation == VK_NULL_HANDLE)
17236 {
17237 return;
17238 }
17239
17240 VMA_DEBUG_LOG("vmaDestroyBuffer");
17241
17242 VMA_DEBUG_GLOBAL_MUTEX_LOCK
17243
17244 if(buffer != VK_NULL_HANDLE)
17245 {
17246 (*allocator->GetVulkanFunctions().vkDestroyBuffer)(allocator->m_hDevice, buffer, allocator->GetAllocationCallbacks());
17247 }
17248
17249 if(allocation != VK_NULL_HANDLE)
17250 {
17251 allocator->FreeMemory(
17252 allocationCount: 1, // allocationCount
17253 pAllocations: &allocation);
17254 }
17255}
17256
17257VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateImage(
17258 VmaAllocator allocator,
17259 const VkImageCreateInfo* pImageCreateInfo,
17260 const VmaAllocationCreateInfo* pAllocationCreateInfo,
17261 VkImage* pImage,
17262 VmaAllocation* pAllocation,
17263 VmaAllocationInfo* pAllocationInfo)
17264{
17265 VMA_ASSERT(allocator && pImageCreateInfo && pAllocationCreateInfo && pImage && pAllocation);
17266
17267 if(pImageCreateInfo->extent.width == 0 ||
17268 pImageCreateInfo->extent.height == 0 ||
17269 pImageCreateInfo->extent.depth == 0 ||
17270 pImageCreateInfo->mipLevels == 0 ||
17271 pImageCreateInfo->arrayLayers == 0)
17272 {
17273 return VK_ERROR_INITIALIZATION_FAILED;
17274 }
17275
17276 VMA_DEBUG_LOG("vmaCreateImage");
17277
17278 VMA_DEBUG_GLOBAL_MUTEX_LOCK
17279
17280 *pImage = VK_NULL_HANDLE;
17281 *pAllocation = VK_NULL_HANDLE;
17282
17283 // 1. Create VkImage.
17284 VkResult res = (*allocator->GetVulkanFunctions().vkCreateImage)(
17285 allocator->m_hDevice,
17286 pImageCreateInfo,
17287 allocator->GetAllocationCallbacks(),
17288 pImage);
17289 if(res >= 0)
17290 {
17291 VmaSuballocationType suballocType = pImageCreateInfo->tiling == VK_IMAGE_TILING_OPTIMAL ?
17292 VMA_SUBALLOCATION_TYPE_IMAGE_OPTIMAL :
17293 VMA_SUBALLOCATION_TYPE_IMAGE_LINEAR;
17294
17295 // 2. Allocate memory using allocator.
17296 VkMemoryRequirements vkMemReq = {};
17297 bool requiresDedicatedAllocation = false;
17298 bool prefersDedicatedAllocation = false;
17299 allocator->GetImageMemoryRequirements(hImage: *pImage, memReq&: vkMemReq,
17300 requiresDedicatedAllocation, prefersDedicatedAllocation);
17301
17302 res = allocator->AllocateMemory(
17303 vkMemReq,
17304 requiresDedicatedAllocation,
17305 prefersDedicatedAllocation,
17306 VK_NULL_HANDLE, // dedicatedBuffer
17307 dedicatedImage: *pImage, // dedicatedImage
17308 dedicatedBufferImageUsage: pImageCreateInfo->usage, // dedicatedBufferImageUsage
17309 createInfo: *pAllocationCreateInfo,
17310 suballocType,
17311 allocationCount: 1, // allocationCount
17312 pAllocations: pAllocation);
17313
17314 if(res >= 0)
17315 {
17316 // 3. Bind image with memory.
17317 if((pAllocationCreateInfo->flags & VMA_ALLOCATION_CREATE_DONT_BIND_BIT) == 0)
17318 {
17319 res = allocator->BindImageMemory(hAllocation: *pAllocation, allocationLocalOffset: 0, hImage: *pImage, VMA_NULL);
17320 }
17321 if(res >= 0)
17322 {
17323 // All steps succeeded.
17324 #if VMA_STATS_STRING_ENABLED
17325 (*pAllocation)->InitBufferImageUsage(bufferImageUsage: pImageCreateInfo->usage);
17326 #endif
17327 if(pAllocationInfo != VMA_NULL)
17328 {
17329 allocator->GetAllocationInfo(hAllocation: *pAllocation, pAllocationInfo);
17330 }
17331
17332 return VK_SUCCESS;
17333 }
17334 allocator->FreeMemory(
17335 allocationCount: 1, // allocationCount
17336 pAllocations: pAllocation);
17337 *pAllocation = VK_NULL_HANDLE;
17338 (*allocator->GetVulkanFunctions().vkDestroyImage)(allocator->m_hDevice, *pImage, allocator->GetAllocationCallbacks());
17339 *pImage = VK_NULL_HANDLE;
17340 return res;
17341 }
17342 (*allocator->GetVulkanFunctions().vkDestroyImage)(allocator->m_hDevice, *pImage, allocator->GetAllocationCallbacks());
17343 *pImage = VK_NULL_HANDLE;
17344 return res;
17345 }
17346 return res;
17347}
17348
17349VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateAliasingImage(
17350 VmaAllocator VMA_NOT_NULL allocator,
17351 VmaAllocation VMA_NOT_NULL allocation,
17352 const VkImageCreateInfo* VMA_NOT_NULL pImageCreateInfo,
17353 VkImage VMA_NULLABLE_NON_DISPATCHABLE* VMA_NOT_NULL pImage)
17354{
17355 VMA_ASSERT(allocator && pImageCreateInfo && pImage && allocation);
17356
17357 *pImage = VK_NULL_HANDLE;
17358
17359 VMA_DEBUG_LOG("vmaCreateImage");
17360
17361 if (pImageCreateInfo->extent.width == 0 ||
17362 pImageCreateInfo->extent.height == 0 ||
17363 pImageCreateInfo->extent.depth == 0 ||
17364 pImageCreateInfo->mipLevels == 0 ||
17365 pImageCreateInfo->arrayLayers == 0)
17366 {
17367 return VK_ERROR_INITIALIZATION_FAILED;
17368 }
17369
17370 VMA_DEBUG_GLOBAL_MUTEX_LOCK
17371
17372 // 1. Create VkImage.
17373 VkResult res = (*allocator->GetVulkanFunctions().vkCreateImage)(
17374 allocator->m_hDevice,
17375 pImageCreateInfo,
17376 allocator->GetAllocationCallbacks(),
17377 pImage);
17378 if (res >= 0)
17379 {
17380 // 2. Bind image with memory.
17381 res = allocator->BindImageMemory(hAllocation: allocation, allocationLocalOffset: 0, hImage: *pImage, VMA_NULL);
17382 if (res >= 0)
17383 {
17384 return VK_SUCCESS;
17385 }
17386 (*allocator->GetVulkanFunctions().vkDestroyImage)(allocator->m_hDevice, *pImage, allocator->GetAllocationCallbacks());
17387 }
17388 return res;
17389}
17390
17391VMA_CALL_PRE void VMA_CALL_POST vmaDestroyImage(
17392 VmaAllocator VMA_NOT_NULL allocator,
17393 VkImage VMA_NULLABLE_NON_DISPATCHABLE image,
17394 VmaAllocation VMA_NULLABLE allocation)
17395{
17396 VMA_ASSERT(allocator);
17397
17398 if(image == VK_NULL_HANDLE && allocation == VK_NULL_HANDLE)
17399 {
17400 return;
17401 }
17402
17403 VMA_DEBUG_LOG("vmaDestroyImage");
17404
17405 VMA_DEBUG_GLOBAL_MUTEX_LOCK
17406
17407 if(image != VK_NULL_HANDLE)
17408 {
17409 (*allocator->GetVulkanFunctions().vkDestroyImage)(allocator->m_hDevice, image, allocator->GetAllocationCallbacks());
17410 }
17411 if(allocation != VK_NULL_HANDLE)
17412 {
17413 allocator->FreeMemory(
17414 allocationCount: 1, // allocationCount
17415 pAllocations: &allocation);
17416 }
17417}
17418
17419VMA_CALL_PRE VkResult VMA_CALL_POST vmaCreateVirtualBlock(
17420 const VmaVirtualBlockCreateInfo* VMA_NOT_NULL pCreateInfo,
17421 VmaVirtualBlock VMA_NULLABLE * VMA_NOT_NULL pVirtualBlock)
17422{
17423 VMA_ASSERT(pCreateInfo && pVirtualBlock);
17424 VMA_ASSERT(pCreateInfo->size > 0);
17425 VMA_DEBUG_LOG("vmaCreateVirtualBlock");
17426 VMA_DEBUG_GLOBAL_MUTEX_LOCK;
17427 *pVirtualBlock = vma_new(pCreateInfo->pAllocationCallbacks, VmaVirtualBlock_T)(*pCreateInfo);
17428 VkResult res = (*pVirtualBlock)->Init();
17429 if(res < 0)
17430 {
17431 vma_delete(pAllocationCallbacks: pCreateInfo->pAllocationCallbacks, ptr: *pVirtualBlock);
17432 *pVirtualBlock = VK_NULL_HANDLE;
17433 }
17434 return res;
17435}
17436
17437VMA_CALL_PRE void VMA_CALL_POST vmaDestroyVirtualBlock(VmaVirtualBlock VMA_NULLABLE virtualBlock)
17438{
17439 if(virtualBlock != VK_NULL_HANDLE)
17440 {
17441 VMA_DEBUG_LOG("vmaDestroyVirtualBlock");
17442 VMA_DEBUG_GLOBAL_MUTEX_LOCK;
17443 VkAllocationCallbacks allocationCallbacks = virtualBlock->m_AllocationCallbacks; // Have to copy the callbacks when destroying.
17444 vma_delete(pAllocationCallbacks: &allocationCallbacks, ptr: virtualBlock);
17445 }
17446}
17447
17448VMA_CALL_PRE VkBool32 VMA_CALL_POST vmaIsVirtualBlockEmpty(VmaVirtualBlock VMA_NOT_NULL virtualBlock)
17449{
17450 VMA_ASSERT(virtualBlock != VK_NULL_HANDLE);
17451 VMA_DEBUG_LOG("vmaIsVirtualBlockEmpty");
17452 VMA_DEBUG_GLOBAL_MUTEX_LOCK;
17453 return virtualBlock->IsEmpty() ? VK_TRUE : VK_FALSE;
17454}
17455
17456VMA_CALL_PRE void VMA_CALL_POST vmaGetVirtualAllocationInfo(VmaVirtualBlock VMA_NOT_NULL virtualBlock,
17457 VmaVirtualAllocation VMA_NOT_NULL_NON_DISPATCHABLE allocation, VmaVirtualAllocationInfo* VMA_NOT_NULL pVirtualAllocInfo)
17458{
17459 VMA_ASSERT(virtualBlock != VK_NULL_HANDLE && pVirtualAllocInfo != VMA_NULL);
17460 VMA_DEBUG_LOG("vmaGetVirtualAllocationInfo");
17461 VMA_DEBUG_GLOBAL_MUTEX_LOCK;
17462 virtualBlock->GetAllocationInfo(allocation, outInfo&: *pVirtualAllocInfo);
17463}
17464
17465VMA_CALL_PRE VkResult VMA_CALL_POST vmaVirtualAllocate(VmaVirtualBlock VMA_NOT_NULL virtualBlock,
17466 const VmaVirtualAllocationCreateInfo* VMA_NOT_NULL pCreateInfo, VmaVirtualAllocation VMA_NULLABLE_NON_DISPATCHABLE* VMA_NOT_NULL pAllocation,
17467 VkDeviceSize* VMA_NULLABLE pOffset)
17468{
17469 VMA_ASSERT(virtualBlock != VK_NULL_HANDLE && pCreateInfo != VMA_NULL && pAllocation != VMA_NULL);
17470 VMA_DEBUG_LOG("vmaVirtualAllocate");
17471 VMA_DEBUG_GLOBAL_MUTEX_LOCK;
17472 return virtualBlock->Allocate(createInfo: *pCreateInfo, outAllocation&: *pAllocation, outOffset: pOffset);
17473}
17474
17475VMA_CALL_PRE void VMA_CALL_POST vmaVirtualFree(VmaVirtualBlock VMA_NOT_NULL virtualBlock, VmaVirtualAllocation VMA_NULLABLE_NON_DISPATCHABLE allocation)
17476{
17477 if(allocation != VK_NULL_HANDLE)
17478 {
17479 VMA_ASSERT(virtualBlock != VK_NULL_HANDLE);
17480 VMA_DEBUG_LOG("vmaVirtualFree");
17481 VMA_DEBUG_GLOBAL_MUTEX_LOCK;
17482 virtualBlock->Free(allocation);
17483 }
17484}
17485
17486VMA_CALL_PRE void VMA_CALL_POST vmaClearVirtualBlock(VmaVirtualBlock VMA_NOT_NULL virtualBlock)
17487{
17488 VMA_ASSERT(virtualBlock != VK_NULL_HANDLE);
17489 VMA_DEBUG_LOG("vmaClearVirtualBlock");
17490 VMA_DEBUG_GLOBAL_MUTEX_LOCK;
17491 virtualBlock->Clear();
17492}
17493
17494VMA_CALL_PRE void VMA_CALL_POST vmaSetVirtualAllocationUserData(VmaVirtualBlock VMA_NOT_NULL virtualBlock,
17495 VmaVirtualAllocation VMA_NOT_NULL_NON_DISPATCHABLE allocation, void* VMA_NULLABLE pUserData)
17496{
17497 VMA_ASSERT(virtualBlock != VK_NULL_HANDLE);
17498 VMA_DEBUG_LOG("vmaSetVirtualAllocationUserData");
17499 VMA_DEBUG_GLOBAL_MUTEX_LOCK;
17500 virtualBlock->SetAllocationUserData(allocation, userData: pUserData);
17501}
17502
17503VMA_CALL_PRE void VMA_CALL_POST vmaGetVirtualBlockStatistics(VmaVirtualBlock VMA_NOT_NULL virtualBlock,
17504 VmaStatistics* VMA_NOT_NULL pStats)
17505{
17506 VMA_ASSERT(virtualBlock != VK_NULL_HANDLE && pStats != VMA_NULL);
17507 VMA_DEBUG_LOG("vmaGetVirtualBlockStatistics");
17508 VMA_DEBUG_GLOBAL_MUTEX_LOCK;
17509 virtualBlock->GetStatistics(outStats&: *pStats);
17510}
17511
17512VMA_CALL_PRE void VMA_CALL_POST vmaCalculateVirtualBlockStatistics(VmaVirtualBlock VMA_NOT_NULL virtualBlock,
17513 VmaDetailedStatistics* VMA_NOT_NULL pStats)
17514{
17515 VMA_ASSERT(virtualBlock != VK_NULL_HANDLE && pStats != VMA_NULL);
17516 VMA_DEBUG_LOG("vmaCalculateVirtualBlockStatistics");
17517 VMA_DEBUG_GLOBAL_MUTEX_LOCK;
17518 virtualBlock->CalculateDetailedStatistics(outStats&: *pStats);
17519}
17520
17521#if VMA_STATS_STRING_ENABLED
17522
17523VMA_CALL_PRE void VMA_CALL_POST vmaBuildVirtualBlockStatsString(VmaVirtualBlock VMA_NOT_NULL virtualBlock,
17524 char* VMA_NULLABLE * VMA_NOT_NULL ppStatsString, VkBool32 detailedMap)
17525{
17526 VMA_ASSERT(virtualBlock != VK_NULL_HANDLE && ppStatsString != VMA_NULL);
17527 VMA_DEBUG_GLOBAL_MUTEX_LOCK;
17528 const VkAllocationCallbacks* allocationCallbacks = virtualBlock->GetAllocationCallbacks();
17529 VmaStringBuilder sb(allocationCallbacks);
17530 virtualBlock->BuildStatsString(detailedMap: detailedMap != VK_FALSE, sb);
17531 *ppStatsString = VmaCreateStringCopy(allocs: allocationCallbacks, srcStr: sb.GetData(), strLen: sb.GetLength());
17532}
17533
17534VMA_CALL_PRE void VMA_CALL_POST vmaFreeVirtualBlockStatsString(VmaVirtualBlock VMA_NOT_NULL virtualBlock,
17535 char* VMA_NULLABLE pStatsString)
17536{
17537 if(pStatsString != VMA_NULL)
17538 {
17539 VMA_ASSERT(virtualBlock != VK_NULL_HANDLE);
17540 VMA_DEBUG_GLOBAL_MUTEX_LOCK;
17541 VmaFreeString(allocs: virtualBlock->GetAllocationCallbacks(), str: pStatsString);
17542 }
17543}
17544#endif // VMA_STATS_STRING_ENABLED
17545#endif // _VMA_PUBLIC_INTERFACE
17546
17547#if defined(__GNUC__) && !defined(__clang__)
17548#pragma GCC diagnostic pop
17549#elif defined(__clang__)
17550#pragma clang diagnostic pop
17551#endif
17552
17553#endif // VMA_IMPLEMENTATION
17554
17555/**
17556\page quick_start Quick start
17557
17558\section quick_start_project_setup Project setup
17559
17560Vulkan Memory Allocator comes in form of a "stb-style" single header file.
17561You don't need to build it as a separate library project.
17562You can add this file directly to your project and submit it to code repository next to your other source files.
17563
17564"Single header" doesn't mean that everything is contained in C/C++ declarations,
17565like it tends to be in case of inline functions or C++ templates.
17566It means that implementation is bundled with interface in a single file and needs to be extracted using preprocessor macro.
17567If you don't do it properly, you will get linker errors.
17568
17569To do it properly:
17570
17571-# Include "vk_mem_alloc.h" file in each CPP file where you want to use the library.
17572 This includes declarations of all members of the library.
17573-# In exactly one CPP file define following macro before this include.
17574 It enables also internal definitions.
17575
17576\code
17577#define VMA_IMPLEMENTATION
17578#include "vk_mem_alloc.h"
17579\endcode
17580
17581It may be a good idea to create dedicated CPP file just for this purpose.
17582
17583This library includes header `<vulkan/vulkan.h>`, which in turn
17584includes `<windows.h>` on Windows. If you need some specific macros defined
17585before including these headers (like `WIN32_LEAN_AND_MEAN` or
17586`WINVER` for Windows, `VK_USE_PLATFORM_WIN32_KHR` for Vulkan), you must define
17587them before every `#include` of this library.
17588
17589This library is written in C++, but has C-compatible interface.
17590Thus you can include and use vk_mem_alloc.h in C or C++ code, but full
17591implementation with `VMA_IMPLEMENTATION` macro must be compiled as C++, NOT as C.
17592Some features of C++14 used. STL containers, RTTI, or C++ exceptions are not used.
17593
17594
17595\section quick_start_initialization Initialization
17596
17597At program startup:
17598
17599-# Initialize Vulkan to have `VkPhysicalDevice`, `VkDevice` and `VkInstance` object.
17600-# Fill VmaAllocatorCreateInfo structure and create #VmaAllocator object by
17601 calling vmaCreateAllocator().
17602
17603Only members `physicalDevice`, `device`, `instance` are required.
17604However, you should inform the library which Vulkan version do you use by setting
17605VmaAllocatorCreateInfo::vulkanApiVersion and which extensions did you enable
17606by setting VmaAllocatorCreateInfo::flags (like #VMA_ALLOCATOR_CREATE_BUFFER_DEVICE_ADDRESS_BIT for VK_KHR_buffer_device_address).
17607Otherwise, VMA would use only features of Vulkan 1.0 core with no extensions.
17608
17609You may need to configure importing Vulkan functions. There are 3 ways to do this:
17610
17611-# **If you link with Vulkan static library** (e.g. "vulkan-1.lib" on Windows):
17612 - You don't need to do anything.
17613 - VMA will use these, as macro `VMA_STATIC_VULKAN_FUNCTIONS` is defined to 1 by default.
17614-# **If you want VMA to fetch pointers to Vulkan functions dynamically** using `vkGetInstanceProcAddr`,
17615 `vkGetDeviceProcAddr` (this is the option presented in the example below):
17616 - Define `VMA_STATIC_VULKAN_FUNCTIONS` to 0, `VMA_DYNAMIC_VULKAN_FUNCTIONS` to 1.
17617 - Provide pointers to these two functions via VmaVulkanFunctions::vkGetInstanceProcAddr,
17618 VmaVulkanFunctions::vkGetDeviceProcAddr.
17619 - The library will fetch pointers to all other functions it needs internally.
17620-# **If you fetch pointers to all Vulkan functions in a custom way**, e.g. using some loader like
17621 [Volk](https://github.com/zeux/volk):
17622 - Define `VMA_STATIC_VULKAN_FUNCTIONS` and `VMA_DYNAMIC_VULKAN_FUNCTIONS` to 0.
17623 - Pass these pointers via structure #VmaVulkanFunctions.
17624
17625\code
17626VmaVulkanFunctions vulkanFunctions = {};
17627vulkanFunctions.vkGetInstanceProcAddr = &vkGetInstanceProcAddr;
17628vulkanFunctions.vkGetDeviceProcAddr = &vkGetDeviceProcAddr;
17629
17630VmaAllocatorCreateInfo allocatorCreateInfo = {};
17631allocatorCreateInfo.vulkanApiVersion = VK_API_VERSION_1_2;
17632allocatorCreateInfo.physicalDevice = physicalDevice;
17633allocatorCreateInfo.device = device;
17634allocatorCreateInfo.instance = instance;
17635allocatorCreateInfo.pVulkanFunctions = &vulkanFunctions;
17636
17637VmaAllocator allocator;
17638vmaCreateAllocator(&allocatorCreateInfo, &allocator);
17639\endcode
17640
17641
17642\section quick_start_resource_allocation Resource allocation
17643
17644When you want to create a buffer or image:
17645
17646-# Fill `VkBufferCreateInfo` / `VkImageCreateInfo` structure.
17647-# Fill VmaAllocationCreateInfo structure.
17648-# Call vmaCreateBuffer() / vmaCreateImage() to get `VkBuffer`/`VkImage` with memory
17649 already allocated and bound to it, plus #VmaAllocation objects that represents its underlying memory.
17650
17651\code
17652VkBufferCreateInfo bufferInfo = { VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO };
17653bufferInfo.size = 65536;
17654bufferInfo.usage = VK_BUFFER_USAGE_VERTEX_BUFFER_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT;
17655
17656VmaAllocationCreateInfo allocInfo = {};
17657allocInfo.usage = VMA_MEMORY_USAGE_AUTO;
17658
17659VkBuffer buffer;
17660VmaAllocation allocation;
17661vmaCreateBuffer(allocator, &bufferInfo, &allocInfo, &buffer, &allocation, nullptr);
17662\endcode
17663
17664Don't forget to destroy your objects when no longer needed:
17665
17666\code
17667vmaDestroyBuffer(allocator, buffer, allocation);
17668vmaDestroyAllocator(allocator);
17669\endcode
17670
17671
17672\page choosing_memory_type Choosing memory type
17673
17674Physical devices in Vulkan support various combinations of memory heaps and
17675types. Help with choosing correct and optimal memory type for your specific
17676resource is one of the key features of this library. You can use it by filling
17677appropriate members of VmaAllocationCreateInfo structure, as described below.
17678You can also combine multiple methods.
17679
17680-# If you just want to find memory type index that meets your requirements, you
17681 can use function: vmaFindMemoryTypeIndexForBufferInfo(),
17682 vmaFindMemoryTypeIndexForImageInfo(), vmaFindMemoryTypeIndex().
17683-# If you want to allocate a region of device memory without association with any
17684 specific image or buffer, you can use function vmaAllocateMemory(). Usage of
17685 this function is not recommended and usually not needed.
17686 vmaAllocateMemoryPages() function is also provided for creating multiple allocations at once,
17687 which may be useful for sparse binding.
17688-# If you already have a buffer or an image created, you want to allocate memory
17689 for it and then you will bind it yourself, you can use function
17690 vmaAllocateMemoryForBuffer(), vmaAllocateMemoryForImage().
17691 For binding you should use functions: vmaBindBufferMemory(), vmaBindImageMemory()
17692 or their extended versions: vmaBindBufferMemory2(), vmaBindImageMemory2().
17693-# **This is the easiest and recommended way to use this library:**
17694 If you want to create a buffer or an image, allocate memory for it and bind
17695 them together, all in one call, you can use function vmaCreateBuffer(),
17696 vmaCreateImage().
17697
17698When using 3. or 4., the library internally queries Vulkan for memory types
17699supported for that buffer or image (function `vkGetBufferMemoryRequirements()`)
17700and uses only one of these types.
17701
17702If no memory type can be found that meets all the requirements, these functions
17703return `VK_ERROR_FEATURE_NOT_PRESENT`.
17704
17705You can leave VmaAllocationCreateInfo structure completely filled with zeros.
17706It means no requirements are specified for memory type.
17707It is valid, although not very useful.
17708
17709\section choosing_memory_type_usage Usage
17710
17711The easiest way to specify memory requirements is to fill member
17712VmaAllocationCreateInfo::usage using one of the values of enum #VmaMemoryUsage.
17713It defines high level, common usage types.
17714Since version 3 of the library, it is recommended to use #VMA_MEMORY_USAGE_AUTO to let it select best memory type for your resource automatically.
17715
17716For example, if you want to create a uniform buffer that will be filled using
17717transfer only once or infrequently and then used for rendering every frame as a uniform buffer, you can
17718do it using following code. The buffer will most likely end up in a memory type with
17719`VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT` to be fast to access by the GPU device.
17720
17721\code
17722VkBufferCreateInfo bufferInfo = { VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO };
17723bufferInfo.size = 65536;
17724bufferInfo.usage = VK_BUFFER_USAGE_UNIFORM_BUFFER_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT;
17725
17726VmaAllocationCreateInfo allocInfo = {};
17727allocInfo.usage = VMA_MEMORY_USAGE_AUTO;
17728
17729VkBuffer buffer;
17730VmaAllocation allocation;
17731vmaCreateBuffer(allocator, &bufferInfo, &allocInfo, &buffer, &allocation, nullptr);
17732\endcode
17733
17734If you have a preference for putting the resource in GPU (device) memory or CPU (host) memory
17735on systems with discrete graphics card that have the memories separate, you can use
17736#VMA_MEMORY_USAGE_AUTO_PREFER_DEVICE or #VMA_MEMORY_USAGE_AUTO_PREFER_HOST.
17737
17738When using `VMA_MEMORY_USAGE_AUTO*` while you want to map the allocated memory,
17739you also need to specify one of the host access flags:
17740#VMA_ALLOCATION_CREATE_HOST_ACCESS_SEQUENTIAL_WRITE_BIT or #VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT.
17741This will help the library decide about preferred memory type to ensure it has `VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT`
17742so you can map it.
17743
17744For example, a staging buffer that will be filled via mapped pointer and then
17745used as a source of transfer to the buffer decribed previously can be created like this.
17746It will likely and up in a memory type that is `HOST_VISIBLE` and `HOST_COHERENT`
17747but not `HOST_CACHED` (meaning uncached, write-combined) and not `DEVICE_LOCAL` (meaning system RAM).
17748
17749\code
17750VkBufferCreateInfo stagingBufferInfo = { VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO };
17751stagingBufferInfo.size = 65536;
17752stagingBufferInfo.usage = VK_BUFFER_USAGE_TRANSFER_SRC_BIT;
17753
17754VmaAllocationCreateInfo stagingAllocInfo = {};
17755stagingAllocInfo.usage = VMA_MEMORY_USAGE_AUTO;
17756stagingAllocInfo.flags = VMA_ALLOCATION_CREATE_HOST_ACCESS_SEQUENTIAL_WRITE_BIT;
17757
17758VkBuffer stagingBuffer;
17759VmaAllocation stagingAllocation;
17760vmaCreateBuffer(allocator, &stagingBufferInfo, &stagingAllocInfo, &stagingBuffer, &stagingAllocation, nullptr);
17761\endcode
17762
17763For more examples of creating different kinds of resources, see chapter \ref usage_patterns.
17764
17765Usage values `VMA_MEMORY_USAGE_AUTO*` are legal to use only when the library knows
17766about the resource being created by having `VkBufferCreateInfo` / `VkImageCreateInfo` passed,
17767so they work with functions like: vmaCreateBuffer(), vmaCreateImage(), vmaFindMemoryTypeIndexForBufferInfo() etc.
17768If you allocate raw memory using function vmaAllocateMemory(), you have to use other means of selecting
17769memory type, as decribed below.
17770
17771\note
17772Old usage values (`VMA_MEMORY_USAGE_GPU_ONLY`, `VMA_MEMORY_USAGE_CPU_ONLY`,
17773`VMA_MEMORY_USAGE_CPU_TO_GPU`, `VMA_MEMORY_USAGE_GPU_TO_CPU`, `VMA_MEMORY_USAGE_CPU_COPY`)
17774are still available and work same way as in previous versions of the library
17775for backward compatibility, but they are not recommended.
17776
17777\section choosing_memory_type_required_preferred_flags Required and preferred flags
17778
17779You can specify more detailed requirements by filling members
17780VmaAllocationCreateInfo::requiredFlags and VmaAllocationCreateInfo::preferredFlags
17781with a combination of bits from enum `VkMemoryPropertyFlags`. For example,
17782if you want to create a buffer that will be persistently mapped on host (so it
17783must be `HOST_VISIBLE`) and preferably will also be `HOST_COHERENT` and `HOST_CACHED`,
17784use following code:
17785
17786\code
17787VmaAllocationCreateInfo allocInfo = {};
17788allocInfo.requiredFlags = VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT;
17789allocInfo.preferredFlags = VK_MEMORY_PROPERTY_HOST_COHERENT_BIT | VK_MEMORY_PROPERTY_HOST_CACHED_BIT;
17790allocInfo.flags = VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT | VMA_ALLOCATION_CREATE_MAPPED_BIT;
17791
17792VkBuffer buffer;
17793VmaAllocation allocation;
17794vmaCreateBuffer(allocator, &bufferInfo, &allocInfo, &buffer, &allocation, nullptr);
17795\endcode
17796
17797A memory type is chosen that has all the required flags and as many preferred
17798flags set as possible.
17799
17800Value passed in VmaAllocationCreateInfo::usage is internally converted to a set of required and preferred flags,
17801plus some extra "magic" (heuristics).
17802
17803\section choosing_memory_type_explicit_memory_types Explicit memory types
17804
17805If you inspected memory types available on the physical device and you have
17806a preference for memory types that you want to use, you can fill member
17807VmaAllocationCreateInfo::memoryTypeBits. It is a bit mask, where each bit set
17808means that a memory type with that index is allowed to be used for the
17809allocation. Special value 0, just like `UINT32_MAX`, means there are no
17810restrictions to memory type index.
17811
17812Please note that this member is NOT just a memory type index.
17813Still you can use it to choose just one, specific memory type.
17814For example, if you already determined that your buffer should be created in
17815memory type 2, use following code:
17816
17817\code
17818uint32_t memoryTypeIndex = 2;
17819
17820VmaAllocationCreateInfo allocInfo = {};
17821allocInfo.memoryTypeBits = 1u << memoryTypeIndex;
17822
17823VkBuffer buffer;
17824VmaAllocation allocation;
17825vmaCreateBuffer(allocator, &bufferInfo, &allocInfo, &buffer, &allocation, nullptr);
17826\endcode
17827
17828
17829\section choosing_memory_type_custom_memory_pools Custom memory pools
17830
17831If you allocate from custom memory pool, all the ways of specifying memory
17832requirements described above are not applicable and the aforementioned members
17833of VmaAllocationCreateInfo structure are ignored. Memory type is selected
17834explicitly when creating the pool and then used to make all the allocations from
17835that pool. For further details, see \ref custom_memory_pools.
17836
17837\section choosing_memory_type_dedicated_allocations Dedicated allocations
17838
17839Memory for allocations is reserved out of larger block of `VkDeviceMemory`
17840allocated from Vulkan internally. That is the main feature of this whole library.
17841You can still request a separate memory block to be created for an allocation,
17842just like you would do in a trivial solution without using any allocator.
17843In that case, a buffer or image is always bound to that memory at offset 0.
17844This is called a "dedicated allocation".
17845You can explicitly request it by using flag #VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT.
17846The library can also internally decide to use dedicated allocation in some cases, e.g.:
17847
17848- When the size of the allocation is large.
17849- When [VK_KHR_dedicated_allocation](@ref vk_khr_dedicated_allocation) extension is enabled
17850 and it reports that dedicated allocation is required or recommended for the resource.
17851- When allocation of next big memory block fails due to not enough device memory,
17852 but allocation with the exact requested size succeeds.
17853
17854
17855\page memory_mapping Memory mapping
17856
17857To "map memory" in Vulkan means to obtain a CPU pointer to `VkDeviceMemory`,
17858to be able to read from it or write to it in CPU code.
17859Mapping is possible only of memory allocated from a memory type that has
17860`VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT` flag.
17861Functions `vkMapMemory()`, `vkUnmapMemory()` are designed for this purpose.
17862You can use them directly with memory allocated by this library,
17863but it is not recommended because of following issue:
17864Mapping the same `VkDeviceMemory` block multiple times is illegal - only one mapping at a time is allowed.
17865This includes mapping disjoint regions. Mapping is not reference-counted internally by Vulkan.
17866Because of this, Vulkan Memory Allocator provides following facilities:
17867
17868\note If you want to be able to map an allocation, you need to specify one of the flags
17869#VMA_ALLOCATION_CREATE_HOST_ACCESS_SEQUENTIAL_WRITE_BIT or #VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT
17870in VmaAllocationCreateInfo::flags. These flags are required for an allocation to be mappable
17871when using #VMA_MEMORY_USAGE_AUTO or other `VMA_MEMORY_USAGE_AUTO*` enum values.
17872For other usage values they are ignored and every such allocation made in `HOST_VISIBLE` memory type is mappable,
17873but they can still be used for consistency.
17874
17875\section memory_mapping_mapping_functions Mapping functions
17876
17877The library provides following functions for mapping of a specific #VmaAllocation: vmaMapMemory(), vmaUnmapMemory().
17878They are safer and more convenient to use than standard Vulkan functions.
17879You can map an allocation multiple times simultaneously - mapping is reference-counted internally.
17880You can also map different allocations simultaneously regardless of whether they use the same `VkDeviceMemory` block.
17881The way it is implemented is that the library always maps entire memory block, not just region of the allocation.
17882For further details, see description of vmaMapMemory() function.
17883Example:
17884
17885\code
17886// Having these objects initialized:
17887struct ConstantBuffer
17888{
17889 ...
17890};
17891ConstantBuffer constantBufferData = ...
17892
17893VmaAllocator allocator = ...
17894VkBuffer constantBuffer = ...
17895VmaAllocation constantBufferAllocation = ...
17896
17897// You can map and fill your buffer using following code:
17898
17899void* mappedData;
17900vmaMapMemory(allocator, constantBufferAllocation, &mappedData);
17901memcpy(mappedData, &constantBufferData, sizeof(constantBufferData));
17902vmaUnmapMemory(allocator, constantBufferAllocation);
17903\endcode
17904
17905When mapping, you may see a warning from Vulkan validation layer similar to this one:
17906
17907<i>Mapping an image with layout VK_IMAGE_LAYOUT_DEPTH_STENCIL_ATTACHMENT_OPTIMAL can result in undefined behavior if this memory is used by the device. Only GENERAL or PREINITIALIZED should be used.</i>
17908
17909It happens because the library maps entire `VkDeviceMemory` block, where different
17910types of images and buffers may end up together, especially on GPUs with unified memory like Intel.
17911You can safely ignore it if you are sure you access only memory of the intended
17912object that you wanted to map.
17913
17914
17915\section memory_mapping_persistently_mapped_memory Persistently mapped memory
17916
17917Kepping your memory persistently mapped is generally OK in Vulkan.
17918You don't need to unmap it before using its data on the GPU.
17919The library provides a special feature designed for that:
17920Allocations made with #VMA_ALLOCATION_CREATE_MAPPED_BIT flag set in
17921VmaAllocationCreateInfo::flags stay mapped all the time,
17922so you can just access CPU pointer to it any time
17923without a need to call any "map" or "unmap" function.
17924Example:
17925
17926\code
17927VkBufferCreateInfo bufCreateInfo = { VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO };
17928bufCreateInfo.size = sizeof(ConstantBuffer);
17929bufCreateInfo.usage = VK_BUFFER_USAGE_TRANSFER_SRC_BIT;
17930
17931VmaAllocationCreateInfo allocCreateInfo = {};
17932allocCreateInfo.usage = VMA_MEMORY_USAGE_AUTO;
17933allocCreateInfo.flags = VMA_ALLOCATION_CREATE_HOST_ACCESS_SEQUENTIAL_WRITE_BIT |
17934 VMA_ALLOCATION_CREATE_MAPPED_BIT;
17935
17936VkBuffer buf;
17937VmaAllocation alloc;
17938VmaAllocationInfo allocInfo;
17939vmaCreateBuffer(allocator, &bufCreateInfo, &allocCreateInfo, &buf, &alloc, &allocInfo);
17940
17941// Buffer is already mapped. You can access its memory.
17942memcpy(allocInfo.pMappedData, &constantBufferData, sizeof(constantBufferData));
17943\endcode
17944
17945\note #VMA_ALLOCATION_CREATE_MAPPED_BIT by itself doesn't guarantee that the allocation will end up
17946in a mappable memory type.
17947For this, you need to also specify #VMA_ALLOCATION_CREATE_HOST_ACCESS_SEQUENTIAL_WRITE_BIT or
17948#VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT.
17949#VMA_ALLOCATION_CREATE_MAPPED_BIT only guarantees that if the memory is `HOST_VISIBLE`, the allocation will be mapped on creation.
17950For an example of how to make use of this fact, see section \ref usage_patterns_advanced_data_uploading.
17951
17952\section memory_mapping_cache_control Cache flush and invalidate
17953
17954Memory in Vulkan doesn't need to be unmapped before using it on GPU,
17955but unless a memory types has `VK_MEMORY_PROPERTY_HOST_COHERENT_BIT` flag set,
17956you need to manually **invalidate** cache before reading of mapped pointer
17957and **flush** cache after writing to mapped pointer.
17958Map/unmap operations don't do that automatically.
17959Vulkan provides following functions for this purpose `vkFlushMappedMemoryRanges()`,
17960`vkInvalidateMappedMemoryRanges()`, but this library provides more convenient
17961functions that refer to given allocation object: vmaFlushAllocation(),
17962vmaInvalidateAllocation(),
17963or multiple objects at once: vmaFlushAllocations(), vmaInvalidateAllocations().
17964
17965Regions of memory specified for flush/invalidate must be aligned to
17966`VkPhysicalDeviceLimits::nonCoherentAtomSize`. This is automatically ensured by the library.
17967In any memory type that is `HOST_VISIBLE` but not `HOST_COHERENT`, all allocations
17968within blocks are aligned to this value, so their offsets are always multiply of
17969`nonCoherentAtomSize` and two different allocations never share same "line" of this size.
17970
17971Also, Windows drivers from all 3 PC GPU vendors (AMD, Intel, NVIDIA)
17972currently provide `HOST_COHERENT` flag on all memory types that are
17973`HOST_VISIBLE`, so on PC you may not need to bother.
17974
17975
17976\page staying_within_budget Staying within budget
17977
17978When developing a graphics-intensive game or program, it is important to avoid allocating
17979more GPU memory than it is physically available. When the memory is over-committed,
17980various bad things can happen, depending on the specific GPU, graphics driver, and
17981operating system:
17982
17983- It may just work without any problems.
17984- The application may slow down because some memory blocks are moved to system RAM
17985 and the GPU has to access them through PCI Express bus.
17986- A new allocation may take very long time to complete, even few seconds, and possibly
17987 freeze entire system.
17988- The new allocation may fail with `VK_ERROR_OUT_OF_DEVICE_MEMORY`.
17989- It may even result in GPU crash (TDR), observed as `VK_ERROR_DEVICE_LOST`
17990 returned somewhere later.
17991
17992\section staying_within_budget_querying_for_budget Querying for budget
17993
17994To query for current memory usage and available budget, use function vmaGetHeapBudgets().
17995Returned structure #VmaBudget contains quantities expressed in bytes, per Vulkan memory heap.
17996
17997Please note that this function returns different information and works faster than
17998vmaCalculateStatistics(). vmaGetHeapBudgets() can be called every frame or even before every
17999allocation, while vmaCalculateStatistics() is intended to be used rarely,
18000only to obtain statistical information, e.g. for debugging purposes.
18001
18002It is recommended to use <b>VK_EXT_memory_budget</b> device extension to obtain information
18003about the budget from Vulkan device. VMA is able to use this extension automatically.
18004When not enabled, the allocator behaves same way, but then it estimates current usage
18005and available budget based on its internal information and Vulkan memory heap sizes,
18006which may be less precise. In order to use this extension:
18007
180081. Make sure extensions VK_EXT_memory_budget and VK_KHR_get_physical_device_properties2
18009 required by it are available and enable them. Please note that the first is a device
18010 extension and the second is instance extension!
180112. Use flag #VMA_ALLOCATOR_CREATE_EXT_MEMORY_BUDGET_BIT when creating #VmaAllocator object.
180123. Make sure to call vmaSetCurrentFrameIndex() every frame. Budget is queried from
18013 Vulkan inside of it to avoid overhead of querying it with every allocation.
18014
18015\section staying_within_budget_controlling_memory_usage Controlling memory usage
18016
18017There are many ways in which you can try to stay within the budget.
18018
18019First, when making new allocation requires allocating a new memory block, the library
18020tries not to exceed the budget automatically. If a block with default recommended size
18021(e.g. 256 MB) would go over budget, a smaller block is allocated, possibly even
18022dedicated memory for just this resource.
18023
18024If the size of the requested resource plus current memory usage is more than the
18025budget, by default the library still tries to create it, leaving it to the Vulkan
18026implementation whether the allocation succeeds or fails. You can change this behavior
18027by using #VMA_ALLOCATION_CREATE_WITHIN_BUDGET_BIT flag. With it, the allocation is
18028not made if it would exceed the budget or if the budget is already exceeded.
18029VMA then tries to make the allocation from the next eligible Vulkan memory type.
18030The all of them fail, the call then fails with `VK_ERROR_OUT_OF_DEVICE_MEMORY`.
18031Example usage pattern may be to pass the #VMA_ALLOCATION_CREATE_WITHIN_BUDGET_BIT flag
18032when creating resources that are not essential for the application (e.g. the texture
18033of a specific object) and not to pass it when creating critically important resources
18034(e.g. render targets).
18035
18036On AMD graphics cards there is a custom vendor extension available: <b>VK_AMD_memory_overallocation_behavior</b>
18037that allows to control the behavior of the Vulkan implementation in out-of-memory cases -
18038whether it should fail with an error code or still allow the allocation.
18039Usage of this extension involves only passing extra structure on Vulkan device creation,
18040so it is out of scope of this library.
18041
18042Finally, you can also use #VMA_ALLOCATION_CREATE_NEVER_ALLOCATE_BIT flag to make sure
18043a new allocation is created only when it fits inside one of the existing memory blocks.
18044If it would require to allocate a new block, if fails instead with `VK_ERROR_OUT_OF_DEVICE_MEMORY`.
18045This also ensures that the function call is very fast because it never goes to Vulkan
18046to obtain a new block.
18047
18048\note Creating \ref custom_memory_pools with VmaPoolCreateInfo::minBlockCount
18049set to more than 0 will currently try to allocate memory blocks without checking whether they
18050fit within budget.
18051
18052
18053\page resource_aliasing Resource aliasing (overlap)
18054
18055New explicit graphics APIs (Vulkan and Direct3D 12), thanks to manual memory
18056management, give an opportunity to alias (overlap) multiple resources in the
18057same region of memory - a feature not available in the old APIs (Direct3D 11, OpenGL).
18058It can be useful to save video memory, but it must be used with caution.
18059
18060For example, if you know the flow of your whole render frame in advance, you
18061are going to use some intermediate textures or buffers only during a small range of render passes,
18062and you know these ranges don't overlap in time, you can bind these resources to
18063the same place in memory, even if they have completely different parameters (width, height, format etc.).
18064
18065![Resource aliasing (overlap)](../gfx/Aliasing.png)
18066
18067Such scenario is possible using VMA, but you need to create your images manually.
18068Then you need to calculate parameters of an allocation to be made using formula:
18069
18070- allocation size = max(size of each image)
18071- allocation alignment = max(alignment of each image)
18072- allocation memoryTypeBits = bitwise AND(memoryTypeBits of each image)
18073
18074Following example shows two different images bound to the same place in memory,
18075allocated to fit largest of them.
18076
18077\code
18078// A 512x512 texture to be sampled.
18079VkImageCreateInfo img1CreateInfo = { VK_STRUCTURE_TYPE_IMAGE_CREATE_INFO };
18080img1CreateInfo.imageType = VK_IMAGE_TYPE_2D;
18081img1CreateInfo.extent.width = 512;
18082img1CreateInfo.extent.height = 512;
18083img1CreateInfo.extent.depth = 1;
18084img1CreateInfo.mipLevels = 10;
18085img1CreateInfo.arrayLayers = 1;
18086img1CreateInfo.format = VK_FORMAT_R8G8B8A8_SRGB;
18087img1CreateInfo.tiling = VK_IMAGE_TILING_OPTIMAL;
18088img1CreateInfo.initialLayout = VK_IMAGE_LAYOUT_UNDEFINED;
18089img1CreateInfo.usage = VK_IMAGE_USAGE_TRANSFER_DST_BIT | VK_IMAGE_USAGE_SAMPLED_BIT;
18090img1CreateInfo.samples = VK_SAMPLE_COUNT_1_BIT;
18091
18092// A full screen texture to be used as color attachment.
18093VkImageCreateInfo img2CreateInfo = { VK_STRUCTURE_TYPE_IMAGE_CREATE_INFO };
18094img2CreateInfo.imageType = VK_IMAGE_TYPE_2D;
18095img2CreateInfo.extent.width = 1920;
18096img2CreateInfo.extent.height = 1080;
18097img2CreateInfo.extent.depth = 1;
18098img2CreateInfo.mipLevels = 1;
18099img2CreateInfo.arrayLayers = 1;
18100img2CreateInfo.format = VK_FORMAT_R8G8B8A8_UNORM;
18101img2CreateInfo.tiling = VK_IMAGE_TILING_OPTIMAL;
18102img2CreateInfo.initialLayout = VK_IMAGE_LAYOUT_UNDEFINED;
18103img2CreateInfo.usage = VK_IMAGE_USAGE_SAMPLED_BIT | VK_IMAGE_USAGE_COLOR_ATTACHMENT_BIT;
18104img2CreateInfo.samples = VK_SAMPLE_COUNT_1_BIT;
18105
18106VkImage img1;
18107res = vkCreateImage(device, &img1CreateInfo, nullptr, &img1);
18108VkImage img2;
18109res = vkCreateImage(device, &img2CreateInfo, nullptr, &img2);
18110
18111VkMemoryRequirements img1MemReq;
18112vkGetImageMemoryRequirements(device, img1, &img1MemReq);
18113VkMemoryRequirements img2MemReq;
18114vkGetImageMemoryRequirements(device, img2, &img2MemReq);
18115
18116VkMemoryRequirements finalMemReq = {};
18117finalMemReq.size = std::max(img1MemReq.size, img2MemReq.size);
18118finalMemReq.alignment = std::max(img1MemReq.alignment, img2MemReq.alignment);
18119finalMemReq.memoryTypeBits = img1MemReq.memoryTypeBits & img2MemReq.memoryTypeBits;
18120// Validate if(finalMemReq.memoryTypeBits != 0)
18121
18122VmaAllocationCreateInfo allocCreateInfo = {};
18123allocCreateInfo.preferredFlags = VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT;
18124
18125VmaAllocation alloc;
18126res = vmaAllocateMemory(allocator, &finalMemReq, &allocCreateInfo, &alloc, nullptr);
18127
18128res = vmaBindImageMemory(allocator, alloc, img1);
18129res = vmaBindImageMemory(allocator, alloc, img2);
18130
18131// You can use img1, img2 here, but not at the same time!
18132
18133vmaFreeMemory(allocator, alloc);
18134vkDestroyImage(allocator, img2, nullptr);
18135vkDestroyImage(allocator, img1, nullptr);
18136\endcode
18137
18138Remember that using resources that alias in memory requires proper synchronization.
18139You need to issue a memory barrier to make sure commands that use `img1` and `img2`
18140don't overlap on GPU timeline.
18141You also need to treat a resource after aliasing as uninitialized - containing garbage data.
18142For example, if you use `img1` and then want to use `img2`, you need to issue
18143an image memory barrier for `img2` with `oldLayout` = `VK_IMAGE_LAYOUT_UNDEFINED`.
18144
18145Additional considerations:
18146
18147- Vulkan also allows to interpret contents of memory between aliasing resources consistently in some cases.
18148See chapter 11.8. "Memory Aliasing" of Vulkan specification or `VK_IMAGE_CREATE_ALIAS_BIT` flag.
18149- You can create more complex layout where different images and buffers are bound
18150at different offsets inside one large allocation. For example, one can imagine
18151a big texture used in some render passes, aliasing with a set of many small buffers
18152used between in some further passes. To bind a resource at non-zero offset in an allocation,
18153use vmaBindBufferMemory2() / vmaBindImageMemory2().
18154- Before allocating memory for the resources you want to alias, check `memoryTypeBits`
18155returned in memory requirements of each resource to make sure the bits overlap.
18156Some GPUs may expose multiple memory types suitable e.g. only for buffers or
18157images with `COLOR_ATTACHMENT` usage, so the sets of memory types supported by your
18158resources may be disjoint. Aliasing them is not possible in that case.
18159
18160
18161\page custom_memory_pools Custom memory pools
18162
18163A memory pool contains a number of `VkDeviceMemory` blocks.
18164The library automatically creates and manages default pool for each memory type available on the device.
18165Default memory pool automatically grows in size.
18166Size of allocated blocks is also variable and managed automatically.
18167
18168You can create custom pool and allocate memory out of it.
18169It can be useful if you want to:
18170
18171- Keep certain kind of allocations separate from others.
18172- Enforce particular, fixed size of Vulkan memory blocks.
18173- Limit maximum amount of Vulkan memory allocated for that pool.
18174- Reserve minimum or fixed amount of Vulkan memory always preallocated for that pool.
18175- Use extra parameters for a set of your allocations that are available in #VmaPoolCreateInfo but not in
18176 #VmaAllocationCreateInfo - e.g., custom minimum alignment, custom `pNext` chain.
18177- Perform defragmentation on a specific subset of your allocations.
18178
18179To use custom memory pools:
18180
18181-# Fill VmaPoolCreateInfo structure.
18182-# Call vmaCreatePool() to obtain #VmaPool handle.
18183-# When making an allocation, set VmaAllocationCreateInfo::pool to this handle.
18184 You don't need to specify any other parameters of this structure, like `usage`.
18185
18186Example:
18187
18188\code
18189// Find memoryTypeIndex for the pool.
18190VkBufferCreateInfo sampleBufCreateInfo = { VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO };
18191sampleBufCreateInfo.size = 0x10000; // Doesn't matter.
18192sampleBufCreateInfo.usage = VK_BUFFER_USAGE_UNIFORM_BUFFER_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT;
18193
18194VmaAllocationCreateInfo sampleAllocCreateInfo = {};
18195sampleAllocCreateInfo.usage = VMA_MEMORY_USAGE_AUTO;
18196
18197uint32_t memTypeIndex;
18198VkResult res = vmaFindMemoryTypeIndexForBufferInfo(allocator,
18199 &sampleBufCreateInfo, &sampleAllocCreateInfo, &memTypeIndex);
18200// Check res...
18201
18202// Create a pool that can have at most 2 blocks, 128 MiB each.
18203VmaPoolCreateInfo poolCreateInfo = {};
18204poolCreateInfo.memoryTypeIndex = memTypeIndex;
18205poolCreateInfo.blockSize = 128ull * 1024 * 1024;
18206poolCreateInfo.maxBlockCount = 2;
18207
18208VmaPool pool;
18209res = vmaCreatePool(allocator, &poolCreateInfo, &pool);
18210// Check res...
18211
18212// Allocate a buffer out of it.
18213VkBufferCreateInfo bufCreateInfo = { VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO };
18214bufCreateInfo.size = 1024;
18215bufCreateInfo.usage = VK_BUFFER_USAGE_UNIFORM_BUFFER_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT;
18216
18217VmaAllocationCreateInfo allocCreateInfo = {};
18218allocCreateInfo.pool = pool;
18219
18220VkBuffer buf;
18221VmaAllocation alloc;
18222res = vmaCreateBuffer(allocator, &bufCreateInfo, &allocCreateInfo, &buf, &alloc, nullptr);
18223// Check res...
18224\endcode
18225
18226You have to free all allocations made from this pool before destroying it.
18227
18228\code
18229vmaDestroyBuffer(allocator, buf, alloc);
18230vmaDestroyPool(allocator, pool);
18231\endcode
18232
18233New versions of this library support creating dedicated allocations in custom pools.
18234It is supported only when VmaPoolCreateInfo::blockSize = 0.
18235To use this feature, set VmaAllocationCreateInfo::pool to the pointer to your custom pool and
18236VmaAllocationCreateInfo::flags to #VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT.
18237
18238\note Excessive use of custom pools is a common mistake when using this library.
18239Custom pools may be useful for special purposes - when you want to
18240keep certain type of resources separate e.g. to reserve minimum amount of memory
18241for them or limit maximum amount of memory they can occupy. For most
18242resources this is not needed and so it is not recommended to create #VmaPool
18243objects and allocations out of them. Allocating from the default pool is sufficient.
18244
18245
18246\section custom_memory_pools_MemTypeIndex Choosing memory type index
18247
18248When creating a pool, you must explicitly specify memory type index.
18249To find the one suitable for your buffers or images, you can use helper functions
18250vmaFindMemoryTypeIndexForBufferInfo(), vmaFindMemoryTypeIndexForImageInfo().
18251You need to provide structures with example parameters of buffers or images
18252that you are going to create in that pool.
18253
18254\code
18255VkBufferCreateInfo exampleBufCreateInfo = { VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO };
18256exampleBufCreateInfo.size = 1024; // Doesn't matter
18257exampleBufCreateInfo.usage = VK_BUFFER_USAGE_UNIFORM_BUFFER_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT;
18258
18259VmaAllocationCreateInfo allocCreateInfo = {};
18260allocCreateInfo.usage = VMA_MEMORY_USAGE_AUTO;
18261
18262uint32_t memTypeIndex;
18263vmaFindMemoryTypeIndexForBufferInfo(allocator, &exampleBufCreateInfo, &allocCreateInfo, &memTypeIndex);
18264
18265VmaPoolCreateInfo poolCreateInfo = {};
18266poolCreateInfo.memoryTypeIndex = memTypeIndex;
18267// ...
18268\endcode
18269
18270When creating buffers/images allocated in that pool, provide following parameters:
18271
18272- `VkBufferCreateInfo`: Prefer to pass same parameters as above.
18273 Otherwise you risk creating resources in a memory type that is not suitable for them, which may result in undefined behavior.
18274 Using different `VK_BUFFER_USAGE_` flags may work, but you shouldn't create images in a pool intended for buffers
18275 or the other way around.
18276- VmaAllocationCreateInfo: You don't need to pass same parameters. Fill only `pool` member.
18277 Other members are ignored anyway.
18278
18279\section linear_algorithm Linear allocation algorithm
18280
18281Each Vulkan memory block managed by this library has accompanying metadata that
18282keeps track of used and unused regions. By default, the metadata structure and
18283algorithm tries to find best place for new allocations among free regions to
18284optimize memory usage. This way you can allocate and free objects in any order.
18285
18286![Default allocation algorithm](../gfx/Linear_allocator_1_algo_default.png)
18287
18288Sometimes there is a need to use simpler, linear allocation algorithm. You can
18289create custom pool that uses such algorithm by adding flag
18290#VMA_POOL_CREATE_LINEAR_ALGORITHM_BIT to VmaPoolCreateInfo::flags while creating
18291#VmaPool object. Then an alternative metadata management is used. It always
18292creates new allocations after last one and doesn't reuse free regions after
18293allocations freed in the middle. It results in better allocation performance and
18294less memory consumed by metadata.
18295
18296![Linear allocation algorithm](../gfx/Linear_allocator_2_algo_linear.png)
18297
18298With this one flag, you can create a custom pool that can be used in many ways:
18299free-at-once, stack, double stack, and ring buffer. See below for details.
18300You don't need to specify explicitly which of these options you are going to use - it is detected automatically.
18301
18302\subsection linear_algorithm_free_at_once Free-at-once
18303
18304In a pool that uses linear algorithm, you still need to free all the allocations
18305individually, e.g. by using vmaFreeMemory() or vmaDestroyBuffer(). You can free
18306them in any order. New allocations are always made after last one - free space
18307in the middle is not reused. However, when you release all the allocation and
18308the pool becomes empty, allocation starts from the beginning again. This way you
18309can use linear algorithm to speed up creation of allocations that you are going
18310to release all at once.
18311
18312![Free-at-once](../gfx/Linear_allocator_3_free_at_once.png)
18313
18314This mode is also available for pools created with VmaPoolCreateInfo::maxBlockCount
18315value that allows multiple memory blocks.
18316
18317\subsection linear_algorithm_stack Stack
18318
18319When you free an allocation that was created last, its space can be reused.
18320Thanks to this, if you always release allocations in the order opposite to their
18321creation (LIFO - Last In First Out), you can achieve behavior of a stack.
18322
18323![Stack](../gfx/Linear_allocator_4_stack.png)
18324
18325This mode is also available for pools created with VmaPoolCreateInfo::maxBlockCount
18326value that allows multiple memory blocks.
18327
18328\subsection linear_algorithm_double_stack Double stack
18329
18330The space reserved by a custom pool with linear algorithm may be used by two
18331stacks:
18332
18333- First, default one, growing up from offset 0.
18334- Second, "upper" one, growing down from the end towards lower offsets.
18335
18336To make allocation from the upper stack, add flag #VMA_ALLOCATION_CREATE_UPPER_ADDRESS_BIT
18337to VmaAllocationCreateInfo::flags.
18338
18339![Double stack](../gfx/Linear_allocator_7_double_stack.png)
18340
18341Double stack is available only in pools with one memory block -
18342VmaPoolCreateInfo::maxBlockCount must be 1. Otherwise behavior is undefined.
18343
18344When the two stacks' ends meet so there is not enough space between them for a
18345new allocation, such allocation fails with usual
18346`VK_ERROR_OUT_OF_DEVICE_MEMORY` error.
18347
18348\subsection linear_algorithm_ring_buffer Ring buffer
18349
18350When you free some allocations from the beginning and there is not enough free space
18351for a new one at the end of a pool, allocator's "cursor" wraps around to the
18352beginning and starts allocation there. Thanks to this, if you always release
18353allocations in the same order as you created them (FIFO - First In First Out),
18354you can achieve behavior of a ring buffer / queue.
18355
18356![Ring buffer](../gfx/Linear_allocator_5_ring_buffer.png)
18357
18358Ring buffer is available only in pools with one memory block -
18359VmaPoolCreateInfo::maxBlockCount must be 1. Otherwise behavior is undefined.
18360
18361\note \ref defragmentation is not supported in custom pools created with #VMA_POOL_CREATE_LINEAR_ALGORITHM_BIT.
18362
18363
18364\page defragmentation Defragmentation
18365
18366Interleaved allocations and deallocations of many objects of varying size can
18367cause fragmentation over time, which can lead to a situation where the library is unable
18368to find a continuous range of free memory for a new allocation despite there is
18369enough free space, just scattered across many small free ranges between existing
18370allocations.
18371
18372To mitigate this problem, you can use defragmentation feature.
18373It doesn't happen automatically though and needs your cooperation,
18374because VMA is a low level library that only allocates memory.
18375It cannot recreate buffers and images in a new place as it doesn't remember the contents of `VkBufferCreateInfo` / `VkImageCreateInfo` structures.
18376It cannot copy their contents as it doesn't record any commands to a command buffer.
18377
18378Example:
18379
18380\code
18381VmaDefragmentationInfo defragInfo = {};
18382defragInfo.pool = myPool;
18383defragInfo.flags = VMA_DEFRAGMENTATION_FLAG_ALGORITHM_FAST_BIT;
18384
18385VmaDefragmentationContext defragCtx;
18386VkResult res = vmaBeginDefragmentation(allocator, &defragInfo, &defragCtx);
18387// Check res...
18388
18389for(;;)
18390{
18391 VmaDefragmentationPassMoveInfo pass;
18392 res = vmaBeginDefragmentationPass(allocator, defragCtx, &pass);
18393 if(res == VK_SUCCESS)
18394 break;
18395 else if(res != VK_INCOMPLETE)
18396 // Handle error...
18397
18398 for(uint32_t i = 0; i < pass.moveCount; ++i)
18399 {
18400 // Inspect pass.pMoves[i].srcAllocation, identify what buffer/image it represents.
18401 VmaAllocationInfo allocInfo;
18402 vmaGetAllocationInfo(allocator, pMoves[i].srcAllocation, &allocInfo);
18403 MyEngineResourceData* resData = (MyEngineResourceData*)allocInfo.pUserData;
18404
18405 // Recreate and bind this buffer/image at: pass.pMoves[i].dstMemory, pass.pMoves[i].dstOffset.
18406 VkImageCreateInfo imgCreateInfo = ...
18407 VkImage newImg;
18408 res = vkCreateImage(device, &imgCreateInfo, nullptr, &newImg);
18409 // Check res...
18410 res = vmaBindImageMemory(allocator, pMoves[i].dstTmpAllocation, newImg);
18411 // Check res...
18412
18413 // Issue a vkCmdCopyBuffer/vkCmdCopyImage to copy its content to the new place.
18414 vkCmdCopyImage(cmdBuf, resData->img, ..., newImg, ...);
18415 }
18416
18417 // Make sure the copy commands finished executing.
18418 vkWaitForFences(...);
18419
18420 // Destroy old buffers/images bound with pass.pMoves[i].srcAllocation.
18421 for(uint32_t i = 0; i < pass.moveCount; ++i)
18422 {
18423 // ...
18424 vkDestroyImage(device, resData->img, nullptr);
18425 }
18426
18427 // Update appropriate descriptors to point to the new places...
18428
18429 res = vmaEndDefragmentationPass(allocator, defragCtx, &pass);
18430 if(res == VK_SUCCESS)
18431 break;
18432 else if(res != VK_INCOMPLETE)
18433 // Handle error...
18434}
18435
18436vmaEndDefragmentation(allocator, defragCtx, nullptr);
18437\endcode
18438
18439Although functions like vmaCreateBuffer(), vmaCreateImage(), vmaDestroyBuffer(), vmaDestroyImage()
18440create/destroy an allocation and a buffer/image at once, these are just a shortcut for
18441creating the resource, allocating memory, and binding them together.
18442Defragmentation works on memory allocations only. You must handle the rest manually.
18443Defragmentation is an iterative process that should repreat "passes" as long as related functions
18444return `VK_INCOMPLETE` not `VK_SUCCESS`.
18445In each pass:
18446
184471. vmaBeginDefragmentationPass() function call:
18448 - Calculates and returns the list of allocations to be moved in this pass.
18449 Note this can be a time-consuming process.
18450 - Reserves destination memory for them by creating temporary destination allocations
18451 that you can query for their `VkDeviceMemory` + offset using vmaGetAllocationInfo().
184522. Inside the pass, **you should**:
18453 - Inspect the returned list of allocations to be moved.
18454 - Create new buffers/images and bind them at the returned destination temporary allocations.
18455 - Copy data from source to destination resources if necessary.
18456 - Destroy the source buffers/images, but NOT their allocations.
184573. vmaEndDefragmentationPass() function call:
18458 - Frees the source memory reserved for the allocations that are moved.
18459 - Modifies source #VmaAllocation objects that are moved to point to the destination reserved memory.
18460 - Frees `VkDeviceMemory` blocks that became empty.
18461
18462Unlike in previous iterations of the defragmentation API, there is no list of "movable" allocations passed as a parameter.
18463Defragmentation algorithm tries to move all suitable allocations.
18464You can, however, refuse to move some of them inside a defragmentation pass, by setting
18465`pass.pMoves[i].operation` to #VMA_DEFRAGMENTATION_MOVE_OPERATION_IGNORE.
18466This is not recommended and may result in suboptimal packing of the allocations after defragmentation.
18467If you cannot ensure any allocation can be moved, it is better to keep movable allocations separate in a custom pool.
18468
18469Inside a pass, for each allocation that should be moved:
18470
18471- You should copy its data from the source to the destination place by calling e.g. `vkCmdCopyBuffer()`, `vkCmdCopyImage()`.
18472 - You need to make sure these commands finished executing before destroying the source buffers/images and before calling vmaEndDefragmentationPass().
18473- If a resource doesn't contain any meaningful data, e.g. it is a transient color attachment image to be cleared,
18474 filled, and used temporarily in each rendering frame, you can just recreate this image
18475 without copying its data.
18476- If the resource is in `HOST_VISIBLE` and `HOST_CACHED` memory, you can copy its data on the CPU
18477 using `memcpy()`.
18478- If you cannot move the allocation, you can set `pass.pMoves[i].operation` to #VMA_DEFRAGMENTATION_MOVE_OPERATION_IGNORE.
18479 This will cancel the move.
18480 - vmaEndDefragmentationPass() will then free the destination memory
18481 not the source memory of the allocation, leaving it unchanged.
18482- If you decide the allocation is unimportant and can be destroyed instead of moved (e.g. it wasn't used for long time),
18483 you can set `pass.pMoves[i].operation` to #VMA_DEFRAGMENTATION_MOVE_OPERATION_DESTROY.
18484 - vmaEndDefragmentationPass() will then free both source and destination memory, and will destroy the source #VmaAllocation object.
18485
18486You can defragment a specific custom pool by setting VmaDefragmentationInfo::pool
18487(like in the example above) or all the default pools by setting this member to null.
18488
18489Defragmentation is always performed in each pool separately.
18490Allocations are never moved between different Vulkan memory types.
18491The size of the destination memory reserved for a moved allocation is the same as the original one.
18492Alignment of an allocation as it was determined using `vkGetBufferMemoryRequirements()` etc. is also respected after defragmentation.
18493Buffers/images should be recreated with the same `VkBufferCreateInfo` / `VkImageCreateInfo` parameters as the original ones.
18494
18495You can perform the defragmentation incrementally to limit the number of allocations and bytes to be moved
18496in each pass, e.g. to call it in sync with render frames and not to experience too big hitches.
18497See members: VmaDefragmentationInfo::maxBytesPerPass, VmaDefragmentationInfo::maxAllocationsPerPass.
18498
18499It is also safe to perform the defragmentation asynchronously to render frames and other Vulkan and VMA
18500usage, possibly from multiple threads, with the exception that allocations
18501returned in VmaDefragmentationPassMoveInfo::pMoves shouldn't be destroyed until the defragmentation pass is ended.
18502
18503<b>Mapping</b> is preserved on allocations that are moved during defragmentation.
18504Whether through #VMA_ALLOCATION_CREATE_MAPPED_BIT or vmaMapMemory(), the allocations
18505are mapped at their new place. Of course, pointer to the mapped data changes, so it needs to be queried
18506using VmaAllocationInfo::pMappedData.
18507
18508\note Defragmentation is not supported in custom pools created with #VMA_POOL_CREATE_LINEAR_ALGORITHM_BIT.
18509
18510
18511\page statistics Statistics
18512
18513This library contains several functions that return information about its internal state,
18514especially the amount of memory allocated from Vulkan.
18515
18516\section statistics_numeric_statistics Numeric statistics
18517
18518If you need to obtain basic statistics about memory usage per heap, together with current budget,
18519you can call function vmaGetHeapBudgets() and inspect structure #VmaBudget.
18520This is useful to keep track of memory usage and stay withing budget
18521(see also \ref staying_within_budget).
18522Example:
18523
18524\code
18525uint32_t heapIndex = ...
18526
18527VmaBudget budgets[VK_MAX_MEMORY_HEAPS];
18528vmaGetHeapBudgets(allocator, budgets);
18529
18530printf("My heap currently has %u allocations taking %llu B,\n",
18531 budgets[heapIndex].statistics.allocationCount,
18532 budgets[heapIndex].statistics.allocationBytes);
18533printf("allocated out of %u Vulkan device memory blocks taking %llu B,\n",
18534 budgets[heapIndex].statistics.blockCount,
18535 budgets[heapIndex].statistics.blockBytes);
18536printf("Vulkan reports total usage %llu B with budget %llu B.\n",
18537 budgets[heapIndex].usage,
18538 budgets[heapIndex].budget);
18539\endcode
18540
18541You can query for more detailed statistics per memory heap, type, and totals,
18542including minimum and maximum allocation size and unused range size,
18543by calling function vmaCalculateStatistics() and inspecting structure #VmaTotalStatistics.
18544This function is slower though, as it has to traverse all the internal data structures,
18545so it should be used only for debugging purposes.
18546
18547You can query for statistics of a custom pool using function vmaGetPoolStatistics()
18548or vmaCalculatePoolStatistics().
18549
18550You can query for information about a specific allocation using function vmaGetAllocationInfo().
18551It fill structure #VmaAllocationInfo.
18552
18553\section statistics_json_dump JSON dump
18554
18555You can dump internal state of the allocator to a string in JSON format using function vmaBuildStatsString().
18556The result is guaranteed to be correct JSON.
18557It uses ANSI encoding.
18558Any strings provided by user (see [Allocation names](@ref allocation_names))
18559are copied as-is and properly escaped for JSON, so if they use UTF-8, ISO-8859-2 or any other encoding,
18560this JSON string can be treated as using this encoding.
18561It must be freed using function vmaFreeStatsString().
18562
18563The format of this JSON string is not part of official documentation of the library,
18564but it will not change in backward-incompatible way without increasing library major version number
18565and appropriate mention in changelog.
18566
18567The JSON string contains all the data that can be obtained using vmaCalculateStatistics().
18568It can also contain detailed map of allocated memory blocks and their regions -
18569free and occupied by allocations.
18570This allows e.g. to visualize the memory or assess fragmentation.
18571
18572
18573\page allocation_annotation Allocation names and user data
18574
18575\section allocation_user_data Allocation user data
18576
18577You can annotate allocations with your own information, e.g. for debugging purposes.
18578To do that, fill VmaAllocationCreateInfo::pUserData field when creating
18579an allocation. It is an opaque `void*` pointer. You can use it e.g. as a pointer,
18580some handle, index, key, ordinal number or any other value that would associate
18581the allocation with your custom metadata.
18582It it useful to identify appropriate data structures in your engine given #VmaAllocation,
18583e.g. when doing \ref defragmentation.
18584
18585\code
18586VkBufferCreateInfo bufCreateInfo = ...
18587
18588MyBufferMetadata* pMetadata = CreateBufferMetadata();
18589
18590VmaAllocationCreateInfo allocCreateInfo = {};
18591allocCreateInfo.usage = VMA_MEMORY_USAGE_AUTO;
18592allocCreateInfo.pUserData = pMetadata;
18593
18594VkBuffer buffer;
18595VmaAllocation allocation;
18596vmaCreateBuffer(allocator, &bufCreateInfo, &allocCreateInfo, &buffer, &allocation, nullptr);
18597\endcode
18598
18599The pointer may be later retrieved as VmaAllocationInfo::pUserData:
18600
18601\code
18602VmaAllocationInfo allocInfo;
18603vmaGetAllocationInfo(allocator, allocation, &allocInfo);
18604MyBufferMetadata* pMetadata = (MyBufferMetadata*)allocInfo.pUserData;
18605\endcode
18606
18607It can also be changed using function vmaSetAllocationUserData().
18608
18609Values of (non-zero) allocations' `pUserData` are printed in JSON report created by
18610vmaBuildStatsString() in hexadecimal form.
18611
18612\section allocation_names Allocation names
18613
18614An allocation can also carry a null-terminated string, giving a name to the allocation.
18615To set it, call vmaSetAllocationName().
18616The library creates internal copy of the string, so the pointer you pass doesn't need
18617to be valid for whole lifetime of the allocation. You can free it after the call.
18618
18619\code
18620std::string imageName = "Texture: ";
18621imageName += fileName;
18622vmaSetAllocationName(allocator, allocation, imageName.c_str());
18623\endcode
18624
18625The string can be later retrieved by inspecting VmaAllocationInfo::pName.
18626It is also printed in JSON report created by vmaBuildStatsString().
18627
18628\note Setting string name to VMA allocation doesn't automatically set it to the Vulkan buffer or image created with it.
18629You must do it manually using an extension like VK_EXT_debug_utils, which is independent of this library.
18630
18631
18632\page virtual_allocator Virtual allocator
18633
18634As an extra feature, the core allocation algorithm of the library is exposed through a simple and convenient API of "virtual allocator".
18635It doesn't allocate any real GPU memory. It just keeps track of used and free regions of a "virtual block".
18636You can use it to allocate your own memory or other objects, even completely unrelated to Vulkan.
18637A common use case is sub-allocation of pieces of one large GPU buffer.
18638
18639\section virtual_allocator_creating_virtual_block Creating virtual block
18640
18641To use this functionality, there is no main "allocator" object.
18642You don't need to have #VmaAllocator object created.
18643All you need to do is to create a separate #VmaVirtualBlock object for each block of memory you want to be managed by the allocator:
18644
18645-# Fill in #VmaVirtualBlockCreateInfo structure.
18646-# Call vmaCreateVirtualBlock(). Get new #VmaVirtualBlock object.
18647
18648Example:
18649
18650\code
18651VmaVirtualBlockCreateInfo blockCreateInfo = {};
18652blockCreateInfo.size = 1048576; // 1 MB
18653
18654VmaVirtualBlock block;
18655VkResult res = vmaCreateVirtualBlock(&blockCreateInfo, &block);
18656\endcode
18657
18658\section virtual_allocator_making_virtual_allocations Making virtual allocations
18659
18660#VmaVirtualBlock object contains internal data structure that keeps track of free and occupied regions
18661using the same code as the main Vulkan memory allocator.
18662Similarly to #VmaAllocation for standard GPU allocations, there is #VmaVirtualAllocation type
18663that represents an opaque handle to an allocation withing the virtual block.
18664
18665In order to make such allocation:
18666
18667-# Fill in #VmaVirtualAllocationCreateInfo structure.
18668-# Call vmaVirtualAllocate(). Get new #VmaVirtualAllocation object that represents the allocation.
18669 You can also receive `VkDeviceSize offset` that was assigned to the allocation.
18670
18671Example:
18672
18673\code
18674VmaVirtualAllocationCreateInfo allocCreateInfo = {};
18675allocCreateInfo.size = 4096; // 4 KB
18676
18677VmaVirtualAllocation alloc;
18678VkDeviceSize offset;
18679res = vmaVirtualAllocate(block, &allocCreateInfo, &alloc, &offset);
18680if(res == VK_SUCCESS)
18681{
18682 // Use the 4 KB of your memory starting at offset.
18683}
18684else
18685{
18686 // Allocation failed - no space for it could be found. Handle this error!
18687}
18688\endcode
18689
18690\section virtual_allocator_deallocation Deallocation
18691
18692When no longer needed, an allocation can be freed by calling vmaVirtualFree().
18693You can only pass to this function an allocation that was previously returned by vmaVirtualAllocate()
18694called for the same #VmaVirtualBlock.
18695
18696When whole block is no longer needed, the block object can be released by calling vmaDestroyVirtualBlock().
18697All allocations must be freed before the block is destroyed, which is checked internally by an assert.
18698However, if you don't want to call vmaVirtualFree() for each allocation, you can use vmaClearVirtualBlock() to free them all at once -
18699a feature not available in normal Vulkan memory allocator. Example:
18700
18701\code
18702vmaVirtualFree(block, alloc);
18703vmaDestroyVirtualBlock(block);
18704\endcode
18705
18706\section virtual_allocator_allocation_parameters Allocation parameters
18707
18708You can attach a custom pointer to each allocation by using vmaSetVirtualAllocationUserData().
18709Its default value is null.
18710It can be used to store any data that needs to be associated with that allocation - e.g. an index, a handle, or a pointer to some
18711larger data structure containing more information. Example:
18712
18713\code
18714struct CustomAllocData
18715{
18716 std::string m_AllocName;
18717};
18718CustomAllocData* allocData = new CustomAllocData();
18719allocData->m_AllocName = "My allocation 1";
18720vmaSetVirtualAllocationUserData(block, alloc, allocData);
18721\endcode
18722
18723The pointer can later be fetched, along with allocation offset and size, by passing the allocation handle to function
18724vmaGetVirtualAllocationInfo() and inspecting returned structure #VmaVirtualAllocationInfo.
18725If you allocated a new object to be used as the custom pointer, don't forget to delete that object before freeing the allocation!
18726Example:
18727
18728\code
18729VmaVirtualAllocationInfo allocInfo;
18730vmaGetVirtualAllocationInfo(block, alloc, &allocInfo);
18731delete (CustomAllocData*)allocInfo.pUserData;
18732
18733vmaVirtualFree(block, alloc);
18734\endcode
18735
18736\section virtual_allocator_alignment_and_units Alignment and units
18737
18738It feels natural to express sizes and offsets in bytes.
18739If an offset of an allocation needs to be aligned to a multiply of some number (e.g. 4 bytes), you can fill optional member
18740VmaVirtualAllocationCreateInfo::alignment to request it. Example:
18741
18742\code
18743VmaVirtualAllocationCreateInfo allocCreateInfo = {};
18744allocCreateInfo.size = 4096; // 4 KB
18745allocCreateInfo.alignment = 4; // Returned offset must be a multiply of 4 B
18746
18747VmaVirtualAllocation alloc;
18748res = vmaVirtualAllocate(block, &allocCreateInfo, &alloc, nullptr);
18749\endcode
18750
18751Alignments of different allocations made from one block may vary.
18752However, if all alignments and sizes are always multiply of some size e.g. 4 B or `sizeof(MyDataStruct)`,
18753you can express all sizes, alignments, and offsets in multiples of that size instead of individual bytes.
18754It might be more convenient, but you need to make sure to use this new unit consistently in all the places:
18755
18756- VmaVirtualBlockCreateInfo::size
18757- VmaVirtualAllocationCreateInfo::size and VmaVirtualAllocationCreateInfo::alignment
18758- Using offset returned by vmaVirtualAllocate() or in VmaVirtualAllocationInfo::offset
18759
18760\section virtual_allocator_statistics Statistics
18761
18762You can obtain statistics of a virtual block using vmaGetVirtualBlockStatistics()
18763(to get brief statistics that are fast to calculate)
18764or vmaCalculateVirtualBlockStatistics() (to get more detailed statistics, slower to calculate).
18765The functions fill structures #VmaStatistics, #VmaDetailedStatistics respectively - same as used by the normal Vulkan memory allocator.
18766Example:
18767
18768\code
18769VmaStatistics stats;
18770vmaGetVirtualBlockStatistics(block, &stats);
18771printf("My virtual block has %llu bytes used by %u virtual allocations\n",
18772 stats.allocationBytes, stats.allocationCount);
18773\endcode
18774
18775You can also request a full list of allocations and free regions as a string in JSON format by calling
18776vmaBuildVirtualBlockStatsString().
18777Returned string must be later freed using vmaFreeVirtualBlockStatsString().
18778The format of this string differs from the one returned by the main Vulkan allocator, but it is similar.
18779
18780\section virtual_allocator_additional_considerations Additional considerations
18781
18782The "virtual allocator" functionality is implemented on a level of individual memory blocks.
18783Keeping track of a whole collection of blocks, allocating new ones when out of free space,
18784deleting empty ones, and deciding which one to try first for a new allocation must be implemented by the user.
18785
18786Alternative allocation algorithms are supported, just like in custom pools of the real GPU memory.
18787See enum #VmaVirtualBlockCreateFlagBits to learn how to specify them (e.g. #VMA_VIRTUAL_BLOCK_CREATE_LINEAR_ALGORITHM_BIT).
18788You can find their description in chapter \ref custom_memory_pools.
18789Allocation strategies are also supported.
18790See enum #VmaVirtualAllocationCreateFlagBits to learn how to specify them (e.g. #VMA_VIRTUAL_ALLOCATION_CREATE_STRATEGY_MIN_TIME_BIT).
18791
18792Following features are supported only by the allocator of the real GPU memory and not by virtual allocations:
18793buffer-image granularity, `VMA_DEBUG_MARGIN`, `VMA_MIN_ALIGNMENT`.
18794
18795
18796\page debugging_memory_usage Debugging incorrect memory usage
18797
18798If you suspect a bug with memory usage, like usage of uninitialized memory or
18799memory being overwritten out of bounds of an allocation,
18800you can use debug features of this library to verify this.
18801
18802\section debugging_memory_usage_initialization Memory initialization
18803
18804If you experience a bug with incorrect and nondeterministic data in your program and you suspect uninitialized memory to be used,
18805you can enable automatic memory initialization to verify this.
18806To do it, define macro `VMA_DEBUG_INITIALIZE_ALLOCATIONS` to 1.
18807
18808\code
18809#define VMA_DEBUG_INITIALIZE_ALLOCATIONS 1
18810#include "vk_mem_alloc.h"
18811\endcode
18812
18813It makes memory of new allocations initialized to bit pattern `0xDCDCDCDC`.
18814Before an allocation is destroyed, its memory is filled with bit pattern `0xEFEFEFEF`.
18815Memory is automatically mapped and unmapped if necessary.
18816
18817If you find these values while debugging your program, good chances are that you incorrectly
18818read Vulkan memory that is allocated but not initialized, or already freed, respectively.
18819
18820Memory initialization works only with memory types that are `HOST_VISIBLE` and with allocations that can be mapped.
18821It works also with dedicated allocations.
18822
18823\section debugging_memory_usage_margins Margins
18824
18825By default, allocations are laid out in memory blocks next to each other if possible
18826(considering required alignment, `bufferImageGranularity`, and `nonCoherentAtomSize`).
18827
18828![Allocations without margin](../gfx/Margins_1.png)
18829
18830Define macro `VMA_DEBUG_MARGIN` to some non-zero value (e.g. 16) to enforce specified
18831number of bytes as a margin after every allocation.
18832
18833\code
18834#define VMA_DEBUG_MARGIN 16
18835#include "vk_mem_alloc.h"
18836\endcode
18837
18838![Allocations with margin](../gfx/Margins_2.png)
18839
18840If your bug goes away after enabling margins, it means it may be caused by memory
18841being overwritten outside of allocation boundaries. It is not 100% certain though.
18842Change in application behavior may also be caused by different order and distribution
18843of allocations across memory blocks after margins are applied.
18844
18845Margins work with all types of memory.
18846
18847Margin is applied only to allocations made out of memory blocks and not to dedicated
18848allocations, which have their own memory block of specific size.
18849It is thus not applied to allocations made using #VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT flag
18850or those automatically decided to put into dedicated allocations, e.g. due to its
18851large size or recommended by VK_KHR_dedicated_allocation extension.
18852
18853Margins appear in [JSON dump](@ref statistics_json_dump) as part of free space.
18854
18855Note that enabling margins increases memory usage and fragmentation.
18856
18857Margins do not apply to \ref virtual_allocator.
18858
18859\section debugging_memory_usage_corruption_detection Corruption detection
18860
18861You can additionally define macro `VMA_DEBUG_DETECT_CORRUPTION` to 1 to enable validation
18862of contents of the margins.
18863
18864\code
18865#define VMA_DEBUG_MARGIN 16
18866#define VMA_DEBUG_DETECT_CORRUPTION 1
18867#include "vk_mem_alloc.h"
18868\endcode
18869
18870When this feature is enabled, number of bytes specified as `VMA_DEBUG_MARGIN`
18871(it must be multiply of 4) after every allocation is filled with a magic number.
18872This idea is also know as "canary".
18873Memory is automatically mapped and unmapped if necessary.
18874
18875This number is validated automatically when the allocation is destroyed.
18876If it is not equal to the expected value, `VMA_ASSERT()` is executed.
18877It clearly means that either CPU or GPU overwritten the memory outside of boundaries of the allocation,
18878which indicates a serious bug.
18879
18880You can also explicitly request checking margins of all allocations in all memory blocks
18881that belong to specified memory types by using function vmaCheckCorruption(),
18882or in memory blocks that belong to specified custom pool, by using function
18883vmaCheckPoolCorruption().
18884
18885Margin validation (corruption detection) works only for memory types that are
18886`HOST_VISIBLE` and `HOST_COHERENT`.
18887
18888
18889\page opengl_interop OpenGL Interop
18890
18891VMA provides some features that help with interoperability with OpenGL.
18892
18893\section opengl_interop_exporting_memory Exporting memory
18894
18895If you want to attach `VkExportMemoryAllocateInfoKHR` structure to `pNext` chain of memory allocations made by the library:
18896
18897It is recommended to create \ref custom_memory_pools for such allocations.
18898Define and fill in your `VkExportMemoryAllocateInfoKHR` structure and attach it to VmaPoolCreateInfo::pMemoryAllocateNext
18899while creating the custom pool.
18900Please note that the structure must remain alive and unchanged for the whole lifetime of the #VmaPool,
18901not only while creating it, as no copy of the structure is made,
18902but its original pointer is used for each allocation instead.
18903
18904If you want to export all memory allocated by the library from certain memory types,
18905also dedicated allocations or other allocations made from default pools,
18906an alternative solution is to fill in VmaAllocatorCreateInfo::pTypeExternalMemoryHandleTypes.
18907It should point to an array with `VkExternalMemoryHandleTypeFlagsKHR` to be automatically passed by the library
18908through `VkExportMemoryAllocateInfoKHR` on each allocation made from a specific memory type.
18909Please note that new versions of the library also support dedicated allocations created in custom pools.
18910
18911You should not mix these two methods in a way that allows to apply both to the same memory type.
18912Otherwise, `VkExportMemoryAllocateInfoKHR` structure would be attached twice to the `pNext` chain of `VkMemoryAllocateInfo`.
18913
18914
18915\section opengl_interop_custom_alignment Custom alignment
18916
18917Buffers or images exported to a different API like OpenGL may require a different alignment,
18918higher than the one used by the library automatically, queried from functions like `vkGetBufferMemoryRequirements`.
18919To impose such alignment:
18920
18921It is recommended to create \ref custom_memory_pools for such allocations.
18922Set VmaPoolCreateInfo::minAllocationAlignment member to the minimum alignment required for each allocation
18923to be made out of this pool.
18924The alignment actually used will be the maximum of this member and the alignment returned for the specific buffer or image
18925from a function like `vkGetBufferMemoryRequirements`, which is called by VMA automatically.
18926
18927If you want to create a buffer with a specific minimum alignment out of default pools,
18928use special function vmaCreateBufferWithAlignment(), which takes additional parameter `minAlignment`.
18929
18930Note the problem of alignment affects only resources placed inside bigger `VkDeviceMemory` blocks and not dedicated
18931allocations, as these, by definition, always have alignment = 0 because the resource is bound to the beginning of its dedicated block.
18932Contrary to Direct3D 12, Vulkan doesn't have a concept of alignment of the entire memory block passed on its allocation.
18933
18934
18935\page usage_patterns Recommended usage patterns
18936
18937Vulkan gives great flexibility in memory allocation.
18938This chapter shows the most common patterns.
18939
18940See also slides from talk:
18941[Sawicki, Adam. Advanced Graphics Techniques Tutorial: Memory management in Vulkan and DX12. Game Developers Conference, 2018](https://www.gdcvault.com/play/1025458/Advanced-Graphics-Techniques-Tutorial-New)
18942
18943
18944\section usage_patterns_gpu_only GPU-only resource
18945
18946<b>When:</b>
18947Any resources that you frequently write and read on GPU,
18948e.g. images used as color attachments (aka "render targets"), depth-stencil attachments,
18949images/buffers used as storage image/buffer (aka "Unordered Access View (UAV)").
18950
18951<b>What to do:</b>
18952Let the library select the optimal memory type, which will likely have `VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT`.
18953
18954\code
18955VkImageCreateInfo imgCreateInfo = { VK_STRUCTURE_TYPE_IMAGE_CREATE_INFO };
18956imgCreateInfo.imageType = VK_IMAGE_TYPE_2D;
18957imgCreateInfo.extent.width = 3840;
18958imgCreateInfo.extent.height = 2160;
18959imgCreateInfo.extent.depth = 1;
18960imgCreateInfo.mipLevels = 1;
18961imgCreateInfo.arrayLayers = 1;
18962imgCreateInfo.format = VK_FORMAT_R8G8B8A8_UNORM;
18963imgCreateInfo.tiling = VK_IMAGE_TILING_OPTIMAL;
18964imgCreateInfo.initialLayout = VK_IMAGE_LAYOUT_UNDEFINED;
18965imgCreateInfo.usage = VK_IMAGE_USAGE_SAMPLED_BIT | VK_IMAGE_USAGE_COLOR_ATTACHMENT_BIT;
18966imgCreateInfo.samples = VK_SAMPLE_COUNT_1_BIT;
18967
18968VmaAllocationCreateInfo allocCreateInfo = {};
18969allocCreateInfo.usage = VMA_MEMORY_USAGE_AUTO;
18970allocCreateInfo.flags = VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT;
18971allocCreateInfo.priority = 1.0f;
18972
18973VkImage img;
18974VmaAllocation alloc;
18975vmaCreateImage(allocator, &imgCreateInfo, &allocCreateInfo, &img, &alloc, nullptr);
18976\endcode
18977
18978<b>Also consider:</b>
18979Consider creating them as dedicated allocations using #VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT,
18980especially if they are large or if you plan to destroy and recreate them with different sizes
18981e.g. when display resolution changes.
18982Prefer to create such resources first and all other GPU resources (like textures and vertex buffers) later.
18983When VK_EXT_memory_priority extension is enabled, it is also worth setting high priority to such allocation
18984to decrease chances to be evicted to system memory by the operating system.
18985
18986\section usage_patterns_staging_copy_upload Staging copy for upload
18987
18988<b>When:</b>
18989A "staging" buffer than you want to map and fill from CPU code, then use as a source od transfer
18990to some GPU resource.
18991
18992<b>What to do:</b>
18993Use flag #VMA_ALLOCATION_CREATE_HOST_ACCESS_SEQUENTIAL_WRITE_BIT.
18994Let the library select the optimal memory type, which will always have `VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT`.
18995
18996\code
18997VkBufferCreateInfo bufCreateInfo = { VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO };
18998bufCreateInfo.size = 65536;
18999bufCreateInfo.usage = VK_BUFFER_USAGE_TRANSFER_SRC_BIT;
19000
19001VmaAllocationCreateInfo allocCreateInfo = {};
19002allocCreateInfo.usage = VMA_MEMORY_USAGE_AUTO;
19003allocCreateInfo.flags = VMA_ALLOCATION_CREATE_HOST_ACCESS_SEQUENTIAL_WRITE_BIT |
19004 VMA_ALLOCATION_CREATE_MAPPED_BIT;
19005
19006VkBuffer buf;
19007VmaAllocation alloc;
19008VmaAllocationInfo allocInfo;
19009vmaCreateBuffer(allocator, &bufCreateInfo, &allocCreateInfo, &buf, &alloc, &allocInfo);
19010
19011...
19012
19013memcpy(allocInfo.pMappedData, myData, myDataSize);
19014\endcode
19015
19016<b>Also consider:</b>
19017You can map the allocation using vmaMapMemory() or you can create it as persistenly mapped
19018using #VMA_ALLOCATION_CREATE_MAPPED_BIT, as in the example above.
19019
19020
19021\section usage_patterns_readback Readback
19022
19023<b>When:</b>
19024Buffers for data written by or transferred from the GPU that you want to read back on the CPU,
19025e.g. results of some computations.
19026
19027<b>What to do:</b>
19028Use flag #VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT.
19029Let the library select the optimal memory type, which will always have `VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT`
19030and `VK_MEMORY_PROPERTY_HOST_CACHED_BIT`.
19031
19032\code
19033VkBufferCreateInfo bufCreateInfo = { VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO };
19034bufCreateInfo.size = 65536;
19035bufCreateInfo.usage = VK_BUFFER_USAGE_TRANSFER_DST_BIT;
19036
19037VmaAllocationCreateInfo allocCreateInfo = {};
19038allocCreateInfo.usage = VMA_MEMORY_USAGE_AUTO;
19039allocCreateInfo.flags = VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT |
19040 VMA_ALLOCATION_CREATE_MAPPED_BIT;
19041
19042VkBuffer buf;
19043VmaAllocation alloc;
19044VmaAllocationInfo allocInfo;
19045vmaCreateBuffer(allocator, &bufCreateInfo, &allocCreateInfo, &buf, &alloc, &allocInfo);
19046
19047...
19048
19049const float* downloadedData = (const float*)allocInfo.pMappedData;
19050\endcode
19051
19052
19053\section usage_patterns_advanced_data_uploading Advanced data uploading
19054
19055For resources that you frequently write on CPU via mapped pointer and
19056freqnently read on GPU e.g. as a uniform buffer (also called "dynamic"), multiple options are possible:
19057
19058-# Easiest solution is to have one copy of the resource in `HOST_VISIBLE` memory,
19059 even if it means system RAM (not `DEVICE_LOCAL`) on systems with a discrete graphics card,
19060 and make the device reach out to that resource directly.
19061 - Reads performed by the device will then go through PCI Express bus.
19062 The performace of this access may be limited, but it may be fine depending on the size
19063 of this resource (whether it is small enough to quickly end up in GPU cache) and the sparsity
19064 of access.
19065-# On systems with unified memory (e.g. AMD APU or Intel integrated graphics, mobile chips),
19066 a memory type may be available that is both `HOST_VISIBLE` (available for mapping) and `DEVICE_LOCAL`
19067 (fast to access from the GPU). Then, it is likely the best choice for such type of resource.
19068-# Systems with a discrete graphics card and separate video memory may or may not expose
19069 a memory type that is both `HOST_VISIBLE` and `DEVICE_LOCAL`, also known as Base Address Register (BAR).
19070 If they do, it represents a piece of VRAM (or entire VRAM, if ReBAR is enabled in the motherboard BIOS)
19071 that is available to CPU for mapping.
19072 - Writes performed by the host to that memory go through PCI Express bus.
19073 The performance of these writes may be limited, but it may be fine, especially on PCIe 4.0,
19074 as long as rules of using uncached and write-combined memory are followed - only sequential writes and no reads.
19075-# Finally, you may need or prefer to create a separate copy of the resource in `DEVICE_LOCAL` memory,
19076 a separate "staging" copy in `HOST_VISIBLE` memory and perform an explicit transfer command between them.
19077
19078Thankfully, VMA offers an aid to create and use such resources in the the way optimal
19079for the current Vulkan device. To help the library make the best choice,
19080use flag #VMA_ALLOCATION_CREATE_HOST_ACCESS_SEQUENTIAL_WRITE_BIT together with
19081#VMA_ALLOCATION_CREATE_HOST_ACCESS_ALLOW_TRANSFER_INSTEAD_BIT.
19082It will then prefer a memory type that is both `DEVICE_LOCAL` and `HOST_VISIBLE` (integrated memory or BAR),
19083but if no such memory type is available or allocation from it fails
19084(PC graphics cards have only 256 MB of BAR by default, unless ReBAR is supported and enabled in BIOS),
19085it will fall back to `DEVICE_LOCAL` memory for fast GPU access.
19086It is then up to you to detect that the allocation ended up in a memory type that is not `HOST_VISIBLE`,
19087so you need to create another "staging" allocation and perform explicit transfers.
19088
19089\code
19090VkBufferCreateInfo bufCreateInfo = { VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO };
19091bufCreateInfo.size = 65536;
19092bufCreateInfo.usage = VK_BUFFER_USAGE_UNIFORM_BUFFER_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT;
19093
19094VmaAllocationCreateInfo allocCreateInfo = {};
19095allocCreateInfo.usage = VMA_MEMORY_USAGE_AUTO;
19096allocCreateInfo.flags = VMA_ALLOCATION_CREATE_HOST_ACCESS_SEQUENTIAL_WRITE_BIT |
19097 VMA_ALLOCATION_CREATE_HOST_ACCESS_ALLOW_TRANSFER_INSTEAD_BIT |
19098 VMA_ALLOCATION_CREATE_MAPPED_BIT;
19099
19100VkBuffer buf;
19101VmaAllocation alloc;
19102VmaAllocationInfo allocInfo;
19103vmaCreateBuffer(allocator, &bufCreateInfo, &allocCreateInfo, &buf, &alloc, &allocInfo);
19104
19105VkMemoryPropertyFlags memPropFlags;
19106vmaGetAllocationMemoryProperties(allocator, alloc, &memPropFlags);
19107
19108if(memPropFlags & VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT)
19109{
19110 // Allocation ended up in a mappable memory and is already mapped - write to it directly.
19111
19112 // [Executed in runtime]:
19113 memcpy(allocInfo.pMappedData, myData, myDataSize);
19114}
19115else
19116{
19117 // Allocation ended up in a non-mappable memory - need to transfer.
19118 VkBufferCreateInfo stagingBufCreateInfo = { VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO };
19119 stagingBufCreateInfo.size = 65536;
19120 stagingBufCreateInfo.usage = VK_BUFFER_USAGE_TRANSFER_SRC_BIT;
19121
19122 VmaAllocationCreateInfo stagingAllocCreateInfo = {};
19123 stagingAllocCreateInfo.usage = VMA_MEMORY_USAGE_AUTO;
19124 stagingAllocCreateInfo.flags = VMA_ALLOCATION_CREATE_HOST_ACCESS_SEQUENTIAL_WRITE_BIT |
19125 VMA_ALLOCATION_CREATE_MAPPED_BIT;
19126
19127 VkBuffer stagingBuf;
19128 VmaAllocation stagingAlloc;
19129 VmaAllocationInfo stagingAllocInfo;
19130 vmaCreateBuffer(allocator, &stagingBufCreateInfo, &stagingAllocCreateInfo,
19131 &stagingBuf, &stagingAlloc, stagingAllocInfo);
19132
19133 // [Executed in runtime]:
19134 memcpy(stagingAllocInfo.pMappedData, myData, myDataSize);
19135 //vkCmdPipelineBarrier: VK_ACCESS_HOST_WRITE_BIT --> VK_ACCESS_TRANSFER_READ_BIT
19136 VkBufferCopy bufCopy = {
19137 0, // srcOffset
19138 0, // dstOffset,
19139 myDataSize); // size
19140 vkCmdCopyBuffer(cmdBuf, stagingBuf, buf, 1, &bufCopy);
19141}
19142\endcode
19143
19144\section usage_patterns_other_use_cases Other use cases
19145
19146Here are some other, less obvious use cases and their recommended settings:
19147
19148- An image that is used only as transfer source and destination, but it should stay on the device,
19149 as it is used to temporarily store a copy of some texture, e.g. from the current to the next frame,
19150 for temporal antialiasing or other temporal effects.
19151 - Use `VkImageCreateInfo::usage = VK_IMAGE_USAGE_TRANSFER_SRC_BIT | VK_IMAGE_USAGE_TRANSFER_DST_BIT`
19152 - Use VmaAllocationCreateInfo::usage = #VMA_MEMORY_USAGE_AUTO
19153- An image that is used only as transfer source and destination, but it should be placed
19154 in the system RAM despite it doesn't need to be mapped, because it serves as a "swap" copy to evict
19155 least recently used textures from VRAM.
19156 - Use `VkImageCreateInfo::usage = VK_IMAGE_USAGE_TRANSFER_SRC_BIT | VK_IMAGE_USAGE_TRANSFER_DST_BIT`
19157 - Use VmaAllocationCreateInfo::usage = #VMA_MEMORY_USAGE_AUTO_PREFER_HOST,
19158 as VMA needs a hint here to differentiate from the previous case.
19159- A buffer that you want to map and write from the CPU, directly read from the GPU
19160 (e.g. as a uniform or vertex buffer), but you have a clear preference to place it in device or
19161 host memory due to its large size.
19162 - Use `VkBufferCreateInfo::usage = VK_BUFFER_USAGE_UNIFORM_BUFFER_BIT`
19163 - Use VmaAllocationCreateInfo::usage = #VMA_MEMORY_USAGE_AUTO_PREFER_DEVICE or #VMA_MEMORY_USAGE_AUTO_PREFER_HOST
19164 - Use VmaAllocationCreateInfo::flags = #VMA_ALLOCATION_CREATE_HOST_ACCESS_SEQUENTIAL_WRITE_BIT
19165
19166
19167\page configuration Configuration
19168
19169Please check "CONFIGURATION SECTION" in the code to find macros that you can define
19170before each include of this file or change directly in this file to provide
19171your own implementation of basic facilities like assert, `min()` and `max()` functions,
19172mutex, atomic etc.
19173The library uses its own implementation of containers by default, but you can switch to using
19174STL containers instead.
19175
19176For example, define `VMA_ASSERT(expr)` before including the library to provide
19177custom implementation of the assertion, compatible with your project.
19178By default it is defined to standard C `assert(expr)` in `_DEBUG` configuration
19179and empty otherwise.
19180
19181\section config_Vulkan_functions Pointers to Vulkan functions
19182
19183There are multiple ways to import pointers to Vulkan functions in the library.
19184In the simplest case you don't need to do anything.
19185If the compilation or linking of your program or the initialization of the #VmaAllocator
19186doesn't work for you, you can try to reconfigure it.
19187
19188First, the allocator tries to fetch pointers to Vulkan functions linked statically,
19189like this:
19190
19191\code
19192m_VulkanFunctions.vkAllocateMemory = (PFN_vkAllocateMemory)vkAllocateMemory;
19193\endcode
19194
19195If you want to disable this feature, set configuration macro: `#define VMA_STATIC_VULKAN_FUNCTIONS 0`.
19196
19197Second, you can provide the pointers yourself by setting member VmaAllocatorCreateInfo::pVulkanFunctions.
19198You can fetch them e.g. using functions `vkGetInstanceProcAddr` and `vkGetDeviceProcAddr` or
19199by using a helper library like [volk](https://github.com/zeux/volk).
19200
19201Third, VMA tries to fetch remaining pointers that are still null by calling
19202`vkGetInstanceProcAddr` and `vkGetDeviceProcAddr` on its own.
19203You need to only fill in VmaVulkanFunctions::vkGetInstanceProcAddr and VmaVulkanFunctions::vkGetDeviceProcAddr.
19204Other pointers will be fetched automatically.
19205If you want to disable this feature, set configuration macro: `#define VMA_DYNAMIC_VULKAN_FUNCTIONS 0`.
19206
19207Finally, all the function pointers required by the library (considering selected
19208Vulkan version and enabled extensions) are checked with `VMA_ASSERT` if they are not null.
19209
19210
19211\section custom_memory_allocator Custom host memory allocator
19212
19213If you use custom allocator for CPU memory rather than default operator `new`
19214and `delete` from C++, you can make this library using your allocator as well
19215by filling optional member VmaAllocatorCreateInfo::pAllocationCallbacks. These
19216functions will be passed to Vulkan, as well as used by the library itself to
19217make any CPU-side allocations.
19218
19219\section allocation_callbacks Device memory allocation callbacks
19220
19221The library makes calls to `vkAllocateMemory()` and `vkFreeMemory()` internally.
19222You can setup callbacks to be informed about these calls, e.g. for the purpose
19223of gathering some statistics. To do it, fill optional member
19224VmaAllocatorCreateInfo::pDeviceMemoryCallbacks.
19225
19226\section heap_memory_limit Device heap memory limit
19227
19228When device memory of certain heap runs out of free space, new allocations may
19229fail (returning error code) or they may succeed, silently pushing some existing_
19230memory blocks from GPU VRAM to system RAM (which degrades performance). This
19231behavior is implementation-dependent - it depends on GPU vendor and graphics
19232driver.
19233
19234On AMD cards it can be controlled while creating Vulkan device object by using
19235VK_AMD_memory_overallocation_behavior extension, if available.
19236
19237Alternatively, if you want to test how your program behaves with limited amount of Vulkan device
19238memory available without switching your graphics card to one that really has
19239smaller VRAM, you can use a feature of this library intended for this purpose.
19240To do it, fill optional member VmaAllocatorCreateInfo::pHeapSizeLimit.
19241
19242
19243
19244\page vk_khr_dedicated_allocation VK_KHR_dedicated_allocation
19245
19246VK_KHR_dedicated_allocation is a Vulkan extension which can be used to improve
19247performance on some GPUs. It augments Vulkan API with possibility to query
19248driver whether it prefers particular buffer or image to have its own, dedicated
19249allocation (separate `VkDeviceMemory` block) for better efficiency - to be able
19250to do some internal optimizations. The extension is supported by this library.
19251It will be used automatically when enabled.
19252
19253It has been promoted to core Vulkan 1.1, so if you use eligible Vulkan version
19254and inform VMA about it by setting VmaAllocatorCreateInfo::vulkanApiVersion,
19255you are all set.
19256
19257Otherwise, if you want to use it as an extension:
19258
192591 . When creating Vulkan device, check if following 2 device extensions are
19260supported (call `vkEnumerateDeviceExtensionProperties()`).
19261If yes, enable them (fill `VkDeviceCreateInfo::ppEnabledExtensionNames`).
19262
19263- VK_KHR_get_memory_requirements2
19264- VK_KHR_dedicated_allocation
19265
19266If you enabled these extensions:
19267
192682 . Use #VMA_ALLOCATOR_CREATE_KHR_DEDICATED_ALLOCATION_BIT flag when creating
19269your #VmaAllocator to inform the library that you enabled required extensions
19270and you want the library to use them.
19271
19272\code
19273allocatorInfo.flags |= VMA_ALLOCATOR_CREATE_KHR_DEDICATED_ALLOCATION_BIT;
19274
19275vmaCreateAllocator(&allocatorInfo, &allocator);
19276\endcode
19277
19278That is all. The extension will be automatically used whenever you create a
19279buffer using vmaCreateBuffer() or image using vmaCreateImage().
19280
19281When using the extension together with Vulkan Validation Layer, you will receive
19282warnings like this:
19283
19284_vkBindBufferMemory(): Binding memory to buffer 0x33 but vkGetBufferMemoryRequirements() has not been called on that buffer._
19285
19286It is OK, you should just ignore it. It happens because you use function
19287`vkGetBufferMemoryRequirements2KHR()` instead of standard
19288`vkGetBufferMemoryRequirements()`, while the validation layer seems to be
19289unaware of it.
19290
19291To learn more about this extension, see:
19292
19293- [VK_KHR_dedicated_allocation in Vulkan specification](https://www.khronos.org/registry/vulkan/specs/1.2-extensions/html/chap50.html#VK_KHR_dedicated_allocation)
19294- [VK_KHR_dedicated_allocation unofficial manual](http://asawicki.info/articles/VK_KHR_dedicated_allocation.php5)
19295
19296
19297
19298\page vk_ext_memory_priority VK_EXT_memory_priority
19299
19300VK_EXT_memory_priority is a device extension that allows to pass additional "priority"
19301value to Vulkan memory allocations that the implementation may use prefer certain
19302buffers and images that are critical for performance to stay in device-local memory
19303in cases when the memory is over-subscribed, while some others may be moved to the system memory.
19304
19305VMA offers convenient usage of this extension.
19306If you enable it, you can pass "priority" parameter when creating allocations or custom pools
19307and the library automatically passes the value to Vulkan using this extension.
19308
19309If you want to use this extension in connection with VMA, follow these steps:
19310
19311\section vk_ext_memory_priority_initialization Initialization
19312
193131) Call `vkEnumerateDeviceExtensionProperties` for the physical device.
19314Check if the extension is supported - if returned array of `VkExtensionProperties` contains "VK_EXT_memory_priority".
19315
193162) Call `vkGetPhysicalDeviceFeatures2` for the physical device instead of old `vkGetPhysicalDeviceFeatures`.
19317Attach additional structure `VkPhysicalDeviceMemoryPriorityFeaturesEXT` to `VkPhysicalDeviceFeatures2::pNext` to be returned.
19318Check if the device feature is really supported - check if `VkPhysicalDeviceMemoryPriorityFeaturesEXT::memoryPriority` is true.
19319
193203) While creating device with `vkCreateDevice`, enable this extension - add "VK_EXT_memory_priority"
19321to the list passed as `VkDeviceCreateInfo::ppEnabledExtensionNames`.
19322
193234) While creating the device, also don't set `VkDeviceCreateInfo::pEnabledFeatures`.
19324Fill in `VkPhysicalDeviceFeatures2` structure instead and pass it as `VkDeviceCreateInfo::pNext`.
19325Enable this device feature - attach additional structure `VkPhysicalDeviceMemoryPriorityFeaturesEXT` to
19326`VkPhysicalDeviceFeatures2::pNext` chain and set its member `memoryPriority` to `VK_TRUE`.
19327
193285) While creating #VmaAllocator with vmaCreateAllocator() inform VMA that you
19329have enabled this extension and feature - add #VMA_ALLOCATOR_CREATE_EXT_MEMORY_PRIORITY_BIT
19330to VmaAllocatorCreateInfo::flags.
19331
19332\section vk_ext_memory_priority_usage Usage
19333
19334When using this extension, you should initialize following member:
19335
19336- VmaAllocationCreateInfo::priority when creating a dedicated allocation with #VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT.
19337- VmaPoolCreateInfo::priority when creating a custom pool.
19338
19339It should be a floating-point value between `0.0f` and `1.0f`, where recommended default is `0.5f`.
19340Memory allocated with higher value can be treated by the Vulkan implementation as higher priority
19341and so it can have lower chances of being pushed out to system memory, experiencing degraded performance.
19342
19343It might be a good idea to create performance-critical resources like color-attachment or depth-stencil images
19344as dedicated and set high priority to them. For example:
19345
19346\code
19347VkImageCreateInfo imgCreateInfo = { VK_STRUCTURE_TYPE_IMAGE_CREATE_INFO };
19348imgCreateInfo.imageType = VK_IMAGE_TYPE_2D;
19349imgCreateInfo.extent.width = 3840;
19350imgCreateInfo.extent.height = 2160;
19351imgCreateInfo.extent.depth = 1;
19352imgCreateInfo.mipLevels = 1;
19353imgCreateInfo.arrayLayers = 1;
19354imgCreateInfo.format = VK_FORMAT_R8G8B8A8_UNORM;
19355imgCreateInfo.tiling = VK_IMAGE_TILING_OPTIMAL;
19356imgCreateInfo.initialLayout = VK_IMAGE_LAYOUT_UNDEFINED;
19357imgCreateInfo.usage = VK_IMAGE_USAGE_SAMPLED_BIT | VK_IMAGE_USAGE_COLOR_ATTACHMENT_BIT;
19358imgCreateInfo.samples = VK_SAMPLE_COUNT_1_BIT;
19359
19360VmaAllocationCreateInfo allocCreateInfo = {};
19361allocCreateInfo.usage = VMA_MEMORY_USAGE_AUTO;
19362allocCreateInfo.flags = VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT;
19363allocCreateInfo.priority = 1.0f;
19364
19365VkImage img;
19366VmaAllocation alloc;
19367vmaCreateImage(allocator, &imgCreateInfo, &allocCreateInfo, &img, &alloc, nullptr);
19368\endcode
19369
19370`priority` member is ignored in the following situations:
19371
19372- Allocations created in custom pools: They inherit the priority, along with all other allocation parameters
19373 from the parametrs passed in #VmaPoolCreateInfo when the pool was created.
19374- Allocations created in default pools: They inherit the priority from the parameters
19375 VMA used when creating default pools, which means `priority == 0.5f`.
19376
19377
19378\page vk_amd_device_coherent_memory VK_AMD_device_coherent_memory
19379
19380VK_AMD_device_coherent_memory is a device extension that enables access to
19381additional memory types with `VK_MEMORY_PROPERTY_DEVICE_COHERENT_BIT_AMD` and
19382`VK_MEMORY_PROPERTY_DEVICE_UNCACHED_BIT_AMD` flag. It is useful mostly for
19383allocation of buffers intended for writing "breadcrumb markers" in between passes
19384or draw calls, which in turn are useful for debugging GPU crash/hang/TDR cases.
19385
19386When the extension is available but has not been enabled, Vulkan physical device
19387still exposes those memory types, but their usage is forbidden. VMA automatically
19388takes care of that - it returns `VK_ERROR_FEATURE_NOT_PRESENT` when an attempt
19389to allocate memory of such type is made.
19390
19391If you want to use this extension in connection with VMA, follow these steps:
19392
19393\section vk_amd_device_coherent_memory_initialization Initialization
19394
193951) Call `vkEnumerateDeviceExtensionProperties` for the physical device.
19396Check if the extension is supported - if returned array of `VkExtensionProperties` contains "VK_AMD_device_coherent_memory".
19397
193982) Call `vkGetPhysicalDeviceFeatures2` for the physical device instead of old `vkGetPhysicalDeviceFeatures`.
19399Attach additional structure `VkPhysicalDeviceCoherentMemoryFeaturesAMD` to `VkPhysicalDeviceFeatures2::pNext` to be returned.
19400Check if the device feature is really supported - check if `VkPhysicalDeviceCoherentMemoryFeaturesAMD::deviceCoherentMemory` is true.
19401
194023) While creating device with `vkCreateDevice`, enable this extension - add "VK_AMD_device_coherent_memory"
19403to the list passed as `VkDeviceCreateInfo::ppEnabledExtensionNames`.
19404
194054) While creating the device, also don't set `VkDeviceCreateInfo::pEnabledFeatures`.
19406Fill in `VkPhysicalDeviceFeatures2` structure instead and pass it as `VkDeviceCreateInfo::pNext`.
19407Enable this device feature - attach additional structure `VkPhysicalDeviceCoherentMemoryFeaturesAMD` to
19408`VkPhysicalDeviceFeatures2::pNext` and set its member `deviceCoherentMemory` to `VK_TRUE`.
19409
194105) While creating #VmaAllocator with vmaCreateAllocator() inform VMA that you
19411have enabled this extension and feature - add #VMA_ALLOCATOR_CREATE_AMD_DEVICE_COHERENT_MEMORY_BIT
19412to VmaAllocatorCreateInfo::flags.
19413
19414\section vk_amd_device_coherent_memory_usage Usage
19415
19416After following steps described above, you can create VMA allocations and custom pools
19417out of the special `DEVICE_COHERENT` and `DEVICE_UNCACHED` memory types on eligible
19418devices. There are multiple ways to do it, for example:
19419
19420- You can request or prefer to allocate out of such memory types by adding
19421 `VK_MEMORY_PROPERTY_DEVICE_UNCACHED_BIT_AMD` to VmaAllocationCreateInfo::requiredFlags
19422 or VmaAllocationCreateInfo::preferredFlags. Those flags can be freely mixed with
19423 other ways of \ref choosing_memory_type, like setting VmaAllocationCreateInfo::usage.
19424- If you manually found memory type index to use for this purpose, force allocation
19425 from this specific index by setting VmaAllocationCreateInfo::memoryTypeBits `= 1u << index`.
19426
19427\section vk_amd_device_coherent_memory_more_information More information
19428
19429To learn more about this extension, see [VK_AMD_device_coherent_memory in Vulkan specification](https://www.khronos.org/registry/vulkan/specs/1.2-extensions/man/html/VK_AMD_device_coherent_memory.html)
19430
19431Example use of this extension can be found in the code of the sample and test suite
19432accompanying this library.
19433
19434
19435\page enabling_buffer_device_address Enabling buffer device address
19436
19437Device extension VK_KHR_buffer_device_address
19438allow to fetch raw GPU pointer to a buffer and pass it for usage in a shader code.
19439It has been promoted to core Vulkan 1.2.
19440
19441If you want to use this feature in connection with VMA, follow these steps:
19442
19443\section enabling_buffer_device_address_initialization Initialization
19444
194451) (For Vulkan version < 1.2) Call `vkEnumerateDeviceExtensionProperties` for the physical device.
19446Check if the extension is supported - if returned array of `VkExtensionProperties` contains
19447"VK_KHR_buffer_device_address".
19448
194492) Call `vkGetPhysicalDeviceFeatures2` for the physical device instead of old `vkGetPhysicalDeviceFeatures`.
19450Attach additional structure `VkPhysicalDeviceBufferDeviceAddressFeatures*` to `VkPhysicalDeviceFeatures2::pNext` to be returned.
19451Check if the device feature is really supported - check if `VkPhysicalDeviceBufferDeviceAddressFeatures::bufferDeviceAddress` is true.
19452
194533) (For Vulkan version < 1.2) While creating device with `vkCreateDevice`, enable this extension - add
19454"VK_KHR_buffer_device_address" to the list passed as `VkDeviceCreateInfo::ppEnabledExtensionNames`.
19455
194564) While creating the device, also don't set `VkDeviceCreateInfo::pEnabledFeatures`.
19457Fill in `VkPhysicalDeviceFeatures2` structure instead and pass it as `VkDeviceCreateInfo::pNext`.
19458Enable this device feature - attach additional structure `VkPhysicalDeviceBufferDeviceAddressFeatures*` to
19459`VkPhysicalDeviceFeatures2::pNext` and set its member `bufferDeviceAddress` to `VK_TRUE`.
19460
194615) While creating #VmaAllocator with vmaCreateAllocator() inform VMA that you
19462have enabled this feature - add #VMA_ALLOCATOR_CREATE_BUFFER_DEVICE_ADDRESS_BIT
19463to VmaAllocatorCreateInfo::flags.
19464
19465\section enabling_buffer_device_address_usage Usage
19466
19467After following steps described above, you can create buffers with `VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT*` using VMA.
19468The library automatically adds `VK_MEMORY_ALLOCATE_DEVICE_ADDRESS_BIT*` to
19469allocated memory blocks wherever it might be needed.
19470
19471Please note that the library supports only `VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT*`.
19472The second part of this functionality related to "capture and replay" is not supported,
19473as it is intended for usage in debugging tools like RenderDoc, not in everyday Vulkan usage.
19474
19475\section enabling_buffer_device_address_more_information More information
19476
19477To learn more about this extension, see [VK_KHR_buffer_device_address in Vulkan specification](https://www.khronos.org/registry/vulkan/specs/1.2-extensions/html/chap46.html#VK_KHR_buffer_device_address)
19478
19479Example use of this extension can be found in the code of the sample and test suite
19480accompanying this library.
19481
19482\page general_considerations General considerations
19483
19484\section general_considerations_thread_safety Thread safety
19485
19486- The library has no global state, so separate #VmaAllocator objects can be used
19487 independently.
19488 There should be no need to create multiple such objects though - one per `VkDevice` is enough.
19489- By default, all calls to functions that take #VmaAllocator as first parameter
19490 are safe to call from multiple threads simultaneously because they are
19491 synchronized internally when needed.
19492 This includes allocation and deallocation from default memory pool, as well as custom #VmaPool.
19493- When the allocator is created with #VMA_ALLOCATOR_CREATE_EXTERNALLY_SYNCHRONIZED_BIT
19494 flag, calls to functions that take such #VmaAllocator object must be
19495 synchronized externally.
19496- Access to a #VmaAllocation object must be externally synchronized. For example,
19497 you must not call vmaGetAllocationInfo() and vmaMapMemory() from different
19498 threads at the same time if you pass the same #VmaAllocation object to these
19499 functions.
19500- #VmaVirtualBlock is not safe to be used from multiple threads simultaneously.
19501
19502\section general_considerations_versioning_and_compatibility Versioning and compatibility
19503
19504The library uses [**Semantic Versioning**](https://semver.org/),
19505which means version numbers follow convention: Major.Minor.Patch (e.g. 2.3.0), where:
19506
19507- Incremented Patch version means a release is backward- and forward-compatible,
19508 introducing only some internal improvements, bug fixes, optimizations etc.
19509 or changes that are out of scope of the official API described in this documentation.
19510- Incremented Minor version means a release is backward-compatible,
19511 so existing code that uses the library should continue to work, while some new
19512 symbols could have been added: new structures, functions, new values in existing
19513 enums and bit flags, new structure members, but not new function parameters.
19514- Incrementing Major version means a release could break some backward compatibility.
19515
19516All changes between official releases are documented in file "CHANGELOG.md".
19517
19518\warning Backward compatiblity is considered on the level of C++ source code, not binary linkage.
19519Adding new members to existing structures is treated as backward compatible if initializing
19520the new members to binary zero results in the old behavior.
19521You should always fully initialize all library structures to zeros and not rely on their
19522exact binary size.
19523
19524\section general_considerations_validation_layer_warnings Validation layer warnings
19525
19526When using this library, you can meet following types of warnings issued by
19527Vulkan validation layer. They don't necessarily indicate a bug, so you may need
19528to just ignore them.
19529
19530- *vkBindBufferMemory(): Binding memory to buffer 0xeb8e4 but vkGetBufferMemoryRequirements() has not been called on that buffer.*
19531 - It happens when VK_KHR_dedicated_allocation extension is enabled.
19532 `vkGetBufferMemoryRequirements2KHR` function is used instead, while validation layer seems to be unaware of it.
19533- *Mapping an image with layout VK_IMAGE_LAYOUT_DEPTH_STENCIL_ATTACHMENT_OPTIMAL can result in undefined behavior if this memory is used by the device. Only GENERAL or PREINITIALIZED should be used.*
19534 - It happens when you map a buffer or image, because the library maps entire
19535 `VkDeviceMemory` block, where different types of images and buffers may end
19536 up together, especially on GPUs with unified memory like Intel.
19537- *Non-linear image 0xebc91 is aliased with linear buffer 0xeb8e4 which may indicate a bug.*
19538 - It may happen when you use [defragmentation](@ref defragmentation).
19539
19540\section general_considerations_allocation_algorithm Allocation algorithm
19541
19542The library uses following algorithm for allocation, in order:
19543
19544-# Try to find free range of memory in existing blocks.
19545-# If failed, try to create a new block of `VkDeviceMemory`, with preferred block size.
19546-# If failed, try to create such block with size / 2, size / 4, size / 8.
19547-# If failed, try to allocate separate `VkDeviceMemory` for this allocation,
19548 just like when you use #VMA_ALLOCATION_CREATE_DEDICATED_MEMORY_BIT.
19549-# If failed, choose other memory type that meets the requirements specified in
19550 VmaAllocationCreateInfo and go to point 1.
19551-# If failed, return `VK_ERROR_OUT_OF_DEVICE_MEMORY`.
19552
19553\section general_considerations_features_not_supported Features not supported
19554
19555Features deliberately excluded from the scope of this library:
19556
19557-# **Data transfer.** Uploading (streaming) and downloading data of buffers and images
19558 between CPU and GPU memory and related synchronization is responsibility of the user.
19559 Defining some "texture" object that would automatically stream its data from a
19560 staging copy in CPU memory to GPU memory would rather be a feature of another,
19561 higher-level library implemented on top of VMA.
19562 VMA doesn't record any commands to a `VkCommandBuffer`. It just allocates memory.
19563-# **Recreation of buffers and images.** Although the library has functions for
19564 buffer and image creation: vmaCreateBuffer(), vmaCreateImage(), you need to
19565 recreate these objects yourself after defragmentation. That is because the big
19566 structures `VkBufferCreateInfo`, `VkImageCreateInfo` are not stored in
19567 #VmaAllocation object.
19568-# **Handling CPU memory allocation failures.** When dynamically creating small C++
19569 objects in CPU memory (not Vulkan memory), allocation failures are not checked
19570 and handled gracefully, because that would complicate code significantly and
19571 is usually not needed in desktop PC applications anyway.
19572 Success of an allocation is just checked with an assert.
19573-# **Code free of any compiler warnings.** Maintaining the library to compile and
19574 work correctly on so many different platforms is hard enough. Being free of
19575 any warnings, on any version of any compiler, is simply not feasible.
19576 There are many preprocessor macros that make some variables unused, function parameters unreferenced,
19577 or conditional expressions constant in some configurations.
19578 The code of this library should not be bigger or more complicated just to silence these warnings.
19579 It is recommended to disable such warnings instead.
19580-# This is a C++ library with C interface. **Bindings or ports to any other programming languages** are welcome as external projects but
19581 are not going to be included into this repository.
19582*/
19583

source code of qtbase/src/3rdparty/VulkanMemoryAllocator/vk_mem_alloc.h