├── .gitignore ├── LICENSE ├── README.md ├── build.zig ├── build.zig.zon ├── libso ├── libjdzglobal.h ├── libjdzglobal.zig ├── libjdzglobalwrap.c └── libjdzshared.zig └── src ├── DeferredSpanList.zig ├── SpanList.zig ├── arena.zig ├── bench.zig ├── bounded_mpmc_queue.zig ├── bounded_mpsc_queue.zig ├── bounded_stack.zig ├── global_allocator.zig ├── global_arena_handler.zig ├── grow_shrink_bench.zig ├── jdz_allocator.zig ├── lock.zig ├── shared_allocator.zig ├── shared_arena_handler.zig ├── span.zig ├── span_cache.zig ├── static_config.zig └── utils.zig /.gitignore: -------------------------------------------------------------------------------- 1 | .zig-cache 2 | zig-out 3 | TODO 4 | bench* 5 | grow_shrink_bench* -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright Joad Nacer 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 4 | 5 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 6 | 7 | THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 8 | 9 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # jdz_allocator: A Zig General Purpose Memory Allocator 2 | jdz_allocator is an original general purpose allocator inspired by Mattias Jansson's [rpmalloc](https://github.com/mjansson/rpmalloc) and Microsoft's [mimalloc](https://github.com/microsoft/mimalloc). Although it currently passes all mimalloc-bench tests without faults, it is not yet battle-tested and may error under certain loads or configurations. 3 | 4 | In its default configuration, it uses no global or threadlocal vars, making it compatible with Zig allocator design. This allows it to function without the need for any `deinitThread` calls while still achieving reasonable multi-threaded performance. It may induce false sharing in the case that `N allocating threads > config.shared_arena_batch_size` - I believe this is unavoidable when not using threadlocal variables. 5 | 6 | For optimal performance, a global-state-based allocator is available under `jdz_allocator.JdzGlobalAllocator`. With this allocator, make sure to call `deinitThread` before thread termination to free thread memory for re-use. This allocator exists as a singleton-per-configuration, with distinct instances existing for different configs. 7 | 8 | Please note that this allocator is a work in progress, and has not yet been thoroughly tested. Usage and bug reports are appreciated and will help contribute to this allocator's completion. Performance is also still being worked on. Contributions are welcome. 9 | 10 | Current focus is on improving default JdzAllocator performance (possibly migrating to an arena/core model where possible with ie use of [rseq](https://google.github.io/tcmalloc/rseq.html)). 11 | 12 | This allocator currently does not support page sizes larger than 64KiB. 13 | 14 | # mimalloc-bench 15 | The allocator has been benchmarked in mimalloc-bench against InKryption's [rpmalloc Zig port](https://github.com/InKryption/rpmalloc-zig-port), dweiller's [zimalloc](https://github.com/dweiller/zimalloc) and c malloc. 16 | 17 | In the following images, jdzw refers to JdzGlobalAllocator, jdzl to JdzAllocator, calc to c_allocator, rpz to rpmalloc-zig, and zmlc to zimalloc. Please note that times of 0.0 for rpz indicate a segfault (as rpz does not correctly handle cross-thread frees). The chart is normalised to jdzw's time and capped at 2.0. 18 | 19 | ### Time 20 | ![image](https://i.imgur.com/6CHHwku.png) 21 | ![image](https://i.imgur.com/pzWlAxa.png) 22 | ### RSS 23 | ![image](https://i.imgur.com/sxspFLm.png) 24 | ![image](https://i.imgur.com/WtO0gMR.png) 25 | 26 | High RSS for jdzw on xmalloc-testN is being investigated. 27 | 28 | Raw results here: https://pastebin.com/QDA2UW67 29 | 30 | # Zig Benchmarks 31 | The allocator has been benchmarked against Zig std's GeneralPurposeAllocator and c_allocator, as well as InKryption's [rpmalloc Zig port](https://github.com/InKryption/rpmalloc-zig-port) and dweiller's [zimalloc](https://github.com/dweiller/zimalloc). Please note that any mention of rpmalloc in these benchmarks is in reference to the Zig port, not to the original C implementation. 32 | 33 | Benchmarks consist of 75,000,000 linearly distributed allocations of 1-80000 bytes per thread, with no cross-thread frees. This is an unrealistic benchmark, and will be improved to further guide development. 34 | 35 | jdz_allocator's default configuration performs competitively but is significantly slower than rpmalloc at high contention. The global allocator's performance is consistently matching rpmalloc's, remaining within the margin of error, with considerably less memory usage. 36 | 37 | Benchmarks were run on an 8 core Intel i7-11800H @2.30GHz on Linux in ReleaseFast mode. 38 | 39 | Benchmarks can be run as follows: `zig run -O ReleaseFast src/bench.zig -lc -- [num_threads...]`. 40 | 41 | The allocator can also be linked via LD_PRELOAD for benchmarking with mimalloc-bench using the shared libraries outputted on build - libjdzglobal, libjdzshared and libjdzglobalwrap - the latter adding threadlocal arena deinit calls on pthread destructor. 42 | 43 | ### Performance 44 | ![image](https://i.imgur.com/X93jgs2.png) 45 | ### Memory Usage 46 | zimalloc was excluded from the memory usage charts due to too high memory usage (at 16 threads, RSS was ~1.5M bytes and VSZ was ~2M). 47 | 48 | ![image](https://i.imgur.com/h0MpuMP.png) 49 | ![image](https://i.imgur.com/MINQn7b.png) 50 | 51 | # Usage 52 | Current master is written for Zig 0.13.0. The allocator can be used as follows: 53 | 54 | Create a build.zig.zon file like this: 55 | ```zig 56 | .{ 57 | .name = "testlib", 58 | .version = "0.0.1", 59 | 60 | .dependencies = .{ 61 | .jdz_allocator = .{ 62 | .url = "https://github.com/joadnacer/jdz_allocator/archive/ea14e0efc3328a6b9e7dd53bb91b9eec2efe3f96.tar.gz", 63 | .hash = "12205353c7f550b1aadf245d70a30d5a0a92eb151f735e15358ff35fcfe53343c93f" }, 64 | }, 65 | } 66 | 67 | ``` 68 | 69 | Add these lines to your build.zig: 70 | ```zig 71 | const jdz_allocator = b.dependency("jdz_allocator", .{ 72 | .target = target, 73 | .optimize = optimize, 74 | }); 75 | 76 | exe.root_module.addImport("jdz_allocator", jdz_allocator.module("jdz_allocator")); 77 | ``` 78 | 79 | Use as follows: 80 | ```zig 81 | const jdz_allocator = @import("jdz_allocator"); 82 | 83 | pub fn main() !void { 84 | var jdz = jdz_allocator.JdzAllocator(.{}).init(); 85 | defer jdz.deinit(); 86 | 87 | const allocator = jdz.allocator(); 88 | 89 | const res = try allocator.alloc(u8, 8); 90 | defer allocator.free(res); 91 | } 92 | ``` 93 | 94 | Or if using the global allocator: 95 | ```zig 96 | const jdz_allocator = @import("jdz_allocator"); 97 | 98 | pub fn main() !void { 99 | const JdzGlobalAllocator = jdz_allocator.JdzGlobalAllocator(.{}); 100 | defer JdzGlobalAllocator.deinit(); 101 | defer JdzGlobalAllocator.deinitThread(); // call this from every thread that makes an allocation 102 | 103 | const allocator = JdzGlobalAllocator.allocator(); 104 | 105 | const res = try allocator.alloc(u8, 8); 106 | defer allocator.free(res); 107 | } 108 | ``` 109 | 110 | # Design 111 | As in rpmalloc, allocations occur from 64 KiB spans guaranteeing at least 16 byte block alignment, with each span serving one size class. The span header can be obtained from an allocation through the application of a simple bitmask. 112 | There exist 5 size categories: 113 | 120 | 121 | Large allocations are made to "large spans" which consist of up to 64 contiguous spans. 122 | 123 | Allocations occur from arenas, which may only have one thread allocating at a time, although threads may free concurrently. 124 | 125 | Arenas are distributed to threads through the arena_handler, which in the default configuration will distribute arenas stored in a concurrent linked list through the use of try-locks. Threads will attempt to claim arenas as theirs via the storing of thread-ids, but arena stealing will consistently occur for `N allocating threads > config.shared_arena_batch_size`, leading to allocator-induced false sharing. 126 | 127 | When used as a global allocator, no arena stealing will occur as arenas will be stored as threadlocal vars, guaranteeing ideal performance. 128 | 129 | Arenas consist of the following: 130 | 136 | 137 | The global allocator also makes use of global caches - one for single spans and one for large spans, implemented as bounded MPMC queues. Upon filling of an arena's local caches, or thread deinit, spans will be freed to the global cache to be reused by other arenas. 138 | -------------------------------------------------------------------------------- /build.zig: -------------------------------------------------------------------------------- 1 | const std = @import("std"); 2 | 3 | pub fn build(b: *std.Build) !void { 4 | const target = b.standardTargetOptions(.{}); 5 | 6 | const optimize = b.standardOptimizeOption(.{}); 7 | 8 | const jdz_allocator = b.addModule("jdz_allocator", .{ 9 | .root_source_file = b.path("src/jdz_allocator.zig"), 10 | }); 11 | 12 | const static_lib = b.addStaticLibrary(.{ 13 | .name = "jdz_allocator", 14 | .root_source_file = b.path("src/jdz_allocator.zig"), 15 | .target = target, 16 | .optimize = optimize, 17 | }); 18 | 19 | b.installArtifact(static_lib); 20 | 21 | const libjdzglobal = b.addSharedLibrary(.{ 22 | .name = "jdzglobal", 23 | .root_source_file = b.path("libso/libjdzglobal.zig"), 24 | .target = target, 25 | .optimize = optimize, 26 | .link_libc = true, 27 | }); 28 | 29 | libjdzglobal.root_module.addImport("jdz_allocator", jdz_allocator); 30 | 31 | b.installArtifact(libjdzglobal); 32 | 33 | const libjdzglobalwrap = b.addSharedLibrary(.{ 34 | .name = "jdzglobalwrap", 35 | .root_source_file = b.path("libso/libjdzglobal.zig"), 36 | .target = target, 37 | .optimize = optimize, 38 | .link_libc = true, 39 | }); 40 | 41 | libjdzglobalwrap.addCSourceFile(.{ 42 | .file = b.path("libso/libjdzglobalwrap.c"), 43 | .flags = &[_][]const u8{}, 44 | }); 45 | 46 | libjdzglobalwrap.root_module.addImport("jdz_allocator", jdz_allocator); 47 | 48 | b.installArtifact(libjdzglobalwrap); 49 | 50 | const libjdzshared = b.addSharedLibrary(.{ 51 | .name = "jdzshared", 52 | .root_source_file = b.path("libso/libjdzshared.zig"), 53 | .target = target, 54 | .optimize = optimize, 55 | .link_libc = true, 56 | }); 57 | 58 | libjdzshared.root_module.addImport("jdz_allocator", jdz_allocator); 59 | 60 | b.installArtifact(libjdzshared); 61 | 62 | const tests = b.addTest(.{ 63 | .root_source_file = b.path("src/jdz_allocator.zig"), 64 | .target = target, 65 | .optimize = optimize, 66 | }); 67 | 68 | const run_tests = b.addRunArtifact(tests); 69 | 70 | const test_step = b.step("test", "Run library tests"); 71 | test_step.dependOn(&run_tests.step); 72 | 73 | const bench_exe = b.addExecutable(.{ 74 | .name = "bench", 75 | .root_source_file = b.path("src/bench.zig"), 76 | .target = target, 77 | .optimize = optimize, 78 | .link_libc = true, 79 | }); 80 | 81 | const grow_shrink_bench_exe = b.addExecutable(.{ 82 | .name = "bench", 83 | .root_source_file = b.path("src/grow_shrink_bench.zig"), 84 | .target = target, 85 | .optimize = optimize, 86 | .link_libc = true, 87 | }); 88 | 89 | b.installArtifact(bench_exe); 90 | b.installArtifact(grow_shrink_bench_exe); 91 | 92 | const run_bench_exe = b.addRunArtifact(bench_exe); 93 | const run_grow_shrink_bench_exe = b.addRunArtifact(grow_shrink_bench_exe); 94 | 95 | const run_bench_step = b.step("run-bench", "Run src/bench.zig"); 96 | run_bench_step.dependOn(&run_bench_exe.step); 97 | 98 | const run_grow_shrink_bench_step = b.step("run-grow-shrink-bench", "Run src/grow_shrink_bench.zig"); 99 | run_grow_shrink_bench_step.dependOn(&run_grow_shrink_bench_exe.step); 100 | } 101 | -------------------------------------------------------------------------------- /build.zig.zon: -------------------------------------------------------------------------------- 1 | .{ 2 | .name = "jdz_allocator", 3 | .version = "0.0.3", 4 | .dependencies = .{}, 5 | .paths = .{""}, 6 | } 7 | -------------------------------------------------------------------------------- /libso/libjdzglobal.h: -------------------------------------------------------------------------------- 1 | #include 2 | 3 | extern void* malloc(uintptr_t const a0); 4 | extern void* realloc(void *const a0, uintptr_t const a1); 5 | extern void free(void *const a0); 6 | extern void* calloc(uintptr_t const a0, uintptr_t const a1); 7 | extern void* aligned_alloc(uintptr_t const a0, uintptr_t const a1); 8 | extern int posix_memalign(void **const a0, uintptr_t const a1, uintptr_t const a2); 9 | extern void* memalign(uintptr_t const a0, uintptr_t const a1); 10 | extern void* valloc(uintptr_t const a0); 11 | extern void* pvalloc(uintptr_t const a0); 12 | extern uintptr_t malloc_usable_size(void *const a0); 13 | extern void jdz_deinit_thread(void); -------------------------------------------------------------------------------- /libso/libjdzglobal.zig: -------------------------------------------------------------------------------- 1 | const std = @import("std"); 2 | const jdz_allocator = @import("jdz_allocator"); 3 | 4 | const assert = std.assert; 5 | const log = std.log.scoped(.jdz_allocator); 6 | 7 | const JdzGlobalAllocator = jdz_allocator.JdzGlobalAllocator(.{}); 8 | var allocator = JdzGlobalAllocator.allocator(); 9 | 10 | /// 11 | /// Slightly modified copy of https://github.com/dweiller/zimalloc/blob/main/src/libzimalloc.zig 12 | /// 13 | export fn malloc(len: usize) ?*anyopaque { 14 | log.debug("malloc {d}", .{len}); 15 | return allocateBytes(len, 1, @returnAddress(), false, false, true); 16 | } 17 | 18 | export fn realloc(ptr_opt: ?*anyopaque, len: usize) ?*anyopaque { 19 | log.debug("realloc {?*} {d}", .{ ptr_opt, len }); 20 | if (ptr_opt) |ptr| { 21 | const old_size = JdzGlobalAllocator.usableSize(ptr); 22 | 23 | const bytes_ptr: [*]u8 = @ptrCast(ptr); 24 | const old_slice = bytes_ptr[0..old_size]; 25 | 26 | if (allocator.resize(old_slice, len)) { 27 | return ptr; 28 | } 29 | 30 | const new_mem = allocateBytes(len, 1, @returnAddress(), false, false, true) orelse 31 | return null; 32 | 33 | const copy_len = @min(len, old_slice.len); 34 | @memcpy(new_mem[0..copy_len], old_slice[0..copy_len]); 35 | 36 | allocator.free(old_slice); 37 | 38 | return new_mem; 39 | } 40 | 41 | return allocateBytes(len, 1, @returnAddress(), false, false, true); 42 | } 43 | 44 | export fn free(ptr_opt: ?*anyopaque) void { 45 | log.debug("free {?*}", .{ptr_opt}); 46 | if (ptr_opt) |ptr| { 47 | const old_size = JdzGlobalAllocator.usableSize(ptr); 48 | const bytes_ptr: [*]u8 = @ptrCast(ptr); 49 | const old_slice = bytes_ptr[0..old_size]; 50 | 51 | allocator.free(old_slice); 52 | } 53 | } 54 | 55 | export fn calloc(size: usize, count: usize) ?*anyopaque { 56 | log.debug("calloc {d} {d}", .{ size, count }); 57 | const bytes = size * count; 58 | return allocateBytes(bytes, 1, @returnAddress(), true, false, true); 59 | } 60 | 61 | export fn aligned_alloc(alignment: usize, size: usize) ?*anyopaque { 62 | log.debug("aligned_alloc alignment={d}, size={d}", .{ alignment, size }); 63 | return allocateBytes(size, alignment, @returnAddress(), false, true, true); 64 | } 65 | 66 | export fn posix_memalign(ptr: *?*anyopaque, alignment: usize, size: usize) c_int { 67 | log.debug("posix_memalign ptr={*}, alignment={d}, size={d}", .{ ptr, alignment, size }); 68 | 69 | if (size == 0) { 70 | ptr.* = null; 71 | return 0; 72 | } 73 | 74 | if (@popCount(alignment) != 1 or alignment < @sizeOf(*anyopaque)) { 75 | return @intFromEnum(std.c.E.INVAL); 76 | } 77 | 78 | if (allocateBytes(size, alignment, @returnAddress(), false, false, false)) |p| { 79 | ptr.* = p; 80 | return 0; 81 | } 82 | 83 | return @intFromEnum(std.c.E.NOMEM); 84 | } 85 | 86 | export fn memalign(alignment: usize, size: usize) ?*anyopaque { 87 | log.debug("memalign alignment={d}, size={d}", .{ alignment, size }); 88 | return allocateBytes(size, alignment, @returnAddress(), false, true, true); 89 | } 90 | 91 | export fn valloc(size: usize) ?*anyopaque { 92 | log.debug("valloc {d}", .{size}); 93 | return allocateBytes(size, std.mem.page_size, @returnAddress(), false, false, true); 94 | } 95 | 96 | export fn pvalloc(size: usize) ?*anyopaque { 97 | log.debug("pvalloc {d}", .{size}); 98 | const aligned_size = std.mem.alignForward(usize, size, std.mem.page_size); 99 | return allocateBytes(aligned_size, std.mem.page_size, @returnAddress(), false, false, true); 100 | } 101 | 102 | export fn malloc_usable_size(ptr_opt: ?*anyopaque) usize { 103 | if (ptr_opt) |ptr| { 104 | return JdzGlobalAllocator.usableSize(ptr); 105 | } 106 | 107 | return 0; 108 | } 109 | 110 | export fn jdz_deinit_thread() void { 111 | JdzGlobalAllocator.deinitThread(); 112 | } 113 | 114 | fn allocateBytes( 115 | byte_count: usize, 116 | alignment: usize, 117 | ret_addr: usize, 118 | comptime zero: bool, 119 | comptime check_alignment: bool, 120 | comptime set_errno: bool, 121 | ) ?[*]u8 { 122 | if (byte_count == 0) return null; 123 | 124 | if (check_alignment) { 125 | if (!set_errno) @compileError("check_alignment requires set_errno to be true"); 126 | if (!std.mem.isValidAlign(alignment)) { 127 | @panic("invalid"); 128 | } 129 | } 130 | 131 | const log2_align = std.math.log2_int(usize, alignment); 132 | if (allocator.rawAlloc(byte_count, log2_align, ret_addr)) |ptr| { 133 | @memset(ptr[0..byte_count], if (zero) 0 else undefined); 134 | log.debug("allocated {*}", .{ptr}); 135 | return ptr; 136 | } 137 | log.debug("out of memory", .{}); 138 | if (set_errno) setErrno(.NOMEM); 139 | return null; 140 | } 141 | 142 | fn setErrno(code: std.c.E) void { 143 | std.c._errno().* = @intFromEnum(code); 144 | } 145 | -------------------------------------------------------------------------------- /libso/libjdzglobalwrap.c: -------------------------------------------------------------------------------- 1 | #define _GNU_SOURCE 2 | 3 | #include "libjdzglobal.h" 4 | 5 | #include 6 | #include 7 | 8 | 9 | pthread_key_t _jdz_default_key = (pthread_key_t) -1; 10 | 11 | static void _jdz_thread_destructor(void* value) { 12 | jdz_deinit_thread(); 13 | } 14 | 15 | void __attribute__ ((constructor)) setup(void) { 16 | pthread_key_create(&_jdz_default_key, &_jdz_thread_destructor); 17 | } 18 | 19 | int (*pthread_create_orig)(pthread_t *, const pthread_attr_t *, void *(*) (void *), void *); 20 | 21 | int pthread_create(pthread_t *thread, const pthread_attr_t *attr, void *(*start) (void *), void *arg) { 22 | if (!pthread_create_orig) 23 | pthread_create_orig = dlsym(RTLD_NEXT, "pthread_create"); 24 | 25 | pthread_setspecific(_jdz_default_key, (void*)1 ); 26 | return pthread_create_orig(thread, attr, start, arg); 27 | } 28 | -------------------------------------------------------------------------------- /libso/libjdzshared.zig: -------------------------------------------------------------------------------- 1 | const std = @import("std"); 2 | const jdz_allocator = @import("jdz_allocator"); 3 | 4 | const assert = std.assert; 5 | const log = std.log.scoped(.jdz_allocator); 6 | 7 | var allocator_instance = jdz_allocator.JdzAllocator(.{}).init(); 8 | var allocator = allocator_instance.allocator(); 9 | 10 | /// 11 | /// Slightly modified copy of https://github.com/dweiller/zimalloc/blob/main/src/libzimalloc.zig 12 | /// 13 | export fn malloc(len: usize) ?*anyopaque { 14 | log.debug("malloc {d}", .{len}); 15 | return allocateBytes(len, 1, @returnAddress(), false, false, true); 16 | } 17 | 18 | export fn realloc(ptr_opt: ?*anyopaque, len: usize) ?*anyopaque { 19 | log.debug("realloc {?*} {d}", .{ ptr_opt, len }); 20 | if (ptr_opt) |ptr| { 21 | const old_size = allocator_instance.usableSize(ptr); 22 | 23 | const bytes_ptr: [*]u8 = @ptrCast(ptr); 24 | const old_slice = bytes_ptr[0..old_size]; 25 | 26 | if (allocator.resize(old_slice, len)) { 27 | return ptr; 28 | } 29 | 30 | const new_mem = allocateBytes(len, 1, @returnAddress(), false, false, true) orelse 31 | return null; 32 | 33 | const copy_len = @min(len, old_slice.len); 34 | @memcpy(new_mem[0..copy_len], old_slice[0..copy_len]); 35 | 36 | allocator.free(old_slice); 37 | 38 | return new_mem; 39 | } 40 | 41 | return allocateBytes(len, 1, @returnAddress(), false, false, true); 42 | } 43 | 44 | export fn free(ptr_opt: ?*anyopaque) void { 45 | log.debug("free {?*}", .{ptr_opt}); 46 | if (ptr_opt) |ptr| { 47 | const old_size = allocator_instance.usableSize(ptr); 48 | const bytes_ptr: [*]u8 = @ptrCast(ptr); 49 | const old_slice = bytes_ptr[0..old_size]; 50 | 51 | allocator.free(old_slice); 52 | } 53 | } 54 | 55 | export fn calloc(size: usize, count: usize) ?*anyopaque { 56 | log.debug("calloc {d} {d}", .{ size, count }); 57 | const bytes = size * count; 58 | return allocateBytes(bytes, 1, @returnAddress(), true, false, true); 59 | } 60 | 61 | export fn aligned_alloc(alignment: usize, size: usize) ?*anyopaque { 62 | log.debug("aligned_alloc alignment={d}, size={d}", .{ alignment, size }); 63 | return allocateBytes(size, alignment, @returnAddress(), false, true, true); 64 | } 65 | 66 | export fn posix_memalign(ptr: *?*anyopaque, alignment: usize, size: usize) c_int { 67 | log.debug("posix_memalign ptr={*}, alignment={d}, size={d}", .{ ptr, alignment, size }); 68 | 69 | if (size == 0) { 70 | ptr.* = null; 71 | return 0; 72 | } 73 | 74 | if (@popCount(alignment) != 1 or alignment < @sizeOf(*anyopaque)) { 75 | return @intFromEnum(std.c.E.INVAL); 76 | } 77 | 78 | if (allocateBytes(size, alignment, @returnAddress(), false, false, false)) |p| { 79 | ptr.* = p; 80 | return 0; 81 | } 82 | 83 | return @intFromEnum(std.c.E.NOMEM); 84 | } 85 | 86 | export fn memalign(alignment: usize, size: usize) ?*anyopaque { 87 | log.debug("memalign alignment={d}, size={d}", .{ alignment, size }); 88 | return allocateBytes(size, alignment, @returnAddress(), false, true, true); 89 | } 90 | 91 | export fn valloc(size: usize) ?*anyopaque { 92 | log.debug("valloc {d}", .{size}); 93 | return allocateBytes(size, std.mem.page_size, @returnAddress(), false, false, true); 94 | } 95 | 96 | export fn pvalloc(size: usize) ?*anyopaque { 97 | log.debug("pvalloc {d}", .{size}); 98 | const aligned_size = std.mem.alignForward(usize, size, std.mem.page_size); 99 | return allocateBytes(aligned_size, std.mem.page_size, @returnAddress(), false, false, true); 100 | } 101 | 102 | export fn malloc_usable_size(ptr_opt: ?*anyopaque) usize { 103 | if (ptr_opt) |ptr| { 104 | return allocator_instance.usableSize(ptr); 105 | } 106 | 107 | return 0; 108 | } 109 | 110 | fn allocateBytes( 111 | byte_count: usize, 112 | alignment: usize, 113 | ret_addr: usize, 114 | comptime zero: bool, 115 | comptime check_alignment: bool, 116 | comptime set_errno: bool, 117 | ) ?[*]u8 { 118 | if (byte_count == 0) return null; 119 | 120 | if (check_alignment) { 121 | if (!set_errno) @compileError("check_alignment requires set_errno to be true"); 122 | if (!std.mem.isValidAlign(alignment)) { 123 | @panic("invalid"); 124 | } 125 | } 126 | 127 | const log2_align = std.math.log2_int(usize, alignment); 128 | if (allocator.rawAlloc(byte_count, log2_align, ret_addr)) |ptr| { 129 | @memset(ptr[0..byte_count], if (zero) 0 else undefined); 130 | log.debug("allocated {*}", .{ptr}); 131 | return ptr; 132 | } 133 | log.debug("out of memory", .{}); 134 | if (set_errno) setErrno(.NOMEM); 135 | return null; 136 | } 137 | 138 | fn setErrno(code: std.c.E) void { 139 | std.c._errno().* = @intFromEnum(code); 140 | } 141 | -------------------------------------------------------------------------------- /src/DeferredSpanList.zig: -------------------------------------------------------------------------------- 1 | const std = @import("std"); 2 | const jdz_allocator = @import("jdz_allocator.zig"); 3 | const utils = @import("utils.zig"); 4 | const span_file = @import("span.zig"); 5 | 6 | const Span = span_file.Span; 7 | const JdzAllocConfig = jdz_allocator.JdzAllocConfig; 8 | const testing = std.testing; 9 | const assert = std.debug.assert; 10 | 11 | head: ?*Span = null, 12 | 13 | const Self = @This(); 14 | 15 | pub fn write(self: *Self, span: *Span) void { 16 | while (true) { 17 | span.next = self.head; 18 | 19 | if (@cmpxchgWeak(?*Span, &self.head, span.next, span, .monotonic, .monotonic) == null) { 20 | return; 21 | } 22 | } 23 | } 24 | 25 | pub fn getAndRemoveList(self: *Self) ?*Span { 26 | return @atomicRmw(?*Span, &self.head, .Xchg, null, .monotonic); 27 | } 28 | -------------------------------------------------------------------------------- /src/SpanList.zig: -------------------------------------------------------------------------------- 1 | const std = @import("std"); 2 | const jdz_allocator = @import("jdz_allocator.zig"); 3 | const utils = @import("utils.zig"); 4 | const static_config = @import("static_config.zig"); 5 | const span_file = @import("span.zig"); 6 | 7 | const Span = span_file.Span; 8 | const JdzAllocConfig = jdz_allocator.JdzAllocConfig; 9 | const testing = std.testing; 10 | const assert = std.debug.assert; 11 | 12 | head: ?*Span = null, 13 | tail: ?*Span = null, 14 | 15 | const Self = @This(); 16 | 17 | pub fn write(self: *Self, span: *Span) void { 18 | assertNotInList(span); 19 | 20 | if (self.tail) |tail| { 21 | tail.next = span; 22 | self.tail = span; 23 | span.prev = tail; 24 | } else { 25 | self.head = span; 26 | self.tail = span; 27 | } 28 | } 29 | 30 | pub inline fn tryRead(self: *Self) ?*Span { 31 | return self.head; 32 | } 33 | 34 | pub fn remove(self: *Self, span: *Span) void { 35 | assert(span.prev == null or span.prev != span.next); 36 | 37 | if (span.prev) |prev| prev.next = span.next else self.head = span.next; 38 | if (span.next) |next| next.prev = span.prev else self.tail = span.prev; 39 | 40 | utils.resetLinkedSpan(span); 41 | } 42 | 43 | pub fn removeHead(self: *Self) *Span { 44 | assert(self.head != null); 45 | 46 | const head = self.head.?; 47 | self.head = head.next; 48 | 49 | if (self.head) |new_head| { 50 | new_head.prev = null; 51 | } else { 52 | self.tail = null; 53 | } 54 | 55 | utils.resetLinkedSpan(head); 56 | 57 | return head; 58 | } 59 | 60 | pub fn writeLinkedSpans(self: *Self, linked_spans: *Span) void { 61 | if (self.tail) |tail| { 62 | tail.next = linked_spans; 63 | linked_spans.prev = tail; 64 | } else { 65 | self.head = linked_spans; 66 | } 67 | 68 | var span = linked_spans; 69 | 70 | while (span.next) |next| { 71 | next.prev = span; 72 | 73 | span = next; 74 | } 75 | 76 | self.tail = span; 77 | } 78 | 79 | pub fn getEmptySpans(self: *Self) ?*Span { 80 | if (self.head == null) return null; 81 | 82 | var empty_spans_head: ?*Span = null; 83 | var empty_spans_cur: ?*Span = null; 84 | 85 | var opt_span = self.head; 86 | 87 | while (opt_span) |span| { 88 | assert(span != span.next); 89 | 90 | if (span.isEmpty()) { 91 | opt_span = self.removeFromListGetNext(span); 92 | 93 | if (empty_spans_cur) |empty_span| { 94 | assert(empty_span != span); 95 | 96 | empty_span.next = span; 97 | empty_spans_cur = span; 98 | span.prev = empty_span; 99 | } else { 100 | empty_spans_head = span; 101 | empty_spans_cur = span; 102 | } 103 | } else { 104 | opt_span = span.next; 105 | } 106 | } 107 | 108 | return empty_spans_head; 109 | } 110 | 111 | pub fn getHeadFreeList(self: *Self) *usize { 112 | if (self.head) |head| { 113 | return &head.free_list; 114 | } else { 115 | return @constCast(&static_config.free_list_null); 116 | } 117 | } 118 | 119 | fn removeFromListGetNext(self: *Self, span: *Span) ?*Span { 120 | const next = span.next; 121 | 122 | self.remove(span); 123 | 124 | return next; 125 | } 126 | 127 | inline fn assertNotInList(span: *Span) void { 128 | assert(span.next == null); 129 | assert(span.prev == null); 130 | } 131 | -------------------------------------------------------------------------------- /src/arena.zig: -------------------------------------------------------------------------------- 1 | const std = @import("std"); 2 | const span_cache = @import("span_cache.zig"); 3 | const stack = @import("bounded_stack.zig"); 4 | const jdz_allocator = @import("jdz_allocator.zig"); 5 | const mpsc_queue = @import("bounded_mpsc_queue.zig"); 6 | const global_allocator = @import("global_allocator.zig"); 7 | const global_arena_handler = @import("global_arena_handler.zig"); 8 | const static_config = @import("static_config.zig"); 9 | const utils = @import("utils.zig"); 10 | const span_file = @import("span.zig"); 11 | 12 | const SpanList = @import("SpanList.zig"); 13 | const Span = span_file.Span; 14 | const DeferredSpanList = @import("DeferredSpanList.zig"); 15 | const JdzAllocConfig = jdz_allocator.JdzAllocConfig; 16 | const SizeClass = static_config.SizeClass; 17 | const Value = std.atomic.Value; 18 | 19 | const assert = std.debug.assert; 20 | 21 | const cache_line = std.atomic.cache_line; 22 | 23 | threadlocal var cached_thread_id: ?std.Thread.Id = null; 24 | 25 | pub fn Arena(comptime config: JdzAllocConfig, comptime is_threadlocal: bool) type { 26 | const ArenaSpanCache = span_cache.SpanCache(config.cache_limit); 27 | 28 | const Lock = utils.getArenaLockType(config); 29 | 30 | const ArenaLargeCache = mpsc_queue.BoundedMpscQueue(*Span, config.large_cache_limit); 31 | 32 | const ArenaMapCache = stack.BoundedStack(*Span, config.map_cache_limit); 33 | 34 | const GlobalArenaHandler = global_arena_handler.GlobalArenaHandler(config); 35 | 36 | return struct { 37 | backing_allocator: std.mem.Allocator, 38 | spans: [size_class_count]SpanList, 39 | free_lists: [size_class_count]*usize, 40 | deferred_partial_spans: [size_class_count]DeferredSpanList, 41 | span_count: Value(usize), 42 | cache: ArenaSpanCache, 43 | large_cache: [large_class_count]ArenaLargeCache, 44 | map_cache: [large_class_count]ArenaMapCache, 45 | writer_lock: Lock align(cache_line), 46 | thread_id: ?std.Thread.Id align(cache_line), 47 | next: ?*Self align(cache_line), 48 | is_alloc_master: bool, 49 | 50 | const GlobalAllocator = if (is_threadlocal) global_allocator.JdzGlobalAllocator(config) else {}; 51 | 52 | const Self = @This(); 53 | 54 | pub fn init(writer_lock: Lock, thread_id: ?std.Thread.Id) Self { 55 | var large_cache: [large_class_count]ArenaLargeCache = undefined; 56 | var map_cache: [large_class_count]ArenaMapCache = undefined; 57 | 58 | for (&map_cache) |*cache| { 59 | cache.* = ArenaMapCache.init(); 60 | } 61 | 62 | for (&large_cache) |*cache| { 63 | cache.* = ArenaLargeCache.init(); 64 | } 65 | 66 | return .{ 67 | .backing_allocator = config.backing_allocator, 68 | .spans = .{.{}} ** size_class_count, 69 | .free_lists = .{@constCast(&free_list_null)} ** size_class_count, 70 | .deferred_partial_spans = .{.{}} ** size_class_count, 71 | .span_count = Value(usize).init(0), 72 | .cache = ArenaSpanCache.init(), 73 | .large_cache = large_cache, 74 | .map_cache = map_cache, 75 | .writer_lock = writer_lock, 76 | .thread_id = thread_id, 77 | .next = null, 78 | .is_alloc_master = false, 79 | }; 80 | } 81 | 82 | pub fn deinit(self: *Self) usize { 83 | self.writer_lock.acquire(); 84 | defer self.writer_lock.release(); 85 | 86 | self.freeEmptySpansFromLists(); 87 | 88 | while (self.cache.tryRead()) |span| { 89 | self.freeSpanOnArenaDeinit(span); 90 | } 91 | 92 | for (&self.large_cache) |*large_cache| { 93 | while (large_cache.tryRead()) |span| { 94 | self.freeSpanOnArenaDeinit(span); 95 | } 96 | } 97 | 98 | for (0..self.map_cache.len) |i| { 99 | while (self.getCachedMapped(i)) |span| { 100 | self.freeSpanOnArenaDeinit(span); 101 | } 102 | } 103 | 104 | return self.span_count.load(.monotonic); 105 | } 106 | 107 | pub fn makeMaster(self: *Self) void { 108 | self.is_alloc_master = true; 109 | } 110 | 111 | pub inline fn tryAcquire(self: *Self) bool { 112 | return self.writer_lock.tryAcquire(); 113 | } 114 | 115 | pub inline fn release(self: *Self) void { 116 | self.writer_lock.release(); 117 | } 118 | 119 | /// 120 | /// Small Or Medium Allocations 121 | /// 122 | pub inline fn allocateToSpan(self: *Self, size_class: SizeClass) ?[*]u8 { 123 | assert(size_class.class_idx != span_class.class_idx); 124 | 125 | if (self.free_lists[size_class.class_idx].* != free_list_null) { 126 | const head = self.spans[size_class.class_idx].head.?; 127 | return @call(.always_inline, Span.popFreeListElement, .{head}); 128 | } 129 | 130 | return self.allocateGeneric(size_class); 131 | } 132 | 133 | fn allocateGeneric(self: *Self, size_class: SizeClass) ?[*]u8 { 134 | return self.allocateFromSpanList(size_class) orelse 135 | self.allocateFromDeferredPartialSpans(size_class) orelse 136 | self.allocateFromCacheOrNew(size_class); 137 | } 138 | 139 | fn allocateFromSpanList(self: *Self, size_class: SizeClass) ?[*]u8 { 140 | while (self.spans[size_class.class_idx].tryRead()) |span| { 141 | if (span.isFull()) { 142 | @atomicStore(bool, &span.full, true, .monotonic); 143 | 144 | _ = self.spans[size_class.class_idx].removeHead(); 145 | self.free_lists[size_class.class_idx] = self.spans[size_class.class_idx].getHeadFreeList(); 146 | } else { 147 | return span.allocate(); 148 | } 149 | } 150 | 151 | return null; 152 | } 153 | 154 | fn allocateFromDeferredPartialSpans(self: *Self, size_class: SizeClass) ?[*]u8 { 155 | const partial_span = self.deferred_partial_spans[size_class.class_idx].getAndRemoveList() orelse { 156 | return null; 157 | }; 158 | 159 | self.spans[size_class.class_idx].writeLinkedSpans(partial_span); 160 | self.free_lists[size_class.class_idx] = self.spans[size_class.class_idx].getHeadFreeList(); 161 | 162 | return partial_span.allocate(); 163 | } 164 | 165 | fn allocateFromCacheOrNew(self: *Self, size_class: SizeClass) ?[*]u8 { 166 | const span = self.getSpanFromCacheOrNew() orelse return null; 167 | 168 | span.initialiseFreshSpan(self, size_class); 169 | 170 | self.spans[size_class.class_idx].write(span); 171 | self.free_lists[size_class.class_idx] = self.spans[size_class.class_idx].getHeadFreeList(); 172 | 173 | return span.allocateFromFreshSpan(); 174 | } 175 | 176 | const getSpanFromCacheOrNew = if (config.split_large_spans_to_one) 177 | getSpanFromCacheOrNewSplitting 178 | else 179 | getSpanFromCacheOrNewNonSplitting; 180 | 181 | fn getSpanFromCacheOrNewSplitting(self: *Self) ?*Span { 182 | return self.cache.tryRead() orelse 183 | self.getSpanFromOneSpanLargeCache() orelse 184 | self.getEmptySpansFromLists() orelse 185 | self.getSpansFromMapCache() orelse 186 | self.getSpansFromLargeCache() orelse 187 | self.getSpansExternal(); 188 | } 189 | 190 | fn getSpanFromCacheOrNewNonSplitting(self: *Self) ?*Span { 191 | return self.cache.tryRead() orelse 192 | self.getSpanFromOneSpanLargeCache() orelse 193 | self.getEmptySpansFromLists() orelse 194 | self.getSpansFromMapCache() orelse 195 | self.getSpansExternal(); 196 | } 197 | 198 | fn getSpansExternal(self: *Self) ?*Span { 199 | if (is_threadlocal) { 200 | if (self.getSpansFromGlobalCaches()) |span| { 201 | return span; 202 | } 203 | } 204 | return self.mapSpan(MapMode.multiple, config.span_alloc_count); 205 | } 206 | 207 | inline fn getSpanFromOneSpanLargeCache(self: *Self) ?*Span { 208 | return self.large_cache[0].tryRead(); 209 | } 210 | 211 | fn getEmptySpansFromLists(self: *Self) ?*Span { 212 | var ret_span: ?*Span = null; 213 | 214 | for (0.., &self.spans) |i, *spans| { 215 | var empty_spans = spans.getEmptySpans() orelse continue; 216 | self.free_lists[i] = spans.getHeadFreeList(); 217 | 218 | if (ret_span) |span| self.cacheSpanOrFree(span); 219 | 220 | ret_span = empty_spans; 221 | 222 | while (empty_spans.next) |next| { 223 | ret_span = next; 224 | 225 | self.cacheSpanOrFree(empty_spans); 226 | 227 | empty_spans = next; 228 | } 229 | } 230 | 231 | return ret_span; 232 | } 233 | 234 | fn getSpansFromLargeCache(self: *Self) ?*Span { 235 | var span_count: usize = large_class_count; 236 | 237 | while (span_count >= 2) : (span_count -= 1) { 238 | const large_span = self.large_cache[span_count - 1].tryRead() orelse continue; 239 | 240 | assert(large_span.span_count == span_count); 241 | 242 | self.getSpansFromLargeSpan(large_span); 243 | 244 | return large_span; 245 | } 246 | 247 | return null; 248 | } 249 | 250 | fn getSpansFromGlobalCaches(self: *Self) ?*Span { 251 | assert(is_threadlocal); 252 | 253 | return self.getSpansFromGlobalSpanCache() orelse { 254 | if (config.split_large_spans_to_one) { 255 | return self.getSpansFromGlobalLargeCache(); 256 | } else { 257 | return null; 258 | } 259 | }; 260 | } 261 | 262 | fn getSpansFromGlobalSpanCache(self: *Self) ?*Span { 263 | assert(is_threadlocal); 264 | 265 | for (0..config.span_alloc_count) |_| { 266 | const span = GlobalAllocator.getCachedSpan() orelse break; 267 | 268 | const written = self.cache.tryWrite(span); 269 | 270 | // should never be called if we have spans in cache 271 | assert(written); 272 | } 273 | 274 | return self.cache.tryRead(); 275 | } 276 | 277 | fn getSpansFromGlobalLargeCache(self: *Self) ?*Span { 278 | assert(is_threadlocal); 279 | 280 | var span_count: usize = large_class_count; 281 | 282 | while (span_count >= 2) : (span_count -= 1) { 283 | const large_span = GlobalAllocator.getCachedLargeSpan(span_count - 1) orelse continue; 284 | 285 | assert(large_span.span_count == span_count); 286 | 287 | self.getSpansFromLargeSpan(large_span); 288 | 289 | return large_span; 290 | } 291 | 292 | return null; 293 | } 294 | 295 | fn getSpansFromLargeSpan(self: *Self, span: *Span) void { 296 | const to_cache = span.splitFirstSpanReturnRemaining(); 297 | 298 | const written = self.cache.tryWrite(to_cache); 299 | 300 | // should never be called if we have spans in cache 301 | assert(written); 302 | } 303 | 304 | /// 305 | /// Large Span Allocations 306 | /// 307 | pub inline fn allocateOneSpan(self: *Self, size_class: SizeClass) ?[*]u8 { 308 | const span = self.getSpanFromCacheOrNew() orelse return null; 309 | 310 | span.initialiseFreshSpan(self, size_class); 311 | 312 | return @ptrFromInt(span.alloc_ptr); 313 | } 314 | 315 | pub inline fn allocateToLargeSpan(self: *Self, span_count: usize) ?[*]u8 { 316 | if (self.getLargeSpan(span_count)) |span| { 317 | span.initialiseFreshLargeSpan(self, span.span_count); 318 | 319 | return span.allocateFromLargeSpan(); 320 | } 321 | 322 | return self.allocateFromNewLargeSpan(span_count); 323 | } 324 | 325 | inline fn getLargeSpan(self: *Self, span_count: usize) ?*Span { 326 | const span_count_float: f32 = @floatFromInt(span_count); 327 | const span_overhead: u32 = @intFromFloat(span_count_float * config.large_span_overhead_mul); 328 | const max_span_count = @min(large_class_count, span_count + span_overhead); 329 | 330 | return self.getLargeSpanFromCaches(span_count, max_span_count); 331 | } 332 | 333 | const getLargeSpanFromCaches = if (config.split_large_spans_to_large) 334 | getLargeSpanFromCachesSplitting 335 | else 336 | getLargeSpanFromCachesNonSplitting; 337 | 338 | fn getLargeSpanFromCachesSplitting(self: *Self, span_count: usize, max_count: u32) ?*Span { 339 | return self.getFromLargeCache(span_count, max_count) orelse 340 | self.getFromMapCache(span_count) orelse 341 | self.splitFromLargeCache(span_count, max_count) orelse { 342 | if (is_threadlocal) { 343 | return getFromGlobalLargeCache(span_count, max_count) orelse 344 | self.splitFromGlobalLargeCache(span_count, max_count); 345 | } else { 346 | return null; 347 | } 348 | }; 349 | } 350 | 351 | fn getLargeSpanFromCachesNonSplitting(self: *Self, span_count: usize, max_count: u32) ?*Span { 352 | return self.getFromLargeCache(span_count, max_count) orelse 353 | self.getFromMapCache(span_count) orelse { 354 | if (is_threadlocal) { 355 | return getFromGlobalLargeCache(span_count, max_count); 356 | } else { 357 | return null; 358 | } 359 | }; 360 | } 361 | 362 | fn getFromLargeCache(self: *Self, span_count: usize, max_span_count: usize) ?*Span { 363 | for (span_count..max_span_count) |count| { 364 | const cached = self.large_cache[count - 1].tryRead() orelse continue; 365 | 366 | assert(cached.span_count == count); 367 | 368 | return cached; 369 | } 370 | 371 | return null; 372 | } 373 | 374 | fn getFromGlobalLargeCache(span_count: usize, max_span_count: usize) ?*Span { 375 | assert(is_threadlocal); 376 | 377 | for (span_count..max_span_count) |count| { 378 | const cached = GlobalAllocator.getCachedLargeSpan(count - 1) orelse continue; 379 | 380 | assert(cached.span_count == count); 381 | 382 | return cached; 383 | } 384 | 385 | return null; 386 | } 387 | 388 | fn splitFromLargeCache(self: *Self, desired_count: usize, from_count: usize) ?*Span { 389 | for (from_count..large_class_count) |count| { 390 | const cached = self.large_cache[count - 1].tryRead() orelse continue; 391 | 392 | assert(cached.span_count == count); 393 | 394 | const remaining = cached.splitFirstSpansReturnRemaining(desired_count); 395 | 396 | if (remaining.span_count > 1) 397 | self.cacheLargeSpanOrFree(remaining) 398 | else 399 | self.cacheSpanOrFree(remaining); 400 | 401 | return cached; 402 | } 403 | 404 | return null; 405 | } 406 | 407 | fn splitFromGlobalLargeCache(self: *Self, desired_count: usize, from_count: usize) ?*Span { 408 | assert(is_threadlocal); 409 | 410 | for (from_count..large_class_count) |count| { 411 | const cached = GlobalAllocator.getCachedLargeSpan(count - 1) orelse continue; 412 | 413 | assert(cached.span_count == count); 414 | 415 | const remaining = cached.splitFirstSpansReturnRemaining(desired_count); 416 | 417 | if (remaining.span_count > 1) 418 | self.cacheLargeSpanOrFree(remaining) 419 | else 420 | self.cacheSpanOrFree(remaining); 421 | 422 | return cached; 423 | } 424 | 425 | return null; 426 | } 427 | 428 | fn allocateFromNewLargeSpan(self: *Self, span_count: usize) ?[*]u8 { 429 | const span = self.mapSpan(MapMode.large, span_count) orelse return null; 430 | 431 | span.initialiseFreshLargeSpan(self, span_count); 432 | 433 | return span.allocateFromLargeSpan(); 434 | } 435 | 436 | /// 437 | /// Huge Allocation/Free 438 | /// 439 | pub fn allocateHuge(self: *Self, span_count: usize) ?[*]u8 { 440 | const span = self.mapSpan(MapMode.large, span_count) orelse return null; 441 | 442 | span.initialiseFreshLargeSpan(self, span_count); 443 | 444 | return @ptrFromInt(span.alloc_ptr); 445 | } 446 | 447 | pub const freeHuge = freeSpan; 448 | 449 | /// 450 | /// Span Mapping 451 | /// 452 | fn mapSpan(self: *Self, comptime map_mode: MapMode, span_count: usize) ?*Span { 453 | var map_count = getMapCount(span_count, map_mode); 454 | 455 | // need padding to guarantee allocating enough spans 456 | if (map_count == span_count) map_count += 1; 457 | 458 | const alloc_size = map_count * span_size; 459 | const span_alloc = self.backing_allocator.rawAlloc(alloc_size, page_alignment, @returnAddress()) orelse { 460 | return null; 461 | }; 462 | const span_alloc_ptr = @intFromPtr(span_alloc); 463 | 464 | if ((span_alloc_ptr & mod_span_size) != 0) map_count -= 1; 465 | 466 | if (config.report_leaks) _ = self.span_count.fetchAdd(map_count, .monotonic); 467 | 468 | const span = self.getSpansCacheRemaining(span_alloc_ptr, alloc_size, map_count, span_count); 469 | 470 | return self.desiredMappingToDesiredSpan(span, map_mode); 471 | } 472 | 473 | fn desiredMappingToDesiredSpan(self: *Self, span: *Span, map_mode: MapMode) *Span { 474 | return switch (map_mode) { 475 | .multiple => self.mapMultipleSpans(span), 476 | .large, .huge => span, 477 | }; 478 | } 479 | 480 | inline fn getMapCount(desired_span_count: usize, map_mode: MapMode) usize { 481 | return switch (map_mode) { 482 | .multiple, .large => @max(page_size / span_size, @max(config.map_alloc_count, desired_span_count)), 483 | .huge => desired_span_count, 484 | }; 485 | } 486 | 487 | fn mapMultipleSpans(self: *Self, span: *Span) *Span { 488 | if (span.span_count > 1) { 489 | const remaining = span.splitFirstSpanReturnRemaining(); 490 | 491 | const could_cache = self.cache.tryWrite(remaining); 492 | 493 | // should never be mapping if have spans in span cache 494 | assert(could_cache); 495 | } 496 | 497 | return span; 498 | } 499 | 500 | /// 501 | /// Arena Map Cache 502 | /// 503 | fn getSpansFromMapCache(self: *Self) ?*Span { 504 | const map_cache_min = 2; 505 | 506 | if (self.getFromMapCache(map_cache_min)) |mapped_span| { 507 | return self.desiredMappingToDesiredSpan(mapped_span, .multiple); 508 | } 509 | 510 | return null; 511 | } 512 | 513 | fn getSpansCacheRemaining(self: *Self, span_alloc_ptr: usize, alloc_size: usize, map_count: usize, desired_span_count: usize) *Span { 514 | const span = instantiateMappedSpan(span_alloc_ptr, alloc_size, map_count); 515 | 516 | if (span.span_count > desired_span_count) { 517 | const remaining = span.splitFirstSpansReturnRemaining(desired_span_count); 518 | 519 | if (remaining.span_count == 1) 520 | self.cacheSpanOrFree(remaining) 521 | else 522 | self.cacheFromMapping(remaining); 523 | } 524 | 525 | return span; 526 | } 527 | 528 | fn instantiateMappedSpan(span_alloc_ptr: usize, alloc_size: usize, map_count: usize) *Span { 529 | const after_pad = span_alloc_ptr & (span_size - 1); 530 | const before_pad = if (after_pad != 0) span_size - after_pad else 0; 531 | const span_ptr = span_alloc_ptr + before_pad; 532 | 533 | const span: *Span = @ptrFromInt(span_ptr); 534 | span.initial_ptr = span_alloc_ptr; 535 | span.alloc_size = alloc_size; 536 | span.span_count = map_count; 537 | 538 | return span; 539 | } 540 | 541 | fn getFromMapCache(self: *Self, span_count: usize) ?*Span { 542 | for (span_count..self.map_cache.len) |count| { 543 | const cached_span = self.getCachedMapped(count); 544 | 545 | if (cached_span) |span| { 546 | assert(count == span.span_count); 547 | 548 | if (count > span_count) { 549 | self.splitMappedSpans(span, span_count); 550 | } 551 | 552 | return span; 553 | } 554 | } 555 | 556 | return null; 557 | } 558 | 559 | inline fn splitMappedSpans(self: *Self, span: *Span, span_count: usize) void { 560 | const remaining = span.splitFirstSpansReturnRemaining(span_count); 561 | 562 | if (remaining.span_count == 1) 563 | self.cacheSpanOrFree(remaining) 564 | else 565 | self.cacheMapped(remaining); 566 | } 567 | 568 | fn cacheFromMapping(self: *Self, span: *Span) void { 569 | const map_cache_max = self.map_cache.len - 1; 570 | 571 | while (span.span_count > map_cache_max) { 572 | const remaining = span.splitLastSpans(map_cache_max); 573 | 574 | self.cacheMapped(remaining); 575 | } 576 | 577 | self.cacheMapped(span); 578 | } 579 | 580 | fn cacheMapped(self: *Self, span: *Span) void { 581 | assert(span.span_count < self.map_cache.len); 582 | 583 | if (span.span_count == 1) { 584 | self.cacheSpanOrFree(span); 585 | } else if (!self.map_cache[span.span_count].tryWrite(span)) { 586 | self.cacheLargeSpanOrFree(span); 587 | } 588 | } 589 | 590 | inline fn getCachedMapped(self: *Self, span_count: usize) ?*Span { 591 | return self.map_cache[span_count].tryRead(); 592 | } 593 | 594 | /// 595 | /// Free/Cache Methods 596 | /// 597 | /// 598 | /// Single Span Free/Cache 599 | /// 600 | pub const freeSmallOrMedium = if (is_threadlocal) 601 | freeSmallOrMediumThreadLocal 602 | else 603 | freeSmallOrMediumShared; 604 | 605 | inline fn freeSmallOrMediumThreadLocal(self: *Self, span: *Span, buf: []u8) void { 606 | if (self == @call(.always_inline, GlobalArenaHandler.getThreadArena, .{})) { 607 | @call(.always_inline, Span.pushFreeList, .{ span, buf }); 608 | 609 | @call(.always_inline, handleSpanNoLongerFull, .{ self, span }); 610 | } else { 611 | @call(.always_inline, Span.pushDeferredFreeList, .{ span, buf }); 612 | 613 | @call(.always_inline, handleSpanNoLongerFullDeferred, .{ self, span }); 614 | } 615 | } 616 | 617 | inline fn freeSmallOrMediumShared(self: *Self, span: *Span, buf: []u8) void { 618 | const tid = @call(.always_inline, getThreadId, .{}); 619 | 620 | if (self.thread_id == tid and @call(.always_inline, Self.tryAcquire, .{self})) { 621 | defer @call(.always_inline, Self.release, .{self}); 622 | 623 | @call(.always_inline, Span.pushFreeList, .{ span, buf }); 624 | 625 | @call(.always_inline, handleSpanNoLongerFull, .{ self, span }); 626 | } else { 627 | @call(.always_inline, Span.pushDeferredFreeList, .{ span, buf }); 628 | 629 | @call(.always_inline, handleSpanNoLongerFullDeferred, .{ self, span }); 630 | } 631 | } 632 | 633 | inline fn handleSpanNoLongerFull(self: *Self, span: *Span) void { 634 | if (span.full and @atomicRmw(bool, &span.full, .Xchg, false, .monotonic)) { 635 | self.spans[span.class.class_idx].write(span); 636 | self.free_lists[span.class.class_idx] = self.spans[span.class.class_idx].getHeadFreeList(); 637 | } 638 | } 639 | 640 | inline fn handleSpanNoLongerFullDeferred(self: *Self, span: *Span) void { 641 | if (span.full and @atomicRmw(bool, &span.full, .Xchg, false, .monotonic)) { 642 | self.deferred_partial_spans[span.class.class_idx].write(span); 643 | } 644 | } 645 | 646 | inline fn getThreadId() std.Thread.Id { 647 | return cached_thread_id orelse { 648 | cached_thread_id = std.Thread.getCurrentId(); 649 | 650 | return cached_thread_id.?; 651 | }; 652 | } 653 | 654 | fn freeSpanOnArenaDeinit(self: *Self, span: *Span) void { 655 | if (is_threadlocal and span.span_count == 1) { 656 | self.cacheSpanToGlobalOrFree(span); 657 | } else if (is_threadlocal) { 658 | self.cacheLargeSpanToGlobalOrFree(span); 659 | } else { 660 | self.freeSpan(span); 661 | } 662 | } 663 | 664 | fn cacheSpanToGlobalOrFree(self: *Self, span: *Span) void { 665 | assert(is_threadlocal); 666 | 667 | if (GlobalAllocator.cacheSpan(span)) { 668 | if (config.report_leaks) _ = self.span_count.fetchSub(span.span_count, .monotonic); 669 | } else { 670 | self.freeSpan(span); 671 | } 672 | } 673 | 674 | fn cacheLargeSpanToGlobalOrFree(self: *Self, span: *Span) void { 675 | assert(is_threadlocal); 676 | 677 | if (GlobalAllocator.cacheLargeSpan(span)) { 678 | if (config.report_leaks) _ = self.span_count.fetchSub(span.span_count, .monotonic); 679 | } else { 680 | self.freeSpan(span); 681 | } 682 | } 683 | 684 | fn freeSpan(self: *Self, span: *Span) void { 685 | assert(span.alloc_size >= span_size); 686 | 687 | if (config.report_leaks) _ = self.span_count.fetchSub(span.span_count, .monotonic); 688 | 689 | const initial_alloc = @as([*]u8, @ptrFromInt(span.initial_ptr))[0..span.alloc_size]; 690 | self.backing_allocator.rawFree(initial_alloc, page_alignment, @returnAddress()); 691 | } 692 | 693 | pub inline fn cacheSpanOrFree(self: *Self, span: *Span) void { 694 | if (!self.cache.tryWrite(span)) { 695 | if (is_threadlocal) { 696 | self.cacheSpanToGlobalOrFree(span); 697 | } else { 698 | self.freeSpan(span); 699 | } 700 | } 701 | } 702 | 703 | fn freeEmptySpansFromLists(self: *Self) void { 704 | for (&self.spans) |*spans| { 705 | self.freeList(spans); 706 | } 707 | 708 | for (&self.deferred_partial_spans) |*deferred_partial_spans| { 709 | self.freeDeferredList(deferred_partial_spans); 710 | } 711 | } 712 | 713 | fn freeList(self: *Self, spans: *SpanList) void { 714 | var empty_spans = spans.getEmptySpans(); 715 | 716 | if (empty_spans) |span| { 717 | self.free_lists[span.class.class_idx] = spans.getHeadFreeList(); 718 | } 719 | 720 | while (empty_spans) |span| { 721 | empty_spans = span.next; 722 | 723 | self.freeSpanFromList(span); 724 | } 725 | } 726 | 727 | fn freeDeferredList(self: *Self, deferred_spans: *DeferredSpanList) void { 728 | var spans = deferred_spans.getAndRemoveList(); 729 | 730 | while (spans) |span| { 731 | spans = span.next; 732 | 733 | if (span.isEmpty()) { 734 | self.freeSpanFromList(span); 735 | } else { 736 | self.spans[span.class.class_idx].write(span); 737 | self.free_lists[span.class.class_idx] = self.spans[span.class.class_idx].getHeadFreeList(); 738 | } 739 | } 740 | } 741 | 742 | fn freeSpanFromList(self: *Self, span: *Span) void { 743 | if (is_threadlocal) { 744 | utils.resetLinkedSpan(span); 745 | 746 | self.cacheSpanToGlobalOrFree(span); 747 | } else { 748 | self.freeSpan(span); 749 | } 750 | } 751 | 752 | /// 753 | /// Large Span Free/Cache 754 | /// 755 | pub inline fn cacheLargeSpanOrFree(self: *Self, span: *Span) void { 756 | const span_count = span.span_count; 757 | 758 | if (!self.large_cache[span_count - 1].tryWrite(span)) { 759 | if (is_threadlocal) { 760 | self.cacheLargeSpanToGlobalOrFree(span); 761 | } else { 762 | self.freeSpan(span); 763 | } 764 | } 765 | } 766 | }; 767 | } 768 | 769 | const MapMode = enum { 770 | multiple, 771 | large, 772 | huge, 773 | }; 774 | 775 | const span_size = static_config.span_size; 776 | const span_max = static_config.span_max; 777 | const span_class = static_config.span_class; 778 | 779 | const page_size = static_config.page_size; 780 | const page_alignment = static_config.page_alignment; 781 | 782 | const span_header_size = static_config.span_header_size; 783 | const mod_span_size = static_config.mod_span_size; 784 | 785 | const size_class_count = static_config.size_class_count; 786 | const large_class_count = static_config.large_class_count; 787 | 788 | const free_list_null = static_config.free_list_null; 789 | -------------------------------------------------------------------------------- /src/bench.zig: -------------------------------------------------------------------------------- 1 | const std = @import("std"); 2 | const jdz = @import("jdz_allocator.zig"); 3 | const static_config = @import("static_config.zig"); 4 | 5 | const mixed_rounds = 10_000_000; 6 | const mixed_min = 1; 7 | const mixed_max = 80000; 8 | 9 | const small_rounds = 10_000_000; 10 | const small_min = 1; 11 | const small_max = static_config.small_max; 12 | 13 | const medium_rounds = 10_000_000; 14 | const medium_min = static_config.small_max + 1; 15 | const medium_max = static_config.medium_max; 16 | 17 | const big_rounds = 10_000_000; 18 | const big_min = static_config.medium_max + 1; 19 | const big_max = static_config.large_max; 20 | 21 | const BUFFER_CAPACITY = 256; 22 | 23 | pub fn main() !void { 24 | const args = try std.process.argsAlloc(std.heap.page_allocator); 25 | defer std.process.argsFree(std.heap.page_allocator, args); 26 | 27 | const num_args = args.len - 1; 28 | 29 | if (num_args == 0) return try bench(1); 30 | 31 | for (0..num_args) |i| { 32 | const num_threads = try std.fmt.parseInt(u32, args[i + 1], 10); 33 | 34 | try bench(num_threads); 35 | } 36 | } 37 | 38 | fn bench(num_threads: u32) !void { 39 | try std.io.getStdOut().writer().print("=== Num Threads={} ===\n", .{num_threads}); 40 | 41 | try std.io.getStdOut().writer().print("==Mixed Alloc==\n", .{}); 42 | try jdz_mixed(num_threads); 43 | try jdz_global_mixed(num_threads); 44 | try c_mixed(num_threads); 45 | try gpa_mixed(num_threads); 46 | 47 | try std.io.getStdOut().writer().print("==Small Alloc==\n", .{}); 48 | try jdz_small(num_threads); 49 | try jdz_global_small(num_threads); 50 | try c_small(num_threads); 51 | try gpa_small(num_threads); 52 | 53 | try std.io.getStdOut().writer().print("==Medium Alloc==\n", .{}); 54 | try jdz_medium(num_threads); 55 | try jdz_global_medium(num_threads); 56 | try c_medium(num_threads); 57 | try gpa_medium(num_threads); 58 | 59 | try std.io.getStdOut().writer().print("==Big Alloc==\n", .{}); 60 | try jdz_big(num_threads); 61 | try jdz_global_big(num_threads); 62 | try c_big(num_threads); 63 | try gpa_big(num_threads); 64 | 65 | try std.io.getStdOut().writer().print("\n", .{}); 66 | } 67 | 68 | /// 69 | /// Mixed 70 | /// 71 | fn jdz_mixed(num_threads: u32) !void { 72 | var jdz_allocator = jdz.JdzAllocator(.{}).init(); 73 | defer jdz_allocator.deinit(); 74 | 75 | const allocator = jdz_allocator.allocator(); 76 | 77 | try runPerfTestAlloc("jdz/mixed", mixed_min, mixed_max, allocator, mixed_rounds, num_threads); 78 | } 79 | 80 | fn jdz_global_mixed(num_threads: u32) !void { 81 | const jdz_allocator = jdz.JdzGlobalAllocator(.{}); 82 | defer jdz.JdzGlobalAllocator(.{}).deinit(); 83 | 84 | const allocator = jdz_allocator.allocator(); 85 | 86 | try runPerfTestAlloc("jdz-global/mixed", mixed_min, mixed_max, allocator, mixed_rounds, num_threads); 87 | } 88 | 89 | fn gpa_mixed(num_threads: u32) !void { 90 | var gpa = std.heap.GeneralPurposeAllocator(.{}){}; 91 | 92 | const allocator = gpa.allocator(); 93 | defer _ = gpa.deinit(); 94 | 95 | try runPerfTestAlloc("gpa/mixed", mixed_min, mixed_max, allocator, mixed_rounds, num_threads); 96 | } 97 | 98 | fn c_mixed(num_threads: u32) !void { 99 | const allocator = std.heap.c_allocator; 100 | 101 | try runPerfTestAlloc("c/mixed", mixed_min, mixed_max, allocator, mixed_rounds, num_threads); 102 | } 103 | 104 | /// 105 | /// Small 106 | /// 107 | fn jdz_small(num_threads: u32) !void { 108 | var jdz_allocator = jdz.JdzAllocator(.{}).init(); 109 | defer jdz_allocator.deinit(); 110 | 111 | const allocator = jdz_allocator.allocator(); 112 | 113 | try runPerfTestAlloc("jdz/small", small_min, small_max, allocator, small_rounds, num_threads); 114 | } 115 | 116 | fn c_small(num_threads: u32) !void { 117 | const allocator = std.heap.c_allocator; 118 | 119 | try runPerfTestAlloc("c/small", small_min, small_max, allocator, small_rounds, num_threads); 120 | } 121 | 122 | fn jdz_global_small(num_threads: u32) !void { 123 | const jdz_allocator = jdz.JdzGlobalAllocator(.{}); 124 | defer jdz.JdzGlobalAllocator(.{}).deinit(); 125 | 126 | const allocator = jdz_allocator.allocator(); 127 | 128 | try runPerfTestAlloc("jdz-global/small", small_min, small_max, allocator, small_rounds, num_threads); 129 | } 130 | 131 | fn gpa_small(num_threads: u32) !void { 132 | var gpa = std.heap.GeneralPurposeAllocator(.{}){}; 133 | 134 | const allocator = gpa.allocator(); 135 | defer _ = gpa.deinit(); 136 | 137 | try runPerfTestAlloc("gpa/small", small_min, small_max, allocator, small_rounds, num_threads); 138 | } 139 | 140 | /// 141 | /// Medium 142 | /// 143 | fn jdz_medium(num_threads: u32) !void { 144 | var jdz_allocator = jdz.JdzAllocator(.{}).init(); 145 | defer jdz_allocator.deinit(); 146 | const allocator = jdz_allocator.allocator(); 147 | 148 | try runPerfTestAlloc("jdz/medium", medium_min, medium_max, allocator, medium_rounds, num_threads); 149 | } 150 | 151 | fn jdz_global_medium(num_threads: u32) !void { 152 | const jdz_allocator = jdz.JdzGlobalAllocator(.{}); 153 | defer jdz.JdzGlobalAllocator(.{}).deinit(); 154 | 155 | const allocator = jdz_allocator.allocator(); 156 | 157 | try runPerfTestAlloc("jdz-global/medium", medium_min, medium_max, allocator, medium_rounds, num_threads); 158 | } 159 | 160 | fn c_medium(num_threads: u32) !void { 161 | const allocator = std.heap.c_allocator; 162 | 163 | try runPerfTestAlloc("c/medium", medium_min, medium_max, allocator, medium_rounds, num_threads); 164 | } 165 | 166 | fn gpa_medium(num_threads: u32) !void { 167 | var gpa = std.heap.GeneralPurposeAllocator(.{}){}; 168 | 169 | const allocator = gpa.allocator(); 170 | defer _ = gpa.deinit(); 171 | 172 | try runPerfTestAlloc("gpa/medium", medium_min, medium_max, allocator, medium_rounds, num_threads); 173 | } 174 | 175 | /// 176 | /// Big 177 | /// 178 | fn jdz_big(num_threads: u32) !void { 179 | var jdz_allocator = jdz.JdzAllocator(.{}).init(); 180 | defer jdz_allocator.deinit(); 181 | 182 | const allocator = jdz_allocator.allocator(); 183 | 184 | try runPerfTestAlloc("jdz/big", big_min, big_max, allocator, big_rounds, num_threads); 185 | } 186 | 187 | fn jdz_global_big(num_threads: u32) !void { 188 | const jdz_allocator = jdz.JdzGlobalAllocator(.{}); 189 | defer jdz.JdzGlobalAllocator(.{}).deinit(); 190 | 191 | const allocator = jdz_allocator.allocator(); 192 | 193 | try runPerfTestAlloc("jdz-global/big", big_min, big_max, allocator, big_rounds, num_threads); 194 | } 195 | 196 | fn c_big(num_threads: u32) !void { 197 | const allocator = std.heap.c_allocator; 198 | 199 | try runPerfTestAlloc("c/big", big_min, big_max, allocator, big_rounds, num_threads); 200 | } 201 | 202 | fn gpa_big(num_threads: u32) !void { 203 | var gpa = std.heap.GeneralPurposeAllocator(.{}){}; 204 | 205 | const allocator = gpa.allocator(); 206 | defer _ = gpa.deinit(); 207 | 208 | try runPerfTestAlloc("gpa/big", big_min, big_max, allocator, big_rounds, num_threads); 209 | } 210 | 211 | /// 212 | /// Bench Methods 213 | /// 214 | fn threadAllocWorker(min: usize, max: usize, allocator: std.mem.Allocator, max_rounds: usize) !void { 215 | var slots = std.BoundedArray([]u8, BUFFER_CAPACITY){}; 216 | var rounds: usize = max_rounds; 217 | 218 | var random_source = std.rand.DefaultPrng.init(1337); 219 | const rng = random_source.random(); 220 | 221 | while (rounds > 0) { 222 | rounds -= 1; 223 | 224 | const free_chance = @as(f32, @floatFromInt(slots.len)) / @as(f32, @floatFromInt(slots.buffer.len - 1)); // more elements => more frees 225 | const alloc_chance = 1.0 - free_chance; // more elements => less allocs 226 | const alloc_amount = rng.intRangeAtMost(usize, min, max); 227 | 228 | if (slots.len > 0) { 229 | if (rng.float(f32) <= free_chance) { 230 | const index = rng.intRangeLessThan(usize, 0, slots.len); 231 | const ptr = slots.swapRemove(index); 232 | allocator.free(ptr); 233 | } 234 | } 235 | 236 | if (slots.len < slots.capacity()) { 237 | if (rng.float(f32) <= alloc_chance) { 238 | const item = try allocator.alloc(u8, alloc_amount); 239 | slots.appendAssumeCapacity(item); 240 | } 241 | } 242 | } 243 | 244 | for (slots.slice()) |ptr| { 245 | allocator.free(ptr); 246 | } 247 | } 248 | 249 | fn runPerfTestAlloc(tag: []const u8, min: usize, max: usize, allocator: std.mem.Allocator, max_rounds: usize, num_threads: u32) !void { 250 | var workers: []std.Thread = try std.heap.page_allocator.alloc(std.Thread, num_threads); 251 | defer std.heap.page_allocator.free(workers); 252 | 253 | const begin_time = std.time.nanoTimestamp(); 254 | 255 | for (0..num_threads) |i| { 256 | workers[i] = try std.Thread.spawn(.{}, threadAllocWorker, .{ min, max, allocator, max_rounds }); 257 | } 258 | 259 | for (0..num_threads) |i| { 260 | workers[i].join(); 261 | } 262 | 263 | const end_time = std.time.nanoTimestamp(); 264 | 265 | try std.io.getStdOut().writer().print("time={d: >10.2}us test={s} num_threads={}\n", .{ 266 | @as(f32, @floatFromInt(end_time - begin_time)) / 1000.0, tag, num_threads, 267 | }); 268 | } 269 | -------------------------------------------------------------------------------- /src/bounded_mpmc_queue.zig: -------------------------------------------------------------------------------- 1 | const std = @import("std"); 2 | const utils = @import("utils.zig"); 3 | const testing = std.testing; 4 | const assert = std.debug.assert; 5 | const Value = std.atomic.Value; 6 | 7 | const cache_line = std.atomic.cache_line; 8 | 9 | /// Array based bounded multiple producer multiple consumer queue 10 | /// This is a Zig port of Dmitry Vyukov's https://www.1024cores.net/home/lock-free-algorithms/queues/bounded-mpmc-queue 11 | pub fn BoundedMpmcQueue(comptime T: type, comptime buffer_size: usize) type { 12 | assert(utils.isPowerOfTwo(buffer_size)); 13 | 14 | const buffer_mask = buffer_size - 1; 15 | 16 | const Cell = struct { 17 | sequence: Value(usize), 18 | data: T, 19 | }; 20 | 21 | return struct { 22 | enqueue_pos: Value(usize) align(cache_line), 23 | dequeue_pos: Value(usize) align(cache_line), 24 | buffer: [buffer_size]Cell, 25 | 26 | const Self = @This(); 27 | 28 | pub fn init() BoundedMpmcQueue(T, buffer_size) { 29 | var buf: [buffer_size]Cell = undefined; 30 | 31 | @setEvalBranchQuota(10000); 32 | for (&buf, 0..) |*cell, i| { 33 | cell.sequence = Value(usize).init(i); 34 | } 35 | 36 | return .{ 37 | .enqueue_pos = Value(usize).init(0), 38 | .dequeue_pos = Value(usize).init(0), 39 | .buffer = buf, 40 | }; 41 | } 42 | 43 | /// Attempts to write to the queue, without overwriting any data 44 | /// Returns `true` if the data is written, `false` if the queue was full 45 | pub fn tryWrite(self: *Self, data: T) bool { 46 | var pos = self.enqueue_pos.load(.monotonic); 47 | 48 | var cell: *Cell = undefined; 49 | 50 | while (true) { 51 | cell = &self.buffer[pos & buffer_mask]; 52 | const seq = cell.sequence.load(.acquire); 53 | const diff = @as(i128, seq) - @as(i128, pos); 54 | 55 | if (diff == 0 and utils.tryCASAddOne(&self.enqueue_pos, pos, .monotonic) == null) { 56 | break; 57 | } else if (diff < 0) { 58 | return false; 59 | } else { 60 | pos = self.enqueue_pos.load(.monotonic); 61 | } 62 | } 63 | 64 | cell.data = data; 65 | cell.sequence.store(pos + 1, .release); 66 | 67 | return true; 68 | } 69 | 70 | /// Attempts to read and remove the head element of the queue 71 | /// Returns `null` if there was no element to read 72 | pub fn tryRead(self: *Self) ?T { 73 | var cell: *Cell = undefined; 74 | var pos = self.dequeue_pos.load(.monotonic); 75 | 76 | while (true) { 77 | cell = &self.buffer[pos & buffer_mask]; 78 | const seq = cell.sequence.load(.acquire); 79 | const diff = @as(i128, seq) - @as(i128, (pos + 1)); 80 | 81 | if (diff == 0 and utils.tryCASAddOne(&self.dequeue_pos, pos, .monotonic) == null) { 82 | break; 83 | } else if (diff < 0) { 84 | return null; 85 | } else { 86 | pos = self.dequeue_pos.load(.monotonic); 87 | } 88 | } 89 | 90 | const res = cell.data; 91 | cell.sequence.store(pos + buffer_mask + 1, .release); 92 | 93 | return res; 94 | } 95 | }; 96 | } 97 | 98 | test "tryWrite/tryRead" { 99 | var queue = BoundedMpmcQueue(u64, 16).init(); 100 | 101 | _ = queue.tryWrite(17); 102 | _ = queue.tryWrite(36); 103 | 104 | try testing.expect(queue.tryRead().? == 17); 105 | try testing.expect(queue.tryRead().? == 36); 106 | } 107 | 108 | test "tryRead empty" { 109 | var queue = BoundedMpmcQueue(u64, 16).init(); 110 | 111 | try testing.expect(queue.tryRead() == null); 112 | } 113 | 114 | test "tryRead emptied" { 115 | var queue = BoundedMpmcQueue(u64, 2).init(); 116 | 117 | _ = queue.tryWrite(1); 118 | _ = queue.tryWrite(2); 119 | 120 | try testing.expect(queue.tryRead().? == 1); 121 | try testing.expect(queue.tryRead().? == 2); 122 | try testing.expect(queue.tryRead() == null); 123 | } 124 | 125 | test "tryWrite to full" { 126 | var queue = BoundedMpmcQueue(u64, 2).init(); 127 | 128 | _ = queue.tryWrite(1); 129 | _ = queue.tryWrite(2); 130 | 131 | try testing.expect(queue.tryWrite(3) == false); 132 | try testing.expect(queue.tryRead().? == 1); 133 | try testing.expect(queue.tryRead().? == 2); 134 | } 135 | -------------------------------------------------------------------------------- /src/bounded_mpsc_queue.zig: -------------------------------------------------------------------------------- 1 | const std = @import("std"); 2 | const jdz_allocator = @import("jdz_allocator.zig"); 3 | const utils = @import("utils.zig"); 4 | 5 | const testing = std.testing; 6 | const assert = std.debug.assert; 7 | const Value = std.atomic.Value; 8 | 9 | const cache_line = std.atomic.cache_line; 10 | 11 | /// Array based bounded multiple producer single consumer queue 12 | /// This is a modification of Dmitry Vyukov's https://www.1024cores.net/home/lock-free-algorithms/queues/bounded-mpmc-queue 13 | pub fn BoundedMpscQueue(comptime T: type, comptime buffer_size: usize) type { 14 | assert(utils.isPowerOfTwo(buffer_size)); 15 | 16 | const buffer_mask = buffer_size - 1; 17 | 18 | const Cell = struct { 19 | sequence: Value(usize), 20 | data: T, 21 | }; 22 | 23 | return struct { 24 | enqueue_pos: Value(usize) align(cache_line), 25 | dequeue_pos: usize align(cache_line), 26 | buffer: [buffer_size]Cell, 27 | 28 | const Self = @This(); 29 | 30 | pub fn init() Self { 31 | var buf: [buffer_size]Cell = undefined; 32 | 33 | for (&buf, 0..) |*cell, i| { 34 | cell.sequence = Value(usize).init(i); 35 | } 36 | 37 | return .{ 38 | .enqueue_pos = Value(usize).init(0), 39 | .dequeue_pos = 0, 40 | .buffer = buf, 41 | }; 42 | } 43 | 44 | /// Attempts to write to the queue, without overwriting any data 45 | /// Returns `true` if the data is written, `false` if the queue was full 46 | pub fn tryWrite(self: *Self, data: T) bool { 47 | var pos = self.enqueue_pos.load(.monotonic); 48 | 49 | var cell: *Cell = undefined; 50 | 51 | while (true) { 52 | cell = &self.buffer[pos & buffer_mask]; 53 | const seq = cell.sequence.load(.acquire); 54 | const diff = @as(i128, seq) - @as(i128, pos); 55 | 56 | if (diff == 0 and utils.tryCASAddOne(&self.enqueue_pos, pos, .monotonic) == null) { 57 | break; 58 | } else if (diff < 0) { 59 | return false; 60 | } else { 61 | pos = self.enqueue_pos.load(.monotonic); 62 | } 63 | } 64 | 65 | cell.data = data; 66 | cell.sequence.store(pos + 1, .release); 67 | 68 | return true; 69 | } 70 | 71 | /// Attempts to read and remove the head element of the queue 72 | /// Returns `null` if there was no element to read 73 | pub fn tryRead(self: *Self) ?T { 74 | const cell = &self.buffer[self.dequeue_pos & buffer_mask]; 75 | const seq = cell.sequence.load(.acquire); 76 | const diff = @as(i128, seq) - @as(i128, (self.dequeue_pos + 1)); 77 | 78 | if (diff == 0) { 79 | self.dequeue_pos += 1; 80 | } else { 81 | return null; 82 | } 83 | 84 | const res = cell.data; 85 | cell.sequence.store(self.dequeue_pos + buffer_mask, .release); 86 | 87 | return res; 88 | } 89 | }; 90 | } 91 | 92 | test "tryWrite/tryRead" { 93 | var queue = BoundedMpscQueue(u64, 16).init(); 94 | 95 | _ = queue.tryWrite(17); 96 | _ = queue.tryWrite(36); 97 | 98 | try testing.expect(queue.tryRead().? == 17); 99 | try testing.expect(queue.tryRead().? == 36); 100 | } 101 | 102 | test "tryRead empty" { 103 | var queue = BoundedMpscQueue(u64, 16).init(); 104 | 105 | try testing.expect(queue.tryRead() == null); 106 | } 107 | 108 | test "tryRead emptied" { 109 | var queue = BoundedMpscQueue(u64, 2).init(); 110 | 111 | _ = queue.tryWrite(1); 112 | _ = queue.tryWrite(2); 113 | 114 | try testing.expect(queue.tryRead().? == 1); 115 | try testing.expect(queue.tryRead().? == 2); 116 | try testing.expect(queue.tryRead() == null); 117 | } 118 | 119 | test "tryWrite to full" { 120 | var queue = BoundedMpscQueue(u64, 2).init(); 121 | 122 | _ = queue.tryWrite(1); 123 | _ = queue.tryWrite(2); 124 | 125 | try testing.expect(queue.tryWrite(3) == false); 126 | try testing.expect(queue.tryRead().? == 1); 127 | try testing.expect(queue.tryRead().? == 2); 128 | } 129 | -------------------------------------------------------------------------------- /src/bounded_stack.zig: -------------------------------------------------------------------------------- 1 | const std = @import("std"); 2 | const utils = @import("utils.zig"); 3 | 4 | const testing = std.testing; 5 | const assert = std.debug.assert; 6 | 7 | pub fn BoundedStack(comptime T: type, comptime buffer_size: usize) type { 8 | return extern struct { 9 | buffer_size: usize, 10 | count: usize, 11 | buffer: [buffer_size]T, 12 | 13 | const Self = @This(); 14 | 15 | pub fn init() BoundedStack(T, buffer_size) { 16 | return .{ 17 | .buffer_size = buffer_size, 18 | .count = 0, 19 | .buffer = undefined, 20 | }; 21 | } 22 | 23 | pub fn tryWrite(self: *Self, data: T) bool { 24 | if (self.count == self.buffer_size) return false; 25 | 26 | self.buffer[self.count] = data; 27 | self.count += 1; 28 | 29 | return true; 30 | } 31 | 32 | pub fn tryRead(self: *Self) ?T { 33 | if (self.count == 0) return null; 34 | 35 | self.count -= 1; 36 | 37 | return self.buffer[self.count]; 38 | } 39 | }; 40 | } 41 | 42 | test "tryWrite/tryRead" { 43 | var queue = BoundedStack(u64, 16).init(); 44 | 45 | _ = queue.tryWrite(17); 46 | _ = queue.tryWrite(36); 47 | 48 | try testing.expect(queue.tryRead().? == 36); 49 | try testing.expect(queue.tryRead().? == 17); 50 | } 51 | 52 | test "tryRead empty" { 53 | var queue = BoundedStack(u64, 16).init(); 54 | 55 | try testing.expect(queue.tryRead() == null); 56 | } 57 | 58 | test "tryRead emptied" { 59 | var queue = BoundedStack(u64, 2).init(); 60 | 61 | _ = queue.tryWrite(1); 62 | _ = queue.tryWrite(2); 63 | 64 | try testing.expect(queue.tryRead().? == 2); 65 | try testing.expect(queue.tryRead().? == 1); 66 | try testing.expect(queue.tryRead() == null); 67 | } 68 | 69 | test "tryWrite to full" { 70 | var queue = BoundedStack(u64, 2).init(); 71 | 72 | _ = queue.tryWrite(1); 73 | _ = queue.tryWrite(2); 74 | 75 | try testing.expect(queue.tryWrite(3) == false); 76 | try testing.expect(queue.tryRead().? == 2); 77 | try testing.expect(queue.tryRead().? == 1); 78 | } 79 | -------------------------------------------------------------------------------- /src/global_allocator.zig: -------------------------------------------------------------------------------- 1 | const std = @import("std"); 2 | const builtin = @import("builtin"); 3 | 4 | const jdz = @import("jdz_allocator.zig"); 5 | const global_handler = @import("global_arena_handler.zig"); 6 | const span_arena = @import("arena.zig"); 7 | const static_config = @import("static_config.zig"); 8 | const bounded_mpmc_queue = @import("bounded_mpmc_queue.zig"); 9 | const utils = @import("utils.zig"); 10 | const span_file = @import("span.zig"); 11 | 12 | const Span = span_file.Span; 13 | const JdzAllocConfig = jdz.JdzAllocConfig; 14 | const Value = std.atomic.Value; 15 | 16 | const log2 = std.math.log2; 17 | const testing = std.testing; 18 | const assert = std.debug.assert; 19 | 20 | pub fn JdzGlobalAllocator(comptime config: JdzAllocConfig) type { 21 | const Arena = span_arena.Arena(config, true); 22 | 23 | const ArenaHandler = global_handler.GlobalArenaHandler(config); 24 | 25 | const GlobalSpanCache = bounded_mpmc_queue.BoundedMpmcQueue(*Span, config.cache_limit * config.global_cache_multiplier); 26 | 27 | const GlobalLargeCache = bounded_mpmc_queue.BoundedMpmcQueue(*Span, config.large_cache_limit * config.global_cache_multiplier); 28 | 29 | assert(config.span_alloc_count >= 1); 30 | 31 | // currently not supporting page sizes greater than 64KiB 32 | assert(page_size <= span_size); 33 | 34 | // -1 as one span gets allocated to span list and not cache 35 | assert(config.span_alloc_count - 1 <= config.cache_limit); 36 | 37 | assert(span_header_size >= @sizeOf(Span)); 38 | assert(config.large_span_overhead_mul >= 0.0); 39 | 40 | // These asserts must be true for alignment purposes 41 | assert(utils.isPowerOfTwo(span_header_size)); 42 | assert(utils.isPowerOfTwo(small_granularity)); 43 | assert(utils.isPowerOfTwo(small_max)); 44 | assert(utils.isPowerOfTwo(medium_granularity)); 45 | assert(medium_granularity <= small_max); 46 | assert(span_header_size % small_granularity == 0); 47 | 48 | // These asserts must be true for MPSC/MPMC queues to work 49 | assert(config.cache_limit > 1); 50 | assert(utils.isPowerOfTwo(config.cache_limit)); 51 | assert(config.large_cache_limit > 1); 52 | assert(utils.isPowerOfTwo(config.large_cache_limit)); 53 | 54 | assert(config.global_cache_multiplier >= 1); 55 | 56 | return struct { 57 | pub var cache: GlobalSpanCache = GlobalSpanCache.init(); 58 | pub var large_cache: [large_class_count]GlobalLargeCache = .{GlobalLargeCache.init()} ** large_class_count; 59 | 60 | var backing_allocator = config.backing_allocator; 61 | 62 | pub fn deinit() void { 63 | ArenaHandler.deinit(); 64 | } 65 | 66 | pub fn deinitThread() void { 67 | ArenaHandler.deinitThread(); 68 | } 69 | 70 | pub fn allocator() std.mem.Allocator { 71 | return .{ 72 | .ptr = undefined, 73 | .vtable = &.{ 74 | .alloc = alloc, 75 | .resize = resize, 76 | .free = free, 77 | }, 78 | }; 79 | } 80 | 81 | fn alloc(ctx: *anyopaque, len: usize, log2_align: u8, ret_addr: usize) ?[*]u8 { 82 | _ = ctx; 83 | _ = ret_addr; 84 | 85 | if (log2_align <= small_granularity_shift) { 86 | return @call(.always_inline, allocate, .{len}); 87 | } 88 | 89 | const alignment = @as(usize, 1) << @intCast(log2_align); 90 | const size = @max(alignment, len); 91 | 92 | if (size <= span_header_size) { 93 | const aligned_block_size: usize = @call(.always_inline, utils.roundUpToPowerOfTwo, .{size}); 94 | 95 | assert(aligned_block_size >= size); 96 | 97 | return @call(.always_inline, allocate, .{aligned_block_size}); 98 | } 99 | 100 | assert(alignment < span_effective_size); 101 | 102 | if (@call(.always_inline, allocate, .{size + alignment})) |block_ptr| { 103 | const align_mask: usize = alignment - 1; 104 | var ptr = block_ptr; 105 | 106 | if (@intFromPtr(ptr) & align_mask != 0) { 107 | ptr = @ptrFromInt((@intFromPtr(ptr) & ~align_mask) + alignment); 108 | const span = @call(.always_inline, utils.getSpan, .{ptr}); 109 | 110 | span.aligned_blocks = true; 111 | } 112 | 113 | return ptr; 114 | } 115 | 116 | return null; 117 | } 118 | 119 | fn resize(ctx: *anyopaque, buf: []u8, log2_align: u8, new_len: usize, ret_addr: usize) bool { 120 | _ = ret_addr; 121 | _ = ctx; 122 | 123 | const alignment = @as(usize, 1) << @intCast(log2_align); 124 | const aligned = (@intFromPtr(buf.ptr) & (alignment - 1)) == 0; 125 | 126 | const span = @call(.always_inline, utils.getSpan, .{buf.ptr}); 127 | 128 | if (buf.len <= span_max) return new_len <= span.class.block_size and aligned; 129 | if (buf.len <= large_max) return new_len <= span.alloc_size - (span.alloc_ptr - span.initial_ptr) and aligned; 130 | 131 | // round up to greater than or equal page size 132 | const max_len = (buf.len - 1 / page_size) * page_size + page_size; 133 | 134 | return aligned and new_len <= max_len; 135 | } 136 | 137 | fn free(ctx: *anyopaque, buf: []u8, log2_align: u8, ret_addr: usize) void { 138 | _ = ctx; 139 | _ = ret_addr; 140 | _ = log2_align; 141 | 142 | const span = @call(.always_inline, utils.getSpan, .{buf.ptr}); 143 | 144 | const arena: *Arena = @ptrCast(@alignCast(span.arena)); 145 | 146 | if (span.class.block_size <= medium_max) { 147 | @call(.always_inline, Arena.freeSmallOrMedium, .{ arena, span, buf }); 148 | } else if (span.class.block_size <= large_max) { 149 | @call(.always_inline, Arena.cacheLargeSpanOrFree, .{ arena, span }); 150 | } else { 151 | @call(.always_inline, Arena.freeHuge, .{ arena, span }); 152 | } 153 | } 154 | 155 | fn allocate(size: usize) ?[*]u8 { 156 | const arena = @call(.always_inline, ArenaHandler.getArena, .{}) orelse return null; 157 | 158 | if (size <= small_max) { 159 | const size_class = @call(.always_inline, utils.getSmallSizeClass, .{size}); 160 | return @call(.always_inline, Arena.allocateToSpan, .{ arena, size_class }); 161 | } else if (size <= medium_max) { 162 | const size_class = @call(.always_inline, utils.getMediumSizeClass, .{size}); 163 | return @call(.always_inline, Arena.allocateToSpan, .{ arena, size_class }); 164 | } else if (size <= span_max) { 165 | return @call(.always_inline, Arena.allocateOneSpan, .{ arena, span_class }); 166 | } else if (size <= large_max) { 167 | const span_count = @call(.always_inline, utils.getSpanCount, .{size}); 168 | return @call(.always_inline, Arena.allocateToLargeSpan, .{ arena, span_count }); 169 | } else { 170 | const span_count = @call(.always_inline, utils.getHugeSpanCount, .{size}) orelse return null; 171 | return @call(.always_inline, Arena.allocateHuge, .{ arena, span_count }); 172 | } 173 | } 174 | 175 | pub fn usableSize(ptr: *anyopaque) usize { 176 | const span = utils.getSpan(ptr); 177 | 178 | if (span.span_count == 1) { 179 | return span.class.block_size; 180 | } 181 | 182 | return span.alloc_size - (span.alloc_ptr - span.initial_ptr); 183 | } 184 | 185 | pub fn cacheSpan(span: *Span) bool { 186 | assert(span.span_count == 1); 187 | 188 | return cache.tryWrite(span); 189 | } 190 | 191 | pub fn cacheLargeSpan(span: *Span) bool { 192 | return large_cache[span.span_count - 1].tryWrite(span); 193 | } 194 | 195 | pub fn getCachedSpan() ?*Span { 196 | return cache.tryRead(); 197 | } 198 | 199 | pub fn getCachedLargeSpan(cache_idx: usize) ?*Span { 200 | return large_cache[cache_idx].tryRead(); 201 | } 202 | }; 203 | } 204 | 205 | const SizeClass = static_config.SizeClass; 206 | 207 | const span_size = static_config.span_size; 208 | const span_effective_size = static_config.span_effective_size; 209 | const span_header_size = static_config.span_header_size; 210 | const span_upper_mask = static_config.span_upper_mask; 211 | 212 | const small_granularity = static_config.small_granularity; 213 | const small_granularity_shift = static_config.small_granularity_shift; 214 | const small_max = static_config.small_max; 215 | 216 | const medium_granularity = static_config.medium_granularity; 217 | const medium_granularity_shift = static_config.medium_granularity_shift; 218 | const medium_max = static_config.medium_max; 219 | 220 | const span_max = static_config.span_max; 221 | const span_class = static_config.span_class; 222 | 223 | const large_max = static_config.large_max; 224 | const large_class_count = static_config.large_class_count; 225 | 226 | const page_size = static_config.page_size; 227 | const page_alignment = static_config.page_alignment; 228 | 229 | const small_size_classes = static_config.small_size_classes; 230 | const medium_size_classes = static_config.medium_size_classes; 231 | 232 | const aligned_size_classes = static_config.aligned_size_classes; 233 | const aligned_spans_offset = static_config.aligned_spans_offset; 234 | const span_align_max = static_config.span_align_max; 235 | 236 | // 237 | // Tests 238 | // 239 | test "small allocations - free in same order" { 240 | const jdz_allocator = JdzGlobalAllocator(.{}); 241 | 242 | const allocator = jdz_allocator.allocator(); 243 | 244 | var list = std.ArrayList(*u64).init(std.testing.allocator); 245 | defer list.deinit(); 246 | 247 | var i: usize = 0; 248 | while (i < 513) : (i += 1) { 249 | const ptr = try allocator.create(u64); 250 | try list.append(ptr); 251 | } 252 | 253 | for (list.items) |ptr| { 254 | allocator.destroy(ptr); 255 | } 256 | } 257 | 258 | test "small allocations - free in reverse order" { 259 | const jdz_allocator = JdzGlobalAllocator(.{}); 260 | 261 | const allocator = jdz_allocator.allocator(); 262 | 263 | var list = std.ArrayList(*u64).init(std.testing.allocator); 264 | defer list.deinit(); 265 | 266 | var i: usize = 0; 267 | while (i < 513) : (i += 1) { 268 | const ptr = try allocator.create(u64); 269 | try list.append(ptr); 270 | } 271 | 272 | while (list.popOrNull()) |ptr| { 273 | allocator.destroy(ptr); 274 | } 275 | } 276 | 277 | test "small allocations - alloc free alloc" { 278 | const jdz_allocator = JdzGlobalAllocator(.{}); 279 | 280 | const allocator = jdz_allocator.allocator(); 281 | 282 | const a = try allocator.create(u64); 283 | allocator.destroy(a); 284 | const b = try allocator.create(u64); 285 | allocator.destroy(b); 286 | } 287 | 288 | test "large allocations" { 289 | const jdz_allocator = JdzGlobalAllocator(.{}); 290 | 291 | const allocator = jdz_allocator.allocator(); 292 | 293 | const ptr1 = try allocator.alloc(u64, 42768); 294 | const ptr2 = try allocator.alloc(u64, 52768); 295 | allocator.free(ptr1); 296 | const ptr3 = try allocator.alloc(u64, 62768); 297 | allocator.free(ptr3); 298 | allocator.free(ptr2); 299 | } 300 | 301 | test "very large allocation" { 302 | const jdz_allocator = JdzGlobalAllocator(.{}); 303 | 304 | const allocator = jdz_allocator.allocator(); 305 | 306 | try std.testing.expectError(error.OutOfMemory, allocator.alloc(u8, std.math.maxInt(usize))); 307 | } 308 | 309 | test "realloc" { 310 | const jdz_allocator = JdzGlobalAllocator(.{}); 311 | 312 | const allocator = jdz_allocator.allocator(); 313 | 314 | var slice = try allocator.alignedAlloc(u8, @alignOf(u32), 1); 315 | defer allocator.free(slice); 316 | slice[0] = 0x12; 317 | 318 | // This reallocation should keep its pointer address. 319 | const old_slice = slice; 320 | slice = try allocator.realloc(slice, 2); 321 | try std.testing.expect(old_slice.ptr == slice.ptr); 322 | try std.testing.expect(slice[0] == 0x12); 323 | slice[1] = 0x34; 324 | 325 | // This requires upgrading to a larger bin size 326 | slice = try allocator.realloc(slice, 17); 327 | try std.testing.expect(old_slice.ptr != slice.ptr); 328 | try std.testing.expect(slice[0] == 0x12); 329 | try std.testing.expect(slice[1] == 0x34); 330 | } 331 | 332 | test "shrink" { 333 | const jdz_allocator = JdzGlobalAllocator(.{}); 334 | 335 | const allocator = jdz_allocator.allocator(); 336 | 337 | var slice = try allocator.alloc(u8, 20); 338 | defer allocator.free(slice); 339 | 340 | @memset(slice, 0x11); 341 | 342 | try std.testing.expect(allocator.resize(slice, 17)); 343 | slice = slice[0..17]; 344 | 345 | for (slice) |b| { 346 | try std.testing.expect(b == 0x11); 347 | } 348 | 349 | try std.testing.expect(allocator.resize(slice, 16)); 350 | 351 | for (slice) |b| { 352 | try std.testing.expect(b == 0x11); 353 | } 354 | } 355 | 356 | test "large object - grow" { 357 | const jdz_allocator = JdzGlobalAllocator(.{}); 358 | 359 | const allocator = jdz_allocator.allocator(); 360 | 361 | var slice1 = try allocator.alloc(u8, 8192 - 10); 362 | defer allocator.free(slice1); 363 | 364 | const old = slice1; 365 | slice1 = try allocator.realloc(slice1, 8192 - 10); 366 | try std.testing.expect(slice1.ptr == old.ptr); 367 | 368 | slice1 = try allocator.realloc(slice1, 8192); 369 | try std.testing.expect(slice1.ptr == old.ptr); 370 | 371 | slice1 = try allocator.realloc(slice1, 8192 + 1); 372 | } 373 | 374 | test "realloc small object to large object" { 375 | const jdz_allocator = JdzGlobalAllocator(.{}); 376 | 377 | const allocator = jdz_allocator.allocator(); 378 | 379 | var slice = try allocator.alloc(u8, 70); 380 | defer allocator.free(slice); 381 | slice[0] = 0x12; 382 | slice[60] = 0x34; 383 | 384 | // This requires upgrading to a large object 385 | const large_object_size = 8192 + 50; 386 | slice = try allocator.realloc(slice, large_object_size); 387 | try std.testing.expect(slice[0] == 0x12); 388 | try std.testing.expect(slice[60] == 0x34); 389 | } 390 | 391 | test "shrink large object to large object" { 392 | const jdz_allocator = JdzGlobalAllocator(.{}); 393 | 394 | const allocator = jdz_allocator.allocator(); 395 | 396 | var slice = try allocator.alloc(u8, 8192 + 50); 397 | defer allocator.free(slice); 398 | slice[0] = 0x12; 399 | slice[60] = 0x34; 400 | 401 | if (!allocator.resize(slice, 8192 + 1)) return; 402 | slice = slice.ptr[0 .. 8192 + 1]; 403 | try std.testing.expect(slice[0] == 0x12); 404 | try std.testing.expect(slice[60] == 0x34); 405 | 406 | try std.testing.expect(allocator.resize(slice, 8192 + 1)); 407 | slice = slice[0 .. 8192 + 1]; 408 | try std.testing.expect(slice[0] == 0x12); 409 | try std.testing.expect(slice[60] == 0x34); 410 | 411 | slice = try allocator.realloc(slice, 8192); 412 | try std.testing.expect(slice[0] == 0x12); 413 | try std.testing.expect(slice[60] == 0x34); 414 | } 415 | 416 | test "shrink large object to large object with larger alignment" { 417 | const jdz_allocator = JdzGlobalAllocator(.{}); 418 | 419 | const allocator = jdz_allocator.allocator(); 420 | 421 | var debug_buffer: [1000]u8 = undefined; 422 | var fba = std.heap.FixedBufferAllocator.init(&debug_buffer); 423 | const debug_allocator = fba.allocator(); 424 | 425 | const alloc_size = 8192 + 50; 426 | var slice = try allocator.alignedAlloc(u8, 16, alloc_size); 427 | defer allocator.free(slice); 428 | 429 | const big_alignment: usize = switch (builtin.os.tag) { 430 | .windows => 65536, // Windows aligns to 64K. 431 | else => 8192, 432 | }; 433 | // This loop allocates until we find a page that is not aligned to the big 434 | // alignment. Then we shrink the allocation after the loop, but increase the 435 | // alignment to the higher one, that we know will force it to realloc. 436 | var stuff_to_free = std.ArrayList([]align(16) u8).init(debug_allocator); 437 | while (std.mem.isAligned(@intFromPtr(slice.ptr), big_alignment)) { 438 | try stuff_to_free.append(slice); 439 | slice = try allocator.alignedAlloc(u8, 16, alloc_size); 440 | } 441 | while (stuff_to_free.popOrNull()) |item| { 442 | allocator.free(item); 443 | } 444 | slice[0] = 0x12; 445 | slice[60] = 0x34; 446 | 447 | slice = try allocator.reallocAdvanced(slice, big_alignment, alloc_size / 2); 448 | try std.testing.expect(slice[0] == 0x12); 449 | try std.testing.expect(slice[60] == 0x34); 450 | } 451 | 452 | test "realloc large object to small object" { 453 | const jdz_allocator = JdzGlobalAllocator(.{}); 454 | 455 | const allocator = jdz_allocator.allocator(); 456 | 457 | var slice = try allocator.alloc(u8, 8192 + 50); 458 | defer allocator.free(slice); 459 | slice[0] = 0x12; 460 | slice[16] = 0x34; 461 | 462 | slice = try allocator.realloc(slice, 19); 463 | try std.testing.expect(slice[0] == 0x12); 464 | try std.testing.expect(slice[16] == 0x34); 465 | } 466 | 467 | test "realloc large object to larger alignment" { 468 | const jdz_allocator = JdzGlobalAllocator(.{}); 469 | 470 | const allocator = jdz_allocator.allocator(); 471 | 472 | var debug_buffer: [1000]u8 = undefined; 473 | var fba = std.heap.FixedBufferAllocator.init(&debug_buffer); 474 | const debug_allocator = fba.allocator(); 475 | 476 | var slice = try allocator.alignedAlloc(u8, 16, 8192 + 50); 477 | defer allocator.free(slice); 478 | 479 | const big_alignment: usize = switch (builtin.os.tag) { 480 | .windows => 65536, // Windows aligns to 64K. 481 | else => 8192, 482 | }; 483 | // This loop allocates until we find a page that is not aligned to the big alignment. 484 | var stuff_to_free = std.ArrayList([]align(16) u8).init(debug_allocator); 485 | while (std.mem.isAligned(@intFromPtr(slice.ptr), big_alignment)) { 486 | try stuff_to_free.append(slice); 487 | slice = try allocator.alignedAlloc(u8, 16, 8192 + 50); 488 | } 489 | while (stuff_to_free.popOrNull()) |item| { 490 | allocator.free(item); 491 | } 492 | slice[0] = 0x12; 493 | slice[16] = 0x34; 494 | 495 | slice = try allocator.reallocAdvanced(slice, 32, 8192 + 100); 496 | try std.testing.expect(slice[0] == 0x12); 497 | try std.testing.expect(slice[16] == 0x34); 498 | 499 | slice = try allocator.reallocAdvanced(slice, 32, 8192 + 25); 500 | try std.testing.expect(slice[0] == 0x12); 501 | try std.testing.expect(slice[16] == 0x34); 502 | 503 | slice = try allocator.reallocAdvanced(slice, big_alignment, 8192 + 100); 504 | try std.testing.expect(slice[0] == 0x12); 505 | try std.testing.expect(slice[16] == 0x34); 506 | } 507 | 508 | test "large object shrinks to small" { 509 | const jdz_allocator = JdzGlobalAllocator(.{}); 510 | 511 | const allocator = jdz_allocator.allocator(); 512 | 513 | const slice = try allocator.alloc(u8, 8192 + 50); 514 | defer allocator.free(slice); 515 | 516 | try std.testing.expect(allocator.resize(slice, 4)); 517 | } 518 | 519 | test "objects of size 1024 and 2048" { 520 | const jdz_allocator = JdzGlobalAllocator(.{}); 521 | 522 | const allocator = jdz_allocator.allocator(); 523 | 524 | const slice = try allocator.alloc(u8, 1025); 525 | const slice2 = try allocator.alloc(u8, 3000); 526 | 527 | allocator.free(slice); 528 | allocator.free(slice2); 529 | } 530 | 531 | test "max large alloc" { 532 | const jdz_allocator = JdzGlobalAllocator(.{}); 533 | 534 | const allocator = jdz_allocator.allocator(); 535 | 536 | const buf = try allocator.alloc(u8, large_max); 537 | defer allocator.free(buf); 538 | } 539 | 540 | test "huge alloc" { 541 | const jdz_allocator = JdzGlobalAllocator(.{}); 542 | 543 | const allocator = jdz_allocator.allocator(); 544 | 545 | const buf = try allocator.alloc(u8, large_max + 1); 546 | defer allocator.free(buf); 547 | } 548 | 549 | test "huge alloc does not try to access span" { 550 | const jdz_allocator = JdzGlobalAllocator(.{}); 551 | 552 | const allocator = jdz_allocator.allocator(); 553 | 554 | const buf1 = try allocator.alloc(u8, large_max + 1); 555 | const buf2 = try allocator.alloc(u8, large_max + 1); 556 | 557 | allocator.free(buf1); 558 | allocator.free(buf2); 559 | } 560 | 561 | test "small alignment small alloc" { 562 | const jdz_allocator = JdzGlobalAllocator(.{}); 563 | 564 | const allocator = jdz_allocator.allocator(); 565 | 566 | const slice = allocator.rawAlloc(1, 4, @returnAddress()).?; 567 | defer allocator.rawFree(slice[0..1], 4, @returnAddress()); 568 | 569 | try std.testing.expect(@intFromPtr(slice) % std.math.pow(u16, 2, 4) == 0); 570 | } 571 | 572 | test "medium alignment small alloc" { 573 | const jdz_allocator = JdzGlobalAllocator(.{}); 574 | 575 | const allocator = jdz_allocator.allocator(); 576 | 577 | const alignment: u8 = @truncate(log2(utils.roundUpToPowerOfTwo(small_max + 1))); 578 | 579 | if (alignment > page_alignment) return error.SkipZigTest; 580 | 581 | const slice = allocator.rawAlloc(1, alignment, @returnAddress()).?; 582 | defer allocator.rawFree(slice[0..1], alignment, @returnAddress()); 583 | 584 | try std.testing.expect(@intFromPtr(slice) % std.math.pow(u16, 2, 4) == 0); 585 | } 586 | 587 | test "page size alignment small alloc" { 588 | const jdz_allocator = JdzGlobalAllocator(.{}); 589 | 590 | const allocator = jdz_allocator.allocator(); 591 | 592 | const slice = allocator.rawAlloc(1, page_alignment, @returnAddress()).?; 593 | defer allocator.rawFree(slice[0..1], page_alignment, @returnAddress()); 594 | 595 | try std.testing.expect(@intFromPtr(slice) % std.math.pow(u16, 2, page_alignment) == 0); 596 | } 597 | 598 | test "small alignment large alloc" { 599 | const jdz_allocator = JdzGlobalAllocator(.{}); 600 | 601 | const allocator = jdz_allocator.allocator(); 602 | 603 | const slice = allocator.rawAlloc(span_max, 4, @returnAddress()).?; 604 | defer allocator.rawFree(slice[0..span_max], 4, @returnAddress()); 605 | 606 | try std.testing.expect(@intFromPtr(slice) % std.math.pow(u16, 2, 4) == 0); 607 | } 608 | 609 | test "medium alignment large alloc" { 610 | const jdz_allocator = JdzGlobalAllocator(.{}); 611 | 612 | const allocator = jdz_allocator.allocator(); 613 | 614 | const alignment: u8 = @truncate(log2(utils.roundUpToPowerOfTwo(small_max + 1))); 615 | 616 | if (alignment > page_alignment) return error.SkipZigTest; 617 | 618 | const slice = allocator.rawAlloc(span_max, alignment, @returnAddress()).?; 619 | defer allocator.rawFree(slice[0..span_max], alignment, @returnAddress()); 620 | 621 | try std.testing.expect(@intFromPtr(slice) % std.math.pow(u16, 2, 4) == 0); 622 | } 623 | 624 | test "page size alignment large alloc" { 625 | const jdz_allocator = JdzGlobalAllocator(.{}); 626 | 627 | const allocator = jdz_allocator.allocator(); 628 | 629 | const slice = allocator.rawAlloc(span_max, page_alignment, @returnAddress()).?; 630 | defer allocator.rawFree(slice[0..span_max], page_alignment, @returnAddress()); 631 | 632 | try std.testing.expect(@intFromPtr(slice) % std.math.pow(u16, 2, page_alignment) == 0); 633 | } 634 | 635 | test "empty span gets cached to global and reused" { 636 | const jdz_allocator = JdzGlobalAllocator(.{ .span_alloc_count = 1 }); 637 | defer jdz_allocator.deinit(); 638 | 639 | const allocator = jdz_allocator.allocator(); 640 | 641 | const alloc_one = try allocator.alloc(u8, 1); 642 | 643 | const span_one = utils.getSpan(alloc_one.ptr); 644 | 645 | allocator.free(alloc_one); 646 | 647 | jdz_allocator.deinitThread(); 648 | 649 | const span_two = jdz_allocator.cache.buffer[0].data; 650 | 651 | try testing.expect(span_one == span_two); 652 | 653 | const alloc_two = try allocator.alloc(u8, 1); 654 | 655 | try testing.expect(alloc_one.ptr == alloc_two.ptr); 656 | 657 | allocator.free(alloc_two); 658 | } 659 | 660 | test "empty large span gets cached to global and reused" { 661 | const jdz_allocator = JdzGlobalAllocator(.{ .map_alloc_count = 1 }); 662 | defer jdz_allocator.deinit(); 663 | 664 | const allocator = jdz_allocator.allocator(); 665 | 666 | const alloc_one = try allocator.alloc(u8, span_size * 2); 667 | 668 | const span_one = utils.getSpan(alloc_one.ptr); 669 | 670 | const cache_idx = span_one.span_count - 1; 671 | 672 | allocator.free(alloc_one); 673 | 674 | jdz_allocator.deinitThread(); 675 | 676 | const span_two = jdz_allocator.large_cache[cache_idx].buffer[0].data; 677 | 678 | try testing.expect(span_one == span_two); 679 | 680 | const alloc_two = try allocator.alloc(u8, span_size * 2); 681 | 682 | try testing.expect(alloc_one.ptr == alloc_two.ptr); 683 | 684 | allocator.free(alloc_two); 685 | } 686 | 687 | test "consecutive overalignment" { 688 | const jdz_allocator = JdzGlobalAllocator(.{}); 689 | defer jdz_allocator.deinit(); 690 | 691 | const allocator = jdz_allocator.allocator(); 692 | 693 | const overaligned_sizes = [_]usize{ 192, 192, 192 }; 694 | var buffers: [overaligned_sizes.len][]const u8 = undefined; 695 | 696 | for (overaligned_sizes, &buffers) |overaligned_size, *buffers_slot| { 697 | buffers_slot.* = try allocator.alignedAlloc(u8, 64, overaligned_size); 698 | } 699 | 700 | for (buffers) |buffer| { 701 | allocator.free(buffer); 702 | } 703 | } 704 | -------------------------------------------------------------------------------- /src/global_arena_handler.zig: -------------------------------------------------------------------------------- 1 | const std = @import("std"); 2 | const span_arena = @import("arena.zig"); 3 | const jdz_allocator = @import("jdz_allocator.zig"); 4 | const utils = @import("utils.zig"); 5 | 6 | const JdzAllocConfig = jdz_allocator.JdzAllocConfig; 7 | 8 | pub fn GlobalArenaHandler(comptime config: JdzAllocConfig) type { 9 | const Mutex = utils.getMutexType(config); 10 | 11 | return struct { 12 | const Arena = span_arena.Arena(config, true); 13 | 14 | var preinit_arena: Arena = Arena.init(.unlocked, 0); 15 | var arena_list: ?*Arena = &preinit_arena; 16 | var mutex: Mutex = .{}; 17 | 18 | threadlocal var thread_arena: ?*Arena = null; 19 | 20 | pub fn deinit() void { 21 | mutex.lock(); 22 | defer mutex.unlock(); 23 | 24 | var opt_arena = arena_list; 25 | 26 | while (opt_arena) |arena| { 27 | const next = arena.next; 28 | 29 | if (arena != &preinit_arena) { 30 | config.backing_allocator.destroy(arena); 31 | } 32 | 33 | opt_arena = next; 34 | } 35 | } 36 | 37 | pub fn deinitThread() void { 38 | const arena = thread_arena orelse return; 39 | 40 | _ = arena.deinit(); 41 | 42 | thread_arena = null; 43 | 44 | addArenaToList(arena); 45 | } 46 | 47 | pub inline fn getArena() ?*Arena { 48 | return thread_arena orelse 49 | getArenaFromList() orelse 50 | createArena(); 51 | } 52 | 53 | pub inline fn getThreadArena() ?*Arena { 54 | return thread_arena; 55 | } 56 | 57 | fn getArenaFromList() ?*Arena { 58 | mutex.lock(); 59 | defer mutex.unlock(); 60 | 61 | if (arena_list) |arena| { 62 | arena_list = arena.next; 63 | arena.next = null; 64 | arena.thread_id = std.Thread.getCurrentId(); 65 | 66 | thread_arena = arena; 67 | 68 | return arena; 69 | } 70 | 71 | return null; 72 | } 73 | 74 | fn createArena() ?*Arena { 75 | const new_arena = config.backing_allocator.create(Arena) catch return null; 76 | 77 | new_arena.* = Arena.init(.unlocked, std.Thread.getCurrentId()); 78 | 79 | thread_arena = new_arena; 80 | 81 | return new_arena; 82 | } 83 | 84 | fn addArenaToList(new_arena: *Arena) void { 85 | mutex.lock(); 86 | defer mutex.unlock(); 87 | 88 | if (arena_list == null) { 89 | arena_list = new_arena; 90 | 91 | return; 92 | } 93 | 94 | var arena = arena_list.?; 95 | 96 | while (arena.next) |next| { 97 | arena = next; 98 | } 99 | 100 | arena.next = new_arena; 101 | } 102 | }; 103 | } 104 | -------------------------------------------------------------------------------- /src/grow_shrink_bench.zig: -------------------------------------------------------------------------------- 1 | const std = @import("std"); 2 | const jdz = @import("jdz_allocator.zig"); 3 | const static_config = @import("static_config.zig"); 4 | 5 | const mixed_rounds = 100_000; 6 | const mixed_min = 1; 7 | const mixed_max = 80000; 8 | 9 | const small_rounds = 100_000; 10 | const small_min = 1; 11 | const small_max = static_config.small_max; 12 | 13 | const medium_rounds = 100_000; 14 | const medium_min = static_config.small_max + 1; 15 | const medium_max = static_config.medium_max; 16 | 17 | const big_rounds = 100_000; 18 | const big_min = static_config.medium_max + 1; 19 | const big_max = static_config.large_max; 20 | const buffer_capacity = @max(mixed_rounds, @max(small_rounds, @max(medium_rounds, big_rounds))); 21 | 22 | var slots = std.BoundedArray([]u8, buffer_capacity){}; 23 | 24 | pub fn main() !void { 25 | const args = try std.process.argsAlloc(std.heap.page_allocator); 26 | defer std.process.argsFree(std.heap.page_allocator, args); 27 | 28 | const num_args = args.len - 1; 29 | 30 | if (num_args == 0) return try bench(1); 31 | 32 | for (0..num_args) |i| { 33 | const num_threads = try std.fmt.parseInt(u32, args[i + 1], 10); 34 | 35 | try bench(num_threads); 36 | } 37 | } 38 | 39 | fn bench(num_threads: u32) !void { 40 | try std.io.getStdOut().writer().print("=== Num Threads={} ===\n", .{num_threads}); 41 | 42 | try std.io.getStdOut().writer().print("==Mixed Alloc==\n", .{}); 43 | try jdz_mixed(num_threads); 44 | try jdz_global_mixed(num_threads); 45 | try c_mixed(num_threads); 46 | try gpa_mixed(num_threads); 47 | 48 | try std.io.getStdOut().writer().print("==Small Alloc==\n", .{}); 49 | try jdz_small(num_threads); 50 | try jdz_global_small(num_threads); 51 | try c_small(num_threads); 52 | try gpa_small(num_threads); 53 | 54 | try std.io.getStdOut().writer().print("==Medium Alloc==\n", .{}); 55 | try jdz_medium(num_threads); 56 | try jdz_global_medium(num_threads); 57 | try c_medium(num_threads); 58 | try gpa_medium(num_threads); 59 | 60 | try std.io.getStdOut().writer().print("==Big Alloc==\n", .{}); 61 | try jdz_big(num_threads); 62 | try jdz_global_big(num_threads); 63 | try c_big(num_threads); 64 | try gpa_big(num_threads); 65 | 66 | try std.io.getStdOut().writer().print("\n", .{}); 67 | } 68 | 69 | /// 70 | /// Mixed 71 | /// 72 | fn jdz_mixed(num_threads: u32) !void { 73 | var jdz_allocator = jdz.JdzAllocator(.{}).init(); 74 | defer jdz_allocator.deinit(); 75 | 76 | const allocator = jdz_allocator.allocator(); 77 | 78 | try runPerfTestAlloc("jdz/mixed", mixed_min, mixed_max, allocator, mixed_rounds, num_threads); 79 | } 80 | 81 | fn jdz_global_mixed(num_threads: u32) !void { 82 | const jdz_allocator = jdz.JdzGlobalAllocator(.{}); 83 | defer jdz.JdzGlobalAllocator(.{}).deinit(); 84 | 85 | const allocator = jdz_allocator.allocator(); 86 | 87 | try runPerfTestAlloc("jdz-global/mixed", mixed_min, mixed_max, allocator, mixed_rounds, num_threads); 88 | } 89 | 90 | fn gpa_mixed(num_threads: u32) !void { 91 | var gpa = std.heap.GeneralPurposeAllocator(.{}){}; 92 | 93 | const allocator = gpa.allocator(); 94 | defer _ = gpa.deinit(); 95 | 96 | try runPerfTestAlloc("gpa/mixed", mixed_min, mixed_max, allocator, mixed_rounds, num_threads); 97 | } 98 | 99 | fn c_mixed(num_threads: u32) !void { 100 | const allocator = std.heap.c_allocator; 101 | 102 | try runPerfTestAlloc("c/mixed", mixed_min, mixed_max, allocator, mixed_rounds, num_threads); 103 | } 104 | 105 | /// 106 | /// Small 107 | /// 108 | fn jdz_small(num_threads: u32) !void { 109 | var jdz_allocator = jdz.JdzAllocator(.{}).init(); 110 | defer jdz_allocator.deinit(); 111 | 112 | const allocator = jdz_allocator.allocator(); 113 | 114 | try runPerfTestAlloc("jdz/small", small_min, small_max, allocator, small_rounds, num_threads); 115 | } 116 | 117 | fn c_small(num_threads: u32) !void { 118 | const allocator = std.heap.c_allocator; 119 | 120 | try runPerfTestAlloc("c/small", small_min, small_max, allocator, small_rounds, num_threads); 121 | } 122 | 123 | fn jdz_global_small(num_threads: u32) !void { 124 | const jdz_allocator = jdz.JdzGlobalAllocator(.{}); 125 | defer jdz.JdzGlobalAllocator(.{}).deinit(); 126 | 127 | const allocator = jdz_allocator.allocator(); 128 | 129 | try runPerfTestAlloc("jdz-global/small", small_min, small_max, allocator, small_rounds, num_threads); 130 | } 131 | 132 | fn gpa_small(num_threads: u32) !void { 133 | var gpa = std.heap.GeneralPurposeAllocator(.{}){}; 134 | 135 | const allocator = gpa.allocator(); 136 | defer _ = gpa.deinit(); 137 | 138 | try runPerfTestAlloc("gpa/small", small_min, small_max, allocator, small_rounds, num_threads); 139 | } 140 | 141 | /// 142 | /// Medium 143 | /// 144 | fn jdz_medium(num_threads: u32) !void { 145 | var jdz_allocator = jdz.JdzAllocator(.{}).init(); 146 | defer jdz_allocator.deinit(); 147 | const allocator = jdz_allocator.allocator(); 148 | 149 | try runPerfTestAlloc("jdz/medium", medium_min, medium_max, allocator, medium_rounds, num_threads); 150 | } 151 | 152 | fn jdz_global_medium(num_threads: u32) !void { 153 | const jdz_allocator = jdz.JdzGlobalAllocator(.{}); 154 | defer jdz.JdzGlobalAllocator(.{}).deinit(); 155 | 156 | const allocator = jdz_allocator.allocator(); 157 | 158 | try runPerfTestAlloc("jdz-global/medium", medium_min, medium_max, allocator, medium_rounds, num_threads); 159 | } 160 | 161 | fn c_medium(num_threads: u32) !void { 162 | const allocator = std.heap.c_allocator; 163 | 164 | try runPerfTestAlloc("c/medium", medium_min, medium_max, allocator, medium_rounds, num_threads); 165 | } 166 | 167 | fn gpa_medium(num_threads: u32) !void { 168 | var gpa = std.heap.GeneralPurposeAllocator(.{}){}; 169 | 170 | const allocator = gpa.allocator(); 171 | defer _ = gpa.deinit(); 172 | 173 | try runPerfTestAlloc("gpa/medium", medium_min, medium_max, allocator, medium_rounds, num_threads); 174 | } 175 | 176 | /// 177 | /// Big 178 | /// 179 | fn jdz_big(num_threads: u32) !void { 180 | var jdz_allocator = jdz.JdzAllocator(.{}).init(); 181 | defer jdz_allocator.deinit(); 182 | 183 | const allocator = jdz_allocator.allocator(); 184 | 185 | try runPerfTestAlloc("jdz/big", big_min, big_max, allocator, big_rounds, num_threads); 186 | } 187 | 188 | fn jdz_global_big(num_threads: u32) !void { 189 | const jdz_allocator = jdz.JdzGlobalAllocator(.{}); 190 | defer jdz.JdzGlobalAllocator(.{}).deinit(); 191 | 192 | const allocator = jdz_allocator.allocator(); 193 | 194 | try runPerfTestAlloc("jdz-global/big", big_min, big_max, allocator, big_rounds, num_threads); 195 | } 196 | 197 | fn c_big(num_threads: u32) !void { 198 | const allocator = std.heap.c_allocator; 199 | 200 | try runPerfTestAlloc("c/big", big_min, big_max, allocator, big_rounds, num_threads); 201 | } 202 | 203 | fn gpa_big(num_threads: u32) !void { 204 | var gpa = std.heap.GeneralPurposeAllocator(.{}){}; 205 | 206 | const allocator = gpa.allocator(); 207 | defer _ = gpa.deinit(); 208 | 209 | try runPerfTestAlloc("gpa/big", big_min, big_max, allocator, big_rounds, num_threads); 210 | } 211 | 212 | /// 213 | /// Bench Methods 214 | /// 215 | fn threadAllocWorker(min: usize, max: usize, allocator: std.mem.Allocator, max_rounds: usize) !void { 216 | var rounds: usize = max_rounds; 217 | 218 | var random_source = std.rand.DefaultPrng.init(1337); 219 | const rng = random_source.random(); 220 | 221 | while (rounds > 0) { 222 | rounds -= 1; 223 | 224 | const alloc_amount = rng.intRangeAtMost(usize, min, max); 225 | 226 | if (slots.len < max_rounds) { 227 | const item = try allocator.alloc(u8, alloc_amount); 228 | slots.appendAssumeCapacity(item); 229 | } 230 | } 231 | 232 | for (slots.slice()) |ptr| { 233 | allocator.free(ptr); 234 | } 235 | 236 | try slots.resize(0); 237 | } 238 | 239 | fn runPerfTestAlloc(tag: []const u8, min: usize, max: usize, allocator: std.mem.Allocator, max_rounds: usize, num_threads: u32) !void { 240 | var workers: []std.Thread = try std.heap.page_allocator.alloc(std.Thread, num_threads); 241 | defer std.heap.page_allocator.free(workers); 242 | 243 | const begin_time = std.time.nanoTimestamp(); 244 | 245 | for (0..num_threads) |i| { 246 | workers[i] = try std.Thread.spawn(.{}, threadAllocWorker, .{ min, max, allocator, max_rounds }); 247 | } 248 | 249 | for (0..num_threads) |i| { 250 | workers[i].join(); 251 | } 252 | 253 | const end_time = std.time.nanoTimestamp(); 254 | 255 | try std.io.getStdOut().writer().print("time={d: >10.2}us test={s} num_threads={}\n", .{ 256 | @as(f32, @floatFromInt(end_time - begin_time)) / 1000.0, tag, num_threads, 257 | }); 258 | } 259 | -------------------------------------------------------------------------------- /src/jdz_allocator.zig: -------------------------------------------------------------------------------- 1 | const std = @import("std"); 2 | const builtin = @import("builtin"); 3 | 4 | const shared_allocator = @import("shared_allocator.zig"); 5 | const global_allocator = @import("global_allocator.zig"); 6 | 7 | pub const JdzAllocator = shared_allocator.JdzAllocator; 8 | pub const JdzGlobalAllocator = global_allocator.JdzGlobalAllocator; 9 | 10 | pub const JdzAllocConfig = struct { 11 | /// backing allocator 12 | backing_allocator: std.mem.Allocator = std.heap.page_allocator, 13 | 14 | /// controls batch span allocation amount for one span allocations 15 | /// will return 1 span to allocating function and all remaining spans will be written to the one span cache 16 | span_alloc_count: u32 = 64, 17 | 18 | /// controls batch memory mapping amount in spans 19 | /// overhead from desired span count will be saved to map_cache for reuse on future map requests 20 | /// as memory mapping will likely not be aligned, we will use 1 span worth of padding per map call 21 | /// padding may be used as a span if alignment allows it 22 | /// minimum is 1 (not recommended) - default is 64, resulting in 4MiB memory mapping + 64KiB padding 23 | map_alloc_count: u32 = 64, 24 | 25 | /// maximum number of spans in arena cache 26 | cache_limit: u32 = 64, 27 | 28 | /// maximum number spans in arena large caches 29 | large_cache_limit: u32 = 64, 30 | 31 | /// maximum number of spans in arena map cache 32 | map_cache_limit: u32 = 16, 33 | 34 | /// global cache multiplier 35 | global_cache_multiplier: u32 = 8, 36 | 37 | /// percentage overhead applied to span count when looking for a large span in cache 38 | /// increases cache hits and memory usage, but does hurt performance 39 | large_span_overhead_mul: f64 = 0.5, 40 | 41 | /// if cached large spans should be split to accomodate small or medium allocations 42 | /// improves memory usage but hurts performance 43 | split_large_spans_to_one: bool = true, 44 | 45 | /// if cached large spans should be split to accomodate smaller large allocations 46 | /// improves memory usage but hurts performance 47 | split_large_spans_to_large: bool = true, 48 | 49 | /// JdzSharedAllocator batch arena instantiation amount 50 | /// prevents allocator-induced false sharing if greater than total number of allocating threads 51 | shared_arena_batch_size: u32 = 8, 52 | 53 | /// if leaks should be reported - only works with JdzSharedAllocator 54 | report_leaks: bool = builtin.mode == .Debug, 55 | 56 | /// Whether to synchronize usage of this allocator. 57 | /// For actual thread safety, the backing allocator must also be thread safe. 58 | thread_safe: bool = !builtin.single_threaded, 59 | }; 60 | 61 | test { 62 | std.testing.refAllDecls(@This()); 63 | } 64 | -------------------------------------------------------------------------------- /src/lock.zig: -------------------------------------------------------------------------------- 1 | const std = @import("std"); 2 | 3 | pub const Lock = enum { 4 | unlocked, 5 | locked, 6 | 7 | pub fn tryAcquire(lock: *Lock) bool { 8 | return @cmpxchgStrong(Lock, lock, .unlocked, .locked, .acquire, .monotonic) == null; 9 | } 10 | 11 | pub fn acquire(lock: *Lock) void { 12 | while (@cmpxchgWeak(Lock, lock, .unlocked, .locked, .acquire, .monotonic) != null) { 13 | std.atomic.spinLoopHint(); 14 | } 15 | } 16 | 17 | pub fn release(lock: *Lock) void { 18 | @atomicStore(Lock, lock, .unlocked, .release); 19 | } 20 | }; 21 | 22 | pub const DummyLock = enum { 23 | unlocked, 24 | locked, 25 | 26 | pub fn tryAcquire(lock: *DummyLock) bool { 27 | _ = lock; 28 | return true; 29 | } 30 | 31 | pub fn acquire(lock: *DummyLock) void { 32 | _ = lock; 33 | } 34 | 35 | pub fn release(lock: *DummyLock) void { 36 | _ = lock; 37 | } 38 | }; 39 | -------------------------------------------------------------------------------- /src/shared_allocator.zig: -------------------------------------------------------------------------------- 1 | const std = @import("std"); 2 | const builtin = @import("builtin"); 3 | 4 | const jdz = @import("jdz_allocator.zig"); 5 | const shared_arena_handler = @import("shared_arena_handler.zig"); 6 | const span_arena = @import("arena.zig"); 7 | const static_config = @import("static_config.zig"); 8 | const utils = @import("utils.zig"); 9 | const span_file = @import("span.zig"); 10 | 11 | const Span = span_file.Span; 12 | const JdzAllocConfig = jdz.JdzAllocConfig; 13 | const Value = std.atomic.Value; 14 | 15 | const log2 = std.math.log2; 16 | const testing = std.testing; 17 | const assert = std.debug.assert; 18 | 19 | pub fn JdzAllocator(comptime config: JdzAllocConfig) type { 20 | const Arena = span_arena.Arena(config, false); 21 | 22 | const SharedArenaHandler = shared_arena_handler.SharedArenaHandler(config); 23 | 24 | assert(config.span_alloc_count >= 1); 25 | 26 | // currently not supporting page sizes greater than 64KiB 27 | assert(page_size <= span_size); 28 | 29 | // -1 as one span gets allocated to span list and not cache 30 | assert(config.span_alloc_count - 1 <= config.cache_limit); 31 | 32 | assert(span_header_size >= @sizeOf(Span)); 33 | assert(config.large_span_overhead_mul >= 0.0); 34 | 35 | // These asserts must be true for alignment purposes 36 | assert(utils.isPowerOfTwo(span_header_size)); 37 | assert(utils.isPowerOfTwo(small_granularity)); 38 | assert(utils.isPowerOfTwo(small_max)); 39 | assert(utils.isPowerOfTwo(medium_granularity)); 40 | assert(medium_granularity <= small_max); 41 | assert(span_header_size % small_granularity == 0); 42 | 43 | // These asserts must be true for MPSC queue to work 44 | assert(config.large_cache_limit > 1); 45 | assert(utils.isPowerOfTwo(config.large_cache_limit)); 46 | 47 | return struct { 48 | backing_allocator: std.mem.Allocator, 49 | arena_handler: SharedArenaHandler, 50 | 51 | const Self = @This(); 52 | 53 | pub fn init() Self { 54 | return .{ 55 | .backing_allocator = config.backing_allocator, 56 | .arena_handler = SharedArenaHandler.init(), 57 | }; 58 | } 59 | 60 | pub fn deinit(self: *Self) void { 61 | const spans_leaked = self.arena_handler.deinit(); 62 | 63 | if (config.report_leaks) { 64 | const log = std.log.scoped(.jdz_allocator); 65 | 66 | if (spans_leaked != 0) { 67 | log.warn("{} leaked 64KiB spans", .{spans_leaked}); 68 | } 69 | } 70 | } 71 | 72 | pub fn allocator(self: *Self) std.mem.Allocator { 73 | return .{ 74 | .ptr = self, 75 | .vtable = &.{ 76 | .alloc = alloc, 77 | .resize = resize, 78 | .free = free, 79 | }, 80 | }; 81 | } 82 | 83 | fn alloc(ctx: *anyopaque, len: usize, log2_align: u8, ret_addr: usize) ?[*]u8 { 84 | _ = ret_addr; 85 | 86 | const self: *Self = @ptrCast(@alignCast(ctx)); 87 | 88 | if (log2_align <= small_granularity_shift) { 89 | return @call(.always_inline, allocate, .{ self, len }); 90 | } 91 | 92 | const alignment = @as(usize, 1) << @intCast(log2_align); 93 | const size = @max(alignment, len); 94 | 95 | if (size <= span_header_size) { 96 | const aligned_block_size: usize = @call(.always_inline, utils.roundUpToPowerOfTwo, .{size}); 97 | 98 | assert(aligned_block_size >= size); 99 | 100 | return @call(.always_inline, allocate, .{ self, aligned_block_size }); 101 | } 102 | 103 | assert(alignment < span_effective_size); 104 | 105 | if (@call(.always_inline, allocate, .{ self, size + alignment })) |block_ptr| { 106 | const align_mask: usize = alignment - 1; 107 | var ptr = block_ptr; 108 | 109 | if (@intFromPtr(ptr) & align_mask != 0) { 110 | ptr = @ptrFromInt((@intFromPtr(ptr) & ~align_mask) + alignment); 111 | const span = @call(.always_inline, utils.getSpan, .{ptr}); 112 | 113 | span.aligned_blocks = true; 114 | } 115 | 116 | return ptr; 117 | } 118 | 119 | return null; 120 | } 121 | 122 | fn resize(ctx: *anyopaque, buf: []u8, log2_align: u8, new_len: usize, ret_addr: usize) bool { 123 | _ = ret_addr; 124 | _ = ctx; 125 | const alignment = @as(usize, 1) << @intCast(log2_align); 126 | const aligned = (@intFromPtr(buf.ptr) & (alignment - 1)) == 0; 127 | 128 | const span = @call(.always_inline, utils.getSpan, .{buf.ptr}); 129 | 130 | if (buf.len <= span_max) return new_len <= span.class.block_size and aligned; 131 | if (buf.len <= large_max) return new_len <= span.alloc_size - (span.alloc_ptr - span.initial_ptr) and aligned; 132 | 133 | // round up to greater than or equal page size 134 | const max_len = (buf.len - 1 / page_size) * page_size + page_size; 135 | 136 | return aligned and new_len <= max_len; 137 | } 138 | 139 | fn free(ctx: *anyopaque, buf: []u8, log2_align: u8, ret_addr: usize) void { 140 | _ = ctx; 141 | _ = ret_addr; 142 | _ = log2_align; 143 | 144 | const span = @call(.always_inline, utils.getSpan, .{buf.ptr}); 145 | 146 | const arena: *Arena = @ptrCast(@alignCast(span.arena)); 147 | 148 | if (span.class.block_size <= medium_max) { 149 | @call(.always_inline, Arena.freeSmallOrMedium, .{ arena, span, buf }); 150 | } else if (span.class.block_size <= large_max) { 151 | @call(.always_inline, Arena.cacheLargeSpanOrFree, .{ arena, span }); 152 | } else { 153 | @call(.always_inline, Arena.freeHuge, .{ arena, span }); 154 | } 155 | } 156 | 157 | fn allocate(self: *Self, size: usize) ?[*]u8 { 158 | const arena = @call(.always_inline, SharedArenaHandler.getArena, .{&self.arena_handler}) orelse return null; 159 | defer @call(.always_inline, Arena.release, .{arena}); 160 | 161 | if (size <= small_max) { 162 | const size_class = @call(.always_inline, utils.getSmallSizeClass, .{size}); 163 | return @call(.always_inline, Arena.allocateToSpan, .{ arena, size_class }); 164 | } else if (size <= medium_max) { 165 | const size_class = @call(.always_inline, utils.getMediumSizeClass, .{size}); 166 | return @call(.always_inline, Arena.allocateToSpan, .{ arena, size_class }); 167 | } else if (size <= span_max) { 168 | return @call(.always_inline, Arena.allocateOneSpan, .{ arena, span_class }); 169 | } else if (size <= large_max) { 170 | const span_count = @call(.always_inline, utils.getSpanCount, .{size}); 171 | return @call(.always_inline, Arena.allocateToLargeSpan, .{ arena, span_count }); 172 | } else { 173 | const span_count = @call(.always_inline, utils.getHugeSpanCount, .{size}) orelse return null; 174 | return @call(.always_inline, Arena.allocateHuge, .{ arena, span_count }); 175 | } 176 | } 177 | 178 | pub fn usableSize(self: *Self, ptr: *anyopaque) usize { 179 | _ = self; 180 | 181 | const span = utils.getSpan(ptr); 182 | 183 | if (span.span_count == 1) { 184 | return span.class.block_size; 185 | } 186 | 187 | return span.alloc_size - (span.alloc_ptr - span.initial_ptr); 188 | } 189 | }; 190 | } 191 | 192 | const SizeClass = static_config.SizeClass; 193 | 194 | const span_size = static_config.span_size; 195 | const span_effective_size = static_config.span_effective_size; 196 | const span_header_size = static_config.span_header_size; 197 | const span_upper_mask = static_config.span_upper_mask; 198 | 199 | const small_granularity = static_config.small_granularity; 200 | const small_granularity_shift = static_config.small_granularity_shift; 201 | const small_max = static_config.small_max; 202 | 203 | const medium_granularity = static_config.medium_granularity; 204 | const medium_granularity_shift = static_config.medium_granularity_shift; 205 | const medium_max = static_config.medium_max; 206 | 207 | const span_max = static_config.span_max; 208 | const span_class = static_config.span_class; 209 | 210 | const large_max = static_config.large_max; 211 | 212 | const page_size = static_config.page_size; 213 | const page_alignment = static_config.page_alignment; 214 | 215 | const small_size_classes = static_config.small_size_classes; 216 | const medium_size_classes = static_config.medium_size_classes; 217 | 218 | const aligned_size_classes = static_config.aligned_size_classes; 219 | const aligned_spans_offset = static_config.aligned_spans_offset; 220 | const span_align_max = static_config.span_align_max; 221 | 222 | // 223 | // Tests 224 | // 225 | test "small allocations - free in same order" { 226 | var jdz_allocator = JdzAllocator(.{}).init(); 227 | defer jdz_allocator.deinit(); 228 | 229 | const allocator = jdz_allocator.allocator(); 230 | 231 | var list = std.ArrayList(*u64).init(std.testing.allocator); 232 | defer list.deinit(); 233 | 234 | var i: usize = 0; 235 | while (i < 513) : (i += 1) { 236 | const ptr = try allocator.create(u64); 237 | try list.append(ptr); 238 | } 239 | 240 | for (list.items) |ptr| { 241 | allocator.destroy(ptr); 242 | } 243 | } 244 | 245 | test "small allocations - free in reverse order" { 246 | var jdz_allocator = JdzAllocator(.{}).init(); 247 | defer jdz_allocator.deinit(); 248 | 249 | const allocator = jdz_allocator.allocator(); 250 | 251 | var list = std.ArrayList(*u64).init(std.testing.allocator); 252 | defer list.deinit(); 253 | 254 | var i: usize = 0; 255 | while (i < 513) : (i += 1) { 256 | const ptr = try allocator.create(u64); 257 | try list.append(ptr); 258 | } 259 | 260 | while (list.popOrNull()) |ptr| { 261 | allocator.destroy(ptr); 262 | } 263 | } 264 | 265 | test "small allocations - alloc free alloc" { 266 | var jdz_allocator = JdzAllocator(.{}).init(); 267 | defer jdz_allocator.deinit(); 268 | 269 | const allocator = jdz_allocator.allocator(); 270 | 271 | const a = try allocator.create(u64); 272 | allocator.destroy(a); 273 | const b = try allocator.create(u64); 274 | allocator.destroy(b); 275 | } 276 | 277 | test "large allocations" { 278 | var jdz_allocator = JdzAllocator(.{}).init(); 279 | defer jdz_allocator.deinit(); 280 | 281 | const allocator = jdz_allocator.allocator(); 282 | 283 | const ptr1 = try allocator.alloc(u64, 42768); 284 | const ptr2 = try allocator.alloc(u64, 52768); 285 | allocator.free(ptr1); 286 | const ptr3 = try allocator.alloc(u64, 62768); 287 | allocator.free(ptr3); 288 | allocator.free(ptr2); 289 | } 290 | 291 | test "very large allocation" { 292 | var jdz_allocator = JdzAllocator(.{}).init(); 293 | defer jdz_allocator.deinit(); 294 | 295 | const allocator = jdz_allocator.allocator(); 296 | 297 | try std.testing.expectError(error.OutOfMemory, allocator.alloc(u8, std.math.maxInt(usize))); 298 | } 299 | 300 | test "realloc" { 301 | var jdz_allocator = JdzAllocator(.{}).init(); 302 | defer jdz_allocator.deinit(); 303 | 304 | const allocator = jdz_allocator.allocator(); 305 | 306 | var slice = try allocator.alignedAlloc(u8, @alignOf(u32), 1); 307 | defer allocator.free(slice); 308 | slice[0] = 0x12; 309 | 310 | // This reallocation should keep its pointer address. 311 | const old_slice = slice; 312 | slice = try allocator.realloc(slice, 2); 313 | try std.testing.expect(old_slice.ptr == slice.ptr); 314 | try std.testing.expect(slice[0] == 0x12); 315 | slice[1] = 0x34; 316 | 317 | // This requires upgrading to a larger bin size 318 | slice = try allocator.realloc(slice, 17); 319 | try std.testing.expect(old_slice.ptr != slice.ptr); 320 | try std.testing.expect(slice[0] == 0x12); 321 | try std.testing.expect(slice[1] == 0x34); 322 | } 323 | 324 | test "shrink" { 325 | var jdz_allocator = JdzAllocator(.{}).init(); 326 | defer jdz_allocator.deinit(); 327 | 328 | const allocator = jdz_allocator.allocator(); 329 | 330 | var slice = try allocator.alloc(u8, 20); 331 | defer allocator.free(slice); 332 | 333 | @memset(slice, 0x11); 334 | 335 | try std.testing.expect(allocator.resize(slice, 17)); 336 | slice = slice[0..17]; 337 | 338 | for (slice) |b| { 339 | try std.testing.expect(b == 0x11); 340 | } 341 | 342 | try std.testing.expect(allocator.resize(slice, 16)); 343 | 344 | for (slice) |b| { 345 | try std.testing.expect(b == 0x11); 346 | } 347 | } 348 | 349 | test "large object - grow" { 350 | var jdz_allocator = JdzAllocator(.{}).init(); 351 | defer jdz_allocator.deinit(); 352 | 353 | const allocator = jdz_allocator.allocator(); 354 | 355 | var slice1 = try allocator.alloc(u8, 8192 - 10); 356 | defer allocator.free(slice1); 357 | 358 | const old = slice1; 359 | slice1 = try allocator.realloc(slice1, 8192 - 10); 360 | try std.testing.expect(slice1.ptr == old.ptr); 361 | 362 | slice1 = try allocator.realloc(slice1, 8192); 363 | try std.testing.expect(slice1.ptr == old.ptr); 364 | 365 | slice1 = try allocator.realloc(slice1, 8192 + 1); 366 | } 367 | 368 | test "realloc small object to large object" { 369 | var jdz_allocator = JdzAllocator(.{}).init(); 370 | defer jdz_allocator.deinit(); 371 | 372 | const allocator = jdz_allocator.allocator(); 373 | 374 | var slice = try allocator.alloc(u8, 70); 375 | defer allocator.free(slice); 376 | slice[0] = 0x12; 377 | slice[60] = 0x34; 378 | 379 | // This requires upgrading to a large object 380 | const large_object_size = 8192 + 50; 381 | slice = try allocator.realloc(slice, large_object_size); 382 | try std.testing.expect(slice[0] == 0x12); 383 | try std.testing.expect(slice[60] == 0x34); 384 | } 385 | 386 | test "shrink large object to large object" { 387 | var jdz_allocator = JdzAllocator(.{}).init(); 388 | defer jdz_allocator.deinit(); 389 | 390 | const allocator = jdz_allocator.allocator(); 391 | 392 | var slice = try allocator.alloc(u8, 8192 + 50); 393 | defer allocator.free(slice); 394 | slice[0] = 0x12; 395 | slice[60] = 0x34; 396 | 397 | if (!allocator.resize(slice, 8192 + 1)) return; 398 | slice = slice.ptr[0 .. 8192 + 1]; 399 | try std.testing.expect(slice[0] == 0x12); 400 | try std.testing.expect(slice[60] == 0x34); 401 | 402 | try std.testing.expect(allocator.resize(slice, 8192 + 1)); 403 | slice = slice[0 .. 8192 + 1]; 404 | try std.testing.expect(slice[0] == 0x12); 405 | try std.testing.expect(slice[60] == 0x34); 406 | 407 | slice = try allocator.realloc(slice, 8192); 408 | try std.testing.expect(slice[0] == 0x12); 409 | try std.testing.expect(slice[60] == 0x34); 410 | } 411 | 412 | test "shrink large object to large object with larger alignment" { 413 | var jdz_allocator = JdzAllocator(.{}).init(); 414 | defer jdz_allocator.deinit(); 415 | 416 | const allocator = jdz_allocator.allocator(); 417 | 418 | var debug_buffer: [1000]u8 = undefined; 419 | var fba = std.heap.FixedBufferAllocator.init(&debug_buffer); 420 | const debug_allocator = fba.allocator(); 421 | 422 | const alloc_size = 8192 + 50; 423 | var slice = try allocator.alignedAlloc(u8, 16, alloc_size); 424 | defer allocator.free(slice); 425 | 426 | const big_alignment: usize = switch (builtin.os.tag) { 427 | .windows => 65536, // Windows aligns to 64K. 428 | else => 8192, 429 | }; 430 | // This loop allocates until we find a page that is not aligned to the big 431 | // alignment. Then we shrink the allocation after the loop, but increase the 432 | // alignment to the higher one, that we know will force it to realloc. 433 | var stuff_to_free = std.ArrayList([]align(16) u8).init(debug_allocator); 434 | while (std.mem.isAligned(@intFromPtr(slice.ptr), big_alignment)) { 435 | try stuff_to_free.append(slice); 436 | slice = try allocator.alignedAlloc(u8, 16, alloc_size); 437 | } 438 | while (stuff_to_free.popOrNull()) |item| { 439 | allocator.free(item); 440 | } 441 | slice[0] = 0x12; 442 | slice[60] = 0x34; 443 | 444 | slice = try allocator.reallocAdvanced(slice, big_alignment, alloc_size / 2); 445 | try std.testing.expect(slice[0] == 0x12); 446 | try std.testing.expect(slice[60] == 0x34); 447 | } 448 | 449 | test "realloc large object to small object" { 450 | var jdz_allocator = JdzAllocator(.{}).init(); 451 | defer jdz_allocator.deinit(); 452 | 453 | const allocator = jdz_allocator.allocator(); 454 | 455 | var slice = try allocator.alloc(u8, 8192 + 50); 456 | defer allocator.free(slice); 457 | slice[0] = 0x12; 458 | slice[16] = 0x34; 459 | 460 | slice = try allocator.realloc(slice, 19); 461 | try std.testing.expect(slice[0] == 0x12); 462 | try std.testing.expect(slice[16] == 0x34); 463 | } 464 | 465 | test "realloc large object to larger alignment" { 466 | var jdz_allocator = JdzAllocator(.{}).init(); 467 | defer jdz_allocator.deinit(); 468 | 469 | const allocator = jdz_allocator.allocator(); 470 | 471 | var debug_buffer: [1000]u8 = undefined; 472 | var fba = std.heap.FixedBufferAllocator.init(&debug_buffer); 473 | const debug_allocator = fba.allocator(); 474 | 475 | var slice = try allocator.alignedAlloc(u8, 16, 8192 + 50); 476 | defer allocator.free(slice); 477 | 478 | const big_alignment: usize = switch (builtin.os.tag) { 479 | .windows => 65536, // Windows aligns to 64K. 480 | else => 8192, 481 | }; 482 | // This loop allocates until we find a page that is not aligned to the big alignment. 483 | var stuff_to_free = std.ArrayList([]align(16) u8).init(debug_allocator); 484 | while (std.mem.isAligned(@intFromPtr(slice.ptr), big_alignment)) { 485 | try stuff_to_free.append(slice); 486 | slice = try allocator.alignedAlloc(u8, 16, 8192 + 50); 487 | } 488 | while (stuff_to_free.popOrNull()) |item| { 489 | allocator.free(item); 490 | } 491 | slice[0] = 0x12; 492 | slice[16] = 0x34; 493 | 494 | slice = try allocator.reallocAdvanced(slice, 32, 8192 + 100); 495 | try std.testing.expect(slice[0] == 0x12); 496 | try std.testing.expect(slice[16] == 0x34); 497 | 498 | slice = try allocator.reallocAdvanced(slice, 32, 8192 + 25); 499 | try std.testing.expect(slice[0] == 0x12); 500 | try std.testing.expect(slice[16] == 0x34); 501 | 502 | slice = try allocator.reallocAdvanced(slice, big_alignment, 8192 + 100); 503 | try std.testing.expect(slice[0] == 0x12); 504 | try std.testing.expect(slice[16] == 0x34); 505 | } 506 | 507 | test "large object shrinks to small" { 508 | var jdz_allocator = JdzAllocator(.{}).init(); 509 | defer jdz_allocator.deinit(); 510 | 511 | const allocator = jdz_allocator.allocator(); 512 | 513 | const slice = try allocator.alloc(u8, 8192 + 50); 514 | defer allocator.free(slice); 515 | 516 | try std.testing.expect(allocator.resize(slice, 4)); 517 | } 518 | 519 | test "objects of size 1024 and 2048" { 520 | var jdz_allocator = JdzAllocator(.{}).init(); 521 | defer jdz_allocator.deinit(); 522 | 523 | const allocator = jdz_allocator.allocator(); 524 | 525 | const slice = try allocator.alloc(u8, 1025); 526 | const slice2 = try allocator.alloc(u8, 3000); 527 | 528 | allocator.free(slice); 529 | allocator.free(slice2); 530 | } 531 | 532 | test "max large alloc" { 533 | var jdz_allocator = JdzAllocator(.{}).init(); 534 | defer jdz_allocator.deinit(); 535 | 536 | const allocator = jdz_allocator.allocator(); 537 | 538 | const buf = try allocator.alloc(u8, large_max); 539 | defer allocator.free(buf); 540 | } 541 | 542 | test "huge alloc" { 543 | var jdz_allocator = JdzAllocator(.{}).init(); 544 | defer jdz_allocator.deinit(); 545 | 546 | const allocator = jdz_allocator.allocator(); 547 | 548 | const buf = try allocator.alloc(u8, large_max + 1); 549 | defer allocator.free(buf); 550 | } 551 | 552 | test "huge alloc does not try to access span" { 553 | var jdz_allocator = JdzAllocator(.{}).init(); 554 | defer jdz_allocator.deinit(); 555 | 556 | const allocator = jdz_allocator.allocator(); 557 | 558 | const buf1 = try allocator.alloc(u8, large_max + 1); 559 | const buf2 = try allocator.alloc(u8, large_max + 1); 560 | 561 | allocator.free(buf1); 562 | allocator.free(buf2); 563 | } 564 | 565 | test "small alignment small alloc" { 566 | var jdz_allocator = JdzAllocator(.{}).init(); 567 | defer jdz_allocator.deinit(); 568 | 569 | const allocator = jdz_allocator.allocator(); 570 | 571 | const slice = allocator.rawAlloc(1, 4, @returnAddress()).?; 572 | defer allocator.rawFree(slice[0..1], 4, @returnAddress()); 573 | 574 | try std.testing.expect(@intFromPtr(slice) % std.math.pow(u16, 2, 4) == 0); 575 | } 576 | 577 | test "medium alignment small alloc" { 578 | var jdz_allocator = JdzAllocator(.{}).init(); 579 | defer jdz_allocator.deinit(); 580 | 581 | const allocator = jdz_allocator.allocator(); 582 | 583 | const alignment: u8 = @truncate(log2(utils.roundUpToPowerOfTwo(small_max + 1))); 584 | 585 | if (alignment > page_alignment) return error.SkipZigTest; 586 | 587 | const slice = allocator.rawAlloc(1, alignment, @returnAddress()).?; 588 | defer allocator.rawFree(slice[0..1], alignment, @returnAddress()); 589 | 590 | try std.testing.expect(@intFromPtr(slice) % std.math.pow(u16, 2, 4) == 0); 591 | } 592 | 593 | test "page size alignment small alloc" { 594 | var jdz_allocator = JdzAllocator(.{}).init(); 595 | defer jdz_allocator.deinit(); 596 | 597 | const allocator = jdz_allocator.allocator(); 598 | 599 | const slice = allocator.rawAlloc(1, page_alignment, @returnAddress()).?; 600 | defer allocator.rawFree(slice[0..1], page_alignment, @returnAddress()); 601 | 602 | try std.testing.expect(@intFromPtr(slice) % std.math.pow(u16, 2, page_alignment) == 0); 603 | } 604 | 605 | test "small alignment large alloc" { 606 | var jdz_allocator = JdzAllocator(.{}).init(); 607 | defer jdz_allocator.deinit(); 608 | 609 | const allocator = jdz_allocator.allocator(); 610 | 611 | const slice = allocator.rawAlloc(span_max, 4, @returnAddress()).?; 612 | defer allocator.rawFree(slice[0..span_max], 4, @returnAddress()); 613 | 614 | try std.testing.expect(@intFromPtr(slice) % std.math.pow(u16, 2, 4) == 0); 615 | } 616 | 617 | test "medium alignment large alloc" { 618 | var jdz_allocator = JdzAllocator(.{}).init(); 619 | defer jdz_allocator.deinit(); 620 | 621 | const allocator = jdz_allocator.allocator(); 622 | 623 | const alignment: u8 = @truncate(log2(utils.roundUpToPowerOfTwo(small_max + 1))); 624 | 625 | if (alignment > page_alignment) return error.SkipZigTest; 626 | 627 | const slice = allocator.rawAlloc(span_max, alignment, @returnAddress()).?; 628 | defer allocator.rawFree(slice[0..span_max], alignment, @returnAddress()); 629 | 630 | try std.testing.expect(@intFromPtr(slice) % std.math.pow(u16, 2, 4) == 0); 631 | } 632 | 633 | test "page size alignment large alloc" { 634 | var jdz_allocator = JdzAllocator(.{}).init(); 635 | defer jdz_allocator.deinit(); 636 | 637 | const allocator = jdz_allocator.allocator(); 638 | 639 | const slice = allocator.rawAlloc(span_max, page_alignment, @returnAddress()).?; 640 | defer allocator.rawFree(slice[0..span_max], page_alignment, @returnAddress()); 641 | 642 | try std.testing.expect(@intFromPtr(slice) % std.math.pow(u16, 2, page_alignment) == 0); 643 | } 644 | 645 | test "consecutive overalignment" { 646 | var jdz_allocator = JdzAllocator(.{}).init(); 647 | defer jdz_allocator.deinit(); 648 | 649 | const allocator = jdz_allocator.allocator(); 650 | 651 | const overaligned_sizes = [_]usize{ 192, 192, 192 }; 652 | var buffers: [overaligned_sizes.len][]const u8 = undefined; 653 | 654 | for (overaligned_sizes, &buffers) |overaligned_size, *buffers_slot| { 655 | buffers_slot.* = try allocator.alignedAlloc(u8, 64, overaligned_size); 656 | } 657 | 658 | for (buffers) |buffer| { 659 | allocator.free(buffer); 660 | } 661 | } 662 | -------------------------------------------------------------------------------- /src/shared_arena_handler.zig: -------------------------------------------------------------------------------- 1 | const std = @import("std"); 2 | const span_arena = @import("arena.zig"); 3 | const jdz_allocator = @import("jdz_allocator.zig"); 4 | const utils = @import("utils.zig"); 5 | 6 | const JdzAllocConfig = jdz_allocator.JdzAllocConfig; 7 | 8 | const assert = std.debug.assert; 9 | 10 | threadlocal var cached_thread_id: ?std.Thread.Id = null; 11 | 12 | pub fn SharedArenaHandler(comptime config: JdzAllocConfig) type { 13 | const Arena = span_arena.Arena(config, false); 14 | 15 | const Mutex = utils.getMutexType(config); 16 | 17 | return struct { 18 | arena_list: ?*Arena, 19 | mutex: Mutex, 20 | arena_batch: u32, 21 | 22 | const Self = @This(); 23 | 24 | pub fn init() Self { 25 | return .{ 26 | .arena_list = null, 27 | .mutex = .{}, 28 | .arena_batch = 0, 29 | }; 30 | } 31 | 32 | pub fn deinit(self: *Self) usize { 33 | self.mutex.lock(); 34 | defer self.mutex.unlock(); 35 | 36 | var spans_leaked: usize = 0; 37 | var opt_arena = self.arena_list; 38 | 39 | while (opt_arena) |arena| { 40 | const next = arena.next; 41 | 42 | spans_leaked += arena.deinit(); 43 | 44 | opt_arena = next; 45 | } 46 | 47 | while (opt_arena) |arena| { 48 | const next = arena.next; 49 | 50 | if (arena.is_alloc_master) { 51 | config.backing_allocator.destroy(arena); 52 | } 53 | 54 | opt_arena = next; 55 | } 56 | 57 | return spans_leaked; 58 | } 59 | 60 | pub inline fn getArena(self: *Self) ?*Arena { 61 | const tid = getThreadId(); 62 | 63 | return self.findOwnedThreadArena(tid) orelse 64 | self.claimOrCreateArena(tid); 65 | } 66 | 67 | inline fn findOwnedThreadArena(self: *Self, tid: std.Thread.Id) ?*Arena { 68 | var opt_arena = self.arena_list; 69 | 70 | while (opt_arena) |list_arena| { 71 | if (list_arena.thread_id == tid or list_arena.thread_id == null) { 72 | return acquireArena(list_arena, tid) orelse continue; 73 | } 74 | 75 | opt_arena = list_arena.next; 76 | } 77 | 78 | return null; 79 | } 80 | 81 | inline fn claimOrCreateArena(self: *Self, tid: std.Thread.Id) ?*Arena { 82 | var opt_arena = self.arena_list; 83 | 84 | while (opt_arena) |arena| { 85 | return acquireArena(arena, tid) orelse { 86 | opt_arena = arena.next; 87 | 88 | continue; 89 | }; 90 | } 91 | 92 | return self.tryCreateArena(); 93 | } 94 | 95 | fn tryCreateArena(self: *Self) ?*Arena { 96 | const arena_batch = self.arena_batch; 97 | 98 | self.mutex.lock(); 99 | defer self.mutex.unlock(); 100 | 101 | if (arena_batch != self.arena_batch) { 102 | return self.getArena(); 103 | } 104 | 105 | return self.createArena(); 106 | } 107 | 108 | fn createArena(self: *Self) ?*Arena { 109 | var new_arenas = config.backing_allocator.alloc(Arena, config.shared_arena_batch_size) catch { 110 | return null; 111 | }; 112 | 113 | self.arena_batch += 1; 114 | 115 | for (new_arenas) |*new_arena| { 116 | new_arena.* = Arena.init(.unlocked, null); 117 | } 118 | 119 | new_arenas[0].makeMaster(); 120 | const acquired = acquireArena(&new_arenas[0], getThreadId()); 121 | 122 | assert(acquired != null); 123 | 124 | for (new_arenas) |*new_arena| { 125 | self.addArenaToList(new_arena); 126 | } 127 | 128 | return acquired; 129 | } 130 | 131 | fn addArenaToList(self: *Self, new_arena: *Arena) void { 132 | if (self.arena_list == null) { 133 | self.arena_list = new_arena; 134 | 135 | return; 136 | } 137 | 138 | var arena = self.arena_list.?; 139 | 140 | while (arena.next) |next| { 141 | arena = next; 142 | } 143 | 144 | arena.next = new_arena; 145 | } 146 | 147 | inline fn acquireArena(arena: *Arena, tid: std.Thread.Id) ?*Arena { 148 | if (arena.tryAcquire()) { 149 | arena.thread_id = tid; 150 | 151 | return arena; 152 | } 153 | 154 | return null; 155 | } 156 | 157 | inline fn getThreadId() std.Thread.Id { 158 | return cached_thread_id orelse { 159 | cached_thread_id = std.Thread.getCurrentId(); 160 | 161 | return cached_thread_id.?; 162 | }; 163 | } 164 | }; 165 | } 166 | -------------------------------------------------------------------------------- /src/span.zig: -------------------------------------------------------------------------------- 1 | const std = @import("std"); 2 | 3 | const jdz_allocator = @import("jdz_allocator.zig"); 4 | const static_config = @import("static_config.zig"); 5 | const utils = @import("utils.zig"); 6 | const span_file = @import("span.zig"); 7 | 8 | const SpanList = @import("SpanList.zig"); 9 | const JdzAllocConfig = jdz_allocator.JdzAllocConfig; 10 | const SizeClass = static_config.SizeClass; 11 | const Value = std.atomic.Value; 12 | 13 | const assert = std.debug.assert; 14 | 15 | const span_size = static_config.span_size; 16 | const span_header_size = static_config.span_header_size; 17 | const page_size = static_config.page_size; 18 | const mod_page_size = static_config.mod_page_size; 19 | const free_list_null = static_config.free_list_null; 20 | 21 | const invalid_pointer: usize = std.mem.alignBackward(usize, std.math.maxInt(usize), static_config.small_granularity); 22 | 23 | pub const Span = extern struct { 24 | free_list: usize, 25 | full: bool, 26 | aligned_blocks: bool, 27 | block_count: u16, 28 | class: SizeClass, 29 | span_count: usize, 30 | arena: *anyopaque, 31 | next: ?*Span, 32 | prev: ?*Span, 33 | alloc_ptr: usize, 34 | initial_ptr: usize, 35 | alloc_size: usize, 36 | 37 | deferred_free_list: usize align(std.atomic.cache_line), 38 | deferred_frees: u16, 39 | 40 | pub inline fn pushFreeList(self: *Span, buf: []u8) void { 41 | const ptr = @call(.always_inline, Span.getBlockPtr, .{ self, buf }); 42 | 43 | @call(.always_inline, Span.pushFreeListElement, .{ self, ptr }); 44 | 45 | self.block_count -= 1; 46 | } 47 | 48 | pub inline fn pushDeferredFreeList(self: *Span, buf: []u8) void { 49 | const ptr = @call(.always_inline, Span.getBlockPtr, .{ self, buf }); 50 | 51 | @call(.always_inline, Span.pushDeferredFreeListElement, .{ self, ptr }); 52 | } 53 | 54 | pub fn allocate(self: *Span) [*]u8 { 55 | if (self.free_list != free_list_null) { 56 | return self.popFreeListElement(); 57 | } 58 | 59 | return self.allocateDeferredOrPtr(); 60 | } 61 | 62 | pub fn allocateFromFreshSpan(self: *Span) [*]u8 { 63 | assert(self.isEmpty()); 64 | 65 | return self.allocateFromAllocPtr(); 66 | } 67 | 68 | pub fn allocateFromAllocPtr(self: *Span) [*]u8 { 69 | assert(self.alloc_ptr <= @intFromPtr(self) + span_size - self.class.block_size); 70 | 71 | self.block_count += 1; 72 | 73 | const next_page = self.alloc_ptr + page_size - (self.alloc_ptr & mod_page_size); 74 | const end_span = @intFromPtr(self) + span_size; 75 | const target = @min(end_span, next_page); 76 | const bytes_to_fill = target - self.alloc_ptr; 77 | const blocks_to_add = bytes_to_fill / self.class.block_size; 78 | 79 | const res: [*]u8 = @ptrFromInt(self.alloc_ptr); 80 | self.alloc_ptr += self.class.block_size; 81 | 82 | if (blocks_to_add > 1) { 83 | self.free_list = self.alloc_ptr; 84 | 85 | for (1..blocks_to_add) |_| { 86 | self.pushFreeListElementForwardPointing(); 87 | } 88 | 89 | @as(*usize, @ptrFromInt(self.alloc_ptr - self.class.block_size)).* = free_list_null; 90 | } 91 | 92 | return res; 93 | } 94 | 95 | pub fn allocateFromLargeSpan(self: *Span) [*]u8 { 96 | assert(self.isEmpty()); 97 | 98 | self.block_count = 1; 99 | 100 | return @as([*]u8, @ptrFromInt(self.alloc_ptr)); 101 | } 102 | 103 | pub inline fn popFreeListElement(self: *Span) [*]u8 { 104 | self.block_count += 1; 105 | 106 | const block = self.free_list; 107 | self.free_list = @as(*usize, @ptrFromInt(block)).*; 108 | 109 | return @ptrFromInt(block); 110 | } 111 | 112 | pub fn initialiseFreshSpan(self: *Span, arena: *anyopaque, size_class: SizeClass) void { 113 | self.* = .{ 114 | .arena = arena, 115 | .initial_ptr = self.initial_ptr, 116 | .alloc_ptr = @intFromPtr(self) + span_header_size, 117 | .alloc_size = self.alloc_size, 118 | .class = size_class, 119 | .free_list = free_list_null, 120 | .deferred_free_list = free_list_null, 121 | .full = false, 122 | .next = null, 123 | .prev = null, 124 | .block_count = 0, 125 | .deferred_frees = 0, 126 | .span_count = 1, 127 | .aligned_blocks = false, 128 | }; 129 | } 130 | 131 | pub fn initialiseFreshLargeSpan(self: *Span, arena: *anyopaque, span_count: usize) void { 132 | assert(static_config.large_max <= std.math.maxInt(u32)); 133 | 134 | self.* = .{ 135 | .arena = arena, 136 | .initial_ptr = self.initial_ptr, 137 | .alloc_ptr = @intFromPtr(self) + span_header_size, 138 | .alloc_size = self.alloc_size, 139 | .class = .{ 140 | .block_size = @truncate(span_count * span_size - span_header_size), 141 | .class_idx = undefined, 142 | .block_max = 1, 143 | }, 144 | .free_list = free_list_null, 145 | .deferred_free_list = free_list_null, 146 | .full = false, 147 | .next = null, 148 | .prev = null, 149 | .block_count = 0, 150 | .deferred_frees = 0, 151 | .span_count = span_count, 152 | .aligned_blocks = false, 153 | }; 154 | } 155 | 156 | pub inline fn isFull(self: *Span) bool { 157 | return self.block_count == self.class.block_max and self.deferred_frees == 0; 158 | } 159 | 160 | pub inline fn isEmpty(self: *Span) bool { 161 | return self.block_count - self.deferred_frees == 0; 162 | } 163 | 164 | pub inline fn splitLastSpans(self: *Span, span_count: usize) *Span { 165 | return self.splitFirstSpansReturnRemaining(self.span_count - span_count); 166 | } 167 | 168 | pub inline fn splitFirstSpanReturnRemaining(self: *Span) *Span { 169 | return self.splitFirstSpansReturnRemaining(1); 170 | } 171 | 172 | pub fn splitFirstSpansReturnRemaining(self: *Span, span_count: usize) *Span { 173 | assert(self.span_count > span_count); 174 | 175 | const remaining_span_addr = @intFromPtr(self) + span_size * span_count; 176 | const remaining_span: *Span = @ptrFromInt(remaining_span_addr); 177 | remaining_span.span_count = self.span_count - span_count; 178 | remaining_span.alloc_size = self.alloc_size - (remaining_span_addr - self.initial_ptr); 179 | remaining_span.initial_ptr = remaining_span_addr; 180 | 181 | self.span_count = span_count; 182 | self.alloc_size = remaining_span.initial_ptr - self.initial_ptr; 183 | 184 | return remaining_span; 185 | } 186 | 187 | inline fn allocateDeferredOrPtr(self: *Span) [*]u8 { 188 | if (self.freeDeferredList()) { 189 | return self.popFreeListElement(); 190 | } else { 191 | return self.allocateFromAllocPtr(); 192 | } 193 | } 194 | 195 | inline fn getBlockPtr(self: *Span, buf: []u8) [*]u8 { 196 | if (!self.aligned_blocks) { 197 | return buf.ptr; 198 | } else { 199 | const start_alloc_ptr = @intFromPtr(self) + span_header_size; 200 | const block_offset = @intFromPtr(buf.ptr) - start_alloc_ptr; 201 | 202 | return buf.ptr - block_offset % self.class.block_size; 203 | } 204 | } 205 | 206 | inline fn pushFreeListElementForwardPointing(self: *Span) void { 207 | const next_block = self.alloc_ptr + self.class.block_size; 208 | @as(*usize, @ptrFromInt(self.alloc_ptr)).* = next_block; 209 | self.alloc_ptr = next_block; 210 | } 211 | 212 | inline fn pushFreeListElement(self: *Span, ptr: [*]u8) void { 213 | const block: *?usize = @ptrCast(@alignCast(ptr)); 214 | block.* = self.free_list; 215 | self.free_list = @intFromPtr(block); 216 | } 217 | 218 | inline fn pushDeferredFreeListElement(self: *Span, ptr: [*]u8) void { 219 | const block: *usize = @ptrCast(@alignCast(ptr)); 220 | 221 | while (true) { 222 | block.* = @atomicRmw(usize, &self.deferred_free_list, .Xchg, invalid_pointer, .acquire); 223 | 224 | if (block.* != invalid_pointer) { 225 | break; 226 | } 227 | } 228 | 229 | self.deferred_frees += 1; 230 | 231 | @atomicStore(usize, &self.deferred_free_list, @intFromPtr(block), .release); 232 | } 233 | 234 | inline fn freeDeferredList(self: *Span) bool { 235 | assert(self.free_list == free_list_null); 236 | 237 | if (self.deferred_free_list == free_list_null) return false; 238 | 239 | while (true) { 240 | self.free_list = @atomicRmw(usize, &self.deferred_free_list, .Xchg, invalid_pointer, .acquire); 241 | 242 | if (self.free_list != invalid_pointer) { 243 | break; 244 | } 245 | } 246 | self.block_count -= self.deferred_frees; 247 | self.deferred_frees = 0; 248 | 249 | @atomicStore(usize, &self.deferred_free_list, free_list_null, .release); 250 | 251 | return true; 252 | } 253 | }; 254 | -------------------------------------------------------------------------------- /src/span_cache.zig: -------------------------------------------------------------------------------- 1 | const std = @import("std"); 2 | 3 | const jdz_allocator = @import("jdz_allocator.zig"); 4 | const bounded_stack = @import("bounded_stack.zig"); 5 | const static_config = @import("static_config.zig"); 6 | const utils = @import("utils.zig"); 7 | const span_file = @import("span.zig"); 8 | 9 | const Span = span_file.Span; 10 | const testing = std.testing; 11 | const assert = std.debug.assert; 12 | const JdzAllocConfig = jdz_allocator.JdzAllocConfig; 13 | 14 | const span_size = static_config.span_size; 15 | 16 | pub fn SpanCache(comptime cache_limit: u32) type { 17 | assert(utils.isPowerOfTwo(cache_limit)); 18 | 19 | const Cache = bounded_stack.BoundedStack(*Span, cache_limit); 20 | 21 | return struct { 22 | cache: Cache, 23 | 24 | const Self = @This(); 25 | 26 | pub fn init() Self { 27 | return .{ 28 | .cache = Cache.init(), 29 | }; 30 | } 31 | 32 | pub fn tryWrite(self: *Self, span: *Span) bool { 33 | return self.cache.tryWrite(span); 34 | } 35 | 36 | pub fn tryRead(self: *Self) ?*Span { 37 | const span = self.cache.tryRead() orelse return null; 38 | 39 | if (span.span_count > 1) { 40 | const split_spans = span.splitFirstSpanReturnRemaining(); 41 | 42 | _ = self.cache.tryWrite(split_spans); 43 | } 44 | 45 | return span; 46 | } 47 | }; 48 | } 49 | -------------------------------------------------------------------------------- /src/static_config.zig: -------------------------------------------------------------------------------- 1 | const std = @import("std"); 2 | const utils = @import("utils.zig"); 3 | const span_file = @import("span.zig"); 4 | 5 | const Span = span_file.Span; 6 | 7 | const log2 = std.math.log2; 8 | const log2_int = std.math.log2_int; 9 | const assert = std.debug.assert; 10 | 11 | pub const SizeClass = extern struct { 12 | block_size: u32, 13 | block_max: u16, 14 | class_idx: u16, 15 | }; 16 | 17 | pub const span_size = 65536; 18 | pub const mod_span_size = span_size - 1; 19 | 20 | // must be a multiple of small_granularity 21 | pub const span_header_size = 256; 22 | pub const span_effective_size = span_size - span_header_size; 23 | pub const span_max = span_effective_size; 24 | 25 | pub const page_size = std.mem.page_size; 26 | pub const mod_page_size = page_size - 1; 27 | pub const page_alignment = log2(page_size); 28 | 29 | pub const span_alignment = log2(span_size); 30 | pub const span_lower_mask: usize = span_size - 1; 31 | pub const span_upper_mask: usize = ~span_lower_mask; 32 | 33 | pub const small_granularity = 16; 34 | pub const small_granularity_shift = log2(small_granularity); 35 | pub const small_max = 2048; 36 | pub const small_class_count = small_max / small_granularity; 37 | 38 | pub const medium_granularity = 256; 39 | pub const medium_granularity_shift = log2(medium_granularity); 40 | // fit at least 2 medium allocs in one span 41 | pub const medium_max = span_effective_size / 2 - ((span_effective_size / 2) % medium_granularity); 42 | pub const medium_class_count = (medium_max - small_max) / medium_granularity; 43 | 44 | pub const large_class_count = 64; 45 | pub const large_max = large_class_count * span_size - span_header_size; 46 | 47 | pub const free_list_null: usize = 0; 48 | 49 | pub const size_class_count = small_class_count + medium_class_count; 50 | 51 | pub const small_size_classes = generateSmallSizeClasses(); 52 | pub const medium_size_classes = generateMediumSizeClasses(); 53 | 54 | pub const span_class = SizeClass{ 55 | .block_max = 1, 56 | .block_size = span_effective_size, 57 | .class_idx = small_class_count + medium_class_count, 58 | }; 59 | 60 | fn generateSmallSizeClasses() [small_class_count]SizeClass { 61 | var size_classes: [small_class_count]SizeClass = undefined; 62 | 63 | for (0..small_class_count) |i| { 64 | size_classes[i].block_size = (i + 1) * small_granularity; 65 | size_classes[i].block_max = span_effective_size / size_classes[i].block_size; 66 | size_classes[i].class_idx = i; 67 | } 68 | 69 | mergeSizeClasses(&size_classes); 70 | 71 | assert(size_classes[0].block_size == small_granularity); 72 | assert(size_classes[size_classes.len - 1].block_size == small_max); 73 | 74 | return size_classes; 75 | } 76 | 77 | fn generateMediumSizeClasses() [medium_class_count]SizeClass { 78 | var size_classes: [medium_class_count]SizeClass = undefined; 79 | 80 | for (0..medium_class_count) |i| { 81 | size_classes[i].block_size = small_max + (i + 1) * medium_granularity; 82 | size_classes[i].block_max = span_effective_size / size_classes[i].block_size; 83 | size_classes[i].class_idx = small_class_count + i; 84 | } 85 | 86 | mergeSizeClasses(&size_classes); 87 | 88 | assert(size_classes[0].block_size == small_max + medium_granularity); 89 | assert(size_classes[size_classes.len - 1].block_size == medium_max); 90 | assert(size_classes[size_classes.len - 1].block_max > 1); 91 | 92 | return size_classes; 93 | } 94 | 95 | // merge size classes with equal block count to higher size 96 | fn mergeSizeClasses(size_classes: []SizeClass) void { 97 | var i = size_classes.len - 1; 98 | while (i > 0) : (i -= 1) { 99 | if (size_classes[i].block_max == size_classes[i - 1].block_max) { 100 | // need to maintain power of 2 classes for alignment 101 | if (utils.isPowerOfTwo(size_classes[i - 1].block_size)) continue; 102 | 103 | size_classes[i - 1].block_size = size_classes[i].block_size; 104 | size_classes[i - 1].class_idx = size_classes[i].class_idx; 105 | } 106 | } 107 | } 108 | -------------------------------------------------------------------------------- /src/utils.zig: -------------------------------------------------------------------------------- 1 | const std = @import("std"); 2 | 3 | const jdz_allocator = @import("jdz_allocator.zig"); 4 | const static_config = @import("static_config.zig"); 5 | const lock = @import("lock.zig"); 6 | const span_file = @import("span.zig"); 7 | 8 | const JdzAllocConfig = jdz_allocator.JdzAllocConfig; 9 | const SizeClass = static_config.SizeClass; 10 | 11 | const Span = span_file.Span; 12 | const Value = std.atomic.Value; 13 | const AtomicOrder = std.builtin.AtomicOrder; 14 | 15 | const assert = std.debug.assert; 16 | 17 | const usize_bits_subbed = @bitSizeOf(usize) - 1; 18 | 19 | const log2_usize_type = @Type(std.builtin.Type{ .Int = std.builtin.Type.Int{ 20 | .signedness = std.builtin.Signedness.unsigned, 21 | .bits = std.math.log2(@bitSizeOf(usize)), 22 | } }); 23 | 24 | const DummyMutex = struct { 25 | pub fn lock(_: @This()) void {} 26 | pub fn unlock(_: @This()) void {} 27 | }; 28 | 29 | pub fn getMutexType(comptime config: JdzAllocConfig) type { 30 | return if (config.thread_safe) 31 | std.Thread.Mutex 32 | else 33 | DummyMutex; 34 | } 35 | 36 | pub fn getArenaLockType(comptime config: JdzAllocConfig) type { 37 | return if (config.thread_safe) 38 | lock.Lock 39 | else 40 | lock.DummyLock; 41 | } 42 | 43 | pub inline fn getSmallSizeClass(len: usize) SizeClass { 44 | assert(len <= small_max); 45 | 46 | return small_size_classes[getSmallSizeIdx(len)]; 47 | } 48 | 49 | pub inline fn getMediumSizeClass(len: usize) SizeClass { 50 | assert(len > small_max and len <= medium_max); 51 | 52 | return medium_size_classes[getMediumSizeIdx(len)]; 53 | } 54 | 55 | pub inline fn getSmallSizeIdx(len: usize) usize { 56 | return (len - 1) >> small_granularity_shift; 57 | } 58 | 59 | pub inline fn getMediumSizeIdx(len: usize) usize { 60 | return (len - small_max - 1) >> medium_granularity_shift; 61 | } 62 | 63 | pub inline fn getSpan(ptr: *anyopaque) *Span { 64 | return @ptrFromInt(@intFromPtr(ptr) & span_upper_mask); 65 | } 66 | 67 | pub inline fn isPowerOfTwo(n: u64) bool { 68 | return (n & (n - 1)) == 0; 69 | } 70 | 71 | pub inline fn getSpanCount(len: usize) usize { 72 | assert(len <= large_max); 73 | 74 | const size: usize = len + span_header_size; 75 | 76 | return (((size - 1) / span_size) * span_size + span_size) / span_size; 77 | } 78 | 79 | pub inline fn getHugeSpanCount(len: usize) ?usize { 80 | if (@addWithOverflow(len, span_header_size).@"0" < len) { 81 | return null; 82 | } 83 | 84 | const size: usize = len + span_header_size; 85 | 86 | return (((size - 1) / span_size) * span_size + span_size) / span_size; 87 | } 88 | 89 | pub inline fn roundUpToPowerOfTwo(n: usize) usize { 90 | assert(n > 1); 91 | assert(n <= std.math.maxInt(usize) / 2 + 1); 92 | 93 | return @as(usize, 1) << @truncate(@as(usize, @bitSizeOf(usize) - @clz(n - 1))); 94 | } 95 | 96 | pub inline fn tryCASAddOne(atomic_ptr: *Value(usize), val: usize, success_ordering: AtomicOrder) ?usize { 97 | return atomic_ptr.cmpxchgWeak(val, val + 1, success_ordering, .monotonic); 98 | } 99 | 100 | pub inline fn resetLinkedSpan(span: *Span) void { 101 | span.next = null; 102 | span.prev = null; 103 | } 104 | 105 | const span_size = static_config.span_size; 106 | const span_upper_mask = static_config.span_upper_mask; 107 | const span_header_size = static_config.span_header_size; 108 | 109 | const small_granularity_shift = static_config.small_granularity_shift; 110 | const small_max = static_config.small_max; 111 | 112 | const medium_granularity_shift = static_config.medium_granularity_shift; 113 | const medium_max = static_config.medium_max; 114 | 115 | const large_max = static_config.large_max; 116 | 117 | const page_alignment = static_config.page_alignment; 118 | 119 | const small_size_classes = static_config.small_size_classes; 120 | const medium_size_classes = static_config.medium_size_classes; 121 | --------------------------------------------------------------------------------