mmap
1.1.0Portable mmap (file memory mapping) utility library.
Table of Contents
About MMAP
This is a utility library providing access to the mmap family of functions in a portable way. It should work on Posix and Windows systems. mmap allows you to directly map a file into the address space of your process without having to manually read it into memory sequentially. Typically this is much more efficient for files that are larger than a few Kb.
Supported operations
The library offers access to the following functions:
It also provides a convenience macro called with-mmap to perform safe, local mappings of files.
(mmap:with-mmap (addr fd size #p"/etc/lsb-release")
2 | (with-output-to-string (out)
3 | (loop for i from 0 below size
4 | for char = (code-char (cffi:mem-aref addr :char i))
5 | do (write-char char out))))
6 | If you're on a system where mmap is supported, :mmap will be in *features*.
System Information
Definition Index
-
MMAP
- ORG.SHIRAKUMO.FRAF.TRIAL.MMAP
No documentation provided.-
EXTERNAL CONDITION MMAP-ERROR
Error signalled if the mmap attempt fails for some reason. 7 | 8 | Possible reasons include, but are not limited to: 9 | - File not found 10 | - File access denied 11 | - Out of memory 12 | - Out of address space 13 | - Mapping not allowed 14 | - Invalid combination of flags 15 | 16 | See MMAP 17 | See CODE 18 | See MESSAGE
-
EXTERNAL FUNCTION MADVISE
- ADDR
- SIZE
- ADVICE
Gives hints about the usage patterns of the memory to better tune mapping behaviour. 19 | 20 | The values passed to this function must be the ones retrieved from a call 21 | to MMAP. 22 | 23 | The following advice hints are supported: 24 | 25 | :NORMAL --- [POSIX] This is the default. 26 | :SEQUENTIAL --- [POSIX] Expect memory to be addressed sequentially. 27 | :RANDOM --- [POSIX] Expect memory to be addressed randomly. 28 | :WILL-NEED --- [POSIX] Expect the memory to be used very soon. 29 | :DONT-NEED --- [POSIX] Expect the memory to not be needed any 30 | time soon. This will most likely cause 31 | pages to be offloaded until they are 32 | accessed again. 33 | :FREE --- [LINUX] The pages in the specified range are no 34 | longer needed and can be freed at any 35 | time, for instance to make space in case 36 | of memory pressure. 37 | :REMOVE --- [LINUX] Free the given pages and the associated 38 | backing store. 39 | :DONT-FORK --- [LINUX] Don't make changes available in children. 40 | :DO-FORK --- [LINUX] Undo :DONT-FORK behaviour. 41 | :MERGEABLE --- [LINUX] The pages in the specified range may be 42 | merged with ones with identical content. 43 | :UNMERGEABLE --- [LINUX] Undo :MERGEABLE behaviour. 44 | :HUGE-PAGE --- [LINUX] Enable transparent huge pages for the 45 | specified page range. 46 | :NO-HUGE-PAGE --- [LINUX] Ensure that the memory in the given 47 | range is not backed by transparent huge 48 | pages. 49 | :DONT-DUMP --- [LINUX] The pages in the specified range should 50 | be excluded from core dumps. 51 | :DO-DUMP --- [LINUX] Undo :DONT-DUMP behaviour. 52 | :WIPE-ON-FORK --- [LINUX] Memory in the given range is zeroed out 53 | for children. 54 | :KEEP-ON-FORK --- [LINUX] Undo :WIPE-ON-FORK behaviour. 55 | :COLD --- [LINUX] Deactivate the given range of 56 | pages. This makes them a more likely 57 | target for reclamation in the presence 58 | of memory pressure. 59 | :PAGEOUT --- [LINUX] The pages in the specified range should 60 | be reclaimed and their data flushed out. 61 | 62 | This function returns nothing useful. 63 | 64 | This function may signal an MMAP-ERROR in case the operating system notices 65 | a problem. 66 | 67 | See MMAP 68 | See MMAP-ERROR 69 | See https://pubs.opengroup.org/onlinepubs/007904875/functions/posix_madvise.html 70 | See https://man7.org/linux/man-pages/man2/madvise.2.html
-
EXTERNAL FUNCTION MMAP
- PATH
- &KEY
- OPEN
- PROTECTION
- MMAP
- SIZE
- OFFSET
Map the given path or number of bytes into the address space. 71 | 72 | PATH can be either a pathname designator, FD, or NIL. If it is NIL, an 73 | anonymous file is mapped and the MMAP flag list must include the flag 74 | :ANONYMOUS. If it is a path or an open POSIX file descriptor, then the 75 | contents of the given file on the file system are mapped into the 76 | address space. The file contents can then be read, written, or 77 | executed depending on the given flags as if normal memory was 78 | accessed. If the file is NIL or its size cannot be automatically 79 | determined, you must pass a valid SIZE. You may optionally pass an 80 | OFFSET (in bytes) into the file from which the mapping begins. 81 | 82 | [POSIX] PATH may also be the symbol :ANONYMOUS, in which case an anonymous 83 | file descriptor is created and returned for you. This can be useful when 84 | sharing a file descriptor with another process without needing to involve 85 | a disk-backed file. 86 | 87 | If the map attempt fails, an error of type MMAP-ERROR is signalled. 88 | If the call succeeds, three values are returned: 89 | 90 | PTR --- A CFFI:FOREIGN-POINTER that points to the start of the place in 91 | memory where the file contents have been mapped. The contents 92 | should be placed in increasing address order, unless the flag 93 | :GROWS-DOWN is active. 94 | FD --- An opaque file descriptor. You should not touch this. 95 | SIZE --- The size of the region of memory that has been mapped in bytes. 96 | 97 | All three values need to be passed on to MUNMAP completely unchanged. Any 98 | change could cause severe issues. 99 | 100 | The three options OPEN, PROTECTION, and MMAP are lists of flags. Not all of 101 | those flags are portable, some are only allowed on Linux, some only on non- 102 | Windows systems. To indicate support, the flags are marked as EVERY, POSIX 103 | (non-Windows), LINUX, or WINDOWS. 104 | 105 | OPEN 106 | :READ --- [EVERY] Opens the file for read access. 107 | :WRITE --- [EVERY] Opens the file for write access. 108 | :CREATE --- [EVERY] Creates the file if it does not exist yet. 109 | :ENSURE-CREATE --- [EVERY] Creates the file if it does not exist yet and 110 | errors if it does. 111 | :TRUNCATE --- [EVERY] Truncates the file and replaces it if it exists. 112 | :DIRECT --- [EVERY] Causes system buffers to be bypassed. 113 | :FILE-SYNC --- [EVERY] Causes writes to the file to be flushed asap. 114 | :DATA-SYNC --- [POSIX] Similar to FILE-SYNC, but uses data integrity 115 | semantics rather than file integrity semantics. 116 | :DONT-CLAIM-TTY--- [POSIX] If the file is a tty and the process does not 117 | already have a controlling tty, this file will 118 | not become the process' controlling tty. 119 | :NON-BLOCK --- [POSIX] Attempt to open the file in non-blocking mode, 120 | causing operations on the fd to return asap. 121 | :NO-FOLLOW --- [LINUX] Errors if the file is a symlink. 122 | :ASYNC --- [LINUX] Enable signal driven IO. 123 | :DIRECTORY --- [LINUX] Errors if the file is not a directory. 124 | :LARGE-FILE --- [LINUX] Allows opening files with size not representable 125 | by a 32 bit unsigned integer. 126 | 127 | PROTECTION 128 | :READ --- [EVERY] Allows reading from the memory region. The OPEN 129 | flag :READ is required for this protection mode. 130 | This flag is required on windows. 131 | :WRITE --- [EVERY] Allows writing to the memory region. 132 | :EXEC --- [EVERY] Allows executing code in the memory region. 133 | :NONE --- [POSIX] Prevents accessing the memory region. 134 | 135 | MMAP 136 | :PRIVATE --- [EVERY] The underlying file is not changed if the memory 137 | area is written to. Copy-on-write is employed to 138 | ensure separation. 139 | :SHARED --- [EVERY] The underlying file is changed if the memory 140 | area is written to and the change will be 141 | visible to other processes. In this case the 142 | OPEN flag :WRITE must be specified. 143 | :ANONYMOUS --- [LINUX/WINDOWS] The path should be a number of bytes to 144 | map to memory. The memory region is then mapped 145 | against an "anonymous" file. 146 | :NO-RESERVE --- [LINUX] Don't reserve swap for this mapping. If memory 147 | runs out, a segfault will be generated instead. 148 | :LOCKED --- [LINUX] Locks the region to RAM, preventing it from 149 | being swapped out. 150 | :GROWS-DOWN --- [LINUX] Causes the memory region to be mapped with a 151 | decreasing address, like in a stack. 152 | :POPULATE --- [LINUX] Pre-populate the memory region with the file 153 | contents, which can help performance. 154 | :NON-BLOCK --- [LINUX] Only useful with :POPULATE -- do not perform a 155 | read-ahead. 156 | 157 | The default values for the flags are: 158 | :OPEN (:READ) :PROTECTION (:READ) :MMAP (:PRIVATE) 159 | 160 | Note that if you are intending to use MPROTECT to change the protection of 161 | the mapped file at a later date, you need to call MMAP with the maximal 162 | combination of protection flags first. If this is not the protection that 163 | you want to start out with, call MPROTECT with the correct combination 164 | immediately after. For instance, if you would like to start with (:READ) and 165 | later want to change it to (:READ :WRITE), call MMAP with (:READ :WRITE), 166 | and immediately after call MPROTECT with (:READ). 167 | 168 | See MUNMAP 169 | See WITH-MMAP 170 | See MMAP-ERROR 171 | See http://pubs.opengroup.org/onlinepubs/7908799/xsh/mmap.html 172 | See http://pubs.opengroup.org/onlinepubs/009604499/functions/stat.html 173 | See http://man7.org/linux/man-pages/man2/mmap.2.html 174 | See http://man7.org/linux/man-pages/man2/stat.2.html 175 | See https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-createfilew 176 | See https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-getfilesize 177 | See https://docs.microsoft.com/en-us/windows/desktop/api/winbase/nf-winbase-createfilemappinga 178 | See https://msdn.microsoft.com/en-us/library/windows/desktop/aa366761(v=vs.85).aspx
-
EXTERNAL FUNCTION MPROTECT
- ADDR
- SIZE
- PROTECTION
Changes the access protection of the mapped memory region. 179 | 180 | The values passed to this function must be the ones retrieved from a call 181 | to MMAP. 182 | 183 | The following protection flags are supported: 184 | 185 | :READ --- [EVERY] Allows reading from the memory region. The OPEN 186 | flag :READ is required for this protection mode. 187 | This flag is required on windows. 188 | :WRITE --- [EVERY] Allows writing to the memory region. 189 | :EXEC --- [EVERY] Allows executing code in the memory region. 190 | :NONE --- [POSIX] Prevents accessing the memory region. 191 | 192 | This function returns nothing useful. 193 | 194 | This function may signal an MMAP-ERROR in case the operating system notices 195 | a problem. 196 | 197 | See MMAP 198 | See MMAP-ERROR 199 | See http://pubs.opengroup.org/onlinepubs/9699919799/functions/mprotect.html 200 | See http://man7.org/linux/man-pages/man2/mprotect.2.html 201 | See https://msdn.microsoft.com/en-us/library/windows/desktop/aa366898(v=vs.85).aspx
-
EXTERNAL FUNCTION MSYNC
- ADDR
- FD
- SIZE
- &KEY
- FLAGS
Causes writes to the mapped file area to be written to disk. 202 | 203 | The values passed to this function must be the ones retrieved from a call 204 | to MMAP. 205 | 206 | The following flags are supported: 207 | 208 | :SYNC --- [EVERY] Writing is synchronous. A call to this function 209 | will not return until the data is flushed to 210 | disk. 211 | :ASYNC --- [EVERY] Writing is asynchronous and a call will return 212 | immediately. 213 | :INVALIDATE --- [POSIX] Asks to invalidate other mappings of the same 214 | file, ensuring the view is synchronised. 215 | 216 | This function returns nothing useful. 217 | 218 | This function may signal an MMAP-ERROR in case the operating system notices 219 | a problem. 220 | 221 | See MMAP 222 | See MMAP-ERROR 223 | See http://pubs.opengroup.org/onlinepubs/000095399/functions/msync.html 224 | See http://man7.org/linux/man-pages/man2/msync.2.html 225 | See https://msdn.microsoft.com/en-us/library/windows/desktop/aa366563(v=vs.85).aspx 226 | See https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-flushfilebuffers
-
EXTERNAL FUNCTION MUNMAP
- ADDR
- FD
- SIZE
Unmaps the memory region, freeing the address space and its file. 227 | 228 | The values passed to this function must be the ones retrieved from a call 229 | to MMAP. Calling MUNMAP with the same values more than once will lead to 230 | undefined consequences and may very well corrupt your system to crash. The 231 | same goes for calling MUNMAP with values not directly returned by MMAP, 232 | calling it with changed values returned by MMAP, or attempting to 233 | dereference the PTR after a call to MUNMAP. 234 | 235 | This function returns nothing useful. 236 | 237 | On POSIX systems you may pass NIL for the FD argument, in which case 238 | the file descriptor is not closed. It is then your responsibility to 239 | close it appropriately at a later point. 240 | 241 | This function may signal an MMAP-ERROR in case the operating system notices 242 | a problem. 243 | 244 | See MMAP 245 | See MMAP-ERROR 246 | See WITH-MMAP 247 | See http://pubs.opengroup.org/onlinepubs/9699919799/functions/mprotect.html 248 | See http://man7.org/linux/man-pages/man2/mprotect.2.html 249 | See https://msdn.microsoft.com/en-us/library/windows/desktop/aa366882(v=vs.85).aspx 250 | See https://msdn.microsoft.com/en-us/library/windows/desktop/ms724211(v=vs.85).aspx
-
EXTERNAL GENERIC-FUNCTION CODE
- CONDITION
The OS-specific error code returned for the mmap failure. 251 | 252 | See MMAP-ERROR
-
EXTERNAL GENERIC-FUNCTION MESSAGE
- CONDITION
The (hopefully) user-readable error message for the mmap failure. 253 | 254 | See MMAP-ERROR
-
EXTERNAL MACRO WITH-MMAP
- ADDR
- FD
- SIZE
- PATH
- &REST
- ARGS
- &KEY
- DONT-CLOSE
- &ALLOW-OTHER-KEYS
- &BODY
- BODY
Map the file or number of bytes to a memory region within the body. 255 | 256 | This is a convenience macro that calls MMAP with the given arguments, 257 | binds the results to the variables ADDR, FD, and SIZE, and automatically 258 | ensures that MUNMAP is called with the correct values when the body is 259 | exited. 260 | 261 | If the flag DONT-CLOSE is set, WITH-MMAP will not free the file 262 | descriptor on unwind. This is useful primarily if you pass in an FD 263 | for the path yourself and are either not responsible for closing it, 264 | or would like to continue using it for other purposes. 265 | 266 | It is safe to change the ADDR, FD, and SIZE bindings, though probably not 267 | very good style to do so. It is NOT safe to save the ADDR and SIZE values 268 | somewhere and use them outside of the dynamic scope of the body. Attempting 269 | to do so is very likely going to burn your process to the ground. 270 | 271 | See MMAP 272 | See MUNMAP