├── LICENSE ├── README.md └── cachetop /LICENSE: -------------------------------------------------------------------------------- 1 | BSD 2-Clause License 2 | 3 | Copyright (c) 2024, Yuchen Zhang 4 | 5 | Redistribution and use in source and binary forms, with or without 6 | modification, are permitted provided that the following conditions are met: 7 | 8 | 1. Redistributions of source code must retain the above copyright notice, this 9 | list of conditions and the following disclaimer. 10 | 11 | 2. Redistributions in binary form must reproduce the above copyright notice, 12 | this list of conditions and the following disclaimer in the documentation 13 | and/or other materials provided with the distribution. 14 | 15 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 16 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 17 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 18 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 19 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 20 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 21 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 22 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 23 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 24 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 25 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # cachetop 2 | 3 | > Author: Xin Shuichen 4 | > support python2 and python3 5 | 6 | ## Description 7 | 8 | `cachetop` is a program that displays the top N files occupying the most cache in the specified path (default is the root directory '/'). It provides insights into cache usage by individual files. 9 | 10 | ## Advantages && Principle 11 | 12 | - **Low memory usage**: The program consumes minimal memory, not exceeding 20M RSS on the author's machine, regardless of any metric changes. 13 | - **Cache coverage**: It displays page cache even if it is not associated with a process. 14 | - **Selective file system support**: The program excludes all virtual file systems and currently only reads ext4 and tmpfs file systems. 15 | 16 | ### Principle and Mechanism 17 | The core principle of this program is based on the mincore system call, which is inspired by `hcache` [https://github.com/djhuahao/hcache]. 18 | The mincore system call allows checking whether each page in a memory segment is cached or not. To utilize this, the file needs to be mapped using mmap, and then the mincore is called with the mmap's address space to obtain the cache information. 19 | 20 | ### Reasons for Not Using Existing Tools 21 | The reason for not directly using tools like `hcache` or `vmtouch`, which are based on `pcstat` [https://github.com/tobert/pcstat], is that these tools obtain the filenames that need cache information from /proc/pid/maps and /proc/pid/fd. 22 | 23 | In Linux kernel, if a process opens a file, reads/writes to it, and then closes the file, all the caches of this file will not be reclaimed even if no one else is using the file. These caches become orphaned and cannot be found under the /proc directory. Therefore, the cache captured by tools like hcache and vmtouch may differ significantly from the cache shown by the free tool. 24 | 25 | To obtain all the cache data, it is necessary to traverse all system files and capture cache data for each file. During the capturing process, virtual file systems like procfs and sysfs do not need to be queried. This program uses a whitelist approach, and currently, it only queries data under ext4 and tmpfs mount points by default. 26 | 27 | An experiment, files in tmpfs are directly stored in the cache. You can try creating a 1GB file in tmpfs with `dd if=/dev/zero of=output.bin bs=1G count=1`, and you will see that the buff/cache in free increases by 1GB. 28 | 29 | image 30 | 31 | **At this point, this file cannot be seen using hcache. But It can be seen using cachetop.** 32 | 33 | hcache: 34 | 35 | image 36 | 37 | cachetop: 38 | 39 | image 40 | 41 | ### Optimizations 42 | 43 | The following optimizations have been made to achieve O(1) space complexity: 44 | 45 | 1. The core scenario for using cache-like tools is to see the top n caches, and it is unnecessary to cache all the data. Therefore, a min-heap is used to complete the top n selection. The size of the top n records in memory remains constant, regardless of the number of files. 46 | 2. The vec is read in segments, with a maximum of 10G per read, corresponding to approximately 2M memory (10G / 4096 = 2560KB). This way, the memory consumption is not related to the file size, and the memory usage of the entire program is fixed. 47 | 48 | The following optimizations have been made to reduce unnecessary reads: 49 | 1. For files that are already smaller than the current top n cache size, they are directly skipped because their cache cannot be larger than the top n files. During testing, this optimization reduced unnecessary mincore operations by an average of **97%**. 50 | 51 | ### Considerations and Performance: 52 | 53 | It is important to note that because this program reads all files, it is impossible to avoid the growth of dentry and inode. However, these are reclaimable slab objects and will not have a significant impact. 54 | 55 | On the author's PC, after dropping the cache, reading all **710,000** files takes approximately **20** seconds (`6.37s user 11.12s system 98% cpu 17.694 total`), with the actual number of mincore operations being **20,000**. 56 | 57 | The dentry and inode cache grew by **1G** during this collection. This should have a linear relationship with the number of files. 58 | 59 | Using the command `watch -n 1 "cat /proc/$(ps -ef | grep cachetop | grep -v grep | awk '{print $2}')/status | grep RSS"`, it can be observed that the memory usage never exceeds **15M**. 60 | 61 | Test Summary: 62 | ``` 63 | File scanned: 710,000 (depend on env) 64 | Mincore executed: 20,000 (depend on env) 65 | Time Spent: 17.694s (depend on scan file num) 66 | Memory Max: 15MB (fixed value) 67 | Dentry & Inode Cache Grew: 1G (depend on scan file num) 68 | ``` 69 | 70 | ## Usage 71 | 72 | ``` 73 | usage: cachetop [-h] [--topn TOPN] [--include_fs INCLUDE_FS] [--exclude_dir EXCLUDE_DIR] [search_path] 74 | 75 | positional arguments: 76 | search_path The directory path to search for files (default: '/') 77 | 78 | optional arguments: 79 | -h, --help show this help message and exit 80 | --topn TOPN, -n topn Print the top N files occupying the most cache (default: 10) 81 | --include_fs INCLUDE_FS, -f INCLUDE_FS Comma-separated list of file systems to include (default: 'ext4,tmpfs') 82 | --exclude_dir EXCLUDE_DIR, -e EXCLUDE_DIR Pipe-separated list of directories to exclude (e.g., '/test|/test1') 83 | ``` 84 | 85 | ## Example 86 | 87 | 1. Search all files under the root directory and print the top 5 files occupying the most cache: 88 | ``` 89 | cachetop -n 5 90 | ``` 91 | 92 | 2. Search all files under the /test directory and print the top 5 files occupying the most cache: 93 | ``` 94 | cachetop /test -n 5 95 | ``` 96 | 97 | 3. Search all files under the root directory, print the top 10 files, and include nfs, bcachefs, and tmpfs file systems: 98 | ``` 99 | cachetop -n 5 --include_fs "nfs,bcachefs,tmpfs" 100 | ``` 101 | 102 | 4. Search files under the root directory, excluding files in the /root and /home directories: 103 | ``` 104 | cachetop --exclude_dir "/root|/home" 105 | ``` 106 | 107 | ## Example Result 108 | 109 | ```shell 110 | $ time ./cachetop -n 20 111 | FILE NAME | CACHE SIZE | FILE SIZE 112 | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 113 | /usr/lib/sysimage/rpm/rpmdb.sqlite | 170.86 MB | 174.05 MB 114 | /var/cache/dnf/fedora-filenames.solvx | 53.64 MB | 53.66 MB 115 | /var/cache/dnf/fedora-376ef8e983c65ce0/repodata/5f86dfe1903316d6b25d494616cf18352204fe70c529b4c97859df6faecd493f-filelists.xml.zck | 49.40 MB | 49.40 MB 116 | /usr/lib64/libclang-cpp.so.16 | 45.96 MB | 66.50 MB 117 | /usr/lib64/libLLVM-16.so | 42.55 MB | 117.79 MB 118 | /var/cache/dnf/updates-filenames.solvx | 34.77 MB | 34.77 MB 119 | /var/cache/dnf/fedora-376ef8e983c65ce0/repodata/dfa2aec8d2e83459677d698542f1a190d92815888668d027b21dfd46fb86ce01-primary.xml.zck | 31.68 MB | 31.68 MB 120 | /var/log/journal/4483974d999a4003bae5281e394062bb/system.journal | 31.02 MB | 32.00 MB 121 | /var/cache/dnf/updates-b7ba662710b98f1a/repodata/f726172e0c8e64f0d991172a3c5945a24cf8d79d315e47e27688f5bc01511b24-filelists.xml.zck | 30.09 MB | 30.09 MB 122 | /root/anaconda3/libexec/gcc/x86_64-conda-linux-gnu/11.2.0/cc1 | 29.62 MB | 29.62 MB 123 | /root/anaconda3/pkgs/gcc_impl_linux-64-11.2.0-h1234567_1/libexec/gcc/x86_64-conda-linux-gnu/11.2.0/cc1 | 29.62 MB | 29.62 MB 124 | /usr/lib/locale/locale-archive | 25.98 MB | 213.97 MB 125 | /usr/lib/locale/locale-archive.real | 25.98 MB | 213.97 MB 126 | /var/cache/dnf/fedora.solv | 25.49 MB | 25.99 MB 127 | /var/lib/docker/overlay2/469328597deff3ff11e139b1b279dd15f802ec66c27984be99877e833bec407f/diff/coze-discord-proxy | 18.50 MB | 28.84 MB 128 | /root/jellyfin/config/data/library.db | 17.88 MB | 17.88 MB 129 | /var/cache/dnf/updates-updateinfo.solvx | 17.47 MB | 17.47 MB 130 | /usr/lib64/libcuda.so.535.146.02 | 17.37 MB | 28.02 MB 131 | /root/.vscode-server/data/CachedExtensionVSIXs/ms-python.vscode-pylance-2024.3.1 | 17.31 MB | 17.31 MB 132 | /var/lib/plocate/plocate.db | 14.47 MB | 14.47 MB 133 | ./cachetop -n 20 5.43s user 6.26s system 98% cpu 11.814 total 134 | ``` 135 | 136 | -------------------------------------------------------------------------------- /cachetop: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | """ 4 | Copyright (c) 2024 Xin Shuichen 5 | 6 | Permission is hereby granted, free of charge, to any person obtaining a copy 7 | of this software and associated documentation files (the "Software"), to deal 8 | in the Software without restriction, including without limitation the rights 9 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 10 | copies of the Software, and to permit persons to whom the Software is 11 | furnished to do so, subject to the following conditions: 12 | 13 | The above copyright notice and this permission notice shall be included in all 14 | copies or substantial portions of the Software. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 17 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 18 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 19 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 20 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 21 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 22 | SOFTWARE. 23 | 24 | Distributed according to the BSD-2-Clause 25 | """ 26 | 27 | 28 | epilog = """ 29 | Examples: 30 | Search all files under the root directory and print the top 5 files occupying the most cache: 31 | cachetop -n 5 32 | 33 | Search all files under the /test directory and print the top 5 files occupying the most cache: 34 | cachetop /test -n 5 35 | 36 | Search all files under the root directory, print the top 10 files, and include nfs, bcachefs, and tmpfs file systems: 37 | cachetop -n 5 --include_fs "nfs,bcachefs,tmpfs" 38 | 39 | Search files under the root directory, excluding files in the /root and /home directories: 40 | cachetop --exclude_dir "/root|/home" 41 | 42 | Advantages: 43 | - Low memory usage, not exceeding 20M RSS on the author's machine. Memory usage does not grow with the num of files to be scanned. 44 | - Displays page cache even if it is not associated with a process. 45 | - Excludes all virtual file systems, currently only reads ext4 and tmpfs file systems. 46 | """ 47 | 48 | import os 49 | import ctypes 50 | import mmap 51 | import heapq 52 | import argparse 53 | 54 | PAGE_SIZE = os.sysconf("SC_PAGESIZE") 55 | libc = ctypes.CDLL("libc.so.6") 56 | vec_length = 10 * 1024 * 1024 * 1024 57 | vec = ( 58 | # 10 GB / 4096 = 2560 KB 59 | ctypes.c_char 60 | * (vec_length // PAGE_SIZE) 61 | )() 62 | 63 | 64 | class MountInfo: 65 | def __init__(self, search_type, exclude_dir=[]): 66 | self.search_type = search_type 67 | self.exclude_dir = exclude_dir 68 | self.search_dir = [] 69 | self.update_mount_info() 70 | 71 | def update_mount_info(self): 72 | with open("/proc/mounts", "r") as f: 73 | mount_info = f.readlines() 74 | 75 | for line in mount_info: 76 | mount_info = line.split() 77 | path, type = mount_info[1], mount_info[2] 78 | if type in self.search_type: 79 | self.search_dir.append(path) 80 | else: 81 | self.exclude_dir.append(path) 82 | 83 | 84 | class SortedQueue: 85 | def __init__(self, max_size): 86 | self.heap = [] 87 | self.max_size = max_size 88 | 89 | def push(self, item_pair): 90 | heapq.heappush(self.heap, list(item_pair)) 91 | if len(self.heap) > self.max_size: 92 | self.pop() 93 | 94 | def pop(self): 95 | if self.heap: 96 | return tuple(heapq.heappop(self.heap)) 97 | else: 98 | raise IndexError("pop from an empty queue") 99 | 100 | def peek(self): 101 | if self.heap: 102 | return tuple(self.heap[0]) 103 | else: 104 | return None 105 | 106 | def walk_all_elements(self): 107 | for item in self.heap: 108 | yield item 109 | 110 | 111 | def bytes_to_readable(size, decimal_places=2): 112 | suffixes = ["B", "KB", "MB", "GB", "TB", "PB"] 113 | index = 0 114 | 115 | while size >= 1024 and index < len(suffixes) - 1: 116 | size /= 1024.0 117 | index += 1 118 | 119 | return "{0:.{1}f} {2}".format(size, decimal_places, suffixes[index]) 120 | 121 | 122 | def walk_files(dir="/", exclude_dir=[]): 123 | exclude_dir = [os.path.abspath(d) for d in exclude_dir] 124 | 125 | for root, dirs, files in os.walk(dir): 126 | dirs[:] = [ 127 | d for d in dirs if os.path.abspath(os.path.join(root, d)) not in exclude_dir 128 | ] 129 | 130 | for filename in files: 131 | file_path = os.path.join(root, filename) 132 | if not os.path.islink(file_path): 133 | yield file_path 134 | 135 | 136 | def get_cached_pages_size(fd, file_size): 137 | cached_pages = 0 138 | mmap_addr = mmap.mmap(fd, file_size, mmap.MAP_PRIVATE | mmap.ACCESS_READ) 139 | 140 | res = ctypes.c_uint8.from_buffer(mmap_addr) 141 | addr = ctypes.byref(res) 142 | addr_ptr_val = ctypes.cast(addr, ctypes.c_void_p).value 143 | 144 | remaining_size = file_size 145 | while remaining_size > 0: 146 | segment_size = min(remaining_size, vec_length) 147 | result = libc.mincore( 148 | ctypes.c_void_p(addr_ptr_val), 149 | ctypes.c_size_t(segment_size), 150 | ctypes.byref(vec), 151 | ) 152 | if result == -1: 153 | errno = ctypes.get_errno() 154 | raise OSError(errno, "mincore failed with error code {0}".format(errno)) 155 | 156 | cached_pages += sum( 157 | bytearray( 158 | vec[: segment_size // PAGE_SIZE + (segment_size % PAGE_SIZE != 0)] 159 | ) 160 | ) 161 | remaining_size -= segment_size 162 | addr_ptr_val += segment_size 163 | 164 | del res, addr 165 | 166 | mmap_addr.close() 167 | 168 | return cached_pages * PAGE_SIZE 169 | 170 | 171 | def main(mount_info, queue, search_path): 172 | queue.push([0, ""]) 173 | count, mincore_count = 0, 0 174 | for file in walk_files(search_path, mount_info.exclude_dir): 175 | count += 1 176 | try: 177 | fd = os.open(file, os.O_RDWR | os.O_EXCL) 178 | except: 179 | continue 180 | 181 | file_size = os.fstat(fd).st_size 182 | if file_size == 0 or file_size < queue.peek()[0]: 183 | os.close(fd) 184 | continue 185 | 186 | try: 187 | mincore_count += 1 188 | cached_size = get_cached_pages_size(fd, file_size) 189 | queue.push((cached_size, file)) 190 | except Exception as e: 191 | print( 192 | "Error: {0} {1} file_size: {2} file_path: {3}".format( 193 | e, type(queue.peek()[0]), file_size, file 194 | ) 195 | ) 196 | finally: 197 | os.close(fd) 198 | 199 | max_len = 0 200 | res = [] 201 | for item in queue.walk_all_elements(): 202 | max_len = max(max_len, len(item[1])) 203 | res.append(item) 204 | 205 | res.sort(key=lambda x: x[0], reverse=True) 206 | 207 | print("Process {0} files. Mincore {1} files.".format(count, mincore_count)) 208 | 209 | print("|{0}---{1}---{1}|".format("-" * max_len, "-" * 20)) 210 | print( 211 | "|{:<{}} | {:<20} | {:<20}|".format( 212 | "FILE NAME", max_len, "CACHE SIZE", "FILE SIZE" 213 | ) 214 | ) 215 | print("|{0}---{1}---{1}|".format("-" * max_len, "-" * 20)) 216 | for item in res: 217 | print( 218 | "|{:<{}} | {:<20} | {:<20}|".format( 219 | item[1], 220 | max_len, 221 | bytes_to_readable(item[0]), 222 | bytes_to_readable(os.path.getsize(item[1])), 223 | ) 224 | ) 225 | print("|{0}---{1}---{1}|".format("-" * max_len, "-" * 20)) 226 | 227 | 228 | if __name__ == "__main__": 229 | 230 | parser = argparse.ArgumentParser( 231 | description="Display the top N files occupying the most cache in the specified path (default is root directory '/').", 232 | epilog=epilog, 233 | formatter_class=argparse.RawTextHelpFormatter, 234 | ) 235 | 236 | parser.add_argument( 237 | "search_path", 238 | nargs="?", 239 | default="/", 240 | help="The directory path to search for files (default: '/')", 241 | ) 242 | parser.add_argument( 243 | "--topn", 244 | "-n", 245 | type=int, 246 | default=10, 247 | help="Print the top N files occupying the most cache (default: 10)", 248 | ) 249 | parser.add_argument( 250 | "--include_fs", 251 | "-f", 252 | type=str, 253 | default="ext4,tmpfs", 254 | help="Comma-separated list of file systems to include (default: 'ext4,tmpfs')", 255 | ) 256 | parser.add_argument( 257 | "--exclude_dir", 258 | "-e", 259 | type=str, 260 | help="Pipe-separated list of directories to exclude (e.g., '/test|/test1')", 261 | ) 262 | 263 | args = parser.parse_args() 264 | 265 | include_fs = args.include_fs.split(",") 266 | exclude_dir = args.exclude_dir.split("|") if args.exclude_dir else [] 267 | mount_info = MountInfo(include_fs, exclude_dir) 268 | 269 | topn = args.topn 270 | queue = SortedQueue(topn) 271 | 272 | search_path = args.search_path 273 | main(mount_info, queue, search_path) 274 | --------------------------------------------------------------------------------