├── .gitignore ├── LICENSE ├── Makefile ├── README ├── compdb.c ├── compdb.h ├── compress.c ├── compress.h ├── debian ├── changelog ├── compat ├── control ├── copyright ├── docs ├── rules └── source │ └── format ├── e2mapper.1 ├── e2mapper.c ├── fatmapper.1 ├── fatmapper.c ├── fiemap.py ├── filemapper.1 ├── filemapper.c ├── filemapper.desktop.in ├── filemapper.h ├── filemapper.in ├── filemapper.png ├── filemapper.py ├── filemapper.ui ├── fmcli.py ├── fmdb.py ├── fmgui.py ├── getfsmap.py ├── ioctl.py ├── mkdoscode ├── ntfsmapper.1 ├── ntfsmapper.c ├── pymod.c ├── shrinkmapper.1 ├── shrinkmapper.c ├── vfs.py ├── xfsmapper.1 └── xfsmapper.c /.gitignore: -------------------------------------------------------------------------------- 1 | *.swp 2 | __pycache__ 3 | *.pyc 4 | e2mapper 5 | filemapper 6 | ntfsmapper 7 | debian/ 8 | *.1.gz 9 | *.desktop 10 | *.o 11 | dosfs.[ch] 12 | fatmapper 13 | xfsmapper 14 | fatcheck.c 15 | libfat.a 16 | db/ 17 | compdb.so 18 | shrinkmapper 19 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | GNU GENERAL PUBLIC LICENSE 2 | Version 2, June 1991 3 | 4 | Copyright (C) 1989, 1991 Free Software Foundation, Inc., 5 | 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA 6 | Everyone is permitted to copy and distribute verbatim copies 7 | of this license document, but changing it is not allowed. 8 | 9 | Preamble 10 | 11 | The licenses for most software are designed to take away your 12 | freedom to share and change it. By contrast, the GNU General Public 13 | License is intended to guarantee your freedom to share and change free 14 | software--to make sure the software is free for all its users. This 15 | General Public License applies to most of the Free Software 16 | Foundation's software and to any other program whose authors commit to 17 | using it. (Some other Free Software Foundation software is covered by 18 | the GNU Lesser General Public License instead.) You can apply it to 19 | your programs, too. 20 | 21 | When we speak of free software, we are referring to freedom, not 22 | price. Our General Public Licenses are designed to make sure that you 23 | have the freedom to distribute copies of free software (and charge for 24 | this service if you wish), that you receive source code or can get it 25 | if you want it, that you can change the software or use pieces of it 26 | in new free programs; and that you know you can do these things. 27 | 28 | To protect your rights, we need to make restrictions that forbid 29 | anyone to deny you these rights or to ask you to surrender the rights. 30 | These restrictions translate to certain responsibilities for you if you 31 | distribute copies of the software, or if you modify it. 32 | 33 | For example, if you distribute copies of such a program, whether 34 | gratis or for a fee, you must give the recipients all the rights that 35 | you have. You must make sure that they, too, receive or can get the 36 | source code. And you must show them these terms so they know their 37 | rights. 38 | 39 | We protect your rights with two steps: (1) copyright the software, and 40 | (2) offer you this license which gives you legal permission to copy, 41 | distribute and/or modify the software. 42 | 43 | Also, for each author's protection and ours, we want to make certain 44 | that everyone understands that there is no warranty for this free 45 | software. If the software is modified by someone else and passed on, we 46 | want its recipients to know that what they have is not the original, so 47 | that any problems introduced by others will not reflect on the original 48 | authors' reputations. 49 | 50 | Finally, any free program is threatened constantly by software 51 | patents. We wish to avoid the danger that redistributors of a free 52 | program will individually obtain patent licenses, in effect making the 53 | program proprietary. To prevent this, we have made it clear that any 54 | patent must be licensed for everyone's free use or not licensed at all. 55 | 56 | The precise terms and conditions for copying, distribution and 57 | modification follow. 58 | 59 | GNU GENERAL PUBLIC LICENSE 60 | TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 61 | 62 | 0. This License applies to any program or other work which contains 63 | a notice placed by the copyright holder saying it may be distributed 64 | under the terms of this General Public License. The "Program", below, 65 | refers to any such program or work, and a "work based on the Program" 66 | means either the Program or any derivative work under copyright law: 67 | that is to say, a work containing the Program or a portion of it, 68 | either verbatim or with modifications and/or translated into another 69 | language. (Hereinafter, translation is included without limitation in 70 | the term "modification".) Each licensee is addressed as "you". 71 | 72 | Activities other than copying, distribution and modification are not 73 | covered by this License; they are outside its scope. The act of 74 | running the Program is not restricted, and the output from the Program 75 | is covered only if its contents constitute a work based on the 76 | Program (independent of having been made by running the Program). 77 | Whether that is true depends on what the Program does. 78 | 79 | 1. You may copy and distribute verbatim copies of the Program's 80 | source code as you receive it, in any medium, provided that you 81 | conspicuously and appropriately publish on each copy an appropriate 82 | copyright notice and disclaimer of warranty; keep intact all the 83 | notices that refer to this License and to the absence of any warranty; 84 | and give any other recipients of the Program a copy of this License 85 | along with the Program. 86 | 87 | You may charge a fee for the physical act of transferring a copy, and 88 | you may at your option offer warranty protection in exchange for a fee. 89 | 90 | 2. You may modify your copy or copies of the Program or any portion 91 | of it, thus forming a work based on the Program, and copy and 92 | distribute such modifications or work under the terms of Section 1 93 | above, provided that you also meet all of these conditions: 94 | 95 | a) You must cause the modified files to carry prominent notices 96 | stating that you changed the files and the date of any change. 97 | 98 | b) You must cause any work that you distribute or publish, that in 99 | whole or in part contains or is derived from the Program or any 100 | part thereof, to be licensed as a whole at no charge to all third 101 | parties under the terms of this License. 102 | 103 | c) If the modified program normally reads commands interactively 104 | when run, you must cause it, when started running for such 105 | interactive use in the most ordinary way, to print or display an 106 | announcement including an appropriate copyright notice and a 107 | notice that there is no warranty (or else, saying that you provide 108 | a warranty) and that users may redistribute the program under 109 | these conditions, and telling the user how to view a copy of this 110 | License. (Exception: if the Program itself is interactive but 111 | does not normally print such an announcement, your work based on 112 | the Program is not required to print an announcement.) 113 | 114 | These requirements apply to the modified work as a whole. If 115 | identifiable sections of that work are not derived from the Program, 116 | and can be reasonably considered independent and separate works in 117 | themselves, then this License, and its terms, do not apply to those 118 | sections when you distribute them as separate works. But when you 119 | distribute the same sections as part of a whole which is a work based 120 | on the Program, the distribution of the whole must be on the terms of 121 | this License, whose permissions for other licensees extend to the 122 | entire whole, and thus to each and every part regardless of who wrote it. 123 | 124 | Thus, it is not the intent of this section to claim rights or contest 125 | your rights to work written entirely by you; rather, the intent is to 126 | exercise the right to control the distribution of derivative or 127 | collective works based on the Program. 128 | 129 | In addition, mere aggregation of another work not based on the Program 130 | with the Program (or with a work based on the Program) on a volume of 131 | a storage or distribution medium does not bring the other work under 132 | the scope of this License. 133 | 134 | 3. You may copy and distribute the Program (or a work based on it, 135 | under Section 2) in object code or executable form under the terms of 136 | Sections 1 and 2 above provided that you also do one of the following: 137 | 138 | a) Accompany it with the complete corresponding machine-readable 139 | source code, which must be distributed under the terms of Sections 140 | 1 and 2 above on a medium customarily used for software interchange; or, 141 | 142 | b) Accompany it with a written offer, valid for at least three 143 | years, to give any third party, for a charge no more than your 144 | cost of physically performing source distribution, a complete 145 | machine-readable copy of the corresponding source code, to be 146 | distributed under the terms of Sections 1 and 2 above on a medium 147 | customarily used for software interchange; or, 148 | 149 | c) Accompany it with the information you received as to the offer 150 | to distribute corresponding source code. (This alternative is 151 | allowed only for noncommercial distribution and only if you 152 | received the program in object code or executable form with such 153 | an offer, in accord with Subsection b above.) 154 | 155 | The source code for a work means the preferred form of the work for 156 | making modifications to it. For an executable work, complete source 157 | code means all the source code for all modules it contains, plus any 158 | associated interface definition files, plus the scripts used to 159 | control compilation and installation of the executable. However, as a 160 | special exception, the source code distributed need not include 161 | anything that is normally distributed (in either source or binary 162 | form) with the major components (compiler, kernel, and so on) of the 163 | operating system on which the executable runs, unless that component 164 | itself accompanies the executable. 165 | 166 | If distribution of executable or object code is made by offering 167 | access to copy from a designated place, then offering equivalent 168 | access to copy the source code from the same place counts as 169 | distribution of the source code, even though third parties are not 170 | compelled to copy the source along with the object code. 171 | 172 | 4. You may not copy, modify, sublicense, or distribute the Program 173 | except as expressly provided under this License. Any attempt 174 | otherwise to copy, modify, sublicense or distribute the Program is 175 | void, and will automatically terminate your rights under this License. 176 | However, parties who have received copies, or rights, from you under 177 | this License will not have their licenses terminated so long as such 178 | parties remain in full compliance. 179 | 180 | 5. You are not required to accept this License, since you have not 181 | signed it. However, nothing else grants you permission to modify or 182 | distribute the Program or its derivative works. These actions are 183 | prohibited by law if you do not accept this License. Therefore, by 184 | modifying or distributing the Program (or any work based on the 185 | Program), you indicate your acceptance of this License to do so, and 186 | all its terms and conditions for copying, distributing or modifying 187 | the Program or works based on it. 188 | 189 | 6. Each time you redistribute the Program (or any work based on the 190 | Program), the recipient automatically receives a license from the 191 | original licensor to copy, distribute or modify the Program subject to 192 | these terms and conditions. You may not impose any further 193 | restrictions on the recipients' exercise of the rights granted herein. 194 | You are not responsible for enforcing compliance by third parties to 195 | this License. 196 | 197 | 7. If, as a consequence of a court judgment or allegation of patent 198 | infringement or for any other reason (not limited to patent issues), 199 | conditions are imposed on you (whether by court order, agreement or 200 | otherwise) that contradict the conditions of this License, they do not 201 | excuse you from the conditions of this License. If you cannot 202 | distribute so as to satisfy simultaneously your obligations under this 203 | License and any other pertinent obligations, then as a consequence you 204 | may not distribute the Program at all. For example, if a patent 205 | license would not permit royalty-free redistribution of the Program by 206 | all those who receive copies directly or indirectly through you, then 207 | the only way you could satisfy both it and this License would be to 208 | refrain entirely from distribution of the Program. 209 | 210 | If any portion of this section is held invalid or unenforceable under 211 | any particular circumstance, the balance of the section is intended to 212 | apply and the section as a whole is intended to apply in other 213 | circumstances. 214 | 215 | It is not the purpose of this section to induce you to infringe any 216 | patents or other property right claims or to contest validity of any 217 | such claims; this section has the sole purpose of protecting the 218 | integrity of the free software distribution system, which is 219 | implemented by public license practices. Many people have made 220 | generous contributions to the wide range of software distributed 221 | through that system in reliance on consistent application of that 222 | system; it is up to the author/donor to decide if he or she is willing 223 | to distribute software through any other system and a licensee cannot 224 | impose that choice. 225 | 226 | This section is intended to make thoroughly clear what is believed to 227 | be a consequence of the rest of this License. 228 | 229 | 8. If the distribution and/or use of the Program is restricted in 230 | certain countries either by patents or by copyrighted interfaces, the 231 | original copyright holder who places the Program under this License 232 | may add an explicit geographical distribution limitation excluding 233 | those countries, so that distribution is permitted only in or among 234 | countries not thus excluded. In such case, this License incorporates 235 | the limitation as if written in the body of this License. 236 | 237 | 9. The Free Software Foundation may publish revised and/or new versions 238 | of the General Public License from time to time. Such new versions will 239 | be similar in spirit to the present version, but may differ in detail to 240 | address new problems or concerns. 241 | 242 | Each version is given a distinguishing version number. If the Program 243 | specifies a version number of this License which applies to it and "any 244 | later version", you have the option of following the terms and conditions 245 | either of that version or of any later version published by the Free 246 | Software Foundation. If the Program does not specify a version number of 247 | this License, you may choose any version ever published by the Free Software 248 | Foundation. 249 | 250 | 10. If you wish to incorporate parts of the Program into other free 251 | programs whose distribution conditions are different, write to the author 252 | to ask for permission. For software which is copyrighted by the Free 253 | Software Foundation, write to the Free Software Foundation; we sometimes 254 | make exceptions for this. Our decision will be guided by the two goals 255 | of preserving the free status of all derivatives of our free software and 256 | of promoting the sharing and reuse of software generally. 257 | 258 | NO WARRANTY 259 | 260 | 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY 261 | FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN 262 | OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES 263 | PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED 264 | OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF 265 | MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS 266 | TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE 267 | PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, 268 | REPAIR OR CORRECTION. 269 | 270 | 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING 271 | WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR 272 | REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, 273 | INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING 274 | OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED 275 | TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY 276 | YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER 277 | PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE 278 | POSSIBILITY OF SUCH DAMAGES. 279 | 280 | END OF TERMS AND CONDITIONS 281 | 282 | How to Apply These Terms to Your New Programs 283 | 284 | If you develop a new program, and you want it to be of the greatest 285 | possible use to the public, the best way to achieve this is to make it 286 | free software which everyone can redistribute and change under these terms. 287 | 288 | To do so, attach the following notices to the program. It is safest 289 | to attach them to the start of each source file to most effectively 290 | convey the exclusion of warranty; and each file should have at least 291 | the "copyright" line and a pointer to where the full notice is found. 292 | 293 | {description} 294 | Copyright (C) {year} {fullname} 295 | 296 | This program is free software; you can redistribute it and/or modify 297 | it under the terms of the GNU General Public License as published by 298 | the Free Software Foundation; either version 2 of the License, or 299 | (at your option) any later version. 300 | 301 | This program is distributed in the hope that it will be useful, 302 | but WITHOUT ANY WARRANTY; without even the implied warranty of 303 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 304 | GNU General Public License for more details. 305 | 306 | You should have received a copy of the GNU General Public License along 307 | with this program; if not, write to the Free Software Foundation, Inc., 308 | 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. 309 | 310 | Also add information on how to contact you by electronic and paper mail. 311 | 312 | If the program is interactive, make it output a short notice like this 313 | when it starts in an interactive mode: 314 | 315 | Gnomovision version 69, Copyright (C) year name of author 316 | Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. 317 | This is free software, and you are welcome to redistribute it 318 | under certain conditions; type `show c' for details. 319 | 320 | The hypothetical commands `show w' and `show c' should show the appropriate 321 | parts of the General Public License. Of course, the commands you use may 322 | be called something other than `show w' and `show c'; they could even be 323 | mouse-clicks or menu items--whatever suits your program. 324 | 325 | You should also get your employer (if you work as a programmer) or your 326 | school, if any, to sign a "copyright disclaimer" for the program, if 327 | necessary. Here is a sample; alter the names: 328 | 329 | Yoyodyne, Inc., hereby disclaims all copyright interest in the program 330 | `Gnomovision' (which makes passes at compilers) written by James Hacker. 331 | 332 | {signature of Ty Coon}, 1 April 1989 333 | Ty Coon, President of Vice 334 | 335 | This General Public License does not permit incorporating your program into 336 | proprietary programs. If your program is a subroutine library, you may 337 | consider it more useful to permit linking proprietary applications with the 338 | library. If this is what you want to do, use the GNU Lesser General 339 | Public License instead of this License. 340 | 341 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | LTO=-flto 2 | CFLAGS=-Wall -O3 -g $(LTO) -std=gnu11 -fPIC 3 | LDFLAGS=-Wall -O3 -g $(LTO) -std=gnu11 -fPIC 4 | LIB_CFLAGS=-Wall -O3 -g -std=gnu11 -shared -fPIC 5 | VERSION=0.9.1 6 | 7 | prefix = /usr 8 | exec_prefix = ${prefix} 9 | bindir = ${exec_prefix}/bin 10 | libdir = ${exec_prefix}/lib 11 | fmlibdir = ${libdir}/filemapper 12 | mandir = ${exec_prefix}/man 13 | man1dir = ${mandir}/man1 14 | appdir = ${exec_prefix}/share/applications 15 | XFSPROGS ?= Please_set_XFSPROGS_to_the_XFS_source_directory 16 | DOSFSTOOLS ?= Please_set_DOSFSTOOLS_to_the_DOS_source_directory 17 | DOSFS_HEADERS=$(DOSFSTOOLS)/src/fsck.fat.h $(DOSFSTOOLS)/src/file.h $(DOSFSTOOLS)/src/fat.h $(DOSFSTOOLS)/src/lfn.h $(DOSFSTOOLS)/src/charconv.h $(DOSFSTOOLS)/src/boot.h $(DOSFSTOOLS)/src/common.h $(DOSFSTOOLS)/src/io.h 18 | PYINCLUDE ?= -I/usr/include/python3.5m/ 19 | COMPDB_LIBS=-lz -llz4 -llzma -lbz2 20 | pyfiles=fiemap.py filemapper.py fmcli.py fmdb.py fmgui.py getfsmap.py ioctl.py vfs.py 21 | 22 | ifeq ("$(notdir $(wildcard $(XFSPROGS)/libxfs/.libs/libxfs.a))", "libxfs.a") 23 | xfsmapper=xfsmapper 24 | endif 25 | ifeq ("$(notdir $(wildcard $(DOSFSTOOLS)/fat.o))", "fat.o") 26 | fatmapper=fatmapper 27 | endif 28 | 29 | progs=filemapper e2mapper ntfsmapper shrinkmapper $(xfsmapper) $(fatmapper) 30 | libs=compdb.so 31 | manpages=$(patsubst %,%.1.gz,$(progs)) 32 | 33 | all: $(progs) $(libs) $(manpages) $(pyfiles) filemapper.desktop 34 | 35 | %.1.gz: %.1 36 | gzip -9n < $< > $@ 37 | 38 | compdb.so: compdb.o compress.o pymod.o 39 | $(CC) $(LIB_CFLAGS) -o $@ $^ -lsqlite3 $(COMPDB_LIBS) 40 | 41 | pymod.o: pymod.c compdb.h compress.h filemapper.h 42 | $(CC) $(CFLAGS) -DPYMOD $(PYINCLUDE) -o $@ -c $< 43 | 44 | compdb.o: compdb.c compdb.h compress.h filemapper.h 45 | 46 | compress.o: compress.h 47 | 48 | filemapper.o: filemapper.h 49 | 50 | shrinkmapper: shrinkmapper.o compress.o compdb.o 51 | $(CC) $(CFLAGS) -o $@ $^ -lsqlite3 $(COMPDB_LIBS) 52 | 53 | shrinkmapper.o: compdb.h filemapper.h 54 | 55 | xfsmapper: filemapper.o xfsmapper.o compress.o compdb.o $(XFSPROGS)/libxfs/.libs/libxfs.a 56 | $(CC) $(CFLAGS) -o $@ $^ $(XFSPROGS)/repair/btree.o -lsqlite3 -lpthread -luuid -lm $(COMPDB_LIBS) 57 | 58 | xfsmapper.o: xfsmapper.c filemapper.h $(XFSPROGS)/include/libxfs.h $(XFSPROGS)/repair/btree.h $(XFSPROGS)/libxfs/libxfs_api_defs.h compdb.h 59 | $(CC) $(CFLAGS) -D_GNU_SOURCE -o $@ -c $< -I$(XFSPROGS)/include/ -I$(XFSPROGS)/libxfs/ -I$(XFSPROGS)/ 60 | 61 | e2mapper: filemapper.o e2mapper.o compress.o compdb.o 62 | $(CC) $(CFLAGS) -o $@ $^ -lsqlite3 -lcom_err -lext2fs -lm $(COMPDB_LIBS) 63 | 64 | e2mapper.o: e2mapper.c filemapper.h compdb.h 65 | 66 | ntfsmapper: filemapper.o ntfsmapper.o compress.o compdb.o 67 | $(CC) $(CFLAGS) -o $@ $^ -lsqlite3 -lntfs-3g -lm $(COMPDB_LIBS) 68 | 69 | ntfsmapper.o: ntfsmapper.c filemapper.h compdb.h 70 | 71 | libfat.a: $(DOSFSTOOLS)/boot.o $(DOSFSTOOLS)/charconv.o $(DOSFSTOOLS)/common.o $(DOSFSTOOLS)/fat.o $(DOSFSTOOLS)/file.o $(DOSFSTOOLS)/io.o $(DOSFSTOOLS)/lfn.o 72 | $(AR) cr libfat.a $^ 73 | 74 | fatmapper: filemapper.o fatmapper.o fatcheck.o libfat.a compress.o compdb.o 75 | $(CC) $(CFLAGS) -o $@ $^ -lsqlite3 -lm $(COMPDB_LIBS) 76 | 77 | fatcheck.c: $(DOSFSTOOLS)/src/check.c $(DOSFS_HEADERS) 78 | sed -e 's/static void add_file/void add_file/g' < $< > $@ 79 | 80 | fatcheck.o: fatcheck.c 81 | $(CC) $(CFLAGS) -o $@ -c $< -I$(DOSFSTOOLS)/src/ 82 | 83 | fatmapper.o: fatmapper.c filemapper.h $(DOSFS_HEADERS) compdb.h 84 | $(CC) $(CFLAGS) -o $@ -c $< -I$(DOSFSTOOLS)/src/ 85 | 86 | clean:; 87 | rm -rf $(progs) $(manpages) $(libs) xfsmapper xfsmapper.1.gz fatmapper fatmapper.1.gz libfat.a fatcheck.c *.pyc __pycache__ filemapper.desktop *.o 88 | 89 | filemapper: filemapper.in 90 | sed -e "s|%libdir%|${fmlibdir}|g" < $< > $@ 91 | 92 | filemapper.desktop: filemapper.desktop.in 93 | sed -e "s|%libdir%|${fmlibdir}|g" < $< > $@ 94 | 95 | install: all 96 | install -d $(DESTDIR)$(bindir) 97 | install -s e2mapper ntfsmapper shrinkmapper $(DESTDIR)$(bindir) 98 | install filemapper $(DESTDIR)$(bindir) 99 | install -d $(DESTDIR)$(fmlibdir) 100 | install -m 0644 $(pyfiles) $(DESTDIR)$(fmlibdir) 101 | install -s compdb.so $(DESTDIR)$(fmlibdir) 102 | install -m 0644 filemapper.png filemapper.ui $(DESTDIR)$(fmlibdir) 103 | install -d $(DESTDIR)$(man1dir) 104 | install -m 0644 e2mapper.1.gz filemapper.1.gz ntfsmapper.1.gz shrinkmapper.1.gz $(DESTDIR)$(man1dir) 105 | install -d $(DESTDIR)$(appdir) 106 | install -m 0644 filemapper.desktop $(DESTDIR)$(appdir) 107 | -test -e fatmapper && install -s fatmapper $(DESTDIR)$(bindir) 108 | -test -e fatmapper && install -m 0644 fatmapper.1.gz $(DESTDIR)$(man1dir) 109 | -test -e xfsmapper && install -s xfsmapper $(DESTDIR)$(bindir) 110 | -test -e xfsmapper && install -m 0644 xfsmapper.1.gz $(DESTDIR)$(man1dir) 111 | 112 | dist: 113 | @if test "`git describe`" != "$(VERSION)" ; then \ 114 | echo 'Update VERSION in the Makefile before running "make dist".' ; \ 115 | exit 1 ; \ 116 | fi 117 | git archive --format=tar --prefix=filemapper-$(VERSION)/ HEAD^{tree} | xz -9 > ../filemapper_$(VERSION).orig.tar.xz 118 | -------------------------------------------------------------------------------- /README: -------------------------------------------------------------------------------- 1 | FileMapper: Draw pictures of the physical layout of a file system. 2 | 3 | This tool can walk every file, directory, metadata object, and symbolic link in 4 | a file system, record the physical locations of everything it finds, and 5 | draw a picture of where everything is located. 6 | 7 | The analyzer tool can use the FIEMAP or FIBMAP ioctls on a mounted file system 8 | to walk the directory tree looking for the data block mappings of files and 9 | directories. There is a separate tool that can examine an unmounted ext[234] 10 | image; it will find symbolic links and metadata as well. All results are 11 | recorded in a sqlite database. 12 | 13 | Once the filesystem state has been snapshotted in the database, a CLI and a 14 | GUI tool can be used to display pictures of where data blocks ended up on the 15 | underlying disk. The GUI tool shows the file system tree, a display of the 16 | contents of the physical extents on the underlying disk similar to the one 17 | in MSDEFRAG, and a list of extents. Selecting a file or a range of physical 18 | blocks shows all extents associated with the selection, although more 19 | specific queries against the extent table can also be made. 20 | 21 | DOWNLOADING 22 | 23 | Clone the git repository at: https://github.com/djwong/filemapper 24 | 25 | (Sorry, no tarballs or releases yet.) 26 | 27 | BUILDING 28 | 29 | Ensure that you have the development files for sqlite3, Qt4, python3, pyqt4, 30 | and e2fslibs installed on your system. Then run `make' to build e2mapper. 31 | Everything else is written in Python. 32 | 33 | RUNNING 34 | 35 | First, index a directory (/home) and store the database (/tmp/some.db): 36 | 37 | # ./filemapper.py -r /home /tmp/some.db 38 | 39 | Then fire up the GUI: 40 | 41 | $ ./filemapper.py -g /tmp/some.db 42 | 43 | Alternately, examine a raw ext4 image: 44 | 45 | # ./e2mapper /tmp/some.db /dev/sda1 46 | 47 | Examine files in the CLI: 48 | 49 | $ ./filemapper.py /tmp/some.db 50 | >> help 51 | 52 | LICENSE 53 | 54 | GPL v2. https://www.gnu.org/licenses/gpl-2.0.html 55 | -------------------------------------------------------------------------------- /compdb.c: -------------------------------------------------------------------------------- 1 | /* 2 | * Compress SQLite databases. 3 | * Copyright 2016 Darrick J. Wong. 4 | * Licensed under the GPLv2. 5 | */ 6 | #include 7 | #include 8 | #include 9 | #include 10 | #include 11 | #include 12 | #include 13 | #include 14 | #include "filemapper.h" 15 | #include "compdb.h" 16 | #include "compress.h" 17 | 18 | /* SQLite engine stuff */ 19 | 20 | #define offsetof(TYPE, MEMBER) ((size_t)&((TYPE *)0)->MEMBER) 21 | #define container_of(ptr, type, member) ({ \ 22 | const typeof( ((type *)0)->member ) *__mptr = (ptr); \ 23 | (type *)( (char *)__mptr - offsetof(type,member) );}) 24 | 25 | struct compdb_vfs { 26 | struct sqlite3_vfs vfs; 27 | struct sqlite3_vfs *oldvfs; 28 | struct compressor_type *compressor; 29 | char compdb_file_header[16]; 30 | 31 | }; 32 | 33 | struct compdb_file { 34 | struct sqlite3_io_methods methods; 35 | struct compdb_vfs *cvfs; 36 | int (*old_read)(sqlite3_file*, void*, 37 | int, sqlite3_int64); 38 | int (*old_write)(sqlite3_file*, const void*, 39 | int, sqlite3_int64); 40 | unsigned int freestart; 41 | unsigned int freelen; 42 | int pagesize; 43 | enum compdb_type db_type; 44 | }; 45 | 46 | /* Convert a sqlite file into a compdb file */ 47 | static inline struct compdb_file * 48 | COMPDB_F( 49 | sqlite3_file *file) 50 | { 51 | return container_of(file->pMethods, struct compdb_file, methods); 52 | } 53 | 54 | /* Figure out database parameters. */ 55 | static int 56 | compdb_sniff( 57 | struct compdb_file *ff, 58 | const struct sqlite3_super *super, 59 | int is_write) 60 | { 61 | int is_sqlite; 62 | int is_compr; 63 | 64 | dbg_printf("%s(%d)\n", __func__, __LINE__); 65 | assert(ff->db_type == DB_UNKNOWN); 66 | 67 | /* Is this really a database? */ 68 | is_sqlite = !memcmp(super->magic, SQLITE_FILE_HEADER, 69 | sizeof(super->magic)); 70 | is_compr = !memcmp(super->magic, ff->cvfs->compdb_file_header, 71 | sizeof(super->magic)); 72 | if ((!is_sqlite && !is_compr) || 73 | super->max_fraction != 64 || super->min_fraction != 32 || 74 | super->leaf_payload != 32 || ntohl(super->schema_format) > 4) 75 | return SQLITE_NOTADB; 76 | 77 | /* Is this a regular uncompressed database? */ 78 | if (is_sqlite && !is_write) { 79 | ff->db_type = DB_REGULAR; 80 | ff->methods.xRead = ff->old_read; 81 | ff->methods.xWrite = ff->old_write; 82 | return SQLITE_OK; 83 | } 84 | 85 | /* Collect some stats. */ 86 | ff->db_type = DB_COMPRESSED; 87 | ff->methods.iVersion = 1; 88 | ff->pagesize = ntohs(super->pagesize); 89 | if (ff->pagesize == 1) 90 | ff->pagesize = 65536; 91 | ff->freestart = ntohl(super->freelist_start); 92 | ff->freelen = ntohl(super->freelist_pages); 93 | 94 | dbg_printf("%s(%d) pagesz %d freepg %u:%u\n", __func__, __LINE__, 95 | ff->pagesize, ff->freestart, ff->freelen); 96 | 97 | return SQLITE_OK; 98 | } 99 | 100 | /* Do some sort of read io. */ 101 | static int 102 | compdb_read( 103 | sqlite3_file *file, 104 | void *ptr, 105 | int iAmt, 106 | sqlite3_int64 iOfst) 107 | { 108 | struct compdb_file *ff; 109 | struct compdb_block_head *bhead; 110 | unsigned int page; 111 | char *buf; 112 | int clen; 113 | int ret; 114 | 115 | ff = COMPDB_F(file); 116 | assert(iOfst == 0 || ff->db_type != DB_UNKNOWN); 117 | 118 | ret = ff->old_read(file, ptr, iAmt, iOfst); 119 | if (ff->db_type == DB_COMPRESSED && iOfst == 0) 120 | memcpy(ptr, SQLITE_FILE_HEADER, sizeof(SQLITE_FILE_HEADER)); 121 | if (ret) 122 | return ret; 123 | 124 | /* We don't compress non-btree pages. */ 125 | bhead = ptr; 126 | page = iOfst / ff->pagesize; 127 | if (ff->db_type == DB_REGULAR || page == 0 || 128 | (page >= ff->freestart && page < ff->freestart + ff->freelen) || 129 | memcmp(bhead->magic, COMPDB_BLOCK_MAGIC, sizeof(bhead->magic))) { 130 | dbg_printf("%s(%d) len=%d off=%llu\n", __func__, __LINE__, 131 | iAmt, iOfst); 132 | return SQLITE_OK; 133 | } 134 | 135 | /* Header sane? */ 136 | assert(ff->db_type == DB_COMPRESSED); 137 | clen = ntohs(bhead->len); 138 | if (clen > ff->pagesize - sizeof(*bhead) || 139 | (unsigned long long)ntohl(bhead->offset) * ff->pagesize != iOfst) { 140 | dbg_printf("%s(%d) header corrupt clen=%d boff=%u iofst=%lld\n", 141 | __func__, __LINE__, clen, ntohl(bhead->offset), 142 | iOfst); 143 | return SQLITE_CORRUPT; 144 | } 145 | 146 | /* Decompress and return. */ 147 | buf = malloc(ff->pagesize); 148 | if (!buf) 149 | return SQLITE_NOMEM; 150 | 151 | ret = ff->cvfs->compressor->decompress(ptr + sizeof(*bhead), buf, clen, 152 | ff->pagesize); 153 | if (ret < 0) { 154 | dbg_printf("%s(%d) decompress failed\n", __func__, __LINE__); 155 | free(buf); 156 | return SQLITE_CORRUPT; 157 | } 158 | 159 | assert(ret <= ff->pagesize); 160 | memcpy(ptr, buf, ret); 161 | memset(ptr + ret, 0, ff->pagesize - ret); 162 | free(buf); 163 | 164 | dbg_printf("%s(%d) len=%d off=%llu clen=%d\n", __func__, __LINE__, 165 | ret, iOfst, clen); 166 | return 0; 167 | } 168 | 169 | /* Do some sort of write. */ 170 | static int 171 | compdb_write( 172 | sqlite3_file *file, 173 | const void *ptr, 174 | int iAmt, 175 | sqlite3_int64 iOfst) 176 | { 177 | struct compdb_file *ff; 178 | struct compdb_block_head *bhead; 179 | char *buf; 180 | sqlite3_int64 isize; 181 | unsigned int page; 182 | int clen; 183 | int ret; 184 | 185 | ff = COMPDB_F(file); 186 | 187 | /* If we don't know db geometry, let's try to pull them in here. */ 188 | if (ff->db_type == DB_UNKNOWN) { 189 | assert(iOfst == 0); 190 | ret = compdb_sniff(ff, ptr, 1); 191 | if (ret) 192 | return ret; 193 | assert(ff->db_type != DB_UNKNOWN); 194 | } 195 | 196 | /* We don't compress non-btree pages. */ 197 | page = iOfst / ff->pagesize; 198 | if (ff->db_type == DB_REGULAR || page == 0 || 199 | (page >= ff->freestart && page < ff->freestart + ff->freelen)) 200 | goto no_compr; 201 | 202 | /* Try to compress data. */ 203 | buf = malloc(ff->pagesize); 204 | if (!buf) 205 | return SQLITE_NOMEM; 206 | 207 | ret = ff->cvfs->compressor->compress(ptr, buf + sizeof(*bhead), iAmt, 208 | ff->pagesize - sizeof(*bhead)); 209 | if (ret == 0) { 210 | free(buf); 211 | goto no_compr; 212 | } 213 | clen = ret + sizeof(*bhead); 214 | 215 | /* Attach compression header. */ 216 | bhead = (struct compdb_block_head *)buf; 217 | memcpy(bhead->magic, COMPDB_BLOCK_MAGIC, sizeof(bhead->magic)); 218 | bhead->len = htons(ret); 219 | bhead->offset = htonl(iOfst / ff->pagesize); 220 | 221 | /* 222 | * Truncate to where the end of the compressed block should be 223 | * so that XFS won't do speculative preallocation. 224 | */ 225 | ret = file->pMethods->xFileSize(file, &isize); 226 | if (ret) 227 | return ret; 228 | if (iOfst + clen > isize) { 229 | ret = file->pMethods->xTruncate(file, iOfst + clen); 230 | if (ret) 231 | return ret; 232 | } 233 | 234 | /* Write compressed data. */ 235 | dbg_printf("%s(%d) len=%d off=%llu clen=%d\n", __func__, __LINE__, 236 | iAmt, iOfst, clen); 237 | ret = ff->old_write(file, buf, clen, iOfst); 238 | free(buf); 239 | if (ret) 240 | goto no_compr; 241 | 242 | /* 243 | * Truncate to the end of the block to avoid short reads. 244 | */ 245 | ret = file->pMethods->xFileSize(file, &isize); 246 | if (ret) 247 | return ret; 248 | if (iOfst + iAmt > isize) { 249 | ret = file->pMethods->xTruncate(file, iOfst + iAmt); 250 | if (ret) 251 | return ret; 252 | } 253 | 254 | return ret; 255 | 256 | no_compr: 257 | /* Compression fails, just write it straight. */ 258 | dbg_printf("%s(%d) len=%d off=%llu\n", __func__, __LINE__, 259 | iAmt, iOfst); 260 | ret = ff->old_write(file, ptr, iAmt, iOfst); 261 | if (ret || iOfst != 0 || ff->db_type == DB_REGULAR) 262 | return ret; 263 | 264 | /* Make sure we write out the compressed magic. */ 265 | return ff->old_write(file, ff->cvfs->compdb_file_header, 266 | sizeof(ff->cvfs->compdb_file_header), 0); 267 | } 268 | 269 | /* 270 | * Open a file. We only care about main db files; everything else 271 | * can just pass through to the underlying VFS. 272 | */ 273 | static int 274 | compdb_open( 275 | sqlite3_vfs *vfs, 276 | const char *zName, 277 | sqlite3_file *file, 278 | int flags, 279 | int *pOutFlags) 280 | { 281 | struct sqlite3_super super; 282 | struct compdb_vfs *cvfs; 283 | struct compdb_file *ff; 284 | int ret; 285 | 286 | cvfs = container_of(vfs, struct compdb_vfs, vfs); 287 | dbg_printf("%s(%d): zName %s flags %xh\n", __func__, __LINE__, 288 | zName, flags); 289 | 290 | /* Open the underlying file. */ 291 | ret = cvfs->oldvfs->xOpen(cvfs->oldvfs, zName, file, flags, pOutFlags); 292 | if (ret || !(flags & SQLITE_OPEN_MAIN_DB)) 293 | return ret; 294 | 295 | /* Shim ourselves in. */ 296 | ff = (struct compdb_file *)(((char *)file) + cvfs->oldvfs->szOsFile); 297 | ff->cvfs = cvfs; 298 | ff->methods = *(file->pMethods); 299 | ff->methods.xRead = compdb_read; 300 | ff->methods.xWrite = compdb_write; 301 | ff->old_read = file->pMethods->xRead; 302 | ff->old_write = file->pMethods->xWrite; 303 | ff->db_type = DB_UNKNOWN; 304 | file->pMethods = &ff->methods; 305 | 306 | /* Read the header so we know a few things. */ 307 | ret = ff->old_read(file, &super, sizeof(super), 0); 308 | if (ret == SQLITE_IOERR_SHORT_READ) { 309 | /* 310 | * Empty db, so disable mmap (there's no way to disable 311 | * it after the fact) and let's see what gets written out. 312 | */ 313 | ff->methods.iVersion = 1; 314 | return SQLITE_OK; 315 | } else if (ret) { 316 | ff->methods.xClose(file); 317 | return ret; 318 | } 319 | 320 | ret = compdb_sniff(ff, &super, 0); 321 | if (ret) { 322 | ff->methods.xClose(file); 323 | return ret; 324 | } 325 | 326 | return SQLITE_OK; 327 | } 328 | 329 | /* Create compdb as a compression shim atop some other VFS. */ 330 | int 331 | compdb_register( 332 | const char *under_vfs, 333 | const char *vfs_name, 334 | const char *compressor) 335 | { 336 | sqlite3_vfs *vfs; 337 | struct compdb_vfs *newvfs; 338 | struct compressor_type *cengine; 339 | int ret; 340 | 341 | /* Find the underlying VFS. */ 342 | vfs = sqlite3_vfs_find(under_vfs); 343 | if (!vfs) 344 | return ENOENT; 345 | 346 | /* Already registered? */ 347 | if (sqlite3_vfs_find(vfs_name)) 348 | return EEXIST; 349 | 350 | /* Find our compressor */ 351 | cengine = compdb_find_compressor(compressor); 352 | if (!cengine) 353 | return ENOENT; 354 | 355 | dbg_printf("%s: Stacking %s ver %d compressor %s\n", vfs_name, 356 | vfs->zName, vfs->iVersion, cengine->name); 357 | 358 | /* Allocate new VFS structure. */ 359 | newvfs = malloc(sizeof(*newvfs)); 360 | if (!newvfs) 361 | return ENOMEM; 362 | 363 | newvfs->oldvfs = vfs; 364 | newvfs->compressor = cengine; 365 | snprintf(newvfs->compdb_file_header, sizeof(newvfs->compdb_file_header), 366 | COMPDB_FILE_TEMPLATE, cengine->name); 367 | newvfs->vfs = *vfs; 368 | newvfs->vfs.zName = strdup(vfs_name); 369 | newvfs->vfs.xOpen = compdb_open; 370 | newvfs->vfs.pNext = NULL; 371 | newvfs->vfs.szOsFile += sizeof(struct compdb_file); 372 | 373 | ret = sqlite3_vfs_register(&newvfs->vfs, under_vfs ? 0 : 1); 374 | if (ret) 375 | return EIO; 376 | 377 | return 0; 378 | } 379 | -------------------------------------------------------------------------------- /compdb.h: -------------------------------------------------------------------------------- 1 | /* 2 | * Compress SQLite databases. 3 | * Copyright 2016 Darrick J. Wong. 4 | * Licensed under the GPLv2. 5 | */ 6 | #ifndef COMPDB_H 7 | #define COMPDB_H 8 | 9 | enum compdb_type { 10 | DB_UNKNOWN, 11 | DB_REGULAR, 12 | DB_COMPRESSED, 13 | }; 14 | 15 | /* 16 | * Put this ahead of every compressed page. btree pages can't have 17 | * 0xDA as the first byte. 18 | */ 19 | static const uint8_t COMPDB_BLOCK_MAGIC[] = {0xDA, 0xAD}; 20 | struct compdb_block_head { 21 | uint8_t magic[2]; 22 | uint16_t len; /* compressed length */ 23 | uint32_t offset; /* page number */ 24 | }; 25 | 26 | /* SQLite superblock format. */ 27 | #define SQLITE_FILE_HEADER "SQLite format 3" 28 | #define COMPDB_FILE_TEMPLATE "SQLite %s v.3" 29 | struct sqlite3_super { 30 | uint8_t magic[16]; 31 | uint16_t pagesize; 32 | uint8_t write_format; 33 | uint8_t read_format; 34 | uint8_t page_reserve; 35 | uint8_t max_fraction; 36 | uint8_t min_fraction; 37 | uint8_t leaf_payload; 38 | uint32_t change_counter; 39 | uint32_t nr_pages; 40 | uint32_t freelist_start; 41 | uint32_t freelist_pages; 42 | uint32_t schema_coookie; 43 | uint32_t schema_format; 44 | uint32_t page_cache_size; 45 | uint32_t highest_btree_root; 46 | uint32_t text_encoding; 47 | uint32_t user_version; 48 | uint32_t vacuum_mode; 49 | uint32_t app_id; 50 | uint8_t reserved[20]; 51 | uint32_t version_valid_for; 52 | uint32_t sqlite_version_number; 53 | }; 54 | 55 | /* Init compressed DB VFS for sqlite3. */ 56 | int compdb_register(const char *under_vfs, const char *vfs_name, 57 | const char *compressor); 58 | 59 | #endif /* COMPDB_H */ 60 | -------------------------------------------------------------------------------- /compress.c: -------------------------------------------------------------------------------- 1 | /* 2 | * Compress SQLite databases. 3 | * Copyright 2016 Darrick J. Wong. 4 | * Licensed under the GPLv2. 5 | */ 6 | #include 7 | #include 8 | #include 9 | #include 10 | #include 11 | #include 12 | #include 13 | #include 14 | #include "compress.h" 15 | 16 | /* bzip2 */ 17 | 18 | static inline int 19 | BZIP_compress( 20 | const char *source, 21 | char *dest, 22 | int sourceSize, 23 | int maxDestSize) 24 | { 25 | bz_stream strm = {0}; 26 | char *endp; 27 | int ret; 28 | 29 | ret = BZ2_bzCompressInit(&strm, 1, 0, 30); 30 | if (ret) 31 | return 0; 32 | 33 | strm.avail_in = sourceSize; 34 | strm.next_in = (char *)source; 35 | strm.next_out = dest; 36 | endp = dest + maxDestSize; 37 | do { 38 | strm.avail_out = endp - (char *)strm.next_out; 39 | ret = BZ2_bzCompress(&strm, BZ_FINISH); 40 | if (ret != BZ_STREAM_END && ret != BZ_OK) { 41 | BZ2_bzCompressEnd(&strm); 42 | return 0; 43 | } 44 | strm.next_out = endp - strm.avail_out; 45 | } while (strm.avail_in && (char *)strm.next_out <= endp); 46 | BZ2_bzCompressEnd(&strm); 47 | 48 | return (char *)strm.next_out - dest; 49 | } 50 | 51 | static inline int 52 | BZIP_decompress( 53 | const char *source, 54 | char *dest, 55 | int compressedSize, 56 | int maxDecompressedSize) 57 | { 58 | bz_stream strm = {0}; 59 | char *endp; 60 | int ret; 61 | 62 | ret = BZ2_bzDecompressInit(&strm, 0, 0); 63 | if (ret != BZ_OK) 64 | return -1; 65 | 66 | strm.avail_in = compressedSize; 67 | strm.next_in = (char *)source; 68 | strm.next_out = dest; 69 | endp = dest + maxDecompressedSize; 70 | do { 71 | strm.avail_out = endp - (char *)strm.next_out; 72 | ret = BZ2_bzDecompress(&strm); 73 | if (ret != BZ_STREAM_END && ret != BZ_OK) { 74 | BZ2_bzDecompressEnd(&strm); 75 | return 0; 76 | } 77 | strm.next_out = endp - strm.avail_out; 78 | } while (strm.avail_in && (char *)strm.next_out <= endp); 79 | BZ2_bzDecompressEnd(&strm); 80 | 81 | if (strm.avail_in) 82 | return -1; 83 | 84 | return (char *)strm.next_out - dest; 85 | } 86 | 87 | /* LZMA */ 88 | 89 | static inline int 90 | LZMA_compress( 91 | const char *source, 92 | char *dest, 93 | int sourceSize, 94 | int maxDestSize) 95 | { 96 | lzma_stream strm = {0}; 97 | char *endp; 98 | int ret; 99 | 100 | ret = lzma_easy_encoder(&strm, 6, LZMA_CHECK_CRC64); 101 | if (ret) 102 | return 0; 103 | 104 | strm.avail_in = sourceSize; 105 | strm.next_in = (unsigned char *)source; 106 | strm.next_out = (unsigned char *)dest; 107 | endp = dest + maxDestSize; 108 | do { 109 | strm.avail_out = endp - (char *)strm.next_out; 110 | ret = lzma_code(&strm, LZMA_FINISH); 111 | if (ret != LZMA_STREAM_END && ret != LZMA_OK) { 112 | lzma_end(&strm); 113 | return 0; 114 | } 115 | strm.next_out = (unsigned char *)endp - strm.avail_out; 116 | } while (strm.avail_in && (char *)strm.next_out <= endp); 117 | lzma_end(&strm); 118 | 119 | return (char *)strm.next_out - dest; 120 | } 121 | 122 | static inline int 123 | LZMA_decompress( 124 | const char *source, 125 | char *dest, 126 | int compressedSize, 127 | int maxDecompressedSize) 128 | { 129 | lzma_stream strm = {0}; 130 | char *endp; 131 | int ret; 132 | 133 | ret = lzma_stream_decoder(&strm, ULONG_MAX, 134 | LZMA_TELL_UNSUPPORTED_CHECK | LZMA_CONCATENATED); 135 | if (ret != LZMA_OK) 136 | return -1; 137 | 138 | strm.avail_in = compressedSize; 139 | strm.next_in = (unsigned char *)source; 140 | strm.next_out = (unsigned char *)dest; 141 | endp = dest + maxDecompressedSize; 142 | do { 143 | strm.avail_out = endp - (char *)strm.next_out; 144 | ret = lzma_code(&strm, LZMA_FINISH); 145 | if (ret != LZMA_STREAM_END && ret != LZMA_OK) { 146 | lzma_end(&strm); 147 | return 0; 148 | } 149 | strm.next_out = (unsigned char *)endp - strm.avail_out; 150 | } while (strm.avail_in && (char *)strm.next_out <= endp); 151 | lzma_end(&strm); 152 | 153 | if (strm.avail_in) 154 | return -1; 155 | 156 | return (char *)strm.next_out - dest; 157 | } 158 | 159 | /* gzip deflate */ 160 | 161 | static inline int 162 | GZIP_compress( 163 | const char *source, 164 | char *dest, 165 | int sourceSize, 166 | int maxDestSize) 167 | { 168 | z_stream strm = {0}; 169 | char *endp; 170 | int ret; 171 | 172 | ret = deflateInit(&strm, 5); 173 | if (ret) 174 | return 0; 175 | 176 | strm.avail_in = sourceSize; 177 | strm.next_in = (unsigned char *)source; 178 | strm.next_out = (unsigned char *)dest; 179 | endp = dest + maxDestSize; 180 | do { 181 | strm.avail_out = endp - (char *)strm.next_out; 182 | ret = deflate(&strm, Z_FINISH); 183 | if (ret != Z_STREAM_END && ret != Z_OK) { 184 | deflateEnd(&strm); 185 | return 0; 186 | } 187 | strm.next_out = (unsigned char *)endp - strm.avail_out; 188 | } while (strm.avail_in && (char *)strm.next_out <= endp); 189 | deflateEnd(&strm); 190 | 191 | return (char *)strm.next_out - dest; 192 | } 193 | 194 | static inline int 195 | GZIP_decompress( 196 | const char *source, 197 | char *dest, 198 | int compressedSize, 199 | int maxDecompressedSize) 200 | { 201 | z_stream strm = {0}; 202 | char *endp; 203 | int ret; 204 | 205 | ret = inflateInit(&strm); 206 | if (ret != Z_OK) 207 | return -1; 208 | 209 | strm.avail_in = compressedSize; 210 | strm.next_in = (unsigned char *)source; 211 | strm.next_out = (unsigned char *)dest; 212 | endp = dest + maxDecompressedSize; 213 | do { 214 | strm.avail_out = endp - (char *)strm.next_out; 215 | ret = inflate(&strm, Z_NO_FLUSH); 216 | if (ret != Z_STREAM_END && ret != Z_OK) { 217 | inflateEnd(&strm); 218 | return 0; 219 | } 220 | strm.next_out = (unsigned char *)endp - strm.avail_out; 221 | } while (strm.avail_in && (char *)strm.next_out <= endp); 222 | inflateEnd(&strm); 223 | 224 | if (strm.avail_in) 225 | return -1; 226 | 227 | return (char *)strm.next_out - dest; 228 | } 229 | 230 | /* LZ4 HC mode */ 231 | 232 | static inline int 233 | LZ4HC_compress( 234 | const char *source, 235 | char *dest, 236 | int sourceSize, 237 | int maxDestSize) 238 | { 239 | return LZ4_compressHC2_limitedOutput(source, dest, sourceSize, 240 | maxDestSize, 8); 241 | } 242 | 243 | /* Generic stuff */ 244 | 245 | static struct compressor_type compressors[] = { 246 | {"GZIP", GZIP_compress, GZIP_decompress}, 247 | {"LZ4D", LZ4_compress_default, LZ4_decompress_safe}, 248 | {"LZ4H", LZ4HC_compress, LZ4_decompress_safe}, 249 | {"LZMA", LZMA_compress, LZMA_decompress}, 250 | {"BZ2A", BZIP_compress, BZIP_decompress}, 251 | {NULL, NULL, NULL}, 252 | }; 253 | 254 | /* Return a comma separated list of compressors. */ 255 | char * 256 | compdb_compressors(void) 257 | { 258 | struct compressor_type *ct; 259 | char *s; 260 | int len = 1; 261 | 262 | for (ct = compressors; ct->name; ct++) 263 | len += strlen(ct->name) + 1; 264 | s = malloc(len); 265 | if (!s) 266 | return NULL; 267 | for (ct = compressors, len = 0; ct->name; ct++) { 268 | if (ct != compressors) { 269 | strcpy(s + len, ","); 270 | len++; 271 | } 272 | strcpy(s + len, ct->name); 273 | len += strlen(ct->name); 274 | } 275 | 276 | return s; 277 | } 278 | 279 | /* Find a compression engine. */ 280 | struct compressor_type * 281 | compdb_find_compressor( 282 | const char *name) 283 | { 284 | struct compressor_type *ct; 285 | 286 | if (!name) 287 | return &compressors[0]; 288 | 289 | for (ct = compressors; ct->name; ct++) 290 | if (strcmp(name, ct->name) == 0) 291 | return ct; 292 | return NULL; 293 | } 294 | -------------------------------------------------------------------------------- /compress.h: -------------------------------------------------------------------------------- 1 | /* 2 | * Compression routines. 3 | * Copyright 2016 Darrick J. Wong. 4 | * Licensed under the GPLv2. 5 | */ 6 | #ifndef COMPRESS_H 7 | #define COMPRESS_H 8 | 9 | struct compressor_type { 10 | const char *name; 11 | int (*compress)(const char *, char *, int, int); 12 | int (*decompress)(const char *, char *, int, int); 13 | }; 14 | 15 | /* Find compression engine. */ 16 | struct compressor_type *compdb_find_compressor(const char *name); 17 | 18 | /* List of supported compressors. */ 19 | char *compdb_compressors(void); 20 | 21 | #endif /* COMPRESS_H */ 22 | -------------------------------------------------------------------------------- /debian/changelog: -------------------------------------------------------------------------------- 1 | filemapper (0.9.1-1) xenial; urgency=medium 2 | 3 | * Fix xfsmapper's handling of xattr extents. 4 | 5 | -- Darrick J. Wong Mon, 10 Apr 2017 19:36:55 -0700 6 | 7 | filemapper (0.9.0-2) xenial; urgency=medium 8 | 9 | * Fix the install target. 10 | 11 | -- Darrick J. Wong Fri, 7 Apr 2017 21:18:29 -0700 12 | 13 | filemapper (0.9.0-1) xenial; urgency=medium 14 | 15 | * Use GETFSMAP to find live fs metadata and unlinked file data. 16 | * Adapt e2mapper/xfsmapper to walk orphaned inodes. 17 | * Add collection of CoW staging extents to xfsmapper. 18 | 19 | -- Darrick J. Wong Fri, 7 Apr 2017 20:24:29 -0700 20 | 21 | filemapper (0.8.5-1) xenial; urgency=medium 22 | 23 | * Default to LZMA for filesystems > 100G. 24 | * Don't compress page zero! 25 | 26 | -- Darrick J. Wong Thu, 29 Dec 2016 23:31:29 -0800 27 | 28 | filemapper (0.8.4-1) xenial; urgency=medium 29 | 30 | * Fix freelist page detection. 31 | 32 | -- Darrick J. Wong Thu, 29 Dec 2016 22:53:29 -0800 33 | 34 | filemapper (0.8.3-1) xenial; urgency=medium 35 | 36 | * Work around qt5 race condition errors causing segfaults. 37 | * Check that shrinkwrapper inputs are recognized. 38 | 39 | -- Darrick J. Wong Thu, 29 Dec 2016 20:27:56 -0800 40 | 41 | filemapper (0.8.2-1) xenial; urgency=medium 42 | 43 | * Allow decompression of databases. 44 | 45 | -- Darrick J. Wong Wed, 28 Dec 2016 18:28:56 -0800 46 | 47 | filemapper (0.8.1-1) xenial; urgency=medium 48 | 49 | * Support various compression formats. 50 | 51 | -- Darrick J. Wong Wed, 28 Dec 2016 15:21:56 -0800 52 | 53 | filemapper (0.8.0-1) xenial; urgency=medium 54 | 55 | * Rebuild with xfsprogs "4.9.0-rc1+" (i.e. up to date rmap/reflink/rtrmapbt) 56 | * Record free space data to distinguish it from unaccounted space. 57 | * Heatmaps in the main display! 58 | 59 | -- Darrick J. Wong Fri, 2 Dec 2016 20:04:56 -0700 60 | 61 | filemapper (0.7.7-3.2) xenial; urgency=medium 62 | 63 | * Rebuild with xfsprogs "4.7.0-rc0" (i.e. up to date rmap/reflink/rtrmapbt) 64 | 65 | -- Darrick J. Wong Wed, 20 Jul 2016 20:52:56 -0700 66 | 67 | filemapper (0.7.7-3.1) xenial; urgency=medium 68 | 69 | * Rebuild with xfsprogs "4.7.0-rc0" (i.e. up to date rmap/reflink) 70 | 71 | -- Darrick J. Wong Sat, 19 Jun 2016 00:57:56 -0700 72 | 73 | filemapper (0.7.7-3) xenial; urgency=medium 74 | 75 | * Rebuild with LTO. 76 | 77 | -- Darrick J. Wong Sun, 20 Mar 2016 13:45:56 -0700 78 | 79 | filemapper (0.7.7-2) xenial; urgency=medium 80 | 81 | * fmgui now supports qt4 and qt5. 82 | 83 | -- Darrick J. Wong Tue, 15 Mar 2016 18:13:56 -0700 84 | 85 | filemapper (0.7.6-2) xenial; urgency=medium 86 | 87 | * Rebuild for Xenial. 88 | * Fix debian/source/format since we don't use quilt. 89 | 90 | -- Darrick J. Wong Tue, 15 Mar 2016 15:47:56 -0700 91 | 92 | filemapper (0.7.6-1) unstable; urgency=low 93 | 94 | * Fix the FIEMAP backend so that xattr fiemap failures don't land us in fibmap for the data fork. 95 | 96 | -- Darrick J. Wong Mon, 25 Jan 2016 00:02:23 -0800 97 | 98 | filemapper (0.7.5-1) unstable; urgency=low 99 | 100 | * Redefine fragmentation as extents / blocks; and calculate the avg travel for the last inode. 101 | 102 | -- Darrick J. Wong Sun, 24 Jan 2016 01:42:52 -0800 103 | 104 | filemapper (0.7.4-1) unstable; urgency=low 105 | 106 | * Fix some xfsmapper bugs reading the rmapbt due to format differences from allocbt. 107 | 108 | -- Darrick J. Wong Sun, 24 Jan 2016 00:42:17 -0800 109 | 110 | filemapper (0.7.3-1) unstable; urgency=low 111 | 112 | * Build and install fatmapper/xfsmapper if possible. 113 | * Update to xfsprogs 4.3 and dosfstools 3.0.28. 114 | 115 | -- Darrick J. Wong Sat, 23 Jan 2016 18:00:00 -0800 116 | 117 | filemapper (0.7.1-1) unstable; urgency=low 118 | 119 | * Build and install fatmapper/xfsmapper if possible. 120 | 121 | -- Darrick J. Wong Wed, 22 Apr 2015 20:19:00 -0700 122 | 123 | filemapper (0.7-1) unstable; urgency=low 124 | 125 | * Add XFS raw support. 126 | * Compute average travel scores for files. 127 | * Handle extents that go "beyond" the end of the FS (i.e. fsstat lied). 128 | * Editable zoom levels and bytes-per-cell zoom levels. 129 | * Export reports in CSV/HTML. 130 | * Record the FS type in the database. 131 | 132 | -- Darrick J. Wong Wed, 22 Apr 2015 19:44:00 -0700 133 | 134 | filemapper (0.6-1) unstable; urgency=low 135 | 136 | * Enable queries of inode data, and update the GUI to retrieve both when 137 | querying the FS 138 | * Sneak in a bunch of performance improvements. 139 | * Increase database page size for faster accesses. 140 | * Add NTFS/FAT32 raw support. 141 | * Wire up a bunch of new query types (dates, inode fields, more extent fields. 142 | * Store more inode metadata. 143 | * Analyze inode->extent relations to assess block allocator performance. 144 | * Reduce memory usage. 145 | * Shift the sorting algorithms for faster query responses. 146 | * Cache overview snapshots for much faster rendering. 147 | * Actual tool tips in the GUI. 148 | * Straighten out the code and group functions together. 149 | * Fix the fragmentation report! 150 | 151 | -- Darrick J. Wong Sat, 21 Feb 2015 01:22:00 -0800 152 | 153 | filemapper (0.5-1) unstable; urgency=low 154 | 155 | * Initial release. 156 | 157 | -- Darrick J. Wong Sun, 01 Feb 2015 23:42:50 -0800 158 | -------------------------------------------------------------------------------- /debian/compat: -------------------------------------------------------------------------------- 1 | 9 2 | -------------------------------------------------------------------------------- /debian/control: -------------------------------------------------------------------------------- 1 | Source: filemapper 2 | Section: admin 3 | Priority: optional 4 | Maintainer: Darrick J. Wong 5 | Build-Depends: debhelper (>= 9.0.0), python3, e2fslibs-dev, libsqlite3-dev, ntfs-3g-dev, python3-dev, liblz4-dev, liblzma-dev, libbz2-dev 6 | Standards-Version: 3.9.5 7 | Homepage: http://djwong.org/ 8 | #Vcs-Git: git://git.debian.org/collab-maint/filemapper.git 9 | #Vcs-Browser: http://git.debian.org/?p=collab-maint/filemapper.git;a=summary 10 | 11 | Package: filemapper 12 | Architecture: any 13 | Depends: ${shlibs:Depends}, ${misc:Depends}, ${python3:Depends}, python3-pyqt5 | python3-pyqt4, python3-dateutil 14 | Description: Explore the physical layout of a filesystem 15 | This program snapshots the extent metadata of a filesystem for later 16 | display and analysis in a graphical program. 17 | -------------------------------------------------------------------------------- /debian/copyright: -------------------------------------------------------------------------------- 1 | Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/ 2 | Upstream-Name: filemapper 3 | Source: http://djwong.org/ 4 | 5 | Files: * 6 | Copyright: 2015 Darrick J. Wong 7 | License: GPL-2+ 8 | This package is free software; you can redistribute it and/or modify 9 | it under the terms of the GNU General Public License as published by 10 | the Free Software Foundation; either version 2 of the License, or 11 | (at your option) any later version. 12 | . 13 | This package is distributed in the hope that it will be useful, 14 | but WITHOUT ANY WARRANTY; without even the implied warranty of 15 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 16 | GNU General Public License for more details. 17 | . 18 | You should have received a copy of the GNU General Public License 19 | along with this program. If not, see 20 | . 21 | On Debian systems, the complete text of the GNU General 22 | Public License version 2 can be found in "/usr/share/common-licenses/GPL-2". 23 | -------------------------------------------------------------------------------- /debian/docs: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/djwong/filemapper/70c455d028cfe9dad3e0cdfcd30262aa70281a96/debian/docs -------------------------------------------------------------------------------- /debian/rules: -------------------------------------------------------------------------------- 1 | #!/usr/bin/make -f 2 | # -*- makefile -*- 3 | 4 | # Uncomment this to turn on verbose mode. 5 | export DH_VERBOSE=1 6 | 7 | # Set XFSPROGS to the xfsprogs source tree and pass '-eXFSPROGS' to debuild to get xfsmapper. 8 | export XFSPROGS 9 | 10 | # Set DOSFSTOOLS to the xfsprogs source tree and pass '-eDOSFSTOOLS' to debuild to get fatmapper. 11 | export DOSFSTOOLS 12 | 13 | %: 14 | dh $@ --with python3 15 | -------------------------------------------------------------------------------- /debian/source/format: -------------------------------------------------------------------------------- 1 | 3.0 (native) 2 | -------------------------------------------------------------------------------- /e2mapper.1: -------------------------------------------------------------------------------- 1 | .\" Hey, EMACS: -*- nroff -*- 2 | .\" (C) Copyright 2015 Darrick J. Wong , 3 | .\" 4 | .\" First parameter, NAME, should be all caps 5 | .\" Second parameter, SECTION, should be 1-8, maybe w/ subsection 6 | .\" other parameters are allowed: see man(7), man(1) 7 | .TH FILEMAPPER 1 "February 1, 2015" 8 | .\" Please adjust this date whenever revising the manpage. 9 | .\" 10 | .\" Some roff macros, for reference: 11 | .\" .nh disable hyphenation 12 | .\" .hy enable hyphenation 13 | .\" .ad l left justify 14 | .\" .ad b justify to both left and right margins 15 | .\" .nf disable filling 16 | .\" .fi enable filling 17 | .\" .br insert line break 18 | .\" .sp insert n+1 empty lines 19 | .\" for manpage-specific macros, see man(7) 20 | .SH NAME 21 | e2mapper \- Analyze an ext[234] image for use with filemapper 22 | .SH SYNOPSIS 23 | .B filemapper 24 | .I dbfile 25 | .I fs_device 26 | .SH DESCRIPTION 27 | This manual page documents briefly the 28 | .B e2mapper 29 | command. 30 | .PP 31 | .\" TeX users may be more comfortable with the \fB\fP and 32 | .\" \fI\fP escape sequences to invode bold face and italics, 33 | .\" respectively. 34 | \fBe2mapper\fP analyzes the internals of an ext[234] filesystem and 35 | generates a snapshot that the filemapper program can use to visualize 36 | and query the physical layout of the filesystem. 37 | .SH OPTIONS 38 | These programs follow the usual GNU command line syntax, with long 39 | options starting with two dashes (`-'). 40 | A summary of options is included below. 41 | For a complete description, see the Info files. 42 | .TP 43 | .I dbfile 44 | Store the filesystem analysis in this file. 45 | .TP 46 | .I fs_device 47 | Open this raw device for analysis. 48 | .TP 49 | .B \-h, \-\-help 50 | Show summary of options. 51 | .SH SEE ALSO 52 | .BR filemapper (1). 53 | .br 54 | -------------------------------------------------------------------------------- /e2mapper.c: -------------------------------------------------------------------------------- 1 | /* 2 | * Generate filemapper databases from ext* filesystems. 3 | * Copyright 2015 Darrick J. Wong. 4 | * Licensed under the GPLv2. 5 | */ 6 | #include 7 | #include 8 | #include 9 | #include 10 | #include 11 | #include 12 | #include 13 | #include 14 | #include 15 | #include 16 | #undef DEBUG 17 | #include "filemapper.h" 18 | #include "compdb.h" 19 | 20 | struct e2map_t { 21 | struct filemapper_t base; 22 | 23 | ext2_filsys fs; 24 | errcode_t err; 25 | ext2fs_inode_bitmap iseen; 26 | ext2_ino_t ino; 27 | struct ext2fs_extent last; 28 | int type; 29 | }; 30 | #define wf_db base.db 31 | #define wf_db_err base.db_err 32 | #define wf_dirpath base.dirpath 33 | #define wf_iconv base.iconv 34 | 35 | #define EXT2_XT_METADATA (EXT2_FT_MAX + 16) 36 | #define EXT2_XT_EXTENT (EXT2_FT_MAX + 17) 37 | #define EXT2_XT_XATTR (EXT2_FT_MAX + 18) 38 | #define EXT2_XT_FREESP (EXT2_FT_MAX + 19) 39 | 40 | static int type_codes[] = { 41 | [EXT2_FT_REG_FILE] = INO_TYPE_FILE, 42 | [EXT2_FT_DIR] = INO_TYPE_DIR, 43 | [EXT2_FT_SYMLINK] = INO_TYPE_SYMLINK, 44 | [EXT2_XT_METADATA] = INO_TYPE_METADATA, 45 | [EXT2_XT_FREESP] = INO_TYPE_FREESP, 46 | }; 47 | 48 | #ifndef EXT4_INLINE_DATA_FL 49 | # define EXT4_INLINE_DATA_FL 0x10000000 /* Inode has inline data */ 50 | #endif 51 | 52 | static int extent_codes[] = { 53 | [EXT2_FT_REG_FILE] = EXT_TYPE_FILE, 54 | [EXT2_FT_DIR] = EXT_TYPE_DIR, 55 | [EXT2_XT_EXTENT] = EXT_TYPE_EXTENT, 56 | [EXT2_XT_METADATA] = EXT_TYPE_METADATA, 57 | [EXT2_XT_XATTR] = EXT_TYPE_XATTR, 58 | [EXT2_FT_SYMLINK] = EXT_TYPE_SYMLINK, 59 | [EXT2_XT_FREESP] = EXT_TYPE_FREESP, 60 | }; 61 | 62 | /* Fake inodes for FS metadata */ 63 | #define INO_METADATA_DIR (-1) 64 | #define STR_METADATA_DIR METADATA_DIR 65 | #define INO_SB_FILE (-2) 66 | #define STR_SB_FILE "superblocks" 67 | #define INO_GDT_FILE (-3) 68 | #define STR_GDT_FILE "group_descriptors" 69 | #define INO_BBITMAP_FILE (-4) 70 | #define STR_BBITMAP_FILE "block_bitmaps" 71 | #define INO_IBITMAP_FILE (-5) 72 | #define STR_IBITMAP_FILE "inode_bitmaps" 73 | #define INO_ITABLE_FILE (-6) 74 | #define STR_ITABLE_FILE "inodes" 75 | #define INO_HIDDEN_DIR (-7) 76 | #define STR_HIDDEN_DIR "hidden_files" 77 | #define INO_FREESP_FILE (-8) 78 | #define STR_FREESP_FILE FREESP_FILE 79 | #define INO_UNLINKED_DIR (-9) 80 | #define STR_UNLINKED_DIR UNLINKED_DIR 81 | /* This must come last */ 82 | #define INO_GROUPS_DIR (-10) 83 | #define STR_GROUPS_DIR "groups" 84 | 85 | /* Hidden inode paths */ 86 | #define INO_BADBLOCKS_FILE EXT2_BAD_INO 87 | #define STR_BADBLOCKS_FILE "badblocks" 88 | #define INO_USR_QUOTA_FILE EXT4_USR_QUOTA_INO 89 | #define STR_USR_QUOTA_FILE "user_quota" 90 | #define INO_GRP_QUOTA_FILE EXT4_GRP_QUOTA_INO 91 | #define STR_GRP_QUOTA_FILE "group_quota" 92 | #define INO_BOOTLOADER_FILE EXT2_BOOT_LOADER_INO 93 | #define STR_BOOTLOADER_FILE "bootloader" 94 | #define INO_UNDELETE_DIR EXT2_UNDEL_DIR_INO 95 | #define STR_UNDELETE_DIR "undelete" 96 | #define INO_RESIZE_FILE EXT2_RESIZE_INO 97 | #define STR_RESIZE_FILE "resize" 98 | #define INO_JOURNAL_FILE EXT2_JOURNAL_INO 99 | #define STR_JOURNAL_FILE "journal" 100 | #define INO_EXCLUDE_FILE EXT2_EXCLUDE_INO 101 | #define STR_EXCLUDE_FILE "exclude" 102 | #define INO_REPLICA_FILE EXT4_REPLICA_INO 103 | #define STR_REPLICA_FILE "replica" 104 | 105 | struct hidden_file { 106 | ext2_ino_t ino; 107 | const char *name; 108 | int type; 109 | }; 110 | 111 | #define H(name, type) {INO_##name, STR_##name, EXT2_##type} 112 | static struct hidden_file hidden_inodes[] = { 113 | H(BADBLOCKS_FILE, XT_METADATA), 114 | H(USR_QUOTA_FILE, XT_METADATA), 115 | H(GRP_QUOTA_FILE, XT_METADATA), 116 | H(BOOTLOADER_FILE, XT_METADATA), 117 | H(UNDELETE_DIR, FT_DIR), 118 | H(RESIZE_FILE, XT_METADATA), 119 | H(JOURNAL_FILE, XT_METADATA), 120 | H(EXCLUDE_FILE, XT_METADATA), 121 | H(REPLICA_FILE, XT_METADATA), 122 | {}, 123 | }; 124 | #undef H 125 | 126 | /* Time handling stuff */ 127 | #define EXT4_EPOCH_BITS 2 128 | #define EXT4_EPOCH_MASK ((1 << EXT4_EPOCH_BITS) - 1) 129 | #define EXT4_NSEC_MASK (~0UL << EXT4_EPOCH_BITS) 130 | 131 | /* 132 | * Extended fields will fit into an inode if the filesystem was formatted 133 | * with large inodes (-I 256 or larger) and there are not currently any EAs 134 | * consuming all of the available space. For new inodes we always reserve 135 | * enough space for the kernel's known extended fields, but for inodes 136 | * created with an old kernel this might not have been the case. None of 137 | * the extended inode fields is critical for correct filesystem operation. 138 | * This macro checks if a certain field fits in the inode. Note that 139 | * inode-size = GOOD_OLD_INODE_SIZE + i_extra_isize 140 | */ 141 | #define EXT4_FITS_IN_INODE(ext4_inode, field) \ 142 | ((offsetof(typeof(*ext4_inode), field) + \ 143 | sizeof((ext4_inode)->field)) \ 144 | <= (EXT2_GOOD_OLD_INODE_SIZE + \ 145 | (ext4_inode)->i_extra_isize)) \ 146 | 147 | static inline __u32 ext4_encode_extra_time(const struct timespec *time) 148 | { 149 | __u32 extra = sizeof(time->tv_sec) > 4 ? 150 | ((time->tv_sec - (__s32)time->tv_sec) >> 32) & 151 | EXT4_EPOCH_MASK : 0; 152 | return extra | (time->tv_nsec << EXT4_EPOCH_BITS); 153 | } 154 | 155 | static inline void ext4_decode_extra_time(struct timespec *time, __u32 extra) 156 | { 157 | if (sizeof(time->tv_sec) > 4 && (extra & EXT4_EPOCH_MASK)) { 158 | __u64 extra_bits = extra & EXT4_EPOCH_MASK; 159 | /* 160 | * Prior to kernel 3.14?, we had a broken decode function, 161 | * wherein we effectively did this: 162 | * if (extra_bits == 3) 163 | * extra_bits = 0; 164 | */ 165 | time->tv_sec += extra_bits << 32; 166 | } 167 | time->tv_nsec = ((extra) & EXT4_NSEC_MASK) >> EXT4_EPOCH_BITS; 168 | } 169 | 170 | #define EXT4_INODE_GET_XTIME(xtime, timespec, raw_inode) \ 171 | do { \ 172 | (timespec)->tv_sec = (signed)((raw_inode)->xtime); \ 173 | if (EXT4_FITS_IN_INODE(raw_inode, xtime ## _extra)) \ 174 | ext4_decode_extra_time((timespec), \ 175 | (raw_inode)->xtime ## _extra); \ 176 | else \ 177 | (timespec)->tv_nsec = 0; \ 178 | } while (0) 179 | 180 | #define EXT4_EINODE_GET_XTIME(xtime, timespec, raw_inode) \ 181 | do { \ 182 | if (EXT4_FITS_IN_INODE(raw_inode, xtime)) \ 183 | (timespec)->tv_sec = \ 184 | (signed)((raw_inode)->xtime); \ 185 | if (EXT4_FITS_IN_INODE(raw_inode, xtime ## _extra)) \ 186 | ext4_decode_extra_time((timespec), \ 187 | (raw_inode)->xtime ## _extra); \ 188 | else \ 189 | (timespec)->tv_nsec = 0; \ 190 | } while (0) 191 | 192 | /* Figure out the physical offset of an inode. */ 193 | static uint64_t inode_offset(ext2_filsys fs, ext2_ino_t ino) 194 | { 195 | blk64_t block, block_nr; 196 | unsigned long offset; 197 | dgrp_t group; 198 | 199 | group = (ino - 1) / EXT2_INODES_PER_GROUP(fs->super); 200 | offset = ((ino - 1) % EXT2_INODES_PER_GROUP(fs->super)) * 201 | EXT2_INODE_SIZE(fs->super); 202 | block = offset >> EXT2_BLOCK_SIZE_BITS(fs->super); 203 | block_nr = ext2fs_inode_table_loc(fs, group) + block; 204 | offset &= (EXT2_BLOCK_SIZE(fs->super) - 1); 205 | 206 | return (block_nr * fs->blocksize) + offset; 207 | } 208 | 209 | /* Help find extents... */ 210 | static int find_blocks(ext2_filsys fs, blk64_t *blocknr, e2_blkcnt_t blockcnt, 211 | blk64_t ref_blk, int ref_offset, void *priv_data) 212 | { 213 | struct e2map_t *wf = priv_data; 214 | unsigned long long max_extent = MAX_EXTENT_LENGTH / fs->blocksize; 215 | uint64_t loff; 216 | 217 | /* Internal node? */ 218 | if (blockcnt < 0) { 219 | if (wf->last.e_len) 220 | loff = (wf->last.e_lblk + wf->last.e_len) * 221 | fs->blocksize; 222 | else 223 | loff = 0; 224 | dbg_printf("R: ino=%d pblk=%llu\n", wf->ino, *blocknr); 225 | insert_extent(&wf->base, wf->ino, *blocknr * fs->blocksize, 226 | &loff, fs->blocksize, 0, 227 | extent_codes[EXT2_XT_EXTENT]); 228 | if (wf->wf_db_err) 229 | goto out; 230 | return 0; 231 | } 232 | 233 | /* Can we attach it to the previous extent? */ 234 | if (wf->last.e_len) { 235 | if (wf->last.e_pblk + wf->last.e_len == *blocknr && 236 | wf->last.e_len + 1 <= max_extent) { 237 | wf->last.e_len++; 238 | dbg_printf("R: ino=%d len=%u\n", wf->ino, 239 | wf->last.e_len); 240 | return 0; 241 | } 242 | 243 | /* Insert the extent */ 244 | dbg_printf("R: ino=%d pblk=%llu lblk=%llu len=%u\n", wf->ino, 245 | wf->last.e_pblk, wf->last.e_lblk, wf->last.e_len); 246 | loff = wf->last.e_lblk * fs->blocksize; 247 | insert_extent(&wf->base, wf->ino, 248 | wf->last.e_pblk * fs->blocksize, 249 | &loff, 250 | wf->last.e_len * fs->blocksize, 251 | 0, extent_codes[wf->type]); 252 | if (wf->wf_db_err) 253 | goto out; 254 | } 255 | 256 | /* Set up the next extent */ 257 | wf->last.e_pblk = *blocknr; 258 | wf->last.e_lblk = blockcnt; 259 | wf->last.e_len = 1; 260 | 261 | out: 262 | if (wf->wf_db_err) 263 | return BLOCK_ABORT; 264 | return 0; 265 | } 266 | 267 | /* Walk a file's extents for extents */ 268 | static void walk_extents(struct e2map_t *wf, ext2_ino_t ino, int type) 269 | { 270 | ext2_filsys fs = wf->fs; 271 | ext2_extent_handle_t handle; 272 | struct ext2fs_extent extent, last; 273 | int flags; 274 | unsigned long long max_extent = MAX_EXTENT_LENGTH / fs->blocksize; 275 | uint64_t loff; 276 | 277 | memset(&last, 0, sizeof(last)); 278 | wf->err = ext2fs_extent_open(fs, ino, &handle); 279 | if (wf->err) 280 | return; 281 | 282 | wf->err = ext2fs_extent_get(handle, EXT2_EXTENT_ROOT, &extent); 283 | if (wf->err) { 284 | if (wf->err == EXT2_ET_EXTENT_NO_NEXT) 285 | wf->err = 0; 286 | goto out; 287 | } 288 | 289 | do { 290 | if (extent.e_flags & EXT2_EXTENT_FLAGS_SECOND_VISIT) 291 | goto next; 292 | 293 | /* Internal node */ 294 | if (!(extent.e_flags & EXT2_EXTENT_FLAGS_LEAF)) { 295 | dbg_printf("ino=%d lblk=%llu\n", wf->ino, 296 | extent.e_pblk); 297 | loff = extent.e_lblk * fs->blocksize; 298 | insert_extent(&wf->base, ino, 299 | extent.e_pblk * fs->blocksize, 300 | &loff, 301 | fs->blocksize, 302 | 0, extent_codes[EXT2_XT_EXTENT]); 303 | if (wf->wf_db_err) 304 | goto out; 305 | goto next; 306 | } 307 | 308 | /* Can we attach it to the previous extent? */ 309 | if (last.e_len) { 310 | if (last.e_pblk + last.e_len == extent.e_pblk && 311 | last.e_lblk + last.e_len == extent.e_lblk && 312 | (last.e_flags & EXT2_EXTENT_FLAGS_UNINIT) == 313 | (extent.e_flags & EXT2_EXTENT_FLAGS_UNINIT) && 314 | last.e_len + extent.e_len <= max_extent) { 315 | last.e_len += extent.e_len; 316 | dbg_printf("R: ino=%d len=%u\n", ino, 317 | last.e_len); 318 | goto next; 319 | } 320 | 321 | /* Insert the extent */ 322 | dbg_printf("R: ino=%d pblk=%llu lblk=%llu len=%u\n", ino, 323 | last.e_pblk, last.e_lblk, last.e_len); 324 | flags = 0; 325 | if (last.e_flags & EXT2_EXTENT_FLAGS_UNINIT) 326 | flags |= EXTENT_UNWRITTEN; 327 | loff = last.e_lblk * fs->blocksize; 328 | insert_extent(&wf->base, ino, 329 | last.e_pblk * fs->blocksize, 330 | &loff, 331 | last.e_len * fs->blocksize, 332 | flags, extent_codes[type]); 333 | if (wf->wf_db_err) 334 | goto out; 335 | } 336 | 337 | /* Start recording extents */ 338 | last = extent; 339 | next: 340 | wf->err = ext2fs_extent_get(handle, EXT2_EXTENT_NEXT, &extent); 341 | } while (wf->err == 0); 342 | 343 | /* Ok if we run off the end */ 344 | if (wf->err == EXT2_ET_EXTENT_NO_NEXT) 345 | wf->err = 0; 346 | if (wf->err) 347 | goto out; 348 | 349 | /* Insert the last extent */ 350 | if (last.e_len) { 351 | dbg_printf("R: ino=%d pblk=%llu lblk=%llu len=%u\n", ino, 352 | last.e_pblk, last.e_lblk, last.e_len); 353 | flags = 0; 354 | if (last.e_flags & EXT2_EXTENT_FLAGS_UNINIT) 355 | flags |= EXTENT_UNWRITTEN; 356 | loff = last.e_lblk * fs->blocksize; 357 | insert_extent(&wf->base, ino, last.e_pblk * fs->blocksize, 358 | &loff, 359 | last.e_len * fs->blocksize, 360 | flags, extent_codes[type]); 361 | if (wf->wf_db_err) 362 | goto out; 363 | } 364 | 365 | out: 366 | ext2fs_extent_free(handle); 367 | return; 368 | } 369 | 370 | /* Walk a file's mappings for extents */ 371 | static void walk_file_mappings(struct e2map_t *wf, ext2_ino_t ino, 372 | int type) 373 | { 374 | struct ext2_inode_large *inode; 375 | struct ext2_inode *inod; 376 | uint32_t *ea_magic; 377 | blk64_t b; 378 | uint64_t ino_offset, inode_end, ino_sz, loff; 379 | uint64_t ib_sz = sizeof(uint32_t) * EXT2_N_BLOCKS; 380 | 381 | if (ext2fs_fast_test_inode_bitmap2(wf->iseen, ino)) 382 | return; 383 | 384 | /* Read the inode */ 385 | ino_sz = EXT2_INODE_SIZE(wf->fs->super); 386 | if (ino_sz < sizeof(struct ext2_inode_large)) 387 | ino_sz = sizeof(struct ext2_inode_large); 388 | wf->err = ext2fs_get_memzero(ino_sz, &inode); 389 | if (wf->err) 390 | return; 391 | inod = (struct ext2_inode *)inode; 392 | wf->err = ext2fs_read_inode_full(wf->fs, ino, inod, ino_sz); 393 | if (wf->err) 394 | goto out; 395 | 396 | /* Where is this inode in the FS? */ 397 | ino_offset = inode_offset(wf->fs, ino); 398 | inode_end = inode->i_extra_isize; 399 | insert_extent(&wf->base, ino, ino_offset, 0, 400 | EXT2_INODE_SIZE(wf->fs->super), 401 | EXTENT_SHARED | EXTENT_NOT_ALIGNED, 402 | extent_codes[EXT2_XT_METADATA]); 403 | if (wf->wf_db_err) 404 | goto out; 405 | 406 | /* inline xattr? */ 407 | ea_magic = (uint32_t *)(((char *)inode) + inode_end); 408 | if (ext2fs_le32_to_cpu(ea_magic) == EXT2_EXT_ATTR_MAGIC) { 409 | insert_extent(&wf->base, ino, ino_offset + inode_end, 410 | NULL, EXT2_INODE_SIZE(wf->fs->super) - inode_end, 411 | EXTENT_SHARED | EXTENT_NOT_ALIGNED, 412 | extent_codes[EXT2_XT_XATTR]); 413 | if (wf->wf_db_err) 414 | goto out; 415 | } 416 | 417 | /* external xattr? */ 418 | b = ext2fs_file_acl_block(wf->fs, inod); 419 | if (b) { 420 | insert_extent(&wf->base, ino, b * wf->fs->blocksize, 421 | NULL, wf->fs->blocksize, 0, 422 | extent_codes[EXT2_XT_XATTR]); 423 | if (wf->wf_db_err) 424 | goto out; 425 | } 426 | 427 | if (inode->i_flags & EXT4_INLINE_DATA_FL || 428 | type == EXT2_FT_SYMLINK) { 429 | /* inline data file or symlink? */ 430 | size_t sz = EXT2_I_SIZE(inode); 431 | loff = 0; 432 | insert_extent(&wf->base, ino, 433 | ino_offset + offsetof(struct ext2_inode, i_block), 434 | &loff, sz > ib_sz ? ib_sz : sz, 435 | EXTENT_SHARED | EXTENT_DATA_INLINE | EXTENT_NOT_ALIGNED, 436 | extent_codes[type]); 437 | if (wf->wf_db_err) 438 | goto out; 439 | 440 | /* inline data in xattr? */ 441 | if (sz <= ib_sz) 442 | goto out; 443 | insert_extent(&wf->base, ino, ino_offset + inode_end, 444 | &ib_sz, EXT2_INODE_SIZE(wf->fs->super) - inode_end, 445 | EXTENT_SHARED | EXTENT_DATA_INLINE | EXTENT_NOT_ALIGNED, 446 | extent_codes[type]); 447 | if (wf->wf_db_err) 448 | goto out; 449 | } else if (inode->i_flags & EXT4_EXTENTS_FL) { 450 | /* extent file */ 451 | walk_extents(wf, ino, type); 452 | } else { 453 | errcode_t err; 454 | 455 | wf->last.e_len = 0; 456 | wf->ino = ino; 457 | wf->type = type; 458 | err = ext2fs_block_iterate3(wf->fs, ino, BLOCK_FLAG_READ_ONLY, 459 | 0, find_blocks, wf); 460 | if (!wf->err) 461 | wf->err = err; 462 | if (wf->last.e_len > 0) { 463 | dbg_printf("R: ino=%d pblk=%llu lblk=%llu len=%u\n", 464 | wf->ino, wf->last.e_pblk, wf->last.e_lblk, 465 | wf->last.e_len); 466 | loff = wf->last.e_lblk * wf->fs->blocksize; 467 | insert_extent(&wf->base, wf->ino, 468 | wf->last.e_pblk * wf->fs->blocksize, 469 | &loff, 470 | wf->last.e_len * wf->fs->blocksize, 471 | 0, extent_codes[wf->type]); 472 | if (wf->wf_db_err) 473 | goto out; 474 | } 475 | } 476 | 477 | out: 478 | ext2fs_free_mem(&inode); 479 | ext2fs_fast_mark_inode_bitmap2(wf->iseen, ino); 480 | return; 481 | } 482 | 483 | /* Handle a directory entry */ 484 | static int walk_fs_helper(ext2_ino_t dir, int entry, 485 | struct ext2_dir_entry *dirent, int offset, 486 | int blocksize, char *buf, void *priv_data) 487 | { 488 | char path[PATH_MAX + 1]; 489 | char name[EXT2_NAME_LEN + 1]; 490 | const char *old_dirpath; 491 | int type, sz; 492 | struct ext2_dir_entry_2 *de2 = (struct ext2_dir_entry_2 *)dirent; 493 | struct e2map_t *wf = priv_data; 494 | struct ext2_inode_large inode; 495 | struct timespec tv; 496 | time_t atime, crtime, ctime, mtime; 497 | time_t *pcrtime; 498 | ssize_t size; 499 | 500 | if (!strcmp(dirent->name, ".") || !strcmp(dirent->name, "..")) 501 | return 0; 502 | 503 | sz = icvt(&wf->base, dirent->name, dirent->name_len & 0xFF, name, 504 | EXT2_NAME_LEN); 505 | if (sz < 0) 506 | return DIRENT_ABORT; 507 | dbg_printf("dir=%d name=%s/%s ino=%d type=%d\n", dir, wf->wf_dirpath, name, 508 | dirent->inode, de2->file_type); 509 | 510 | memset(&inode, 0, sizeof(inode)); 511 | wf->err = ext2fs_read_inode_full(wf->fs, dirent->inode, 512 | (struct ext2_inode *)&inode, 513 | sizeof(inode)); 514 | if (wf->err) 515 | return DIRENT_ABORT; 516 | 517 | if (de2->file_type != 0) { 518 | switch(de2->file_type) { 519 | case EXT2_FT_REG_FILE: 520 | case EXT2_FT_DIR: 521 | case EXT2_FT_SYMLINK: 522 | type = de2->file_type; 523 | break; 524 | default: 525 | return 0; 526 | } 527 | } else { 528 | if (S_ISREG(inode.i_mode)) 529 | type = EXT2_FT_REG_FILE; 530 | else if (S_ISDIR(inode.i_mode)) 531 | type = EXT2_FT_DIR; 532 | else if (S_ISLNK(inode.i_mode)) 533 | type = EXT2_FT_SYMLINK; 534 | else 535 | return 0; 536 | } 537 | 538 | EXT4_INODE_GET_XTIME(i_atime, &tv, &inode); 539 | atime = tv.tv_sec; 540 | EXT4_INODE_GET_XTIME(i_mtime, &tv, &inode); 541 | mtime = tv.tv_sec; 542 | EXT4_INODE_GET_XTIME(i_ctime, &tv, &inode); 543 | ctime = tv.tv_sec; 544 | EXT4_EINODE_GET_XTIME(i_crtime, &tv, &inode); 545 | crtime = tv.tv_sec; 546 | pcrtime = (EXT4_FITS_IN_INODE(&inode, i_crtime) ? &crtime : NULL); 547 | size = EXT2_I_SIZE(&inode); 548 | 549 | if (dir) 550 | snprintf(path, PATH_MAX, "%s/%s", wf->wf_dirpath, name); 551 | else 552 | path[0] = 0; 553 | insert_inode(&wf->base, dirent->inode, type_codes[type], path, &atime, 554 | pcrtime, &ctime, &mtime, &size); 555 | if (wf->wf_db_err) 556 | return DIRENT_ABORT; 557 | if (dir) 558 | insert_dentry(&wf->base, (signed)dir, name, dirent->inode); 559 | if (wf->wf_db_err) 560 | return DIRENT_ABORT; 561 | 562 | walk_file_mappings(wf, dirent->inode, type); 563 | if (wf->err || wf->wf_db_err) 564 | return DIRENT_ABORT; 565 | 566 | if (type == EXT2_FT_DIR) { 567 | errcode_t err; 568 | 569 | old_dirpath = wf->wf_dirpath; 570 | wf->wf_dirpath = path; 571 | err = ext2fs_dir_iterate2(wf->fs, dirent->inode, 0, NULL, 572 | walk_fs_helper, wf); 573 | if (!wf->err) 574 | wf->err = err; 575 | wf->wf_dirpath = old_dirpath; 576 | } 577 | if (wf->err || wf->wf_db_err) 578 | return DIRENT_ABORT; 579 | 580 | return 0; 581 | } 582 | 583 | /* Walk the whole FS, looking for inodes to analyze. */ 584 | static void walk_fs(struct e2map_t *wf) 585 | { 586 | ext2_filsys fs = wf->fs; 587 | struct ext2_dir_entry dirent; 588 | 589 | wf->wf_dirpath = ""; 590 | 591 | dirent.inode = EXT2_ROOT_INO; 592 | ext2fs_set_rec_len(fs, EXT2_DIR_REC_LEN(1), &dirent); 593 | dirent.name_len = (EXT2_FT_DIR << 8) | 1; 594 | dirent.name[0] = '/'; 595 | dirent.name[1] = 0; 596 | walk_fs_helper(0, 0, &dirent, 0, 0, NULL, wf); 597 | } 598 | 599 | /* Walk the orphaned inode chain. */ 600 | static void 601 | walk_unlinked_inode_chain(struct e2map_t *wf, ext2_ino_t ino) 602 | { 603 | struct ext2_dir_entry dirent; 604 | struct ext2_inode inode; 605 | 606 | wf->wf_dirpath = "/" STR_METADATA_DIR "/" STR_UNLINKED_DIR; 607 | while (ino != 0) { 608 | snprintf(dirent.name, EXT2_NAME_LEN, "%u", ino); 609 | 610 | /* Walk the unlinked inode */ 611 | dirent.inode = ino; 612 | dirent.rec_len = sizeof(dirent); 613 | dirent.name_len = strlen(dirent.name); 614 | walk_fs_helper(INO_UNLINKED_DIR, 0, &dirent, 0, 0, NULL, wf); 615 | if (wf->err || wf->wf_db_err) 616 | break; 617 | 618 | /* Grab the on-disk buffer to read next unlinked */ 619 | wf->err = ext2fs_read_inode(wf->fs, ino, &inode); 620 | if (wf->err) 621 | break; 622 | 623 | ino = inode.i_dtime; 624 | } 625 | } 626 | 627 | #define INJECT_METADATA(parent_ino, path, ino, name, type) \ 628 | do { \ 629 | inject_metadata(&wf->base, (parent_ino), (path), (ino), (name), type_codes[(type)]); \ 630 | if (wf->wf_db_err) \ 631 | goto out; \ 632 | } while(0); 633 | 634 | #define INJECT_ROOT_METADATA(suffix, type) \ 635 | INJECT_METADATA(INO_METADATA_DIR, "/" STR_METADATA_DIR, INO_##suffix, STR_##suffix, type) 636 | 637 | #define INJECT_GROUP(ino, path, type) \ 638 | INJECT_METADATA(INO_GROUPS_DIR, "/" STR_METADATA_DIR "/" STR_GROUPS_DIR, (ino), (path), (type)) 639 | 640 | /* Insert extents for a file, given a bitmap */ 641 | static void walk_bitmap(struct e2map_t *wf, int64_t ino, ext2fs_block_bitmap bm) 642 | { 643 | blk64_t start, end, out = 0, loff = 0; 644 | 645 | start = 0; 646 | end = ext2fs_blocks_count(wf->fs->super) - 1; 647 | 648 | wf->err = ext2fs_find_first_set_block_bitmap2(bm, start, end, &out); 649 | while (wf->err == 0) { 650 | start = out; 651 | wf->err = ext2fs_find_first_zero_block_bitmap2(bm, start, 652 | end, &out); 653 | if (wf->err == ENOENT) { 654 | out = end; 655 | wf->err = 0; 656 | } else if (wf->err) 657 | break; 658 | 659 | insert_extent(&wf->base, ino, start * wf->fs->blocksize, 660 | NULL, (out - start) * wf->fs->blocksize, 661 | EXTENT_SHARED, extent_codes[EXT2_XT_METADATA]); 662 | 663 | if (wf->wf_db_err) 664 | break; 665 | start = out; 666 | loff += (out - start) * wf->fs->blocksize; 667 | wf->err = ext2fs_find_first_set_block_bitmap2(bm, start, 668 | end, &out); 669 | } 670 | 671 | if (wf->err == ENOENT) 672 | wf->err = 0; 673 | } 674 | 675 | /* Invent a FS tree for metadata. */ 676 | static void walk_metadata(struct e2map_t *wf) 677 | { 678 | ext2_filsys fs = wf->fs; 679 | dgrp_t group; 680 | int64_t ino, group_ino; 681 | blk64_t s, o, n, y, first_data_block; 682 | blk_t u; 683 | struct ext2_inode inode; 684 | char path[PATH_MAX + 1]; 685 | uint32_t zero_buf[EXT2_N_BLOCKS]; 686 | ext2fs_block_bitmap sb_bmap, sb_gdt, sb_bbitmap, sb_ibitmap, sb_itable; 687 | struct hidden_file *hf; 688 | int w; 689 | 690 | sb_bmap = sb_gdt = sb_bbitmap = sb_ibitmap = sb_itable = NULL; 691 | INJECT_METADATA(EXT2_ROOT_INO, "", INO_METADATA_DIR, \ 692 | STR_METADATA_DIR, EXT2_FT_DIR); 693 | INJECT_ROOT_METADATA(SB_FILE, EXT2_XT_METADATA); 694 | INJECT_ROOT_METADATA(GDT_FILE, EXT2_XT_METADATA); 695 | INJECT_ROOT_METADATA(BBITMAP_FILE, EXT2_XT_METADATA); 696 | INJECT_ROOT_METADATA(IBITMAP_FILE, EXT2_XT_METADATA); 697 | INJECT_ROOT_METADATA(ITABLE_FILE, EXT2_XT_METADATA); 698 | INJECT_ROOT_METADATA(GROUPS_DIR, EXT2_FT_DIR); 699 | INJECT_ROOT_METADATA(HIDDEN_DIR, EXT2_FT_DIR); 700 | INJECT_ROOT_METADATA(FREESP_FILE, EXT2_XT_FREESP); 701 | 702 | wf->err = ext2fs_read_block_bitmap(fs); 703 | if (wf->err) 704 | goto out; 705 | 706 | first_data_block = fs->super->s_first_data_block; 707 | fs->super->s_first_data_block = 0; 708 | wf->err = ext2fs_allocate_block_bitmap(fs, "superblock", &sb_bmap); 709 | if (wf->err) 710 | goto out; 711 | 712 | wf->err = ext2fs_allocate_block_bitmap(fs, "group descriptors", 713 | &sb_gdt); 714 | if (wf->err) 715 | goto out; 716 | 717 | wf->err = ext2fs_allocate_block_bitmap(fs, "block bitmaps", 718 | &sb_bbitmap); 719 | if (wf->err) 720 | goto out; 721 | 722 | wf->err = ext2fs_allocate_block_bitmap(fs, "inode bitmaps", 723 | &sb_ibitmap); 724 | if (wf->err) 725 | goto out; 726 | 727 | wf->err = ext2fs_allocate_block_bitmap(fs, "inode tables", 728 | &sb_itable); 729 | if (wf->err) 730 | goto out; 731 | fs->super->s_first_data_block = first_data_block; 732 | 733 | ino = INO_GROUPS_DIR - 1; 734 | snprintf(path, PATH_MAX, "%d", fs->group_desc_count); 735 | w = strlen(path); 736 | for (group = 0; group < fs->group_desc_count; group++) { 737 | snprintf(path, PATH_MAX, "%0*d", w, group); 738 | group_ino = ino; 739 | ino--; 740 | INJECT_GROUP(group_ino, path, EXT2_FT_DIR); 741 | wf->err = ext2fs_super_and_bgd_loc2(fs, group, &s, &o, &n, &u); 742 | if (wf->err) 743 | goto out; 744 | snprintf(path, PATH_MAX, "/%s/%s/%0*d", STR_METADATA_DIR, 745 | STR_GROUPS_DIR, w, group); 746 | 747 | /* Record the superblock */ 748 | if (s || group == 0) { 749 | ext2fs_fast_mark_block_bitmap2(sb_bmap, s); 750 | INJECT_METADATA(group_ino, path, ino, "superblock", 751 | EXT2_XT_METADATA); 752 | insert_extent(&wf->base, ino, s * fs->blocksize, 753 | NULL, fs->blocksize, EXTENT_SHARED, 754 | extent_codes[EXT2_XT_METADATA]); 755 | if (wf->wf_db_err) 756 | goto out; 757 | ino--; 758 | u--; 759 | } 760 | 761 | /* Record old style group descriptors */ 762 | if (o) { 763 | ext2fs_fast_mark_block_bitmap_range2(sb_gdt, o, u); 764 | INJECT_METADATA(group_ino, path, ino, "descriptor", 765 | EXT2_XT_METADATA); 766 | insert_extent(&wf->base, ino, o * fs->blocksize, 767 | NULL, u * fs->blocksize, EXTENT_SHARED, 768 | extent_codes[EXT2_XT_METADATA]); 769 | if (wf->wf_db_err) 770 | goto out; 771 | ino--; 772 | } 773 | 774 | /* Record new style group descriptors */ 775 | if (n) { 776 | ext2fs_fast_mark_block_bitmap_range2(sb_gdt, n, u); 777 | INJECT_METADATA(group_ino, path, ino, "descriptor", 778 | EXT2_XT_METADATA); 779 | insert_extent(&wf->base, ino, n * fs->blocksize, 780 | NULL, u * fs->blocksize, EXTENT_SHARED, 781 | extent_codes[EXT2_XT_METADATA]); 782 | if (wf->wf_db_err) 783 | goto out; 784 | ino--; 785 | } 786 | 787 | /* Record block bitmap */ 788 | s = ext2fs_block_bitmap_loc(fs, group); 789 | ext2fs_fast_mark_block_bitmap2(sb_bbitmap, s); 790 | INJECT_METADATA(group_ino, path, ino, "block_bitmap", 791 | EXT2_XT_METADATA); 792 | insert_extent(&wf->base, ino, s * fs->blocksize, NULL, 793 | fs->blocksize, EXTENT_SHARED, 794 | extent_codes[EXT2_XT_METADATA]); 795 | if (wf->wf_db_err) 796 | goto out; 797 | ino--; 798 | 799 | /* Record free space. */ 800 | s = ext2fs_group_first_block2(fs, group); 801 | n = ext2fs_group_last_block2(fs, group); 802 | INJECT_METADATA(group_ino, path, ino, "freespace", 803 | EXT2_XT_FREESP); 804 | while (s <= n) { 805 | wf->err = ext2fs_find_first_zero_block_bitmap2( 806 | fs->block_map, s, n, &o); 807 | if (wf->err == ENOENT) { 808 | wf->err = 0; 809 | break; 810 | } 811 | if (wf->err) 812 | goto out; 813 | y = o; 814 | wf->err = ext2fs_find_first_set_block_bitmap2( 815 | fs->block_map, o, n, &y); 816 | if (wf->err == ENOENT) { 817 | y = n; 818 | wf->err = 0; 819 | } 820 | if (wf->err) 821 | goto out; 822 | if (o == y) 823 | break; 824 | insert_extent(&wf->base, ino, o * fs->blocksize, NULL, 825 | (y - o) * fs->blocksize, EXTENT_SHARED, 826 | extent_codes[EXT2_XT_FREESP]); 827 | if (wf->wf_db_err) 828 | goto out; 829 | insert_extent(&wf->base, INO_FREESP_FILE, 830 | o * wf->fs->blocksize, NULL, 831 | (y - o) * wf->fs->blocksize, 832 | EXTENT_SHARED, 833 | extent_codes[EXT2_XT_FREESP]); 834 | if (wf->wf_db_err) 835 | goto out; 836 | s = y + 1; 837 | } 838 | ino--; 839 | 840 | /* Record inode bitmap */ 841 | s = ext2fs_inode_bitmap_loc(fs, group); 842 | ext2fs_fast_mark_block_bitmap2(sb_ibitmap, s); 843 | INJECT_METADATA(group_ino, path, ino, "inode_bitmap", 844 | EXT2_XT_METADATA); 845 | insert_extent(&wf->base, ino, s * fs->blocksize, NULL, 846 | fs->blocksize, EXTENT_SHARED, 847 | extent_codes[EXT2_XT_METADATA]); 848 | if (wf->wf_db_err) 849 | goto out; 850 | ino--; 851 | 852 | /* Record inode table */ 853 | s = ext2fs_inode_table_loc(fs, group); 854 | ext2fs_fast_mark_block_bitmap_range2(sb_itable, s, 855 | fs->inode_blocks_per_group); 856 | INJECT_METADATA(group_ino, path, ino, "inodes", 857 | EXT2_XT_METADATA); 858 | insert_extent(&wf->base, ino, s * fs->blocksize, NULL, 859 | fs->inode_blocks_per_group * fs->blocksize, 860 | EXTENT_SHARED, extent_codes[EXT2_XT_METADATA]); 861 | if (wf->wf_db_err) 862 | goto out; 863 | ino--; 864 | } 865 | 866 | /* Unlinked inodes. */ 867 | if (fs->super->s_last_orphan) { 868 | INJECT_ROOT_METADATA(UNLINKED_DIR, EXT2_FT_DIR); 869 | walk_unlinked_inode_chain(wf, fs->super->s_last_orphan); 870 | if (wf->err || wf->wf_db_err) 871 | goto out; 872 | } 873 | 874 | /* Emit extents for the overall files */ 875 | walk_bitmap(wf, INO_SB_FILE, sb_bmap); 876 | if (wf->err || wf->wf_db_err) 877 | goto out; 878 | walk_bitmap(wf, INO_GDT_FILE, sb_gdt); 879 | if (wf->err || wf->wf_db_err) 880 | goto out; 881 | walk_bitmap(wf, INO_BBITMAP_FILE, sb_bbitmap); 882 | if (wf->err || wf->wf_db_err) 883 | goto out; 884 | walk_bitmap(wf, INO_IBITMAP_FILE, sb_ibitmap); 885 | if (wf->err || wf->wf_db_err) 886 | goto out; 887 | walk_bitmap(wf, INO_ITABLE_FILE, sb_itable); 888 | if (wf->err || wf->wf_db_err) 889 | goto out; 890 | /* Now go for the hidden files */ 891 | memset(zero_buf, 0, sizeof(zero_buf)); 892 | snprintf(path, PATH_MAX, "/%s/%s", STR_METADATA_DIR, STR_HIDDEN_DIR); 893 | for (hf = hidden_inodes; hf->ino != 0; hf++) { 894 | wf->err = ext2fs_read_inode(wf->fs, hf->ino, &inode); 895 | if (wf->err) 896 | goto out; 897 | if (!memcmp(zero_buf, inode.i_block, sizeof(zero_buf))) 898 | continue; 899 | 900 | INJECT_METADATA(INO_HIDDEN_DIR, path, hf->ino, hf->name, 901 | hf->type); 902 | 903 | walk_file_mappings(wf, hf->ino, hf->type); 904 | if (wf->err || wf->wf_db_err) 905 | goto out; 906 | 907 | if (hf->type == EXT2_FT_DIR) { 908 | errcode_t err; 909 | 910 | err = ext2fs_dir_iterate2(fs, hf->ino, 0, NULL, 911 | walk_fs_helper, wf); 912 | if (!wf->err) 913 | wf->err = err; 914 | if (wf->err || wf->wf_db_err) 915 | goto out; 916 | } 917 | } 918 | out: 919 | ext2fs_free_block_bitmap(sb_itable); 920 | ext2fs_free_block_bitmap(sb_ibitmap); 921 | ext2fs_free_block_bitmap(sb_bbitmap); 922 | ext2fs_free_block_bitmap(sb_gdt); 923 | ext2fs_free_block_bitmap(sb_bmap); 924 | return; 925 | } 926 | 927 | #define CHECK_ERROR(msg) \ 928 | do { \ 929 | if (wf.err) { \ 930 | com_err(fsdev, wf.err, (msg)); \ 931 | goto out; \ 932 | } \ 933 | if (wf.wf_db_err) { \ 934 | com_err(dbfile, 0, "%s %s", sqlite3_errstr(wf.wf_db_err), (msg)); \ 935 | goto out; \ 936 | } \ 937 | } while (0); 938 | 939 | int main(int argc, char *argv[]) 940 | { 941 | const char *dbfile; 942 | const char *fsdev; 943 | char *errm; 944 | struct e2map_t wf; 945 | sqlite3 *db = NULL; 946 | ext2_filsys fs = NULL; 947 | int db_err = 0; 948 | errcode_t err = 0, err2; 949 | uint64_t total_bytes; 950 | 951 | if (argc != 3) { 952 | printf("Usage: %s dbfile fsdevice\n", argv[0]); 953 | return 0; 954 | } 955 | 956 | add_error_table(&et_ext2_error_table); 957 | 958 | /* Open things */ 959 | memset(&wf, 0, sizeof(wf)); 960 | dbfile = argv[1]; 961 | fsdev = argv[2]; 962 | 963 | db_err = truncate(dbfile, 0); 964 | if (db_err && errno != ENOENT) { 965 | com_err(fsdev, errno, "while truncating database."); 966 | goto out; 967 | } 968 | 969 | err = ext2fs_open2(fsdev, NULL, EXT2_FLAG_64BITS | EXT2_FLAG_SKIP_MMP, 970 | 0, 0, unix_io_manager, &fs); 971 | if (err) { 972 | com_err(fsdev, err, "while opening filesystem."); 973 | goto out; 974 | } 975 | fs->default_bitmap_type = EXT2FS_BMAP64_RBTREE; 976 | 977 | total_bytes = ext2fs_blocks_count(fs->super) * fs->blocksize; 978 | err = compdb_register("unix-excl", "comp-unix-excl", 979 | total_bytes > 100000000000ULL ? "LZMA" : "GZIP"); 980 | if (err) { 981 | com_err(dbfile, 0, "%s while setting up compressed db", 982 | sqlite3_errstr(err)); 983 | goto out; 984 | } 985 | 986 | err = sqlite3_open_v2(dbfile, &db, 987 | SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE, 988 | "comp-unix-excl"); 989 | if (err) { 990 | com_err(dbfile, 0, "%s while opening database", 991 | sqlite3_errstr(err)); 992 | goto out; 993 | } 994 | 995 | wf.wf_iconv = iconv_open("UTF-8", "UTF-8"); 996 | wf.wf_db = db; 997 | wf.fs = fs; 998 | 999 | /* Prepare and clean out database. */ 1000 | prepare_db(&wf.base); 1001 | CHECK_ERROR("while preparing database"); 1002 | wf.wf_db_err = sqlite3_exec(db, "BEGIN TRANSACTION", NULL, NULL, &errm); 1003 | if (errm) { 1004 | com_err(dbfile, 0, "%s while starting transaction", errm); 1005 | free(errm); 1006 | goto out; 1007 | } 1008 | CHECK_ERROR("while starting fs analysis database transaction"); 1009 | 1010 | total_bytes = ext2fs_blocks_count(fs->super) * fs->blocksize; 1011 | collect_fs_stats(&wf.base, fs->device_name, fs->blocksize, 1012 | fs->fragsize, total_bytes, 1013 | ext2fs_free_blocks_count(fs->super) * fs->blocksize, 1014 | fs->super->s_inodes_count, 1015 | fs->super->s_free_inodes_count, 1016 | EXT2_NAME_LEN, "ext4"); 1017 | CHECK_ERROR("while storing fs stats"); 1018 | 1019 | /* Walk the filesystem */ 1020 | wf.err = ext2fs_allocate_inode_bitmap(fs, "visited inodes", &wf.iseen); 1021 | CHECK_ERROR("while allocating scanned inode bitmap"); 1022 | walk_fs(&wf); 1023 | CHECK_ERROR("while analyzing filesystem"); 1024 | 1025 | /* Walk the metadata */ 1026 | walk_metadata(&wf); 1027 | CHECK_ERROR("while analyzing metadata"); 1028 | 1029 | /* Generate indexes and finalize. */ 1030 | index_db(&wf.base); 1031 | CHECK_ERROR("while indexing database"); 1032 | finalize_fs_stats(&wf.base, fs->device_name); 1033 | CHECK_ERROR("while finalizing database"); 1034 | calc_inode_stats(&wf.base); 1035 | CHECK_ERROR("while calculating inode statistics"); 1036 | 1037 | wf.wf_db_err = sqlite3_exec(db, "END TRANSACTION", NULL, NULL, &errm); 1038 | if (errm) { 1039 | fprintf(stderr, "%s %s", errm, "while ending transaction"); 1040 | free(errm); 1041 | goto out; 1042 | } 1043 | CHECK_ERROR("while flushing fs analysis database transaction"); 1044 | 1045 | wf.wf_db_err = sqlite3_exec(db, "BEGIN TRANSACTION", NULL, NULL, &errm); 1046 | if (errm) { 1047 | fprintf(stderr, "%s %s", errm, "while starting transaction"); 1048 | free(errm); 1049 | goto out; 1050 | } 1051 | CHECK_ERROR("while starting overview cache database transaction"); 1052 | 1053 | /* Cache overviews. */ 1054 | cache_overview(&wf.base, 2048); 1055 | CHECK_ERROR("while caching CLI overview"); 1056 | cache_overview(&wf.base, 65536); 1057 | CHECK_ERROR("while caching GUI overview"); 1058 | wf.wf_db_err = sqlite3_exec(db, "END TRANSACTION", NULL, NULL, &errm); 1059 | if (errm) { 1060 | com_err(dbfile, 0, "%s while ending transaction", errm); 1061 | free(errm); 1062 | goto out; 1063 | } 1064 | CHECK_ERROR("while flushing overview cache database transaction"); 1065 | 1066 | out: 1067 | if (wf.iseen) 1068 | ext2fs_free_inode_bitmap(wf.iseen); 1069 | if (wf.wf_iconv) 1070 | iconv_close(wf.wf_iconv); 1071 | 1072 | err2 = sqlite3_close(db); 1073 | if (err2) 1074 | com_err(dbfile, 0, "%s while closing database", 1075 | sqlite3_errstr(err2)); 1076 | if (!err && err2) 1077 | err = err2; 1078 | 1079 | err2 = fs ? ext2fs_close_free(&fs) : 0; 1080 | if (err2) 1081 | com_err(fsdev, err2, "while closing filesystem."); 1082 | 1083 | if (!err && err2) 1084 | err = err2; 1085 | 1086 | return err; 1087 | } 1088 | -------------------------------------------------------------------------------- /fatmapper.1: -------------------------------------------------------------------------------- 1 | .\" Hey, EMACS: -*- nroff -*- 2 | .\" (C) Copyright 2015 Darrick J. Wong , 3 | .\" 4 | .\" First parameter, NAME, should be all caps 5 | .\" Second parameter, SECTION, should be 1-8, maybe w/ subsection 6 | .\" other parameters are allowed: see man(7), man(1) 7 | .TH FILEMAPPER 1 "April 22, 2015" 8 | .\" Please adjust this date whenever revising the manpage. 9 | .\" 10 | .\" Some roff macros, for reference: 11 | .\" .nh disable hyphenation 12 | .\" .hy enable hyphenation 13 | .\" .ad l left justify 14 | .\" .ad b justify to both left and right margins 15 | .\" .nf disable filling 16 | .\" .fi enable filling 17 | .\" .br insert line break 18 | .\" .sp insert n+1 empty lines 19 | .\" for manpage-specific macros, see man(7) 20 | .SH NAME 21 | fatmapper \- Analyze an ext[234] image for use with filemapper 22 | .SH SYNOPSIS 23 | .B filemapper 24 | .I dbfile 25 | .I fs_device 26 | .SH DESCRIPTION 27 | This manual page documents briefly the 28 | .B fatmapper 29 | command. 30 | .PP 31 | .\" TeX users may be more comfortable with the \fB\fP and 32 | .\" \fI\fP escape sequences to invode bold face and italics, 33 | .\" respectively. 34 | \fBfatmapper\fP analyzes the internals of a FAT12/16/32 filesystem and 35 | generates a snapshot that the filemapper program can use to visualize 36 | and query the physical layout of the filesystem. 37 | .SH OPTIONS 38 | These programs follow the usual GNU command line syntax, with long 39 | options starting with two dashes (`-'). 40 | A summary of options is included below. 41 | For a complete description, see the Info files. 42 | .TP 43 | .I dbfile 44 | Store the filesystem analysis in this file. 45 | .TP 46 | .I fs_device 47 | Open this raw device for analysis. 48 | .TP 49 | .B \-h, \-\-help 50 | Show summary of options. 51 | .SH SEE ALSO 52 | .BR filemapper (1). 53 | .br 54 | -------------------------------------------------------------------------------- /fatmapper.c: -------------------------------------------------------------------------------- 1 | /* 2 | * Generate filemapper databases from fat filesystems. 3 | * Copyright 2015 Darrick J. Wong. 4 | * Licensed under the GPLv2+. 5 | */ 6 | #define _XOPEN_SOURCE 600 7 | #define _FILE_OFFSET_BITS 64 8 | #define _LARGEFILE64_SOURCE 1 9 | #define _GNU_SOURCE 1 10 | 11 | #include 12 | #include 13 | #include 14 | #include 15 | #include 16 | #include 17 | #include 18 | #include 19 | #include 20 | #include 21 | 22 | #include 23 | #include 24 | #include 25 | #include 26 | #include 27 | #include 28 | #include 29 | #include 30 | #undef DEBUG 31 | #include "filemapper.h" 32 | #include "compdb.h" 33 | 34 | /* from check.h */ 35 | void add_file(DOS_FS * fs, DOS_FILE *** chain, DOS_FILE * parent, 36 | loff_t offset, FDSC ** cp); 37 | 38 | struct fatmap_t { 39 | struct filemapper_t base; 40 | 41 | DOS_FS *fs; 42 | int err; 43 | uint64_t ino, dir_ino; /* fake inode numbers */ 44 | }; 45 | #define wf_db base.db 46 | #define wf_db_err base.db_err 47 | #define wf_dirpath base.dirpath 48 | #define wf_iconv base.iconv 49 | 50 | /* misc. dosfs stuff */ 51 | #define FAT_MAX_NAME_LEN 255 52 | int interactive = 0, rw = 0, list = 0, test = 0, verbose = 0, write_immed = 0; 53 | int atari_format = 0, boot_only = 0; 54 | unsigned n_files = 0; 55 | void *mem_queue = NULL; 56 | static DOS_FILE *root; 57 | 58 | #define ROOT_DIR_INO 1 59 | #define INO_METADATA_DIR (-1) 60 | #define STR_METADATA_DIR METADATA_DIR 61 | #define INO_SB_FILE (-2) 62 | #define STR_SB_FILE "superblock" 63 | #define INO_PRIMARY_FAT_FILE (-3) 64 | #define STR_PRIMARY_FAT_FILE "primary_fat" 65 | #define INO_BACKUP_FAT_FILE (-4) 66 | #define STR_BACKUP_FAT_FILE "backup_fat" 67 | #define INO_FREESP_FILE (-5) 68 | #define STR_FREESP_FILE FREESP_FILE 69 | 70 | #define FSTART(p,fs) \ 71 | ((uint32_t)le16toh(p->dir_ent.start) | \ 72 | (fs->fat_bits == 32 ? le16toh(p->dir_ent.starthi) << 16 : 0)) 73 | 74 | typedef int (*walk_file_fn)(DOS_FS *fs, DOS_FILE *file, FDSC **cp, void *private); 75 | 76 | /* Walk a file's mappings for extents */ 77 | static void walk_file_mappings(struct fatmap_t *wf, DOS_FILE *file) 78 | { 79 | DOS_FS *fs = wf->fs; 80 | uint32_t curr, lcurr; 81 | uint64_t pclus, lclus, len; 82 | unsigned long long max_extent = MAX_EXTENT_LENGTH / fs->cluster_size; 83 | int type; 84 | uint64_t loff; 85 | 86 | if (file->dir_ent.attr & ATTR_DIR) 87 | type = EXT_TYPE_DIR; 88 | else if (file->dir_ent.attr & ATTR_VOLUME) 89 | type = EXT_TYPE_METADATA; 90 | else 91 | type = EXT_TYPE_FILE; 92 | 93 | len = 0; 94 | for (curr = FSTART(file, fs) ? FSTART(file, fs) : -1, lcurr = 0; 95 | curr != -1; 96 | curr = next_cluster(fs, curr), lcurr++) { 97 | if (len) { 98 | /* Lengthen extent */ 99 | if (pclus + len == curr && len + 1 <= max_extent) { 100 | len++; 101 | dbg_printf("R: ino=%llu len=%u\n", wf->ino, len); 102 | continue; 103 | } 104 | 105 | /* Insert the extent */ 106 | dbg_printf("R: ino=%llu pblk=%llu lblk=%llu len=%u\n", 107 | wf->ino, cluster_start(fs, pclus), 108 | lclus * fs->cluster_size, 109 | len * fs->cluster_size); 110 | loff = lclus * fs->cluster_size; 111 | insert_extent(&wf->base, wf->ino, 112 | cluster_start(fs, pclus), 113 | &loff, 114 | len * fs->cluster_size, 115 | 0, type); 116 | if (wf->wf_db_err) 117 | goto out; 118 | } 119 | 120 | /* Set up the next extent */ 121 | pclus = curr; 122 | lclus = lcurr; 123 | len = 1; 124 | } 125 | 126 | if (len) { 127 | /* Insert the extent */ 128 | dbg_printf("R: ino=%llu pblk=%llu lblk=%llu len=%u\n", 129 | wf->ino, cluster_start(fs, pclus), 130 | lclus * fs->cluster_size, len * fs->cluster_size); 131 | loff = lclus * fs->cluster_size; 132 | insert_extent(&wf->base, wf->ino, 133 | cluster_start(fs, pclus), 134 | &loff, 135 | len * fs->cluster_size, 136 | 0, type); 137 | if (wf->wf_db_err) 138 | goto out; 139 | } 140 | out: 141 | return; 142 | } 143 | 144 | /* Load directory entries from a directory */ 145 | static int scan_dir(DOS_FS * fs, DOS_FILE * this, FDSC ** cp) 146 | { 147 | DOS_FILE **chain; 148 | int i; 149 | uint32_t clu_num; 150 | 151 | chain = &this->first; 152 | i = 0; 153 | clu_num = FSTART(this, fs); 154 | lfn_reset(); 155 | while (clu_num > 0 && clu_num != -1) { 156 | add_file(fs, &chain, this, 157 | cluster_start(fs, clu_num) + (i % fs->cluster_size), cp); 158 | i += sizeof(DIR_ENT); 159 | if (!(i % fs->cluster_size)) 160 | if ((clu_num = next_cluster(fs, clu_num)) == 0 || 161 | clu_num == -1) 162 | break; 163 | } 164 | lfn_reset(); 165 | 166 | return 0; 167 | } 168 | 169 | /* Walk each directory entry */ 170 | static int walk_dir(DOS_FS *fs, DOS_FILE *start, FDSC **cp, walk_file_fn fn, 171 | void *private) 172 | { 173 | while (start) { 174 | if (!strncmp((const char *)(start->dir_ent.name), MSDOS_DOT, 175 | MSDOS_NAME) || 176 | !strncmp((const char *)(start->dir_ent.name), MSDOS_DOTDOT, 177 | MSDOS_NAME)) { 178 | start = start->next; 179 | continue; 180 | } 181 | if (fn(fs, start, cp, private)) 182 | return 1; 183 | start = start->next; 184 | } 185 | return 0; 186 | } 187 | 188 | static time_t decode_time(uint16_t date, uint16_t time) 189 | { 190 | struct tm ret; 191 | 192 | memset(&ret, 0, sizeof(ret)); 193 | date = le16toh(date); 194 | time = le16toh(time); 195 | 196 | ret.tm_year = ((date >> 9) & 0x7F) + 80; 197 | ret.tm_mon = ((date >> 5) & 0xF) - 1; 198 | ret.tm_mday = (date & 0x1F); 199 | ret.tm_hour = ((time >> 11) & 0x1F); 200 | ret.tm_min = ((time >> 5) & 0x3F); 201 | ret.tm_sec = (time & 0x1F); 202 | 203 | return mktime(&ret); 204 | } 205 | 206 | /* Handle a directory entry */ 207 | static int walk_fs_helper(DOS_FS *fs, DOS_FILE *file, FDSC **cp, void *priv_data) 208 | { 209 | char path[PATH_MAX + 1]; 210 | char name[FAT_MAX_NAME_LEN + 1]; 211 | int type; 212 | uint64_t ino; 213 | struct fatmap_t *wf = priv_data; 214 | time_t atime, crtime, mtime; 215 | ssize_t size; 216 | 217 | /* Ignore volume labels */ 218 | if (file->dir_ent.attr & 0x8) 219 | return 0; 220 | 221 | if (file->lfn) 222 | snprintf(name, FAT_MAX_NAME_LEN, "%s", file->lfn); 223 | else 224 | snprintf(name, FAT_MAX_NAME_LEN, "%s", 225 | file_name(file->dir_ent.name)); 226 | 227 | if (!strcmp(name, ".") || !strcmp(name, "..")) 228 | return 0; 229 | 230 | if (file->dir_ent.attr & ATTR_DIR) 231 | type = INO_TYPE_DIR; 232 | else if (file->dir_ent.attr & ATTR_VOLUME) 233 | type = INO_TYPE_METADATA; 234 | else 235 | type = INO_TYPE_FILE; 236 | ino = wf->ino; 237 | 238 | dbg_printf("dir=%"PRIu64" name=%s/%s attr=0x%x ino=%"PRIu64" type=%d\n", 239 | wf->dir_ino, wf->wf_dirpath, name, file->dir_ent.attr, 240 | ino, type); 241 | 242 | atime = decode_time(file->dir_ent.adate, 0); 243 | crtime = decode_time(file->dir_ent.cdate, file->dir_ent.ctime); 244 | mtime = decode_time(file->dir_ent.date, file->dir_ent.time); 245 | size = le32toh(file->dir_ent.size); 246 | 247 | snprintf(path, PATH_MAX, "%s/%s", wf->wf_dirpath, name); 248 | insert_inode(&wf->base, ino, type, path, &atime, &crtime, NULL, &mtime, 249 | &size); 250 | if (wf->wf_db_err) 251 | goto err; 252 | 253 | insert_dentry(&wf->base, wf->dir_ino, name, ino); 254 | if (wf->wf_db_err) 255 | goto err; 256 | 257 | walk_file_mappings(wf, file); 258 | if (wf->err || wf->wf_db_err) 259 | goto err; 260 | 261 | wf->ino++; 262 | if (type == INO_TYPE_DIR) { 263 | FDSC **n; 264 | const char *old_dirpath; 265 | uint64_t old_dir_ino; 266 | 267 | old_dir_ino = wf->dir_ino; 268 | old_dirpath = wf->wf_dirpath; 269 | wf->wf_dirpath = path; 270 | wf->dir_ino = ino; 271 | n = file_cd(cp, (char *)file->dir_ent.name); 272 | wf->err = scan_dir(fs, file, n); 273 | if (wf->err) 274 | goto err; 275 | 276 | walk_dir(fs, file->first, n, walk_fs_helper, wf); 277 | if (wf->err || wf->wf_db_err) 278 | goto err; 279 | 280 | wf->wf_dirpath = old_dirpath; 281 | wf->dir_ino = old_dir_ino; 282 | } 283 | if (wf->err || wf->wf_db_err) 284 | goto err; 285 | 286 | return 0; 287 | err: 288 | return -1; 289 | } 290 | 291 | /* Walk the whole FS, looking for inodes to analyze. */ 292 | static void walk_fs(struct fatmap_t *wf) 293 | { 294 | DOS_FS *fs = wf->fs; 295 | DOS_FILE **chain; 296 | FDSC **n; 297 | int i; 298 | 299 | wf->wf_dirpath = ""; 300 | wf->ino = wf->dir_ino = ROOT_DIR_INO; 301 | 302 | /* Create a fake root-root inode that points to the root dir */ 303 | root = NULL; 304 | chain = &root; 305 | lfn_reset(); 306 | if (fs->root_cluster) { 307 | add_file(fs, &chain, NULL, 0, &fp_root); 308 | } else { 309 | for (i = 0; i < fs->root_entries; i++) 310 | add_file(fs, &chain, NULL, fs->root_start + i * sizeof(DIR_ENT), 311 | &fp_root); 312 | } 313 | lfn_reset(); 314 | 315 | /* Walk the root dir */ 316 | walk_file_mappings(wf, root); 317 | if (wf->err || wf->wf_db_err) 318 | goto out; 319 | 320 | /* Now inject the root inode */ 321 | insert_inode(&wf->base, wf->ino, INO_TYPE_DIR, wf->wf_dirpath, NULL, 322 | NULL, NULL, NULL, NULL); 323 | if (wf->wf_db_err) 324 | goto out; 325 | wf->ino++; 326 | 327 | /* Now walk it */ 328 | n = file_cd(&fp_root, (char *)root->dir_ent.name); 329 | wf->err = scan_dir(fs, root, n); 330 | if (wf->err) 331 | goto out; 332 | 333 | walk_dir(fs, root->first, n, walk_fs_helper, wf); 334 | if (wf->err || wf->wf_db_err) 335 | goto out; 336 | out: 337 | return; 338 | } 339 | 340 | #define INVALID_CLUSTER (~0U) 341 | static void walk_freesp(struct fatmap_t *wf) 342 | { 343 | DOS_FS *fs = wf->fs; 344 | FAT_ENTRY ent; 345 | uint32_t cluster; 346 | uint32_t freestart = INVALID_CLUSTER; 347 | 348 | for (cluster = 0; cluster < fs->clusters; cluster++) { 349 | get_fat(&ent, fs->fat, cluster, fs); 350 | if (ent.value == 0 && freestart == INVALID_CLUSTER) 351 | freestart = cluster; 352 | else if (ent.value && freestart != INVALID_CLUSTER) { 353 | insert_extent(&wf->base, INO_FREESP_FILE, 354 | freestart * fs->cluster_size, NULL, 355 | (cluster - freestart) * fs->cluster_size, 356 | 0, EXT_TYPE_FREESP); 357 | if (wf->wf_db_err) 358 | goto out; 359 | freestart = INVALID_CLUSTER; 360 | } 361 | } 362 | if (freestart != INVALID_CLUSTER) { 363 | insert_extent(&wf->base, INO_FREESP_FILE, 364 | freestart * fs->cluster_size, NULL, 365 | (cluster - freestart) * fs->cluster_size, 366 | 0, EXT_TYPE_FREESP); 367 | if (wf->wf_db_err) 368 | goto out; 369 | } 370 | out: 371 | return; 372 | } 373 | 374 | #define INJECT_METADATA(parent_ino, path, ino, name, type) \ 375 | do { \ 376 | inject_metadata(&wf->base, (parent_ino), (path), (ino), (name), (type)); \ 377 | if (wf->wf_db_err) \ 378 | goto out; \ 379 | } while(0); 380 | 381 | #define INJECT_ROOT_METADATA(suffix, type) \ 382 | INJECT_METADATA(INO_METADATA_DIR, "/" STR_METADATA_DIR, INO_##suffix, STR_##suffix, type) 383 | 384 | /* Walk the metadata */ 385 | static void walk_metadata(struct fatmap_t *wf) 386 | { 387 | DOS_FS *fs = wf->fs; 388 | 389 | INJECT_METADATA(ROOT_DIR_INO, "", INO_METADATA_DIR, \ 390 | STR_METADATA_DIR, INO_TYPE_DIR); 391 | INJECT_ROOT_METADATA(SB_FILE, INO_TYPE_METADATA); 392 | insert_extent(&wf->base, INO_SB_FILE, 0, NULL, 393 | fs->cluster_size, 0, EXT_TYPE_METADATA); 394 | if (wf->wf_db_err) 395 | goto out; 396 | INJECT_ROOT_METADATA(PRIMARY_FAT_FILE, INO_TYPE_METADATA); 397 | insert_extent(&wf->base, INO_PRIMARY_FAT_FILE, fs->fat_start, NULL, 398 | fs->fat_size, 0, EXT_TYPE_METADATA); 399 | if (wf->wf_db_err) 400 | goto out; 401 | INJECT_ROOT_METADATA(BACKUP_FAT_FILE, INO_TYPE_METADATA); 402 | insert_extent(&wf->base, INO_BACKUP_FAT_FILE, 403 | fs->fat_start + fs->fat_size, NULL, 404 | fs->fat_size, 0, EXT_TYPE_METADATA); 405 | if (wf->wf_db_err) 406 | goto out; 407 | INJECT_ROOT_METADATA(FREESP_FILE, INO_TYPE_FREESP); 408 | walk_freesp(wf); 409 | if (wf->wf_db_err) 410 | goto out; 411 | out: 412 | return; 413 | } 414 | 415 | #define CHECK_ERROR(msg) \ 416 | do { \ 417 | if (wf.err) { \ 418 | pdie("%s %s", strerror(errno), (msg)); \ 419 | goto out; \ 420 | } \ 421 | if (wf.wf_db_err) { \ 422 | die("%s %s", sqlite3_errstr(wf.wf_db_err), (msg)); \ 423 | goto out; \ 424 | } \ 425 | } while (0); 426 | 427 | int main(int argc, char *argv[]) 428 | { 429 | const char *dbfile; 430 | const char *fsdev; 431 | char *errm; 432 | struct fatmap_t wf; 433 | sqlite3 *db = NULL; 434 | DOS_FS fsb, *fs = &fsb; 435 | int db_err = 0; 436 | uint64_t total_bytes; 437 | int err = 0, err2; 438 | 439 | if (argc != 3) { 440 | printf("Usage: %s dbfile fsdevice\n", argv[0]); 441 | return 0; 442 | } 443 | 444 | set_dos_codepage(-1); 445 | 446 | /* Open things */ 447 | memset(&fsb, 0, sizeof(fsb)); 448 | memset(&wf, 0, sizeof(wf)); 449 | dbfile = argv[1]; 450 | fsdev = argv[2]; 451 | 452 | db_err = truncate(dbfile, 0); 453 | if (db_err && errno != ENOENT) { 454 | perror(dbfile); 455 | goto out; 456 | } 457 | 458 | fs_open((char *)fsdev, rw); 459 | read_boot(fs); 460 | read_fat(fs); 461 | 462 | total_bytes = (uint64_t)fs->clusters * fs->cluster_size; 463 | err = compdb_register("unix-excl", "comp-unix-excl", 464 | total_bytes > 100000000000ULL ? "LZMA" : "GZIP"); 465 | if (err) { 466 | die("%s while setting up compressed db", 467 | sqlite3_errstr(err)); 468 | goto out; 469 | } 470 | 471 | err = sqlite3_open_v2(dbfile, &db, 472 | SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE, 473 | "comp-unix-excl"); 474 | if (err) { 475 | die("%s while opening database", 476 | sqlite3_errstr(err)); 477 | goto out; 478 | } 479 | 480 | wf.wf_iconv = iconv_open("UTF-8", "UTF-8"); 481 | wf.fs = fs; 482 | wf.wf_db = db; 483 | wf.ino = ROOT_DIR_INO; 484 | 485 | /* Prepare and clean out database. */ 486 | prepare_db(&wf.base); 487 | CHECK_ERROR("while preparing database"); 488 | wf.wf_db_err = sqlite3_exec(db, "BEGIN TRANSACTION", NULL, NULL, &errm); 489 | if (errm) { 490 | pdie("%s while starting transaction", errm); 491 | free(errm); 492 | goto out; 493 | } 494 | CHECK_ERROR("while starting fs analysis database transaction"); 495 | 496 | total_bytes = (uint64_t)fs->clusters * fs->cluster_size; 497 | 498 | collect_fs_stats(&wf.base, (char *)fsdev, fs->cluster_size, 499 | fs->cluster_size, total_bytes, 500 | (uint64_t)fs->free_clusters * fs->cluster_size, 501 | 0, 0, FAT_MAX_NAME_LEN, "FAT"); 502 | CHECK_ERROR("while storing fs stats"); 503 | 504 | /* Walk the filesystem */ 505 | walk_fs(&wf); 506 | CHECK_ERROR("while analyzing filesystem"); 507 | 508 | walk_metadata(&wf); 509 | CHECK_ERROR("while walking metadata"); 510 | 511 | /* Generate indexes and finalize. */ 512 | index_db(&wf.base); 513 | CHECK_ERROR("while indexing database"); 514 | finalize_fs_stats(&wf.base, (char *)fsdev); 515 | CHECK_ERROR("while finalizing database"); 516 | calc_inode_stats(&wf.base); 517 | CHECK_ERROR("while calculating inode statistics"); 518 | 519 | wf.wf_db_err = sqlite3_exec(db, "END TRANSACTION", NULL, NULL, &errm); 520 | if (errm) { 521 | fprintf(stderr, "%s %s", errm, "while ending transaction"); 522 | free(errm); 523 | goto out; 524 | } 525 | CHECK_ERROR("while flushing fs analysis database transaction"); 526 | 527 | wf.wf_db_err = sqlite3_exec(db, "BEGIN TRANSACTION", NULL, NULL, &errm); 528 | if (errm) { 529 | fprintf(stderr, "%s %s", errm, "while starting transaction"); 530 | free(errm); 531 | goto out; 532 | } 533 | CHECK_ERROR("while starting overview cache database transaction"); 534 | 535 | /* Cache overviews. */ 536 | cache_overview(&wf.base, 2048); 537 | CHECK_ERROR("while caching CLI overview"); 538 | cache_overview(&wf.base, 65536); 539 | CHECK_ERROR("while caching GUI overview"); 540 | wf.wf_db_err = sqlite3_exec(db, "END TRANSACTION", NULL, NULL, &errm); 541 | if (errm) { 542 | pdie("%s while ending transaction", errm); 543 | free(errm); 544 | goto out; 545 | } 546 | CHECK_ERROR("while flushing overview cache database transaction"); 547 | 548 | out: 549 | if (wf.wf_iconv) 550 | iconv_close(wf.wf_iconv); 551 | 552 | err2 = sqlite3_close(db); 553 | if (err2) 554 | die("%s while closing database", 555 | sqlite3_errstr(err2)); 556 | if (!err && err2) 557 | err = err2; 558 | 559 | return err; 560 | } 561 | -------------------------------------------------------------------------------- /fiemap.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # Python wrapper of FIEMAP/FIBMAP ioctls. 3 | # Copyright (C) 2015 Darrick J. Wong. All rights reserved. 4 | # Licensed under the GPLv2. 5 | 6 | import array 7 | import fcntl 8 | import struct 9 | import collections 10 | import os 11 | import errno 12 | import stat 13 | import itertools 14 | import ioctl 15 | 16 | # From linux/fiemap.h 17 | FIEMAP_FLAG_SYNC = 0x0001 18 | FIEMAP_FLAG_XATTR = 0x0002 19 | FIEMAP_FLAGS_COMPAT = FIEMAP_FLAG_SYNC | FIEMAP_FLAG_XATTR 20 | # Bogus flag added by djwong 21 | FIEMAP_FLAG_FORCE_FIBMAP = 0x8000000 22 | 23 | FIEMAP_EXTENT_LAST = 0x0001 24 | FIEMAP_EXTENT_UNKNOWN = 0x0002 25 | FIEMAP_EXTENT_DELALLOC = 0x0004 26 | FIEMAP_EXTENT_ENCODED = 0x0008 27 | FIEMAP_EXTENT_DATA_ENCRYPTED = 0x0080 28 | FIEMAP_EXTENT_NOT_ALIGNED = 0x0100 29 | FIEMAP_EXTENT_DATA_INLINE = 0x0200 30 | FIEMAP_EXTENT_DATA_TAIL = 0x0400 31 | FIEMAP_EXTENT_UNWRITTEN = 0x0800 32 | FIEMAP_EXTENT_MERGED = 0x1000 33 | FIEMAP_EXTENT_SHARED = 0x2000 34 | 35 | # Bogus flag added by djwong 36 | FIEMAP_EXTENT_HOLE = 0x8000 37 | 38 | # Derived from linux/fiemap.h 39 | _struct_fiemap = struct.Struct('=QQLLLL') 40 | _struct_fiemap_extent = struct.Struct('=QQQQQLLLL') 41 | _struct_fibmap = struct.Struct('=L') 42 | 43 | # From linux/fs.h 44 | _FS_IOC_FIEMAP = ioctl._IOWR(ord('f'), 11, _struct_fiemap) 45 | _FIBMAP = ioctl._IO(0x00, 1) 46 | 47 | fiemap_rec = collections.namedtuple('fiemap_rec', 48 | 'logical physical length flags hdr_flags') 49 | 50 | MAX_EXTENT_LENGTH = 2**60 51 | 52 | def fiemap2(fd, start = 0, length = ioctl._UINT64_MAX, flags = 0, count = 10000): 53 | while length > 0: 54 | hdr = _struct_fiemap.pack(start, length, flags, 0, count, 0) 55 | try: 56 | buf = bytearray(hdr) + bytearray(_struct_fiemap_extent.size * count) 57 | ret = fcntl.ioctl(fd, _FS_IOC_FIEMAP, buf) 58 | except TypeError: 59 | # Turn into mutable C-level array of chars 60 | s = '%s%s' % (hdr, '\0' * (_struct_fiemap_extent.size * count)) 61 | buf = array.array('c', s) 62 | ret = fcntl.ioctl(fd, _FS_IOC_FIEMAP, buf) 63 | 64 | if ret < 0: 65 | raise IOError('FIEMAP') 66 | 67 | meh = _struct_fiemap.unpack_from(buf) 68 | oflags = meh[2] 69 | entries = meh[3] 70 | if entries == 0: 71 | return 72 | 73 | bufsz = _struct_fiemap.size + (_struct_fiemap_extent.size * entries) 74 | assert len(buf) >= bufsz 75 | for offset in range(_struct_fiemap.size, bufsz, _struct_fiemap_extent.size): 76 | x = _struct_fiemap_extent.unpack_from(buf, offset) 77 | rec = fiemap_rec(x[0], x[1], x[2], x[5], oflags) 78 | yield rec 79 | 80 | if rec.flags & FIEMAP_EXTENT_LAST: 81 | return 82 | length -= rec.length 83 | start += rec.length 84 | 85 | def fibmap2(fd, start = 0, end = None, flags = 0): 86 | xstat = os.fstat(fd) 87 | if not stat.S_ISREG(xstat.st_mode): 88 | return 89 | if flags & FIEMAP_FLAG_XATTR: 90 | return 91 | 92 | if end is None: 93 | end = xstat.st_size 94 | 95 | statvfs = os.fstatvfs(fd) 96 | block_size = statvfs.f_bsize 97 | start_block = start // block_size 98 | end_block = (end + block_size - 1) // block_size 99 | 100 | block = start_block 101 | fe_pblk = None 102 | fe_lblk = None 103 | fe_len = None 104 | max_extent = MAX_EXTENT_LENGTH / block_size 105 | while block <= end_block: 106 | indata = struct.pack('i', block) 107 | res = fcntl.ioctl(fd, _FIBMAP, indata) 108 | pblock = struct.unpack('i', res)[0] 109 | if fe_pblk is not None: 110 | if pblock > 0 and pblock == fe_pblk + fe_len and \ 111 | fe_len <= max_extent: 112 | fe_len += 1 113 | else: 114 | yield fiemap_rec(fe_lblk * block_size, \ 115 | fe_pblk * block_size, \ 116 | fe_len * block_size, 0, \ 117 | FIEMAP_EXTENT_MERGED) 118 | fe_pblk = fe_lblk = fe_len = None 119 | else: 120 | if pblock > 0: 121 | fe_pblk = pblock 122 | fe_lblk = block 123 | fe_len = 1 124 | block += 1 125 | 126 | if fe_pblk is not None: 127 | yield fiemap_rec(fe_lblk * block_size, \ 128 | fe_pblk * block_size, fe_len * block_size, 0, \ 129 | FIEMAP_EXTENT_MERGED) 130 | 131 | def file_mappings(fd, start = 0, length = None, flags = 0): 132 | if flags & FIEMAP_FLAG_FORCE_FIBMAP: 133 | return fibmap2(fd, start, length, flags) 134 | try: 135 | return fiemap2(fd, start, length, flags) 136 | except: 137 | return fibmap2(fd, start, length, flags) 138 | 139 | if __name__ == '__main__': 140 | import sys 141 | import pprint 142 | 143 | if len(sys.argv) < 2: 144 | sys.stderr.write('No filename(s) given') 145 | sys.exit(1) 146 | 147 | for file_ in sys.argv[1:]: 148 | with open(file_, 'r') as fd: 149 | stat = os.fstat(fd.fileno()) 150 | print(file_) 151 | print('-' * len(file_)) 152 | for mapping in file_mappings(fd, length = stat.st_size, \ 153 | flags = FIEMAP_FLAG_SYNC): 154 | pprint.pprint(mapping) 155 | -------------------------------------------------------------------------------- /filemapper.1: -------------------------------------------------------------------------------- 1 | .\" Hey, EMACS: -*- nroff -*- 2 | .\" (C) Copyright 2015 Darrick J. Wong , 3 | .\" 4 | .\" First parameter, NAME, should be all caps 5 | .\" Second parameter, SECTION, should be 1-8, maybe w/ subsection 6 | .\" other parameters are allowed: see man(7), man(1) 7 | .TH FILEMAPPER 1 "February 1, 2015" 8 | .\" Please adjust this date whenever revising the manpage. 9 | .\" 10 | .\" Some roff macros, for reference: 11 | .\" .nh disable hyphenation 12 | .\" .hy enable hyphenation 13 | .\" .ad l left justify 14 | .\" .ad b justify to both left and right margins 15 | .\" .nf disable filling 16 | .\" .fi enable filling 17 | .\" .br insert line break 18 | .\" .sp insert n+1 empty lines 19 | .\" for manpage-specific macros, see man(7) 20 | .SH NAME 21 | filemapper \- Analyze, query, and display the physical layout of a filesystem. 22 | .SH SYNOPSIS 23 | .B filemapper 24 | .I 25 | .RI [ options ] 26 | .RI database 27 | .RI commands... 28 | .SH DESCRIPTION 29 | This manual page documents briefly the 30 | .B filemapper 31 | command. 32 | .PP 33 | .\" TeX users may be more comfortable with the \fB\fP and 34 | .\" \fI\fP escape sequences to invode bold face and italics, 35 | .\" respectively. 36 | \fBfilemapper\fP analyzes a filesystem's physical layout, which allows 37 | the user to graphically explore and query the positioning of all data 38 | within the filesystem. Its builtin analysis module can use FIEMAP for 39 | realtime analysis; see the 40 | .B e2mapper 41 | program to obtain more details for ext[234] filesystems. 42 | .SH OPTIONS 43 | These programs follow the usual GNU command line syntax, with long 44 | options starting with two dashes (`-'). 45 | A summary of options is included below. 46 | .TP 47 | .B \-h, \-\-help 48 | Show summary of options. 49 | .TP 50 | .B \-m 51 | Enable machine-friendly CSV output in the CLI. 52 | .TP 53 | .B \-l length 54 | Set the overview to be this many characters long. CLI only. 55 | .TP 56 | .B \-q 57 | If the -r option is given, exit immediately afterwards. 58 | .TP 59 | .B \-r fspath 60 | Perform a live analysis of the filesystem mounted at a given path via FIEMAP. 61 | This will erase all database contents. 62 | .TP 63 | .B \-g 64 | Start up the GUI. 65 | .SH SEE ALSO 66 | .BR e2mapper (1). 67 | .br 68 | -------------------------------------------------------------------------------- /filemapper.c: -------------------------------------------------------------------------------- 1 | /* 2 | * FileMapper definitions for C. 3 | * Copyright 2015 Darrick J. Wong. 4 | * Licensed under the GPLv2+. 5 | */ 6 | #undef DEBUG 7 | #undef PROGRESS_REPORT 8 | #include 9 | #include 10 | #include 11 | #include "filemapper.h" 12 | 13 | static char *opschema = "\ 14 | PRAGMA cache_size = 4096;\ 15 | PRAGMA mmap_size = 1073741824;\ 16 | PRAGMA journal_mode = MEMORY;\ 17 | PRAGMA synchronous = OFF;\ 18 | PRAGMA locking_mode = EXCLUSIVE;\ 19 | PRAGMA case_sensitive_like = ON;\ 20 | "; 21 | 22 | static char *dbschema = "PRAGMA page_size = 65536;\ 23 | PRAGMA application_id = 61272;\ 24 | PRAGMA journal_mode = MEMORY;\ 25 | DROP VIEW IF EXISTS dentry_t;\ 26 | DROP VIEW IF EXISTS path_extent_v;\ 27 | DROP VIEW IF EXISTS path_inode_v;\ 28 | DROP TABLE IF EXISTS overview_t;\ 29 | DROP TABLE IF EXISTS dentry_t;\ 30 | DROP TABLE IF EXISTS extent_t;\ 31 | DROP TABLE IF EXISTS inode_t;\ 32 | DROP TABLE IF EXISTS path_t;\ 33 | DROP TABLE IF EXISTS dir_t;\ 34 | DROP TABLE IF EXISTS fs_t;\ 35 | CREATE TABLE fs_t(path TEXT PRIMARY KEY NOT NULL, block_size INTEGER NOT NULL, frag_size INTEGER NOT NULL, total_bytes INTEGER NOT NULL, free_bytes INTEGER NOT NULL, avail_bytes INTEGER NOT NULL, total_inodes INTEGER NOT NULL, free_inodes INTEGER NOT NULL, avail_inodes INTEGER NOT NULL, max_len INTEGER NOT NULL, timestamp INTEGER NOT NULL, finished INTEGER NOT NULL, path_separator TEXT NOT NULL, fstype TEXT);\ 36 | CREATE TABLE inode_type_t(id INTEGER PRIMARY KEY UNIQUE, code TEXT NOT NULL);\ 37 | INSERT INTO inode_type_t VALUES (0, 'f');\ 38 | INSERT INTO inode_type_t VALUES (1, 'd');\ 39 | INSERT INTO inode_type_t VALUES (2, 'm');\ 40 | INSERT INTO inode_type_t VALUES (3, 's');\ 41 | INSERT INTO inode_type_t VALUES (4, 'u');\ 42 | CREATE TABLE inode_t(ino INTEGER PRIMARY KEY UNIQUE NOT NULL, type INTEGER NOT NULL, nr_extents INTEGER, travel_score REAL, atime INTEGER, crtime INTEGER, ctime INTEGER, mtime INTEGER, size INTEGER, FOREIGN KEY(type) REFERENCES inode_type_t(id));\ 43 | CREATE TABLE dir_t(dir_ino INTEGER NOT NULL, name TEXT NOT NULL, name_ino INTEGER NOT NULL, FOREIGN KEY(dir_ino) REFERENCES inode_t(ino), FOREIGN KEY(name_ino) REFERENCES inode_t(ino));\ 44 | CREATE TABLE path_t(path TEXT PRIMARY KEY UNIQUE NOT NULL, ino INTEGER NOT NULL, FOREIGN KEY(ino) REFERENCES inode_t(ino));\ 45 | CREATE TABLE extent_type_t (id INTEGER PRIMARY KEY UNIQUE, code TEXT NOT NULL);\ 46 | INSERT INTO extent_type_t VALUES (0, 'f');\ 47 | INSERT INTO extent_type_t VALUES (1, 'd');\ 48 | INSERT INTO extent_type_t VALUES (2, 'e');\ 49 | INSERT INTO extent_type_t VALUES (3, 'm');\ 50 | INSERT INTO extent_type_t VALUES (4, 'x');\ 51 | INSERT INTO extent_type_t VALUES (5, 's');\ 52 | INSERT INTO extent_type_t VALUES (6, 'u');\ 53 | CREATE TABLE extent_t(ino INTEGER NOT NULL, p_off INTEGER NOT NULL, l_off INTEGER, flags INTEGER NOT NULL, length INTEGER NOT NULL, type INTEGER NOT NULL, p_end INTEGER NOT NULL, FOREIGN KEY(ino) REFERENCES inode_t(ino), FOREIGN KEY(type) REFERENCES extent_type_t(id));\ 54 | CREATE TABLE overview_t(length INTEGER NOT NULL, cell_no INTEGER NOT NULL, files INTEGER NOT NULL, dirs INTEGER NOT NULL, mappings INTEGER NOT NULL, metadata INTEGER NOT NULL, xattrs INTEGER NOT NULL, symlinks INTEGER NOT NULL, freesp INTEGER NOT NULL, CONSTRAINT pk_overview PRIMARY KEY (length, cell_no));\ 55 | CREATE VIEW path_extent_v AS SELECT path_t.path, extent_t.p_off, extent_t.l_off, extent_t.length, extent_t.flags, extent_t.type, extent_t.p_end, extent_t.ino FROM extent_t, path_t WHERE extent_t.ino = path_t.ino;\ 56 | CREATE VIEW path_inode_v AS SELECT path_t.path, inode_t.ino, inode_t.type, inode_t.nr_extents, inode_t.travel_score, inode_t.atime, inode_t.crtime, inode_t.ctime, inode_t.mtime, inode_t.size FROM path_t, inode_t WHERE inode_t.ino = path_t.ino;\ 57 | CREATE VIEW dentry_t AS SELECT dir_t.dir_ino, dir_t.name, dir_t.name_ino, inode_t.type FROM dir_t, inode_t WHERE dir_t.name_ino = inode_t.ino;"; 58 | 59 | static char *dbindex = "CREATE INDEX inode_i ON inode_t(ino);\ 60 | CREATE INDEX path_ino_i ON path_t(ino);\ 61 | CREATE INDEX path_path_i ON path_t(path);\ 62 | CREATE INDEX dir_ino_i ON dir_t(dir_ino);\ 63 | CREATE INDEX dir_nino_i ON dir_t(name_ino);\ 64 | CREATE INDEX extent_poff_i ON extent_t(p_off, p_end);\ 65 | CREATE INDEX extent_loff_i ON extent_t(l_off, length);\ 66 | CREATE INDEX extent_ino_i ON extent_t(ino);\ 67 | CREATE INDEX overview_cell_i ON overview_t(length, cell_no);\ 68 | CREATE INDEX inode_ino_i ON inode_t(ino);\ 69 | CREATE INDEX extent_type_i ON extent_t(type);\ 70 | PRAGMA foreign_key_check;"; 71 | 72 | static int primary_extent_type_for_inode[] = { 73 | [INO_TYPE_FILE] = EXT_TYPE_FILE, 74 | [INO_TYPE_DIR] = EXT_TYPE_DIR, 75 | [INO_TYPE_METADATA] = EXT_TYPE_METADATA, 76 | [INO_TYPE_SYMLINK] = EXT_TYPE_SYMLINK, 77 | [INO_TYPE_FREESP] = EXT_TYPE_FREESP, 78 | }; 79 | 80 | /* Convert a directory pathname */ 81 | int icvt(struct filemapper_t *wf, char *in, size_t inl, char *out, size_t outl) 82 | { 83 | size_t x; 84 | 85 | while (inl) { 86 | x = iconv(wf->iconv, &in, &inl, &out, &outl); 87 | if (x == -1) { 88 | if (errno == EILSEQ || errno == EINVAL) { 89 | if (outl < 3) 90 | return -1; 91 | *out = 0xEF; 92 | out++; 93 | *out = 0xBF; 94 | out++; 95 | *out = 0xBD; 96 | out++; 97 | outl -= 3; 98 | in++; 99 | inl--; 100 | } else { 101 | return -1; 102 | } 103 | } 104 | } 105 | 106 | if (*out && outl < 1) { 107 | errno = EFBIG; 108 | return -1; 109 | } 110 | *out = 0; 111 | return 0; 112 | } 113 | 114 | /* Run a bunch of queries */ 115 | void run_batch_query(struct filemapper_t *wf, const char *sql) 116 | { 117 | sqlite3 *db = wf->db; 118 | sqlite3_stmt *stmt = NULL; 119 | const char *tail, *p; 120 | int err, err2 = 0; 121 | 122 | p = sql; 123 | err = sqlite3_prepare_v2(db, p, -1, &stmt, &tail); 124 | while (err == 0 && stmt) { 125 | do { 126 | err = sqlite3_step(stmt); 127 | } while (err == SQLITE_ROW); 128 | if (err != SQLITE_DONE) 129 | break; 130 | err = sqlite3_finalize(stmt); 131 | stmt = NULL; 132 | if (err) 133 | break; 134 | p = tail; 135 | err = sqlite3_prepare_v2(db, p, -1, &stmt, &tail); 136 | } 137 | if (stmt) 138 | err2 = sqlite3_finalize(stmt); 139 | if (!err && err2) 140 | err = err2; 141 | 142 | if (err) 143 | dbg_printf("err=%d p=%s\n", err, tail); 144 | 145 | wf->db_err = err; 146 | } 147 | 148 | /* Insert an inode record into the inode and path tables */ 149 | void insert_inode(struct filemapper_t *wf, int64_t ino, int type, 150 | const char *path, time_t *atime, time_t *crtime, 151 | time_t *ctime, time_t *mtime, int64_t *size) 152 | { 153 | const char *ino_sql = "INSERT OR REPLACE INTO inode_t VALUES(?, ?, NULL, NULL, ?, ?, ?, ?, ?);"; 154 | const char *path_sql = "INSERT INTO path_t VALUES(?, ?);"; 155 | sqlite3_stmt *stmt = NULL; 156 | int err, err2, col = 1; 157 | 158 | #ifdef PROGRESS_REPORT 159 | {static int i = 0; 160 | if (!(i++ % 23)) 161 | printf("%s: ino=%"PRId64" type=%d path=%s \r", 162 | __func__, ino, type, path); 163 | } 164 | #else 165 | dbg_printf("%s: ino=%"PRId64" type=%d path=%s\n", __func__, ino, 166 | type, path); 167 | #endif 168 | 169 | /* Update the inode table */ 170 | err = sqlite3_prepare_v2(wf->db, ino_sql, -1, &stmt, NULL); 171 | if (err) 172 | goto out; 173 | err = sqlite3_bind_int64(stmt, col++, ino); 174 | if (err) 175 | goto out; 176 | err = sqlite3_bind_int(stmt, col++, type); 177 | if (err) 178 | goto out; 179 | if (atime) 180 | err = sqlite3_bind_int64(stmt, col++, *atime); 181 | else 182 | err = sqlite3_bind_null(stmt, col++); 183 | if (err) 184 | goto out; 185 | if (crtime) 186 | err = sqlite3_bind_int64(stmt, col++, *crtime); 187 | else 188 | err = sqlite3_bind_null(stmt, col++); 189 | if (err) 190 | goto out; 191 | if (ctime) 192 | err = sqlite3_bind_int64(stmt, col++, *ctime); 193 | else 194 | err = sqlite3_bind_null(stmt, col++); 195 | if (err) 196 | goto out; 197 | if (mtime) 198 | err = sqlite3_bind_int64(stmt, col++, *mtime); 199 | else 200 | err = sqlite3_bind_null(stmt, col++); 201 | if (err) 202 | goto out; 203 | if (size) 204 | err = sqlite3_bind_int64(stmt, col++, *size); 205 | else 206 | err = sqlite3_bind_null(stmt, col++); 207 | if (err) 208 | goto out; 209 | err = sqlite3_step(stmt); 210 | if (err && err != SQLITE_DONE) 211 | goto out; 212 | err = sqlite3_finalize(stmt); 213 | if (err) 214 | goto out; 215 | stmt = NULL; 216 | 217 | /* Update the path table */ 218 | col = 1; 219 | err = sqlite3_prepare_v2(wf->db, path_sql, -1, &stmt, NULL); 220 | if (err) 221 | goto out; 222 | err = sqlite3_bind_text(stmt, col++, path, -1, SQLITE_STATIC); 223 | if (err) 224 | goto out; 225 | err = sqlite3_bind_int64(stmt, col++, ino); 226 | if (err) 227 | goto out; 228 | err = sqlite3_step(stmt); 229 | if (err && err != SQLITE_DONE) 230 | goto out; 231 | err = 0; 232 | out: 233 | err2 = sqlite3_finalize(stmt); 234 | if (!err && err2) 235 | err = err2; 236 | wf->db_err = err; 237 | } 238 | 239 | /* Insert a directory entry into the database. */ 240 | void insert_dentry(struct filemapper_t *wf, int64_t dir_ino, 241 | const char *name, int64_t ino) 242 | { 243 | const char *dentry_sql = "INSERT INTO dir_t VALUES(?, ?, ?);"; 244 | sqlite3_stmt *stmt = NULL; 245 | int err, err2, col = 1; 246 | 247 | dbg_printf("%s: dir=%"PRId64" name=%s ino=%"PRId64"\n", __func__, 248 | dir_ino, name, ino); 249 | 250 | /* Update the dentry table */ 251 | err = sqlite3_prepare_v2(wf->db, dentry_sql, -1, &stmt, NULL); 252 | if (err) 253 | goto out; 254 | err = sqlite3_bind_int64(stmt, col++, dir_ino); 255 | if (err) 256 | goto out; 257 | err = sqlite3_bind_text(stmt, col++, name, -1, SQLITE_STATIC); 258 | if (err) 259 | goto out; 260 | err = sqlite3_bind_int64(stmt, col++, ino); 261 | if (err) 262 | goto out; 263 | err = sqlite3_step(stmt); 264 | if (err && err != SQLITE_DONE) 265 | goto out; 266 | err = 0; 267 | out: 268 | err2 = sqlite3_finalize(stmt); 269 | if (!err && err2) 270 | err = err2; 271 | wf->db_err = err; 272 | } 273 | 274 | /* Insert an extent into the database. */ 275 | void insert_extent(struct filemapper_t *wf, int64_t ino, uint64_t physical, 276 | uint64_t *logical, uint64_t length, int flags, int type) 277 | { 278 | const char *extent_sql = "INSERT INTO extent_t VALUES(?, ?, ?, ?, ?, ?, ?);"; 279 | sqlite3_stmt *stmt = NULL; 280 | int err, err2, col = 1; 281 | 282 | dbg_printf("%s: ino=%"PRId64" phys=%"PRIu64" logical=%"PRIu64" len=%"PRIu64" flags=0x%x type=%d\n", __func__, 283 | ino, physical, logical ? *logical : 0, length, flags, type); 284 | 285 | /* Update the dentry table */ 286 | err = sqlite3_prepare_v2(wf->db, extent_sql, -1, &stmt, NULL); 287 | if (err) 288 | goto out; 289 | err = sqlite3_bind_int64(stmt, col++, ino); 290 | if (err) 291 | goto out; 292 | err = sqlite3_bind_int64(stmt, col++, physical); 293 | if (err) 294 | goto out; 295 | if (logical) { 296 | err = sqlite3_bind_int64(stmt, col++, *logical); 297 | if (err) 298 | goto out; 299 | } else { 300 | err = sqlite3_bind_null(stmt, col++); 301 | if (err) 302 | goto out; 303 | } 304 | err = sqlite3_bind_int(stmt, col++, flags); 305 | if (err) 306 | goto out; 307 | err = sqlite3_bind_int64(stmt, col++, length); 308 | if (err) 309 | goto out; 310 | err = sqlite3_bind_int(stmt, col++, type); 311 | if (err) 312 | goto out; 313 | err = sqlite3_bind_int64(stmt, col++, physical + length - 1); 314 | if (err) 315 | goto out; 316 | err = sqlite3_step(stmt); 317 | if (err && err != SQLITE_DONE) 318 | goto out; 319 | err = 0; 320 | out: 321 | err2 = sqlite3_finalize(stmt); 322 | if (!err && err2) 323 | err = err2; 324 | wf->db_err = err; 325 | } 326 | 327 | void inject_metadata(struct filemapper_t *wf, int64_t parent_ino, 328 | const char *path, int64_t ino, const char *name, 329 | int type) 330 | { 331 | char __path[PATH_MAX + 1]; 332 | 333 | snprintf(__path, PATH_MAX, "%s/%s", path, name); 334 | wf->dirpath = path; 335 | insert_inode(wf, ino, type, __path, NULL, NULL, NULL, NULL, NULL); 336 | if (wf->db_err) 337 | return; 338 | insert_dentry(wf, parent_ino, name, ino); 339 | if (wf->db_err) 340 | return; 341 | } 342 | 343 | /* Store fs statistics in the database */ 344 | void collect_fs_stats(struct filemapper_t *wf, char *fs_name, 345 | uint32_t blocksize, uint32_t fragsize, 346 | uint64_t total_bytes, uint64_t free_bytes, 347 | uint64_t total_inodes, uint64_t free_inodes, 348 | unsigned int max_name_len, const char *fstype) 349 | { 350 | const char *sql = "INSERT INTO fs_t VALUES(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 0, ?, ?);"; 351 | char p[PATH_MAX + 1]; 352 | sqlite3_stmt *stmt; 353 | time_t t; 354 | int err, err2, col = 1; 355 | 356 | err = sqlite3_prepare_v2(wf->db, sql, -1, &stmt, NULL); 357 | if (err) 358 | goto out; 359 | err = icvt(wf, fs_name, strlen(fs_name), p, PATH_MAX); 360 | if (err) 361 | goto out; 362 | err = sqlite3_bind_text(stmt, col++, p, -1, SQLITE_STATIC); 363 | if (err) 364 | goto out; 365 | err = sqlite3_bind_int(stmt, col++, blocksize); 366 | if (err) 367 | goto out; 368 | err = sqlite3_bind_int(stmt, col++, fragsize); 369 | if (err) 370 | goto out; 371 | err = sqlite3_bind_int64(stmt, col++, total_bytes); 372 | if (err) 373 | goto out; 374 | err = sqlite3_bind_int64(stmt, col++, free_bytes); 375 | if (err) 376 | goto out; 377 | err = sqlite3_bind_int64(stmt, col++, free_bytes); 378 | if (err) 379 | goto out; 380 | err = sqlite3_bind_int64(stmt, col++, total_inodes); 381 | if (err) 382 | goto out; 383 | err = sqlite3_bind_int64(stmt, col++, free_inodes); 384 | if (err) 385 | goto out; 386 | err = sqlite3_bind_int64(stmt, col++, free_inodes); 387 | if (err) 388 | goto out; 389 | err = sqlite3_bind_int(stmt, col++, max_name_len); 390 | if (err) 391 | goto out; 392 | t = time(NULL); 393 | err = sqlite3_bind_int64(stmt, col++, t); 394 | if (err) 395 | goto out; 396 | err = sqlite3_bind_text(stmt, col++, "/", -1, SQLITE_STATIC); 397 | if (err) 398 | goto out; 399 | err = sqlite3_bind_text(stmt, col++, fstype, -1, SQLITE_STATIC); 400 | if (err) 401 | goto out; 402 | err = sqlite3_step(stmt); 403 | if (err && err != SQLITE_DONE) 404 | goto out; 405 | err = 0; 406 | out: 407 | err2 = sqlite3_finalize(stmt); 408 | if (!err && err2) 409 | err = err2; 410 | wf->db_err = err; 411 | } 412 | 413 | /* Mark the database as complete. */ 414 | void finalize_fs_stats(struct filemapper_t *wf, char *fs_name) 415 | { 416 | const char *sql = "UPDATE fs_t SET finished = 1 WHERE path = ?;"; 417 | char p[PATH_MAX + 1]; 418 | sqlite3_stmt *stmt; 419 | int64_t total_bytes, max_pend; 420 | int err, err2, col = 1; 421 | 422 | err = sqlite3_prepare_v2(wf->db, sql, -1, &stmt, NULL); 423 | if (err) 424 | goto out; 425 | err = icvt(wf, fs_name, strlen(fs_name), p, PATH_MAX); 426 | if (err) 427 | goto out; 428 | err = sqlite3_bind_text(stmt, col++, p, -1, SQLITE_STATIC); 429 | if (err) 430 | goto out; 431 | err = sqlite3_step(stmt); 432 | if (err && err != SQLITE_DONE) 433 | goto out; 434 | err = sqlite3_finalize(stmt); 435 | stmt = NULL; 436 | if (err) 437 | goto out; 438 | 439 | /* Make sure the extents don't "overflow" the end of the FS. */ 440 | sql = "SELECT MAX(p_end) FROM extent_t"; 441 | err = sqlite3_prepare_v2(wf->db, sql, -1, &stmt, NULL); 442 | if (err) 443 | goto out; 444 | err = sqlite3_step(stmt); 445 | if (err && err != SQLITE_ROW) 446 | goto out; 447 | max_pend = sqlite3_column_int64(stmt, 0); 448 | err = sqlite3_step(stmt); 449 | if (err && err != SQLITE_DONE) 450 | goto out; 451 | err = sqlite3_finalize(stmt); 452 | stmt = NULL; 453 | if (err) 454 | goto out; 455 | 456 | sql = "SELECT total_bytes FROM fs_t"; 457 | err = sqlite3_prepare_v2(wf->db, sql, -1, &stmt, NULL); 458 | if (err) 459 | goto out; 460 | err = sqlite3_step(stmt); 461 | if (err && err != SQLITE_ROW) 462 | goto out; 463 | total_bytes = sqlite3_column_int64(stmt, 0); 464 | err = sqlite3_step(stmt); 465 | if (err && err != SQLITE_DONE) 466 | goto out; 467 | err = sqlite3_finalize(stmt); 468 | stmt = NULL; 469 | if (err) 470 | goto out; 471 | 472 | if (total_bytes <= max_pend) { 473 | sql = "UPDATE fs_t SET total_bytes = ? WHERE path = ?"; 474 | err = sqlite3_prepare_v2(wf->db, sql, -1, &stmt, NULL); 475 | if (err) 476 | goto out; 477 | err = sqlite3_bind_int64(stmt, 1, max_pend + 1); 478 | if (err) 479 | goto out; 480 | err = sqlite3_bind_text(stmt, col++, p, -1, SQLITE_STATIC); 481 | if (err) 482 | goto out; 483 | err = sqlite3_step(stmt); 484 | if (err && err != SQLITE_DONE) 485 | goto out; 486 | err = sqlite3_finalize(stmt); 487 | stmt = NULL; 488 | if (err) 489 | goto out; 490 | } 491 | out: 492 | err2 = (stmt ? sqlite3_finalize(stmt) : 0); 493 | if (!err && err2) 494 | err = err2; 495 | wf->db_err = err; 496 | } 497 | 498 | /* Generate an overview cache. */ 499 | void cache_overview(struct filemapper_t *wf, uint64_t length) 500 | { 501 | sqlite3 *db = wf->db; 502 | uint64_t start_cell, end_cell, i; 503 | uint64_t e_p_off, e_p_end; 504 | double bytes_per_cell; 505 | int e_type; 506 | sqlite3_stmt *stmt = NULL; 507 | struct overview_t *overview = NULL; 508 | int err, err2; 509 | uint64_t total_bytes; 510 | char *sql; 511 | 512 | /* How many bytes do we know about? */ 513 | sql = "SELECT total_bytes FROM fs_t"; 514 | err = sqlite3_prepare_v2(db, sql, -1, &stmt, NULL); 515 | if (err) 516 | goto out; 517 | err = sqlite3_step(stmt); 518 | if (err && err != SQLITE_ROW) 519 | goto out; 520 | total_bytes = sqlite3_column_int64(stmt, 0); 521 | err = sqlite3_step(stmt); 522 | if (err && err != SQLITE_DONE) 523 | goto out; 524 | err = sqlite3_finalize(stmt); 525 | stmt = NULL; 526 | if (err) 527 | goto out; 528 | 529 | /* Allocate memory */ 530 | overview = calloc(length, sizeof(*overview)); 531 | if (overview == NULL) { 532 | err = SQLITE_NOMEM; 533 | goto out; 534 | } 535 | 536 | bytes_per_cell = (double)total_bytes / length; 537 | 538 | /* Aggregate the extents */ 539 | sql = "SELECT p_off, p_end, type FROM extent_t;"; 540 | err = sqlite3_prepare_v2(db, sql, -1, &stmt, NULL); 541 | if (err) 542 | goto out; 543 | err = sqlite3_step(stmt); 544 | while (err == SQLITE_ROW) { 545 | e_p_off = sqlite3_column_int64(stmt, 0); 546 | e_p_end = sqlite3_column_int64(stmt, 1); 547 | e_type = sqlite3_column_int(stmt, 2); 548 | start_cell = e_p_off / bytes_per_cell; 549 | end_cell = e_p_end / bytes_per_cell; 550 | assert(start_cell < length && end_cell < length); 551 | 552 | switch (e_type) { 553 | case EXT_TYPE_FILE: 554 | for (i = start_cell; i <= end_cell; i++) 555 | overview[i].files++; 556 | break; 557 | case EXT_TYPE_DIR: 558 | for (i = start_cell; i <= end_cell; i++) 559 | overview[i].dirs++; 560 | break; 561 | case EXT_TYPE_EXTENT: 562 | for (i = start_cell; i <= end_cell; i++) 563 | overview[i].mappings++; 564 | break; 565 | case EXT_TYPE_METADATA: 566 | for (i = start_cell; i <= end_cell; i++) 567 | overview[i].metadata++; 568 | break; 569 | case EXT_TYPE_XATTR: 570 | for (i = start_cell; i <= end_cell; i++) 571 | overview[i].xattrs++; 572 | break; 573 | case EXT_TYPE_SYMLINK: 574 | for (i = start_cell; i <= end_cell; i++) 575 | overview[i].symlinks++; 576 | break; 577 | case EXT_TYPE_FREESP: 578 | for (i = start_cell; i <= end_cell; i++) 579 | overview[i].freesp++; 580 | break; 581 | } 582 | err = sqlite3_step(stmt); 583 | } 584 | if (err && err != SQLITE_DONE) 585 | goto out; 586 | err = sqlite3_finalize(stmt); 587 | if (err) 588 | goto out; 589 | stmt = NULL; 590 | 591 | /* Now spit it back to the database */ 592 | err = sqlite3_prepare_v2(db, "INSERT OR REPLACE INTO overview_t VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?);", 593 | -1, &stmt, NULL); 594 | if (err) 595 | goto out; 596 | for (i = 0; i < length; i++) { 597 | err = sqlite3_bind_int64(stmt, 1, length); 598 | if (err) 599 | goto out; 600 | err = sqlite3_bind_int64(stmt, 2, i); 601 | if (err) 602 | goto out; 603 | err = sqlite3_bind_int64(stmt, 3, overview[i].files); 604 | if (err) 605 | goto out; 606 | err = sqlite3_bind_int64(stmt, 4, overview[i].dirs); 607 | if (err) 608 | goto out; 609 | err = sqlite3_bind_int64(stmt, 5, overview[i].mappings); 610 | if (err) 611 | goto out; 612 | err = sqlite3_bind_int64(stmt, 6, overview[i].metadata); 613 | if (err) 614 | goto out; 615 | err = sqlite3_bind_int64(stmt, 7, overview[i].xattrs); 616 | if (err) 617 | goto out; 618 | err = sqlite3_bind_int64(stmt, 8, overview[i].symlinks); 619 | if (err) 620 | goto out; 621 | err = sqlite3_bind_int64(stmt, 9, overview[i].freesp); 622 | if (err) 623 | goto out; 624 | err = sqlite3_step(stmt); 625 | if (err && err != SQLITE_DONE) 626 | goto out; 627 | err = sqlite3_reset(stmt); 628 | if (err) 629 | goto out; 630 | } 631 | out: 632 | err2 = sqlite3_finalize(stmt); 633 | if (!err) 634 | err = err2; 635 | free(overview); 636 | wf->db_err = err; 637 | } 638 | 639 | /* Prepare database to receive new data */ 640 | void prepare_db(struct filemapper_t *wf) 641 | { 642 | run_batch_query(wf, opschema); 643 | if (wf->db_err) 644 | return; 645 | run_batch_query(wf, dbschema); 646 | if (wf->db_err) 647 | return; 648 | } 649 | 650 | /* Index database data */ 651 | void index_db(struct filemapper_t *wf) 652 | { 653 | run_batch_query(wf, dbindex); 654 | } 655 | 656 | /* Calculate the number of extents and travel score data */ 657 | void calc_inode_stats(struct filemapper_t *wf) 658 | { 659 | sqlite3 *db = wf->db; 660 | sqlite3_stmt *ino_stmt = NULL, *upd_stmt = NULL; 661 | int64_t extents, p_dist, l_dist, last_poff, last_loff; 662 | int64_t p_off, l_off, length; 663 | int64_t last_ino, ino; 664 | int etype, itype, has_ino; 665 | int err, err2; 666 | 667 | /* For each inode... */ 668 | err = sqlite3_prepare_v2(db, "SELECT extent_t.ino, inode_t.type AS itype, extent_t.type AS etype, p_off, l_off, length FROM extent_t INNER JOIN inode_t WHERE extent_t.l_off IS NOT NULL AND extent_t.ino = inode_t.ino AND inode_t.ino IN (SELECT ino FROM inode_t WHERE travel_score IS NULL OR nr_extents IS NULL) ORDER BY extent_t.ino, l_off;", 669 | -1, &ino_stmt, NULL); 670 | if (err) 671 | goto out; 672 | 673 | /* Update inode table... */ 674 | err = sqlite3_prepare_v2(db, "UPDATE inode_t SET nr_extents = ?, travel_score = ? WHERE ino = ?;", 675 | -1, &upd_stmt, NULL); 676 | if (err) 677 | goto out; 678 | 679 | /* For each inode... */ 680 | extents = p_dist = l_dist = 0; 681 | last_poff = last_loff = 0; 682 | last_ino = 0; 683 | has_ino = 0; 684 | err = sqlite3_step(ino_stmt); 685 | while (err == SQLITE_ROW) { 686 | ino = sqlite3_column_int64(ino_stmt, 0); 687 | itype = sqlite3_column_int(ino_stmt, 1); 688 | etype = sqlite3_column_int(ino_stmt, 2); 689 | p_off = sqlite3_column_int64(ino_stmt, 3); 690 | l_off = sqlite3_column_int64(ino_stmt, 4); 691 | length = sqlite3_column_int64(ino_stmt, 5); 692 | dbg_printf("%s: ino=%"PRId64" itype=%d etype=%d poff=%"PRId64 693 | " loff=%"PRId64" len=%"PRId64"\n", 694 | __func__, ino, itype, etype, p_off, l_off, length); 695 | 696 | if (etype != primary_extent_type_for_inode[itype]) 697 | goto next; 698 | 699 | if (!has_ino || ino != last_ino) { 700 | if (has_ino) { 701 | err = sqlite3_reset(upd_stmt); 702 | if (err) 703 | goto out; 704 | err = sqlite3_bind_int64(upd_stmt, 1, extents); 705 | if (err) 706 | goto out; 707 | err = sqlite3_bind_double(upd_stmt, 2, (double)p_dist / l_dist); 708 | if (err) 709 | goto out; 710 | err = sqlite3_bind_int64(upd_stmt, 3, last_ino); 711 | if (err) 712 | goto out; 713 | err = sqlite3_step(upd_stmt); 714 | if (err && err != SQLITE_DONE) 715 | goto out; 716 | } 717 | extents = p_dist = l_dist = 0; 718 | last_poff = last_loff = 0; 719 | has_ino = 1; 720 | last_ino = ino; 721 | } 722 | 723 | if (extents) { 724 | p_dist += abs(p_off - last_poff); 725 | l_dist += l_off - last_loff; 726 | } 727 | extents++; 728 | p_dist += length; 729 | l_dist += length; 730 | last_poff = p_off + length - 1; 731 | last_loff = l_off + length - 1; 732 | 733 | next: 734 | err = sqlite3_step(ino_stmt); 735 | } 736 | if (has_ino) { 737 | err = sqlite3_reset(upd_stmt); 738 | if (err) 739 | goto out; 740 | err = sqlite3_bind_int64(upd_stmt, 1, extents); 741 | if (err) 742 | goto out; 743 | err = sqlite3_bind_double(upd_stmt, 2, (double)p_dist / l_dist); 744 | if (err) 745 | goto out; 746 | err = sqlite3_bind_int64(upd_stmt, 3, last_ino); 747 | if (err) 748 | goto out; 749 | err = sqlite3_step(upd_stmt); 750 | if (err && err != SQLITE_DONE) 751 | goto out; 752 | } 753 | if (err && err != SQLITE_DONE) 754 | goto out; 755 | err = 0; 756 | out: 757 | err2 = sqlite3_finalize(upd_stmt); 758 | if (err2 && !err) 759 | err = err2; 760 | err2 = sqlite3_finalize(ino_stmt); 761 | if (err2 && !err) 762 | err = err2; 763 | wf->db_err = err; 764 | } 765 | 766 | /* Simple bitmap functions */ 767 | int fm_test_bit(const uint8_t *bmap, const uint64_t bit) 768 | { 769 | return (bmap[bit >> 3] >> (bit & 7)) & 1; 770 | } 771 | 772 | void fm_set_bit(uint8_t *bmap, const uint64_t bit, const int new_value) 773 | { 774 | if (new_value) 775 | bmap[bit >> 3] |= (1 << (bit & 7)); 776 | else 777 | bmap[bit >> 3] &= ~(1 << (bit & 7)); 778 | } 779 | -------------------------------------------------------------------------------- /filemapper.desktop.in: -------------------------------------------------------------------------------- 1 | [Desktop Entry] 2 | Version=1.0 3 | Terminal=false 4 | Icon=%libdir%/filemapper.png 5 | Type=Application 6 | Categories=System 7 | Exec=filemapper -g %f 8 | Name=FileMapper 9 | GenericName=Filesystem Analysis 10 | Comment=Analyze, visualize, and query the physical layout of a filesystem. 11 | StartupNotify=true 12 | Keywords=Filesystem;SQL; 13 | InitialPreference=5 14 | X-KDE-Protocols=file 15 | -------------------------------------------------------------------------------- /filemapper.h: -------------------------------------------------------------------------------- 1 | /* 2 | * FileMapper declarations for C. 3 | * Copyright 2015 Darrick J. Wong. 4 | * Licensed under the GPLv2+. 5 | */ 6 | #ifndef FILEMAPPER_H_ 7 | #define FILEMAPPER_H_ 8 | #include 9 | #include 10 | #include 11 | #include 12 | #include 13 | #include 14 | #include 15 | #include 16 | #include 17 | #include 18 | 19 | #ifdef DEBUG 20 | # define dbg_printf(f, a...) do {printf(f, ## a); fflush(stdout); } while (0) 21 | #else 22 | # define dbg_printf(f, a...) 23 | #endif 24 | 25 | struct filemapper_t { 26 | sqlite3 *db; 27 | int db_err; 28 | const char *dirpath; 29 | iconv_t iconv; 30 | }; 31 | 32 | struct overview_t { 33 | uint64_t files; 34 | uint64_t dirs; 35 | uint64_t mappings; 36 | uint64_t metadata; 37 | uint64_t xattrs; 38 | uint64_t symlinks; 39 | uint64_t freesp; 40 | }; 41 | 42 | #define INO_TYPE_FILE 0 43 | #define INO_TYPE_DIR 1 44 | #define INO_TYPE_METADATA 2 45 | #define INO_TYPE_SYMLINK 3 46 | #define INO_TYPE_FREESP 4 47 | 48 | #define METADATA_DIR "$metadata" 49 | #define FREESP_FILE "freespace" 50 | #define METADATA_FILE "metadata" 51 | #define UNLINKED_DIR "orphaned" 52 | 53 | #define EXT_TYPE_FILE 0 54 | #define EXT_TYPE_DIR 1 55 | #define EXT_TYPE_EXTENT 2 56 | #define EXT_TYPE_METADATA 3 57 | #define EXT_TYPE_XATTR 4 58 | #define EXT_TYPE_SYMLINK 5 59 | #define EXT_TYPE_FREESP 6 60 | 61 | /* Extent flags. Yes, these are the FIEMAP flags. */ 62 | #define EXTENT_LAST 0x00000001 /* Last extent in file. */ 63 | #define EXTENT_UNKNOWN 0x00000002 /* Data location unknown. */ 64 | #define EXTENT_DELALLOC 0x00000004 /* Location still pending. 65 | * Sets EXTENT_UNKNOWN. */ 66 | #define EXTENT_ENCODED 0x00000008 /* Data can not be read 67 | * while fs is unmounted */ 68 | #define EXTENT_DATA_ENCRYPTED 0x00000080 /* Data is encrypted by fs. 69 | * Sets EXTENT_NO_BYPASS. */ 70 | #define EXTENT_NOT_ALIGNED 0x00000100 /* Extent offsets may not be 71 | * block aligned. */ 72 | #define EXTENT_DATA_INLINE 0x00000200 /* Data mixed with metadata. 73 | * Sets EXTENT_NOT_ALIGNED.*/ 74 | #define EXTENT_DATA_TAIL 0x00000400 /* Multiple files in block. 75 | * Sets EXTENT_NOT_ALIGNED.*/ 76 | #define EXTENT_UNWRITTEN 0x00000800 /* Space allocated, but 77 | * no data (i.e. zero). */ 78 | #define EXTENT_MERGED 0x00001000 /* File does not natively 79 | * support extents. Result 80 | * merged for efficiency. */ 81 | #define EXTENT_SHARED 0x00002000 /* Space shared with other 82 | * files. */ 83 | 84 | #define MAX_EXTENT_LENGTH (1ULL << 60) 85 | 86 | /* Convert a directory pathname */ 87 | int icvt(struct filemapper_t *wf, char *in, size_t inl, char *out, size_t outl); 88 | 89 | /* Run a bunch of queries */ 90 | void run_batch_query(struct filemapper_t *wf, const char *sql); 91 | 92 | /* Insert an inode record into the inode and path tables */ 93 | void insert_inode(struct filemapper_t *wf, int64_t ino, int type, 94 | const char *path, time_t *atime, time_t *crtime, 95 | time_t *ctime, time_t *mtime, int64_t *size); 96 | 97 | /* Insert a directory entry into the database. */ 98 | void insert_dentry(struct filemapper_t *wf, int64_t dir_ino, 99 | const char *name, int64_t ino); 100 | 101 | /* Insert an extent into the database. */ 102 | void insert_extent(struct filemapper_t *wf, int64_t ino, uint64_t physical, 103 | uint64_t *logical, uint64_t length, int flags, int type); 104 | 105 | void inject_metadata(struct filemapper_t *wf, int64_t parent_ino, 106 | const char *path, int64_t ino, const char *name, 107 | int type); 108 | 109 | /* Store fs statistics in the database */ 110 | void collect_fs_stats(struct filemapper_t *wf, char *fs_name, 111 | uint32_t blocksize, uint32_t fragsize, 112 | uint64_t total_bytes, uint64_t free_bytes, 113 | uint64_t total_inodes, uint64_t free_inodes, 114 | unsigned int max_name_len, const char *fstype); 115 | 116 | /* Mark the database as complete. */ 117 | void finalize_fs_stats(struct filemapper_t *wf, char *fs_name); 118 | 119 | /* Generate an overview cache. */ 120 | void cache_overview(struct filemapper_t *wf, uint64_t length); 121 | 122 | /* Prepare database to receive new data. */ 123 | void prepare_db(struct filemapper_t *wf); 124 | 125 | /* Index database. */ 126 | void index_db(struct filemapper_t *wf); 127 | 128 | /* Calculate inode statistics */ 129 | void calc_inode_stats(struct filemapper_t *wf); 130 | 131 | /* Simple bitmap functions */ 132 | int fm_test_bit(const uint8_t *bmap, const uint64_t bit); 133 | void fm_set_bit(uint8_t *bmap, const uint64_t bit, const int new_value); 134 | 135 | #endif /* ifdef FM_H_ */ 136 | -------------------------------------------------------------------------------- /filemapper.in: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | PYTHONPATH="%libdir%:${PYTHONPATH}" 4 | export PYTHONPATH 5 | FM_LIB_DIR="%libdir%" 6 | export FM_LIB_DIR 7 | exec python3 %libdir%/filemapper.py "$@" 8 | -------------------------------------------------------------------------------- /filemapper.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/djwong/filemapper/70c455d028cfe9dad3e0cdfcd30262aa70281a96/filemapper.png -------------------------------------------------------------------------------- /filemapper.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # Generate/view a sqlite database of FS data via FIEMAP or other. 3 | # Copyright (C) 2015 Darrick J. Wong. All rights reserved. 4 | # Licensed under the GPLv2. 5 | 6 | import os 7 | import fmdb 8 | import fmcli 9 | import sys 10 | import argparse 11 | 12 | VER = 'filemapper v0.9.1' 13 | 14 | if __name__ == "__main__": 15 | parser = argparse.ArgumentParser(prog = sys.argv[0], 16 | description = 'Display an overview of commands.') 17 | parser.add_argument('-m', action = 'store_true', help = 'Enable machine-friendly outputs (CLI).') 18 | parser.add_argument('-l', default = 2048, metavar = 'length', type = int, help = 'Initial overview length (CLI).') 19 | parser.add_argument('-r', nargs = 1, metavar = 'fspath', help = 'Analyze a filesystem using the FIEMAP backend.') 20 | parser.add_argument('-q', action = 'store_true', help = 'If -r is specified, exit after analyzing.') 21 | parser.add_argument('-g', action = 'store_true', help = 'Start the GUI.') 22 | parser.add_argument('-s', action = 'store_true', help = 'Generate SQL schemas and index definitions.') 23 | parser.add_argument('database', nargs = '?', default = None, help = 'Database file to store snapshots.') 24 | parser.add_argument('commands', nargs = '*', \ 25 | help = 'Commands to run (CLI). -g cannot be specified.') 26 | args = parser.parse_args(sys.argv[1:]) 27 | 28 | if args.g: 29 | import fmgui 30 | app = fmgui.start_qt() 31 | if args.database == None and args.g: 32 | args.database = fmgui.get_db_fname() 33 | if args.database == '' or args.database == None: 34 | parser.print_help() 35 | print('%s: error: no database file specified.' % sys.argv[0]) 36 | sys.exit(1) 37 | 38 | if args.g and len(args.commands) > 0: 39 | parser.print_help() 40 | print('%s: error: -g cannot be specified with commands to run.' % sys.argv[0]) 41 | sys.exit(1) 42 | 43 | if args.s: 44 | print(fmdb.generate_op_sql()) 45 | print(fmdb.generate_schema_sql()) 46 | print(fmdb.generate_index_sql()) 47 | 48 | if args.r is not None: 49 | fmdb = fmdb.fiemap_db(args.r[0], args.database, True) 50 | fmdb.analyze(True) 51 | fmdb.cache_overview(2048) 52 | fmdb.cache_overview(65536) 53 | fmdb.calc_inode_stats() 54 | if args.q: 55 | sys.exit(0) 56 | else: 57 | fmdb = fmdb.fmdb(None, args.database, os.access(args.database, os.W_OK)) 58 | 59 | if args.g: 60 | fmgui = fmgui.fmgui(fmdb) 61 | app.exec_() 62 | # Qt 5.6 has a race wherein exit() handlers get called 63 | # before QApplication is destroyed, so force a gc run 64 | # in the hopes of racing with the race to get the 65 | # ordering right. Otherwise we segfault. 66 | app = None 67 | import gc 68 | gc.collect(0) 69 | gc.collect(1) 70 | gc.collect(2) 71 | sys.exit(0) 72 | else: 73 | fmdb.set_overview_length(args.l) 74 | fmcli = fmcli.fmcli(fmdb) 75 | fmcli.machine = args.m 76 | for c in args.commands: 77 | fmcli.push(c) 78 | if len(args.commands) == 0: 79 | fmcli.interact(VER) 80 | -------------------------------------------------------------------------------- /fmcli.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # filemapper CLI 3 | # Copyright (C) 2015 Darrick J. Wong 4 | # Licensed under GPLv2. 5 | 6 | import code 7 | import readline 8 | import atexit 9 | import os 10 | import argparse 11 | import sys 12 | import fmdb 13 | from collections import namedtuple 14 | import datetime 15 | from dateutil import tz 16 | 17 | units = namedtuple('units', ['abbrev', 'label', 'factor']) 18 | 19 | # Units for regular numbers 20 | units_none = units('', '', 1) 21 | units_k = units('K', 'K', 10 ** 3) 22 | units_m = units('M', 'M', 10 ** 6) 23 | units_g = units('G', 'G', 10 ** 9) 24 | units_t = units('T', 'T', 10 ** 12) 25 | 26 | # Units for storage quantities 27 | units_bytes = units('', 'bytes', 1) 28 | units_sectors = units('s', 'sectors', 2 ** 9) 29 | units_kib = units('K', 'KiB', 2 ** 10) 30 | units_mib = units('M', 'MiB', 2 ** 20) 31 | units_gib = units('G', 'GiB', 2 ** 30) 32 | units_tib = units('T', 'TiB', 2 ** 40) 33 | 34 | units_auto = units('a', 'auto', None) 35 | 36 | def format_size(units, num): 37 | '''Pretty-format a number with base-2 suffixes.''' 38 | if num is None: 39 | return '' 40 | if units.factor is not None: 41 | if units.factor == 1 and type(num) == int: 42 | return "{:,}{}{}".format(int(num), \ 43 | ' ' if len(units.label) > 0 else '', units.label[:-1] if num == 1 else units.label) 44 | return "{:,.1f} {}".format(float(num) / units.factor, units.label) 45 | units_scale = [units_bytes, units_kib, units_mib, units_gib, units_tib] 46 | for i in range(0, len(units_scale) - 1): 47 | if num < units_scale[i + 1].factor: 48 | return format_size(units_scale[i], num) 49 | return format_size(units_scale[-1], num) 50 | 51 | def format_number(units, num): 52 | '''Pretty-format a number with base-10 suffixes.''' 53 | if num is None: 54 | return '' 55 | if units.factor is not None: 56 | if units.factor == 1 and type(num) == int: 57 | return "{:,}{}{}".format(int(num), \ 58 | ' ' if len(units.label) > 0 else '', units.label) 59 | return "{:,.1f} {}".format(float(num) / units.factor, units.label) 60 | units_scale = [units_none, units_k, units_m, units_g, units_t] 61 | for i in range(0, len(units_scale) - 1): 62 | if num < units_scale[i + 1].factor: 63 | return format_number(units_scale[i], num) 64 | return format_number(units_scale[-1], num) 65 | 66 | def posix_timestamp_str(dt, pretty = False): 67 | '''Generate a string from a datetime object.''' 68 | if dt is None: 69 | return '' 70 | if pretty: 71 | dt = dt.astimezone(fmdb.tz_local) 72 | return dt.strftime('%d-%b-%Y %H:%M:%S') 73 | dt = dt.astimezone(fmdb.tz_gmt) 74 | return dt.isoformat() 75 | 76 | def n2p(maximum, num): 77 | '''Convert a suffixed number to an integer.''' 78 | conv = [ 79 | units('%', 'percent', maximum / 100.0), 80 | units_none, 81 | units_k, 82 | units_m, 83 | units_g, 84 | units_t, 85 | ] 86 | for unit in conv: 87 | if num[-1].lower() == unit.abbrev.lower(): 88 | return int(unit.factor * float(num[:-1])) 89 | return int(num) 90 | 91 | def s2p(fs, num): 92 | '''Convert a suffixed size to an integer.''' 93 | conv = [ 94 | units('%', 'percent', fs.total_bytes / 100.0), 95 | units('B', 'blocks', fs.block_size), 96 | units_bytes, 97 | units_sectors, 98 | units_kib, 99 | units_mib, 100 | units_gib, 101 | units_tib, 102 | ] 103 | for unit in conv: 104 | if num[-1].lower() == unit.abbrev.lower(): 105 | return int(unit.factor * float(num[:-1])) 106 | return int(num) 107 | 108 | def parse_ranges(args, fn): 109 | '''Parse string arguments into numeric ranges.''' 110 | ranges = [] 111 | if 'all' in args: 112 | return ranges 113 | for arg in args: 114 | if ':' in arg: 115 | pos = arg.index(':') 116 | ranges.append((fn(arg[:pos]), fn(arg[pos+1:]))) 117 | elif '-' in arg: 118 | start = 1 if arg[0] == '-' else 0 119 | pos = arg.index('-', start) 120 | ranges.append((fn(arg[:pos]), fn(arg[pos+1:]))) 121 | else: 122 | ranges.append(fn(arg)) 123 | return ranges 124 | 125 | def split_unescape(s, delim, str_delim, escape='\\', unescape=True): 126 | """Split a string into a an argv array, with string support. 127 | >>> split_unescape('foo,bar', ',') 128 | ['foo', 'bar'] 129 | >>> split_unescape('foo$,bar', ',', '$') 130 | ['foo,bar'] 131 | >>> split_unescape('foo$$,bar', ',', '$', unescape=True) 132 | ['foo$', 'bar'] 133 | >>> split_unescape('foo$$,bar', ',', '$', unescape=False) 134 | ['foo$$', 'bar'] 135 | >>> split_unescape('foo$', ',', '$', unescape=True) 136 | ['foo$'] 137 | """ 138 | ret = [] 139 | current = [] 140 | in_str = False 141 | itr = iter(s) 142 | for ch in itr: 143 | if ch == escape: 144 | try: 145 | n = next(itr) 146 | if (in_str and n in str_delim) or \ 147 | (not in_str and (n in delim or n in str_delim)): 148 | if not unescape: 149 | current.append(ch) 150 | current.append(n) 151 | else: 152 | current.append(ch) 153 | current.append(n) 154 | except StopIteration: 155 | current.append(escape) 156 | elif ch in str_delim: 157 | in_str = not in_str 158 | elif ch == delim and not in_str: 159 | # split! (add current to the list and reset it) 160 | if len(current) > 0: 161 | ret.append(''.join(current)) 162 | current = [] 163 | else: 164 | current.append(ch) 165 | if len(current) > 0: 166 | ret.append(''.join(current)) 167 | return ret 168 | 169 | class fmcli(code.InteractiveConsole): 170 | '''Interactive command line client.''' 171 | def __init__(self, fmdb, locals=None, filename="", \ 172 | histfile=os.path.join(os.path.expanduser('~'), '.config', \ 173 | 'fmcli-history')): 174 | # In Python 2.x this didn't inherit from object, so we're 175 | # stuck with the old superclass syntax for now. 176 | # super(fmcli, self).__init__(locals, filename) 177 | code.InteractiveConsole.__init__(self, locals, filename) 178 | self.init_history(histfile) 179 | self.fmdb = fmdb 180 | readline.set_history_length(1000) 181 | self.commands = { 182 | ('cache', 'a'): self.do_cache_overview, 183 | ('clear_calculated', 'cc'): self.do_clear_calculated, 184 | ('calc_inode_stats', 'cs'): self.do_calc_inode_stats, 185 | ('help', 'h', '?'): self.do_help, 186 | ('ls', ): self.do_ls, 187 | ('machine', 'm'): self.do_machine, 188 | ('quit', 'exit', 'q'): self.do_exit, 189 | ('summary', 's'): self.do_summary, 190 | ('units', 'u'): self.do_set_units, 191 | 192 | ('overview', 'o'): self.do_overview, 193 | ('overview_cell', 'oc'): self.do_cell_to_extents, 194 | ('overview_types', 'ot'): self.do_overview_extent_types, 195 | 196 | ('extent_flag', 'ef'): self.do_extent_flag, 197 | ('extent_inode', 'ei'): self.do_inodes, 198 | ('extent_logical', 'el'): self.do_loff_to_extents, 199 | ('extent_physical', 'ep'): self.do_poff_to_extents, 200 | ('extent_length', 'esz'): self.do_lengths, 201 | ('extent_type', 'et'): self.do_extent_type, 202 | ('extent_paths', 'p'): self.do_paths, 203 | 204 | ('inode_extents', 'ie'): self.do_nr_extents, 205 | ('inode_type', 'it'): self.do_inode_type, 206 | ('inode_size', 'isz'): self.do_inode_sizes, 207 | ('inode_travel_score', 'its'): self.do_travel_scores, 208 | ('inode_paths', 'ip'): self.do_paths_stats, 209 | } 210 | self.done = False 211 | self.units = units_auto 212 | self.machine = False 213 | self.fs = self.fmdb.query_summary() 214 | 215 | ## Interpreter stuff 216 | 217 | def init_history(self, histfile): 218 | '''Initializes readline history.''' 219 | readline.parse_and_bind("tab: complete") 220 | if not hasattr(readline, "read_history_file"): 221 | return 222 | try: 223 | readline.read_history_file(histfile) 224 | except IOError: 225 | pass 226 | atexit.register(self.save_history, histfile) 227 | 228 | def save_history(self, histfile): 229 | '''Save readline history to disk.''' 230 | readline.write_history_file(histfile) 231 | 232 | def raw_input(self, prompt): 233 | return code.InteractiveConsole.raw_input(self, 'fm' + prompt) 234 | 235 | def runsource(self, source, filename=''): 236 | args = split_unescape(source, ' ', ('"', "'")) 237 | if len(args) == 0: 238 | return 239 | for key in self.commands: 240 | if args[0] in key: 241 | try: 242 | self.commands[key](args) 243 | except SystemExit: 244 | return 245 | if self.done: 246 | sys.exit(0) 247 | return 248 | print("Command '%s' not recognized." % args[0]) 249 | self.do_help(args) 250 | 251 | ## Utility 252 | 253 | def parse_size_ranges(self, args): 254 | '''Parse string arguments into size ranges.''' 255 | return parse_ranges(args, lambda x: s2p(self.fs, x)) 256 | 257 | def parse_number_ranges(self, args, maximum): 258 | '''Parse string arguments into number ranges.''' 259 | return parse_ranges(args, lambda x: n2p(maximum, x)) 260 | 261 | ## Pretty printers 262 | 263 | def print_extent(self, ext): 264 | '''Pretty-print an extent.''' 265 | if self.machine: 266 | print("'%s',%d,%s,%d,'%s','%s'" % \ 267 | (ext.path if ext.path != '' else self.fs.pathsep, \ 268 | ext.p_off, '' if ext.l_off is None else ext.l_off, \ 269 | ext.length, \ 270 | fmdb.extent_flagstr(ext), \ 271 | fmdb.extent_typestr(ext))) 272 | return 273 | print("'%s', %s, %s, %s, '%s', '%s'" % \ 274 | (ext.path if ext.path != '' else self.fs.pathsep, \ 275 | format_size(self.units, ext.p_off), \ 276 | format_size(self.units, ext.l_off), \ 277 | format_size(self.units, ext.length), \ 278 | fmdb.extent_flagstr(ext), \ 279 | fmdb.extent_typestr(ext))) 280 | 281 | def print_dentry(self, de): 282 | '''Pretty-print a dentry.''' 283 | if self.machine: 284 | print("'%s',%d,'%s'" % \ 285 | (de.name, de.ino, fmdb.dentry_typestr(de))) 286 | return 287 | print("'%s', %s, '%s'" % \ 288 | (de.name, format_number(units_none, de.ino), \ 289 | fmdb.dentry_typestr(de))) 290 | 291 | def print_inode_stats(self, inode): 292 | '''Pretty-print inode statistics.''' 293 | p = inode.path if inode.path != '' else self.fs.pathsep 294 | ts = '' if inode.travel_score is None else '%.02f' % inode.travel_score 295 | nr = '' if inode.nr_extents is None else '%d' % inode.nr_extents 296 | if self.machine: 297 | iss = '' if inode.size is None else inode.size 298 | print("'%s',%d,%s,%s,'%s',%s,%s,%s,%s,%s" % \ 299 | (p, inode.ino, nr, ts, \ 300 | fmdb.inode_typestr(inode), \ 301 | posix_timestamp_str(inode.atime), \ 302 | posix_timestamp_str(inode.crtime), \ 303 | posix_timestamp_str(inode.ctime), \ 304 | posix_timestamp_str(inode.mtime), \ 305 | iss)) 306 | return 307 | print("'%s', %d, %s, %s, %s, '%s', '%s', '%s', '%s', %s" % \ 308 | (p, inode.ino, nr, ts, \ 309 | fmdb.inode_typestr(inode), \ 310 | posix_timestamp_str(inode.atime, True), \ 311 | posix_timestamp_str(inode.crtime, True), \ 312 | posix_timestamp_str(inode.ctime, True), \ 313 | posix_timestamp_str(inode.mtime, True), \ 314 | format_size(self.units, inode.size))) 315 | 316 | ## Misc. Commands 317 | 318 | def do_help(self, argv): 319 | parser = argparse.ArgumentParser(prog = argv[0], 320 | description = 'Display an overview of commands.') 321 | parser.add_argument('commands', nargs = '*', \ 322 | help = 'Commands to look up.') 323 | args = parser.parse_args(argv[1:]) 324 | print_cmds = False 325 | if len(args.commands) == 0: 326 | print_cmds = True 327 | for key in self.commands: 328 | for cmd in args.commands: 329 | if cmd in key: 330 | self.commands[key]([cmd, '-h']) 331 | else: 332 | print_cmds = True 333 | if print_cmds: 334 | print("Available commands:") 335 | for key in sorted(self.commands): 336 | print(key[0]) 337 | 338 | def do_exit(self, argv): 339 | parser = argparse.ArgumentParser(prog = argv[0], 340 | description = 'Exit the program.') 341 | parser.parse_args(argv[1:]) 342 | self.done = True 343 | 344 | def do_summary(self, argv): 345 | parser = argparse.ArgumentParser(prog = argv[0], 346 | description = 'Display a summary of the filesystem.') 347 | parser.parse_args(argv[1:]) 348 | res = self.fs 349 | tb = self.fs.total_bytes 350 | fb = self.fs.free_bytes 351 | if tb == 0: 352 | fb = 1 353 | tb = 1 354 | ti = self.fs.total_inodes 355 | fi = self.fs.free_inodes 356 | if ti == 0: 357 | fi = 1 358 | ti = 1 359 | print("Summary of '%s':" % res.path) 360 | print("Type:\t\t\t%s" % res.fstype) 361 | print("Block size:\t\t%s" % format_size(units_auto, res.block_size)) 362 | print("Fragment size:\t\t%s" % format_size(units_auto, res.frag_size)) 363 | print("Total space:\t\t%s" % format_size(self.units, res.total_bytes)) 364 | print("Used space:\t\t%s (%.0f%%)" % \ 365 | (format_size(self.units, res.total_bytes - res.free_bytes), \ 366 | 100 * (1.0 - (float(fb) / tb)))) 367 | print("Free space:\t\t%s" % format_size(self.units, res.free_bytes)) 368 | print("Total inodes:\t\t%s" % format_number(units_none, res.total_inodes)) 369 | print("Used inodes:\t\t%s (%.0f%%)" % \ 370 | (format_number(units_none, res.total_inodes - res.free_inodes), \ 371 | 100 * (1.0 - (float(fi) / ti)))) 372 | print("Free inodes:\t\t%s" % format_number(units_none, res.free_inodes)) 373 | print("Overview cells:\t\t%s each" % format_size(units_auto, float(res.total_bytes) / self.fmdb.overview_len)) 374 | print("Extents:\t\t%s" % format_number(units_none, res.extents)) 375 | print("Inodes w/ extents:\t%s" % format_number(units_none, res.inodes)) 376 | inodes = res.inodes if res.inodes != 0 else 1 377 | extents = res.extents if res.extents != 0 else 1 378 | extents_blocks = self.fs.extents_bytes / self.fs.block_size if self.fs.extents_bytes != 0 else 1 379 | print("Fragmentation:\t\t%.1f%%" % (100.0 * extents / extents_blocks)) 380 | print("Avg. Travel Score:\t%.02f bytes" % self.fmdb.query_avg_travel_score()) 381 | 382 | def do_set_units(self, argv): 383 | avail_units = [ 384 | units_auto, 385 | units_bytes, 386 | units_sectors, 387 | units('B', 'blocks', self.fs.block_size), 388 | units_kib, 389 | units_mib, 390 | units_gib, 391 | units_tib, 392 | ] 393 | parser = argparse.ArgumentParser(prog = argv[0], 394 | description = 'Set display units.') 395 | unit_list = [x[1] for x in avail_units] 396 | parser.add_argument('units', \ 397 | help = 'Units for display output. Default is bytes.', \ 398 | choices = [x[1] for x in avail_units]) 399 | args = parser.parse_args(argv[1:]) 400 | for u in avail_units: 401 | if args.units.lower() == u.abbrev.lower() or \ 402 | args.units.lower() == u.label.lower(): 403 | self.units = u 404 | print("Units set to '%s'." % self.units.label) 405 | return 406 | print("Unrecognized unit '%s'. Available units:" % args.units) 407 | print(', '.join(unit_list)) 408 | 409 | def do_machine(self, argv): 410 | parser = argparse.ArgumentParser(prog = argv[0], 411 | description = 'Toggle machine-friendly output mode.') 412 | parser.add_argument('mode', choices = ['yes', 'no', 'on', 'off', '1', '0'], \ 413 | help = 'Paths to look up.') 414 | args = parser.parse_args(argv[1:]) 415 | if args.mode == 'yes' or args.mode == 'on' or args.mode == '1': 416 | self.machine = True 417 | else: 418 | self.machine = False 419 | 420 | def do_ls(self, argv): 421 | parser = argparse.ArgumentParser(prog = argv[0], 422 | description = 'Look up directories in the filesystem tree.') 423 | parser.add_argument('dirnames', nargs = '+', \ 424 | help = 'Directory names to look up.') 425 | args = parser.parse_args(argv[1:]) 426 | x = [x if x[-1] != self.fs.pathsep else x[:-1] for x in args.dirnames] 427 | for de in self.fmdb.query_ls(x): 428 | self.print_dentry(de) 429 | 430 | def do_clear_calculated(self, argv): 431 | parser = argparse.ArgumentParser(prog = argv[0], 432 | description = 'Erase all calculated values.') 433 | args = parser.parse_args(argv[1:]) 434 | self.fmdb.clear_calculated_values() 435 | 436 | def do_calc_inode_stats(self, argv): 437 | parser = argparse.ArgumentParser(prog = argv[0], 438 | description = 'Calculate inode statistics.') 439 | parser.add_argument('-f', '--force', help = 'Force recalculation.', action = 'store_true') 440 | args = parser.parse_args(argv[1:]) 441 | self.fmdb.calc_inode_stats(args.force) 442 | 443 | ## Overview management 444 | 445 | def do_overview(self, argv): 446 | parser = argparse.ArgumentParser(prog = argv[0], 447 | description = 'Show the block overview.') 448 | parser.add_argument('blocks', nargs='?', metavar = 'N', \ 449 | type = int, default = None, \ 450 | help = 'Number of blocks to print. Default is 2048.') 451 | args = parser.parse_args(argv[1:]) 452 | if args.blocks is not None: 453 | self.fmdb.set_overview_length(args.blocks) 454 | for ov in self.fmdb.query_overview(): 455 | sys.stdout.write(ov.to_letter()) 456 | sys.stdout.write('\n') 457 | 458 | def do_cache_overview(self, argv): 459 | parser = argparse.ArgumentParser(prog = argv[0], 460 | description = 'Create caches of the overview table.') 461 | parser.add_argument('lengths', nargs = '+', \ 462 | help = 'Lengths of the overview tables.') 463 | args = parser.parse_args(argv[1:]) 464 | for arg in args.lengths: 465 | self.fmdb.cache_overview(arg) 466 | 467 | def do_cell_to_extents(self, argv): 468 | parser = argparse.ArgumentParser(prog = argv[0], 469 | description = 'Look up extents of a given range of overview cells.') 470 | parser.add_argument('cells', nargs = '+', \ 471 | help = 'Cell ranges to look up. This can be a single number or a range (e.g. 0-10).') 472 | args = parser.parse_args(argv[1:]) 473 | ranges = self.parse_number_args(args.cells, self.fmdb.overview_len) 474 | r = list(self.fmdb.pick_cells(ranges)) 475 | for x in self.fmdb.query_poff_range(r): 476 | self.print_extent(x) 477 | 478 | def do_overview_extent_types(self, argv): 479 | parser = argparse.ArgumentParser(prog = argv[0], 480 | description = 'Restrict the overview display to particular types of extents.') 481 | t = [x for x in sorted(fmdb.extent_type_strings.keys())] 482 | t.append('all') 483 | parser.add_argument('types', nargs = '+', \ 484 | help = 'Type codes to look up. Valid values are: (d)irectory, (e)xtent map, (f)ile, FS (m)etadata, (s)ymbolic links, and e(x)tended attributes. Use "all" to display all types.', \ 485 | choices = t) 486 | args = parser.parse_args(argv[1:]) 487 | if 'all' in args: 488 | self.fmdb.set_extent_types_to_show(None) 489 | return 490 | types = set() 491 | for arg in args.types: 492 | types.add(fmdb.extent_type_strings[arg]) 493 | self.fmdb.set_extent_types_to_show(types) 494 | 495 | ## Queries 496 | 497 | def do_poff_to_extents(self, argv): 498 | parser = argparse.ArgumentParser(prog = argv[0], 499 | description = 'Look up extents of a given range of physical offsets.') 500 | parser.add_argument('offsets', nargs = '+', \ 501 | help = 'Physical offsets to look up. This can be a single number or a range (e.g. 0-10k).') 502 | args = parser.parse_args(argv[1:]) 503 | ranges = self.parse_size_ranges(args.offsets) 504 | for x in self.fmdb.query_poff_range(ranges): 505 | self.print_extent(x) 506 | 507 | def do_loff_to_extents(self, argv): 508 | parser = argparse.ArgumentParser(prog = argv[0], 509 | description = 'Look up extents of a given range of logical offsets.') 510 | parser.add_argument('offsets', nargs = '+', \ 511 | help = 'Logical offsets to look up. This can be a single number or a range (e.g. 0-10m).') 512 | args = parser.parse_args(argv[1:]) 513 | ranges = self.parse_size_ranges(args.offsets) 514 | for x in self.fmdb.query_loff_range(ranges): 515 | self.print_extent(x) 516 | 517 | def do_inodes(self, argv): 518 | parser = argparse.ArgumentParser(prog = argv[0], 519 | description = 'Look up extents of a given range of inodes.') 520 | parser.add_argument('inodes', nargs = '+', \ 521 | help = 'Inodes to look up. This can be a single number or a range (e.g. 0-10).') 522 | args = parser.parse_args(argv[1:]) 523 | ranges = self.parse_number_ranges(args.inodes, self.fs.total_inodes) 524 | for x in self.fmdb.query_inums(ranges): 525 | self.print_extent(x) 526 | 527 | def do_paths(self, argv): 528 | parser = argparse.ArgumentParser(prog = argv[0], 529 | description = 'Look up extents of a given path.') 530 | parser.add_argument('paths', nargs = '+', \ 531 | help = 'Paths to look up.') 532 | parser.add_argument('-q', action = 'store_true', help = 'Quiet mode.') 533 | args = parser.parse_args(argv[1:]) 534 | it = self.fmdb.query_paths(args.paths) 535 | if args.q: 536 | x = list(it) 537 | return 538 | for ext in it: 539 | self.print_extent(ext) 540 | 541 | def do_lengths(self, argv): 542 | parser = argparse.ArgumentParser(prog = argv[0], 543 | description = 'Look up extents of a given range of lengths.') 544 | parser.add_argument('lengths', nargs = '+', \ 545 | help = 'Lengths to look up. This can be a single number or a range (e.g. 0-16k).') 546 | args = parser.parse_args(argv[1:]) 547 | ranges = self.parse_size_ranges(args.lengths) 548 | for x in self.fmdb.query_lengths(ranges): 549 | self.print_extent(x) 550 | 551 | def do_extent_type(self, argv): 552 | parser = argparse.ArgumentParser(prog = argv[0], 553 | description = 'Look up extents with a particular type.') 554 | parser.add_argument('types', nargs = '+', \ 555 | help = 'Type codes to look up. Valid values are: (d)irectory, (e)xtent map, (f)ile, FS (m)etadata, (s)ymbolic links, and e(x)tended attributes.', \ 556 | choices = [x for x in sorted(fmdb.extent_type_strings.keys())]) 557 | args = parser.parse_args(argv[1:]) 558 | types = set() 559 | for arg in args.types: 560 | types.add(fmdb.extent_type_strings[arg]) 561 | for x in self.fmdb.query_extent_types(list(types)): 562 | self.print_extent(x) 563 | 564 | def do_extent_flag(self, argv): 565 | parser = argparse.ArgumentParser(prog = argv[0], 566 | description = 'Look up extents with a particular set of flags.') 567 | parser.add_argument('flags', nargs = '*', \ 568 | help = 'Flag codes to look up. Valid values are: u(n)known, (d)elayed allocation, (e)ncoded, (E)ncrypted, (u)naligned, (i)nline, (t)ail-packed, (U)nwritten, (m)erged, (s)hared, or no flag code at all.', \ 569 | choices = [x for x in sorted(fmdb.extent_flags_strings.keys())]) 570 | parser.add_argument('-e', action = 'store_true', help = 'Flags must match exactly.') 571 | args = parser.parse_args(argv[1:]) 572 | flags = 0 573 | for arg in args.flags: 574 | flags |= fmdb.extent_flags_strings[arg] 575 | for x in self.fmdb.query_extent_flags(flags, args.e): 576 | self.print_extent(x) 577 | 578 | def do_paths_stats(self, argv): 579 | parser = argparse.ArgumentParser(prog = argv[0], 580 | description = 'Calculate inode statistics for given paths.') 581 | parser.add_argument('paths', nargs = '+', \ 582 | help = 'Paths to look up.') 583 | parser.add_argument('-q', action = 'store_true', help = 'Quiet mode. Calculate and cache the results, but do not print them.') 584 | args = parser.parse_args(argv[1:]) 585 | i = self.fmdb.query_paths_inodes(args.paths) 586 | if args.q: 587 | list(i) 588 | return 589 | for x in i: 590 | self.print_inode_stats(x) 591 | 592 | def do_travel_scores(self, argv): 593 | parser = argparse.ArgumentParser(prog = argv[0], 594 | description = 'Look up inodes of a given range of travel scores.') 595 | parser.add_argument('scores', nargs = '+', \ 596 | help = 'Scores to look up. This can be a single number or a range (e.g. 0-16k).') 597 | args = parser.parse_args(argv[1:]) 598 | ranges = self.parse_size_ranges(args.scores) 599 | for x in self.fmdb.query_travel_scores_inodes(ranges): 600 | self.print_inode_stats(x) 601 | 602 | def do_nr_extents(self, argv): 603 | parser = argparse.ArgumentParser(prog = argv[0], 604 | description = 'Look up inodes of a given range of primary extent counts. The extent type of a primary extent matches the type of the inode, e.g. file extents for files.') 605 | parser.add_argument('counts', nargs = '+', \ 606 | help = 'Extent counts to look up. This can be a single number or a range (e.g. 0-16k).') 607 | args = parser.parse_args(argv[1:]) 608 | ranges = self.parse_number_ranges(args.counts, 2**64) 609 | for x in self.fmdb.query_nr_extents_inodes(ranges): 610 | self.print_inode_stats(x) 611 | 612 | def do_inode_sizes(self, argv): 613 | parser = argparse.ArgumentParser(prog = argv[0], 614 | description = 'Look up inodes of a given range of inode sizes.') 615 | parser.add_argument('sizes', nargs = '+', \ 616 | help = 'Sizes to look up. This can be a single number or a range (e.g. 0-16k).') 617 | args = parser.parse_args(argv[1:]) 618 | ranges = self.parse_size_ranges(args.sizes) 619 | for x in self.fmdb.query_sizes_inodes(ranges): 620 | self.print_inode_stats(x) 621 | 622 | def do_inode_type(self, argv): 623 | parser = argparse.ArgumentParser(prog = argv[0], 624 | description = 'Look up inodes with a particular type.') 625 | parser.add_argument('types', nargs = '+', \ 626 | help = 'Type codes to look up. Valid values are: (d)irectory, (f)ile, FS (m)etadata, and (s)ymbolic links.', \ 627 | choices = [x for x in sorted(fmdb.inode_type_strings.keys())]) 628 | args = parser.parse_args(argv[1:]) 629 | types = set() 630 | for arg in args.types: 631 | types.add(fmdb.inode_type_strings[arg]) 632 | for x in self.fmdb.query_inode_types_inodes(list(types)): 633 | self.print_inode_stats(x) 634 | -------------------------------------------------------------------------------- /getfsmap.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # Python wrapper of GETFSMAP ioctls. 3 | # Copyright (C) 2017 Darrick J. Wong. All rights reserved. 4 | # Licensed under the GPLv2. 5 | 6 | import struct 7 | import collections 8 | import ioctl 9 | import fcntl 10 | import array 11 | 12 | # Derived from linux/fsmap.h 13 | _struct_fsmap_string = 'LLQQQQQQQ' 14 | _struct_fsmap = struct.Struct('=' + _struct_fsmap_string) 15 | _struct_fsmap_head_string = 'LLLLQQQQQQ' + (2 * _struct_fsmap_string) 16 | _struct_fsmap_head = struct.Struct('=' + _struct_fsmap_head_string) 17 | 18 | FMH_IF_VALID = 0 19 | FMH_OF_DEV_T = 0x1 20 | 21 | FMR_OF_PREALLOC = 0x1 22 | FMR_OF_ATTR_FORK = 0x2 23 | FMR_OF_EXTENT_MAP = 0x4 24 | FMR_OF_SHARED = 0x8 25 | FMR_OF_SPECIAL_OWNER = 0x10 26 | FMR_OF_LAST = 0x20 27 | 28 | def FMR_OWNER(fstype, code): 29 | return (fstype << 32) | (code & 0xFFFFFFFF) 30 | def FMR_OWNER_TYPE(owner): 31 | return owner >> 32 32 | def FMR_OWNER_CODE(owner): 33 | return owner & 0xFFFFFFFF 34 | 35 | FMR_OWN_FREE = FMR_OWNER(0, 1) 36 | FMR_OWN_UNKNOWN = FMR_OWNER(0, 2) 37 | FMR_OWN_METADATA = FMR_OWNER(0, 3) 38 | 39 | _FS_IOC_GETFSMAP = ioctl._IOWR(ord('X'), 59, _struct_fsmap_head) 40 | 41 | fsmap_key = collections.namedtuple('fsmap_key', 42 | 'device flags physical owner offset') 43 | fsmap_rec = collections.namedtuple('fsmap_rec', 44 | 'device flags physical owner offset length hdr_flags') 45 | 46 | def getfsmap(fd, getfsmap_keys = None, count = 10000): 47 | '''Iterable GETFSMAP generator...''' 48 | length = 0 49 | 50 | # Prepare keys 51 | try: 52 | key0 = getfsmap_keys[0] 53 | except: 54 | key0 = fsmap_key(0, 0, 0, 0, 0) 55 | try: 56 | key1 = getfsmap_keys[1] 57 | except: 58 | key1 = fsmap_key(ioctl._UINT32_MAX, ioctl._UINT32_MAX, \ 59 | ioctl._UINT64_MAX, ioctl._UINT64_MAX, \ 60 | ioctl._UINT64_MAX) 61 | 62 | while True: 63 | hdr = _struct_fsmap_head.pack(0, 0, count, 0, 0, 0, 0, 0, 0, 0, \ 64 | key0.device, key0.flags, key0.physical, \ 65 | key0.owner, key0.offset, length, 0, 0, 0, \ 66 | key1.device, key1.flags, key1.physical, \ 67 | key1.owner, key1.offset, 0, 0, 0, 0) 68 | buf = bytearray(hdr) + bytearray(_struct_fsmap.size * count) 69 | try: 70 | ret = fcntl.ioctl(fd, _FS_IOC_GETFSMAP, buf) 71 | except TypeError: 72 | # Turn into mutable C-level array of chars 73 | s = '%s%s' % (hdr, '\0' * (_struct_fsmap.size * count)) 74 | buf = array.array('c', s) 75 | ret = fcntl.ioctl(fd, _FS_IOC_GETFSMAP, buf) 76 | if ret < 0: 77 | raise IOError('GETFSMAP') 78 | 79 | meh = _struct_fsmap_head.unpack_from(buf) 80 | oflags = meh[1] 81 | entries = meh[3] 82 | if entries == 0: 83 | return 84 | 85 | bufsz = _struct_fsmap_head.size + (_struct_fsmap.size * entries) 86 | assert len(buf) >= bufsz 87 | for offset in range(_struct_fsmap_head.size, bufsz, _struct_fsmap.size): 88 | x = _struct_fsmap.unpack_from(buf, offset) 89 | rec = fsmap_rec(x[0], x[1], x[2], x[3], x[4], x[5], oflags) 90 | yield rec 91 | 92 | if rec.flags & FMR_OF_LAST: 93 | return 94 | key0 = rec 95 | length = rec.length 96 | 97 | XFS_FMR_OWN_TYPE = ord('X') 98 | XFS_FMR_OWN_FS = FMR_OWNER(XFS_FMR_OWN_TYPE, 1) 99 | XFS_FMR_OWN_LOG = FMR_OWNER(XFS_FMR_OWN_TYPE, 2) 100 | XFS_FMR_OWN_AG = FMR_OWNER(XFS_FMR_OWN_TYPE, 3) 101 | XFS_FMR_OWN_INOBT = FMR_OWNER(XFS_FMR_OWN_TYPE, 4) 102 | XFS_FMR_OWN_INODES = FMR_OWNER(XFS_FMR_OWN_TYPE, 5) 103 | XFS_FMR_OWN_REFC = FMR_OWNER(XFS_FMR_OWN_TYPE, 6) 104 | XFS_FMR_OWN_COW = FMR_OWNER(XFS_FMR_OWN_TYPE, 7) 105 | XFS_FMR_OWN_DEFECTIVE = FMR_OWNER(XFS_FMR_OWN_TYPE, 8) 106 | 107 | EXT4_FMR_OWN_TYPE = ord('f') 108 | EXT4_FMR_OWN_GDT = FMR_OWNER(EXT4_FMR_OWN_TYPE, 1) 109 | EXT4_FMR_OWN_RESV_GDT = FMR_OWNER(EXT4_FMR_OWN_TYPE, 2) 110 | EXT4_FMR_OWN_BLKBM = FMR_OWNER(EXT4_FMR_OWN_TYPE, 3) 111 | EXT4_FMR_OWN_INOBM = FMR_OWNER(EXT4_FMR_OWN_TYPE, 4) 112 | 113 | special_owner_types = { 114 | XFS_FMR_OWN_TYPE: 'xfs', 115 | EXT4_FMR_OWN_TYPE: 'ext4', 116 | } 117 | 118 | special_owner_codes = { 119 | XFS_FMR_OWN_FS: 'fsdata', 120 | XFS_FMR_OWN_LOG: 'log', 121 | XFS_FMR_OWN_AG: 'bnobt-cntbt-rmapbt', 122 | XFS_FMR_OWN_INOBT: 'inobt', 123 | XFS_FMR_OWN_INODES: 'inodes', 124 | XFS_FMR_OWN_REFC: 'refcountbt', 125 | XFS_FMR_OWN_COW: 'cow', 126 | XFS_FMR_OWN_DEFECTIVE: 'defective', 127 | EXT4_FMR_OWN_GDT: 'group_descriptors', 128 | EXT4_FMR_OWN_RESV_GDT: 'reserved_gdt_blocks', 129 | EXT4_FMR_OWN_BLKBM: 'block_bitmap', 130 | EXT4_FMR_OWN_INOBM: 'inode_bitmap', 131 | } 132 | 133 | def special_owner_name(owner): 134 | '''Formulate a name for a special owner.''' 135 | t = FMR_OWNER_TYPE(owner) 136 | c = FMR_OWNER_CODE(owner) 137 | if owner in special_owner_codes: 138 | return '%s:%s' % (special_owner_types[t], \ 139 | special_owner_codes[owner]) 140 | return '%d:%d' % (t, c) 141 | 142 | if __name__ == '__main__': 143 | import sys 144 | import pprint 145 | 146 | if len(sys.argv) < 2: 147 | sys.stderr.write('No filename(s) given\n') 148 | sys.exit(1) 149 | 150 | for file_ in sys.argv[1:]: 151 | with open(file_, 'r') as fd: 152 | for fmr in getfsmap(fd): 153 | pprint.pprint(fmr) 154 | -------------------------------------------------------------------------------- /ioctl.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # Python wrapper of ioctl support code. 3 | # Copyright (C) 2017 Darrick J. Wong. All rights reserved. 4 | # Licensed under the GPLv2. 5 | 6 | # Internals and plumbing 7 | # From asm-generic/ioctl.h 8 | _IOC_NRBITS = 8 9 | _IOC_TYPEBITS = 8 10 | _IOC_SIZEBITS = 14 11 | 12 | _IOC_NRSHIFT = 0 13 | _IOC_TYPESHIFT = _IOC_NRSHIFT + _IOC_NRBITS 14 | _IOC_SIZESHIFT = _IOC_TYPESHIFT + _IOC_TYPEBITS 15 | _IOC_DIRSHIFT = _IOC_SIZESHIFT + _IOC_SIZEBITS 16 | 17 | _IOC_TYPECHECK = lambda struct: struct.size 18 | 19 | _IOC = lambda dir_, type_, nr, size: \ 20 | (dir_ << _IOC_DIRSHIFT) | (type_ << _IOC_TYPESHIFT) | \ 21 | (nr << _IOC_NRSHIFT) | (size << _IOC_SIZESHIFT) 22 | _IOC_NONE = 0 23 | _IOC_WRITE = 1 24 | _IOC_READ = 2 25 | _IOWR = lambda type_, nr, size: \ 26 | _IOC(_IOC_READ | _IOC_WRITE, type_, nr, _IOC_TYPECHECK(size)) 27 | _IO = lambda type_, nr: \ 28 | _IOC(_IOC_NONE, type_, nr, 0) 29 | 30 | _UINT32_MAX = (2 ** 32) - 1 31 | _UINT64_MAX = (2 ** 64) - 1 32 | -------------------------------------------------------------------------------- /mkdoscode: -------------------------------------------------------------------------------- 1 | #!/bin/sh -x 2 | 3 | # Create dosfs.[ch] from dosfstools... 4 | 5 | if [ ! -d "$1" ]; then 6 | echo "Usage: $0 dosfstools-src-dir" 7 | exit 1 8 | fi 9 | 10 | dir="${PWD}" 11 | cd "$1" 12 | 13 | cat fsck.fat.h boot.h charconv.h check.h common.h fat.h file.h io.h lfn.h version.h > "${dir}/dosfs.h" 14 | cat >> "${dir}/dosfs.h" << ENDL 15 | #ifndef FAT_EXTRA_H_ 16 | #define FAT_EXTRA_H_ 17 | void add_file(DOS_FS * fs, DOS_FILE *** chain, DOS_FILE * parent, loff_t offset, FDSC ** cp); 18 | #endif 19 | ENDL 20 | cat > "${dir}/dosfs.c" << ENDL 21 | #define _XOPEN_SOURCE 600 22 | #define _FILE_OFFSET_BITS 64 23 | #define _LARGEFILE64_SOURCE 1 24 | #define _GNU_SOURCE 1 25 | #include "dosfs.h" 26 | ENDL 27 | cat boot.c charconv.c check.c common.c fat.c file.c io.c lfn.c | sed -e '/^#include "/d' -e '/^#define _LARGEFILE64_SOURCE/d' -e 's/static void add_file/void add_file/g' >> "${dir}/dosfs.c" 28 | exit 1 29 | -------------------------------------------------------------------------------- /ntfsmapper.1: -------------------------------------------------------------------------------- 1 | .\" Hey, EMACS: -*- nroff -*- 2 | .\" (C) Copyright 2015 Darrick J. Wong , 3 | .\" 4 | .\" First parameter, NAME, should be all caps 5 | .\" Second parameter, SECTION, should be 1-8, maybe w/ subsection 6 | .\" other parameters are allowed: see man(7), man(1) 7 | .TH FILEMAPPER 1 "February 1, 2015" 8 | .\" Please adjust this date whenever revising the manpage. 9 | .\" 10 | .\" Some roff macros, for reference: 11 | .\" .nh disable hyphenation 12 | .\" .hy enable hyphenation 13 | .\" .ad l left justify 14 | .\" .ad b justify to both left and right margins 15 | .\" .nf disable filling 16 | .\" .fi enable filling 17 | .\" .br insert line break 18 | .\" .sp insert n+1 empty lines 19 | .\" for manpage-specific macros, see man(7) 20 | .SH NAME 21 | ntfsmapper \- Analyze an NTFS image for use with filemapper 22 | .SH SYNOPSIS 23 | .B filemapper 24 | .I dbfile 25 | .I fs_device 26 | .SH DESCRIPTION 27 | This manual page documents briefly the 28 | .B ntfsmapper 29 | command. 30 | .PP 31 | .\" TeX users may be more comfortable with the \fB\fP and 32 | .\" \fI\fP escape sequences to invode bold face and italics, 33 | .\" respectively. 34 | \fBntfsmapper\fP analyzes the internals of an NTFS filesystem and 35 | generates a snapshot that the filemapper program can use to visualize 36 | and query the physical layout of the filesystem. 37 | .SH OPTIONS 38 | These programs follow the usual GNU command line syntax, with long 39 | options starting with two dashes (`-'). 40 | A summary of options is included below. 41 | For a complete description, see the Info files. 42 | .TP 43 | .I dbfile 44 | Store the filesystem analysis in this file. 45 | .TP 46 | .I fs_device 47 | Open this raw device for analysis. 48 | .TP 49 | .B \-h, \-\-help 50 | Show summary of options. 51 | .SH SEE ALSO 52 | .BR filemapper (1). 53 | .br 54 | -------------------------------------------------------------------------------- /ntfsmapper.c: -------------------------------------------------------------------------------- 1 | /* 2 | * Generate filemapper databases from ntfs filesystems. 3 | * Copyright 2015 Darrick J. Wong. 4 | * Licensed under the GPLv2. 5 | */ 6 | #define _XOPEN_SOURCE 600 7 | #define _FILE_OFFSET_BITS 64 8 | #define _LARGEFILE64_SOURCE 1 9 | #define _GNU_SOURCE 1 10 | 11 | #include 12 | #include 13 | #include 14 | #include 15 | #include 16 | #include 17 | #include 18 | #include 19 | #include 20 | #include 21 | #include 22 | #include 23 | #include 24 | #include 25 | #undef DEBUG 26 | #include "filemapper.h" 27 | #include "compdb.h" 28 | 29 | struct ntfsmap_t { 30 | struct filemapper_t base; 31 | 32 | ntfs_volume *fs; 33 | int err; 34 | u64 dir_ino; 35 | u64 total_inodes; 36 | u8 *ino_bmap; 37 | }; 38 | #define wf_db base.db 39 | #define wf_db_err base.db_err 40 | #define wf_dirpath base.dirpath 41 | #define wf_iconv base.iconv 42 | 43 | /* Fake inodes for metadata */ 44 | 45 | #define INO_METADATA_DIR (-1) 46 | #define STR_METADATA_DIR METADATA_DIR 47 | #define INO_FREESP_FILE (-2) 48 | #define STR_FREESP_FILE FREESP_FILE 49 | 50 | /* These bits are more or less copied from ntfsprogs. */ 51 | 52 | static const char *invalid_ntfs_msg = 53 | "The device '%s' doesn't have a valid NTFS.\n" 54 | "Maybe you selected the wrong device? Or the whole disk instead of a\n" 55 | "partition (e.g. /dev/hda, not /dev/hda1)? Or the other way around?\n"; 56 | 57 | static const char *corrupt_volume_msg = 58 | "NTFS is inconsistent. Run chkdsk /f on Windows then reboot it TWICE!\n" 59 | "The usage of the /f parameter is very IMPORTANT! No modification was\n" 60 | "made to NTFS by this software.\n"; 61 | 62 | static const char *hibernated_volume_msg = 63 | "The NTFS partition is hibernated. Please resume Windows and turned it \n" 64 | "off properly, so mounting could be done safely.\n"; 65 | 66 | static const char *unclean_journal_msg = 67 | "Access is denied because the NTFS journal file is unclean. Choices are:\n" 68 | " A) Shutdown Windows properly.\n" 69 | " B) Click the 'Safely Remove Hardware' icon in the Windows taskbar\n" 70 | " notification area before disconnecting the device.\n" 71 | " C) Use 'Eject' from Windows Explorer to safely remove the device.\n" 72 | " D) If you ran chkdsk previously then boot Windows again which will\n" 73 | " automatically initialize the journal.\n" 74 | " E) Submit 'force' option (WARNING: This solution it not recommended).\n" 75 | " F) ntfsmount: Mount the volume read-only by using the 'ro' mount option.\n"; 76 | 77 | static const char *opened_volume_msg = 78 | "Access is denied because the NTFS volume is already exclusively opened.\n" 79 | "The volume may be already mounted, or another software may use it which\n" 80 | "could be identified for example by the help of the 'fuser' command.\n"; 81 | 82 | static const char *fakeraid_msg = 83 | "You seem to have a SoftRAID/FakeRAID hardware and must use an activated,\n" 84 | "different device under /dev/mapper, (e.g. /dev/mapper/nvidia_eahaabcc1)\n" 85 | "to mount NTFS. Please see the 'dmraid' documentation for help.\n"; 86 | 87 | /* Map NTFS attributes to extent types */ 88 | static int extent_codes(ntfs_inode *inode, int attr_type) 89 | { 90 | if (MREF(inode->mft_no) < FILE_first_user) 91 | return EXT_TYPE_METADATA; 92 | 93 | switch (attr_type) { 94 | case AT_EA: 95 | case AT_EA_INFORMATION: 96 | case AT_ATTRIBUTE_LIST: 97 | return EXT_TYPE_XATTR; 98 | case AT_DATA: 99 | return EXT_TYPE_FILE; 100 | case AT_INDEX_ALLOCATION: 101 | return EXT_TYPE_DIR; 102 | case AT_BITMAP: 103 | case AT_REPARSE_POINT: 104 | case AT_LOGGED_UTILITY_STREAM: 105 | return EXT_TYPE_METADATA; 106 | default: 107 | abort(); 108 | } 109 | } 110 | 111 | /* Walk a file's mappings for extents */ 112 | static void walk_file_mappings(struct ntfsmap_t *wf, ntfs_inode *inode) 113 | { 114 | ntfs_attr_search_ctx *ctx; 115 | runlist *runs = NULL, *r; 116 | unsigned long long p_block, l_block, e_len; 117 | unsigned long long max_extent = MAX_EXTENT_LENGTH / wf->fs->cluster_size; 118 | uint64_t loff; 119 | 120 | if (ntfs_bit_get(wf->ino_bmap, inode->mft_no)) 121 | return; 122 | 123 | ctx = ntfs_attr_get_search_ctx(inode, NULL); 124 | if (!ctx) { 125 | wf->err = errno; 126 | return; 127 | } 128 | 129 | while (!ntfs_attr_lookup(AT_UNUSED, NULL, 0, 0, 0, NULL, 0, ctx)) { 130 | if (!ctx->attr->non_resident) 131 | continue; 132 | runs = ntfs_mapping_pairs_decompress(wf->fs, ctx->attr, NULL); 133 | if (!runs) { 134 | wf->err = errno; 135 | goto out; 136 | } 137 | p_block = l_block = e_len = 0; 138 | for (r = runs; r->length > 0; r++) { 139 | if (r->lcn < 0) 140 | continue; 141 | if (e_len > 0) { 142 | if (p_block + e_len == r->lcn && 143 | l_block + e_len == r->vcn && 144 | e_len + r->length <= max_extent) { 145 | e_len += r->length; 146 | dbg_printf("R: ino=%d len=%u\n", 147 | inode->mft_no, e_len); 148 | continue; 149 | } 150 | 151 | dbg_printf("R: ino=%"PRIu64" type=0x%x vcn=%"PRIu64" lcn=%"PRIu64" len=%"PRIu64"\n", 152 | inode->mft_no, ctx->attr->type, 153 | p_block, l_block, e_len); 154 | loff = l_block * wf->fs->cluster_size; 155 | insert_extent(&wf->base, inode->mft_no, 156 | p_block * wf->fs->cluster_size, 157 | &loff, 158 | e_len * wf->fs->cluster_size, 159 | 0, 160 | extent_codes(inode, ctx->attr->type)); 161 | if (wf->wf_db_err) 162 | goto out; 163 | } 164 | p_block = r->lcn; 165 | l_block = r->vcn; 166 | e_len = r->length; 167 | } 168 | 169 | if (e_len > 0) { 170 | dbg_printf("R: ino=%"PRIu64" type=0x%x vcn=%"PRIu64" lcn=%"PRIu64" len=%"PRIu64"\n", 171 | inode->mft_no, ctx->attr->type, 172 | p_block, l_block, e_len); 173 | loff = l_block * wf->fs->cluster_size; 174 | insert_extent(&wf->base, inode->mft_no, 175 | p_block * wf->fs->cluster_size, 176 | &loff, 177 | e_len * wf->fs->cluster_size, 178 | 0, 179 | extent_codes(inode, ctx->attr->type)); 180 | if (wf->wf_db_err) 181 | goto out; 182 | } 183 | 184 | free(runs); 185 | runs = NULL; 186 | } 187 | 188 | out: 189 | free(runs); 190 | ntfs_bit_set(wf->ino_bmap, inode->mft_no, 1); 191 | ntfs_attr_put_search_ctx(ctx); 192 | return; 193 | } 194 | 195 | /* Handle a directory entry */ 196 | static int walk_fs_helper(void *priv_data, const ntfschar * de_name, 197 | const int de_name_len, const int name_type, 198 | const s64 pos, const MFT_REF mref, 199 | const unsigned dt_type) 200 | { 201 | char path[PATH_MAX + 1]; 202 | char name[NTFS_MAX_NAME_LEN + 1], *p; 203 | int type; 204 | struct ntfsmap_t *wf = priv_data; 205 | ntfs_inode *ni = NULL; 206 | time_t atime, crtime, ctime, mtime; 207 | struct timespec ts; 208 | u64 parent_ino; 209 | 210 | /* Skip the 8.3 names */ 211 | if ((name_type & FILE_NAME_WIN32_AND_DOS) == FILE_NAME_DOS) 212 | return 0; 213 | 214 | p = name; 215 | if (de_name) { 216 | if (ntfs_ucstombs(de_name, de_name_len, &p, NTFS_MAX_NAME_LEN) < 0) { 217 | wf->err = errno; 218 | ntfs_log_error("Cannot represent filename in locale."); 219 | return -1; 220 | } 221 | } else { 222 | name[0] = 0; 223 | } 224 | 225 | if (!strcmp(name, ".") || !strcmp(name, "..")) 226 | return 0; 227 | 228 | ni = ntfs_inode_open(wf->fs, mref); 229 | if (!ni) { 230 | wf->err = errno; 231 | return -1; 232 | } 233 | 234 | if (de_name && MREF(mref) < FILE_first_user && dt_type == NTFS_DT_REG) { 235 | type = INO_TYPE_METADATA; 236 | goto have_type; 237 | } 238 | 239 | switch (dt_type) { 240 | case NTFS_DT_REG: 241 | type = INO_TYPE_FILE; 242 | break; 243 | case NTFS_DT_DIR: 244 | type = INO_TYPE_DIR; 245 | break; 246 | case NTFS_DT_LNK: 247 | type = INO_TYPE_SYMLINK; 248 | break; 249 | default: 250 | ntfs_inode_close(ni); 251 | return 0; 252 | } 253 | 254 | have_type: 255 | dbg_printf("dir=%"PRIu64" name=%s/%s nametype=0x%x ino=%"PRIu64" type=%d:%d\n", 256 | wf->dir_ino, wf->wf_dirpath, name, name_type, ni->mft_no, type, dt_type); 257 | 258 | ts = ntfs2timespec(ni->last_access_time); 259 | atime = ts.tv_sec; 260 | ts = ntfs2timespec(ni->creation_time); 261 | crtime = ts.tv_sec; 262 | ts = ntfs2timespec(ni->last_mft_change_time); 263 | ctime = ts.tv_sec; 264 | ts = ntfs2timespec(ni->last_data_change_time); 265 | mtime = ts.tv_sec; 266 | 267 | 268 | if (de_name) 269 | snprintf(path, PATH_MAX, "%s/%s", wf->wf_dirpath, name); 270 | else 271 | path[0] = 0; 272 | if (MREF(mref) < FILE_first_user && wf->dir_ino == FILE_root) { 273 | parent_ino = INO_METADATA_DIR; 274 | snprintf(path, PATH_MAX, "/%s/%s", STR_METADATA_DIR, name); 275 | } else 276 | parent_ino = wf->dir_ino; 277 | 278 | dbg_printf("parent_ino = %lld path = %s\n", parent_ino, path); 279 | insert_inode(&wf->base, ni->mft_no, type, path, &atime, &crtime, &ctime, 280 | &mtime, &ni->data_size); 281 | if (wf->wf_db_err) 282 | goto err; 283 | if (de_name) 284 | insert_dentry(&wf->base, parent_ino, name, ni->mft_no); 285 | if (wf->wf_db_err) 286 | goto err; 287 | 288 | walk_file_mappings(wf, ni); 289 | if (wf->err || wf->wf_db_err) 290 | goto err; 291 | 292 | if (type == INO_TYPE_DIR) { 293 | const char *old_dirpath; 294 | u64 old_dir_ino; 295 | s64 pos = 0; 296 | int err; 297 | 298 | old_dir_ino = wf->dir_ino; 299 | old_dirpath = wf->wf_dirpath; 300 | wf->wf_dirpath = path; 301 | wf->dir_ino = ni->mft_no; 302 | err = ntfs_readdir(ni, &pos, wf, walk_fs_helper); 303 | if (!wf->err) 304 | wf->err = err; 305 | wf->wf_dirpath = old_dirpath; 306 | wf->dir_ino = old_dir_ino; 307 | } 308 | if (wf->err || wf->wf_db_err) 309 | goto err; 310 | 311 | ntfs_inode_close(ni); 312 | return 0; 313 | err: 314 | ntfs_inode_close(ni); 315 | return -1; 316 | } 317 | 318 | /* Walk the whole FS, looking for inodes to analyze. */ 319 | static void walk_fs(struct ntfsmap_t *wf) 320 | { 321 | wf->wf_dirpath = ""; 322 | wf->ino_bmap = calloc(1, wf->total_inodes / 8); 323 | if (!wf->ino_bmap) { 324 | wf->err = ENOMEM; 325 | return; 326 | } 327 | 328 | inject_metadata(&wf->base, FILE_root, "", INO_METADATA_DIR, 329 | STR_METADATA_DIR, INO_TYPE_DIR); 330 | if (wf->wf_db_err) 331 | return; 332 | 333 | walk_fs_helper(wf, NULL, 0, FILE_NAME_WIN32, 0, FILE_root, NTFS_DT_DIR); 334 | 335 | free(wf->ino_bmap); 336 | wf->ino_bmap = NULL; 337 | } 338 | 339 | /* Walk a byte in the bitmap... */ 340 | #define CL(c) ((c) * wf->fs->cluster_size) 341 | static inline void walk_block_bitmap_byte(struct ntfsmap_t *wf, 342 | unsigned char b, long long blknow, 343 | long long *freestart) 344 | { 345 | int j; 346 | 347 | switch (b) { 348 | case 0: 349 | if (*freestart < 0) 350 | *freestart = blknow; 351 | break; 352 | case 255: 353 | if (*freestart >= 0) { 354 | insert_extent(&wf->base, INO_FREESP_FILE, 355 | CL(*freestart), NULL, 356 | CL(blknow - *freestart), 0, 357 | EXT_TYPE_FREESP); 358 | dbg_printf("freestart %llu blknow %llu\n", 359 | *freestart, blknow); 360 | *freestart = -1; 361 | } 362 | break; 363 | default: 364 | for (j = 0; j < 8; j++, blknow++) { 365 | if ( (b & (1 << j)) && *freestart >= 0) { 366 | insert_extent(&wf->base, INO_FREESP_FILE, 367 | CL(*freestart), NULL, 368 | CL(blknow - *freestart), 369 | 0, EXT_TYPE_FREESP); 370 | dbg_printf("freestart %llu blknow %llu\n", 371 | *freestart, blknow); 372 | *freestart = -1; 373 | } else if ( !(b & (1 << j)) && *freestart < 0) { 374 | *freestart = blknow; 375 | } 376 | } 377 | break; 378 | } 379 | } 380 | 381 | /* Read bitmap? */ 382 | #define BITMAP_LEN 65536 383 | static void walk_block_bitmap(struct ntfsmap_t *wf) 384 | { 385 | ntfs_attr *bmp_na = wf->fs->lcnbmp_na; 386 | unsigned char buf[BITMAP_LEN]; 387 | unsigned char *b; 388 | long long bmp_offset = 0; 389 | long long br; 390 | long long freestart; 391 | long long blk; 392 | long long blknow; 393 | 394 | inject_metadata(&wf->base, INO_METADATA_DIR, "/" STR_METADATA_DIR, 395 | INO_FREESP_FILE, STR_FREESP_FILE, INO_TYPE_FREESP); 396 | if (wf->wf_db_err) 397 | return; 398 | 399 | freestart = -1; 400 | while (1) { 401 | br = ntfs_attr_pread(bmp_na, bmp_offset, BITMAP_LEN, buf); 402 | if (br <= 0) 403 | break; 404 | 405 | /* Look for zero bits == free clusters. */ 406 | for (b = buf, blk = 0; b < buf + br; b++, blk += 8) { 407 | blknow = (bmp_offset << 3) + blk; 408 | walk_block_bitmap_byte(wf, *b, blknow, &freestart); 409 | } 410 | 411 | bmp_offset += br; 412 | } 413 | if (freestart >= 0) { 414 | insert_extent(&wf->base, INO_FREESP_FILE, 415 | CL(freestart), NULL, 416 | CL(blknow - freestart), 417 | 0, EXT_TYPE_FREESP); 418 | dbg_printf("freestart %llu eofs %llu\n", freestart, 419 | wf->fs->nr_clusters); 420 | } 421 | } 422 | #undef CL 423 | 424 | #define CHECK_ERROR(msg) \ 425 | do { \ 426 | if (wf.err) { \ 427 | ntfs_log_error("%s %s", strerror(errno), (msg)); \ 428 | goto out; \ 429 | } \ 430 | if (wf.wf_db_err) { \ 431 | ntfs_log_error("%s %s", sqlite3_errstr(wf.wf_db_err), (msg)); \ 432 | goto out; \ 433 | } \ 434 | } while (0); 435 | 436 | int main(int argc, char *argv[]) 437 | { 438 | const char *dbfile; 439 | const char *fsdev; 440 | char *errm; 441 | struct ntfsmap_t wf; 442 | sqlite3 *db = NULL; 443 | ntfs_volume *fs = NULL; 444 | int db_err = 0; 445 | uint64_t total_bytes, size; 446 | int err = 0, err2, delta_bits; 447 | 448 | if (argc != 3) { 449 | printf("Usage: %s dbfile fsdevice\n", argv[0]); 450 | return 0; 451 | } 452 | 453 | /* Open things */ 454 | memset(&wf, 0, sizeof(wf)); 455 | dbfile = argv[1]; 456 | fsdev = argv[2]; 457 | 458 | db_err = truncate(dbfile, 0); 459 | if (db_err && errno != ENOENT) { 460 | perror(dbfile); 461 | goto out; 462 | } 463 | 464 | ntfs_log_set_handler(ntfs_log_handler_stderr); 465 | 466 | fs = ntfs_mount(fsdev, NTFS_MNT_RDONLY | NTFS_MNT_FORENSIC); 467 | if (!fs) { 468 | ntfs_log_perror("Failed to mount '%s'", fsdev); 469 | if (errno == EINVAL) 470 | ntfs_log_error(invalid_ntfs_msg, fsdev); 471 | else if (errno == EIO) 472 | ntfs_log_error("%s", corrupt_volume_msg); 473 | else if (errno == EPERM) 474 | ntfs_log_error("%s", hibernated_volume_msg); 475 | else if (errno == EOPNOTSUPP) 476 | ntfs_log_error("%s", unclean_journal_msg); 477 | else if (errno == EBUSY) 478 | ntfs_log_error("%s", opened_volume_msg); 479 | else if (errno == ENXIO) 480 | ntfs_log_error("%s", fakeraid_msg); 481 | goto out; 482 | } 483 | 484 | total_bytes = fs->nr_clusters * fs->cluster_size; 485 | err = compdb_register("unix-excl", "comp-unix-excl", 486 | total_bytes > 100000000000ULL ? "LZMA" : "GZIP"); 487 | if (err) { 488 | ntfs_log_error("%s while setting up compressed db", 489 | sqlite3_errstr(err)); 490 | goto out; 491 | } 492 | 493 | err = sqlite3_open_v2(dbfile, &db, 494 | SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE, 495 | "comp-unix-excl"); 496 | if (err) { 497 | ntfs_log_error("%s while opening database", 498 | sqlite3_errstr(err)); 499 | goto out; 500 | } 501 | 502 | ntfs_set_char_encoding("utf8"); 503 | wf.wf_iconv = iconv_open("UTF-8", "UTF-8"); 504 | wf.fs = fs; 505 | wf.wf_db = db; 506 | 507 | /* Prepare and clean out database. */ 508 | prepare_db(&wf.base); 509 | CHECK_ERROR("while preparing database"); 510 | wf.wf_db_err = sqlite3_exec(db, "BEGIN TRANSACTION", NULL, NULL, &errm); 511 | if (errm) { 512 | ntfs_log_error("%s while starting transaction", errm); 513 | free(errm); 514 | goto out; 515 | } 516 | CHECK_ERROR("while starting fs analysis database transaction"); 517 | 518 | ntfs_volume_get_free_space(fs); 519 | total_bytes = fs->nr_clusters * fs->cluster_size; 520 | 521 | /* Free inodes on the free space */ 522 | size = fs->free_clusters; 523 | delta_bits = fs->cluster_size_bits - fs->mft_record_size_bits; 524 | if (delta_bits >= 0) 525 | size <<= delta_bits; 526 | else 527 | size >>= -delta_bits; 528 | 529 | /* Number of inodes at this point in time. */ 530 | wf.total_inodes = (fs->mftbmp_na->allocated_size << 3) + size; 531 | 532 | /* Free inodes available for all and for non-privileged processes. */ 533 | size += fs->free_mft_records; 534 | if (size < 0) 535 | size = 0; 536 | 537 | collect_fs_stats(&wf.base, fs->dev->d_name, fs->cluster_size, 538 | fs->cluster_size, total_bytes, 539 | fs->free_clusters * fs->cluster_size, 540 | wf.total_inodes, size, NTFS_MAX_NAME_LEN, "NTFS"); 541 | CHECK_ERROR("while storing fs stats"); 542 | 543 | /* Walk the filesystem */ 544 | walk_fs(&wf); 545 | CHECK_ERROR("while analyzing filesystem"); 546 | walk_block_bitmap(&wf); 547 | CHECK_ERROR("while analyzing free space"); 548 | 549 | /* Generate indexes and finalize. */ 550 | index_db(&wf.base); 551 | CHECK_ERROR("while indexing database"); 552 | finalize_fs_stats(&wf.base, fs->dev->d_name); 553 | CHECK_ERROR("while finalizing database"); 554 | calc_inode_stats(&wf.base); 555 | CHECK_ERROR("while calculating inode statistics"); 556 | 557 | wf.wf_db_err = sqlite3_exec(db, "END TRANSACTION", NULL, NULL, &errm); 558 | if (errm) { 559 | fprintf(stderr, "%s %s", errm, "while ending transaction"); 560 | free(errm); 561 | goto out; 562 | } 563 | CHECK_ERROR("while flushing fs analysis database transaction"); 564 | 565 | wf.wf_db_err = sqlite3_exec(db, "BEGIN TRANSACTION", NULL, NULL, &errm); 566 | if (errm) { 567 | fprintf(stderr, "%s %s", errm, "while starting transaction"); 568 | free(errm); 569 | goto out; 570 | } 571 | CHECK_ERROR("while starting overview cache database transaction"); 572 | 573 | /* Cache overviews. */ 574 | cache_overview(&wf.base, 2048); 575 | CHECK_ERROR("while caching CLI overview"); 576 | cache_overview(&wf.base, 65536); 577 | CHECK_ERROR("while caching GUI overview"); 578 | wf.wf_db_err = sqlite3_exec(db, "END TRANSACTION", NULL, NULL, &errm); 579 | if (errm) { 580 | ntfs_log_error("%s while ending transaction", errm); 581 | free(errm); 582 | goto out; 583 | } 584 | CHECK_ERROR("while flushing overview cache database transaction"); 585 | 586 | out: 587 | if (wf.wf_iconv) 588 | iconv_close(wf.wf_iconv); 589 | 590 | err2 = sqlite3_close(db); 591 | if (err2) 592 | ntfs_log_error("%s while closing database", 593 | sqlite3_errstr(err2)); 594 | if (!err && err2) 595 | err = err2; 596 | 597 | err2 = ntfs_umount(fs, FALSE); 598 | if (err2) 599 | ntfs_log_error("%s %s", strerror(err2), 600 | "while closing filesystem.\n"); 601 | 602 | if (!err && err2) 603 | err = err2; 604 | 605 | return err; 606 | } 607 | -------------------------------------------------------------------------------- /pymod.c: -------------------------------------------------------------------------------- 1 | /* 2 | * Python module code... 3 | * Copyright 2016 Darrick J. Wong. 4 | * Licensed under the GPLv2. 5 | */ 6 | #include 7 | #include "filemapper.h" 8 | #include "compdb.h" 9 | #include "compress.h" 10 | 11 | #define MOD_NAME "compdb" 12 | 13 | /* Get a list of supported compression algorithms. */ 14 | static PyObject * 15 | compdb_compressors_py( 16 | PyObject *self, 17 | PyObject *args) 18 | { 19 | char *s; 20 | PyObject *o; 21 | 22 | s = compdb_compressors(); 23 | o = Py_BuildValue("s", s); 24 | free(s); 25 | 26 | return o; 27 | } 28 | 29 | /* Register a compressed-VFS */ 30 | static PyObject * 31 | compdb_register_py( 32 | PyObject *self, 33 | PyObject *args) 34 | { 35 | char *under; 36 | char *name; 37 | char *compr; 38 | int err; 39 | 40 | if (!PyArg_ParseTuple(args, "zsz", &under, &name, &compr)) 41 | return NULL; 42 | 43 | err = compdb_register(under, name, compr); 44 | if (err) 45 | PyErr_SetString(PyExc_RuntimeError, strerror(err)); 46 | 47 | return Py_BuildValue("i", err); 48 | } 49 | 50 | /* Unregister a VFS */ 51 | static PyObject * 52 | compdb_unregister_py( 53 | PyObject *self, 54 | PyObject *args) 55 | { 56 | sqlite3_vfs *vfs; 57 | char *name; 58 | int err; 59 | 60 | if (!PyArg_ParseTuple(args, "z", &name)) 61 | return NULL; 62 | 63 | vfs = sqlite3_vfs_find(name); 64 | if (!vfs) { 65 | err = ENOENT; 66 | PyErr_SetString(PyExc_RuntimeError, strerror(err)); 67 | goto out; 68 | } 69 | 70 | err = sqlite3_vfs_unregister(vfs); 71 | if (err != SQLITE_OK) { 72 | err = EIO; 73 | PyErr_SetString(PyExc_RuntimeError, strerror(err)); 74 | } 75 | 76 | out: 77 | return Py_BuildValue("i", err); 78 | } 79 | 80 | static PyMethodDef compdb_methods[] = { 81 | {"register", compdb_register_py, METH_VARARGS, NULL}, 82 | {"unregister", compdb_unregister_py, METH_VARARGS, NULL}, 83 | {"compressors", compdb_compressors_py, METH_NOARGS, NULL}, 84 | {NULL, NULL, 0, NULL} 85 | }; 86 | 87 | static struct PyModuleDef moduledef = { 88 | PyModuleDef_HEAD_INIT, 89 | MOD_NAME, 90 | NULL, 91 | 0, 92 | compdb_methods, 93 | NULL, 94 | NULL, 95 | NULL, 96 | NULL 97 | }; 98 | 99 | #if PY_MAJOR_VERSION >= 3 100 | PyMODINIT_FUNC 101 | PyInit_compdb(void) 102 | { 103 | PyObject *m; 104 | 105 | m = PyModule_Create(&moduledef); 106 | if (!m) 107 | return NULL; 108 | 109 | return m; 110 | } 111 | #else 112 | PyMODINIT_FUNC 113 | init_compdb(void) 114 | { 115 | PyObject *m; 116 | 117 | m = Py_InitModule(MOD_NAME, compdb_methods); 118 | if (!m) 119 | return; 120 | 121 | return; 122 | } 123 | #endif /* PY_MAJOR_VERSION */ 124 | -------------------------------------------------------------------------------- /shrinkmapper.1: -------------------------------------------------------------------------------- 1 | .\" (C) Copyright 2016 Darrick J. Wong , 2 | .TH SHRINKMAPPER 1 "December 28, 2016" 3 | .SH NAME 4 | shrinkmapper \- Recompress filemapper databases. 5 | .SH SYNOPSIS 6 | .B shrinkmapper 7 | .I infile 8 | .I outfile 9 | .BI "[" incompressor "]" 10 | .BI "[" outcompressor "]" 11 | .SH DESCRIPTION 12 | This manual page documents briefly the 13 | .B shrinkmapper 14 | command. 15 | .PP 16 | \fBshrinkmapper\fP compresses a filemapper database to use less disk 17 | space. 18 | It can operate on any SQLite3 database, so long as one uses the compdb 19 | SQLite VFS shim to provide compressed file support. 20 | .SH OPTIONS 21 | These programs follow the usual GNU command line syntax, with long 22 | options starting with two dashes (`-'). 23 | A summary of options is included below. 24 | For a complete description, see the Info files. 25 | .TP 26 | .I infile 27 | Read data from this database. 28 | .TP 29 | .I outfile 30 | Write database to this file. 31 | .TP 32 | .I incompressor 33 | If specified, use this compression algorithm to read in pages from the database. 34 | An empty string or no string selects the default (gzip). 35 | .TP 36 | .I outcompressor 37 | If specified, use this compression algorithm to write pages to the database. 38 | An empty string or no string selects the default (gzip). 39 | .TP 40 | .B \-l 41 | Print the names of all supported compression algorithms. 42 | .TP 43 | .B \-h, \-\-help 44 | Show summary of options. 45 | .SH SEE ALSO 46 | .BR filemapper (1). 47 | .br 48 | -------------------------------------------------------------------------------- /shrinkmapper.c: -------------------------------------------------------------------------------- 1 | /* 2 | * Compress existing SQLite databases. 3 | * Copyright 2016 Darrick J. Wong. 4 | * Licensed under the GPLv2. 5 | */ 6 | #include 7 | #include 8 | #include 9 | #include 10 | #include 11 | #include 12 | #include 13 | #include 14 | #include 15 | #include 16 | #include 17 | #include 18 | #include 19 | #include 20 | #include "filemapper.h" 21 | #include "compdb.h" 22 | #include "compress.h" 23 | 24 | #define swap(a, b) \ 25 | do { typeof(a) __tmp = (a); (a) = (b); (b) = __tmp; } while (0) 26 | 27 | struct compdb_info { 28 | enum compdb_type type; 29 | int pagesize; 30 | unsigned int freestart; 31 | unsigned int freelen; 32 | char in_file_header[16]; 33 | char out_file_header[16]; 34 | }; 35 | 36 | /* Figure out database parameters. */ 37 | static int 38 | sniff( 39 | const struct sqlite3_super *super, 40 | struct compdb_info *cdb) 41 | { 42 | int is_sqlite; 43 | int is_compr; 44 | 45 | /* Is this really a database? */ 46 | is_sqlite = !memcmp(super->magic, SQLITE_FILE_HEADER, 47 | sizeof(super->magic)); 48 | is_compr = !memcmp(super->magic, cdb->in_file_header, 49 | sizeof(super->magic)); 50 | if ((!is_sqlite && !is_compr) || 51 | super->max_fraction != 64 || super->min_fraction != 32 || 52 | super->leaf_payload != 32 || ntohl(super->schema_format) > 4) 53 | return EUCLEAN; 54 | 55 | if (is_sqlite) 56 | cdb->type = DB_REGULAR; 57 | else if (is_compr) 58 | cdb->type = DB_COMPRESSED; 59 | 60 | /* Collect some stats. */ 61 | cdb->pagesize = ntohs(super->pagesize); 62 | if (cdb->pagesize == 1) 63 | cdb->pagesize = 65536; 64 | cdb->freestart = ntohl(super->freelist_start); 65 | cdb->freelen = ntohl(super->freelist_pages); 66 | dbg_printf("%s(%d): pagesize=%d freestart=%llu:%llu\n", __func__, __LINE__, 67 | cdb->pagesize, cdb->freestart, cdb->freelen); 68 | return 0; 69 | } 70 | 71 | static inline int 72 | NOOP_compress( 73 | const char *source, 74 | char *dest, 75 | int sourceSize, 76 | int maxDestSize) 77 | { 78 | return 0; 79 | } 80 | 81 | int 82 | main( 83 | int argc, 84 | char *argv[]) 85 | { 86 | struct sqlite3_super super; 87 | struct compdb_info cdb; 88 | struct stat sb; 89 | char *name; 90 | void *bin, *bout; 91 | void *outp; 92 | struct compdb_block_head *bhead; 93 | struct compressor_type *inc; 94 | struct compressor_type *outc; 95 | struct compressor_type none_type; 96 | int fdin, fdout; 97 | int try_compress; 98 | size_t outlen; 99 | size_t page; 100 | size_t nr_pages; 101 | ssize_t ret; 102 | 103 | if (argc == 2 && !strcmp(argv[1], "-l")) { 104 | printf("Supported: NONE,%s\n", compdb_compressors()); 105 | return 0; 106 | } 107 | 108 | if (argc < 3 || argc > 5) { 109 | printf("Usage: %s infile outfile [compressor] [compressor]\n", argv[0]); 110 | return 1; 111 | } 112 | 113 | /* Open files */ 114 | fdin = open(argv[1], O_RDONLY); 115 | if (fdin < 0) { 116 | perror(argv[1]); 117 | return 2; 118 | } 119 | 120 | fdout = open(argv[2], O_RDWR | O_CREAT | O_TRUNC, 0644); 121 | if (fdout < 0) { 122 | perror(argv[2]); 123 | return 2; 124 | } 125 | 126 | ret = fstat(fdin, &sb); 127 | if (ret) { 128 | perror(argv[1]); 129 | return 2; 130 | } 131 | 132 | /* Select compressor(s) */ 133 | if (argc >= 4 && argv[3][0] != 0) 134 | name = argv[3]; 135 | else 136 | name = NULL; 137 | 138 | inc = compdb_find_compressor(name); 139 | if (!inc) { 140 | printf("%s: no such compressor?\n", name); 141 | return 2; 142 | } 143 | snprintf(cdb.in_file_header, sizeof(cdb.in_file_header), 144 | COMPDB_FILE_TEMPLATE, inc->name); 145 | 146 | if (argc == 5 && argv[4][0] != 0) 147 | name = argv[4]; 148 | else 149 | name = NULL; 150 | 151 | if (name && !strcmp(name, "NONE")) { 152 | none_type.name = "none"; 153 | none_type.compress = NOOP_compress; 154 | none_type.decompress = NULL; 155 | outc = &none_type; 156 | memcpy(cdb.out_file_header, SQLITE_FILE_HEADER, 157 | sizeof(cdb.out_file_header)); 158 | } else { 159 | outc = compdb_find_compressor(name); 160 | if (!outc) { 161 | printf("%s: no such compressor?\n", name); 162 | return 2; 163 | } 164 | snprintf(cdb.out_file_header, sizeof(cdb.out_file_header), 165 | COMPDB_FILE_TEMPLATE, outc->name); 166 | } 167 | dbg_printf("recompress %s(%s) -> %s(%s)\n", argv[1], inc->name, 168 | argv[2], outc->name); 169 | 170 | /* Verify superblock */ 171 | ret = pread(fdin, &super, sizeof(super), 0); 172 | if (ret < 0) { 173 | perror(argv[1]); 174 | return 2; 175 | } else if (ret < sizeof(super)) { 176 | printf("%s: Short super read??\n", argv[1]); 177 | return 2; 178 | } 179 | 180 | cdb.type = DB_UNKNOWN; 181 | ret = sniff(&super, &cdb); 182 | if (ret < 0) { 183 | perror(argv[1]); 184 | return 2; 185 | } 186 | 187 | if (cdb.type == DB_UNKNOWN) { 188 | printf("%s: Unrecognized file type.\n", argv[1]); 189 | return 2; 190 | } 191 | 192 | /* Allocate buffers */ 193 | bin = malloc(cdb.pagesize); 194 | if (!bin) { 195 | perror("malloc"); 196 | return 2; 197 | } 198 | 199 | bout = malloc(cdb.pagesize); 200 | if (!bout) { 201 | perror("malloc"); 202 | return 2; 203 | } 204 | 205 | /* Copy pages */ 206 | nr_pages = (sb.st_size + cdb.pagesize - 1) / cdb.pagesize; 207 | for (page = 0; page < nr_pages; page++) { 208 | /* Read buffer. */ 209 | dbg_printf("%s(%d) off=%zu len=%u\n", __func__, __LINE__, 210 | page * cdb.pagesize, cdb.pagesize); 211 | ret = pread(fdin, bin, cdb.pagesize, page * cdb.pagesize); 212 | if (ret < 0) { 213 | perror(argv[1]); 214 | return 3; 215 | } else if (ret < cdb.pagesize && page != nr_pages - 1) { 216 | printf("%s: Short page %zu read?\n", argv[1], page); 217 | return 3; 218 | } 219 | 220 | /* Transform buffer. */ 221 | outlen = cdb.pagesize; 222 | outp = bin; 223 | bhead = bin; 224 | try_compress = 0; 225 | if (page == 0) { 226 | /* Do we need to change the header? */ 227 | if (cdb.type == DB_REGULAR || inc != outc) 228 | memcpy(outp, cdb.out_file_header, 229 | sizeof(cdb.out_file_header)); 230 | } else if (cdb.type == DB_COMPRESSED && 231 | !memcmp(bhead->magic, COMPDB_BLOCK_MAGIC, 232 | sizeof(COMPDB_BLOCK_MAGIC)) && 233 | ntohl(bhead->offset) == page) { 234 | if (inc == outc) { 235 | /* Compressed page, send it along. */ 236 | outlen = ntohs(bhead->len) + sizeof(*bhead); 237 | } else { 238 | /* Changing compression types; decompress. */ 239 | ret = inc->decompress(bin + sizeof(*bhead), 240 | bout, 241 | ntohs(bhead->len), 242 | cdb.pagesize); 243 | if (ret <= 0) { 244 | printf("%s: Decompression failed at " 245 | " page %zu\n", argv[1], page); 246 | return 3; 247 | } 248 | dbg_printf("%s(%d) off=%zu len=%d\n", __func__, 249 | __LINE__, page * cdb.pagesize, 250 | ntohs(bhead->len)); 251 | swap(bin, bout); 252 | outp = bin; 253 | try_compress = 1; 254 | } 255 | } else if (page < cdb.freestart || 256 | page >= cdb.freestart + cdb.freelen) { 257 | /* Uncompressed; try to compress. */ 258 | try_compress = 1; 259 | } 260 | 261 | if (try_compress) { 262 | /* Try to compress this page? */ 263 | ret = outc->compress(bin, bout + sizeof(*bhead), 264 | cdb.pagesize, 265 | cdb.pagesize - sizeof(*bhead)); 266 | if (ret > 0) { 267 | bhead = bout; 268 | memcpy(bhead->magic, COMPDB_BLOCK_MAGIC, 269 | sizeof(bhead->magic)); 270 | bhead->len = htons(ret); 271 | bhead->offset = htonl(page); 272 | outp = bout; 273 | outlen = ret + sizeof(*bhead); 274 | } 275 | } 276 | 277 | /* 278 | * Truncate to where the end of the compressed block 279 | * should be so that XFS won't do speculative preallocation. 280 | */ 281 | ret = ftruncate(fdout, (page * cdb.pagesize) + outlen); 282 | if (ret) { 283 | perror(argv[2]); 284 | return 2; 285 | } 286 | 287 | /* Write to disk. */ 288 | dbg_printf("%s(%d) off=%zu len=%zu\n", __func__, __LINE__, 289 | page * cdb.pagesize, outlen); 290 | ret = pwrite(fdout, outp, outlen, page * cdb.pagesize); 291 | if (ret < 0) { 292 | perror(argv[2]); 293 | return 3; 294 | } else if (ret < outlen) { 295 | printf("%s: Short page %zu write?\n", argv[2], page); 296 | return 3; 297 | } 298 | 299 | /* 300 | * Truncate to the end of the block to avoid short reads. 301 | */ 302 | if (outlen != cdb.pagesize) { 303 | ret = ftruncate(fdout, (page + 1) * cdb.pagesize); 304 | if (ret) { 305 | perror(argv[2]); 306 | return 2; 307 | } 308 | } 309 | } 310 | 311 | free(bout); 312 | free(bin); 313 | close(fdout); 314 | close(fdin); 315 | 316 | return 0; 317 | } 318 | -------------------------------------------------------------------------------- /vfs.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # online filesystem mapper scanning 3 | # Copyright (C) 2017 Darrick J. Wong. All rights reserved. 4 | # Licensed under the GPLv2. 5 | 6 | import os 7 | import stat 8 | import fiemap 9 | import getfsmap 10 | import collections 11 | import fmdb 12 | 13 | fake_stat = collections.namedtuple('fake_stat', 14 | 'st_ino st_mode st_size st_atime st_mtime st_ctime') 15 | 16 | def walk_fs(path, dir_fn, ino_fn, extent_fn): 17 | '''Iterate the filesystem, looking for extent data.''' 18 | def do_map(fstat, path): 19 | if fstat.st_ino in seen: 20 | return 21 | seen.add(fstat.st_ino) 22 | fd = os.open(path, os.O_RDONLY) 23 | try: 24 | if do_map.fiemap_broken: 25 | for extent in fiemap.fibmap2(fd, flags = flags): 26 | extent_fn(fstat, extent, False) 27 | return 28 | try: 29 | for extent in fiemap.fiemap2(fd, flags = flags): 30 | extent_fn(fstat, extent, False) 31 | except: 32 | if stat.S_ISREG(fstat.st_mode): 33 | do_map.fiemap_broken = True 34 | for extent in fiemap.fibmap2(fd, flags = flags): 35 | extent_fn(fstat, extent, False) 36 | try: 37 | for extent in fiemap.fiemap2(fd, flags = flags | fiemap.FIEMAP_FLAG_XATTR): 38 | extent_fn(fstat, extent, True) 39 | except: 40 | pass 41 | finally: 42 | os.close(fd) 43 | 44 | def ensure_metadir(stat_dict): 45 | if fmdb.METADATA_DIR in stat_dict: 46 | return stat_dict[fmdb.METADATA_DIR] 47 | metadir_stat = fake_stat(-1, -fmdb.INO_TYPE_DIR, None, None, None, None) 48 | ino_fn(metadir_stat, '/' + fmdb.METADATA_DIR) 49 | root_stat = os.lstat(path) 50 | dir_fn(root_stat, [(fmdb.METADATA_DIR, metadir_stat)]) 51 | stat_dict[fmdb.METADATA_DIR] = metadir_stat 52 | return metadir_stat 53 | 54 | def ensure_metadir_file(stat_dict, owner, mode, name): 55 | if owner in stat_dict: 56 | return stat_dict[owner] 57 | sb = fake_stat(owner, mode, None, None, None, None) 58 | metadir_stat = ensure_metadir(stat_dict) 59 | ino_fn(sb, '/' + fmdb.METADATA_DIR + '/' + name) 60 | dir_fn(metadir_stat, [(name, sb)]) 61 | stat_dict[owner] = sb 62 | return sb 63 | 64 | def ensure_unlinked_file(stat_dict, owner): 65 | if owner in stat_dict: 66 | return stat_dict[owner] 67 | sb = fake_stat(owner, stat.S_IFREG, None, None, None, None) 68 | udir_stat = ensure_metadir_file(stat_dict, -4, stat.S_IFDIR, fmdb.UNLINKED_DIR) 69 | ino_fn(sb, '/' + fmdb.METADATA_DIR + '/' + fmdb.UNLINKED_DIR + '/%d' % owner) 70 | dir_fn(udir_stat, [('%d' % owner, sb)]) 71 | stat_dict[owner] = sb 72 | return sb 73 | 74 | do_map.fiemap_broken = False 75 | seen = set() 76 | # Careful - we have to pass a byte string to os.walk so that 77 | # it'll return byte strings, which we can then decode ourselves. 78 | # Otherwise the automatic Unicode decoding will error out. 79 | # 80 | # Strip out the trailing / so that root is '' 81 | if path != os.sep and path[-1] == os.sep: 82 | path = path[:-1] 83 | prefix_len = 0 if path == os.sep else len(path) 84 | root_stat = os.lstat(path) 85 | flags = fiemap.FIEMAP_FLAG_SYNC 86 | 87 | # Walk the directory tree for dir and extent information 88 | seen_inodes = set() 89 | for root, dirs, files in os.walk(path.encode('utf-8', 'surrogateescape')): 90 | rstat = os.lstat(root) 91 | if rstat.st_dev != root_stat.st_dev: 92 | continue 93 | if root.decode('utf-8', 'replace') == os.sep: 94 | plen = 1 95 | else: 96 | plen = prefix_len 97 | seen_inodes.add(rstat.st_ino) 98 | ino_fn(rstat, root[plen:].decode('utf-8', 'replace')) 99 | do_map(rstat, root) 100 | dentries = [] 101 | for xdir in dirs: 102 | dname = os.path.join(root, xdir) 103 | try: 104 | dstat = os.lstat(dname) 105 | except Exception as e: 106 | print(e) 107 | continue 108 | 109 | if dstat.st_dev != root_stat.st_dev: 110 | dirs.remove(xdir) 111 | continue 112 | dentries.append((xdir.decode('utf-8', 'replace'), dstat)) 113 | for xfile in files: 114 | fname = os.path.join(root, xfile) 115 | try: 116 | fstat = os.lstat(fname) 117 | except Exception as e: 118 | print(e) 119 | continue 120 | 121 | if not stat.S_ISREG(fstat.st_mode) and \ 122 | not stat.S_ISDIR(fstat.st_mode): 123 | continue 124 | 125 | if fstat.st_dev != root_stat.st_dev: 126 | continue 127 | seen_inodes.add(fstat.st_ino) 128 | ino_fn(fstat, fname[prefix_len:].decode('utf-8', 'replace')) 129 | do_map(fstat, fname) 130 | dentries.append((xfile.decode('utf-8', 'replace'), fstat)) 131 | dir_fn(rstat, dentries) 132 | 133 | # Now try to walk the space map for metadata and unlinked inodes 134 | stat_dict = {} 135 | try: 136 | fd = os.open(path, os.O_RDONLY) 137 | for rmap in getfsmap.getfsmap(fd): 138 | hdr_flags = 0 139 | if (rmap.hdr_flags & getfsmap.FMH_OF_DEV_T) and \ 140 | rmap.device != root_stat.st_dev: 141 | # Not this device; skip 142 | continue 143 | elif (rmap.flags & getfsmap.FMR_OF_SPECIAL_OWNER) == 0: 144 | if rmap.owner in seen_inodes: 145 | continue 146 | # Capture unlinked inode extents 147 | if rmap.flags & getfsmap.FMR_OF_ATTR_FORK: 148 | hdr_flags |= fiemap.FIEMAP_FLAG_XATTR 149 | sb = ensure_unlinked_file(stat_dict, rmap.owner) 150 | elif rmap.owner == getfsmap.FMR_OWN_FREE: 151 | # Free space 152 | sb = ensure_metadir_file(stat_dict, -2, -fmdb.INO_TYPE_FREESP, fmdb.FREESP_FILE) 153 | elif rmap.owner == getfsmap.FMR_OWN_METADATA: 154 | # Generic metadata 155 | sb = ensure_metadir_file(stat_dict, -3, -fmdb.INO_TYPE_METADATA, fmdb.METADATA_FILE) 156 | elif getfsmap.FMR_OWNER_TYPE(rmap.owner) != 0: 157 | # FS-specific metadata 158 | sb = ensure_metadir_file(stat_dict, rmap.owner, -fmdb.INO_TYPE_METADATA, getfsmap.special_owner_name(rmap.owner)) 159 | else: 160 | # Unknown generic special owner 161 | continue 162 | ext = fiemap.fiemap_rec(rmap.offset, rmap.physical, \ 163 | rmap.length, 0, hdr_flags) 164 | extent_fn(sb, ext, False) 165 | except OSError: 166 | pass 167 | finally: 168 | os.close(fd) 169 | -------------------------------------------------------------------------------- /xfsmapper.1: -------------------------------------------------------------------------------- 1 | .\" Hey, EMACS: -*- nroff -*- 2 | .\" (C) Copyright 2015 Darrick J. Wong , 3 | .\" 4 | .\" First parameter, NAME, should be all caps 5 | .\" Second parameter, SECTION, should be 1-8, maybe w/ subsection 6 | .\" other parameters are allowed: see man(7), man(1) 7 | .TH FILEMAPPER 1 "April 22, 2015" 8 | .\" Please adjust this date whenever revising the manpage. 9 | .\" 10 | .\" Some roff macros, for reference: 11 | .\" .nh disable hyphenation 12 | .\" .hy enable hyphenation 13 | .\" .ad l left justify 14 | .\" .ad b justify to both left and right margins 15 | .\" .nf disable filling 16 | .\" .fi enable filling 17 | .\" .br insert line break 18 | .\" .sp insert n+1 empty lines 19 | .\" for manpage-specific macros, see man(7) 20 | .SH NAME 21 | xfsmapper \- Analyze an XFS image for use with filemapper 22 | .SH SYNOPSIS 23 | .B filemapper 24 | .I dbfile 25 | .I fs_device 26 | .SH DESCRIPTION 27 | This manual page documents briefly the 28 | .B xfsmapper 29 | command. 30 | .PP 31 | .\" TeX users may be more comfortable with the \fB\fP and 32 | .\" \fI\fP escape sequences to invode bold face and italics, 33 | .\" respectively. 34 | \fBxfsmapper\fP analyzes the internals of an XFS filesystem and 35 | generates a snapshot that the filemapper program can use to visualize 36 | and query the physical layout of the filesystem. 37 | .SH OPTIONS 38 | These programs follow the usual GNU command line syntax, with long 39 | options starting with two dashes (`-'). 40 | A summary of options is included below. 41 | For a complete description, see the Info files. 42 | .TP 43 | .I dbfile 44 | Store the filesystem analysis in this file. 45 | .TP 46 | .I fs_device 47 | Open this raw device for analysis. 48 | .TP 49 | .B \-h, \-\-help 50 | Show summary of options. 51 | .SH SEE ALSO 52 | .BR filemapper (1). 53 | .br 54 | --------------------------------------------------------------------------------