├── .github └── workflows │ └── ci.yml ├── .gitignore ├── .pre-commit-config.yaml ├── CHANGELOG ├── CONTRIBUTORS.txt ├── LICENSE ├── Makefile ├── README ├── indxparse ├── BinaryParser.py ├── FileMap.py ├── INDXFind.py ├── INDXParse.py ├── MFT.py ├── MFTINDX.py ├── MFTView.py ├── Progress.py ├── SDS.py ├── SDS_get_index.py ├── SortedCollection.py ├── __init__.py ├── carve_mft_records.py ├── extract_mft_record_slack.py ├── fuse-mft.py ├── get_file_info.py ├── list_mft.py ├── py.typed └── tree_mft.py ├── setup.cfg ├── setup.py ├── tests ├── MFTINDX │ ├── 7-ntfs-undel.dd.d.txt │ ├── 7-ntfs-undel.dd.l.txt │ ├── 7-ntfs-undel.dd.m.txt │ ├── 7-ntfs-undel.dd.mft.d.txt │ ├── 7-ntfs-undel.dd.mft.m.txt │ ├── 7-ntfs-undel.dd.s.txt │ └── Makefile ├── Makefile ├── list_mft │ ├── 7-ntfs-undel.dd.bodyfile │ ├── 7-ntfs-undel.dd.json │ ├── 7-ntfs-undel.dd.mft.bodyfile │ ├── 7-ntfs-undel.dd.mft.json │ └── Makefile └── tree_mft │ ├── 7-ntfs-undel.dd.mft.txt │ └── Makefile └── third_party ├── .gitignore ├── 7-ntfs-undel.dd.md5 ├── 7-ntfs-undel.dd.mft.gz ├── 7-ntfs-undel.dd.mft.sha2-256 ├── 7-ntfs-undel.dd.mft.sha3-256 ├── 7-ntfs-undel.dd.sha2-256 ├── 7-ntfs-undel.dd.sha3-256 ├── 7-undel-ntfs.zip ├── 7-undel-ntfs.zip.sha2-256 ├── 7-undel-ntfs.zip.sha3-256 └── Makefile /.github/workflows/ci.yml: -------------------------------------------------------------------------------- 1 | # Portions of this file contributed by NIST are governed by the following 2 | # statement: 3 | # 4 | # This software was developed at the National Institute of Standards 5 | # and Technology by employees of the Federal Government in the course 6 | # of their official duties. Pursuant to title 17 Section 105 of the 7 | # United States Code this software is not subject to copyright 8 | # protection and is in the public domain. NIST assumes no 9 | # responsibility whatsoever for its use by other parties, and makes 10 | # no guarantees, expressed or implied, about its quality, 11 | # reliability, or any other characteristic. 12 | # 13 | # We would appreciate acknowledgement if the software is used. 14 | 15 | name: Continuous Integration 16 | 17 | on: 18 | push: 19 | branches: 20 | - master 21 | pull_request: 22 | branches: 23 | - master 24 | 25 | jobs: 26 | build: 27 | 28 | runs-on: ubuntu-latest 29 | strategy: 30 | matrix: 31 | python-version: 32 | - '3.8' 33 | - '3.12' 34 | 35 | steps: 36 | - uses: actions/checkout@v3 37 | - name: Set up Python ${{ matrix.python-version }} 38 | uses: actions/setup-python@v4 39 | with: 40 | python-version: ${{ matrix.python-version }} 41 | - name: Pre-commit Checks 42 | run: | 43 | pip -q install pre-commit 44 | pre-commit run --all-files 45 | - name: Run tests 46 | run: make PYTHON3=python check 47 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *~ 2 | *pyc 3 | *.egg-info 4 | *.zip 5 | .idea/* 6 | .venv.done.log 7 | build 8 | venv 9 | -------------------------------------------------------------------------------- /.pre-commit-config.yaml: -------------------------------------------------------------------------------- 1 | repos: 2 | - repo: https://github.com/PyCQA/isort 3 | rev: 5.12.0 4 | hooks: 5 | - id: isort 6 | - repo: https://github.com/psf/black 7 | rev: 23.7.0 8 | hooks: 9 | - id: black 10 | - repo: https://github.com/pycqa/flake8 11 | rev: 6.1.0 12 | hooks: 13 | - id: flake8 14 | -------------------------------------------------------------------------------- /CHANGELOG: -------------------------------------------------------------------------------- 1 | v1.3 2 | - fix bug in parsing INDX_ALLOCATION attributes with an embedded null block reported by Jerome Leseinne 3 | - add fuse-mft.py MFT FUSE driver 4 | v1.3.1 5 | - fix bug in parsing INDX_ALLOCATION attributes for some volume root directories, as reported by Andrew Case 6 | -------------------------------------------------------------------------------- /CONTRIBUTORS.txt: -------------------------------------------------------------------------------- 1 | - Phill Moore, for reporting a bug in list_mft.py that prevent users from providing a path prefix 2 | - Glenn Edwards, for reporing and providing fixes to multiple bugs 3 | - Andrew Case, for reporting an issue when parsing empty INDX allocation attributes 4 | - Jacob Garner, for providing the INDXFind.py script 5 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | 177 | END OF TERMS AND CONDITIONS 178 | 179 | APPENDIX: How to apply the Apache License to your work. 180 | 181 | To apply the Apache License to your work, attach the following 182 | boilerplate notice, with the fields enclosed by brackets "[]" 183 | replaced with your own identifying information. (Don't include 184 | the brackets!) The text should be enclosed in the appropriate 185 | comment syntax for the file format. We also recommend that a 186 | file or class name and description of purpose be included on the 187 | same "printed page" as the copyright notice for easier 188 | identification within third-party archives. 189 | 190 | Copyright [yyyy] [name of copyright owner] 191 | 192 | Licensed under the Apache License, Version 2.0 (the "License"); 193 | you may not use this file except in compliance with the License. 194 | You may obtain a copy of the License at 195 | 196 | http://www.apache.org/licenses/LICENSE-2.0 197 | 198 | Unless required by applicable law or agreed to in writing, software 199 | distributed under the License is distributed on an "AS IS" BASIS, 200 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 201 | See the License for the specific language governing permissions and 202 | limitations under the License. 203 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | #!/usr/bin/make -f 2 | 3 | # Portions of this file contributed by NIST are governed by the 4 | # following statement: 5 | # 6 | # This software was developed at the National Institute of Standards 7 | # and Technology by employees of the Federal Government in the course 8 | # of their official duties. Pursuant to Title 17 Seciton 105 of the 9 | # United States Code, this software is not subject to copyright 10 | # protection within the United States. NIST assumes no responsibility 11 | # whatsoever for its use by other parties, and makes no guarantees, 12 | # expressed or implied, about its quality, reliability, or any other 13 | # characteristic. 14 | # 15 | # We would appreciate acknowledgement if the software is used. 16 | 17 | SHELL := /bin/bash 18 | 19 | PYTHON3 ?= python3 20 | 21 | all: 22 | 23 | .PHONY: \ 24 | check-mypy \ 25 | check-third_party 26 | 27 | .venv.done.log: \ 28 | setup.cfg \ 29 | setup.py 30 | rm -rf venv 31 | $(PYTHON3) -m venv venv 32 | source venv/bin/activate \ 33 | && pip install \ 34 | --upgrade \ 35 | pip \ 36 | setuptools \ 37 | wheel 38 | source venv/bin/activate \ 39 | && pip install \ 40 | --editable \ 41 | .[testing] 42 | touch $@ 43 | 44 | check: \ 45 | check-mypy \ 46 | check-third_party 47 | $(MAKE) \ 48 | --directory tests \ 49 | check 50 | 51 | check-mypy: \ 52 | .venv.done.log 53 | source venv/bin/activate \ 54 | && mypy \ 55 | indxparse 56 | source venv/bin/activate \ 57 | && mypy \ 58 | --strict \ 59 | indxparse/INDXFind.py \ 60 | indxparse/MFTINDX.py \ 61 | indxparse/__init__.py \ 62 | indxparse/extract_mft_record_slack.py \ 63 | indxparse/list_mft.py \ 64 | indxparse/tree_mft.py 65 | 66 | check-third_party: 67 | $(MAKE) \ 68 | --directory third_party \ 69 | check 70 | 71 | clean: 72 | @$(MAKE) \ 73 | --directory tests \ 74 | clean 75 | @$(MAKE) \ 76 | --directory third_party \ 77 | clean 78 | @rm -f \ 79 | .venv.done.log 80 | -------------------------------------------------------------------------------- /README: -------------------------------------------------------------------------------- 1 | INDXParse 2 | =============== 3 | 4 | Introduction 5 | ------------ 6 | INDX files are features of the Windows NTFS file system. They can 7 | be thought of as nodes in a B+ tree, where each directory has an 8 | INDX file. The INDX files contain records for each file within a 9 | directory. Records contain at least the following information: 10 | 11 | - Filename 12 | - Physical size of file 13 | - Logical size of file 14 | - Modified timestamp 15 | - Accessed timestamp 16 | - Changed timestamp 17 | - Created timestamp 18 | 19 | INDX files are interesting to forensic investigators for a number 20 | of reasons. First, an investigator may use INDX files as a source 21 | of timestamps to develop a timeline of activity. Secondly, these 22 | files have significant slack spaces. With careful parsing, an 23 | investigator may recover old or deleted records from within these 24 | data chunks. In other words, the investigator may be able to show 25 | a file existed even if it has been deleted. 26 | 27 | INDX files are not usually accessible from within the Windows 28 | operating system. Forensic utilties such as the FTK Imager may 29 | allow a user to extract the file by accessing the raw hard disk. 30 | FTK names the INDX file "$I30". Tools like the Sleuthkit can 31 | extract the directory entries from a forensic image. INDXParse 32 | will not work against a live system. 33 | 34 | Previous work & tools 35 | --------------------- 36 | I'd like to first mention John McCash, who mentioned he was 37 | unaware of any non-EnCase tools that parse INDX files in a SANS 38 | blog post. That got my mental gears turning. 39 | 40 | I started out with a document called NTFS Forensics: A 41 | Programmers View of Raw Filesystem Data Extraction by Jason 42 | Medeiros. Unfortunately, while this document describes parsing 43 | INDX files in detail, a number of steps in the explanation were 44 | wrong. 45 | 46 | The second resource I used, and used extensively, was Forensic 47 | computing by A. J. Sammes, Tony Sammes, and Brian Jenkinson. I 48 | found the relevent section was available for free via Google 49 | books. This was an excellent document, and I now plan on buying 50 | the full book. 51 | 52 | 42 LLC provides the INDX Extractor Enpack as a compiled EnScript 53 | for EnCase. This was not useful to me, because I was unable to 54 | get to the logic of the script. 55 | 56 | The Sleuthkit has INDX structures defined in the tsk_ntfs.h 57 | header files. I didn't do much digging in the code to see if 58 | TSK does any parsing of the INDX files (I suspect it does), 59 | but I did use it to verify the file structure. 60 | 61 | Usage 62 | ----- 63 | INDXParse.py accepts a number of command line parameters and 64 | switches that determine what data is parsed and output format. 65 | INDXParse.py currently supports both CSV (default) and 66 | Bodyfile (v3) output formats. The CSV schema is as follows: 67 | 68 | - Filename 69 | - Physical size of file 70 | - Logical size of file 71 | - Modified timestamp 72 | - Accessed timestamp 73 | - Changed timestamp 74 | - Created timestamp 75 | 76 | INDXParse.py will parse INDX structure slack space if provided 77 | the '-d' flag. Entries identified in the slack space will be 78 | tagged with a string of the form "(slack at ###)" where ### is 79 | the hex offset to the slack entry. Note that slack entries will 80 | have separate timestamps from the live entries, and could be 81 | used to show the state of the system at a point in time. 82 | 83 | If the program encounters an error while parsing the filename, 84 | the filename field will contain a best guess, and the comment 85 | "(error decoding filename)". If the program encounters an error 86 | while parsing timestamps, a timestamp corresponding to the UNIX 87 | epoch will be printed instead. 88 | 89 | The full command line help is included here: 90 | 91 | INDX $ python INDXParse.py -h 92 | usage: INDXParse.py [-h] [-c | -b] [-d] filename 93 | 94 | Parse NTFS INDX files. 95 | 96 | positional arguments: 97 | filename Input INDX file path 98 | 99 | optional arguments: 100 | -h, --help show this help message and exit 101 | -c Output CSV 102 | -b Output Bodyfile 103 | -d Find entries in slack space 104 | 105 | INDXTemplate.bt is a template file for the useful 010 Editor. 106 | Use it as you would any other template by applying it to INDX files. 107 | 108 | TODO 109 | ---- 110 | - Brainstorm more features ;-) 111 | 112 | License 113 | ------- 114 | INDXParse is released under the Apache 2.0 license. 115 | 116 | 117 | Contributors 118 | ------------ 119 | 120 | - Jerome Leseinne for identifying a bug in the is_valid constraint and null blocks 121 | -------------------------------------------------------------------------------- /indxparse/FileMap.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | 3 | # Alex Nelson, NIST, contributed to this file. Contributions of NIST 4 | # are not subject to US Copyright. 5 | 6 | import sys 7 | from collections import OrderedDict 8 | from struct import calcsize 9 | from struct import unpack_from as old_unpack_from 10 | 11 | # From: http://code.activestate.com/recipes/577197-sortedcollection/ 12 | from indxparse.SortedCollection import SortedCollection 13 | 14 | MEGABYTE = 1024 * 1024 15 | 16 | 17 | class LRUQueue(object): 18 | """ 19 | LRUQueue is a data structure that orders objects by 20 | their insertion time, and supports an update/touch operation 21 | that resets an item to the newest slot. 22 | 23 | This is an example of a priority queue, ordered by 24 | insertion time, with explicit support for "touch". 25 | """ 26 | 27 | def __init__(self, key=lambda n: n): 28 | """ 29 | The `key` parameter may be provided if the 30 | items in the queue are complex. 31 | The `key` parameter should select a unique "id" field from 32 | each item. 33 | """ 34 | super(LRUQueue, self).__init__() 35 | self._q = OrderedDict() 36 | self._key = key 37 | 38 | def push(self, v): 39 | k = self._key(v) 40 | self._q[k] = v 41 | 42 | def pop(self): 43 | return self._q.popitem(last=False)[1] 44 | 45 | def touch(self, v): 46 | """ 47 | Reset the given value back to the newest slot. 48 | """ 49 | k = self._key(v) 50 | del self._q[k] 51 | self._q[k] = v 52 | 53 | def size(self): 54 | return len(self._q) 55 | 56 | def __len__(self): 57 | return self.size() 58 | 59 | @staticmethod 60 | def test(): 61 | q = LRUQueue() 62 | assert q.size() == 0 63 | assert len(q) == 0 64 | 65 | q.push(0) 66 | assert q.size() == 1 67 | assert len(q) == 1 68 | 69 | assert q.pop() == 0 70 | assert q.size() == 0 71 | assert len(q) == 0 72 | 73 | q.push(0) 74 | q.push(1) 75 | assert q.pop() == 0 76 | assert q.pop() == 1 77 | 78 | q.push(0) 79 | q.push(1) 80 | q.touch(0) 81 | assert q.pop() == 1 82 | assert q.pop() == 0 83 | 84 | q = LRUQueue(key=lambda n: n[0]) 85 | q.push([0]) 86 | assert q.pop() == [0] 87 | 88 | q.push([0]) 89 | q.push([1]) 90 | assert q.pop() == [0] 91 | assert q.pop() == [1] 92 | return True 93 | 94 | 95 | class BoundedLRUQueue(object): 96 | """ 97 | BoundedLRUQueue is a LRUQueue with a finite capacity. 98 | When an item is pushed that causes the capacity to be exceeded, 99 | the LRU item is automatically popped. 100 | 101 | Otherwise, this class behaves just like the LRUQueue. 102 | """ 103 | 104 | def __init__(self, capacity, key=lambda n: n): 105 | """ 106 | The `key` parameter may be provided if the 107 | items in the queue are complex. 108 | The `key` parameter should select a unique "id" field from 109 | each item. 110 | """ 111 | super(BoundedLRUQueue, self).__init__() 112 | self._q = LRUQueue(key) 113 | self._capacity = capacity 114 | 115 | def push(self, v): 116 | self._q.push(v) 117 | if len(self._q) > self._capacity: 118 | return self._q.pop() 119 | 120 | def pop(self): 121 | return self._q.pop() 122 | 123 | def touch(self, v): 124 | self._q.touch(v) 125 | 126 | def size(self): 127 | return len(self._q) 128 | 129 | def __len__(self): 130 | return self.size() 131 | 132 | @staticmethod 133 | def test(): 134 | q = BoundedLRUQueue(5) 135 | assert q.size() == 0 136 | assert len(q) == 0 137 | 138 | q.push(0) 139 | assert q.size() == 1 140 | assert len(q) == 1 141 | 142 | assert q.pop() == 0 143 | assert q.size() == 0 144 | assert len(q) == 0 145 | 146 | q.push(0) 147 | q.push(1) 148 | assert q.pop() == 0 149 | assert q.pop() == 1 150 | 151 | q.push(0) 152 | q.push(1) 153 | q.touch(0) 154 | assert q.pop() == 1 155 | assert q.pop() == 0 156 | 157 | q = BoundedLRUQueue(5, key=lambda n: n[0]) 158 | q.push([0]) 159 | assert q.pop() == [0] 160 | 161 | q.push([0]) 162 | q.push([1]) 163 | assert q.pop() == [0] 164 | assert q.pop() == [1] 165 | 166 | q = BoundedLRUQueue(2) 167 | assert q.push(0) is None 168 | assert q.push(1) is None 169 | assert q.push(2) == 0 170 | assert q.pop() == 1 171 | assert q.pop() == 2 172 | return True 173 | 174 | 175 | class RangeCache(object): 176 | """ 177 | RangeCache is a data structure that tracks a finite set of 178 | ranges (a range is a 2-tuple consisting of a numeric start 179 | and numeric length). New ranges can be added via the `push` 180 | method, and if such a call causes the capacity to be exceeded, 181 | then the "oldest" range is removed. The `get` method implements 182 | an efficient lookup for a single value that may be found within 183 | one of the ranges. 184 | """ 185 | 186 | def __init__(self, capacity, start_key=lambda o: o[0], length_key=lambda o: o[1]): 187 | """ 188 | @param key: A function that fetches the range start from an item. 189 | """ 190 | super(RangeCache, self).__init__() 191 | self._ranges = SortedCollection(key=start_key) 192 | self._lru = BoundedLRUQueue(capacity, key=start_key) 193 | self._start_key = start_key 194 | self._length_key = length_key 195 | 196 | def push(self, o): 197 | """ 198 | Add a range to the cache. 199 | 200 | If `key` is not provided to the constructor, then 201 | `o` should be a 3-tuple: 202 | - range start (numeric) 203 | - range length (numeric) 204 | - range item (object) 205 | """ 206 | self._ranges.insert(o) 207 | popped = self._lru.push(o) 208 | if popped is not None: 209 | self._ranges.remove(popped) 210 | 211 | def touch(self, o): 212 | self._lru.touch(o) 213 | 214 | def get(self, value): 215 | """ 216 | Search for the numeric `value` within the ranges 217 | tracked by this cache. 218 | @raise ValueError: if the value is not found in the range cache. 219 | """ 220 | hit = self._ranges.find_le(value) 221 | if value < self._start_key(hit) + self._length_key(hit): 222 | return hit 223 | raise ValueError("%s not found in range cache" % value) 224 | 225 | @staticmethod 226 | def test(): 227 | q = RangeCache(2) 228 | 229 | x = None 230 | try: 231 | x = q.get(0) 232 | except ValueError: 233 | pass 234 | assert x is None 235 | 236 | x = None 237 | try: 238 | x = q.get(1) 239 | except ValueError: 240 | pass 241 | assert x is None 242 | 243 | q.push((1, 1, [0])) 244 | 245 | x = None 246 | try: 247 | x = q.get(0) 248 | except ValueError: 249 | pass 250 | assert x is None 251 | 252 | assert q.get(1) == (1, 1, [0]) 253 | assert q.get(1.99) == (1, 1, [0]) 254 | x = None 255 | try: 256 | x = q.get(2.01) 257 | except ValueError: 258 | pass 259 | assert x is None 260 | 261 | q.push((3, 1, [1])) 262 | assert q.get(1) == (1, 1, [0]) 263 | assert q.get(3) == (3, 1, [1]) 264 | 265 | q.push((5, 1, [2])) 266 | x = None 267 | try: 268 | x = q.get(1) 269 | except ValueError: 270 | pass 271 | assert x is None 272 | 273 | assert q.get(3) == (3, 1, [1]) 274 | assert q.get(5) == (5, 1, [2]) 275 | 276 | q.touch((3, 1, [1])) 277 | q.push((7, 1, [3])) 278 | 279 | assert q.get(3) == (3, 1, [1]) 280 | assert q.get(7) == (7, 1, [3]) 281 | x = None 282 | try: 283 | x = q.get(5) 284 | except ValueError: 285 | pass 286 | assert x is None 287 | 288 | return True 289 | 290 | 291 | class FileMap(object): 292 | """ 293 | FileMap is a wrapper for a file-like object that satisfies the 294 | buffer interface. This is essentially the inverse of StringIO. 295 | It implements a caching layer over the calls to the OS seek/read 296 | functions for improved performance. 297 | 298 | Q: Why might you want this over mmap? 299 | A: 1) Its pure Python 300 | 2) You can stack this over any Python file-like objects. 301 | eg. FileMap over ZipFile gives you a random access buffer 302 | thats backed by a compressed image on the file system. 303 | """ 304 | 305 | def __init__(self, filelike, block_size=MEGABYTE, cache_size=10, size=None): 306 | """ 307 | If `size` is not provided, then `filelike` must have the 308 | `seek` and `tell` methods implemented. 309 | """ 310 | super(FileMap, self).__init__() 311 | if size is None: 312 | import os 313 | 314 | filelike.seek(0, os.SEEK_END) 315 | size = filelike.tell() 316 | self._f = filelike 317 | self._block_size = block_size 318 | self._size = size 319 | self._block_cache = RangeCache(cache_size) 320 | 321 | def __getitem__(self, index): 322 | if index < 0: 323 | index = self._size + index 324 | block_index = index % self._block_size 325 | block_start = index - block_index 326 | 327 | try: 328 | hit = self._block_cache.get(index) 329 | buf = hit[2] 330 | self._block_cache.touch(hit) 331 | return buf[block_index] 332 | except ValueError: 333 | self._f.seek(block_start) 334 | buf = self._f.read(self._block_size) 335 | self._block_cache.push((block_start, self._block_size, buf)) 336 | return buf[block_index] 337 | 338 | def _get_containing_block(self, index): 339 | """ 340 | Given an index, return block-aligned block that contains it, 341 | updating the appropriate caches. 342 | """ 343 | block_index = index % self._block_size 344 | block_start = index - block_index 345 | 346 | try: 347 | hit = self._block_cache.get(block_start) 348 | buf = hit[2] 349 | self._block_cache.touch(hit) 350 | return buf 351 | except ValueError: 352 | self._f.seek(block_start) 353 | buf = self._f.read(self._block_size) 354 | self._block_cache.push((block_start, self._block_size, buf)) 355 | return buf 356 | 357 | def __getslice__(self, start, end): 358 | if end == sys.maxsize: 359 | end = self._size 360 | 361 | start_block_index = start % self._block_size 362 | start_block_start = start - start_block_index 363 | 364 | end_block_index = end % self._block_size 365 | end_block_start = end - end_block_index 366 | 367 | if start_block_start == end_block_start: 368 | # easy case, everything falls within the same block 369 | buf = self._get_containing_block(start) 370 | return buf[start_block_index:end_block_index] 371 | else: 372 | # hard case, slice goes over one or more block boundaries 373 | ret = "" 374 | 375 | # phase 1, start to block boundary 376 | buf = self._get_containing_block(start_block_start) 377 | s = start_block_index 378 | e = start_block_start + self._block_size 379 | ret += buf[s:e] 380 | 381 | # phase 2, any complete blocks 382 | cur_block_start = start_block_start + self._block_size 383 | while cur_block_start + self._block_size < end_block_start: 384 | buf = self._get_containing_block(cur_block_start) 385 | ret += buf 386 | cur_block_start += self._block_size 387 | 388 | # phase 3, block boundary to end 389 | buf = self._get_containing_block(cur_block_start) 390 | s = 0 391 | e = end_block_index or self._block_size 392 | ret += buf[0:e] 393 | return ret 394 | 395 | def __len__(self): 396 | return self._size 397 | 398 | @staticmethod 399 | def test(): 400 | from io import StringIO 401 | 402 | f = StringIO("0123abcd4567efgh") 403 | buf = FileMap(f, block_size=4, cache_size=2) 404 | 405 | assert len(buf) == 16 406 | 407 | assert buf[0] == "0" 408 | assert buf[1] == "1" 409 | assert buf[0:2] == "01" 410 | 411 | assert buf[4] == "a" 412 | assert buf[5] == "b" 413 | assert buf[4:6] == "ab" 414 | 415 | assert buf[2:6] == "23ab" 416 | assert buf[0:8] == "0123abcd" 417 | 418 | assert buf[0:12] == "0123abcd4567" 419 | assert buf[0:16] == "0123abcd4567efgh" 420 | assert buf[:] == "0123abcd4567efgh" 421 | 422 | assert buf[-1] == "h" 423 | assert buf[-2:] == "gh" 424 | assert buf[-4:] == "efgh" 425 | assert buf[-8:] == "4567efgh" 426 | 427 | return True 428 | 429 | 430 | def unpack_from(fmt, buffer, off=0): 431 | """ 432 | Shim struct.unpack_from and divert unpacking of FileMaps. 433 | 434 | Otherwise, you'd get an exception like: 435 | TypeError: unpack_from() argument 1 must be convertible to a buffer, not FileMap 436 | 437 | So, we extract a true sub-buffer from the FileMap, and feed this 438 | back into the old unpack function. 439 | Theres an extra allocation and copy, but there's no getting 440 | around that. 441 | """ 442 | if not isinstance(buffer, FileMap): 443 | return old_unpack_from(fmt, buffer, off) 444 | size = calcsize(fmt) 445 | buf = buffer[off : off + size] 446 | return old_unpack_from(fmt, buf, 0x0) 447 | 448 | 449 | def struct_test(): 450 | from io import StringIO 451 | 452 | f = StringIO("\x04\x03\x02\x01") 453 | buf = FileMap(f) 454 | assert unpack_from(" 8 | # while at Mandiant 9 | # 10 | # Licensed under the Apache License, Version 2.0 (the "License"); 11 | # you may not use this file except in compliance with the License. 12 | # You may obtain a copy of the License at 13 | # 14 | # http://www.apache.org/licenses/LICENSE-2.0 15 | # 16 | # Unless required by applicable law or agreed to in writing, software 17 | # distributed under the License is distributed on an "AS IS" BASIS, 18 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 19 | # See the License for the specific language governing permissions and 20 | # limitations under the License. 21 | # 22 | # 23 | # Alex Nelson, NIST, contributed to this file. Contributions of NIST 24 | # are not subject to US Copyright. 25 | 26 | import sys 27 | 28 | if sys.argv[1] == "-h": 29 | print("\tpython ./INDXfind.py ") 30 | print("\tex:\tpython ./INDXfind.py /mnt/ewf/ewf1") 31 | sys.exit() 32 | 33 | f = open( 34 | sys.argv[1], "rb" 35 | ) # ewfmount'd drive expected as first argument on command line 36 | indxBytes = ( 37 | b"\x49\x4e\x44\x58\x28\x00\x09\x00" # 49 4e 44 58 28 00 09 00 "INDX( header" 38 | ) 39 | offset = 0 # data processed 40 | byteChunk = b"go" # cheap do-while 41 | recordsFound = 0 # for progress 42 | outFile = open("INDX_records.raw", "wb") # output file 43 | 44 | print( 45 | "\n\tRunning... progress will output every GigaByte. In testing this was every 15-20 seconds.\n" 46 | '\tThe output file is named "INDX_records.raw".\n' 47 | "\tINDX_records.raw should be parsed with INDXparser.py which can be found at:\thttps://github.com/williballenthin/INDXParse\n" 48 | ) 49 | 50 | while byteChunk != b"": 51 | byteChunk = f.read( 52 | 4096 53 | ) # Only searching for cluster aligned (4096 on Windows Server 2003) INDX records... records all appear to be 4096 bytes 54 | compare = byteChunk[0:8] # Compare INDX header to first 8 bytes of the byteChunk 55 | if compare == indxBytes: 56 | recordsFound = recordsFound + 1 57 | outFile.write(byteChunk) # Write the byteChunk to the output file 58 | 59 | offset = offset + 4096 # Update offset for progress tracking 60 | 61 | # Progress 62 | if offset % 1073741824 == 0: 63 | print( 64 | "Processed: %d GB. INDX records found: %d" 65 | % ((offset / 1073741824), recordsFound) 66 | ) 67 | 68 | outFile.close() 69 | -------------------------------------------------------------------------------- /indxparse/MFTINDX.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | 3 | # This file is part of INDXParse. 4 | # 5 | # Copyright 2011 Will Ballenthin 6 | # while at Mandiant 7 | # 8 | # Licensed under the Apache License, Version 2.0 (the "License"); 9 | # you may not use this file except in compliance with the License. 10 | # You may obtain a copy of the License at 11 | # 12 | # http://www.apache.org/licenses/LICENSE-2.0 13 | # 14 | # Unless required by applicable law or agreed to in writing, software 15 | # distributed under the License is distributed on an "AS IS" BASIS, 16 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 17 | # See the License for the specific language governing permissions and 18 | # limitations under the License. 19 | # 20 | # 21 | # Sheldon Douglas, NIST Associate, and Alex Nelson, NIST, contributed 22 | # to this file. Contributions of NIST are not subject to US 23 | # Copyright. 24 | # 25 | # 26 | # Version v.1.2.0 27 | import argparse 28 | import array 29 | import calendar 30 | import logging 31 | import re 32 | import sys 33 | from datetime import datetime 34 | from typing import Any, List, Optional, Union 35 | 36 | from indxparse.BinaryParser import OverrunBufferException 37 | from indxparse.MFT import ( 38 | ATTR_TYPE, 39 | MREF, 40 | MSEQNO, 41 | NTATTR_STANDARD_INDEX_HEADER, 42 | Attribute, 43 | FilenameAttribute, 44 | IndexRecordHeader, 45 | IndexRootHeader, 46 | InvalidAttributeException, 47 | MFTRecord, 48 | NTFSFile, 49 | StandardInformation, 50 | StandardInformationFieldDoesNotExist, 51 | ) 52 | 53 | if sys.version_info >= (3, 9): 54 | from collections.abc import MutableSequence 55 | else: 56 | from typing import MutableSequence 57 | 58 | verbose = False 59 | 60 | 61 | def information_bodyfile( 62 | path: str, 63 | size: int, 64 | inode: int, 65 | owner_id: int, 66 | info: Union[FilenameAttribute, StandardInformation], 67 | attributes: Optional[List[str]] = None, 68 | ) -> str: 69 | if not attributes: 70 | attributes = [] 71 | try: 72 | modified = int(calendar.timegm(info.modified_time().timetuple())) 73 | except (ValueError, AttributeError): 74 | modified = int(calendar.timegm(datetime(1970, 1, 1, 0, 0, 0).timetuple())) 75 | try: 76 | accessed = int(calendar.timegm(info.accessed_time().timetuple())) 77 | except (ValueError, AttributeError): 78 | accessed = int(calendar.timegm(datetime(1970, 1, 1, 0, 0, 0).timetuple())) 79 | try: 80 | changed = int(calendar.timegm(info.changed_time().timetuple())) 81 | except (ValueError, AttributeError): 82 | changed = int(calendar.timegm(datetime(1970, 1, 1, 0, 0, 0).timetuple())) 83 | try: 84 | created = int(calendar.timegm(info.created_time().timetuple())) 85 | except (ValueError, AttributeError): 86 | created = int(calendar.timegm(datetime.min.timetuple())) 87 | attributes_text = "" 88 | if len(attributes) > 0: 89 | attributes_text = " (%s)" % (", ".join(attributes)) 90 | return "0|%s|%s|0|%d|0|%s|%s|%s|%s|%s\n" % ( 91 | path + attributes_text, 92 | inode, 93 | owner_id, 94 | size, 95 | accessed, 96 | modified, 97 | changed, 98 | created, 99 | ) 100 | 101 | 102 | def record_bodyfile( 103 | ntfsfile: NTFSFile, record: MFTRecord, attributes: Optional[List[str]] = None 104 | ) -> str: 105 | """ 106 | Return a bodyfile formatted string for the given MFT record. 107 | The string contains metadata for the one file described by the record. 108 | The string may have multiple lines, which cover $SI and 109 | $FN timestamp entries, and entries for each ADS. 110 | """ 111 | ret = "" 112 | if not attributes: 113 | attributes = [] 114 | path = ntfsfile.mft_record_build_path(record, {}) 115 | si = record.standard_information() 116 | fn = record.filename_information() 117 | if not fn: 118 | raise InvalidAttributeException("Unable to parse attribute") 119 | inode = record.inode or record.mft_record_number() 120 | if record.is_directory(): 121 | size = 0 122 | else: 123 | data_attr = record.data_attribute() 124 | if data_attr and data_attr.non_resident() > 0: 125 | size = data_attr.data_size() 126 | else: 127 | size = fn.logical_size() 128 | 129 | ADSs = [] 130 | for attr in record.attributes(): 131 | if attr.type() != ATTR_TYPE.DATA or len(attr.name()) == 0: 132 | continue 133 | if attr.non_resident() > 0: 134 | size = attr.data_size() 135 | else: 136 | size = attr.value_length() 137 | # sys.stderr.write("|%s, %s|\n" % (attr.name(), len(attr.name()))) 138 | ADSs.append((attr.name(), size)) 139 | 140 | if si: 141 | try: 142 | si_index = si.security_id() 143 | except StandardInformationFieldDoesNotExist: 144 | si_index = 0 145 | ret += "%s" % ( 146 | information_bodyfile(path, size, inode, si_index, si, attributes) 147 | ) 148 | for ads in ADSs: 149 | ret += "%s" % ( 150 | information_bodyfile( 151 | path + ":" + ads[0], ads[1], inode, si_index, si, attributes 152 | ) 153 | ) 154 | 155 | # sys.stderr.write(str(ADSs) + "\n") 156 | attributes.append("filename") 157 | if si: 158 | try: 159 | si_index = si.security_id() 160 | except StandardInformationFieldDoesNotExist: 161 | si_index = 0 162 | ret += "%s" % ( 163 | information_bodyfile(path, size, inode, si_index, fn, attributes=attributes) 164 | ) 165 | for ads in ADSs: 166 | ret += "%s" % ( 167 | information_bodyfile( 168 | path + ":" + ads[0], 169 | ads[1], 170 | inode, 171 | si_index, 172 | fn, 173 | attributes=attributes, 174 | ) 175 | ) 176 | 177 | return ret 178 | 179 | 180 | def node_header_bodyfile( 181 | node_header: NTATTR_STANDARD_INDEX_HEADER, 182 | basepath: str, 183 | *args: Any, 184 | indxlist: bool, 185 | slack: bool, 186 | **kwargs: Any, 187 | ) -> str: 188 | """ 189 | Returns a bodyfile formatted string for all INDX entries following the 190 | given INDX node header. 191 | """ 192 | ret = "" 193 | attrs = ["filename", "INDX"] 194 | if indxlist: 195 | for e in node_header.entries(): 196 | path = basepath + "\\" + e.filename_information().filename() 197 | size = e.filename_information().logical_size() 198 | inode = 0 199 | ret += information_bodyfile( 200 | path, size, inode, 0, e.filename_information(), attributes=attrs 201 | ) 202 | attrs.append("slack") 203 | if slack: 204 | for e in node_header.slack_entries(): 205 | path = basepath + "\\" + e.filename_information().filename() 206 | size = e.filename_information().logical_size() 207 | inode = 0 208 | ret += information_bodyfile( 209 | path, size, inode, 0, e.filename_information(), attributes=attrs 210 | ) 211 | return ret 212 | 213 | 214 | def record_indx_entries_bodyfile( 215 | ntfsfile: NTFSFile, 216 | record: MFTRecord, 217 | *args: Any, 218 | clustersize: int, 219 | indxlist: bool, 220 | offset: int, 221 | slack: bool, 222 | **kwargs: Any, 223 | ) -> str: 224 | """ 225 | Returns a bodyfile formatted string for all INDX entries associated with 226 | the given MFT record 227 | """ 228 | # TODO handle all possible errors here 229 | f = ntfsfile 230 | ret = "" 231 | if not record: 232 | return ret 233 | basepath = f.mft_record_build_path(record, {}) 234 | indxroot = record.attribute(ATTR_TYPE.INDEX_ROOT) 235 | if indxroot: 236 | if indxroot.non_resident() != 0: 237 | # TODO this shouldn't happen. 238 | pass 239 | else: 240 | iroh = IndexRootHeader(indxroot.value(), 0, False) 241 | nh = iroh.node_header() 242 | ret += node_header_bodyfile(nh, basepath, indxlist=indxlist, slack=slack) 243 | extractbuf = array.array("B") 244 | for attr in record.attributes(): 245 | if attr.type() != ATTR_TYPE.INDEX_ALLOCATION: 246 | continue 247 | if attr.non_resident() != 0: 248 | for offset, length in attr.runlist().runs(): 249 | try: 250 | ooff = offset * clustersize + offset 251 | llen = length * clustersize 252 | extractbuf += f.read(ooff, llen) 253 | except IOError: 254 | pass 255 | else: 256 | extractbuf += array.array("B", attr.value()) 257 | if len(extractbuf) < 4096: # TODO make this INDX record size 258 | return ret 259 | offset = 0 260 | try: 261 | ireh = IndexRecordHeader(extractbuf, offset, False) 262 | except OverrunBufferException: 263 | return ret 264 | # TODO could miss something if there is an empty, valid record at the end 265 | while ireh.magic() == 0x58444E49: 266 | nh = ireh.node_header() 267 | ret += node_header_bodyfile(nh, basepath, indxlist=indxlist, slack=slack) 268 | # TODO get this from the boot record 269 | offset += clustersize 270 | if offset + 4096 > len(extractbuf): # TODO make this INDX record size 271 | return ret 272 | try: 273 | ireh = IndexRecordHeader(extractbuf, offset, False) 274 | except OverrunBufferException: 275 | return ret 276 | return ret 277 | 278 | 279 | def try_write(s: str) -> None: 280 | try: 281 | sys.stdout.write(s) 282 | except (UnicodeEncodeError, UnicodeDecodeError): 283 | logging.warning( 284 | "Failed to write string " "due to encoding issue: " + str(list(s)) 285 | ) 286 | 287 | 288 | def print_nonresident_indx_bodyfile( 289 | buf: MutableSequence[int], 290 | basepath: str = "", 291 | *args: Any, 292 | clustersize: int, 293 | indxlist: bool, 294 | slack: bool, 295 | **kwargs: Any, 296 | ) -> None: 297 | offset = 0 298 | try: 299 | irh = IndexRecordHeader(buf, offset, False) 300 | except OverrunBufferException: 301 | return 302 | # TODO could miss something if there is an empty, valid record at the end 303 | while irh.magic() == 0x58444E49: 304 | nh = irh.node_header() 305 | try_write(node_header_bodyfile(nh, basepath, indxlist=indxlist, slack=slack)) 306 | offset += clustersize 307 | if offset + 4096 > len(buf): # TODO make this INDX record size 308 | return 309 | try: 310 | irh = IndexRecordHeader(buf, offset, False) 311 | except OverrunBufferException: 312 | return 313 | return 314 | 315 | 316 | def print_bodyfile( 317 | *args: Any, 318 | clustersize: int, 319 | deleted: bool, 320 | filename: str, 321 | filetype: str, 322 | filter_pattern: str, 323 | indxlist: bool, 324 | mftlist: bool, 325 | offset: int, 326 | prefix: str, 327 | progress: bool, 328 | slack: bool, 329 | **kwargs: Any, 330 | ) -> None: 331 | if filetype == "mft" or filetype == "image": 332 | ntfs_file = NTFSFile( 333 | clustersize=clustersize, 334 | filename=filename, 335 | filetype=filetype, 336 | offset=offset, 337 | prefix=prefix, 338 | progress=progress, 339 | ) 340 | if filter_pattern: 341 | refilter = re.compile(filter_pattern) 342 | for record in ntfs_file.record_generator(): 343 | logging.debug("Considering MFT record %s" % (record.mft_record_number())) 344 | try: 345 | if record.magic() != 0x454C4946: 346 | logging.debug("Record has a bad magic value") 347 | continue 348 | if filter_pattern: 349 | path = ntfs_file.mft_record_build_path(record, {}) 350 | if not refilter.search(path): 351 | logging.debug( 352 | "Skipping listing path " "due to regex filter: " + path 353 | ) 354 | continue 355 | if record.is_active() and mftlist: 356 | try_write(record_bodyfile(ntfs_file, record)) 357 | if indxlist or slack: 358 | try_write( 359 | record_indx_entries_bodyfile( 360 | ntfs_file, 361 | record, 362 | clustersize=clustersize, 363 | indxlist=indxlist, 364 | offset=offset, 365 | slack=slack, 366 | ) 367 | ) 368 | elif (not record.is_active()) and deleted: 369 | try_write( 370 | record_bodyfile(ntfs_file, record, attributes=["deleted"]) 371 | ) 372 | if filetype == "image" and (indxlist or slack): 373 | extractbuf = array.array("B") 374 | found_indxalloc = False 375 | for attr in record.attributes(): 376 | if attr.type() != ATTR_TYPE.INDEX_ALLOCATION: 377 | continue 378 | found_indxalloc = True 379 | if attr.non_resident() != 0: 380 | for offset, length in attr.runlist().runs(): 381 | ooff = offset * clustersize + offset 382 | llen = length * clustersize 383 | extractbuf += ntfs_file.read(ooff, llen) 384 | else: 385 | pass # This shouldn't happen. 386 | if found_indxalloc and len(extractbuf) > 0: 387 | path = ntfs_file.mft_record_build_path(record, {}) 388 | print_nonresident_indx_bodyfile( 389 | extractbuf, 390 | basepath=path, 391 | clustersize=clustersize, 392 | indxlist=indxlist, 393 | slack=slack, 394 | ) 395 | except InvalidAttributeException: 396 | pass 397 | elif filetype == "indx": 398 | with open(filename, "rb") as fh: 399 | buf = array.array("B", fh.read()) 400 | print_nonresident_indx_bodyfile( 401 | buf, clustersize=clustersize, indxlist=indxlist, slack=slack 402 | ) 403 | 404 | 405 | def print_indx_info( 406 | *args: Any, 407 | clustersize: int, 408 | extract: str, 409 | filename: str, 410 | filetype: str, 411 | infomode: str, 412 | offset: int, 413 | prefix: str, 414 | progress: bool, 415 | **kwargs: Any, 416 | ) -> None: 417 | f = NTFSFile( 418 | clustersize=clustersize, 419 | filename=filename, 420 | filetype=filetype, 421 | offset=offset, 422 | prefix=prefix, 423 | progress=progress, 424 | ) 425 | record: Optional[MFTRecord] = None 426 | try: 427 | record_num = int(infomode) 428 | record_buf = f.mft_get_record_buf(record_num) 429 | record = MFTRecord(record_buf, 0, False) 430 | except ValueError: 431 | record = f.mft_get_record_by_path(infomode) 432 | if record is None: 433 | print("Did not find directory entry for " + infomode) 434 | return 435 | print("Found directory entry for: " + infomode) 436 | 437 | if record.magic() != 0x454C4946: 438 | if record.magic() == int("0xBAAD", 0x10): 439 | print("BAAD Record") 440 | else: 441 | print("Invalid magic header: ", hex(record.magic())) 442 | return 443 | 444 | print("Path: " + f.mft_record_build_path(record, {})) 445 | print("MFT Record: " + str(record.mft_record_number())) 446 | 447 | print("Metadata: ") 448 | if record.is_active(): 449 | print(" active file") 450 | else: 451 | print(" deleted file") 452 | 453 | if record.is_directory(): 454 | print(" type: directory") 455 | else: 456 | print(" type: file") 457 | 458 | if not record.is_directory(): 459 | data_attr = record.data_attribute() 460 | if data_attr and data_attr.non_resident() > 0: 461 | print(" size: %d bytes" % (data_attr.data_size())) 462 | else: 463 | rfni = record.filename_information() 464 | if rfni is not None: 465 | print(" size: %d bytes" % (rfni.logical_size())) 466 | 467 | def get_flags(flags: int) -> List[str]: 468 | attributes = [] 469 | if flags & 0x01: 470 | attributes.append("readonly") 471 | if flags & 0x02: 472 | attributes.append("hidden") 473 | if flags & 0x04: 474 | attributes.append("system") 475 | if flags & 0x08: 476 | attributes.append("unused-dos") 477 | if flags & 0x10: 478 | attributes.append("directory-dos") 479 | if flags & 0x20: 480 | attributes.append("archive") 481 | if flags & 0x40: 482 | attributes.append("device") 483 | if flags & 0x80: 484 | attributes.append("normal") 485 | if flags & 0x100: 486 | attributes.append("temporary") 487 | if flags & 0x200: 488 | attributes.append("sparse") 489 | if flags & 0x400: 490 | attributes.append("reparse-point") 491 | if flags & 0x800: 492 | attributes.append("compressed") 493 | if flags & 0x1000: 494 | attributes.append("offline") 495 | if flags & 0x2000: 496 | attributes.append("not-indexed") 497 | if flags & 0x4000: 498 | attributes.append("encrypted") 499 | if flags & 0x10000000: 500 | attributes.append("has-indx") 501 | if flags & 0x20000000: 502 | attributes.append("has-view-index") 503 | return attributes 504 | 505 | rsi = record.standard_information() 506 | if rsi is None: 507 | print(" SI not found") 508 | else: 509 | print(" attributes: " + ", ".join(get_flags(rsi.attributes()))) 510 | 511 | crtime = rsi.created_time().isoformat("T") + "Z" 512 | mtime = rsi.modified_time().isoformat("T") + "Z" 513 | chtime = rsi.changed_time().isoformat("T") + "Z" 514 | atime = rsi.accessed_time().isoformat("T") + "Z" 515 | 516 | print(" SI modified: %s" % (mtime)) 517 | print(" SI accessed: %s" % (atime)) 518 | print(" SI changed: %s" % (chtime)) 519 | print(" SI birthed: %s" % (crtime)) 520 | 521 | try: 522 | # since the fields are sequential, we can handle an exception half way through here 523 | # and then ignore the remaining items. Dont have to worry about individual try/catches 524 | print(" owner id (quota info): %d" % (rsi.owner_id())) 525 | print(" security id: %d" % (rsi.security_id())) 526 | print(" quota charged: %d" % (rsi.quota_charged())) 527 | print(" USN: %d" % (rsi.usn())) 528 | except StandardInformationFieldDoesNotExist: 529 | pass 530 | 531 | print("Filenames:") 532 | for b in record.attributes(): 533 | if b.type() != ATTR_TYPE.FILENAME_INFORMATION: 534 | continue 535 | try: 536 | fnattr = FilenameAttribute(b.value(), 0, record) 537 | a = fnattr.filename_type() 538 | print(" Type: %s" % (["POSIX", "WIN32", "DOS 8.3", "WIN32 + DOS 8.3"][a])) 539 | print(" name: %s" % (str(fnattr.filename()))) 540 | print(" attributes: " + ", ".join(get_flags(fnattr.flags()))) 541 | print(" logical size: %d bytes" % (fnattr.logical_size())) 542 | print(" physical size: %d bytes" % (fnattr.physical_size())) 543 | 544 | crtime = fnattr.created_time().isoformat("T") + "Z" 545 | mtime = fnattr.modified_time().isoformat("T") + "Z" 546 | chtime = fnattr.changed_time().isoformat("T") + "Z" 547 | atime = fnattr.accessed_time().isoformat("T") + "Z" 548 | 549 | print(" modified: %s" % (mtime)) 550 | print(" accessed: %s" % (atime)) 551 | print(" changed: %s" % (chtime)) 552 | print(" birthed: %s" % (crtime)) 553 | print(" parent ref: %d" % (MREF(fnattr.mft_parent_reference()))) 554 | print(" parent seq: %d" % (MSEQNO(fnattr.mft_parent_reference()))) 555 | except ZeroDivisionError: 556 | continue 557 | 558 | print("Attributes:") 559 | for b in record.attributes(): 560 | print(" %s" % (Attribute.TYPES[b.type()])) 561 | print(" attribute name: %s" % (b.name() or "")) 562 | print(" attribute flags: " + ", ".join(get_flags(b.flags()))) 563 | if b.non_resident() > 0: 564 | print(" resident: no") 565 | print(" data size: %d" % (b.data_size())) 566 | print(" allocated size: %d" % (b.allocated_size())) 567 | 568 | if b.allocated_size() > 0: 569 | print(" runlist:") 570 | for offset, length in b.runlist().runs(): 571 | print(" Cluster %s, length %s" % (hex(offset), hex(length))) 572 | print( 573 | " %s (%s) bytes for %s (%s) bytes" 574 | % ( 575 | offset * clustersize, 576 | hex(offset * clustersize), 577 | length * clustersize, 578 | hex(length * clustersize), 579 | ) 580 | ) 581 | else: 582 | print(" resident: yes") 583 | print(" size: %d bytes" % (b.value_length())) 584 | 585 | # INDX stuff 586 | indxroot = record.attribute(ATTR_TYPE.INDEX_ROOT) 587 | if not indxroot: 588 | print("No INDX_ROOT attribute") 589 | return 590 | print("Found INDX_ROOT attribute") 591 | if indxroot.non_resident() != 0: 592 | # This shouldn't happen. 593 | print("INDX_ROOT attribute is non-resident") 594 | for rle in indxroot.runlist()._entries(): 595 | print("Cluster %s, length %s" % (hex(rle.offset()), hex(rle.length()))) 596 | else: 597 | print("INDX_ROOT attribute is resident") 598 | irh = IndexRootHeader(indxroot.value(), 0, False) 599 | someentries = False 600 | for nhe in irh.node_header().entries(): 601 | if not someentries: 602 | print("INDX_ROOT entries:") 603 | someentries = True 604 | print(" " + nhe.filename_information().filename()) 605 | print( 606 | " " 607 | + str(nhe.filename_information().logical_size()) 608 | + " bytes in size" 609 | ) 610 | print( 611 | " b " 612 | + nhe.filename_information().created_time().isoformat("T") 613 | + "Z" 614 | ) 615 | print( 616 | " m " 617 | + nhe.filename_information().modified_time().isoformat("T") 618 | + "Z" 619 | ) 620 | print( 621 | " c " 622 | + nhe.filename_information().changed_time().isoformat("T") 623 | + "Z" 624 | ) 625 | print( 626 | " a " 627 | + nhe.filename_information().accessed_time().isoformat("T") 628 | + "Z" 629 | ) 630 | 631 | if not someentries: 632 | print("INDX_ROOT entries: (none)") 633 | someentries = False 634 | for e in irh.node_header().slack_entries(): 635 | if not someentries: 636 | print("INDX_ROOT slack entries:") 637 | someentries = True 638 | print(" " + e.filename_information().filename()) 639 | if not someentries: 640 | print("INDX_ROOT slack entries: (none)") 641 | extractbuf = array.array("B") 642 | found_indxalloc = False 643 | for rattr in record.attributes(): 644 | if rattr.type() != ATTR_TYPE.INDEX_ALLOCATION: 645 | continue 646 | found_indxalloc = True 647 | print("Found INDX_ALLOCATION attribute") 648 | if rattr.non_resident() != 0: 649 | print("INDX_ALLOCATION is non-resident") 650 | for offset, length in rattr.runlist().runs(): 651 | print("Cluster %s, length %s" % (hex(offset), hex(length))) 652 | print( 653 | " Using clustersize %s (%s) bytes and volume offset %s (%s) bytes: \n %s (%s) bytes for %s (%s) bytes" 654 | % ( 655 | clustersize, 656 | hex(clustersize), 657 | offset, 658 | hex(offset), 659 | (offset * clustersize) + offset, 660 | hex((offset * clustersize) + offset), 661 | length * clustersize, 662 | hex(length * clustersize), 663 | ) 664 | ) 665 | ooff = offset * clustersize + offset 666 | llen = length * clustersize 667 | extractbuf += f.read(ooff, llen) 668 | else: 669 | # This shouldn't happen. 670 | print("INDX_ALLOCATION is resident") 671 | if not found_indxalloc: 672 | print("No INDX_ALLOCATION attribute found") 673 | return 674 | if extract: 675 | with open(extract, "wb") as g: 676 | g.write(extractbuf) 677 | return 678 | 679 | 680 | def main() -> None: 681 | parser = argparse.ArgumentParser(description="Parse NTFS " "filesystem structures.") 682 | parser.add_argument( 683 | "-t", 684 | action="store", 685 | metavar="type", 686 | nargs=1, 687 | dest="filetype", 688 | choices=["image", "MFT", "INDX", "auto"], 689 | default="auto", 690 | help="The type of data provided.", 691 | ) 692 | parser.add_argument( 693 | "-c", 694 | action="store", 695 | metavar="size", 696 | nargs=1, 697 | type=int, 698 | dest="clustersize", 699 | help="Use this cluster size in bytes " "(default 4096 bytes)", 700 | ) 701 | parser.add_argument( 702 | "-o", 703 | action="store", 704 | metavar="offset", 705 | nargs=1, 706 | type=int, 707 | dest="offset", 708 | help="Offset in bytes to volume in image " "(default 32256 bytes)", 709 | ) 710 | parser.add_argument( 711 | "-l", 712 | action="store_true", 713 | dest="indxlist", 714 | help="List file entries in INDX records", 715 | ) 716 | parser.add_argument( 717 | "-s", 718 | action="store_true", 719 | dest="slack", 720 | help="List file entries in INDX slack space", 721 | ) 722 | parser.add_argument( 723 | "-m", 724 | action="store_true", 725 | dest="mftlist", 726 | help="List file entries for active MFT records", 727 | ) 728 | parser.add_argument( 729 | "-d", 730 | action="store_true", 731 | dest="deleted", 732 | help="List file entries for MFT records " "marked as deleted", 733 | ) 734 | parser.add_argument( 735 | "-i", 736 | action="store", 737 | metavar="path|inode", 738 | nargs=1, 739 | dest="infomode", 740 | help="Print information about a path's INDX records", 741 | ) 742 | parser.add_argument( 743 | "-e", 744 | action="store", 745 | metavar="i30", 746 | nargs=1, 747 | dest="extract", 748 | help="Used with -i, extract INDX_ALLOCATION " "attribute to a file", 749 | ) 750 | parser.add_argument( 751 | "-f", 752 | action="store", 753 | metavar="regex", 754 | nargs=1, 755 | dest="filter_pattern", 756 | help="Only consider entries whose path " "matches this regular expression", 757 | ) 758 | parser.add_argument( 759 | "-p", 760 | action="store", 761 | metavar="prefix", 762 | nargs=1, 763 | dest="prefix", 764 | help="Prefix paths with `prefix` rather than \\.\\", 765 | ) 766 | parser.add_argument( 767 | "--progress", 768 | action="store_true", 769 | dest="progress", 770 | help="Update a status indicator on STDERR " "if STDOUT is redirected", 771 | ) 772 | parser.add_argument( 773 | "-v", action="store_true", dest="verbose", help="Print debugging information" 774 | ) 775 | parser.add_argument("filename", action="store", help="Input INDX file path") 776 | 777 | results = parser.parse_args() 778 | 779 | global verbose 780 | verbose = results.verbose 781 | 782 | if results.filetype and results.filetype != "auto": 783 | results.filetype = results.filetype[0].lower() 784 | logging.info("Asked to process a file with type: " + results.filetype) 785 | else: 786 | with open(results.filename, "rb") as f: 787 | b = f.read(1024) 788 | if b[0:4] == b"FILE": 789 | results.filetype = "mft" 790 | elif b[0:4] == b"INDX": 791 | results.filetype = "indx" 792 | else: 793 | results.filetype = "image" 794 | logging.info("Auto-detected input file type: " + results.filetype) 795 | 796 | if results.clustersize: 797 | results.clustersize = results.clustersize[0] 798 | logging.info( 799 | "Using explicit file system cluster size %s (%s) bytes" 800 | % (str(results.clustersize), hex(results.clustersize)) 801 | ) 802 | else: 803 | results.clustersize = 4096 804 | logging.info( 805 | "Assuming file system cluster size %s (%s) bytes" 806 | % (str(results.clustersize), hex(results.clustersize)) 807 | ) 808 | 809 | if results.offset: 810 | results.offset = results.offset[0] 811 | logging.info( 812 | "Using explicit volume offset %s (%s) bytes" 813 | % (str(results.offset), hex(results.offset)) 814 | ) 815 | else: 816 | results.offset = 32256 817 | logging.info( 818 | "Assuming volume offset %s (%s) bytes" 819 | % (str(results.offset), hex(results.offset)) 820 | ) 821 | 822 | if results.prefix: 823 | results.prefix = results.prefix[0] 824 | logging.info("Using path prefix " + results.prefix) 825 | 826 | if results.indxlist: 827 | logging.info("Asked to list entries in INDX records") 828 | if results.filetype == "mft": 829 | logging.info( 830 | " Note, only resident INDX records can be processed " 831 | "with an MFT input file" 832 | ) 833 | logging.info( 834 | " If you find an interesting record, " 835 | "use -i to identify the relevant INDX record clusters" 836 | ) 837 | elif results.filetype == "indx": 838 | logging.info(" Note, only records in this INDX record will be listed") 839 | elif results.filetype == "image": 840 | pass 841 | else: 842 | pass 843 | 844 | if results.slack: 845 | logging.info("Asked to list slack entries in INDX records") 846 | logging.info( 847 | " Note, this uses a scanning heuristic to identify records. " 848 | "These records may be corrupt or out-of-date." 849 | ) 850 | 851 | if results.mftlist: 852 | logging.info("Asked to list active file entries in the MFT") 853 | if results.filetype == "indx": 854 | raise ValueError("Cannot list MFT entries of an INDX record") 855 | 856 | if results.deleted: 857 | logging.info("Asked to list deleted file entries in the MFT") 858 | if results.filetype == "indx": 859 | raise ValueError("Cannot list MFT entries of an INDX record") 860 | 861 | if results.infomode: 862 | results.infomode = results.infomode[0] 863 | logging.info("Asked to list information about path " + results.infomode) 864 | if results.indxlist or results.slack or results.mftlist or results.deleted: 865 | raise ValueError( 866 | "Information mode (-i) cannot be run " 867 | "with file entry list modes (-l/-s/-m/-d)" 868 | ) 869 | 870 | if results.extract: 871 | results.extract = results.extract[0] 872 | logging.info( 873 | "Asked to extract INDX_ALLOCATION attribute " 874 | "for the path " + results.infomode 875 | ) 876 | 877 | if results.extract and not results.infomode: 878 | logging.warning( 879 | "Extract (-e) doesn't make sense " "without information mode (-i)" 880 | ) 881 | 882 | if results.extract and not results.filetype == "image": 883 | raise ValueError( 884 | "Cannot extract non-resident attributes " "from anything but an image" 885 | ) 886 | 887 | if not ( 888 | results.indxlist 889 | or results.slack 890 | or results.mftlist 891 | or results.deleted 892 | or results.infomode 893 | ): 894 | raise ValueError("You must choose a mode (-i/-l/-s/-m/-d)") 895 | 896 | if results.filter_pattern: 897 | results.filter_pattern = results.filter_pattern[0] 898 | logging.info( 899 | "Asked to only list file entry information " 900 | "for paths matching the regular expression: " + results.filter_pattern 901 | ) 902 | if results.infomode: 903 | logging.warning("This filter has no meaning with information mode (-i)") 904 | 905 | if results.infomode: 906 | print_indx_info( 907 | clustersize=results.clustersize, 908 | extract=results.extract, 909 | filename=results.filename, 910 | filetype=results.filetype, 911 | infomode=results.infomode, 912 | offset=results.offset, 913 | prefix=results.prefix, 914 | progress=results.progress, 915 | ) 916 | elif results.indxlist or results.slack or results.mftlist or results.deleted: 917 | print_bodyfile( 918 | clustersize=results.clustersize, 919 | deleted=results.deleted, 920 | filename=results.filename, 921 | filetype=results.filetype, 922 | filter_pattern=results.filter_pattern, 923 | indxlist=results.indxlist, 924 | mftlist=results.mftlist, 925 | offset=results.offset, 926 | prefix=results.prefix, 927 | progress=results.progress, 928 | slack=results.slack, 929 | ) 930 | 931 | 932 | if __name__ == "__main__": 933 | main() 934 | -------------------------------------------------------------------------------- /indxparse/Progress.py: -------------------------------------------------------------------------------- 1 | #!/bin/python 2 | # This file is part of INDXParse. 3 | # 4 | # Copyright 2013 Willi Ballenthin 5 | # while at Mandiant 6 | # 7 | # Licensed under the Apache License, Version 2.0 (the "License"); 8 | # you may not use this file except in compliance with the License. 9 | # You may obtain a copy of the License at 10 | # 11 | # http://www.apache.org/licenses/LICENSE-2.0 12 | # 13 | # Unless required by applicable law or agreed to in writing, software 14 | # distributed under the License is distributed on an "AS IS" BASIS, 15 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | # See the License for the specific language governing permissions and 17 | # limitations under the License. 18 | # 19 | # 20 | # Sheldon Douglas, NIST Associate, and Alex Nelson, NIST, contributed 21 | # to this file. Contributions of NIST are not subject to US 22 | # Copyright. 23 | 24 | 25 | class Progress(object): 26 | """ 27 | An interface to things that track the progress of a long running task. 28 | """ 29 | 30 | def __init__(self, max_: int) -> None: 31 | super(Progress, self).__init__() 32 | self._max = max_ 33 | self._current = 0 34 | 35 | def set_current(self, current: int) -> None: 36 | """ 37 | Set the number of steps that this task has completed. 38 | 39 | @type current: int 40 | """ 41 | self._current = current 42 | 43 | def set_complete(self) -> None: 44 | """ 45 | Convenience method to set the task as having completed all steps. 46 | """ 47 | self._current = self._max 48 | 49 | 50 | class NullProgress(Progress): 51 | """ 52 | A Progress class that ignores any updates. 53 | """ 54 | 55 | def __init__(self, max_: int) -> None: 56 | super(NullProgress, self).__init__(max_) 57 | 58 | def set_current(self, current: int) -> None: 59 | pass 60 | 61 | 62 | class ProgressBarProgress(Progress): 63 | def __init__(self, max_): 64 | from progressbar import ETA, Bar, ProgressBar # type: ignore 65 | 66 | super(ProgressBarProgress, self).__init__(max_) 67 | 68 | widgets = [ 69 | "Progress: ", 70 | Bar(marker="=", left="[", right="]"), 71 | " ", 72 | ETA(), 73 | " ", 74 | ] 75 | self._pbar = ProgressBar(widgets=widgets, maxval=self._max) 76 | self._has_notified_started = False 77 | 78 | def set_current(self, current: int) -> None: 79 | if not self._has_notified_started: 80 | self._pbar.start() 81 | self._has_notified_started = True 82 | 83 | self._pbar.update(current) 84 | 85 | def set_complete(self) -> None: 86 | self._pbar.finish() 87 | -------------------------------------------------------------------------------- /indxparse/SDS.py: -------------------------------------------------------------------------------- 1 | #!/bin/python 2 | 3 | # This file is part of INDXParse. 4 | # 5 | # Copyright 2011-13 Will Ballenthin 6 | # while at Mandiant 7 | # 8 | # Licensed under the Apache License, Version 2.0 (the "License"); 9 | # you may not use this file except in compliance with the License. 10 | # You may obtain a copy of the License at 11 | # 12 | # http://www.apache.org/licenses/LICENSE-2.0 13 | # 14 | # Unless required by applicable law or agreed to in writing, software 15 | # distributed under the License is distributed on an "AS IS" BASIS, 16 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 17 | # See the License for the specific language governing permissions and 18 | # limitations under the License. 19 | # 20 | # 21 | # Alex Nelson, NIST, contributed to this file. Contributions of NIST 22 | # are not subject to US Copyright. 23 | # 24 | # 25 | # Version v.1.2 26 | from indxparse.BinaryParser import ( 27 | Block, 28 | Nestable, 29 | ParseException, 30 | align, 31 | read_byte, 32 | read_dword, 33 | read_word, 34 | ) 35 | 36 | 37 | class NULL_OBJECT(object): 38 | def __init__(self): 39 | super(NULL_OBJECT, self).__init__() 40 | 41 | @staticmethod 42 | def structure_size(buf, offset, parent): 43 | return 0 44 | 45 | def __len__(self): 46 | return 0 47 | 48 | 49 | null_object = NULL_OBJECT() 50 | 51 | 52 | class SECURITY_DESCRIPTOR_CONTROL: 53 | SE_OWNER_DEFAULTED = 1 << 0 54 | SE_GROUP_DEFAULTED = 1 << 1 55 | SE_DACL_PRESENT = 1 << 2 56 | SE_DACL_DEFAULTED = 1 << 3 57 | SE_SACL_PRESENT = 1 << 4 58 | SE_SACL_DEFAULTED = 1 << 5 59 | SE_SACL_UNUSED0 = 1 << 6 60 | SE_SACL_UNUSED1 = 1 << 7 61 | SE_DACL_AUTO_INHERIT_REQ = 1 << 8 62 | SE_SACL_AUTO_INHERIT_REQ = 1 << 9 63 | SE_DACL_AUTO_INHERITED = 1 << 10 64 | SE_SACL_AUTO_INHERITED = 1 << 11 65 | SE_DACL_PROTECTED = 1 << 12 66 | SE_SACL_PROTECTED = 1 << 13 67 | SE_RM_CONTROL_VALID = 1 << 14 68 | SE_SELF_RELATIVE = 1 << 15 69 | 70 | 71 | class SID_IDENTIFIER_AUTHORITY(Block, Nestable): 72 | def __init__(self, buf, offset, parent): 73 | super(SID_IDENTIFIER_AUTHORITY, self).__init__(buf, offset) 74 | self.declare_field("word_be", "high_part", 0x0) 75 | self.declare_field("dword_be", "low_part") 76 | 77 | @staticmethod 78 | def structure_size(buf, offset, parent): 79 | return 6 80 | 81 | def __len__(self): 82 | return SID_IDENTIFIER_AUTHORITY.structure_size( 83 | self._buf, self.absolute_offset(0x0), None 84 | ) 85 | 86 | def __str__(self): 87 | return "%s" % ((self.high_part() << 32) + self.low_part()) 88 | 89 | 90 | class SID(Block, Nestable): 91 | def __init__(self, buf, offset, parent): 92 | super(SID, self).__init__(buf, offset) 93 | self.declare_field("byte", "revision", 0x0) 94 | self.declare_field("byte", "sub_authority_count") 95 | self.declare_field(SID_IDENTIFIER_AUTHORITY, "identifier_authority") 96 | self.declare_field("dword", "sub_authorities", count=self.sub_authority_count()) 97 | 98 | @staticmethod 99 | def structure_size(buf, offset, parent): 100 | sub_auth_count = read_byte(buf, offset + 1) 101 | auth_size = SID_IDENTIFIER_AUTHORITY.structure_size(buf, offset + 2, parent) 102 | return 2 + auth_size + (sub_auth_count * 4) 103 | 104 | def __len__(self): 105 | return self._off_sub_authorities + (self.sub_authority_count() * 4) 106 | 107 | def string(self): 108 | ret = "S-%d-%s" % (self.revision(), self.identifier_authority()) 109 | for sub_auth in self.sub_authorities(): 110 | ret += "-%s" % (str(sub_auth)) 111 | return ret 112 | 113 | 114 | class ACE_TYPES: 115 | """ 116 | One byte. 117 | """ 118 | 119 | ACCESS_MIN_MS_ACE_TYPE = 0 120 | ACCESS_ALLOWED_ACE_TYPE = 0 121 | ACCESS_DENIED_ACE_TYPE = 1 122 | SYSTEM_AUDIT_ACE_TYPE = 2 123 | SYSTEM_ALARM_ACE_TYPE = 3 # Not implemented as of Win2k. 124 | ACCESS_MAX_MS_V2_ACE_TYPE = 3 125 | 126 | ACCESS_ALLOWED_COMPOUND_ACE_TYPE = 4 127 | ACCESS_MAX_MS_V3_ACE_TYPE = 4 128 | 129 | # The following are Win2k only. 130 | ACCESS_MIN_MS_OBJECT_ACE_TYPE = 5 131 | ACCESS_ALLOWED_OBJECT_ACE_TYPE = 5 132 | ACCESS_DENIED_OBJECT_ACE_TYPE = 6 133 | SYSTEM_AUDIT_OBJECT_ACE_TYPE = 7 134 | SYSTEM_ALARM_OBJECT_ACE_TYPE = 8 135 | ACCESS_MAX_MS_OBJECT_ACE_TYPE = 8 136 | ACCESS_MAX_MS_V4_ACE_TYPE = 8 137 | 138 | # This one is for WinNT/2k. 139 | ACCESS_MAX_MS_ACE_TYPE = 8 140 | 141 | 142 | class ACE_FLAGS: 143 | """ 144 | One byte. 145 | """ 146 | 147 | OBJECT_INHERIT_ACE = 0x01 148 | CONTAINER_INHERIT_ACE = 0x02 149 | NO_PROPAGATE_INHERIT_ACE = 0x04 150 | INHERIT_ONLY_ACE = 0x08 151 | INHERITED_ACE = 0x10 # Win2k only. 152 | VALID_INHERIT_FLAGS = 0x1F 153 | 154 | # The audit flags. 155 | SUCCESSFUL_ACCESS_ACE_FLAG = 0x40 156 | FAILED_ACCESS_ACE_FLAG = 0x80 157 | 158 | 159 | class ACCESS_MASK: 160 | """ 161 | DWORD. 162 | """ 163 | 164 | FILE_READ_DATA = 0x00000001 165 | FILE_LIST_DIRECTORY = 0x00000001 166 | FILE_WRITE_DATA = 0x00000002 167 | FILE_ADD_FILE = 0x00000002 168 | FILE_APPEND_DATA = 0x00000004 169 | FILE_ADD_SUBDIRECTORY = 0x00000004 170 | FILE_READ_EA = 0x00000008 171 | FILE_WRITE_EA = 0x00000010 172 | FILE_EXECUTE = 0x00000020 173 | FILE_TRAVERSE = 0x00000020 174 | FILE_DELETE_CHILD = 0x00000040 175 | FILE_READ_ATTRIBUTES = 0x00000080 176 | FILE_WRITE_ATTRIBUTES = 0x00000100 177 | DELETE = 0x00010000 178 | READ_CONTROL = 0x00020000 179 | WRITE_DAC = 0x00040000 180 | WRITE_OWNER = 0x00080000 181 | SYNCHRONIZE = 0x00100000 182 | STANDARD_RIGHTS_READ = 0x00020000 183 | STANDARD_RIGHTS_WRITE = 0x00020000 184 | STANDARD_RIGHTS_EXECUTE = 0x00020000 185 | STANDARD_RIGHTS_REQUIRED = 0x000F0000 186 | STANDARD_RIGHTS_ALL = 0x001F0000 187 | ACCESS_SYSTEM_SECURITY = 0x01000000 188 | MAXIMUM_ALLOWED = 0x02000000 189 | GENERIC_ALL = 0x10000000 190 | GENERIC_EXECUTE = 0x20000000 191 | GENERIC_WRITE = 0x40000000 192 | GENERIC_READ = 0x80000000 193 | 194 | 195 | class ACE(Block): 196 | def __init__(self, buf, offset, parent): 197 | super(ACE, self).__init__(buf, offset) 198 | self.declare_field("byte", "ace_type", 0x0) 199 | self.declare_field("byte", "ace_flags") 200 | 201 | @staticmethod 202 | def get_ace(buf, offset, parent): 203 | header = ACE(buf, offset, parent) 204 | if header.ace_type() == ACE_TYPES.ACCESS_ALLOWED_ACE_TYPE: 205 | return ACCESS_ALLOWED_ACE(buf, offset, parent) 206 | elif header.ace_type() == ACE_TYPES.ACCESS_DENIED_ACE_TYPE: 207 | return ACCESS_DENIED_ACE(buf, offset, parent) 208 | elif header.ace_type() == ACE_TYPES.SYSTEM_AUDIT_ACE_TYPE: 209 | return SYSTEM_AUDIT_ACE(buf, offset, parent) 210 | elif header.ace_type() == ACE_TYPES.SYSTEM_ALARM_ACE_TYPE: 211 | return SYSTEM_ALARM_ACE(buf, offset, parent) 212 | elif header.ace_type() == ACE_TYPES.ACCESS_ALLOWED_OBJECT_ACE_TYPE: 213 | return ACCESS_ALLOWED_OBJECT_ACE(buf, offset, parent) 214 | elif header.ace_type() == ACE_TYPES.ACCESS_DENIED_OBJECT_ACE_TYPE: 215 | return ACCESS_DENIED_OBJECT_ACE(buf, offset, parent) 216 | elif header.ace_type() == ACE_TYPES.SYSTEM_AUDIT_OBJECT_ACE_TYPE: 217 | return SYSTEM_AUDIT_OBJECT_ACE(buf, offset, parent) 218 | elif header.ace_type() == ACE_TYPES.SYSTEM_ALARM_OBJECT_ACE_TYPE: 219 | return SYSTEM_ALARM_OBJECT_ACE(buf, offset, parent) 220 | else: 221 | raise ParseException("unknown ACE type") 222 | 223 | 224 | class StandardACE(ACE, Nestable): 225 | def __init__(self, buf, offset, parent): 226 | super(StandardACE, self).__init__(buf, offset, parent) 227 | self.declare_field("word", "size", 0x2) 228 | self.declare_field("dword", "access_mask") 229 | self.declare_field(SID, "sid") 230 | 231 | @staticmethod 232 | def structure_size(buf, offset, parent): 233 | return read_word(buf, offset + 0x2) 234 | 235 | def __len__(self): 236 | return self.size() 237 | 238 | 239 | class ACCESS_ALLOWED_ACE(StandardACE): 240 | def __init__(self, buf, offset, parent): 241 | super(ACCESS_ALLOWED_ACE, self).__init__(buf, offset, parent) 242 | 243 | 244 | class ACCESS_DENIED_ACE(StandardACE): 245 | def __init__(self, buf, offset, parent): 246 | super(ACCESS_DENIED_ACE, self).__init__(buf, offset, parent) 247 | 248 | 249 | class SYSTEM_AUDIT_ACE(StandardACE): 250 | def __init__(self, buf, offset, parent): 251 | super(SYSTEM_AUDIT_ACE, self).__init__(buf, offset, parent) 252 | 253 | 254 | class SYSTEM_ALARM_ACE(StandardACE): 255 | def __init__(self, buf, offset, parent): 256 | super(SYSTEM_ALARM_ACE, self).__init__(buf, offset, parent) 257 | 258 | 259 | class OBJECT_ACE_FLAGS: 260 | """ 261 | DWORD. 262 | """ 263 | 264 | ACE_OBJECT_TYPE_PRESENT = 1 265 | ACE_INHERITED_OBJECT_TYPE_PRESENT = 2 266 | 267 | 268 | class ObjectACE(ACE, Nestable): 269 | def __init__(self, buf, offset, parent): 270 | super(ObjectACE, self).__init__(buf, offset, parent) 271 | self.declare_field("word", "size", 0x2) 272 | self.declare_field("dword", "access_mask") 273 | self.declare_field("dword", "object_flags") 274 | self.declare_field("guid", "object_type") 275 | self.declare_field("guid", "inherited_object_type") 276 | 277 | @staticmethod 278 | def structure_size(buf, offset, parent): 279 | return read_word(buf, offset + 0x2) 280 | 281 | def __len__(self): 282 | return self.size() 283 | 284 | 285 | class ACCESS_ALLOWED_OBJECT_ACE(ObjectACE): 286 | def __init__(self, buf, offset, parent): 287 | super(ACCESS_ALLOWED_OBJECT_ACE, self).__init__(buf, offset, parent) 288 | 289 | 290 | class ACCESS_DENIED_OBJECT_ACE(ObjectACE): 291 | def __init__(self, buf, offset, parent): 292 | super(ACCESS_DENIED_OBJECT_ACE, self).__init__(buf, offset, parent) 293 | 294 | 295 | class SYSTEM_AUDIT_OBJECT_ACE(ObjectACE): 296 | def __init__(self, buf, offset, parent): 297 | super(SYSTEM_AUDIT_OBJECT_ACE, self).__init__(buf, offset, parent) 298 | 299 | 300 | class SYSTEM_ALARM_OBJECT_ACE(ObjectACE): 301 | def __init__(self, buf, offset, parent): 302 | super(SYSTEM_ALARM_OBJECT_ACE, self).__init__(buf, offset, parent) 303 | 304 | 305 | class ACL(Block, Nestable): 306 | def __init__(self, buf, offset, parent): 307 | super(ACL, self).__init__(buf, offset) 308 | self.declare_field("byte", "revision", 0x0) 309 | self.declare_field("byte", "alignment1") 310 | self.declare_field("word", "size") 311 | self.declare_field("word", "ace_count") 312 | self.declare_field("word", "alignment2") 313 | self._off_ACEs = self.current_field_offset() 314 | self.add_explicit_field(self._off_ACEs, ACE, "ACEs") 315 | 316 | @staticmethod 317 | def structure_size(buf, offset, parent): 318 | return read_word(buf, offset + 0x2) 319 | 320 | def __len__(self): 321 | return self.size() 322 | 323 | def ACEs(self): 324 | ofs = self._off_ACEs 325 | for _ in range(self.ace_count()): 326 | a = ACE.get_ace(self._buf, self.offset() + ofs, self) 327 | yield a 328 | ofs += a.size() 329 | ofs = align(ofs, 4) 330 | 331 | 332 | class NULL_ACL(object): 333 | """ 334 | TODO(wb): Not actually sure what the NULL ACL is... 335 | just guessing at the values here. 336 | """ 337 | 338 | def __init__(self): 339 | super(NULL_ACL, self).__init__() 340 | 341 | def revision(self): 342 | return 1 343 | 344 | def alignment1(self): 345 | return 0 346 | 347 | def size(self): 348 | return 0 349 | 350 | def ace_count(self): 351 | return 0 352 | 353 | def ACEs(self): 354 | return 355 | 356 | @staticmethod 357 | def structure_size(buf, offset, parent): 358 | return 0 359 | 360 | def __len__(self): 361 | return 0 362 | 363 | 364 | class SECURITY_DESCRIPTOR_RELATIVE(Block, Nestable): 365 | def __init__(self, buf, offset, parent): 366 | super(SECURITY_DESCRIPTOR_RELATIVE, self).__init__(buf, offset) 367 | self.declare_field("byte", "revision", 0x0) 368 | self.declare_field("byte", "alignment") 369 | self.declare_field("word", "control") 370 | self.declare_field("dword", "owner_offset") 371 | self.declare_field("dword", "group_offset") 372 | self.declare_field("dword", "sacl_offset") 373 | self.declare_field("dword", "dacl_offset") 374 | 375 | self.add_explicit_field(self.owner_offset(), "SID", "owner") 376 | self.add_explicit_field(self.group_offset(), "SID", "group") 377 | if self.control() & SECURITY_DESCRIPTOR_CONTROL.SE_SACL_PRESENT: 378 | self.add_explicit_field(self.sacl_offset(), "ACL", "sacl") 379 | if self.control() & SECURITY_DESCRIPTOR_CONTROL.SE_DACL_PRESENT: 380 | self.add_explicit_field(self.dacl_offset(), "ACL", "dacl") 381 | 382 | @staticmethod 383 | def structure_size(buf, offset, parent): 384 | return len(SECURITY_DESCRIPTOR_RELATIVE(buf, offset, parent)) 385 | 386 | def __len__(self): 387 | ret = 20 388 | ret += len((self.owner() or null_object)) 389 | ret += len((self.group() or null_object)) 390 | ret += len((self.sacl() or null_object)) 391 | ret += len((self.dacl() or null_object)) 392 | return ret 393 | 394 | def owner(self): 395 | if self.owner_offset() != 0: 396 | return SID(self._buf, self.absolute_offset(self.owner_offset()), self) 397 | else: 398 | return None 399 | 400 | def group(self): 401 | if self.group_offset() != 0: 402 | return SID(self._buf, self.absolute_offset(self.group_offset()), self) 403 | else: 404 | return None 405 | 406 | def sacl(self): 407 | if self.control() & SECURITY_DESCRIPTOR_CONTROL.SE_SACL_PRESENT: 408 | if self.sacl_offset() > 0: 409 | return ACL(self._buf, self.absolute_offset(self.sacl_offset()), self) 410 | else: 411 | return NULL_ACL() 412 | else: 413 | return None 414 | 415 | def dacl(self): 416 | if self.control() & SECURITY_DESCRIPTOR_CONTROL.SE_DACL_PRESENT: 417 | if self.dacl_offset() > 0: 418 | return ACL(self._buf, self.absolute_offset(self.dacl_offset()), self) 419 | else: 420 | return NULL_ACL() 421 | else: 422 | return None 423 | 424 | 425 | class SDS_ENTRY(Block, Nestable): 426 | def __init__(self, buf, offset, parent): 427 | super(SDS_ENTRY, self).__init__(buf, offset) 428 | self.declare_field("dword", "hash", 0x0) 429 | self.declare_field("dword", "security_id") 430 | self.declare_field("qword", "offset") 431 | self.declare_field("dword", "length") 432 | self.declare_field(SECURITY_DESCRIPTOR_RELATIVE, "sid") 433 | 434 | @staticmethod 435 | def structure_size(buf, offset, parent): 436 | return read_dword(buf, offset + 0x10) 437 | 438 | def __len__(self): 439 | return self.length() 440 | 441 | 442 | class SDS(Block): 443 | def __init__(self, buf, offset, parent): 444 | super(SDS, self).__init__(buf, offset) 445 | self.add_explicit_field(0, SDS, "sds_entries") 446 | 447 | def sds_entries(self): 448 | ofs = 0 449 | while len(self._buf) > self.offset() + ofs + 0x14: 450 | s = SDS_ENTRY(self._buf, self.offset() + ofs, self) 451 | if len(s) != 0: 452 | yield s 453 | ofs += len(s) 454 | ofs = align(ofs, 0x10) 455 | else: 456 | if ofs % 0x10000 == 0: 457 | return 458 | else: 459 | ofs = align(ofs, 0x10000) 460 | 461 | 462 | def main(): 463 | import contextlib 464 | import mmap 465 | import sys 466 | 467 | with open(sys.argv[1], "r") as f: 468 | with contextlib.closing( 469 | mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) 470 | ) as buf: 471 | s = SDS(buf, 0, None) 472 | print("SDS") 473 | for e in s.sds_entries(): 474 | print(" SDS_ENTRY") 475 | print((e.get_all_string(indent=2))) 476 | 477 | 478 | if __name__ == "__main__": 479 | main() 480 | -------------------------------------------------------------------------------- /indxparse/SDS_get_index.py: -------------------------------------------------------------------------------- 1 | #!/bin/python 2 | 3 | # This file is part of INDXParse. 4 | # 5 | # Copyright 2011-13 Will Ballenthin 6 | # while at Mandiant 7 | # 8 | # Licensed under the Apache License, Version 2.0 (the "License"); 9 | # you may not use this file except in compliance with the License. 10 | # You may obtain a copy of the License at 11 | # 12 | # http://www.apache.org/licenses/LICENSE-2.0 13 | # 14 | # Unless required by applicable law or agreed to in writing, software 15 | # distributed under the License is distributed on an "AS IS" BASIS, 16 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 17 | # See the License for the specific language governing permissions and 18 | # limitations under the License. 19 | # 20 | # 21 | # Alex Nelson, NIST, contributed to this file. Contributions of NIST 22 | # are not subject to US Copyright. 23 | # 24 | # 25 | # Version v.1.2 26 | from indxparse.SDS import SDS 27 | 28 | 29 | def main(): 30 | import argparse 31 | import contextlib 32 | import mmap 33 | 34 | parser = argparse.ArgumentParser(description="Get an SDS record by index.") 35 | parser.add_argument( 36 | "-v", action="store_true", dest="verbose", help="Print debugging information" 37 | ) 38 | parser.add_argument("SDS", action="store", help="Input SDS file path") 39 | parser.add_argument("index", action="store", type=int, help="Entry index to fetch") 40 | results = parser.parse_args() 41 | 42 | with open(results.SDS, "r") as f: 43 | with contextlib.closing( 44 | mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) 45 | ) as buf: 46 | s = SDS(buf, 0, None) 47 | print("SDS") 48 | for e in s.sds_entries(): 49 | print(" SDS_ENTRY") 50 | print((e.get_all_string(indent=2))) 51 | 52 | 53 | if __name__ == "__main__": 54 | main() 55 | -------------------------------------------------------------------------------- /indxparse/SortedCollection.py: -------------------------------------------------------------------------------- 1 | """ 2 | From http://code.activestate.com/recipes/577197-sortedcollection/ 3 | """ 4 | from bisect import bisect_left, bisect_right 5 | 6 | 7 | class SortedCollection(object): 8 | """Sequence sorted by a key function. 9 | 10 | SortedCollection() is much easier to work with than using bisect() directly. 11 | It supports key functions like those use in sorted(), min(), and max(). 12 | The result of the key function call is saved so that keys can be searched 13 | efficiently. 14 | 15 | Instead of returning an insertion-point which can be hard to interpret, the 16 | five find-methods return a specific item in the sequence. They can scan for 17 | exact matches, the last item less-than-or-equal to a key, or the first item 18 | greater-than-or-equal to a key. 19 | 20 | Once found, an item's ordinal position can be located with the index() method. 21 | New items can be added with the insert() and insert_right() methods. 22 | Old items can be deleted with the remove() method. 23 | 24 | The usual sequence methods are provided to support indexing, slicing, 25 | length lookup, clearing, copying, forward and reverse iteration, contains 26 | checking, item counts, item removal, and a nice looking repr. 27 | 28 | Finding and indexing are O(log n) operations while iteration and insertion 29 | are O(n). The initial sort is O(n log n). 30 | 31 | The key function is stored in the 'key' attibute for easy introspection or 32 | so that you can assign a new key function (triggering an automatic re-sort). 33 | 34 | In short, the class was designed to handle all of the common use cases for 35 | bisect but with a simpler API and support for key functions. 36 | 37 | >>> from pprint import pprint 38 | >>> from operator import itemgetter 39 | 40 | >>> s = SortedCollection(key=itemgetter(2)) 41 | >>> for record in [ 42 | ... ('roger', 'young', 30), 43 | ... ('angela', 'jones', 28), 44 | ... ('bill', 'smith', 22), 45 | ... ('david', 'thomas', 32)]: 46 | ... s.insert(record) 47 | 48 | >>> pprint(list(s)) # show records sorted by age 49 | [('bill', 'smith', 22), 50 | ('angela', 'jones', 28), 51 | ('roger', 'young', 30), 52 | ('david', 'thomas', 32)] 53 | 54 | >>> s.find_le(29) # find oldest person aged 29 or younger 55 | ('angela', 'jones', 28) 56 | >>> s.find_lt(28) # find oldest person under 28 57 | ('bill', 'smith', 22) 58 | >>> s.find_gt(28) # find youngest person over 28 59 | ('roger', 'young', 30) 60 | 61 | >>> r = s.find_ge(32) # find youngest person aged 32 or older 62 | >>> s.index(r) # get the index of their record 63 | 3 64 | >>> s[3] # fetch the record at that index 65 | ('david', 'thomas', 32) 66 | 67 | >>> s.key = itemgetter(0) # now sort by first name 68 | >>> pprint(list(s)) 69 | [('angela', 'jones', 28), 70 | ('bill', 'smith', 22), 71 | ('david', 'thomas', 32), 72 | ('roger', 'young', 30)] 73 | 74 | """ 75 | 76 | def __init__(self, iterable=(), key=None): 77 | self._given_key = key 78 | key = (lambda x: x) if key is None else key 79 | decorated = sorted((key(item), item) for item in iterable) 80 | self._keys = [k for k, item in decorated] 81 | self._items = [item for k, item in decorated] 82 | self._key = key 83 | 84 | def _getkey(self): 85 | return self._key 86 | 87 | def _setkey(self, key): 88 | if key is not self._key: 89 | self.__init__(self._items, key=key) 90 | 91 | def _delkey(self): 92 | self._setkey(None) 93 | 94 | key = property(_getkey, _setkey, _delkey, "key function") 95 | 96 | def clear(self): 97 | self.__init__([], self._key) 98 | 99 | def copy(self): 100 | return self.__class__(self, self._key) 101 | 102 | def __len__(self): 103 | return len(self._items) 104 | 105 | def __getitem__(self, i): 106 | return self._items[i] 107 | 108 | def __iter__(self): 109 | return iter(self._items) 110 | 111 | def __reversed__(self): 112 | return reversed(self._items) 113 | 114 | def __repr__(self): 115 | return "%s(%r, key=%s)" % ( 116 | self.__class__.__name__, 117 | self._items, 118 | getattr(self._given_key, "__name__", repr(self._given_key)), 119 | ) 120 | 121 | def __reduce__(self): 122 | return self.__class__, (self._items, self._given_key) 123 | 124 | def __contains__(self, item): 125 | k = self._key(item) 126 | i = bisect_left(self._keys, k) 127 | j = bisect_right(self._keys, k) 128 | return item in self._items[i:j] 129 | 130 | def index(self, item): 131 | "Find the position of an item. Raise ValueError if not found." 132 | k = self._key(item) 133 | i = bisect_left(self._keys, k) 134 | j = bisect_right(self._keys, k) 135 | return self._items[i:j].index(item) + i 136 | 137 | def count(self, item): 138 | "Return number of occurrences of item" 139 | k = self._key(item) 140 | i = bisect_left(self._keys, k) 141 | j = bisect_right(self._keys, k) 142 | return self._items[i:j].count(item) 143 | 144 | def insert(self, item): 145 | "Insert a new item. If equal keys are found, add to the left" 146 | k = self._key(item) 147 | i = bisect_left(self._keys, k) 148 | self._keys.insert(i, k) 149 | self._items.insert(i, item) 150 | 151 | def insert_right(self, item): 152 | "Insert a new item. If equal keys are found, add to the right" 153 | k = self._key(item) 154 | i = bisect_right(self._keys, k) 155 | self._keys.insert(i, k) 156 | self._items.insert(i, item) 157 | 158 | def remove(self, item): 159 | "Remove first occurence of item. Raise ValueError if not found" 160 | i = self.index(item) 161 | del self._keys[i] 162 | del self._items[i] 163 | 164 | def find(self, k): 165 | "Return first item with a key == k. Raise ValueError if not found." 166 | i = bisect_left(self._keys, k) 167 | if i != len(self) and self._keys[i] == k: 168 | return self._items[i] 169 | raise ValueError("No item found with key equal to: %r" % (k,)) 170 | 171 | def find_le(self, k): 172 | "Return last item with a key <= k. Raise ValueError if not found." 173 | i = bisect_right(self._keys, k) 174 | if i: 175 | return self._items[i - 1] 176 | raise ValueError("No item found with key at or below: %r" % (k,)) 177 | 178 | def find_lt(self, k): 179 | "Return last item with a key < k. Raise ValueError if not found." 180 | i = bisect_left(self._keys, k) 181 | if i: 182 | return self._items[i - 1] 183 | raise ValueError("No item found with key below: %r" % (k,)) 184 | 185 | def find_ge(self, k): 186 | "Return first item with a key >= equal to k. Raise ValueError if not found" 187 | i = bisect_left(self._keys, k) 188 | if i != len(self): 189 | return self._items[i] 190 | raise ValueError("No item found with key at or above: %r" % (k,)) 191 | 192 | def find_gt(self, k): 193 | "Return first item with a key > k. Raise ValueError if not found" 194 | i = bisect_right(self._keys, k) 195 | if i != len(self): 196 | return self._items[i] 197 | raise ValueError("No item found with key above: %r" % (k,)) 198 | -------------------------------------------------------------------------------- /indxparse/__init__.py: -------------------------------------------------------------------------------- 1 | # Alex Nelson, NIST, contributed to this file. Contributions of NIST 2 | # are not subject to US Copyright. 3 | 4 | __version__ = "1.1.9" 5 | 6 | from . import ( # noqa: F401 7 | MFT, 8 | SDS, 9 | BinaryParser, 10 | FileMap, 11 | Progress, 12 | SortedCollection, 13 | get_file_info, 14 | ) 15 | -------------------------------------------------------------------------------- /indxparse/carve_mft_records.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python2 2 | 3 | # Alex Nelson, NIST, contributed to this file. Contributions of NIST 4 | # are not subject to US Copyright. 5 | 6 | """ 7 | Carve MFT records from arbitrary binary data. 8 | 9 | author: Willi Ballenthin 10 | email: william.ballenthin@fireeye.com 11 | """ 12 | import argparse 13 | import array 14 | import contextlib 15 | import logging 16 | import mmap 17 | import os 18 | import sys 19 | 20 | import indxparse.MFT 21 | 22 | logger = logging.getLogger(__name__) 23 | 24 | 25 | def sizeof_fmt(num, suffix="B"): 26 | """ 27 | via: http://stackoverflow.com/a/1094933/87207 28 | """ 29 | 30 | for unit in ["", "Ki", "Mi", "Gi", "Ti", "Pi", "Ei", "Zi"]: 31 | if abs(num) < 1024.0: 32 | return "%3.1f%s%s" % (num, unit, suffix) 33 | num /= 1024.0 34 | return "%.1f%s%s" % (num, "Yi", suffix) 35 | 36 | 37 | class BadRecord(Exception): 38 | pass 39 | 40 | 41 | def output_record(record_offset, record): 42 | ret = [] 43 | 44 | ret.append(hex(record_offset)) 45 | ret.append(record.filename_information().filename()) 46 | 47 | data = record.data_attribute() 48 | ret.append(hex(data.allocated_size())) 49 | 50 | if data.non_resident() == 0: 51 | logger.warn("unexpected resident data") 52 | raise BadRecord() 53 | 54 | if data.allocated_size() == 0: 55 | logger.warn("unexpected zero length") 56 | raise BadRecord() 57 | 58 | for offset, length in data.runlist().runs(): 59 | logger.debug( 60 | "run offset: %s clusters, length: %s clusters (%s / %s in bytes)", 61 | offset, 62 | length, 63 | offset * 4096, 64 | length * 4096, 65 | ) 66 | 67 | off, size = list(data.runlist().runs())[0] 68 | ret.append(hex(off)) 69 | ret.append(hex(size)) 70 | 71 | print((",".join(ret))) 72 | 73 | 74 | def main(argv=None): 75 | if argv is None: 76 | argv = sys.argv[1:] 77 | 78 | parser = argparse.ArgumentParser(description="A program.") 79 | parser.add_argument("input", type=str, help="Path to input file") 80 | parser.add_argument( 81 | "-v", "--verbose", action="store_true", help="Enable debug logging" 82 | ) 83 | parser.add_argument( 84 | "-q", "--quiet", action="store_true", help="Disable all output but errors" 85 | ) 86 | args = parser.parse_args() 87 | 88 | if args.verbose: 89 | logging.basicConfig(level=logging.DEBUG) 90 | elif args.quiet: 91 | logging.basicConfig(level=logging.ERROR) 92 | else: 93 | logging.basicConfig(level=logging.INFO) 94 | 95 | HEADER = "FILE0" 96 | total_size = os.path.getsize(args.input) 97 | 98 | count = 0 99 | with open(args.input, "rb") as f: 100 | with contextlib.closing(mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)) as m: 101 | offset = 0 102 | while True: 103 | offset = m.find(HEADER, offset) 104 | 105 | if offset == -1: 106 | break 107 | 108 | if offset % 0x10 != 0: 109 | offset += 1 110 | continue 111 | 112 | buf = array.array("B", m[offset : offset + 1024]) 113 | record = indxparse.MFT.MFTRecord(buf, 0, None) 114 | output_record(offset, record) 115 | count += 1 116 | 117 | if count % 1000 == 0: 118 | logger.info( 119 | "%s: found %d records over %s bytes (of %s total), %.2f complete", 120 | hex(offset), 121 | count, 122 | sizeof_fmt(offset), 123 | sizeof_fmt(total_size), 124 | float(offset) / total_size, 125 | ) 126 | offset += 1 127 | 128 | 129 | if __name__ == "__main__": 130 | sys.exit(main()) 131 | -------------------------------------------------------------------------------- /indxparse/extract_mft_record_slack.py: -------------------------------------------------------------------------------- 1 | #!/bin/python 2 | 3 | # This file is part of INDXParse. 4 | # 5 | # Copyright 2014 Will Ballenthin 6 | # while at FireEye 7 | # 8 | # Licensed under the Apache License, Version 2.0 (the "License"); 9 | # you may not use this file except in compliance with the License. 10 | # You may obtain a copy of the License at 11 | # 12 | # http://www.apache.org/licenses/LICENSE-2.0 13 | # 14 | # Unless required by applicable law or agreed to in writing, software 15 | # distributed under the License is distributed on an "AS IS" BASIS, 16 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 17 | # See the License for the specific language governing permissions and 18 | # limitations under the License. 19 | # 20 | # 21 | # Alex Nelson, NIST, contributed to this file. Contributions of NIST 22 | # are not subject to US Copyright. 23 | import mmap 24 | import sys 25 | 26 | from indxparse.MFT import MFTEnumerator 27 | 28 | 29 | def main() -> None: 30 | filename = sys.argv[1] 31 | 32 | with open(filename, "rb") as fh: 33 | with mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ) as mm: 34 | enum = MFTEnumerator(mm) 35 | for record in enum.enumerate_records(): 36 | slack = record.slack_data() 37 | sys.stdout.buffer.write(b"\x00" * (1024 - len(slack))) 38 | sys.stdout.buffer.write(slack) 39 | 40 | 41 | if __name__ == "__main__": 42 | main() 43 | -------------------------------------------------------------------------------- /indxparse/fuse-mft.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # Alex Nelson, NIST, contributed to this file. Contributions of NIST 4 | # are not subject to US Copyright. 5 | 6 | 7 | import calendar 8 | import errno 9 | import inspect 10 | import mmap 11 | import os 12 | import stat 13 | import sys 14 | from typing import Dict 15 | 16 | from fuse import FUSE, FuseOSError, Operations, fuse_get_context # type: ignore 17 | 18 | from indxparse.get_file_info import format_record 19 | from indxparse.MFT import Cache, MFTEnumerator, MFTRecord, MFTTree 20 | from indxparse.Progress import ProgressBarProgress 21 | 22 | PERMISSION_ALL_READ = int("444", 8) 23 | 24 | 25 | def unixtimestamp(ts): 26 | """ 27 | unixtimestamp converts a datetime.datetime to a UNIX timestamp. 28 | @type ts: datetime.datetime 29 | @rtype: int 30 | """ 31 | return calendar.timegm(ts.utctimetuple()) 32 | 33 | 34 | def log(func): 35 | """ 36 | log is a decorator that logs the a function call with its 37 | parameters and return value. 38 | """ 39 | 40 | def inner(*args, **kwargs): 41 | func_name = inspect.stack()[3][3] 42 | if func_name == "_wrapper": 43 | func_name = inspect.stack()[2][3] 44 | (uid, gid, pid) = fuse_get_context() 45 | pre = "(%s: UID=%d GID=%d PID=%d ARGS=(%s) KWARGS=(%s))" % ( 46 | func_name, 47 | uid, 48 | gid, 49 | pid, 50 | ", ".join(map(str, list(args)[1:])), 51 | str(**kwargs), 52 | ) 53 | try: 54 | ret = func(*args, **kwargs) 55 | post = " +--> %s" % (str(ret)) 56 | sys.stderr.write("%s\n%s\n" % (pre, post)) 57 | return ret 58 | except Exception as e: 59 | post = " +--> %s" % (str(e)) 60 | sys.stderr.write("%s\n%s" % (pre, post)) 61 | raise e 62 | 63 | return inner 64 | 65 | 66 | class FH(object): 67 | """ 68 | FH is a class used to represent a file handle. 69 | Subclass it and override the get_data and get_size methods 70 | for specific behavior. 71 | """ 72 | 73 | def __init__(self, fh, record): 74 | super(FH, self).__init__() 75 | self._fh = fh 76 | self._record = record 77 | 78 | def get_data(self): 79 | """ 80 | Return a bytestring containing the data of the opened file. 81 | @rtype: str 82 | """ 83 | raise RuntimeError("FH.get_data not implemented") 84 | 85 | def get_size(self): 86 | """ 87 | @rtype: int 88 | """ 89 | raise RuntimeError("FH.get_size not implemented") 90 | 91 | def get_fh(self): 92 | return self._fh 93 | 94 | 95 | class RegularFH(FH): 96 | """ 97 | RegularFH is a class used to represent an open file. 98 | """ 99 | 100 | def __init__(self, fh, record): 101 | super(RegularFH, self).__init__(fh, record) 102 | 103 | def get_data(self): 104 | data_attribute = self._record.data_attribute() 105 | if data_attribute is not None and data_attribute.non_resident() == 0: 106 | return data_attribute.value() 107 | return "" 108 | 109 | def get_size(self): 110 | data_attribute = self._record.data_attribute() 111 | if data_attribute is not None: 112 | if data_attribute.non_resident() == 0: 113 | return len(self.get_data()) 114 | else: 115 | return data_attribute.data_size() 116 | else: 117 | return self._record.standard_information.logical_size() 118 | 119 | 120 | def get_meta_for_file(record: MFTRecord, path: str) -> str: 121 | """ 122 | Given an MFT record, print out metadata about the relevant file. 123 | @type record: MFT.MFTRecord 124 | @type path: str 125 | @rtype: str 126 | """ 127 | return format_record(record, path) 128 | 129 | 130 | class MetaFH(FH): 131 | """ 132 | A class used to represent a virtual file containing metadata 133 | for a regular file. 134 | """ 135 | 136 | def __init__(self, fh, record, path, record_buf): 137 | super(MetaFH, self).__init__(fh, record) 138 | self._path = path 139 | self._record_buf = record_buf 140 | 141 | def get_data(self): 142 | return get_meta_for_file(self._record, self._path) 143 | 144 | def get_size(self): 145 | return len(self.get_data()) 146 | 147 | 148 | def is_special_file(path): 149 | """ 150 | is_special_file returns true if the file path is a special/virtual file. 151 | @type path: str 152 | @rtype: boolean 153 | """ 154 | return "::" in path.rpartition("/")[2] 155 | 156 | 157 | def explode_special_file(path): 158 | """ 159 | explode_special_file breaks apart the path of a special/virtual file into 160 | its base path and special file identifier. 161 | @type path: str 162 | @rtype: (str, str) 163 | """ 164 | (base, _, special) = path.rpartition("::") 165 | return base, special 166 | 167 | 168 | class MFTFuseOperations(Operations): 169 | """ 170 | MFTFuseOperations is a FUSE driver for NTFS MFT files. 171 | """ 172 | 173 | def __init__(self, root, mfttree, buf: mmap.mmap) -> None: 174 | self._root = root 175 | self._tree = mfttree 176 | self._buf = buf 177 | self._opened_files: Dict[int, FH] = {} 178 | 179 | record_cache = Cache(1024) 180 | path_cache = Cache(1024) 181 | 182 | self._enumerator = MFTEnumerator( 183 | buf, record_cache=record_cache, path_cache=path_cache 184 | ) 185 | 186 | # Helpers 187 | # ======= 188 | def _get_node(self, path): 189 | """ 190 | _get_node returns the MFTTreeNode associated with a path. 191 | @type path: str 192 | @rtype: MFT.MFTTreeNode 193 | @raises: FuseOSError(errno.ENOENT) 194 | """ 195 | if path.startswith("/"): 196 | path = path[1:] 197 | 198 | current_node = self._tree.get_root() 199 | for component in path.split("/"): 200 | if component == "": 201 | continue 202 | try: 203 | current_node = current_node.get_child_node(component) 204 | except KeyError: 205 | raise FuseOSError(errno.ENOENT) 206 | 207 | return current_node 208 | 209 | def _get_record(self, path): 210 | """ 211 | _get_record returns the MFTRecord associated with a path. 212 | @type path: str 213 | @rtype: MFT.MFTRecord 214 | """ 215 | return self._enumerator.get_record(self._get_node(path).get_record_number()) 216 | 217 | # Filesystem methods 218 | # ================== 219 | @log 220 | def getattr(self, path, fh=None): 221 | (uid, gid, pid) = fuse_get_context() 222 | 223 | working_path = path 224 | 225 | if is_special_file(path): 226 | (working_path, special) = explode_special_file(working_path) 227 | 228 | record = self._get_record(working_path) 229 | if record.is_directory(): 230 | mode = stat.S_IFDIR | PERMISSION_ALL_READ 231 | nlink = 2 232 | else: 233 | mode = stat.S_IFREG | PERMISSION_ALL_READ 234 | nlink = 1 235 | 236 | # TODO(wb): fix the duplication of this code with the FH classes 237 | if is_special_file(path): 238 | size = 0 239 | (working_path, special) = explode_special_file(path) 240 | if special == "meta": 241 | node = self._get_node(working_path) 242 | # TODO - Check to see if anything further in the control flow relies on a side-effect of this call. 243 | _ = self._enumerator.get_record_buf(node.get_record_number()) 244 | size = len(get_meta_for_file(record, working_path)) 245 | else: 246 | data_attribute = record.data_attribute() 247 | if data_attribute is not None: 248 | if data_attribute.non_resident() == 0: 249 | size = len(data_attribute.value()) 250 | else: 251 | size = data_attribute.data_size() 252 | else: 253 | size = record.filename_information().logical_size() 254 | 255 | return { 256 | "st_atime": unixtimestamp(record.standard_information().accessed_time()), 257 | "st_ctime": unixtimestamp(record.standard_information().changed_time()), 258 | # "st_crtime": unixtimestamp(record.standard_information().created_time()), 259 | "st_mtime": unixtimestamp(record.standard_information().modified_time()), 260 | "st_size": size, 261 | "st_uid": uid, 262 | "st_gid": gid, 263 | "st_mode": mode, 264 | "st_nlink": nlink, 265 | } 266 | 267 | @log 268 | def readdir(self, path, fh): 269 | dirents = [".", ".."] 270 | record = self._get_node(path) 271 | dirents.extend([r.get_filename() for r in record.get_children_nodes()]) 272 | for r in dirents: 273 | yield r 274 | 275 | @log 276 | def readlink(self, path): 277 | return path 278 | 279 | @log 280 | def statfs(self, path): 281 | return dict( 282 | (key, 0) 283 | for key in ( 284 | "f_bavail", 285 | "f_bfree", 286 | "f_blocks", 287 | "f_bsize", 288 | "f_favail", 289 | "f_ffree", 290 | "f_files", 291 | "f_flag", 292 | "f_frsize", 293 | "f_namemax", 294 | ) 295 | ) 296 | 297 | @log 298 | def chmod(self, path, mode): 299 | return errno.EROFS 300 | 301 | @log 302 | def chown(self, path, uid, gid): 303 | return errno.EROFS 304 | 305 | @log 306 | def mknod(self, path, mode, dev): 307 | return errno.EROFS 308 | 309 | @log 310 | def rmdir(self, path): 311 | return errno.EROFS 312 | 313 | @log 314 | def mkdir(self, path, mode): 315 | return errno.EROFS 316 | 317 | @log 318 | def unlink(self, path): 319 | return errno.EROFS 320 | 321 | @log 322 | def symlink(self, target, name): 323 | return errno.EROFS 324 | 325 | @log 326 | def rename(self, old, new): 327 | return errno.EROFS 328 | 329 | @log 330 | def link(self, target, name): 331 | return errno.EROFS 332 | 333 | @log 334 | def utimens(self, path, times=None): 335 | return errno.EROFS 336 | 337 | # File methods 338 | # ============ 339 | 340 | def _get_available_fh(self): 341 | """ 342 | _get_available_fh returns an unused fh 343 | The caller must be careful to handle race conditions. 344 | @rtype: int 345 | """ 346 | for i in range(65534): 347 | if i not in self._opened_files: 348 | return i 349 | 350 | @log 351 | def open(self, path, flags): 352 | if flags & os.O_WRONLY > 0: 353 | return errno.EROFS 354 | if flags & os.O_RDWR > 0: 355 | return errno.EROFS 356 | 357 | # TODO(wb): race here on fh used/unused 358 | fh = self._get_available_fh() 359 | if is_special_file(path): 360 | (path, special) = explode_special_file(path) 361 | if special == "meta": 362 | record = self._get_record(path) 363 | node = self._get_node(path) 364 | record_buf = self._enumerator.get_record_buf(node.get_record_number()) 365 | self._opened_files[fh] = MetaFH(fh, record, path, record_buf) 366 | else: 367 | raise FuseOSError(errno.ENOENT) 368 | else: 369 | self._opened_files[fh] = RegularFH(fh, self._get_record(path)) 370 | 371 | return fh 372 | 373 | @log 374 | def read(self, path, length, offset, fh): 375 | txt = self._opened_files[fh].get_data().encode("utf-8") 376 | return txt[offset : offset + length] 377 | 378 | @log 379 | def flush(self, path, fh): 380 | return "" 381 | 382 | @log 383 | def release(self, path, fh): 384 | del self._opened_files[fh] 385 | 386 | @log 387 | def create(self, path, mode, fi=None): 388 | return errno.EROFS 389 | 390 | @log 391 | def write(self, path, buf, offset, fh): 392 | return errno.EROFS 393 | 394 | @log 395 | def truncate(self, path, length, fh=None): 396 | return errno.EROFS 397 | 398 | @log 399 | def fsync(self, path, fdatasync, fh): 400 | return errno.EPERM 401 | 402 | 403 | def main(): 404 | mft_filename = sys.argv[1] 405 | mountpoint = sys.argv[2] 406 | with open(mft_filename, "rb") as fh: 407 | with mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ) as mm: 408 | tree = MFTTree(mm) 409 | tree.build(progress_class=ProgressBarProgress) 410 | handler = MFTFuseOperations(mountpoint, tree, mm) 411 | FUSE(handler, mountpoint, foreground=True) 412 | 413 | 414 | if __name__ == "__main__": 415 | main() 416 | -------------------------------------------------------------------------------- /indxparse/get_file_info.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | 3 | # Sheldon Douglas, NIST Associate, and Alex Nelson, NIST, contributed 4 | # to this file. Contributions of NIST are not subject to US 5 | # Copyright. 6 | 7 | import argparse 8 | import array 9 | import datetime 10 | import logging 11 | import mmap 12 | import re 13 | from string import printable 14 | from typing import Any, Dict 15 | 16 | from jinja2 import Template 17 | 18 | from indxparse.MFT import ( 19 | ATTR_TYPE, 20 | MREF, 21 | MSEQNO, 22 | Attribute, 23 | Cache, 24 | FilenameAttribute, 25 | IndexRootHeader, 26 | MFTEnumerator, 27 | MFTRecord, 28 | StandardInformationFieldDoesNotExist, 29 | ) 30 | 31 | ASCII_BYTE: bytes = printable.encode("ascii") 32 | 33 | 34 | def ascii_strings(buf, n=4): 35 | reg = b"([%s]{%d,})" % (ASCII_BYTE, n) 36 | ascii_re = re.compile(reg) 37 | for match in ascii_re.finditer(buf): 38 | if isinstance(match.group(), array.array): 39 | yield match.group().tostring().decode("ascii") 40 | else: 41 | yield match.group().decode("ascii") 42 | 43 | 44 | def unicode_strings(buf, n=4): 45 | reg = b"((?:[%s]\x00){4,})" % (ASCII_BYTE) 46 | ascii_re = re.compile(reg) 47 | for match in ascii_re.finditer(buf): 48 | try: 49 | if isinstance(match.group(), array.array): 50 | yield match.group().tostring().decode("utf-16") 51 | else: 52 | yield match.group().decode("utf-16") 53 | except UnicodeDecodeError: 54 | pass 55 | 56 | 57 | def get_flags(flags): 58 | """ 59 | Get readable list of attribute flags. 60 | """ 61 | attributes = [] 62 | for flag in list(Attribute.FLAGS.keys()): 63 | if flags & flag: 64 | attributes.append(Attribute.FLAGS[flag]) 65 | return attributes 66 | 67 | 68 | def create_safe_datetime(fn): 69 | try: 70 | return fn() 71 | except ValueError: 72 | return datetime.datetime(1970, 1, 1, 0, 0, 0) 73 | 74 | 75 | def create_safe_timeline_entry(fn, type_, source, path): 76 | return { 77 | "timestamp": create_safe_datetime(fn), 78 | "type": type_, 79 | "source": source, 80 | "path": path, 81 | } 82 | 83 | 84 | def create_safe_timeline_entries(attr, source, path): 85 | return [ 86 | create_safe_timeline_entry(attr.created_time, "birthed", source, path), 87 | create_safe_timeline_entry(attr.accessed_time, "accessed", source, path), 88 | create_safe_timeline_entry(attr.modified_time, "modified", source, path), 89 | create_safe_timeline_entry(attr.changed_time, "changed", source, path), 90 | ] 91 | 92 | 93 | def get_timeline_entries(record): 94 | entries = [] 95 | si = record.standard_information() 96 | fn = record.filename_information() 97 | 98 | if si and fn: 99 | filename = fn.filename() 100 | entries.extend(create_safe_timeline_entries(si, "$SI", filename)) 101 | 102 | for b in record.attributes(): 103 | if b.type() != ATTR_TYPE.FILENAME_INFORMATION: 104 | continue 105 | attr = FilenameAttribute(b.value(), 0, record) 106 | attr_filename = attr.filename() 107 | entries.extend(create_safe_timeline_entries(attr, "$FN", attr_filename)) 108 | 109 | indxroot = record.attribute(ATTR_TYPE.INDEX_ROOT) 110 | if indxroot and indxroot.non_resident() == 0: 111 | irh = IndexRootHeader(indxroot.value(), 0, False) 112 | for e in irh.node_header().entries(): 113 | fn = e.filename_information() 114 | fn_filename = fn.filename() 115 | entries.extend(create_safe_timeline_entries(fn, "INDX", fn_filename)) 116 | 117 | for e in irh.node_header().slack_entries(): 118 | fn = e.filename_information() 119 | fn_filename = fn.filename() 120 | entries.extend(create_safe_timeline_entries(fn, "slack-INDX", fn_filename)) 121 | 122 | return sorted( 123 | entries, key=lambda x: x["timestamp"] or datetime.datetime(1970, 1, 1, 0, 0, 0) 124 | ) 125 | 126 | 127 | def make_filename_information_model(attr): 128 | if attr is None: 129 | return None 130 | 131 | return { 132 | "type": ["POSIX", "WIN32", "DOS 8.3", "WIN32 + DOS 8.3"][attr.filename_type()], 133 | "name": attr.filename(), 134 | "flags": get_flags(attr.flags()), 135 | "logical_size": attr.logical_size(), 136 | "physical_size": attr.physical_size(), 137 | "modified": create_safe_datetime(attr.modified_time), 138 | "accessed": create_safe_datetime(attr.accessed_time), 139 | "changed": create_safe_datetime(attr.changed_time), 140 | "created": create_safe_datetime(attr.created_time), 141 | "parent_ref": MREF(attr.mft_parent_reference()), 142 | "parent_seq": MSEQNO(attr.mft_parent_reference()), 143 | } 144 | 145 | 146 | def make_standard_information_model(attr): 147 | if attr is None: 148 | return None 149 | # if attr is None: 150 | # default_time = datetime.datetime(1970, 1, 1, 0, 0, 0) 151 | # return { 152 | # "created": default_time, 153 | # "modified": default_time, 154 | # "changed": default_time, 155 | # "accessed": default_time, 156 | # "owner_id": 0, 157 | # "security_id": "", 158 | # "quota_charged": 0, 159 | # "usn": 0 160 | # } 161 | ret = { 162 | "created": create_safe_datetime(attr.created_time), 163 | "modified": create_safe_datetime(attr.modified_time), 164 | "changed": create_safe_datetime(attr.changed_time), 165 | "accessed": create_safe_datetime(attr.accessed_time), 166 | "flags": get_flags(attr.attributes()), 167 | } 168 | 169 | # since the fields are sequential, we can handle an exception half way through here 170 | # and then ignore the remaining items. Dont have to worry about individual try/catches 171 | try: 172 | ret["owner_id"] = attr.owner_id() 173 | ret["security_id"] = attr.security_id() 174 | ret["quota_charged"] = attr.quota_charged() 175 | ret["usn"] = attr.usn() 176 | except StandardInformationFieldDoesNotExist: 177 | pass 178 | 179 | return ret 180 | 181 | 182 | def make_attribute_model(attr): 183 | ret = { 184 | "type": Attribute.TYPES[attr.type()], 185 | "name": attr.name(), 186 | "flags": get_flags(attr.flags()), 187 | "is_resident": attr.non_resident() == 0, 188 | "data_size": 0, 189 | "allocated_size": 0, 190 | "value_size": 0, 191 | "runs": [], 192 | } 193 | 194 | if attr.non_resident() > 0: 195 | ret["data_size"] = attr.data_size() 196 | ret["allocated_size"] = attr.allocated_size() 197 | 198 | if attr.allocated_size() > 0: 199 | for offset, length in attr.runlist().runs(): 200 | ret["runs"].append( 201 | { 202 | "offset": offset, 203 | "length": length, 204 | } 205 | ) 206 | else: 207 | ret["value_size"] = attr.value_length() 208 | return ret 209 | 210 | 211 | def make_model(record: MFTRecord, path: str) -> Dict[str, Any]: 212 | active_data = record.active_data() 213 | slack_data = record.slack_data() 214 | model = { 215 | "magic": record.magic(), 216 | "path": path, 217 | "inode": record.inode, 218 | "is_active": record.is_active(), 219 | "is_directory": record.is_directory(), 220 | "size": 0, # updated below 221 | "standard_information": make_standard_information_model( 222 | record.standard_information() 223 | ), 224 | "filename_information": make_filename_information_model( 225 | record.filename_information() 226 | ), 227 | "owner_id": 0, # updated below 228 | "security_id": 0, # updated below 229 | "quota_charged": 0, # updated below 230 | "usn": 0, # updated below 231 | "filenames": [], 232 | "attributes": [], 233 | "indx_entries": [], 234 | "slack_indx_entries": [], 235 | "timeline": get_timeline_entries(record), 236 | "active_ascii_strings": ascii_strings(active_data), 237 | "active_unicode_strings": unicode_strings(active_data), 238 | "slack_ascii_strings": ascii_strings(slack_data), 239 | "slack_unicode_strings": unicode_strings(slack_data), 240 | } 241 | 242 | if not record.is_directory(): 243 | data_attr = record.data_attribute() 244 | if data_attr and data_attr.non_resident() > 0: 245 | model["size"] = data_attr.data_size() 246 | else: 247 | filename_attr = record.filename_information() 248 | if filename_attr is not None: 249 | model["size"] = filename_attr.logical_size() 250 | else: 251 | model["size"] = 0 252 | 253 | for b in record.attributes(): 254 | if b.type() != ATTR_TYPE.FILENAME_INFORMATION: 255 | continue 256 | attr = FilenameAttribute(b.value(), 0, record) 257 | model["filenames"].append(make_filename_information_model(attr)) 258 | 259 | for b in record.attributes(): 260 | model["attributes"].append(make_attribute_model(b)) 261 | 262 | indxroot = record.attribute(ATTR_TYPE.INDEX_ROOT) 263 | if indxroot and indxroot.non_resident() == 0: 264 | irh = IndexRootHeader(indxroot.value(), 0, False) 265 | for e in irh.node_header().entries(): 266 | m = make_filename_information_model(e.filename_information()) 267 | m["inode"] = MREF(e.mft_reference()) 268 | m["sequence_num"] = MSEQNO(e.mft_reference()) 269 | model["indx_entries"].append(m) 270 | 271 | for e in irh.node_header().slack_entries(): 272 | m = make_filename_information_model(e.filename_information()) 273 | m["inode"] = MREF(e.mft_reference()) 274 | m["sequence_num"] = MSEQNO(e.mft_reference()) 275 | model["slack_indx_entries"].append(m) 276 | return model 277 | 278 | 279 | def format_record(record: MFTRecord, path: str) -> str: 280 | template = Template( 281 | """\ 282 | MFT Record: {{ record.inode }} 283 | Path: {{ record.path }} 284 | Metadata: 285 | Active: {{ record.is_active }} 286 | {% if record.is_directory %}\ 287 | Type: directory\ 288 | {% else %}\ 289 | Type: file\ 290 | {% endif %} 291 | Flags: {{ record.standard_information.flags|join(', ') }} 292 | $SI Modified: {{ record.standard_information.modified }} 293 | $SI Accessed: {{ record.standard_information.accessed }} 294 | $SI Changed: {{ record.standard_information.changed }} 295 | $SI Birthed: {{ record.standard_information.created }} 296 | Owner ID: {{ record.standard_information.owner_id }} 297 | Security ID: {{ record.standard_information.security_id }} 298 | Quota charged: {{ record.standard_information.quota_charged }} 299 | USN: {{ record.standard_information.usn }} 300 | Filenames: \ 301 | {% for filename in record.filenames %} 302 | Type: {{ filename.type }} 303 | Name: {{ filename.name }} 304 | Flags: {{ filename.flags|join(', ') }} 305 | Logical size: {{ filename.logical_size }} 306 | Physical size: {{ filename.physical_size }} 307 | Modified: {{ filename.modified }} 308 | Accessed: {{ filename.accessed }} 309 | Changed: {{ filename.changed }} 310 | Birthed: {{ filename.created }} 311 | Parent reference: {{ filename.parent_ref }} 312 | Parent sequence number: {{ filename.parent_seq }}\ 313 | {% endfor %} 314 | Attributes: \ 315 | {% for attribute in record.attributes %} 316 | Type: {{ attribute.type }} 317 | Name: {{ attribute.name }} 318 | Flags: {{ attribute.flags|join(', ') }} 319 | Resident: {{ attribute.is_resident }} 320 | Data size: {{ attribute.data_size }} 321 | Allocated size: {{ attribute.allocated_size }} 322 | Value size: {{ attribute.value_size }} \ 323 | {% if attribute.runs %} 324 | Data runs: {% for run in attribute.runs %} 325 | Offset (clusters): {{ run.offset }} Length (clusters): {{ run.length }} \ 326 | {% endfor %}\ 327 | {% endif %}\ 328 | {% endfor %} 329 | INDX root entries:\ 330 | {% if not record.indx_entries %}\ 331 | \ 332 | {% endif %}\ 333 | {% for indx in record.indx_entries %} 334 | Name: {{ indx.filename }} 335 | Size: {{ indx.size }} 336 | Modified: {{ indx.modified }} 337 | Accessed: {{ indx.accessed }} 338 | Changed: {{ indx.changed }} 339 | Birthed: {{ indx.created }} 340 | Reference: {{ indx.inode }} 341 | Sequence number: {{ indx.sequence_num }}\ 342 | {% endfor %} 343 | INDX root slack entries:\ 344 | {% if not record.slack_indx_entries %}\ 345 | \ 346 | {% endif %}\ 347 | {% for indx in record.slack_indx_entries %} 348 | Name: {{ indx.filename }} 349 | Size: {{ indx.size }} 350 | Modified: {{ indx.modified }} 351 | Accessed: {{ indx.accessed }} 352 | Changed: {{ indx.changed }} 353 | Birthed: {{ indx.created }} 354 | Reference: {{ indx.inode }} 355 | Sequence number: {{ indx.sequence_num }}\ 356 | {% endfor %} 357 | Timeline: 358 | {% for entry in record.timeline %}\ 359 | {{ "%-30s%-12s%-8s%s"|format(entry.timestamp, entry.type, entry.source, entry.path) }} 360 | {% endfor %}\ 361 | Active strings: 362 | ASCII strings: 363 | {% for string in record.active_ascii_strings %}\ 364 | {{ string }} 365 | {% endfor %}\ 366 | Unicode strings: 367 | {% for string in record.active_unicode_strings %}\ 368 | {{ string }} 369 | {% endfor %}\ 370 | Slack strings: 371 | ASCII strings: 372 | {% for string in record.slack_ascii_strings %}\ 373 | {{ string }} 374 | {% endfor %}\ 375 | Unicode strings: 376 | {% for string in record.slack_unicode_strings %}\ 377 | {{ string }} 378 | {% endfor %}\ 379 | """ 380 | ) 381 | return template.render(record=make_model(record, path)) 382 | 383 | 384 | def print_indx_info(record, path): 385 | print(format_record(record, path)) 386 | 387 | 388 | def main(): 389 | parser = argparse.ArgumentParser(description="Inspect " "a given MFT file record.") 390 | parser.add_argument( 391 | "-a", 392 | action="store", 393 | metavar="cache_size", 394 | type=int, 395 | dest="cache_size", 396 | default=1024, 397 | help="Size of cache.", 398 | ) 399 | parser.add_argument( 400 | "-p", 401 | action="store", 402 | metavar="prefix", 403 | nargs=1, 404 | dest="prefix", 405 | default="\\.", 406 | help="Prefix paths with `prefix` rather than \\.\\", 407 | ) 408 | parser.add_argument( 409 | "-v", action="store_true", dest="verbose", help="Print debugging information" 410 | ) 411 | parser.add_argument("mft", action="store", help="Path to MFT") 412 | parser.add_argument( 413 | "record_or_path", action="store", help="MFT record or file path to inspect" 414 | ) 415 | 416 | results = parser.parse_args() 417 | 418 | if results.verbose: 419 | logging.basicConfig(level=logging.DEBUG) 420 | 421 | with open(results.mft, "rb") as fh: 422 | with mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ) as mm: 423 | record_cache = Cache(results.cache_size) 424 | path_cache = Cache(results.cache_size) 425 | 426 | enum = MFTEnumerator(mm, record_cache=record_cache, path_cache=path_cache) 427 | 428 | should_use_inode = False 429 | try: 430 | record_num = int(results.record_or_path) 431 | should_use_inode = True 432 | except ValueError: 433 | should_use_inode = False 434 | 435 | if should_use_inode: 436 | record = enum.get_record(record_num) 437 | path = results.prefix + enum.get_path(record) 438 | print_indx_info(record, path) 439 | else: 440 | path = results.record_or_path 441 | record = enum.get_record_by_path(path) 442 | print_indx_info(record, results.prefix + path) 443 | 444 | 445 | if __name__ == "__main__": 446 | main() 447 | -------------------------------------------------------------------------------- /indxparse/list_mft.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | 3 | # This file is part of INDXParse. 4 | # 5 | # Copyright 2014 Willi Ballenthin 6 | # while at FireEye 7 | # 8 | # Licensed under the Apache License, Version 2.0 (the "License"); 9 | # you may not use this file except in compliance with the License. 10 | # You may obtain a copy of the License at 11 | # 12 | # http://www.apache.org/licenses/LICENSE-2.0 13 | # 14 | # Unless required by applicable law or agreed to in writing, software 15 | # distributed under the License is distributed on an "AS IS" BASIS, 16 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 17 | # See the License for the specific language governing permissions and 18 | # limitations under the License. 19 | # 20 | # 21 | # Alex Nelson, NIST, contributed to this file. Contributions of NIST 22 | # are not subject to US Copyright. 23 | # 24 | # 25 | # This is a straightforward script; the complexity comes about in supporting 26 | # a variety of output formats. The default output (Bodyfile), is by far 27 | # the fastest (by a factor of about two). The user defined formats are 28 | # implemented with Jinja2 templating. This makes our job fairly easy. 29 | # The final option is JSON output. 30 | # 31 | # TODO: 32 | # - use logging instead of #-prefixed comment lines 33 | # - display inactive record tags 34 | 35 | import argparse 36 | import calendar 37 | import datetime 38 | import json 39 | import logging 40 | import mmap 41 | import sys 42 | import types 43 | from typing import Any, Dict, List, Optional, Type, Union 44 | 45 | from jinja2 import Environment, Template 46 | 47 | from indxparse.get_file_info import make_model 48 | from indxparse.MFT import ( 49 | ATTR_TYPE, 50 | MREF, 51 | Cache, 52 | FilenameAttribute, 53 | IndexRootHeader, 54 | MFTEnumerator, 55 | MFTRecord, 56 | StandardInformation, 57 | StandardInformationFieldDoesNotExist, 58 | ) 59 | from indxparse.Progress import NullProgress, Progress, ProgressBarProgress 60 | 61 | 62 | def format_bodyfile( 63 | path: str, 64 | size: int, 65 | inode: int, 66 | owner_id: int, 67 | info: Union[FilenameAttribute, StandardInformation, Dict[Any, Any]], 68 | attributes: List[str], 69 | ) -> str: 70 | """ 71 | Format a single line of Bodyfile output. 72 | Arguments: 73 | - `info`: This is optionally a dictionary to handle a special case 74 | for Alternate Data Streams. 75 | """ 76 | default_int = int( 77 | calendar.timegm(datetime.datetime(1970, 1, 1, 0, 0, 0).timetuple()) 78 | ) 79 | if not attributes: 80 | attributes = [] 81 | if isinstance(info, dict): 82 | modified = default_int 83 | accessed = default_int 84 | changed = default_int 85 | created = default_int 86 | else: 87 | try: 88 | modified = int(calendar.timegm(info.modified_time().timetuple())) 89 | except (ValueError, AttributeError): 90 | modified = default_int 91 | try: 92 | accessed = int(calendar.timegm(info.accessed_time().timetuple())) 93 | except (ValueError, AttributeError): 94 | accessed = default_int 95 | try: 96 | changed = int(calendar.timegm(info.changed_time().timetuple())) 97 | except (ValueError, AttributeError): 98 | changed = default_int 99 | try: 100 | created = int(calendar.timegm(info.created_time().timetuple())) 101 | except (ValueError, AttributeError): 102 | created = int(calendar.timegm(datetime.datetime.min.timetuple())) 103 | attributes_text = "" 104 | if len(attributes) > 0: 105 | attributes_text = " (%s)" % (", ".join(attributes)) 106 | return "0|%s|%s|0|%d|0|%s|%s|%s|%s|%s" % ( 107 | path + attributes_text, 108 | inode, 109 | owner_id, 110 | size, 111 | accessed, 112 | modified, 113 | changed, 114 | created, 115 | ) 116 | 117 | 118 | def output_mft_record( 119 | mft_enumerator: MFTEnumerator, record: MFTRecord, prefix: str 120 | ) -> None: 121 | """ 122 | Print to STDOUT all the Bodyfile formatted lines 123 | associated with a single record. This includes 124 | a line for standard information, filename information, 125 | and any resident directory index entries. 126 | """ 127 | tags = [] 128 | if not record.is_active(): 129 | tags.append("inactive") 130 | 131 | path = prefix + "\\" + mft_enumerator.get_path(record) 132 | si = record.standard_information() 133 | fn = record.filename_information() 134 | 135 | if not record.is_active() and not fn: 136 | return 137 | 138 | inode = record.mft_record_number() 139 | if record.is_directory(): 140 | size = 0 141 | else: 142 | data_attr = record.data_attribute() 143 | if data_attr and data_attr.non_resident() > 0: 144 | size = data_attr.data_size() 145 | elif fn: 146 | size = fn.logical_size() 147 | else: 148 | size = 0 149 | 150 | ADSs = [] # list of (name, size) 151 | for attr in record.attributes(): 152 | if attr.type() != ATTR_TYPE.DATA or len(attr.name()) == 0: 153 | continue 154 | if attr.non_resident() > 0: 155 | size = attr.data_size() 156 | else: 157 | size = attr.value_length() 158 | ADSs.append((attr.name(), size)) 159 | 160 | si_index = 0 161 | if si: 162 | try: 163 | si_index = si.security_id() 164 | except StandardInformationFieldDoesNotExist: 165 | pass 166 | 167 | indices = [] # list of (filename, size, reference, info) 168 | slack_indices = [] # list of (filename, size, reference, info) 169 | indxroot = record.attribute(ATTR_TYPE.INDEX_ROOT) 170 | if indxroot and indxroot.non_resident() == 0: 171 | # TODO(wb): don't use IndxRootHeader 172 | irh = IndexRootHeader(indxroot.value(), 0, False) 173 | for e in irh.node_header().entries(): 174 | indices.append( 175 | ( 176 | e.filename_information().filename(), 177 | e.mft_reference(), 178 | e.filename_information().logical_size(), 179 | e.filename_information(), 180 | ) 181 | ) 182 | 183 | for e in irh.node_header().slack_entries(): 184 | slack_indices.append( 185 | ( 186 | e.filename_information().filename(), 187 | e.mft_reference(), 188 | e.filename_information().logical_size(), 189 | e.filename_information(), 190 | ) 191 | ) 192 | 193 | # si 194 | if si: 195 | try: 196 | print(format_bodyfile(path, size, inode, si_index, si, tags)) 197 | except UnicodeEncodeError: 198 | print("# failed to print: %s" % (list(path))) 199 | 200 | # fn 201 | if fn: 202 | tags = ["filename"] 203 | if not record.is_active(): 204 | tags.append("inactive") 205 | try: 206 | print(format_bodyfile(path, size, inode, si_index, fn, tags)) 207 | except UnicodeEncodeError: 208 | print("# failed to print: %s" % (list(path))) 209 | 210 | # ADS 211 | for ads in ADSs: 212 | tags = [] 213 | if not record.is_active(): 214 | tags.append("inactive") 215 | try: 216 | print( 217 | format_bodyfile( 218 | path + ":" + ads[0], ads[1], inode, si_index, si or {}, tags 219 | ), 220 | ) 221 | except UnicodeEncodeError: 222 | print("# failed to print: %s" % (list(path))) 223 | 224 | # INDX 225 | for indx in indices: 226 | tags = ["indx"] 227 | try: 228 | print( 229 | format_bodyfile( 230 | path + "\\" + indx[0], indx[1], MREF(indx[2]), 0, indx[3], tags 231 | ), 232 | ) 233 | except UnicodeEncodeError: 234 | print("# failed to print: %s" % (list(path))) 235 | 236 | for indx in slack_indices: 237 | tags = ["indx", "slack"] 238 | try: 239 | print( 240 | format_bodyfile( 241 | path + "\\" + indx[0], indx[1], MREF(indx[2]), 0, indx[3], tags 242 | ), 243 | ) 244 | except UnicodeEncodeError: 245 | print("# failed to print: %s" % (list(path))) 246 | 247 | 248 | def unixtimestampformat(value: Optional[datetime.datetime]) -> int: 249 | """ 250 | A custom Jinja2 filter for converting a datetime.datetime 251 | a UNIX timestamp integer. 252 | """ 253 | if value is None: 254 | return 0 255 | return int(calendar.timegm(value.timetuple())) 256 | 257 | 258 | def get_default_template(env: Environment) -> Template: 259 | """ 260 | Return a Jinja2 Template instance that formats an 261 | MFT record into bodyfile format. 262 | Slower than the format_bodyfile() function above, so 263 | this format is provided here for reference. 264 | """ 265 | return env.from_string( 266 | """\ 267 | {% if record.standard_information and record.filename_information %} 268 | 0|{{ prefix }}{{ record.path }}|{{ record.inode }}|0|{{ record.standard_information.owner_id }}|0|{{ record.size }}|{{ record.standard_information.accessed|unixtimestampformat }}|{{ record.standard_information.modified|unixtimestampformat }}|{{ record.standard_information.changed|unixtimestampformat }}|{{ record.standard_information.created|unixtimestampformat }} 269 | {% endif %} 270 | {% if record.standard_information and record.filename_information %} 271 | 0|{{ prefix }}{{ record.path }} (filename)|{{ record.inode }}|0|{{ record.standard_information.owner_id }}|0|{{ record.size }}|{{ record.filename_information.accessed|unixtimestampformat }}|{{ record.filename_information.modified|unixtimestampformat }}|{{ record.filename_information.changed|unixtimestampformat }}|{{ record.filename_information.created|unixtimestampformat }} 272 | {% endif %} 273 | {% for e in record.indx_entries %} 274 | 0|{{ prefix }}{{ record.path }}\\{{ e.name }} (INDX)|{{ e.inode }}|0|0|0|{{ e.logical_size }}|{{ e.accessed|unixtimestampformat }}|{{ e.modified|unixtimestampformat }}|{{ e.changed|unixtimestampformat }}|{{ e.created|unixtimestampformat }} 275 | {% endfor %} 276 | {% for e in record.slack_indx_entries %} 277 | 0|{{ prefix }}{{ record.path }}\\{{ e.name }} (slack-INDX)|{{ e.inode }}|0|0|0|{{ e.logical_size }}|{{ e.accessed|unixtimestampformat }}|{{ e.modified|unixtimestampformat }}|{{ e.changed|unixtimestampformat }}|{{ e.created|unixtimestampformat }} 278 | {% endfor %} 279 | """ 280 | ) 281 | 282 | 283 | def main() -> None: 284 | parser = argparse.ArgumentParser(description="Parse MFT " "filesystem structures.") 285 | parser.add_argument( 286 | "-c", 287 | action="store", 288 | metavar="cache_size", 289 | type=int, 290 | dest="cache_size", 291 | default=1024, 292 | help="Size of cache.", 293 | ) 294 | parser.add_argument( 295 | "-p", 296 | action="store", 297 | metavar="prefix", 298 | nargs=1, 299 | dest="prefix", 300 | default="\\.", 301 | help="Prefix paths with `prefix` rather than \\.\\", 302 | ) 303 | parser.add_argument( 304 | "-v", action="store_true", dest="verbose", help="Print debugging information" 305 | ) 306 | parser.add_argument( 307 | "--progress", 308 | action="store_true", 309 | dest="progress", 310 | help="Update a status indicator on STDERR " "if STDOUT is redirected", 311 | ) 312 | parser.add_argument( 313 | "--format", 314 | action="store", 315 | metavar="format", 316 | nargs=1, 317 | dest="format", 318 | help="Output format specification", 319 | ) 320 | parser.add_argument( 321 | "--format_file", 322 | action="store", 323 | metavar="format_file", 324 | nargs=1, 325 | dest="format_file", 326 | help="File containing output format specification", 327 | ) 328 | parser.add_argument( 329 | "--json", action="store_true", dest="json", help="Output in JSON format" 330 | ) 331 | parser.add_argument( 332 | "-f", 333 | action="store", 334 | metavar="regex", 335 | nargs=1, 336 | dest="filter", 337 | help="Only consider entries whose path " "matches this regular expression", 338 | ) 339 | parser.add_argument("filename", action="store", help="Input MFT file path") 340 | results = parser.parse_args() 341 | use_default_output = True 342 | 343 | if results.verbose: 344 | logging.basicConfig(level=logging.DEBUG) 345 | 346 | env = Environment(trim_blocks=True, lstrip_blocks=True) 347 | env.filters["unixtimestampformat"] = unixtimestampformat 348 | 349 | flags_count = 0 350 | if results.format: 351 | flags_count += 1 352 | template = env.from_string(results.format[0]) 353 | if results.format_file: 354 | flags_count += 1 355 | with open(results.format_file[0], "r", encoding="utf-8") as f: 356 | template = env.from_string(f.read()) 357 | if results.json: 358 | flags_count += 1 359 | pass 360 | 361 | if flags_count > 1: 362 | sys.stderr.write( 363 | "Only one of --format, --format_file, --json may be provided.\n" 364 | ) 365 | sys.exit(-1) 366 | elif flags_count == 1: 367 | use_default_output = False 368 | elif flags_count == 0: 369 | flags_count += 1 370 | template = get_default_template(env) 371 | use_default_output = True 372 | 373 | # Syntax note for variable holding a class reference: 374 | # https://docs.python.org/3/library/typing.html#the-type-of-class-objects 375 | progress_cls: Type[Progress] 376 | if results.progress: 377 | progress_cls = ProgressBarProgress 378 | else: 379 | progress_cls = NullProgress 380 | 381 | with open(results.filename, "rb") as fh: 382 | with mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ) as mm: 383 | record_cache = Cache(results.cache_size) 384 | path_cache = Cache(results.cache_size) 385 | 386 | enum = MFTEnumerator(mm, record_cache=record_cache, path_cache=path_cache) 387 | progress = progress_cls(enum.len()) 388 | if use_default_output: 389 | for record, record_path in enum.enumerate_paths(): 390 | output_mft_record(enum, record, results.prefix[0]) 391 | progress.set_current(record.inode) 392 | elif results.json: 393 | 394 | class MFTEncoder(json.JSONEncoder): 395 | def default(self, obj: Any) -> Any: 396 | if isinstance(obj, datetime.datetime): 397 | return obj.isoformat("T") + "Z" 398 | elif isinstance(obj, types.GeneratorType): 399 | return [o for o in obj] 400 | return json.JSONEncoder.default(self, obj) 401 | 402 | print("[") 403 | record_count = 0 404 | for record, record_path in enum.enumerate_paths(): 405 | if record_count > 0: 406 | print(",") 407 | record_count += 1 408 | m = make_model(record, record_path) 409 | print(json.dumps(m, cls=MFTEncoder, indent=2)) 410 | progress.set_current(record.inode) 411 | print("]") 412 | else: 413 | for record, record_path in enum.enumerate_paths(): 414 | sys.stdout.write( 415 | template.render( 416 | record=make_model(record, record_path), 417 | prefix=results.prefix[0], 418 | ) 419 | + "\n" 420 | ) 421 | progress.set_current(record.inode) 422 | progress.set_complete() 423 | 424 | 425 | if __name__ == "__main__": 426 | main() 427 | -------------------------------------------------------------------------------- /indxparse/py.typed: -------------------------------------------------------------------------------- 1 | # Alex Nelson, NIST, contributed to this file. Contributions of NIST 2 | # are not subject to US Copyright. 3 | 4 | # This file is defined to support PEP 561: 5 | # https://www.python.org/dev/peps/pep-0561/ 6 | -------------------------------------------------------------------------------- /indxparse/tree_mft.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | 3 | # Alex Nelson, NIST, contributed to this file. Contributions of NIST 4 | # are not subject to US Copyright. 5 | 6 | import argparse 7 | import logging 8 | import mmap 9 | 10 | from indxparse.MFT import Cache, MFTTree, MFTTreeNode 11 | 12 | 13 | def main() -> None: 14 | parser = argparse.ArgumentParser(description="Parse MFT " "filesystem structures.") 15 | parser.add_argument( 16 | "-c", 17 | action="store", 18 | metavar="cache_size", 19 | type=int, 20 | dest="cache_size", 21 | default=1024, 22 | help="Size of cache.", 23 | ) 24 | parser.add_argument( 25 | "-v", action="store_true", dest="verbose", help="Print debugging information" 26 | ) 27 | parser.add_argument("filename", action="store", help="Input MFT file path") 28 | 29 | results = parser.parse_args() 30 | 31 | if results.verbose: 32 | logging.basicConfig(level=logging.DEBUG) 33 | 34 | with open(results.filename, "rb") as fh: 35 | with mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ) as mm: 36 | record_cache = Cache(results.cache_size) 37 | path_cache = Cache(results.cache_size) 38 | 39 | tree = MFTTree(mm) 40 | tree.build(record_cache=record_cache, path_cache=path_cache) 41 | 42 | def rec(node: MFTTreeNode, prefix: str) -> None: 43 | print(prefix + node.get_filename()) 44 | for child in node.get_children_nodes(): 45 | rec(child, prefix + " ") 46 | 47 | rec(tree.get_root(), "") 48 | 49 | 50 | if __name__ == "__main__": 51 | main() 52 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [metadata] 2 | name = indxparse 3 | version = attr: indxparse.__version__ 4 | author = Willi Ballenthin 5 | url = https://github.com/williballenthin/INDXParse 6 | classifiers = 7 | License :: OSI Approved :: Apache Software License 8 | Programming Language :: Python :: 3 9 | license_files = 10 | LICENSE 11 | THIRD_PARTY_LICENSES.md 12 | 13 | [options] 14 | install_requires = 15 | jinja2 16 | packages = find: 17 | python_requires = >=3.8 18 | 19 | [options.entry_points] 20 | console_scripts = 21 | INDXParse.py = indxparse.INDXParse:main 22 | MFTINDX.py = indxparse.MFTINDX:main 23 | SDS_get_index.py = indxparse.SDS_get_index:main 24 | extract_mft_record_slack.py = indxparse.extract_mft_record_slack:main 25 | get_file_info.py = indxparse.get_file_info:main 26 | list_mft.py = indxparse.list_mft:main 27 | tree_mft.py = indxparse.tree_mft:main 28 | # NOTE: fuse-mft.py will still be exposed as a script if the fuse 29 | # feature is not installed, but it will not run successfully unless 30 | # the dependencies of that feature are installed. This is a known 31 | # non-obvious dependency specification issue: 32 | # https://github.com/pypa/pip/issues/9726 33 | fuse-mft.py = indxparse.fuse_mft:main [fuse] 34 | # NOTE: As with fuse-mft.py, MFTView.py requires the wx feature. 35 | MFTView.py = indxparse.MFTView:main [wx] 36 | 37 | [options.extras_require] 38 | fuse = 39 | fuse-python 40 | testing = 41 | mypy 42 | wx = 43 | wxPython 44 | 45 | [options.package_data] 46 | indxparse = py.typed 47 | 48 | [flake8] 49 | # https://black.readthedocs.io/en/stable/guides/using_black_with_other_tools.html#flake8 50 | extend-ignore = 51 | E203 52 | E501 53 | 54 | [isort] 55 | # https://pycqa.github.io/isort/docs/configuration/black_compatibility.html 56 | profile = black 57 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | # Alex Nelson, NIST, contributed to this file. Contributions of NIST 2 | # are not subject to US Copyright. 3 | 4 | import setuptools 5 | 6 | if __name__ == "__main__": 7 | setuptools.setup() 8 | -------------------------------------------------------------------------------- /tests/MFTINDX/7-ntfs-undel.dd.d.txt: -------------------------------------------------------------------------------- 1 | 0|\.\frag1.dat (deleted)|2255|0|260|0|1584|1078084840|1078084840|1078084840|1078084817 2 | 0|\.\frag1.dat (deleted, filename)|2255|0|260|0|1584|1078084817|1078084817|1078084817|1078084817 3 | 0|\.\frag2.dat (deleted)|2256|0|260|0|3873|1078084974|1078084974|1078084974|1078084829 4 | 0|\.\frag2.dat (deleted, filename)|2256|0|260|0|3873|1078084829|1078084829|1078084829|1078084829 5 | 0|\.\sing1.dat (deleted)|2257|0|260|0|780|1078084884|1078084884|1078084884|1078084884 6 | 0|\.\sing1.dat (deleted, filename)|2257|0|260|0|780|1078084884|1078084884|1078084884|1078084884 7 | 0|\.\mult1.dat (deleted)|2258|0|260|0|1234|1078084942|1078084942|1078084942|1078084897 8 | 0|\.\mult1.dat:ADS (deleted)|2258|0|260|0|1234|1078084942|1078084942|1078084942|1078084897 9 | 0|\.\mult1.dat (deleted, filename)|2258|0|260|0|1234|1078084897|1078084897|1078084897|1078084897 10 | 0|\.\mult1.dat:ADS (deleted, filename)|2258|0|260|0|1234|1078084897|1078084897|1078084897|1078084897 11 | 0|\.\dir1 (deleted)|2259|0|261|0|0|1078084980|1078084980|1078084980|1078084980 12 | 0|\.\dir1 (deleted, filename)|2259|0|261|0|0|1078084980|1078084980|1078084980|1078084980 13 | 0|\$OrphanFiles\dir2 (deleted)|2260|0|261|0|0|1078084983|1078084983|1078084983|1078084983 14 | 0|\$OrphanFiles\dir2 (deleted, filename)|2260|0|261|0|0|1078084983|1078084983|1078084983|1078084983 15 | 0|\$OrphanFiles\frag3.dat (deleted)|2261|0|260|0|2027|1078085029|1078085029|1078085029|1078085004 16 | 0|\$OrphanFiles\frag3.dat (deleted, filename)|2261|0|260|0|2027|1078085004|1078085004|1078085004|1078085004 17 | 0|\$OrphanFiles\mult2.dat (deleted)|2262|0|260|0|1715|1078085018|1078085018|1078085018|1078085018 18 | 0|\$OrphanFiles\mult2.dat (deleted, filename)|2262|0|260|0|1715|1078085018|1078085018|1078085018|1078085018 19 | 0|\.\res1.dat (deleted)|2263|0|260|0|0|1078085137|1078085137|1078085137|1078085137 20 | 0|\.\res1.dat (deleted, filename)|2263|0|260|0|0|1078085137|1078085137|1078085137|1078085137 21 | 0|\$OrphanFiles\sing2.dat (deleted)|2264|0|260|0|1005|1078085055|1078085055|1078085055|1078085055 22 | 0|\$OrphanFiles\sing2.dat (deleted, filename)|2264|0|260|0|1005|1078085055|1078085055|1078085055|1078085055 23 | -------------------------------------------------------------------------------- /tests/MFTINDX/7-ntfs-undel.dd.l.txt: -------------------------------------------------------------------------------- 1 | 0|\.\$Extend\$ObjId (filename, INDX)|0|0|0|0|0|1078084684|1078084684|1078084684|1078084684 2 | 0|\.\$Extend\$Quota (filename, INDX)|0|0|0|0|0|1078084684|1078084684|1078084684|1078084684 3 | 0|\.\$Extend\$Reparse (filename, INDX)|0|0|0|0|0|1078084684|1078084684|1078084684|1078084684 4 | 0|\.\$Extend\$ObjId\ (filename, INDX)|0|0|0|0|0|0|-11560031107|0|0 5 | 0|\.\System Volume Information\tracking.log (filename, INDX)|0|0|0|0|20480|1078085172|1078085172|1078085172|1078084750 6 | -------------------------------------------------------------------------------- /tests/MFTINDX/7-ntfs-undel.dd.m.txt: -------------------------------------------------------------------------------- 1 | 0|\.\$MFT|0|0|256|0|39936|1078084677|1078084677|1078084677|1078084677 2 | 0|\.\$MFT (filename)|0|0|256|0|39936|1078084677|1078084677|1078084677|1078084677 3 | 0|\.\$MFTMirr|1|0|256|0|4096|1078084677|1078084677|1078084677|1078084677 4 | 0|\.\$MFTMirr (filename)|1|0|256|0|4096|1078084677|1078084677|1078084677|1078084677 5 | 0|\.\$LogFile|2|0|256|0|2097152|1078084677|1078084677|1078084677|1078084677 6 | 0|\.\$LogFile (filename)|2|0|256|0|2097152|1078084677|1078084677|1078084677|1078084677 7 | 0|\.\$Volume|3|0|0|0|0|1078084677|1078084677|1078084677|1078084677 8 | 0|\.\$Volume (filename)|3|0|0|0|0|1078084677|1078084677|1078084677|1078084677 9 | 0|\.\$AttrDef|4|0|0|0|2560|1078084677|1078084677|1078084677|1078084677 10 | 0|\.\$AttrDef (filename)|4|0|0|0|2560|1078084677|1078084677|1078084677|1078084677 11 | 0|\.|5|0|0|0|0|1078085971|1078085971|1078085971|1078084677 12 | 0|\. (filename)|5|0|0|0|0|1078084677|1078084677|1078084677|1078084677 13 | 0|\.\$Bitmap|6|0|256|0|752|1078084677|1078084677|1078084677|1078084677 14 | 0|\.\$Bitmap (filename)|6|0|256|0|752|1078084677|1078084677|1078084677|1078084677 15 | 0|\.\$Boot|7|0|0|0|8192|1078084677|1078084677|1078084677|1078084677 16 | 0|\.\$Boot (filename)|7|0|0|0|8192|1078084677|1078084677|1078084677|1078084677 17 | 0|\.\$BadClus|8|0|256|0|6160384|1078084677|1078084677|1078084677|1078084677 18 | 0|\.\$BadClus:$Bad|8|0|256|0|6160384|1078084677|1078084677|1078084677|1078084677 19 | 0|\.\$BadClus (filename)|8|0|256|0|6160384|1078084677|1078084677|1078084677|1078084677 20 | 0|\.\$BadClus:$Bad (filename)|8|0|256|0|6160384|1078084677|1078084677|1078084677|1078084677 21 | 0|\.\$Secure|9|0|257|0|263140|1078084677|1078084677|1078084677|1078084677 22 | 0|\.\$Secure:$SDS|9|0|257|0|263140|1078084677|1078084677|1078084677|1078084677 23 | 0|\.\$Secure (filename)|9|0|257|0|263140|0|0|0|-62135596800 24 | 0|\.\$Secure:$SDS (filename)|9|0|257|0|263140|0|0|0|-62135596800 25 | 0|\.\$UpCase|10|0|256|0|131072|1078084677|1078084677|1078084677|1078084677 26 | 0|\.\$UpCase (filename)|10|0|256|0|131072|1078084677|1078084677|1078084677|1078084677 27 | 0|\.\$Extend|11|0|257|0|0|1078084677|1078084677|1078084677|1078084677 28 | 0|\.\$Extend (filename)|11|0|257|0|0|1078084677|1078084677|1078084677|1078084677 29 | 0|\.\mult1.dat|162|0|260|0|0|1078084897|1078084897|1078084897|1078084897 30 | 0|\.\mult1.dat (filename)|162|0|260|0|0|1078084897|1078084897|1078084897|1078084897 31 | 0|\.\$MFT|2064|0|256|0|39936|1078084677|1078084677|1078084677|1078084677 32 | 0|\.\$MFT (filename)|2064|0|256|0|39936|1078084677|1078084677|1078084677|1078084677 33 | 0|\.\$MFTMirr|2065|0|256|0|4096|1078084677|1078084677|1078084677|1078084677 34 | 0|\.\$MFTMirr (filename)|2065|0|256|0|4096|1078084677|1078084677|1078084677|1078084677 35 | 0|\.\$LogFile|2066|0|256|0|2097152|1078084677|1078084677|1078084677|1078084677 36 | 0|\.\$LogFile (filename)|2066|0|256|0|2097152|1078084677|1078084677|1078084677|1078084677 37 | 0|\.\$Volume|2067|0|0|0|0|1078084677|1078084677|1078084677|1078084677 38 | 0|\.\$Volume (filename)|2067|0|0|0|0|1078084677|1078084677|1078084677|1078084677 39 | 0|\.\$Extend\$Quota|2250|0|257|0|0|1078084684|1078084684|1078084684|1078084684 40 | 0|\.\$Extend\$Quota (filename)|2250|0|257|0|0|1078084684|1078084684|1078084684|1078084684 41 | 0|\.\$Extend\$ObjId|2251|0|257|0|0|1078084684|1078084684|1078084684|1078084684 42 | 0|\.\$Extend\$ObjId (filename)|2251|0|257|0|0|1078084684|1078084684|1078084684|1078084684 43 | 0|\.\$Extend\$Reparse|2252|0|257|0|0|1078084684|1078084684|1078084684|1078084684 44 | 0|\.\$Extend\$Reparse (filename)|2252|0|257|0|0|1078084684|1078084684|1078084684|1078084684 45 | 0|\.\System Volume Information|2253|0|258|0|0|1078085916|1078084751|1078084751|1078084750 46 | 0|\.\System Volume Information (filename)|2253|0|258|0|0|1078084750|1078084750|1078084750|1078084750 47 | 0|\$OrphanFiles\tracking.log|2254|0|259|0|20480|1078085172|1078085172|1078085172|1078084750 48 | 0|\$OrphanFiles\tracking.log (filename)|2254|0|259|0|20480|1078084751|1078084751|1078084751|1078084750 49 | -------------------------------------------------------------------------------- /tests/MFTINDX/7-ntfs-undel.dd.mft.d.txt: -------------------------------------------------------------------------------- 1 | 0|\.\frag1.dat (deleted)|29|0|260|0|1584|1078084840|1078084840|1078084840|1078084817 2 | 0|\.\frag1.dat (deleted, filename)|29|0|260|0|1584|1078084817|1078084817|1078084817|1078084817 3 | 0|\.\frag2.dat (deleted)|30|0|260|0|3873|1078084974|1078084974|1078084974|1078084829 4 | 0|\.\frag2.dat (deleted, filename)|30|0|260|0|3873|1078084829|1078084829|1078084829|1078084829 5 | 0|\.\sing1.dat (deleted)|31|0|260|0|780|1078084884|1078084884|1078084884|1078084884 6 | 0|\.\sing1.dat (deleted, filename)|31|0|260|0|780|1078084884|1078084884|1078084884|1078084884 7 | 0|\.\mult1.dat (deleted)|32|0|260|0|1234|1078084942|1078084942|1078084942|1078084897 8 | 0|\.\mult1.dat:ADS (deleted)|32|0|260|0|1234|1078084942|1078084942|1078084942|1078084897 9 | 0|\.\mult1.dat (deleted, filename)|32|0|260|0|1234|1078084897|1078084897|1078084897|1078084897 10 | 0|\.\mult1.dat:ADS (deleted, filename)|32|0|260|0|1234|1078084897|1078084897|1078084897|1078084897 11 | 0|\.\dir1 (deleted)|33|0|261|0|0|1078084980|1078084980|1078084980|1078084980 12 | 0|\.\dir1 (deleted, filename)|33|0|261|0|0|1078084980|1078084980|1078084980|1078084980 13 | 0|\$OrphanFiles\dir2 (deleted)|34|0|261|0|0|1078084983|1078084983|1078084983|1078084983 14 | 0|\$OrphanFiles\dir2 (deleted, filename)|34|0|261|0|0|1078084983|1078084983|1078084983|1078084983 15 | 0|\$OrphanFiles\frag3.dat (deleted)|35|0|260|0|2027|1078085029|1078085029|1078085029|1078085004 16 | 0|\$OrphanFiles\frag3.dat (deleted, filename)|35|0|260|0|2027|1078085004|1078085004|1078085004|1078085004 17 | 0|\$OrphanFiles\mult2.dat (deleted)|36|0|260|0|1715|1078085018|1078085018|1078085018|1078085018 18 | 0|\$OrphanFiles\mult2.dat (deleted, filename)|36|0|260|0|1715|1078085018|1078085018|1078085018|1078085018 19 | 0|\.\res1.dat (deleted)|37|0|260|0|0|1078085137|1078085137|1078085137|1078085137 20 | 0|\.\res1.dat (deleted, filename)|37|0|260|0|0|1078085137|1078085137|1078085137|1078085137 21 | 0|\$OrphanFiles\sing2.dat (deleted)|38|0|260|0|1005|1078085055|1078085055|1078085055|1078085055 22 | 0|\$OrphanFiles\sing2.dat (deleted, filename)|38|0|260|0|1005|1078085055|1078085055|1078085055|1078085055 23 | -------------------------------------------------------------------------------- /tests/MFTINDX/7-ntfs-undel.dd.mft.m.txt: -------------------------------------------------------------------------------- 1 | 0|\.\$MFT|0|0|256|0|39936|1078084677|1078084677|1078084677|1078084677 2 | 0|\.\$MFT (filename)|0|0|256|0|39936|1078084677|1078084677|1078084677|1078084677 3 | 0|\.\$MFTMirr|1|0|256|0|4096|1078084677|1078084677|1078084677|1078084677 4 | 0|\.\$MFTMirr (filename)|1|0|256|0|4096|1078084677|1078084677|1078084677|1078084677 5 | 0|\.\$LogFile|2|0|256|0|2097152|1078084677|1078084677|1078084677|1078084677 6 | 0|\.\$LogFile (filename)|2|0|256|0|2097152|1078084677|1078084677|1078084677|1078084677 7 | 0|\.\$Volume|3|0|0|0|0|1078084677|1078084677|1078084677|1078084677 8 | 0|\.\$Volume (filename)|3|0|0|0|0|1078084677|1078084677|1078084677|1078084677 9 | 0|\.\$AttrDef|4|0|0|0|2560|1078084677|1078084677|1078084677|1078084677 10 | 0|\.\$AttrDef (filename)|4|0|0|0|2560|1078084677|1078084677|1078084677|1078084677 11 | 0|\.|5|0|0|0|0|1078085971|1078085971|1078085971|1078084677 12 | 0|\. (filename)|5|0|0|0|0|1078084677|1078084677|1078084677|1078084677 13 | 0|\.\$Bitmap|6|0|256|0|752|1078084677|1078084677|1078084677|1078084677 14 | 0|\.\$Bitmap (filename)|6|0|256|0|752|1078084677|1078084677|1078084677|1078084677 15 | 0|\.\$Boot|7|0|0|0|8192|1078084677|1078084677|1078084677|1078084677 16 | 0|\.\$Boot (filename)|7|0|0|0|8192|1078084677|1078084677|1078084677|1078084677 17 | 0|\.\$BadClus|8|0|256|0|6160384|1078084677|1078084677|1078084677|1078084677 18 | 0|\.\$BadClus:$Bad|8|0|256|0|6160384|1078084677|1078084677|1078084677|1078084677 19 | 0|\.\$BadClus (filename)|8|0|256|0|6160384|1078084677|1078084677|1078084677|1078084677 20 | 0|\.\$BadClus:$Bad (filename)|8|0|256|0|6160384|1078084677|1078084677|1078084677|1078084677 21 | 0|\.\$Secure|9|0|257|0|263140|1078084677|1078084677|1078084677|1078084677 22 | 0|\.\$Secure:$SDS|9|0|257|0|263140|1078084677|1078084677|1078084677|1078084677 23 | 0|\.\$Secure (filename)|9|0|257|0|263140|0|0|0|-62135596800 24 | 0|\.\$Secure:$SDS (filename)|9|0|257|0|263140|0|0|0|-62135596800 25 | 0|\.\$UpCase|10|0|256|0|131072|1078084677|1078084677|1078084677|1078084677 26 | 0|\.\$UpCase (filename)|10|0|256|0|131072|1078084677|1078084677|1078084677|1078084677 27 | 0|\.\$Extend|11|0|257|0|0|1078084677|1078084677|1078084677|1078084677 28 | 0|\.\$Extend (filename)|11|0|257|0|0|1078084677|1078084677|1078084677|1078084677 29 | 0|\.\$Extend\$Quota|24|0|257|0|0|1078084684|1078084684|1078084684|1078084684 30 | 0|\.\$Extend\$Quota (filename)|24|0|257|0|0|1078084684|1078084684|1078084684|1078084684 31 | 0|\.\$Extend\$ObjId|25|0|257|0|0|1078084684|1078084684|1078084684|1078084684 32 | 0|\.\$Extend\$ObjId (filename)|25|0|257|0|0|1078084684|1078084684|1078084684|1078084684 33 | 0|\.\$Extend\$Reparse|26|0|257|0|0|1078084684|1078084684|1078084684|1078084684 34 | 0|\.\$Extend\$Reparse (filename)|26|0|257|0|0|1078084684|1078084684|1078084684|1078084684 35 | 0|\.\System Volume Information|27|0|258|0|0|1078085916|1078084751|1078084751|1078084750 36 | 0|\.\System Volume Information (filename)|27|0|258|0|0|1078084750|1078084750|1078084750|1078084750 37 | 0|\.\System Volume Information\tracking.log|28|0|259|0|20480|1078085172|1078085172|1078085172|1078084750 38 | 0|\.\System Volume Information\tracking.log (filename)|28|0|259|0|20480|1078084751|1078084751|1078084751|1078084750 39 | -------------------------------------------------------------------------------- /tests/MFTINDX/7-ntfs-undel.dd.s.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/williballenthin/INDXParse/038e8ec836cf23600124db74b40757b7184c08c5/tests/MFTINDX/7-ntfs-undel.dd.s.txt -------------------------------------------------------------------------------- /tests/MFTINDX/Makefile: -------------------------------------------------------------------------------- 1 | #!/usr/bin/make -f 2 | 3 | # Portions of this file contributed by NIST are governed by the 4 | # following statement: 5 | # 6 | # This software was developed at the National Institute of Standards 7 | # and Technology by employees of the Federal Government in the course 8 | # of their official duties. Pursuant to Title 17 Seciton 105 of the 9 | # United States Code, this software is not subject to copyright 10 | # protection within the United States. NIST assumes no responsibility 11 | # whatsoever for its use by other parties, and makes no guarantees, 12 | # expressed or implied, about its quality, reliability, or any other 13 | # characteristic. 14 | # 15 | # We would appreciate acknowledgement if the software is used. 16 | 17 | SHELL := /bin/bash 18 | 19 | top_srcdir := $(shell cd ../.. ; pwd) 20 | 21 | 7_ntfs_undel_dd := $(top_srcdir)/third_party/7-undel-ntfs/7-ntfs-undel.dd 22 | 23 | 7_ntfs_undel_dd_mft := $(top_srcdir)/third_party/7-ntfs-undel.dd.mft 24 | 25 | # TODO - This test-set does not yet contain an INDX sample. 26 | 27 | all: \ 28 | 7-ntfs-undel.dd.d.txt \ 29 | 7-ntfs-undel.dd.l.txt \ 30 | 7-ntfs-undel.dd.m.txt \ 31 | 7-ntfs-undel.dd.s.txt \ 32 | 7-ntfs-undel.dd.mft.d.txt \ 33 | 7-ntfs-undel.dd.mft.m.txt 34 | 35 | 7-ntfs-undel.dd.d.txt: \ 36 | $(7_ntfs_undel_dd) \ 37 | $(top_srcdir)/.venv.done.log \ 38 | $(top_srcdir)/indxparse/BinaryParser.py \ 39 | $(top_srcdir)/indxparse/MFT.py \ 40 | $(top_srcdir)/indxparse/Progress.py \ 41 | $(top_srcdir)/indxparse/MFTINDX.py 42 | rm -f _$@ 43 | source $(top_srcdir)/venv/bin/activate \ 44 | && MFTINDX.py \ 45 | -c 1024 \ 46 | -d \ 47 | -o 0 \ 48 | -t image \ 49 | $< \ 50 | > _$@ 51 | mv _$@ $@ 52 | 53 | 7-ntfs-undel.dd.l.txt: \ 54 | $(7_ntfs_undel_dd) \ 55 | $(top_srcdir)/.venv.done.log \ 56 | $(top_srcdir)/indxparse/BinaryParser.py \ 57 | $(top_srcdir)/indxparse/MFT.py \ 58 | $(top_srcdir)/indxparse/Progress.py \ 59 | $(top_srcdir)/indxparse/MFTINDX.py 60 | rm -f _$@ 61 | source $(top_srcdir)/venv/bin/activate \ 62 | && MFTINDX.py \ 63 | -c 1024 \ 64 | -l \ 65 | -o 0 \ 66 | -t image \ 67 | $< \ 68 | > _$@ 69 | mv _$@ $@ 70 | 71 | 7-ntfs-undel.dd.m.txt: \ 72 | $(7_ntfs_undel_dd) \ 73 | $(top_srcdir)/.venv.done.log \ 74 | $(top_srcdir)/indxparse/BinaryParser.py \ 75 | $(top_srcdir)/indxparse/MFT.py \ 76 | $(top_srcdir)/indxparse/Progress.py \ 77 | $(top_srcdir)/indxparse/MFTINDX.py 78 | rm -f _$@ 79 | source $(top_srcdir)/venv/bin/activate \ 80 | && MFTINDX.py \ 81 | -c 1024 \ 82 | -m \ 83 | -o 0 \ 84 | -t image \ 85 | $< \ 86 | > _$@ 87 | mv _$@ $@ 88 | 89 | 7-ntfs-undel.dd.mft.d.txt: \ 90 | $(7_ntfs_undel_dd_mft) \ 91 | $(top_srcdir)/.venv.done.log \ 92 | $(top_srcdir)/indxparse/BinaryParser.py \ 93 | $(top_srcdir)/indxparse/MFT.py \ 94 | $(top_srcdir)/indxparse/Progress.py \ 95 | $(top_srcdir)/indxparse/MFTINDX.py 96 | rm -f _$@ 97 | source $(top_srcdir)/venv/bin/activate \ 98 | && MFTINDX.py \ 99 | -c 1024 \ 100 | -d \ 101 | -o 0 \ 102 | -t MFT \ 103 | $< \ 104 | > _$@ 105 | mv _$@ $@ 106 | 107 | 7-ntfs-undel.dd.mft.m.txt: \ 108 | $(7_ntfs_undel_dd_mft) \ 109 | $(top_srcdir)/.venv.done.log \ 110 | $(top_srcdir)/indxparse/BinaryParser.py \ 111 | $(top_srcdir)/indxparse/MFT.py \ 112 | $(top_srcdir)/indxparse/Progress.py \ 113 | $(top_srcdir)/indxparse/MFTINDX.py 114 | rm -f _$@ 115 | source $(top_srcdir)/venv/bin/activate \ 116 | && MFTINDX.py \ 117 | -c 1024 \ 118 | -m \ 119 | -o 0 \ 120 | -t MFT \ 121 | $< \ 122 | > _$@ 123 | mv _$@ $@ 124 | 125 | 7-ntfs-undel.dd.s.txt: \ 126 | $(7_ntfs_undel_dd) \ 127 | $(top_srcdir)/.venv.done.log \ 128 | $(top_srcdir)/indxparse/BinaryParser.py \ 129 | $(top_srcdir)/indxparse/MFT.py \ 130 | $(top_srcdir)/indxparse/Progress.py \ 131 | $(top_srcdir)/indxparse/MFTINDX.py 132 | rm -f _$@ 133 | source $(top_srcdir)/venv/bin/activate \ 134 | && MFTINDX.py \ 135 | -c 1024 \ 136 | -o 0 \ 137 | -s \ 138 | -t image \ 139 | $< \ 140 | > _$@ 141 | mv _$@ $@ 142 | 143 | check: \ 144 | all 145 | 146 | clean: 147 | @rm -f \ 148 | *.txt \ 149 | _* 150 | -------------------------------------------------------------------------------- /tests/Makefile: -------------------------------------------------------------------------------- 1 | #!/usr/bin/make -f 2 | 3 | # Portions of this file contributed by NIST are governed by the 4 | # following statement: 5 | # 6 | # This software was developed at the National Institute of Standards 7 | # and Technology by employees of the Federal Government in the course 8 | # of their official duties. Pursuant to Title 17 Seciton 105 of the 9 | # United States Code, this software is not subject to copyright 10 | # protection within the United States. NIST assumes no responsibility 11 | # whatsoever for its use by other parties, and makes no guarantees, 12 | # expressed or implied, about its quality, reliability, or any other 13 | # characteristic. 14 | # 15 | # We would appreciate acknowledgement if the software is used. 16 | 17 | SHELL := /bin/bash 18 | 19 | all: \ 20 | all-console_scripts 21 | 22 | .PHONY: \ 23 | all-MFTINDX \ 24 | all-console_scripts \ 25 | all-list_mft \ 26 | all-tree_mft 27 | 28 | all-MFTINDX: 29 | $(MAKE) \ 30 | --directory MFTINDX 31 | 32 | # TODO: This list of unimplemented tests lines up with setup.cfg. 33 | # * all-INDXParse 34 | # * all-MFTView 35 | # * all-SDS_get_index 36 | # * all-extract_mft_record_slack 37 | # * all-fuse-mft 38 | # * all-get_file_info 39 | all-console_scripts: \ 40 | all-list_mft \ 41 | all-tree_mft \ 42 | all-MFTINDX 43 | 44 | all-list_mft: 45 | $(MAKE) \ 46 | --directory list_mft 47 | 48 | all-tree_mft: 49 | $(MAKE) \ 50 | --directory tree_mft 51 | 52 | check: \ 53 | all 54 | 55 | clean: 56 | @$(MAKE) \ 57 | --directory MFTINDX \ 58 | clean 59 | @$(MAKE) \ 60 | --directory tree_mft \ 61 | clean 62 | @$(MAKE) \ 63 | --directory list_mft \ 64 | clean 65 | -------------------------------------------------------------------------------- /tests/list_mft/7-ntfs-undel.dd.bodyfile: -------------------------------------------------------------------------------- 1 | 0|\\$ORPHAN\$MFT|0|0|256|0|39936|1078084677|1078084677|1078084677|1078084677 2 | 0|\\$ORPHAN\$MFT (filename)|0|0|256|0|39936|1078084677|1078084677|1078084677|1078084677 3 | 0|\\$ORPHAN\$MFTMirr|1|0|256|0|4096|1078084677|1078084677|1078084677|1078084677 4 | 0|\\$ORPHAN\$MFTMirr (filename)|1|0|256|0|4096|1078084677|1078084677|1078084677|1078084677 5 | 0|\\$ORPHAN\$LogFile|2|0|256|0|2097152|1078084677|1078084677|1078084677|1078084677 6 | 0|\\$ORPHAN\$LogFile (filename)|2|0|256|0|2097152|1078084677|1078084677|1078084677|1078084677 7 | 0|\\$ORPHAN\$Volume|3|0|0|0|0|1078084677|1078084677|1078084677|1078084677 8 | 0|\\$ORPHAN\$Volume (filename)|3|0|0|0|0|1078084677|1078084677|1078084677|1078084677 9 | 0|\\$ORPHAN\$AttrDef|4|0|0|0|2560|1078084677|1078084677|1078084677|1078084677 10 | 0|\\$ORPHAN\$AttrDef (filename)|4|0|0|0|2560|1078084677|1078084677|1078084677|1078084677 11 | 0|\\|5|0|0|0|0|1078085971|1078085971|1078085971|1078084677 12 | 0|\\ (filename)|5|0|0|0|0|1078084677|1078084677|1078084677|1078084677 13 | 0|\\$ORPHAN\$Bitmap|6|0|256|0|752|1078084677|1078084677|1078084677|1078084677 14 | 0|\\$ORPHAN\$Bitmap (filename)|6|0|256|0|752|1078084677|1078084677|1078084677|1078084677 15 | 0|\\$ORPHAN\$Boot|7|0|0|0|8192|1078084677|1078084677|1078084677|1078084677 16 | 0|\\$ORPHAN\$Boot (filename)|7|0|0|0|8192|1078084677|1078084677|1078084677|1078084677 17 | 0|\\$ORPHAN\$BadClus|8|0|256|0|6160384|1078084677|1078084677|1078084677|1078084677 18 | 0|\\$ORPHAN\$BadClus (filename)|8|0|256|0|6160384|1078084677|1078084677|1078084677|1078084677 19 | 0|\\$ORPHAN\$BadClus:$Bad|8|0|256|0|6160384|1078084677|1078084677|1078084677|1078084677 20 | 0|\\$ORPHAN\$Secure|9|0|257|0|263140|1078084677|1078084677|1078084677|1078084677 21 | 0|\\$ORPHAN\$Secure (filename)|9|0|257|0|263140|0|0|0|-62135596800 22 | 0|\\$ORPHAN\$Secure:$SDS|9|0|257|0|263140|1078084677|1078084677|1078084677|1078084677 23 | 0|\\$ORPHAN\$UpCase|10|0|256|0|131072|1078084677|1078084677|1078084677|1078084677 24 | 0|\\$ORPHAN\$UpCase (filename)|10|0|256|0|131072|1078084677|1078084677|1078084677|1078084677 25 | 0|\\$ORPHAN\$Extend|11|0|257|0|0|1078084677|1078084677|1078084677|1078084677 26 | 0|\\$ORPHAN\$Extend (filename)|11|0|257|0|0|1078084677|1078084677|1078084677|1078084677 27 | 0|\\$ORPHAN\$Extend\$ObjId (indx)|0|0|0|0|281474976710681|1078084684|1078084684|1078084684|1078084684 28 | 0|\\$ORPHAN\$Extend\$Quota (indx)|0|0|0|0|281474976710680|1078084684|1078084684|1078084684|1078084684 29 | 0|\\$ORPHAN\$Extend\$Reparse (indx)|0|0|0|0|281474976710682|1078084684|1078084684|1078084684|1078084684 30 | 0|\\??|12|0|0|0|0|1078084677|1078084677|1078084677|1078084677 31 | 0|\\??|13|0|0|0|0|1078084677|1078084677|1078084677|1078084677 32 | 0|\\??|14|0|0|0|0|1078084677|1078084677|1078084677|1078084677 33 | 0|\\??|15|0|0|0|0|1078084677|1078084677|1078084677|1078084677 34 | 0|\\$ORPHAN\mult1.dat|32|0|260|0|0|1078084897|1078084897|1078084897|1078084897 35 | 0|\\$ORPHAN\mult1.dat (filename)|32|0|260|0|0|1078084897|1078084897|1078084897|1078084897 36 | 0|\\$ORPHAN\$MFT|0|0|256|0|39936|1078084677|1078084677|1078084677|1078084677 37 | 0|\\$ORPHAN\$MFT (filename)|0|0|256|0|39936|1078084677|1078084677|1078084677|1078084677 38 | 0|\\$ORPHAN\$MFTMirr|1|0|256|0|4096|1078084677|1078084677|1078084677|1078084677 39 | 0|\\$ORPHAN\$MFTMirr (filename)|1|0|256|0|4096|1078084677|1078084677|1078084677|1078084677 40 | 0|\\$ORPHAN\$LogFile|2|0|256|0|2097152|1078084677|1078084677|1078084677|1078084677 41 | 0|\\$ORPHAN\$LogFile (filename)|2|0|256|0|2097152|1078084677|1078084677|1078084677|1078084677 42 | 0|\\$ORPHAN\$Volume|3|0|0|0|0|1078084677|1078084677|1078084677|1078084677 43 | 0|\\$ORPHAN\$Volume (filename)|3|0|0|0|0|1078084677|1078084677|1078084677|1078084677 44 | 0|\\$ORPHAN\$Quota|24|0|257|0|0|1078084684|1078084684|1078084684|1078084684 45 | 0|\\$ORPHAN\$Quota (filename)|24|0|257|0|0|1078084684|1078084684|1078084684|1078084684 46 | 0|\\$ORPHAN\$ObjId|25|0|257|0|0|1078084684|1078084684|1078084684|1078084684 47 | 0|\\$ORPHAN\$ObjId (filename)|25|0|257|0|0|1078084684|1078084684|1078084684|1078084684 48 | 0|\\$ORPHAN\$ObjId\ (indx)|0|0|0|0|3670048|0|-11560031107|0|0 49 | 0|\\$ORPHAN\$Reparse|26|0|257|0|0|1078084684|1078084684|1078084684|1078084684 50 | 0|\\$ORPHAN\$Reparse (filename)|26|0|257|0|0|1078084684|1078084684|1078084684|1078084684 51 | 0|\\$ORPHAN\System Volume Information|27|0|258|0|0|1078085916|1078084751|1078084751|1078084750 52 | 0|\\$ORPHAN\System Volume Information (filename)|27|0|258|0|0|1078084750|1078084750|1078084750|1078084750 53 | 0|\\$ORPHAN\System Volume Information\tracking.log (indx)|20480|0|0|0|562949953421340|1078085172|1078085172|1078085172|1078084750 54 | 0|\\$ORPHAN\tracking.log|28|0|259|0|20480|1078085172|1078085172|1078085172|1078084750 55 | 0|\\$ORPHAN\tracking.log (filename)|28|0|259|0|20480|1078084751|1078084751|1078084751|1078084750 56 | 0|\\$ORPHAN\frag1.dat (inactive)|29|0|260|0|1584|1078084840|1078084840|1078084840|1078084817 57 | 0|\\$ORPHAN\frag1.dat (filename, inactive)|29|0|260|0|1584|1078084817|1078084817|1078084817|1078084817 58 | 0|\\$ORPHAN\frag2.dat (inactive)|30|0|260|0|3873|1078084974|1078084974|1078084974|1078084829 59 | 0|\\$ORPHAN\frag2.dat (filename, inactive)|30|0|260|0|3873|1078084829|1078084829|1078084829|1078084829 60 | 0|\\$ORPHAN\sing1.dat (inactive)|31|0|260|0|780|1078084884|1078084884|1078084884|1078084884 61 | 0|\\$ORPHAN\sing1.dat (filename, inactive)|31|0|260|0|780|1078084884|1078084884|1078084884|1078084884 62 | 0|\\$ORPHAN\mult1.dat (inactive)|32|0|260|0|1234|1078084942|1078084942|1078084942|1078084897 63 | 0|\\$ORPHAN\mult1.dat (filename, inactive)|32|0|260|0|1234|1078084897|1078084897|1078084897|1078084897 64 | 0|\\$ORPHAN\mult1.dat:ADS (inactive)|32|0|260|0|1234|1078084942|1078084942|1078084942|1078084897 65 | 0|\\$ORPHAN\dir1 (inactive)|33|0|261|0|0|1078084980|1078084980|1078084980|1078084980 66 | 0|\\$ORPHAN\dir1 (filename, inactive)|33|0|261|0|0|1078084980|1078084980|1078084980|1078084980 67 | 0|\\$ORPHAN\dir2 (inactive)|34|0|261|0|0|1078084983|1078084983|1078084983|1078084983 68 | 0|\\$ORPHAN\dir2 (filename, inactive)|34|0|261|0|0|1078084983|1078084983|1078084983|1078084983 69 | 0|\\$ORPHAN\frag3.dat (inactive)|35|0|260|0|2027|1078085029|1078085029|1078085029|1078085004 70 | 0|\\$ORPHAN\frag3.dat (filename, inactive)|35|0|260|0|2027|1078085004|1078085004|1078085004|1078085004 71 | 0|\\$ORPHAN\mult2.dat (inactive)|36|0|260|0|1715|1078085018|1078085018|1078085018|1078085018 72 | 0|\\$ORPHAN\mult2.dat (filename, inactive)|36|0|260|0|1715|1078085018|1078085018|1078085018|1078085018 73 | 0|\\$ORPHAN\res1.dat (inactive)|37|0|260|0|0|1078085137|1078085137|1078085137|1078085137 74 | 0|\\$ORPHAN\res1.dat (filename, inactive)|37|0|260|0|0|1078085137|1078085137|1078085137|1078085137 75 | 0|\\$ORPHAN\sing2.dat (inactive)|38|0|260|0|1005|1078085055|1078085055|1078085055|1078085055 76 | 0|\\$ORPHAN\sing2.dat (filename, inactive)|38|0|260|0|1005|1078085055|1078085055|1078085055|1078085055 77 | -------------------------------------------------------------------------------- /tests/list_mft/7-ntfs-undel.dd.mft.bodyfile: -------------------------------------------------------------------------------- 1 | 0|\\\$MFT|0|0|256|0|39936|1078084677|1078084677|1078084677|1078084677 2 | 0|\\\$MFT (filename)|0|0|256|0|39936|1078084677|1078084677|1078084677|1078084677 3 | 0|\\\$MFTMirr|1|0|256|0|4096|1078084677|1078084677|1078084677|1078084677 4 | 0|\\\$MFTMirr (filename)|1|0|256|0|4096|1078084677|1078084677|1078084677|1078084677 5 | 0|\\\$LogFile|2|0|256|0|2097152|1078084677|1078084677|1078084677|1078084677 6 | 0|\\\$LogFile (filename)|2|0|256|0|2097152|1078084677|1078084677|1078084677|1078084677 7 | 0|\\\$Volume|3|0|0|0|0|1078084677|1078084677|1078084677|1078084677 8 | 0|\\\$Volume (filename)|3|0|0|0|0|1078084677|1078084677|1078084677|1078084677 9 | 0|\\\$AttrDef|4|0|0|0|2560|1078084677|1078084677|1078084677|1078084677 10 | 0|\\\$AttrDef (filename)|4|0|0|0|2560|1078084677|1078084677|1078084677|1078084677 11 | 0|\\|5|0|0|0|0|1078085971|1078085971|1078085971|1078084677 12 | 0|\\ (filename)|5|0|0|0|0|1078084677|1078084677|1078084677|1078084677 13 | 0|\\\$Bitmap|6|0|256|0|752|1078084677|1078084677|1078084677|1078084677 14 | 0|\\\$Bitmap (filename)|6|0|256|0|752|1078084677|1078084677|1078084677|1078084677 15 | 0|\\\$Boot|7|0|0|0|8192|1078084677|1078084677|1078084677|1078084677 16 | 0|\\\$Boot (filename)|7|0|0|0|8192|1078084677|1078084677|1078084677|1078084677 17 | 0|\\\$BadClus|8|0|256|0|6160384|1078084677|1078084677|1078084677|1078084677 18 | 0|\\\$BadClus (filename)|8|0|256|0|6160384|1078084677|1078084677|1078084677|1078084677 19 | 0|\\\$BadClus:$Bad|8|0|256|0|6160384|1078084677|1078084677|1078084677|1078084677 20 | 0|\\\$Secure|9|0|257|0|263140|1078084677|1078084677|1078084677|1078084677 21 | 0|\\\$Secure (filename)|9|0|257|0|263140|0|0|0|-62135596800 22 | 0|\\\$Secure:$SDS|9|0|257|0|263140|1078084677|1078084677|1078084677|1078084677 23 | 0|\\\$UpCase|10|0|256|0|131072|1078084677|1078084677|1078084677|1078084677 24 | 0|\\\$UpCase (filename)|10|0|256|0|131072|1078084677|1078084677|1078084677|1078084677 25 | 0|\\\$Extend|11|0|257|0|0|1078084677|1078084677|1078084677|1078084677 26 | 0|\\\$Extend (filename)|11|0|257|0|0|1078084677|1078084677|1078084677|1078084677 27 | 0|\\\$Extend\$ObjId (indx)|0|0|0|0|281474976710681|1078084684|1078084684|1078084684|1078084684 28 | 0|\\\$Extend\$Quota (indx)|0|0|0|0|281474976710680|1078084684|1078084684|1078084684|1078084684 29 | 0|\\\$Extend\$Reparse (indx)|0|0|0|0|281474976710682|1078084684|1078084684|1078084684|1078084684 30 | 0|\\\$Extend\$Quota|24|0|257|0|0|1078084684|1078084684|1078084684|1078084684 31 | 0|\\\$Extend\$Quota (filename)|24|0|257|0|0|1078084684|1078084684|1078084684|1078084684 32 | 0|\\\$Extend\$ObjId|25|0|257|0|0|1078084684|1078084684|1078084684|1078084684 33 | 0|\\\$Extend\$ObjId (filename)|25|0|257|0|0|1078084684|1078084684|1078084684|1078084684 34 | 0|\\\$Extend\$ObjId\ (indx)|0|0|0|0|3670048|0|-11560031107|0|0 35 | 0|\\\$Extend\$Reparse|26|0|257|0|0|1078084684|1078084684|1078084684|1078084684 36 | 0|\\\$Extend\$Reparse (filename)|26|0|257|0|0|1078084684|1078084684|1078084684|1078084684 37 | 0|\\\System Volume Information|27|0|258|0|0|1078085916|1078084751|1078084751|1078084750 38 | 0|\\\System Volume Information (filename)|27|0|258|0|0|1078084750|1078084750|1078084750|1078084750 39 | 0|\\\System Volume Information\tracking.log (indx)|20480|0|0|0|562949953421340|1078085172|1078085172|1078085172|1078084750 40 | 0|\\\System Volume Information\tracking.log|28|0|259|0|20480|1078085172|1078085172|1078085172|1078084750 41 | 0|\\\System Volume Information\tracking.log (filename)|28|0|259|0|20480|1078084751|1078084751|1078084751|1078084750 42 | 0|\\\frag1.dat (inactive)|29|0|260|0|1584|1078084840|1078084840|1078084840|1078084817 43 | 0|\\\frag1.dat (filename, inactive)|29|0|260|0|1584|1078084817|1078084817|1078084817|1078084817 44 | 0|\\\frag2.dat (inactive)|30|0|260|0|3873|1078084974|1078084974|1078084974|1078084829 45 | 0|\\\frag2.dat (filename, inactive)|30|0|260|0|3873|1078084829|1078084829|1078084829|1078084829 46 | 0|\\\sing1.dat (inactive)|31|0|260|0|780|1078084884|1078084884|1078084884|1078084884 47 | 0|\\\sing1.dat (filename, inactive)|31|0|260|0|780|1078084884|1078084884|1078084884|1078084884 48 | 0|\\\mult1.dat (inactive)|32|0|260|0|1234|1078084942|1078084942|1078084942|1078084897 49 | 0|\\\mult1.dat (filename, inactive)|32|0|260|0|1234|1078084897|1078084897|1078084897|1078084897 50 | 0|\\\mult1.dat:ADS (inactive)|32|0|260|0|1234|1078084942|1078084942|1078084942|1078084897 51 | 0|\\\dir1 (inactive)|33|0|261|0|0|1078084980|1078084980|1078084980|1078084980 52 | 0|\\\dir1 (filename, inactive)|33|0|261|0|0|1078084980|1078084980|1078084980|1078084980 53 | 0|\\$ORPHAN\dir2 (inactive)|34|0|261|0|0|1078084983|1078084983|1078084983|1078084983 54 | 0|\\$ORPHAN\dir2 (filename, inactive)|34|0|261|0|0|1078084983|1078084983|1078084983|1078084983 55 | 0|\\$ORPHAN\frag3.dat (inactive)|35|0|260|0|2027|1078085029|1078085029|1078085029|1078085004 56 | 0|\\$ORPHAN\frag3.dat (filename, inactive)|35|0|260|0|2027|1078085004|1078085004|1078085004|1078085004 57 | 0|\\$ORPHAN\mult2.dat (inactive)|36|0|260|0|1715|1078085018|1078085018|1078085018|1078085018 58 | 0|\\$ORPHAN\mult2.dat (filename, inactive)|36|0|260|0|1715|1078085018|1078085018|1078085018|1078085018 59 | 0|\\\res1.dat (inactive)|37|0|260|0|0|1078085137|1078085137|1078085137|1078085137 60 | 0|\\\res1.dat (filename, inactive)|37|0|260|0|0|1078085137|1078085137|1078085137|1078085137 61 | 0|\\$ORPHAN\sing2.dat (inactive)|38|0|260|0|1005|1078085055|1078085055|1078085055|1078085055 62 | 0|\\$ORPHAN\sing2.dat (filename, inactive)|38|0|260|0|1005|1078085055|1078085055|1078085055|1078085055 63 | -------------------------------------------------------------------------------- /tests/list_mft/Makefile: -------------------------------------------------------------------------------- 1 | #!/usr/bin/make -f 2 | 3 | # Portions of this file contributed by NIST are governed by the 4 | # following statement: 5 | # 6 | # This software was developed at the National Institute of Standards 7 | # and Technology by employees of the Federal Government in the course 8 | # of their official duties. Pursuant to Title 17 Seciton 105 of the 9 | # United States Code, this software is not subject to copyright 10 | # protection within the United States. NIST assumes no responsibility 11 | # whatsoever for its use by other parties, and makes no guarantees, 12 | # expressed or implied, about its quality, reliability, or any other 13 | # characteristic. 14 | # 15 | # We would appreciate acknowledgement if the software is used. 16 | 17 | SHELL := /bin/bash 18 | 19 | top_srcdir := $(shell cd ../.. ; pwd) 20 | 21 | 7_ntfs_undel_dd := $(top_srcdir)/third_party/7-undel-ntfs/7-ntfs-undel.dd 22 | 23 | 7_ntfs_undel_dd_mft := $(top_srcdir)/third_party/7-ntfs-undel.dd.mft 24 | 25 | all: \ 26 | 7-ntfs-undel.dd.bodyfile \ 27 | 7-ntfs-undel.dd.json \ 28 | 7-ntfs-undel.dd.mft.bodyfile \ 29 | 7-ntfs-undel.dd.mft.json 30 | 31 | 7-ntfs-undel.dd.bodyfile: \ 32 | $(7_ntfs_undel_dd) \ 33 | $(top_srcdir)/.venv.done.log \ 34 | $(top_srcdir)/indxparse/list_mft.py 35 | rm -f _$@ 36 | source $(top_srcdir)/venv/bin/activate \ 37 | && list_mft.py \ 38 | $< \ 39 | > _$@ 40 | mv _$@ $@ 41 | 42 | 7-ntfs-undel.dd.json: \ 43 | $(7_ntfs_undel_dd) \ 44 | $(top_srcdir)/.venv.done.log \ 45 | $(top_srcdir)/indxparse/list_mft.py 46 | rm -f __$@ _$@ 47 | source $(top_srcdir)/venv/bin/activate \ 48 | && list_mft.py \ 49 | --json \ 50 | $< \ 51 | > __$@ 52 | python3 -m json.tool \ 53 | __$@ \ 54 | _$@ 55 | rm __$@ 56 | mv _$@ $@ 57 | 58 | 7-ntfs-undel.dd.mft.bodyfile: \ 59 | $(7_ntfs_undel_dd_mft) \ 60 | $(top_srcdir)/.venv.done.log \ 61 | $(top_srcdir)/indxparse/list_mft.py 62 | rm -f _$@ 63 | source $(top_srcdir)/venv/bin/activate \ 64 | && list_mft.py \ 65 | $< \ 66 | > _$@ 67 | mv _$@ $@ 68 | 69 | 7-ntfs-undel.dd.mft.json: \ 70 | $(7_ntfs_undel_dd_mft) \ 71 | $(top_srcdir)/.venv.done.log \ 72 | $(top_srcdir)/indxparse/list_mft.py 73 | rm -f __$@ _$@ 74 | source $(top_srcdir)/venv/bin/activate \ 75 | && list_mft.py \ 76 | --json \ 77 | $< \ 78 | > __$@ 79 | python3 -m json.tool \ 80 | __$@ \ 81 | _$@ 82 | rm __$@ 83 | mv _$@ $@ 84 | 85 | check: \ 86 | all 87 | 88 | clean: 89 | @rm -f \ 90 | *.bodyfile \ 91 | *.json \ 92 | _* 93 | -------------------------------------------------------------------------------- /tests/tree_mft/7-ntfs-undel.dd.mft.txt: -------------------------------------------------------------------------------- 1 | \. 2 | $MFT 3 | $MFTMirr 4 | $LogFile 5 | $Volume 6 | $AttrDef 7 | $Bitmap 8 | $Boot 9 | $BadClus 10 | $Secure 11 | $UpCase 12 | $Extend 13 | $Quota 14 | $ObjId 15 | $Reparse 16 | System Volume Information 17 | tracking.log 18 | frag1.dat 19 | frag2.dat 20 | sing1.dat 21 | mult1.dat 22 | dir1 23 | res1.dat 24 | -------------------------------------------------------------------------------- /tests/tree_mft/Makefile: -------------------------------------------------------------------------------- 1 | #!/usr/bin/make -f 2 | 3 | # Portions of this file contributed by NIST are governed by the 4 | # following statement: 5 | # 6 | # This software was developed at the National Institute of Standards 7 | # and Technology by employees of the Federal Government in the course 8 | # of their official duties. Pursuant to Title 17 Seciton 105 of the 9 | # United States Code, this software is not subject to copyright 10 | # protection within the United States. NIST assumes no responsibility 11 | # whatsoever for its use by other parties, and makes no guarantees, 12 | # expressed or implied, about its quality, reliability, or any other 13 | # characteristic. 14 | # 15 | # We would appreciate acknowledgement if the software is used. 16 | 17 | SHELL := /bin/bash 18 | 19 | top_srcdir := $(shell cd ../.. ; pwd) 20 | 21 | 7_ntfs_undel_dd_mft := $(top_srcdir)/third_party/7-ntfs-undel.dd.mft 22 | 23 | all: \ 24 | 7-ntfs-undel.dd.mft.txt 25 | 26 | 7-ntfs-undel.dd.mft.txt: \ 27 | $(7_ntfs_undel_dd_mft) \ 28 | $(top_srcdir)/.venv.done.log \ 29 | $(top_srcdir)/indxparse/BinaryParser.py \ 30 | $(top_srcdir)/indxparse/MFT.py \ 31 | $(top_srcdir)/indxparse/Progress.py \ 32 | $(top_srcdir)/indxparse/tree_mft.py 33 | rm -f _$@ 34 | source $(top_srcdir)/venv/bin/activate \ 35 | && tree_mft.py \ 36 | $< \ 37 | > _$@ 38 | mv _$@ $@ 39 | 40 | check: \ 41 | all 42 | 43 | clean: 44 | @rm -f \ 45 | *.txt \ 46 | _* 47 | -------------------------------------------------------------------------------- /third_party/.gitignore: -------------------------------------------------------------------------------- 1 | *.mft 2 | .*.done.log 3 | 7-undel-ntfs 4 | _* 5 | -------------------------------------------------------------------------------- /third_party/7-ntfs-undel.dd.md5: -------------------------------------------------------------------------------- 1 | e7dbb96759d9cd62b729463ebfe61dab 2 | -------------------------------------------------------------------------------- /third_party/7-ntfs-undel.dd.mft.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/williballenthin/INDXParse/038e8ec836cf23600124db74b40757b7184c08c5/third_party/7-ntfs-undel.dd.mft.gz -------------------------------------------------------------------------------- /third_party/7-ntfs-undel.dd.mft.sha2-256: -------------------------------------------------------------------------------- 1 | 61599172a16abbd697f1ef4ed2c1baa810c7b354af31014ed2f74c854a11f6a3 2 | -------------------------------------------------------------------------------- /third_party/7-ntfs-undel.dd.mft.sha3-256: -------------------------------------------------------------------------------- 1 | caf99e699d01b03066561e2ac724337a2db3a4c24d65521e3710e29c042aae14 2 | -------------------------------------------------------------------------------- /third_party/7-ntfs-undel.dd.sha2-256: -------------------------------------------------------------------------------- 1 | 4138cc42148e3381e3c66eb50090f4a30416ae8c20247c2b9bad27cf8c764d88 2 | -------------------------------------------------------------------------------- /third_party/7-ntfs-undel.dd.sha3-256: -------------------------------------------------------------------------------- 1 | 7fe01ab6190fef07a53c1179d67afd3094502a01ca62c0a88c6dd14e2e88bbba 2 | -------------------------------------------------------------------------------- /third_party/7-undel-ntfs.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/williballenthin/INDXParse/038e8ec836cf23600124db74b40757b7184c08c5/third_party/7-undel-ntfs.zip -------------------------------------------------------------------------------- /third_party/7-undel-ntfs.zip.sha2-256: -------------------------------------------------------------------------------- 1 | be2a85eb4e4e210d0c974254483d5ecacd4ea23f1e19f2110249ba306f72b5b7 2 | -------------------------------------------------------------------------------- /third_party/7-undel-ntfs.zip.sha3-256: -------------------------------------------------------------------------------- 1 | 1c686165e12f56da002c9b552c6f2ce0db5e65337ce2878f293a723e62bdaa6c 2 | -------------------------------------------------------------------------------- /third_party/Makefile: -------------------------------------------------------------------------------- 1 | #!/usr/bin/make -f 2 | 3 | # Portions of this file contributed by NIST are governed by the 4 | # following statement: 5 | # 6 | # This software was developed at the National Institute of Standards 7 | # and Technology by employees of the Federal Government in the course 8 | # of their official duties. Pursuant to Title 17 Seciton 105 of the 9 | # United States Code, this software is not subject to copyright 10 | # protection within the United States. NIST assumes no responsibility 11 | # whatsoever for its use by other parties, and makes no guarantees, 12 | # expressed or implied, about its quality, reliability, or any other 13 | # characteristic. 14 | # 15 | # We would appreciate acknowledgement if the software is used. 16 | 17 | SHELL := /bin/bash 18 | 19 | all: \ 20 | .7-undel-ntfs.zip.done.log \ 21 | 7-ntfs-undel.dd.mft 22 | 23 | .PRECIOUS: \ 24 | 7-undel-ntfs.zip 25 | 26 | .7-undel-ntfs.zip.done.log: \ 27 | 7-undel-ntfs.zip 28 | rm -rf 7-undel-ntfs 29 | unzip $< 30 | # Verify hashes of test disk image. 31 | test \ 32 | "x$$(openssl dgst -md5 7-undel-ntfs/7-ntfs-undel.dd | awk '{print($$NF)}')" \ 33 | == \ 34 | "x$$(head -n1 7-ntfs-undel.dd.md5)" 35 | test \ 36 | "x$$(openssl dgst -sha256 7-undel-ntfs/7-ntfs-undel.dd | awk '{print($$NF)}')" \ 37 | == \ 38 | "x$$(head -n1 7-ntfs-undel.dd.sha2-256)" 39 | test \ 40 | "x$$(openssl dgst -sha3-256 7-undel-ntfs/7-ntfs-undel.dd | awk '{print($$NF)}')" \ 41 | == \ 42 | "x$$(head -n1 7-ntfs-undel.dd.sha3-256)" 43 | touch $@ 44 | 45 | # This file was originally extracted from 7-undel-ntfs.zip's contents, 46 | # using The SleuthKit 4.12.1's icat with this command: 47 | # 48 | # icat 7-undel-ntfs.zip/7-ntfs-undel.dd 0 > 7-ntfs-undel.dd.mft 49 | # 50 | # To avoid adding build or test dependencies, the extracted file and its 51 | # hashes are stored in this repository. The file was gzip'd using the 52 | # Python gzip module, symmetrically to the module's usage in this 53 | # recipe: 54 | # 55 | # python3 -m gzip --best 7-ntfs-undel.dd.mft > 7-ntfs-undel.dd.mft.gz 56 | # 57 | # The initial file-copy operation is because the "gunzip" mode of the 58 | # Python gzip module takes "$x.gz" and creates "$x", leaving "$x.gz" in 59 | # place and printing nothing to stdout. (I.e. there is no equivalent to 60 | # "gunzip --to-stdout $x.gz > $x".) 61 | 7-ntfs-undel.dd.mft: \ 62 | 7-ntfs-undel.dd.mft.gz 63 | test -r $@.sha2-256 \ 64 | || (echo "ERROR:third_party/Makefile:Recorded SHA2-256 hash not found." >&2 ; exit 1) 65 | test -r $@.sha3-256 \ 66 | || (echo "ERROR:third_party/Makefile:Recorded SHA3-256 hash not found." >&2 ; exit 1) 67 | cp \ 68 | $< \ 69 | _$@.gz 70 | python3 -m gzip \ 71 | --decompress \ 72 | _$@.gz 73 | rm _$@.gz 74 | test \ 75 | "x$$(openssl dgst -sha256 _$@ | awk '{print($$NF)}')" \ 76 | == \ 77 | "x$$(head -n1 $@.sha2-256)" 78 | test \ 79 | "x$$(openssl dgst -sha3-256 _$@ | awk '{print($$NF)}')" \ 80 | == \ 81 | "x$$(head -n1 $@.sha3-256)" 82 | mv _$@ $@ 83 | 84 | # This zip file is part of a dataset with this home page: 85 | # https://dftt.sourceforge.net/test7/index.html 86 | 7-undel-ntfs.zip: 87 | test -r $@.sha2-256 \ 88 | || (echo "ERROR:third_party/Makefile:Recorded SHA2-256 hash not found." >&2 ; exit 1) 89 | test -r $@.sha3-256 \ 90 | || (echo "ERROR:third_party/Makefile:Recorded SHA3-256 hash not found." >&2 ; exit 1) 91 | # Verify hashes of downloaded zip. 92 | wget \ 93 | --output-document _$@ \ 94 | https://prdownloads.sourceforge.net/dftt/7-undel-ntfs.zip?download 95 | test \ 96 | "x$$(openssl dgst -sha256 _$@ | awk '{print($$NF)}')" \ 97 | == \ 98 | "x$$(head -n1 $@.sha2-256)" 99 | test \ 100 | "x$$(openssl dgst -sha3-256 _$@ | awk '{print($$NF)}')" \ 101 | == \ 102 | "x$$(head -n1 $@.sha3-256)" 103 | mv _$@ $@ 104 | 105 | check: \ 106 | .7-undel-ntfs.zip.done.log \ 107 | 7-ntfs-undel.dd.mft 108 | 109 | clean: 110 | @rm -f \ 111 | *.mft \ 112 | .*.done.log 113 | @rm -rf \ 114 | 7-undel-ntfs 115 | --------------------------------------------------------------------------------