├── .gitignore ├── LICENSE ├── MANIFEST.in ├── README.md ├── leancrawler ├── __init__.py ├── crawler.lean └── crawler.py ├── requirements.txt ├── requirements_tests.txt ├── setup.py ├── tests ├── test.yaml └── test_dummy.py └── tox.ini /.gitignore: -------------------------------------------------------------------------------- 1 | *.htm* 2 | __pycache__ 3 | *.sw* 4 | *.db 5 | .coverage 6 | *.mypy_cache 7 | .pytest_cache 8 | .tox 9 | .python-version 10 | src/Lean_Crawler.egg-info/ 11 | *.olean 12 | *.gephi 13 | *.gexf 14 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "{}" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright {yyyy} {name of copyright owner} 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include leancrawler/crawler.lean 2 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Lean crawler 2 | 3 | **Warning:** This tool is a Lean 3 tool, it has no relevance to today’s world. Using it makes sense only for paleontologists. 4 | 5 | This is a python library which gathers statistics and relational information 6 | about Lean libraries. It is at a very early experimental stage, 7 | but already usable for fun. 8 | 9 | ## Installation 10 | 11 | You need Python 3.6 or later, and Lean 3.26 or later. Make sure the python package 12 | manager `pip` is installed. Clone this repository, go to its root directory 13 | and run `pip install .` (using `sudo` if needed). It's also recommended to 14 | install `ipython` for interactive use. Alternatively, if you don't want to mess 15 | up with your global python environment, you can use a dedicated virtual 16 | environment, as explained below. 17 | 18 | ### Optional: setting up a virtual python environment 19 | Use `pip install --user virtualenvwrapper`, and add to your `.bashrc` something like: 20 | ```bash 21 | # Python virtualenvwrapper 22 | export WORKON_HOME=$HOME/.virtualenvs 23 | export VIRTUALENVWRAPPER_VIRTUALENV=/usr/local/bin/virtualenv 24 | source $HOME/.local/bin/virtualenvwrapper.sh 25 | ``` 26 | You can then run `mkvirtualenv --python=/usr/bin/python3.6 leancrawler` to 27 | create your virtual environment. Once inside this environment (either because 28 | you just created it or after running `workon leancrawler`), you can pip 29 | install. Note that you can leave the environment by running `deactivate`. 30 | 31 | 32 | ## Usage 33 | 34 | Inside a valid Lean project folder, if you want to inspect the theory 35 | visible in `my_file.lean`, run `leancrawler my_file` (if `my_file` is not at 36 | the root of you project source then you can indicate a dot separated path, just 37 | as you would do when importing it in Lean). This will 38 | create a file `crawl.lean` in the current folder. Then you need to make sure 39 | you have up to date olean files for `my_file` and all its dependencies. 40 | Then you can run `lean --run crawl.lean`. 41 | This will take a while if you want to inspect a large theory. In the end you 42 | should have a new file called `data.yaml` (you can change this name by providing a 43 | second argument to the `leancrawler` invocation). 44 | 45 | Then run `ipython` (or regular `python` if masochistic), and try something like: 46 | 47 | ```python 48 | from leancrawler import LeanLib, LeanDeclGraph 49 | 50 | lib = LeanLib.from_yaml('My nice lib', 'my_data.yaml') 51 | ``` 52 | 53 | This will also take a noticable time, but much less than Lean's work 54 | above. You can save that work for later use by typing 55 | `lib.dump('my_py_data')` and retrieve it in a later python session using 56 | `lib = LeanLib.load_dump('my_py_data')`. 57 | 58 | Then `lib` is basically a dictionary of whose keys are Lean names (as 59 | python strings) and values are `LeanDecl` which contain a bunch a 60 | informations about a Lean declaration. Try `lib['group']` to see what is 61 | stored about the `group` declaration from Lean's core library. 62 | 63 | One way of playing with the lib is to make a graph. 64 | 65 | ```python 66 | G = LeanDeclGraph.from_lib(lib) 67 | ``` 68 | 69 | This graph will include lots of nodes coming from basic logic (like 70 | `eq`, `Exists`, etc.) which can be seen as noise. You can get rid of 71 | some of them using `lib.prune_foundations()` before creating the graph. 72 | This function takes an optional argument `files` which is a list of substrings 73 | whose appearance in a declaration filename flags it for removal. 74 | For instance `lib.prune_foundations(files=['mathlib', 'elan'])` will remove 75 | any declaration from mathlib and the core library. 76 | Similarly, you can use the optional argument `prefixes` to indicate a list 77 | of declaration name prefixes to remove. 78 | For instance `lib.prune_foundations(prefixes=['list', 'set', 'finset'])` will remove 79 | any declaration whose name starts with either `list`, `set` or `finset`. 80 | 81 | If you are interested only in nodes leading up to `group`, you can try 82 | `group_graph = G.component_of('group')`. Then you can export it as a 83 | gexf file using `group_graph.write('group.gexf')`. 84 | 85 | 86 | You can also explore the graph using networkx's API, for instance 87 | `nx.dag_longest_path(G)` will show the longest path in the graph 88 | while `nx.shortest_path(gr, 'my_def', 'my_lemma')` will show the shortest 89 | path from `my_def` to `my_lemma`. 90 | 91 | ## Contributing 92 | 93 | In order to setup a dev environment, run `pip install -r requirements_tests.txt`. 94 | You will probably want to install `leancrawler` in dev 95 | mode using `pip install -e .` (adding the `-e` switch compared to instructions 96 | above). You also need to make sure both python 3.6 and 3.7 are installed, since 97 | tests are run against both version. See [pyenv](https://github.com/pyenv/pyenv) 98 | if unsure how to ensure that. Then use `tox` to run tests and linting. If you only want to 99 | * test for python 3.6: `tox -e py36` 100 | * test for python 3.7: `tox -e py37` 101 | * run [static type checking](http://mypy-lang.org/): `tox -e mypy` 102 | * run [PEP8](https://www.python.org/dev/peps/pep-0008/) linting: `tox -e flake8` 103 | 104 | Note that the testing setup is done, but currently there is only one trivial test. 105 | -------------------------------------------------------------------------------- /leancrawler/__init__.py: -------------------------------------------------------------------------------- 1 | from .crawler import LeanDecl, LeanLib, LeanDeclGraph 2 | 3 | import networkx as nx 4 | 5 | __all__ = ['LeanDecl', 'LeanLib', 'LeanDeclGraph', 'nx'] 6 | -------------------------------------------------------------------------------- /leancrawler/crawler.lean: -------------------------------------------------------------------------------- 1 | 2 | import meta.expr 3 | import system.io 4 | 5 | open tactic declaration environment io io.fs (put_str_ln close) 6 | 7 | -- The next instance is there to prevent PyYAML trying to be too smart 8 | meta def my_name_to_string : has_to_string name := 9 | ⟨λ n, "\"" ++ to_string n ++ "\""⟩ 10 | 11 | local attribute [instance] my_name_to_string 12 | 13 | meta def expr.get_pi_app_fn : expr → expr 14 | | (expr.pi _ _ _ e) := e.get_pi_app_fn 15 | | e := e.get_app_fn 16 | 17 | namespace name_set 18 | meta def partition (P : name → bool) (s : name_set) : name_set × name_set := 19 | s.fold (s, s) (λ a m, if P a then (m.1, m.2.erase a) else (m.1.erase a, m.2)) 20 | end name_set 21 | 22 | /-- 23 | `pre.list_items_aux nm` returns the list of names occuring in the declaration `nm` or (recusively) 24 | in any declarations occurring in the value of `nm` with namespace `pre` 25 | and whose last component starts with `_`. 26 | Auxiliary function for `list_items`. -/ 27 | meta def list_items_aux (pre : name) : name → tactic name_set | nm := do 28 | env ← get_env, 29 | decl ← get_decl nm, 30 | let l := decl.value.list_constant, 31 | let (aux, l₂) := l.partition (λ nm : name, nm.get_prefix = pre ∧ nm.last.front = '_'), 32 | aux.mfold l₂ (λ nm l', list_items_aux nm >>= λ l'', return (l'.union l'')) 33 | 34 | /-- `list_value_items nm` returns the list of names occuring in the declaration `nm` or (recusively) 35 | in any declarations `nm._proof_i` (or to be more precise: any declaration in namespace `nm` 36 | whose last part of the name starts with `_`). -/ 37 | meta def list_value_items (nm : name) : tactic (list name) := do 38 | l ← list_items_aux nm nm, 39 | return l.to_list 40 | -- let l := l.to_list.map (λ nm : name, if nm.last.front = '_' then nm.get_prefix else nm), 41 | -- return l.dedup 42 | 43 | /-- `list_value_items nm` returns the list of names occuring in the declaration `nm` or (recusively) 44 | in any declarations `nm._proof_i` (or to be more precise: any declaration in namespace `nm` 45 | whose last part of the name starts with `_`). -/ 46 | meta def list_type_items (nm₀ : name) : tactic (list name) := do 47 | env ← get_env, 48 | decl ← get_decl nm₀, 49 | let l := decl.type.list_constant, 50 | let (aux, l₂) := l.partition (λ nm : name, nm.get_prefix = nm₀ ∧ nm.last.front = '_'), 51 | l₃ ← aux.mfold l₂ (λ nm l', list_items_aux nm₀ nm >>= λ l'', return (l'.union l'')), 52 | return l₃.to_list 53 | 54 | meta def mnot : bool → tactic bool := λ p, return (¬ p) 55 | 56 | meta def pos_line (p : option pos) : string := 57 | match p with 58 | | some x := to_string x.line 59 | | _ := "" 60 | end 61 | 62 | meta def file_name (p : option string) : string := 63 | match p with 64 | | some x := x 65 | | _ := "Unknown file" 66 | end 67 | 68 | section 69 | 70 | structure declaration.modifiers := 71 | (Class := ff) 72 | (Structure := ff) 73 | (StructureField := ff) 74 | (Inductive := ff) 75 | (Instance := ff) 76 | (IsRecursor := ff) 77 | (IsConstructor := ff) 78 | 79 | def bool.to_string_python : has_to_string bool := ⟨λ k, match k with tt := "True" | ff := "False" end⟩ 80 | local attribute [instance] bool.to_string_python 81 | 82 | instance : has_to_string declaration.modifiers := ⟨λ m, 83 | "{ class: " ++ to_string m.Class ++ 84 | ", structure: " ++ to_string m.Structure ++ 85 | ", structure_field: " ++ to_string m.StructureField ++ 86 | ", is_recursor: " ++ to_string m.IsRecursor ++ 87 | ", is_constructor: " ++ to_string m.IsConstructor ++ 88 | ", inductive: " ++ to_string m.Inductive ++ 89 | ", instance: " ++ to_string m.Instance ++ " }"⟩ 90 | 91 | open tactic declaration environment 92 | 93 | meta def declaration.get_kind_string : declaration → string 94 | | (thm _ _ _ _) := "lemma" 95 | | (defn _ _ _ _ _ _) := "definition" 96 | | (cnst _ _ _ _) := "constant" 97 | | (ax _ _ _) := "axiom" 98 | 99 | 100 | meta def environment.get_modifiers (env : environment) (n : name) : tactic declaration.modifiers := 101 | do 102 | c ← (has_attribute `class n >> return tt) <|> return ff, 103 | i ← (has_attribute `instance n >> return tt) <|> return ff, 104 | return { 105 | Class := c, 106 | Structure := env.is_structure n, 107 | StructureField := (env.is_projection n).is_some, 108 | IsConstructor := env.is_constructor n, 109 | IsRecursor := env.is_recursor n, 110 | Inductive := env.is_ginductive n, 111 | Instance := i } 112 | end 113 | 114 | 115 | meta def print_item_crawl (env : environment) (decl : declaration) : tactic string := 116 | let name := decl.to_name, 117 | pos := pos_line (env.decl_pos name), 118 | fname := file_name (env.decl_olean name) in 119 | do 120 | let res := "- Name: " ++ to_string name ++ "\n", 121 | let res := res ++ " File: " ++ fname ++ "\n", 122 | let res := res ++ " Line: " ++ pos ++ "\n", 123 | let res := res ++ " Kind: " ++ decl.get_kind_string ++ "\n", 124 | mods ← env.get_modifiers name, 125 | let res := res ++ " Modifiers: " ++ to_string mods ++ "\n", 126 | 127 | pp_type ← pp decl.type, 128 | let res := res ++ " Type: " ++ (to_string pp_type).quote ++ "\n", 129 | type_decls ← list_type_items name, 130 | type_proofs ← type_decls.mfilter $ λ c, mk_const c >>= is_proof, 131 | type_others ← type_decls.mfilter $ λ c, mk_const c >>= is_proof >>= mnot, 132 | let res := res ++ " Type uses proofs: " ++ to_string type_proofs ++ "\n", 133 | let res := res ++ " Type uses others: " ++ to_string type_others ++ "\n", 134 | 135 | pp_value ← pp decl.value, 136 | let res := res ++ " Value: " ++ (to_string pp_value).quote ++ "\n", 137 | value_decls ← list_value_items name, 138 | value_proofs ← value_decls.mfilter $ λ c, mk_const c >>= is_proof, 139 | value_others ← value_decls.mfilter $ λ c, mk_const c >>= is_proof >>= mnot, 140 | let res := res ++ " Value uses proofs: " ++ to_string value_proofs ++ "\n", 141 | let res := res ++ " Value uses others: " ++ to_string value_others ++ "\n", 142 | 143 | let res := res ++ (" Target class: " ++ (if mods.Instance then to_string decl.type.get_pi_app_fn else "") ++ "\n"), 144 | let res := res ++ (" Parent: " ++ match env.is_projection name with 145 | | some info := to_string info.cname ++ "\n" 146 | | none := "\n" 147 | end), 148 | let res := res ++ (" Fields: " ++ (to_string $ (env.structure_fields_full name).get_or_else []) ++ "\n"), 149 | return res 150 | 151 | 152 | meta def main : tactic unit := 153 | do curr_env ← get_env, 154 | h ← unsafe_run_io (mk_file_handle "data.yaml" mode.write), 155 | let decls := curr_env.fold [] list.cons, 156 | let filtered_decls := decls.filter 157 | (λ x, not (to_name x).is_internal), 158 | filtered_decls.mmap' (λ d, 159 | do s ← (print_item_crawl curr_env d), 160 | unsafe_run_io (do io.fs.put_str_ln h s, 161 | io.fs.flush h), 162 | skip), 163 | unsafe_run_io (close h), 164 | skip 165 | -------------------------------------------------------------------------------- /leancrawler/crawler.py: -------------------------------------------------------------------------------- 1 | """ 2 | Python classes storing information about Lean libraries 3 | """ 4 | 5 | from dataclasses import dataclass, field 6 | from typing import List, Dict, Optional, Set 7 | from copy import deepcopy 8 | import pickle 9 | import sys 10 | from pathlib import Path 11 | 12 | from yaml import safe_load 13 | import networkx as nx 14 | from networkx.drawing.nx_pydot import graphviz_layout 15 | 16 | 17 | def strip_name(name): 18 | """ Remove the last component of a Lean name.""" 19 | return '.'.join(name.split('.')[:-1]) 20 | 21 | 22 | # Lean names ending with the following are auxilliary definitions 23 | AUX_DEF_SUFFIX = ('.rec', '.brec', '.brec_on', '.mk', '.rec_on', '.inj_on', 24 | '.has_sizeof_inst', '.no_confusion_type', '.no_confusion', 25 | '.cases_on', '.inj_arrow', '.sizeof', '.inj', 26 | '.inj_eq', '.sizeof_spec', '.drec', '.dcases_on', 27 | '.drec_on', '.below', '.ibelow', '.binduction_on',) 28 | 29 | 30 | @dataclass 31 | class LeanDecl: 32 | """ A Lean declaration """ 33 | # pylint: disable=too-many-instance-attributes 34 | name: str 35 | filename: str 36 | line_nb: int = 0 37 | kind: str = '' 38 | is_inductive: bool = False 39 | is_structure: bool = False 40 | is_structure_field: bool = False 41 | is_class: bool = False 42 | is_instance: bool = False 43 | is_recursor: bool = False 44 | is_constructor: bool = False 45 | Type: str = '' 46 | type_uses_proofs: Set[str] = field(default_factory=set) 47 | type_uses_others: Set[str] = field(default_factory=set) 48 | type_size: int = 0 49 | type_dedup_size: int = 0 50 | type_pp_size: int = 0 51 | Value: str = '' 52 | value_uses_proofs: Set[str] = field(default_factory=set) 53 | value_uses_others: Set[str] = field(default_factory=set) 54 | value_size: int = 0 55 | value_dedup_size: int = 0 56 | value_pp_size: int = 0 57 | target_class: Optional[str] = None 58 | parent: Optional[str] = None 59 | fields: Optional[List[str]] = None 60 | 61 | @classmethod 62 | def from_dict(cls, d): 63 | """ Create a LeanDecl from a dictionary coming from a YaML file. """ 64 | decl = cls(name=d['Name'], 65 | filename=d['File'], 66 | line_nb=d['Line'], 67 | kind=d['Kind'], 68 | is_inductive=d['Modifiers']['inductive'], 69 | is_structure=d['Modifiers']['structure'], 70 | is_structure_field=d['Modifiers']['structure_field'], 71 | is_class=d['Modifiers']['class'], 72 | is_instance=d['Modifiers']['instance'], 73 | is_recursor=d['Modifiers']['is_recursor'], 74 | is_constructor=d['Modifiers']['is_constructor'], 75 | Type=d.get('Type', ''), 76 | type_uses_proofs=set(d.get('Type uses proofs', [])), 77 | type_uses_others=set(d.get('Type uses others', [])), 78 | type_size=d.get('Type size', 0), 79 | type_dedup_size=d.get('Type dedup size', 0), 80 | type_pp_size=d.get('Type pp size', 0), 81 | Value=d.get('Value', ''), 82 | value_uses_proofs=set(d.get('Value uses proofs', [])), 83 | value_uses_others=set(d.get('Value uses others', [])), 84 | value_size=d.get('Value size', 0), 85 | value_dedup_size=d.get('Value dedup size', 0), 86 | value_pp_size=d.get('Value pp size', 0), 87 | target_class=d.get('Target class', 0), 88 | parent=d.get('Parent', ''), 89 | fields=d.get('Fields', None)) 90 | if decl.is_structure_field: 91 | # Remove the trailing ".mk" 92 | decl.parent = strip_name(decl.parent) 93 | elif decl.is_constructor: 94 | decl.parent = strip_name(decl.name) 95 | return decl 96 | 97 | @property 98 | def user_kind(self) -> str: 99 | """ A heuristic more informative kind of declaration. """ 100 | if self.is_class: 101 | return 'class' 102 | if self.is_instance: 103 | return 'instance' 104 | if self.is_structure: 105 | return 'structure' 106 | if self.is_inductive: 107 | return 'inductive' 108 | return self.kind or 'unknown' 109 | 110 | @property 111 | def type_uses(self) -> Set[str]: 112 | """ Aggregated type declaration uses. """ 113 | return self.type_uses_others.union(self.type_uses_proofs) 114 | 115 | @property 116 | def value_uses(self) -> Set[str]: 117 | """ Aggregated value declaration uses. """ 118 | return self.value_uses_others.union(self.value_uses_proofs) 119 | 120 | @property 121 | def uses(self) -> Set[str]: 122 | """ Aggregated uses. """ 123 | return self.type_uses.union(self.value_uses) 124 | 125 | def __str__(self): 126 | return str(self.name) 127 | 128 | 129 | @dataclass 130 | class LeanLib: 131 | """ A Lean library, seen as a collection of Lean declarations.""" 132 | name: str 133 | items: Dict[str, LeanDecl] = field(default_factory=dict) 134 | 135 | def __getitem__(self, key: str) -> LeanDecl: 136 | return self.items[key] 137 | 138 | def __setitem__(self, key: str, value: LeanDecl) -> None: 139 | self.items[key] = value 140 | 141 | def __delitem__(self, key: str) -> None: 142 | del self.items[key] 143 | 144 | def __contains__(self, key: str) -> bool: 145 | return key in self.items 146 | 147 | def __iter__(self): 148 | return iter(self.items.values()) 149 | 150 | def get(self, key: str, default: LeanDecl = None) -> Optional[LeanDecl]: 151 | """ Return the Lean item whose name is "key", or the default value. """ 152 | return self.items.get(key, default) 153 | 154 | @classmethod 155 | def from_yaml(cls, name: str, filename: str) -> 'LeanLib': 156 | """Create a Lean lib from a name and a Lean-exported YaML filepath.""" 157 | lib = cls(name) 158 | with open(filename, 'rb') as f: 159 | lean_output = f.read() 160 | for decl in safe_load(lean_output): 161 | lib[decl["Name"]] = LeanDecl.from_dict(decl) 162 | 163 | # Now aggregate data about inductives using constructors 164 | for decl in filter(lambda d: d.is_constructor, lib): 165 | parent = lib[decl.parent] 166 | parent.type_uses_proofs.update(decl.type_uses_proofs) 167 | parent.type_uses_others.update(decl.type_uses_others) 168 | parent.value_uses_proofs.update(decl.value_uses_proofs) 169 | parent.value_uses_others.update(decl.value_uses_others) 170 | 171 | def rself(s): 172 | """ remove self from uses.""" 173 | s.difference_update(set([decl.parent])) 174 | rself(parent.type_uses_proofs) 175 | rself(parent.type_uses_others) 176 | rself(parent.value_uses_proofs) 177 | rself(parent.value_uses_others) 178 | parent.type_size += decl.type_size 179 | parent.type_dedup_size += decl.type_dedup_size 180 | parent.type_pp_size += decl.type_pp_size 181 | parent.value_size += decl.value_size 182 | parent.value_dedup_size += decl.value_dedup_size 183 | parent.value_pp_size += decl.value_pp_size 184 | 185 | lib[parent.name] = parent 186 | return lib 187 | 188 | @staticmethod 189 | def load_dump(name: str) -> 'LeanLib': 190 | """ Create a Lean library from a pickle dump filename. """ 191 | with open(name, 'rb') as f: 192 | return pickle.load(f) 193 | 194 | def dump(self, name): 195 | """ Pickle dump a Lean library to a named file. """ 196 | with open(name, 'wb') as f: 197 | pickle.dump(self, f) 198 | 199 | def prune_foundations(self, 200 | files: Optional[List[str]] = None, 201 | prefixes: Optional[List[str]] = None): 202 | """ Remove items that are too dependant on foundations and artificially 203 | create hubs. """ 204 | forbidden_file_part = ['logic', 'classical', 'meta', 'tactic'] +\ 205 | (files or []) 206 | forbidden_prefixes = ['has_', 'set.', 'quot.', 'quotient.'] +\ 207 | (prefixes or []) 208 | remove = set() 209 | for item in self: 210 | for part in forbidden_file_part: 211 | if part in item.filename: 212 | remove.add(item.name) 213 | for prefix in forbidden_prefixes: 214 | if item.name.startswith(prefix): 215 | remove.add(item.name) 216 | for name in remove.union(set([ 217 | 'eq', 'eq.refl', 'eq.mpr', 'eq.rec', 'eq.trans', 'eq.subst', 218 | 'eq.symm', 'eq_self_iff_true', 'eq.mp', 219 | 'ne', 'not', 'true', 'false', 'trivial', 'rfl', 220 | 'congr', 'congr_arg', 'propext', 'funext', 221 | 'and', 'and.intro', 'and.elim', 222 | 'or', 'or.inl', 'or.inr', 'or.elim', 223 | 'iff', 'iff.intro', 'iff.mp', 'iff.mpr', 'iff_true_intro', 224 | 'iff_self', 'iff.refl', 'iff.rfl', 225 | 'classical.choice', 'classical.indefinite_description', 226 | 'classical.some', 'nonempty', 227 | 'decidable', 'decidable_eq', 'decidable_rel', 228 | 'imp_congr_eq', 'forall_congr_eq', 229 | 'auto_param', 230 | 'Exists', 'Exists.intro', 'subtype', 'subtype.val', 231 | 'id_rhs', 232 | 'set', 'set.has_mem', 'set_of', 233 | 'prod', 'prod.fst', 'prod.snd', 'prod.mk', 234 | 'coe', 'coe_to_lift', 'coe_base', 'coe_fn', 'coe_sort', 235 | 'coe_t', 'coe_trans', 'quotient', 'quot'])): 236 | self.items.pop(name, None) 237 | 238 | 239 | COLORS = {'theorem': {'a': 1, 'r': 9, 'b': 200, 'g': 200}, 240 | 'lemma': {'a': 1, 'r': 9, 'b': 200, 'g': 200}, 241 | 'definition': {'a': 1, 'r': 9, 'b': 236, 'g': 173}, 242 | 'structure': {'a': 1, 'r': 9, 'b': 236, 'g': 173}, 243 | 'constant': {'a': 1, 'r': 9, 'b': 236, 'g': 173}, 244 | 'axiom': {'a': 1, 'r': 9, 'b': 236, 'g': 173}, 245 | 'class': {'a': 1, 'r': 9, 'b': 236, 'g': 173}, 246 | 'inductive': {'a': 1, 'r': 9, 'b': 236, 'g': 173}, 247 | 'instance': {'a': 1, 'r': 9, 'b': 136, 'g': 253}, 248 | 'unknown': {'a': 1, 'r': 10, 'b': 10, 'g': 10}} 249 | 250 | 251 | class LeanDeclGraph(nx.DiGraph): 252 | """ A Lean declarations graph. """ 253 | @classmethod 254 | def from_lib( 255 | cls, lean: LeanLib, types_only: bool = False, **kwargs 256 | ) -> 'LeanDeclGraph': 257 | """ Creates a graph from a LeanLib object """ 258 | graph = cls(**kwargs) 259 | lib = deepcopy(lean) 260 | 261 | # Let us now drop some unwanted declarations from the lib 262 | for name, item in lean.items.items(): 263 | if (name.endswith(AUX_DEF_SUFFIX) or item.is_structure_field or 264 | item.is_constructor or item.is_recursor): 265 | lib.items.pop(name) 266 | 267 | # before adding nodes and edges 268 | for name, item in lib.items.items(): 269 | graph.add_node(name) 270 | graph.nodes[name]['id'] = item.name 271 | graph.nodes[name]['label'] = item.name 272 | graph.nodes[name]['kind'] = item.user_kind 273 | graph.nodes[name]['viz'] = {'color': COLORS[item.user_kind]} 274 | 275 | for dep in item.type_uses if types_only else item.uses: 276 | if dep in lib: 277 | graph.add_edge(dep, name) 278 | else: 279 | stripped = '.'.join(dep.split('.')[:-1]) 280 | if stripped != name and stripped in lib: 281 | graph.add_edge(stripped, name) 282 | return graph 283 | 284 | def layout(self, root): 285 | """ Slowly sets node positions using graphviz, with given root. """ 286 | for node, (x, y) in graphviz_layout(self, 'dot', root).items(): 287 | self.nodes[node]['viz']['position'] = {'x': x, 'y': y, 'z': 0} 288 | 289 | def component_of(self, key): 290 | """ The subgraph containing everything needed to define key. """ 291 | return self.subgraph(nx.ancestors(self, key).union([key])) 292 | 293 | def write(self, name: str): 294 | """ Saves declaration graph in GEXF format in a file named name. """ 295 | nx.write_gexf(self, name) 296 | 297 | 298 | def crawl(): 299 | """Command line utility""" 300 | args = sys.argv 301 | if len(args) == 1: 302 | print("You need to provide a module using dot separated file names as " 303 | "in the Lean import command.") 304 | sys.exit(1) 305 | elif len(args) == 2: 306 | mod = args[1] 307 | yaml_file = 'data.yaml' 308 | elif len(args) == 3: 309 | mod = args[1] 310 | yaml_file = args[2] 311 | else: 312 | print("Too many arguments.") 313 | sys.exit(1) 314 | 315 | with open('crawl.lean', 'w') as out: 316 | out.write(f'import {mod}\n') 317 | with (Path(__file__).parent/'crawler.lean').open() as inp: 318 | for line in inp: 319 | out.write(line.replace('data.yaml', yaml_file)) 320 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | peewee>=3.6.4 2 | networkx>=2.1 3 | pyyaml>=3.13 4 | regex>=2018.7.11 5 | -------------------------------------------------------------------------------- /requirements_tests.txt: -------------------------------------------------------------------------------- 1 | pytest 2 | pytest-cov 3 | pytest-mock 4 | flake8 5 | mypy 6 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from setuptools import setup, find_packages 2 | 3 | with open("README.md", "r") as fh: 4 | long_description = fh.read() 5 | 6 | setup( 7 | name='leancrawler', 8 | version='0.0.3', 9 | url='https://github.com/PatrickMassot/leancrawler', 10 | author='Patrick Massot', 11 | author_email='patrickmassot@free.fr', 12 | description='A Lean prover library crawler', 13 | long_description=long_description, 14 | long_description_content_type="text/markdown", 15 | packages=find_packages(), 16 | package_data={ 17 | 'leancrawler': ['crawl.lean'], 18 | }, 19 | include_package_data=True, 20 | entry_points={ 21 | "console_scripts": [ 22 | "leancrawler = leancrawler.crawler:crawl", 23 | ]}, 24 | classifiers=[ 25 | "Programming Language :: Python :: 3", 26 | "License :: OSI Approved :: Apache Software License", 27 | "Operating System :: OS Independent" ], 28 | python_requires='>=3.6', 29 | install_requires=['networkx >= 2.1', 'pyyaml >= 3.13', 'pydot >= 1.4.1']) 30 | -------------------------------------------------------------------------------- /tests/test.yaml: -------------------------------------------------------------------------------- 1 | - Name: toto.rec_on 2 | Line: 9 3 | Type: definition 4 | Uses: [toto.mk, toto] 5 | Size: 137 6 | Line: 17 7 | Type: constant 8 | Size: 4 9 | - Name: xnat.zero.sizeof_spec 10 | Line: 23 11 | Type: definition 12 | Uses: [nat.has_one, has_one.one, xnat.zero, xnat.sizeof, nat, eq] 13 | Size: 39 14 | - Name: toto.sizeof 15 | Line: 9 16 | Type: definition 17 | Uses: [nat, toto] 18 | Size: 289 19 | - Name: tata.fielda2 20 | Line: 17 21 | Type: structure_field 22 | Parent: tata.mk 23 | Uses: [tata] 24 | Size: 32 25 | - Name: tata.no_confusion_type 26 | Line: 17 27 | Type: definition 28 | Uses: [tata] 29 | Size: 322 30 | - Name: toto.mk.inj 31 | Line: 9 32 | Type: definition 33 | Uses: [and, toto.mk, toto, eq] 34 | Size: 497 35 | - Name: tata.mk.inj_eq 36 | Line: 17 37 | Type: definition 38 | Uses: [and, tata.mk, tata, eq] 39 | Size: 1279 40 | Line: 23 41 | Type: constant 42 | Size: 4 43 | - Name: xnat.zero.inj_eq 44 | Line: 23 45 | Type: definition 46 | Uses: [true, xnat.zero, xnat, eq] 47 | Size: 207 48 | - Name: xnat.zero.inj 49 | Line: 23 50 | Type: definition 51 | Uses: [true, xnat.zero, xnat, eq] 52 | Size: 53 53 | Line: 9 54 | Type: constant 55 | Size: 4 56 | - Name: xnat.ibelow 57 | Line: 23 58 | Type: definition 59 | Uses: [xnat] 60 | Size: 135 61 | - Name: toto.has_sizeof_inst 62 | Line: 9 63 | Type: instance 64 | Target: has_sizeof.{2} 65 | Uses: [toto, has_sizeof] 66 | Size: 34 67 | - Name: tata.no_confusion 68 | Line: 17 69 | Type: definition 70 | Uses: [tata.no_confusion_type, eq, tata] 71 | Size: 466 72 | - Name: xnat.no_confusion 73 | Line: 23 74 | Type: definition 75 | Uses: [xnat.no_confusion_type, eq, xnat] 76 | Size: 386 77 | Line: 23 78 | Type: constant 79 | Size: 4 80 | - Name: tata 81 | Line: 17 82 | Type: structure 83 | Fields: [fielda1, fielda2] 84 | Size: 4 85 | - Name: xnat.succ.inj 86 | Line: 23 87 | Type: definition 88 | Uses: [xnat.succ, eq, xnat] 89 | Size: 195 90 | - Name: toto.no_confusion 91 | Line: 9 92 | Type: definition 93 | Uses: [toto.no_confusion_type, eq, toto] 94 | Size: 466 95 | - Name: xnat.rec_on 96 | Line: 23 97 | Type: definition 98 | Uses: [xnat.succ, xnat.zero, xnat] 99 | Size: 137 100 | - Name: toto.fieldo2 101 | Line: 9 102 | Type: structure_field 103 | Parent: toto.mk 104 | Uses: [toto] 105 | Size: 32 106 | - Name: xnat.succ.sizeof_spec 107 | Line: 23 108 | Type: definition 109 | Uses: [xnat.has_sizeof_inst, sizeof, nat.has_one, has_one.one, nat.has_add, has_add.add, xnat.succ, xnat.sizeof, nat, eq, xnat] 110 | Size: 61 111 | - Name: xnat.sizeof 112 | Line: 23 113 | Type: definition 114 | Uses: [nat, xnat] 115 | Size: 168 116 | - Name: toto.fieldo1 117 | Line: 9 118 | Type: structure_field 119 | Parent: toto.mk 120 | Uses: [toto] 121 | Size: 32 122 | - Name: tata.sizeof 123 | Line: 17 124 | Type: definition 125 | Uses: [nat, tata] 126 | Size: 289 127 | - Name: tata.has_sizeof_inst 128 | Line: 17 129 | Type: instance 130 | Target: has_sizeof.{2} 131 | Uses: [tata, has_sizeof] 132 | Size: 34 133 | Line: 23 134 | Type: constant 135 | Size: 4 136 | - Name: xnat.cases_on 137 | Line: 23 138 | Type: definition 139 | Uses: [xnat.succ, xnat.zero, xnat] 140 | Size: 159 141 | - Name: tata.rec_on 142 | Line: 17 143 | Type: definition 144 | Uses: [tata.mk, tata] 145 | Size: 137 146 | - Name: xnat 147 | Line: 23 148 | Type: inductive 149 | Size: 4 150 | - Name: x 151 | Line: 7 152 | Type: definition 153 | Uses: [nat] 154 | Size: 70 155 | - Name: ex_toto 156 | Line: 13 157 | Type: instance 158 | Target: toto 159 | Uses: [toto] 160 | Size: 16 161 | - Name: toto 162 | Line: 9 163 | Type: class 164 | Fields: [fieldo1, fieldo2] 165 | Size: 4 166 | - Name: tata.mk.sizeof_spec 167 | Line: 17 168 | Type: definition 169 | Uses: [nat.has_one, has_one.one, tata.mk, tata.sizeof, nat, eq] 170 | Size: 96 171 | - Name: toto.mk.sizeof_spec 172 | Line: 9 173 | Type: definition 174 | Uses: [nat.has_one, has_one.one, toto.mk, toto.sizeof, nat, eq] 175 | Size: 96 176 | - Name: toto.cases_on 177 | Line: 9 178 | Type: definition 179 | Uses: [toto.mk, toto] 180 | Size: 137 181 | - Name: xnat.binduction_on 182 | Line: 23 183 | Type: definition 184 | Uses: [xnat.ibelow, xnat] 185 | Size: 519 186 | - Name: xnat.zero.inj_arrow 187 | Line: 23 188 | Type: definition 189 | Uses: [true, xnat.zero, xnat, eq] 190 | Size: 101 191 | - Name: tata.cases_on 192 | Line: 17 193 | Type: definition 194 | Uses: [tata.mk, tata] 195 | Size: 137 196 | Line: 9 197 | Type: constant 198 | Size: 4 199 | - Name: xnat.no_confusion_type 200 | Line: 23 201 | Type: definition 202 | Uses: [xnat] 203 | Size: 314 204 | - Name: two_plus_two 205 | Line: 29 206 | Type: theorem 207 | Statement uses: [nat.has_one, has_one.one, bit0, nat.has_add, has_add.add, nat, eq] 208 | Size: 0 209 | Proof uses lemmas: [trivial, of_as_true] 210 | and uses: [nat.decidable_eq, nat.has_one, has_one.one, bit0, nat.has_add, has_add.add, nat, eq] 211 | Proof size: 538 212 | - Name: one_plus_one 213 | Line: 27 214 | Type: theorem 215 | Statement uses: [bit0, nat.has_one, has_one.one, nat.has_add, has_add.add, nat, eq] 216 | Size: 0 217 | Proof uses lemmas: [trivial, of_as_true] 218 | and uses: [nat.decidable_eq, bit0, nat.has_one, has_one.one, nat.has_add, has_add.add, nat, eq] 219 | Proof size: 376 220 | - Name: xnat.succ.inj_arrow 221 | Line: 23 222 | Type: definition 223 | Uses: [xnat.succ, eq, xnat] 224 | Size: 160 225 | - Name: toto.mk.inj_arrow 226 | Line: 9 227 | Type: definition 228 | Uses: [toto.mk, toto, eq] 229 | Size: 518 230 | - Name: triv 231 | Line: 5 232 | Type: axiom 233 | Uses: [true, eq] 234 | Size: 4 235 | - Name: tata.mk.inj 236 | Line: 17 237 | Type: definition 238 | Uses: [and, tata.mk, tata, eq] 239 | Size: 497 240 | - Name: tata.fielda1 241 | Line: 17 242 | Type: structure_field 243 | Parent: tata.mk 244 | Uses: [tata] 245 | Size: 32 246 | - Name: xnat.has_sizeof_inst 247 | Line: 23 248 | Type: instance 249 | Target: has_sizeof.{1} 250 | Uses: [xnat, has_sizeof] 251 | Size: 34 252 | - Name: xnat.brec_on 253 | Line: 23 254 | Type: definition 255 | Uses: [xnat.below, xnat] 256 | Size: 834 257 | Line: 17 258 | Type: constant 259 | Size: 4 260 | - Name: toto.no_confusion_type 261 | Line: 9 262 | Type: definition 263 | Uses: [toto] 264 | Size: 322 265 | - Name: xnat.succ.inj_eq 266 | Line: 23 267 | Type: definition 268 | Uses: [xnat.succ, eq, xnat] 269 | Size: 407 270 | - Name: tata.mk.inj_arrow 271 | Line: 17 272 | Type: definition 273 | Uses: [tata.mk, tata, eq] 274 | Size: 518 275 | - Name: ex_tata 276 | Line: 21 277 | Type: definition 278 | Uses: [tata] 279 | Size: 17 280 | - Name: z 281 | Line: 3 282 | Type: constant 283 | Size: 4 284 | - Name: xnat.below 285 | Line: 23 286 | Type: definition 287 | Uses: [xnat] 288 | Size: 238 289 | -------------------------------------------------------------------------------- /tests/test_dummy.py: -------------------------------------------------------------------------------- 1 | from leancrawler import * 2 | -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | [tox] 2 | envlist = py36, py37, flake8, mypy 3 | 4 | [testenv] 5 | commands= 6 | pip install -r requirements_tests.txt 7 | pytest -v --cov-report= --cov=leancrawler 8 | deps = -rrequirements.txt 9 | 10 | [testenv:flake8] 11 | deps=flake8 12 | commands=flake8 13 | 14 | [testenv:mypy] 15 | deps=mypy 16 | commands=mypy --ignore-missing-imports --follow-imports=skip src/leancrawler 17 | --------------------------------------------------------------------------------