├── .gitignore ├── .travis.yml ├── AUTHORS ├── CHANGELOG.rst ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.rst ├── ISSUE_TEMPLATE.md ├── LICENSE ├── MANIFEST.in ├── Makefile ├── PULL_REQUEST_TEMPLATE.md ├── README.md ├── TODO.md ├── docs ├── CHANGELOG.rst ├── CONTRIBUTING.rst ├── LICENSE.rst ├── README.rst ├── conf.py ├── html │ └── index.html └── index.rst ├── halive ├── __init__.py ├── __main__.py ├── cl_parser.py └── main.py ├── img ├── halive_demo.gif └── halive_scheme.png ├── release ├── requirements-dev.txt ├── requirements.txt ├── setup.cfg ├── setup.py └── tests ├── __init__.py ├── data ├── urls.txt ├── urls2.txt └── urls3.txt ├── test_cl_parser.py └── test_sanity.py /.gitignore: -------------------------------------------------------------------------------- 1 | # Reports for some tests 2 | report_test*.txt 3 | 4 | 5 | *.py[cod] 6 | __pycache__ 7 | 8 | # Virtual Environment 9 | env/ 10 | venv/ 11 | 12 | # C extensions 13 | *.so 14 | 15 | # Vim 16 | [._]*.s[a-w][a-z] 17 | [._]s[a-w][a-z] 18 | *.un~ 19 | Session.vim 20 | 21 | # Packages 22 | *.egg 23 | *.egg-info 24 | dist 25 | build 26 | eggs 27 | parts 28 | var 29 | sdist 30 | develop-eggs 31 | .installed.cfg 32 | lib 33 | lib64 34 | 35 | # Installer logs 36 | pip-log.txt 37 | 38 | # Unit test / coverage reports 39 | .coverage 40 | .tox 41 | nosetests.xml 42 | 43 | # Mr Developer 44 | .mr.developer.cfg 45 | .project 46 | .pydevproject 47 | 48 | # GitHub token file 49 | .pypt/gh-token 50 | 51 | #  52 | .DS_Store 53 | 54 | # vim session files 55 | *.vim 56 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: python 2 | sudo: false 3 | python: 4 | - "3.3" 5 | - "3.4" 6 | - "3.5" 7 | - "3.6" 8 | - "3.6\7" 9 | - "pypy" 10 | 11 | install: 12 | - "pip install -r requirements.txt" 13 | - "pip install ." 14 | 15 | script: 16 | - "py.test --cov halive --cov-report term-missing tests/" 17 | 18 | notifications: 19 | email: 20 | on_success: change 21 | on_failure: always 22 | -------------------------------------------------------------------------------- /AUTHORS: -------------------------------------------------------------------------------- 1 | gnc 2 | -------------------------------------------------------------------------------- /CHANGELOG.rst: -------------------------------------------------------------------------------- 1 | ===================== 2 | Appendix C. Changelog 3 | ===================== 4 | :Info: This is the changelog for halive. 5 | :Author: gnc 6 | :Copyright: © 2019, gnc. 7 | :License: BSD (see /LICENSE or :doc:`Appendix B `.) 8 | :Date: 2019-08-03 9 | :Version: 0.1.0 10 | 11 | .. index:: CHANGELOG 12 | 13 | GitHub holds releases, too 14 | ========================== 15 | 16 | More information can be found on GitHub in the `releases section 17 | `_. 18 | 19 | Version History 20 | =============== 21 | 22 | 0.1.0 23 | Initial release. 24 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Contributor Covenant Code of Conduct 2 | 3 | ## Our Pledge 4 | 5 | In the interest of fostering an open and welcoming environment, we as 6 | contributors and maintainers pledge to making participation in our project and 7 | our community a harassment-free experience for everyone, regardless of age, body 8 | size, disability, ethnicity, sex characteristics, gender identity and expression, 9 | level of experience, education, socio-economic status, nationality, personal 10 | appearance, race, religion, or sexual identity and orientation. 11 | 12 | ## Our Standards 13 | 14 | Examples of behavior that contributes to creating a positive environment 15 | include: 16 | 17 | * Using welcoming and inclusive language 18 | * Being respectful of differing viewpoints and experiences 19 | * Gracefully accepting constructive criticism 20 | * Focusing on what is best for the community 21 | * Showing empathy towards other community members 22 | 23 | Examples of unacceptable behavior by participants include: 24 | 25 | * The use of sexualized language or imagery and unwelcome sexual attention or 26 | advances 27 | * Trolling, insulting/derogatory comments, and personal or political attacks 28 | * Public or private harassment 29 | * Publishing others' private information, such as a physical or electronic 30 | address, without explicit permission 31 | * Other conduct which could reasonably be considered inappropriate in a 32 | professional setting 33 | 34 | ## Our Responsibilities 35 | 36 | Project maintainers are responsible for clarifying the standards of acceptable 37 | behavior and are expected to take appropriate and fair corrective action in 38 | response to any instances of unacceptable behavior. 39 | 40 | Project maintainers have the right and responsibility to remove, edit, or 41 | reject comments, commits, code, wiki edits, issues, and other contributions 42 | that are not aligned to this Code of Conduct, or to ban temporarily or 43 | permanently any contributor for other behaviors that they deem inappropriate, 44 | threatening, offensive, or harmful. 45 | 46 | ## Scope 47 | 48 | This Code of Conduct applies both within project spaces and in public spaces 49 | when an individual is representing the project or its community. Examples of 50 | representing a project or community include using an official project e-mail 51 | address, posting via an official social media account, or acting as an appointed 52 | representative at an online or offline event. Representation of a project may be 53 | further defined and clarified by project maintainers. 54 | 55 | ## Enforcement 56 | 57 | Instances of abusive, harassing, or otherwise unacceptable behavior may be 58 | reported by contacting the project team at nebbionegiuseppe@gmail.com. All 59 | complaints will be reviewed and investigated and will result in a response that 60 | is deemed necessary and appropriate to the circumstances. The project team is 61 | obligated to maintain confidentiality with regard to the reporter of an incident. 62 | Further details of specific enforcement policies may be posted separately. 63 | 64 | Project maintainers who do not follow or enforce the Code of Conduct in good 65 | faith may face temporary or permanent repercussions as determined by other 66 | members of the project's leadership. 67 | 68 | ## Attribution 69 | 70 | This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, 71 | available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html 72 | 73 | [homepage]: https://www.contributor-covenant.org 74 | 75 | For answers to common questions about this code of conduct, see 76 | https://www.contributor-covenant.org/faq 77 | -------------------------------------------------------------------------------- /CONTRIBUTING.rst: -------------------------------------------------------------------------------- 1 | ============================== 2 | Appendix A. Contribution rules 3 | ============================== 4 | :Info: Those are the contribution rules for halive. 5 | :Copyright: © 2012-2018, Chris Warrick. 6 | :License: 3-clause BSD 7 | 8 | .. index:: contributing 9 | 10 | Do you want to contribute to this project? Great! I’d love to see some help, 11 | but you must comply with some rules. 12 | 13 | The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL 14 | NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and 15 | “OPTIONAL” in this document are to be interpreted as described in 16 | RFC 2119. 17 | 18 | --------------- 19 | Issue reporting 20 | --------------- 21 | 22 | .. index:: issues 23 | 24 | GitHub Issues are the recommended way to report an issue. If you do not have an 25 | account there, get one or mail me. 26 | 27 | When pasting console sessions, you must paste them fully, *prompt-to-prompt*, 28 | to see all the messages and your input. Trim only stuff that you are 1000% 29 | sure that is not related to the project in question. 30 | 31 | -------------------------------------------- 32 | General preparations, rules and pull process 33 | -------------------------------------------- 34 | 35 | Prepare 36 | ======= 37 | 38 | A GitHub account is recommended. Patches by mail are accepted, but I’d prefer 39 | to work via GitHub. 40 | 41 | .. _Rules: 42 | 43 | Rules 44 | ===== 45 | 46 | 1. Commits must have short, informative and logical messages. Signoffs and 47 | long messages are recommended. “Fix #xxx” is required if an issue 48 | exists. 49 | 2. The following fancy Unicode characters should be used when 50 | needed: ``— “ ” ‘ ’``. ``…`` should not appear in console output, but may 51 | appear elsewhere. 52 | 3. For Python code, use the PEP 8 coding style and PEP 257 documentation style. 53 | For other languages, K&R style applies. Braces are mandatory in all blocks 54 | (even one-line blocks). Braces are on the same lines as class names and 55 | function signatures. Use 4-space indents. 56 | 57 | Request a Pull 58 | ============== 59 | 60 | Done? Go hit the **Pull Request** button over on GitHub! And if you don’t 61 | use GitHub, ``git format-patch``. Other formats are not accepted. 62 | 63 | Your commit should be pulled up in a (longer) while. If I like it. Because 64 | some commits may be bad. So, do your best not to do those bad commits. 65 | -------------------------------------------------------------------------------- /ISSUE_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | ## Expected Behavior 2 | 3 | 4 | ## Actual Behavior 5 | 6 | 7 | ## Steps to Reproduce the Problem 8 | 9 | 1. 10 | 1. 11 | 1. 12 | 13 | ## Specifications 14 | 15 | - Version: 16 | - Platform: 17 | - Subsystem: 18 | 19 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright © 2019, gnc. 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without 5 | modification, are permitted provided that the following conditions are 6 | met: 7 | 8 | 1. Redistributions of source code must retain the above copyright 9 | notice, this list of conditions, and the following disclaimer. 10 | 11 | 2. Redistributions in binary form must reproduce the above copyright 12 | notice, this list of conditions, and the following disclaimer in the 13 | documentation and/or other materials provided with the distribution. 14 | 15 | 3. Neither the name of the author of this software nor the names of 16 | contributors to this software may be used to endorse or promote 17 | products derived from this software without specific prior written 18 | consent. 19 | 20 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 21 | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 22 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 23 | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 24 | OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 25 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 26 | LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 27 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 28 | THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 29 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 30 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 31 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | graft halive 2 | graft docs 3 | graft tests 4 | include README.rst AUTHORS LICENSE CHANGELOG.rst setup.py setup.cfg requirements.txt 5 | global-exclude __pycache__ *.pyc 6 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | .PHONY: default,dev 2 | 3 | 4 | default: 5 | python -m halive 6 | dev: 7 | vim -S Session.vim 8 | test1: 9 | python -m halive tests/data/urls.txt tests/data/urls2.txt 10 | test2: 11 | python -m halive tests/data/urls.txt tests/data/urls2.txt -t 2 -s 12 | test3: 13 | python -m halive tests/data/urls.txt tests/data/urls2.txt -t 2 -s -o report_test_3.txt 14 | test4: 15 | python -m halive tests/data/urls.txt tests/data/urls2.txt -t 2 -s --only-urls -o report_test_4.txt 16 | test5: 17 | python -m halive tests/data/urls3.txt 18 | -------------------------------------------------------------------------------- /PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | What does this implement/fix? Explain your changes. 2 | --------------------------------------------------- 3 | ... 4 | 5 | Does this close any currently open issues? 6 | ------------------------------------------ 7 | ... 8 | 9 | 10 | Any relevant logs, error output, etc? 11 | ------------------------------------- 12 | (If it’s long, please paste to https://ghostbin.com/ or https://bpaste.net and insert the link here.) 13 | 14 | Any other comments? 15 | ------------------- 16 | ... 17 | 18 | Where has this been tested? 19 | --------------------------- 20 | 21 | **Operating System:** ... 22 | 23 | **Platform:** ... 24 | 25 | **Target Platform:** ... 26 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # halive. An http/s prober, to check which URLs are alive. 2 | 3 | 4 | Author: gnc 5 | 6 | Copyright: © 2019, gnc. 7 | 8 | Date: 2019-08-03 9 | 10 | Version: 0.1.0 11 | 12 | 13 | ## PURPOSE 14 | 15 | Halive is a software used to get a list of alive hosts starting from a list of 16 | hosts whose status is unknown in a short amount of time. It's fast, indeed 17 | halive achieve its speed through its asynchronous design. 18 | 19 | Once we have a list of domains/subdomains gathered from the reconnaissance phase, 20 | (for example through [pdlist](https://github.com/gnebbia/pdlist)) we want to check 21 | fastly which subdomains are alive and which are not. 22 | To this purpose we can use halive and obtain from the initial list of hostnames 23 | only the alive subdomains. 24 | 25 | ![](img/halive_scheme.png) 26 | 27 | 28 | 29 | ## INSTALLATION 30 | 31 | We can install halive simply by doing: 32 | ```sh 33 | git clone https://github.com/gnebbia/halive 34 | cd halive 35 | pip install -r requirements.txt 36 | python setup.py install 37 | ``` 38 | 39 | 40 | ## USAGE 41 | 42 | In order to get the list of alive hosts with also the response status code we 43 | can do: 44 | ```sh 45 | halive big_list_of_urls.txt 46 | ``` 47 | 48 | The default speed is set through the use of 20 (default) workers, we can 49 | increase the speed by increasing the number of workers, by doing: 50 | ```sh 51 | halive big_list_of_urls.txt -t 100 52 | # in this case we use 100 workers 53 | ``` 54 | 55 | We can also print only URLs without response code and store results in a text 56 | file, by doing: 57 | ```sh 58 | halive big_lists_of_urls.txt -t 100 --only-urls --output report.txt 59 | ``` 60 | 61 | We can also only filter URLs which do not reply with 4XX HTTP codes, this can be 62 | done by executing halive in this way: 63 | ```sh 64 | halive big_lists_of_urls.txt -t 100 --only-success --only-urls --output report.txt 65 | ``` 66 | 67 | A usage example in the gif below: 68 | ![](img/halive_demo.gif) 69 | 70 | 71 | 72 | ## COPYRIGHT 73 | 74 | Copyright © 2019, gnc. 75 | All rights reserved. 76 | 77 | Redistribution and use in source and binary forms, with or without 78 | modification, are permitted provided that the following conditions are 79 | met: 80 | 81 | 1. Redistributions of source code must retain the above copyright 82 | notice, this list of conditions, and the following disclaimer. 83 | 84 | 2. Redistributions in binary form must reproduce the above copyright 85 | notice, this list of conditions, and the following disclaimer in the 86 | documentation and/or other materials provided with the distribution. 87 | 88 | 3. Neither the name of the author of this software nor the names of 89 | contributors to this software may be used to endorse or promote 90 | products derived from this software without specific prior written 91 | consent. 92 | 93 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 94 | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 95 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 96 | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 97 | OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 98 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 99 | LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 100 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 101 | THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 102 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 103 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 104 | -------------------------------------------------------------------------------- /TODO.md: -------------------------------------------------------------------------------- 1 | - Refactor download with a printing function 2 | - Add real-time printing by moving the printing code in the function 3 | which performs the HTTP/S request with requests 4 | -------------------------------------------------------------------------------- /docs/CHANGELOG.rst: -------------------------------------------------------------------------------- 1 | ===================== 2 | Appendix C. Changelog 3 | ===================== 4 | :Info: This is the changelog for halive. 5 | :Author: gnc 6 | :Copyright: © 2019, gnc. 7 | :License: BSD (see /LICENSE or :doc:`Appendix B `.) 8 | :Date: 2019-08-03 9 | :Version: 0.1.0 10 | 11 | .. index:: CHANGELOG 12 | 13 | GitHub holds releases, too 14 | ========================== 15 | 16 | More information can be found on GitHub in the `releases section 17 | `_. 18 | 19 | Version History 20 | =============== 21 | 22 | 0.1.0 23 | Initial release. 24 | -------------------------------------------------------------------------------- /docs/CONTRIBUTING.rst: -------------------------------------------------------------------------------- 1 | ============================== 2 | Appendix A. Contribution rules 3 | ============================== 4 | :Info: Those are the contribution rules for halive. 5 | :Copyright: © 2012-2018, Chris Warrick. 6 | :License: 3-clause BSD 7 | 8 | .. index:: contributing 9 | 10 | Do you want to contribute to this project? Great! I’d love to see some help, 11 | but you must comply with some rules. 12 | 13 | The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL 14 | NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and 15 | “OPTIONAL” in this document are to be interpreted as described in 16 | RFC 2119. 17 | 18 | --------------- 19 | Issue reporting 20 | --------------- 21 | 22 | .. index:: issues 23 | 24 | GitHub Issues are the recommended way to report an issue. If you do not have an 25 | account there, get one or mail me. 26 | 27 | When pasting console sessions, you must paste them fully, *prompt-to-prompt*, 28 | to see all the messages and your input. Trim only stuff that you are 1000% 29 | sure that is not related to the project in question. 30 | 31 | -------------------------------------------- 32 | General preparations, rules and pull process 33 | -------------------------------------------- 34 | 35 | Prepare 36 | ======= 37 | 38 | A GitHub account is recommended. Patches by mail are accepted, but I’d prefer 39 | to work via GitHub. 40 | 41 | .. _Rules: 42 | 43 | Rules 44 | ===== 45 | 46 | 1. Commits must have short, informative and logical messages. Signoffs and 47 | long messages are recommended. “Fix #xxx” is required if an issue 48 | exists. 49 | 2. The following fancy Unicode characters should be used when 50 | needed: ``— “ ” ‘ ’``. ``…`` should not appear in console output, but may 51 | appear elsewhere. 52 | 3. For Python code, use the PEP 8 coding style and PEP 257 documentation style. 53 | For other languages, K&R style applies. Braces are mandatory in all blocks 54 | (even one-line blocks). Braces are on the same lines as class names and 55 | function signatures. Use 4-space indents. 56 | 57 | Request a Pull 58 | ============== 59 | 60 | Done? Go hit the **Pull Request** button over on GitHub! And if you don’t 61 | use GitHub, ``git format-patch``. Other formats are not accepted. 62 | 63 | Your commit should be pulled up in a (longer) while. If I like it. Because 64 | some commits may be bad. So, do your best not to do those bad commits. 65 | -------------------------------------------------------------------------------- /docs/LICENSE.rst: -------------------------------------------------------------------------------- 1 | ======================================================= 2 | Appendix B. License for halive 3 | ======================================================= 4 | :Info: This is the license for halive. 5 | :Author: gnc 6 | :Copyright: © 2019, gnc. 7 | :License: BSD (see /LICENSE or :doc:`Appendix B `.) 8 | :Date: 2019-08-03 9 | :Version: 0.1.0 10 | 11 | .. index:: LICENSE 12 | 13 | Copyright © 2019, gnc. 14 | All rights reserved. 15 | 16 | Redistribution and use in source and binary forms, with or without 17 | modification, are permitted provided that the following conditions are 18 | met: 19 | 20 | 1. Redistributions of source code must retain the above copyright 21 | notice, this list of conditions, and the following disclaimer. 22 | 23 | 2. Redistributions in binary form must reproduce the above copyright 24 | notice, this list of conditions, and the following disclaimer in the 25 | documentation and/or other materials provided with the distribution. 26 | 27 | 3. Neither the name of the author of this software nor the names of 28 | contributors to this software may be used to endorse or promote 29 | products derived from this software without specific prior written 30 | consent. 31 | 32 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 33 | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 34 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 35 | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 36 | OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 37 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 38 | LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 39 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 40 | THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 41 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 42 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 43 | -------------------------------------------------------------------------------- /docs/README.rst: -------------------------------------------------------------------------------- 1 | ============================================================================== 2 | halive. An http/s prober, to check which URLs are alive. 3 | ============================================================================== 4 | :Info: This is the README file for halive. 5 | :Author: gnc 6 | :Copyright: © 2019, gnc. 7 | :Date: 2019-08-03 8 | :Version: 0.1.0 9 | 10 | .. index: README 11 | .. image:: https://travis-ci.org/gnebbia/halive.svg?branch=master 12 | :target: https://travis-ci.org/gnebbia/halive 13 | 14 | PURPOSE 15 | ------- 16 | 17 | INSTALLATION 18 | ------------ 19 | 20 | USAGE 21 | ----- 22 | 23 | NOTES 24 | ----- 25 | 26 | COPYRIGHT 27 | --------- 28 | Copyright © 2019, gnc. 29 | All rights reserved. 30 | 31 | Redistribution and use in source and binary forms, with or without 32 | modification, are permitted provided that the following conditions are 33 | met: 34 | 35 | 1. Redistributions of source code must retain the above copyright 36 | notice, this list of conditions, and the following disclaimer. 37 | 38 | 2. Redistributions in binary form must reproduce the above copyright 39 | notice, this list of conditions, and the following disclaimer in the 40 | documentation and/or other materials provided with the distribution. 41 | 42 | 3. Neither the name of the author of this software nor the names of 43 | contributors to this software may be used to endorse or promote 44 | products derived from this software without specific prior written 45 | consent. 46 | 47 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 48 | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 49 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 50 | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 51 | OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 52 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 53 | LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 54 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 55 | THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 56 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 57 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 58 | -------------------------------------------------------------------------------- /docs/conf.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # halive documentation build configuration file, created by 4 | # sphinx-quickstart on Fri Dec 14 21:02:58 2012. 5 | # 6 | # This file is execfile()d with the current directory set to its containing dir. 7 | # 8 | # Note that not all possible configuration values are present in this 9 | # autogenerated file. 10 | # 11 | # All configuration values have a default; values that are commented out 12 | # serve to show the default. 13 | 14 | import sys, os 15 | 16 | # If extensions (or modules to document with autodoc) are in another directory, 17 | # add these directories to sys.path here. If the directory is relative to the 18 | # documentation root, use os.path.abspath to make it absolute, like shown here. 19 | #sys.path.insert(0, os.path.abspath('.')) 20 | 21 | # -- General configuration ----------------------------------------------------- 22 | 23 | # If your documentation needs a minimal Sphinx version, state it here. 24 | #needs_sphinx = '1.0' 25 | 26 | # Add any Sphinx extension module names here, as strings. They can be extensions 27 | # coming with Sphinx (named 'sphinx.ext.*') or your custom ones. 28 | extensions = ['sphinx.ext.autodoc', 'sphinx.ext.doctest', 'sphinx.ext.intersphinx', 'sphinx.ext.todo', 'sphinx.ext.coverage', 'sphinx.ext.ifconfig', 'sphinx.ext.viewcode'] 29 | 30 | # Add any paths that contain templates here, relative to this directory. 31 | templates_path = ['_templates'] 32 | 33 | # The suffix of source filenames. 34 | source_suffix = '.rst' 35 | 36 | # The encoding of source files. 37 | #source_encoding = 'utf-8-sig' 38 | 39 | # The master toctree document. 40 | master_doc = 'index' 41 | 42 | # General information about the project. 43 | project = u'halive' 44 | copyright = u'2019, gnc' 45 | 46 | # The version info for the project you're documenting, acts as replacement for 47 | # |version| and |release|, also used in various other places throughout the 48 | # built documents. 49 | # 50 | # The short X.Y version. 51 | version = '0.1.0' 52 | # The full version, including alpha/beta/rc tags. 53 | release = '0.1.0' 54 | 55 | # The language for content autogenerated by Sphinx. Refer to documentation 56 | # for a list of supported languages. 57 | #language = None 58 | 59 | # There are two options for replacing |today|: either, you set today to some 60 | # non-false value, then it is used: 61 | #today = '' 62 | # Else, today_fmt is used as the format for a strftime call. 63 | #today_fmt = '%B %d, %Y' 64 | 65 | # List of patterns, relative to source directory, that match files and 66 | # directories to ignore when looking for source files. 67 | exclude_patterns = ['_build'] 68 | 69 | # The reST default role (used for this markup: `text`) to use for all documents. 70 | #default_role = None 71 | 72 | # If true, '()' will be appended to :func: etc. cross-reference text. 73 | #add_function_parentheses = True 74 | 75 | # If true, the current module name will be prepended to all description 76 | # unit titles (such as .. function::). 77 | #add_module_names = True 78 | 79 | # If true, sectionauthor and moduleauthor directives will be shown in the 80 | # output. They are ignored by default. 81 | #show_authors = False 82 | 83 | # The name of the Pygments (syntax highlighting) style to use. 84 | pygments_style = 'sphinx' 85 | 86 | # A list of ignored prefixes for module index sorting. 87 | #modindex_common_prefix = [] 88 | 89 | 90 | # -- Options for HTML output --------------------------------------------------- 91 | 92 | # The theme to use for HTML and HTML Help pages. See the documentation for 93 | # a list of builtin themes. 94 | html_theme = 'default' 95 | 96 | # Theme options are theme-specific and customize the look and feel of a theme 97 | # further. For a list of options available for each theme, see the 98 | # documentation. 99 | #html_theme_options = {} 100 | 101 | # Add any paths that contain custom themes here, relative to this directory. 102 | #html_theme_path = [] 103 | 104 | # The name for this set of Sphinx documents. If None, it defaults to 105 | # " v documentation". 106 | #html_title = None 107 | 108 | # A shorter title for the navigation bar. Default is the same as html_title. 109 | #html_short_title = None 110 | 111 | # The name of an image file (relative to this directory) to place at the top 112 | # of the sidebar. 113 | #html_logo = None 114 | 115 | # The name of an image file (within the static path) to use as favicon of the 116 | # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 117 | # pixels large. 118 | #html_favicon = None 119 | 120 | # Add any paths that contain custom static files (such as style sheets) here, 121 | # relative to this directory. They are copied after the builtin static files, 122 | # so a file named "default.css" will overwrite the builtin "default.css". 123 | html_static_path = ['_static'] 124 | 125 | # If not '', a 'Last updated on:' timestamp is inserted at every page bottom, 126 | # using the given strftime format. 127 | #html_last_updated_fmt = '%b %d, %Y' 128 | 129 | # If true, SmartyPants will be used to convert quotes and dashes to 130 | # typographically correct entities. 131 | #html_use_smartypants = True 132 | 133 | # Custom sidebar templates, maps document names to template names. 134 | #html_sidebars = {} 135 | 136 | # Additional templates that should be rendered to pages, maps page names to 137 | # template names. 138 | #html_additional_pages = {} 139 | 140 | # If false, no module index is generated. 141 | #html_domain_indices = True 142 | 143 | # If false, no index is generated. 144 | #html_use_index = True 145 | 146 | # If true, the index is split into individual pages for each letter. 147 | #html_split_index = False 148 | 149 | # If true, links to the reST sources are added to the pages. 150 | #html_show_sourcelink = True 151 | 152 | # If true, "Created using Sphinx" is shown in the HTML footer. Default is True. 153 | #html_show_sphinx = True 154 | 155 | # If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. 156 | #html_show_copyright = True 157 | 158 | # If true, an OpenSearch description file will be output, and all pages will 159 | # contain a tag referring to it. The value of this option must be the 160 | # base URL from which the finished HTML is served. 161 | #html_use_opensearch = '' 162 | 163 | # This is the file name suffix for HTML files (e.g. ".xhtml"). 164 | #html_file_suffix = None 165 | 166 | # Output file base name for HTML help builder. 167 | htmlhelp_basename = 'halivedoc' 168 | 169 | 170 | # -- Options for LaTeX output -------------------------------------------------- 171 | 172 | # The paper size ('letter' or 'a4'). 173 | #latex_paper_size = 'letter' 174 | 175 | # The font size ('10pt', '11pt' or '12pt'). 176 | #latex_font_size = '10pt' 177 | 178 | # Grouping the document tree into LaTeX files. List of tuples 179 | # (source start file, target name, title, author, documentclass [howto/manual]). 180 | latex_documents = [ 181 | ('index', 'halive.tex', u'halive Documentation', 182 | u'gnc', 'manual'), 183 | ] 184 | 185 | latex_elements = {'papersize': 'a4paper', 'fontpkg': '\\usepackage{tgheros}', 186 | 'fncychap': '\\usepackage[Sonny]{fncychap}'} 187 | 188 | # The name of an image file (relative to this directory) to place at the top of 189 | # the title page. 190 | #latex_logo = None 191 | 192 | # For "manual" documents, if this is true, then toplevel headings are parts, 193 | # not chapters. 194 | #latex_use_parts = False 195 | 196 | # If true, show page references after internal links. 197 | #latex_show_pagerefs = False 198 | 199 | # If true, show URL addresses after external links. 200 | #latex_show_urls = False 201 | 202 | # Additional stuff for the LaTeX preamble. 203 | #latex_preamble = '' 204 | 205 | # Documents to append as an appendix to all manuals. 206 | #latex_appendices = [] 207 | 208 | # If false, no module index is generated. 209 | #latex_domain_indices = True 210 | 211 | 212 | # -- Options for manual page output -------------------------------------------- 213 | 214 | # One entry per manual page. List of tuples 215 | # (source start file, name, description, authors, manual section). 216 | man_pages = [ 217 | ('index', 'halive', u'halive Documentation', 218 | [u'gnc'], 1) 219 | ] 220 | 221 | 222 | # Example configuration for intersphinx: refer to the Python standard library. 223 | intersphinx_mapping = {'http://docs.python.org/': None} 224 | -------------------------------------------------------------------------------- /docs/html/index.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | Redirecting to Read The Docs 4 | 5 | The docs are at halive.readthedocs.org. 6 | You will be redirected there in a while. If not, click the link above. 7 | -------------------------------------------------------------------------------- /docs/index.rst: -------------------------------------------------------------------------------- 1 | =============================== 2 | halive 3 | =============================== 4 | 5 | .. toctree:: 6 | :maxdepth: 2 7 | 8 | README for halive 9 | CONTRIBUTING 10 | LICENSE 11 | CHANGELOG 12 | 13 | Indices and tables 14 | ================== 15 | 16 | * :ref:`genindex` 17 | * :ref:`modindex` 18 | * :ref:`search` 19 | -------------------------------------------------------------------------------- /halive/__init__.py: -------------------------------------------------------------------------------- 1 | # -*- encoding: utf-8 -*- 2 | # halive v0.1.0 3 | # An http/s prober, to check which URLs are alive. 4 | # Copyright © 2019, gnc. 5 | # All rights reserved. 6 | # 7 | # Redistribution and use in source and binary forms, with or without 8 | # modification, are permitted provided that the following conditions are 9 | # met: 10 | # 11 | # 1. Redistributions of source code must retain the above copyright 12 | # notice, this list of conditions, and the following disclaimer. 13 | # 14 | # 2. Redistributions in binary form must reproduce the above copyright 15 | # notice, this list of conditions, and the following disclaimer in the 16 | # documentation and/or other materials provided with the distribution. 17 | # 18 | # 3. Neither the name of the author of this software nor the names of 19 | # contributors to this software may be used to endorse or promote 20 | # products derived from this software without specific prior written 21 | # consent. 22 | # 23 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 24 | # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 25 | # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 26 | # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 27 | # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 28 | # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 29 | # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 30 | # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 31 | # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 32 | # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 33 | # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 34 | 35 | """ 36 | An http/s prober, to check which URLs are alive. 37 | 38 | :Copyright: © 2019, gnc. 39 | :License: BSD (see /LICENSE). 40 | """ 41 | 42 | __title__ = 'halive' 43 | __version__ = '0.1.0' 44 | __author__ = 'gnc' 45 | __license__ = '3-clause BSD' 46 | __docformat__ = 'restructuredtext en' 47 | 48 | __all__ = () 49 | 50 | # import gettext 51 | # G = gettext.translation('halive', '/usr/share/locale', fallback='C') 52 | # _ = G.gettext 53 | -------------------------------------------------------------------------------- /halive/__main__.py: -------------------------------------------------------------------------------- 1 | # -*- encoding: utf-8 -*- 2 | # halive v0.1.0 3 | # An http/s prober, to check which URLs are alive. 4 | # Copyright © 2019, gnc. 5 | # See /LICENSE for licensing information. 6 | 7 | """ 8 | Main routine of halive. 9 | 10 | :Copyright: © 2019, gnc. 11 | :License: BSD (see /LICENSE). 12 | """ 13 | from halive.main import main 14 | 15 | 16 | __all__ = ('main',) 17 | 18 | 19 | 20 | if __name__ == '__main__': 21 | main() 22 | -------------------------------------------------------------------------------- /halive/cl_parser.py: -------------------------------------------------------------------------------- 1 | # -*- encoding: utf-8 -*- 2 | # halive v0.1.0 3 | # An http/s prober, to check which URLs are alive. 4 | # Copyright © 2019, gnc. 5 | # See /LICENSE for licensing information. 6 | 7 | """ 8 | Command Line Parsing Module for halive 9 | 10 | :Copyright: © 2019, gnc. 11 | :License: BSD (see /LICENSE). 12 | """ 13 | 14 | import argparse 15 | 16 | 17 | def parse_args(args): 18 | """ 19 | This function parses the arguments which have been passed from the command 20 | line, these can be easily retrieved for example by using "sys.argv[1:]". 21 | It returns a parser object as with argparse. 22 | 23 | Arguments: 24 | args -- the list of arguments passed from the command line as the sys.argv 25 | format 26 | 27 | Returns: a parser with the provided arguments, which can be used in a 28 | simpler format 29 | """ 30 | parser = argparse.ArgumentParser( 31 | prog='halive', 32 | description='An http/s prober, to check which URLs are alive.') 33 | 34 | parser.add_argument( 35 | "inputfiles", 36 | help="input file with one url per line", 37 | type=argparse.FileType('r'), 38 | nargs='+', 39 | ) 40 | parser.add_argument( 41 | "-o", "--output", 42 | dest='outputfile', 43 | help="save results to the specified file", 44 | default=None, 45 | nargs='?', 46 | type=argparse.FileType('w'), 47 | ) 48 | parser.add_argument( 49 | "-t", "--concurrency", 50 | help="number of concurrent http requests", 51 | default=20, 52 | type=int, 53 | ) 54 | parser.add_argument( 55 | "-s", "--only-success", 56 | help="show only responses which are not 4XX errors", 57 | action='store_true', 58 | default=False, 59 | ) 60 | parser.add_argument( 61 | "-u", "--only-urls", 62 | help="show only active URLs, without response status codes", 63 | action='store_true', 64 | default=False, 65 | ) 66 | 67 | return parser.parse_args(args) 68 | -------------------------------------------------------------------------------- /halive/main.py: -------------------------------------------------------------------------------- 1 | # -*- encoding: utf-8 -*- 2 | # halive v0.1.0 3 | # An http/s prober, to check which URLs are alive. 4 | # Copyright © 2019, gnc. 5 | # See /LICENSE for licensing information. 6 | 7 | """ 8 | Main module for halive. 9 | 10 | :Copyright: © 2019, gnc. 11 | :License: BSD (see /LICENSE). 12 | """ 13 | 14 | import re 15 | import sys 16 | import asyncio 17 | import concurrent.futures 18 | import requests 19 | from halive.cl_parser import parse_args 20 | 21 | 22 | def show_banner(): 23 | """ 24 | Shows the banner of Halive 25 | """ 26 | print(""" 27 | _ _ _ _ _____ _______ 28 | | | | | / \ | | |_ _\ \ / / ____| 29 | | |_| | / _ \ | | | | \ \ / /| _| 30 | | _ |/ ___ \| |___ | | \ V / | |___ 31 | |_| |_/_/ \_\_____|___| \_/ |_____| 32 | 33 | 34 | A super fast asynchronous http and https prober, to check who is (h)alive. 35 | Developed by gnc 36 | """) 37 | 38 | 39 | def get_urls(inputfiles): 40 | """ 41 | This function takes as input the list of files containing the hostnames 42 | and normalizes the format of the hostnames in order to be able to perform 43 | valid HTTP/HTTPS requests. 44 | 45 | Args: 46 | inputfiles -- list of inputfiles 47 | 48 | Returns: 49 | urls -- list of normalized URLs which can be queries 50 | """ 51 | urls = [] 52 | scheme_rgx = re.compile(r'^https?://') 53 | for ifile in inputfiles: 54 | urls.append(ifile.read().splitlines()) 55 | urls = set([n for l in urls for n in l]) 56 | urls = list(filter(None, urls)) 57 | for i in range(len(urls)): 58 | if not scheme_rgx.match(urls[i]): 59 | urls[i] = 'http://' + urls[i] 60 | return urls 61 | 62 | 63 | async def download(urls, num_workers, show_only_success, outputfile, only_urls): 64 | """ 65 | This function is responsible for performing the asynchrounous requests to 66 | the list of URLs. 67 | 68 | Args: 69 | urls -- list of URLs to query 70 | num_workers -- this is an integer determining the degree of concurrency 71 | show_only_success -- this is a boolean indicating if only the success 72 | responses (not 4XX) should be shown 73 | outputfile -- the name of the output file where we want to save the 74 | list of valid URLs 75 | only_urls -- this is a boolean indicating if we want to ouput only 76 | URLs without any response code 77 | (this happens when it is set to True) 78 | """ 79 | outputfiledata = [] 80 | with concurrent.futures.ThreadPoolExecutor(max_workers=num_workers) as executor: 81 | loop = asyncio.get_event_loop() 82 | futures = [] 83 | response = [] 84 | for url in urls: 85 | futures.append(loop.run_in_executor(executor, make_request, url)) 86 | 87 | for response in await asyncio.gather(*futures): 88 | outputfiledata.append(response) 89 | if not response['status'] == -1: 90 | if show_only_success: 91 | if response['status'] < 400 or response['status'] >= 500: 92 | if only_urls: 93 | print(response['url']) 94 | else: 95 | print( 96 | '{:70.70} {}'.format( 97 | response['url'], 98 | response['status'])) 99 | else: 100 | if only_urls: 101 | print(response['url']) 102 | else: 103 | print( 104 | '{:70.70} {}'.format( 105 | response['url'], 106 | response['status'])) 107 | if outputfile: 108 | for host in outputfiledata: 109 | if not host['status'] == -1: 110 | if only_urls: 111 | outputfile.write('{}\n'.format(host['url'])) 112 | else: 113 | outputfile.write( 114 | '{},{}\n'.format( 115 | host['url'], host['status'])) 116 | 117 | 118 | def make_request(url): 119 | """ 120 | This is an internal utility function which is responsible for performing 121 | te request, it saves the results in a dictionary. 122 | """ 123 | response = {} 124 | try: 125 | resp = requests.head(url, allow_redirects=False, timeout=1) 126 | response['url'] = resp.url 127 | response['status'] = resp.status_code 128 | except BaseException: 129 | response['url'] = url 130 | response['status'] = -1 131 | 132 | return response 133 | 134 | 135 | def main(): 136 | """Main routine of halive.""" 137 | show_banner() 138 | args = parse_args(sys.argv[1:]) 139 | urls = get_urls(args.inputfiles) 140 | if args.only_urls: 141 | print("URL") 142 | else: 143 | print('{:70.70} {}'.format("URL", "Response")) 144 | 145 | loop = asyncio.get_event_loop() 146 | loop.run_until_complete(download(urls, 147 | args.concurrency, 148 | args.only_success, 149 | args.outputfile, 150 | args.only_urls)) 151 | -------------------------------------------------------------------------------- /img/halive_demo.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gnebbia/halive/f449693084526806b36761ca8c5afea0e9f3c64e/img/halive_demo.gif -------------------------------------------------------------------------------- /img/halive_scheme.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gnebbia/halive/f449693084526806b36761ca8c5afea0e9f3c64e/img/halive_scheme.png -------------------------------------------------------------------------------- /release: -------------------------------------------------------------------------------- 1 | #!/bin/zsh 2 | # The Release Script 3 | # Part of the Python Project Template. 4 | # Copyright © 2013-2018, Chris Warrick. 5 | # All rights reserved. 6 | # 7 | # Redistribution and use in source and binary forms, with or without 8 | # modification, are permitted provided that the following conditions are 9 | # met: 10 | # 11 | # 1. Redistributions of source code must retain the above copyright 12 | # notice, this list of conditions, and the following disclaimer. 13 | # 14 | # 2. Redistributions in binary form must reproduce the above copyright 15 | # notice, this list of conditions, and the following disclaimer in the 16 | # documentation and/or other materials provided with the distribution. 17 | # 18 | # 3. Neither the name of the author of this software nor the names of 19 | # contributors to this software may be used to endorse or promote 20 | # products derived from this software without specific prior written 21 | # consent. 22 | # 23 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 24 | # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 25 | # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 26 | # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 27 | # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 28 | # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 29 | # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 30 | # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 31 | # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 32 | # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 33 | # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 34 | 35 | . .pypt/config 36 | function status { 37 | echo $@ 38 | } 39 | 40 | function warning { 41 | echo 'WARNING: '$@ 42 | } 43 | 44 | function error { 45 | echo 'ERROR: '$@ 46 | } 47 | 48 | function cleanup { 49 | rm -rf $PROJECTLC.egg-info build || true 50 | rm -rf **/__pycache__ || true 51 | } 52 | 53 | function cleanup_cmfn { 54 | [[ -e $cmfn ]] && rm $cmfn || true 55 | [[ -e $cmfn2 ]] && rm $cmfn2 || true 56 | } 57 | 58 | status '*** Chris Warrick’s Release Scripts (PyPT)' 59 | 60 | echo -n "Version number: " 61 | read version 62 | 63 | echo $version | grep '^[0-9]\{1,\}\.[0-9]\{1,\}\.[0-9]\{1,\}$' > /dev/null 64 | 65 | if [[ $? != 0 ]]; then 66 | echo $version | grep '^[0-9]\{1,\}\.[0-9]\{1,\}\.[0-9]\{1,\}\-[0-9A-Za-z-]\{1,\}$' > /dev/null 67 | if [[ $? != 0 ]]; then 68 | warning 'version number is not compliant with versioning scheme (Semantic Versioning 2.0)' 69 | echo -n 'Continue? [y/N] ' 70 | read vercont 71 | if [[ $vercont == 'y' || $vercont == 'Y' ]]; then 72 | echo 'Continuing.' 73 | else 74 | exit 2 75 | fi 76 | else 77 | status 'NOTICE: pre-release version number in use.' 78 | echo -n 'Continue? [Y/n] ' 79 | read vercont 80 | if [[ $vercont == 'n' || $vercont == 'N' ]]; then 81 | exit 2 82 | else 83 | echo 'Continuing.' 84 | fi 85 | fi 86 | fi 87 | 88 | # Creating all the dates at the exact same time. 89 | date=$(date '+%Y-%m-%d') 90 | datel=$(date '+%Y-%m-%d %H:%M%z') 91 | #datep=$(date '+%Y%m%d') 92 | dates=$(date '+%s') 93 | 94 | cmfn=$PWD/.git/kwrs-$dates 95 | cmfn2=$cmfn"-commit" 96 | 97 | cleanup 98 | cleanup_cmfn 99 | git add -A --ignore-errors . 100 | 101 | cat > $cmfn <> $cmfn 121 | 122 | if [[ "$EDITOR" == 'vim' || "$EDITOR" == '/usr/bin/vim' ]]; then 123 | $EDITOR -c 'set filetype=gitcommit' $cmfn 124 | elif [[ $EDITOR == '' ]]; then 125 | vim -c 'set filetype=gitcommit' $cmfn 126 | else 127 | $EDITOR $cmfn 128 | fi 129 | 130 | .pypt/commitlog $cmfn $PWD $version 131 | 132 | status 'Replacing versions and dates in files...' 133 | sed "s/version=.*/version='$version',/g" setup.py -i 134 | sed "s/version = .*/version = '$version'/g" docs/conf.py -i 135 | sed "s/release = .*/release = '$version'/g" docs/conf.py -i 136 | sed "s/:Version: .*/:Version: $version/g" docs/**/*.rst -i 137 | sed "s/# $PROJECT v.*/# $PROJECT v$version/" $PROJECTLC/**/*.py -i 138 | sed "s/__version__ = .*/__version__ = '$version'/g" $PROJECTLC/__init__.py -i 139 | sed "s/:Date: .*/:Date: $date/g" docs/*.rst -i 140 | 141 | [[ -e "PKGBUILD" ]] && sed "s/pkgver=.*/pkgver=$version/g" PKGBUILD -i || true 142 | [[ -e "PKGBUILD-git" ]] && sed "s/pkgver=.*/pkgver=$version/g" PKGBUILD-git -i || true 143 | 144 | cp docs/README.rst docs/CHANGELOG.rst docs/CONTRIBUTING.rst . 145 | cp docs/README.rst README 146 | 147 | status 'Generating locales...' 148 | ./.pypt/localegen 149 | 150 | status 'Importing...' 151 | python -c "import $PROJECTLC" 152 | if [[ $? != 0 ]]; then 153 | error "Import failed. Fix your code or don't come back." 154 | exit 1 155 | fi 156 | 157 | if [[ -e tests ]]; then 158 | status 'Running tests...' 159 | pytest tests/ 160 | if [[ $? != 0 ]]; then 161 | error "Tests failed. Fix your code or don't come back." 162 | exit 1 163 | fi 164 | fi 165 | 166 | status 'Running pre-sdist.hook...' 167 | 168 | . .pypt/hooks/pre-sdist.hook 169 | 170 | status 'This is the last chance to quit. Hit ^C now if you want to.' 171 | read bailout 172 | 173 | ./setup.py sdist bdist_wheel || exit $? 174 | twine upload -s dist/$PROJECTLC-$version.tar.gz dist/$PROJECTLC-$version*.whl || exit $? 175 | 176 | if [[ -e PKGBUILD ]]; then 177 | status 'Updating AUR PKGBUILDs...' 178 | md5out=$(md5sum 'dist/'$PROJECTLC'-'$version'.tar.gz'|awk '{print $1}') 179 | sed "s/md5sums=.*/md5sums=('$md5out')/" PKGBUILD -i 180 | fi 181 | 182 | cleanup 183 | 184 | git add -A --ignore-errors . || exit $? 185 | 186 | git commit -S -asF $cmfn2 || exit $? 187 | git tag -sm "Version $version" v$version || exit $? 188 | git push --follow-tags origin master || exit $? 189 | 190 | .pypt/ghrel $cmfn $PWD $GITUSER/$GITREPO v$version 191 | 192 | cleanup_cmfn 193 | 194 | status "Done!" 195 | . .pypt/hooks/post-release.hook 196 | -------------------------------------------------------------------------------- /requirements-dev.txt: -------------------------------------------------------------------------------- 1 | requests 2 | twine 3 | pytest 4 | coverage 5 | pytest-cov 6 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | requests 2 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [flake8] 2 | ignore = E741 3 | 4 | [wheel] 5 | universal = 1 6 | 7 | [tool:pytest] 8 | norecursedirs = .git 9 | addopts = --cov=halive --cov-report term-missing 10 | 11 | [coverage:run] 12 | branch = True 13 | omit = tests/* 14 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- encoding: utf-8 -*- 3 | import io 4 | from setuptools import setup, find_packages 5 | 6 | 7 | setup(name='halive', 8 | version='0.1.0', 9 | description='An http/s prober, to check which URLs are alive.', 10 | keywords='halive', 11 | author='gnc', 12 | author_email='nebbionegiuseppe@gmail.com', 13 | url='https://github.com/gnebbia/halive', 14 | license='3-clause BSD', 15 | long_description=io.open( 16 | './docs/README.rst', 'r', encoding='utf-8').read(), 17 | platforms='any', 18 | zip_safe=False, 19 | # http://pypi.python.org/pypi?%3Aaction=list_classifiers 20 | classifiers=['Development Status :: 1 - Planning', 21 | 'Programming Language :: Python', 22 | 'Programming Language :: Python :: 3', 23 | 'Programming Language :: Python :: 3.3', 24 | 'Programming Language :: Python :: 3.4', 25 | 'Programming Language :: Python :: 3.5', 26 | 'Programming Language :: Python :: 3.6', 27 | 'Programming Language :: Python :: 3.7', 28 | ], 29 | packages=find_packages(exclude=('tests',)), 30 | include_package_data=True, 31 | install_requires=[], 32 | entry_points={ 33 | 'console_scripts':[ 34 | 'halive = halive.main:main', 35 | ] 36 | }, 37 | ) 38 | -------------------------------------------------------------------------------- /tests/__init__.py: -------------------------------------------------------------------------------- 1 | # -*- encoding: utf-8 -*- 2 | # halive test suite 3 | # Copyright © 2019, gnc. 4 | # All rights reserved. 5 | # 6 | # Redistribution and use in source and binary forms, with or without 7 | # modification, are permitted provided that the following conditions are 8 | # met: 9 | # 10 | # 1. Redistributions of source code must retain the above copyright 11 | # notice, this list of conditions, and the following disclaimer. 12 | # 13 | # 2. Redistributions in binary form must reproduce the above copyright 14 | # notice, this list of conditions, and the following disclaimer in the 15 | # documentation and/or other materials provided with the distribution. 16 | # 17 | # 3. Neither the name of the author of this software nor the names of 18 | # contributors to this software may be used to endorse or promote 19 | # products derived from this software without specific prior written 20 | # consent. 21 | # 22 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 23 | # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 24 | # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 25 | # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 26 | # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 27 | # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 28 | # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 29 | # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 30 | # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 31 | # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 32 | # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 33 | -------------------------------------------------------------------------------- /tests/data/urls.txt: -------------------------------------------------------------------------------- 1 | example.com 2 | pelabinfosec.ml 3 | github.com/gnebbia/pdlist 4 | github.com/gnebbia/awdawdwdawd 5 | https://github.com 6 | nasa.gov 7 | -------------------------------------------------------------------------------- /tests/data/urls2.txt: -------------------------------------------------------------------------------- 1 | example.com 2 | pelabinfosec.ml 3 | http://ciao.it/adwakdniand 4 | https://github.com/gnebbia/pdlist 5 | github.com/gnebbia/lallora 6 | ciao.it 7 | cvedetails.com 8 | aowdawodnawidwaddwaid.yxia 9 | -------------------------------------------------------------------------------- /tests/data/urls3.txt: -------------------------------------------------------------------------------- 1 | https://github.com 2 | http://ciao.it 3 | https://example.com 4 | -------------------------------------------------------------------------------- /tests/test_cl_parser.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- encoding: utf-8 -*- 3 | # halive test suite 4 | # Copyright © 2019, gnc. 5 | # See /LICENSE for licensing information. 6 | 7 | 8 | import halive 9 | from halive.cl_parser import parse_args 10 | 11 | 12 | def test_parse_args_one_file(files=None): 13 | # Test single file as input 14 | if files is None: 15 | args = parse_args(['tests/data/urls.txt']) 16 | else: 17 | args = parse_args(files) 18 | assert args.inputfiles 19 | urls = [] 20 | for f in args.inputfiles: 21 | lines = f.read().splitlines() 22 | print(lines) 23 | urls.append(lines) 24 | urls = [n for l in urls for n in l] 25 | 26 | 27 | def test_parse_args_two_files(files=None): 28 | # Test multiple files as input 29 | if args is None: 30 | args = parse_args(['tests/data/urls.txt','tests/data/urls2.txt']) 31 | assert len(args.inputfiles) == 2 32 | else: 33 | args = parse_args(files) 34 | urls = [] 35 | for f in args.inputfiles: 36 | lines = f.read().splitlines() 37 | print(lines) 38 | urls.append(lines) 39 | urls = [n for l in urls for n in l] 40 | 41 | 42 | 43 | 44 | def test_import(): 45 | """Test imports.""" 46 | halive 47 | -------------------------------------------------------------------------------- /tests/test_sanity.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- encoding: utf-8 -*- 3 | # halive test suite 4 | # Copyright © 2019, gnc. 5 | # See /LICENSE for licensing information. 6 | 7 | 8 | import halive 9 | import halive.cl_parser 10 | 11 | 12 | def test_true(): 13 | """Test if True is truthy.""" 14 | assert True 15 | 16 | 17 | def test_false(): 18 | """Test if False is falsey.""" 19 | assert not False 20 | 21 | 22 | def test_trueexpr(): 23 | """Test if this evaluates as True.""" 24 | assert 1 == 1 25 | 26 | 27 | def test_falseexpr(): 28 | """Test if this evaluates as False.""" 29 | assert 1 != 2 30 | 31 | 32 | def test_math(): 33 | """Test basic arithmetic skills of Python.""" 34 | assert 2 + 2 == 4 35 | assert 2 - 2 == 0 36 | assert 2 * 2 == 4 37 | assert 2 / 2 == 1 38 | assert 3 % 2 == 1 39 | 40 | 41 | def test_bitwise(): 42 | """Test bitwise operators.""" 43 | assert 0b11 ^ 0b10 == 0b01 44 | assert 0b100 | 0b010 == 0b110 45 | assert 0b101 & 0b011 == 0b001 46 | assert 0b10 << 2 == 0b1000 47 | assert 0b1111 >> 2 == 0b11 48 | 49 | 50 | def test_import(): 51 | """Test imports.""" 52 | halive 53 | halive.cl_parser 54 | --------------------------------------------------------------------------------