├── .gitignore ├── README.md ├── reduce_ram.ipynb └── LICENSE /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | .eggs/ 17 | lib/ 18 | lib64/ 19 | parts/ 20 | sdist/ 21 | var/ 22 | wheels/ 23 | pip-wheel-metadata/ 24 | share/python-wheels/ 25 | *.egg-info/ 26 | .installed.cfg 27 | *.egg 28 | MANIFEST 29 | 30 | # PyInstaller 31 | # Usually these files are written by a python script from a template 32 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 33 | *.manifest 34 | *.spec 35 | 36 | # Installer logs 37 | pip-log.txt 38 | pip-delete-this-directory.txt 39 | 40 | # Unit test / coverage reports 41 | htmlcov/ 42 | .tox/ 43 | .nox/ 44 | .coverage 45 | .coverage.* 46 | .cache 47 | nosetests.xml 48 | coverage.xml 49 | *.cover 50 | *.py,cover 51 | .hypothesis/ 52 | .pytest_cache/ 53 | 54 | # Translations 55 | *.mo 56 | *.pot 57 | 58 | # Django stuff: 59 | *.log 60 | local_settings.py 61 | db.sqlite3 62 | db.sqlite3-journal 63 | 64 | # Flask stuff: 65 | instance/ 66 | .webassets-cache 67 | 68 | # Scrapy stuff: 69 | .scrapy 70 | 71 | # Sphinx documentation 72 | docs/_build/ 73 | 74 | # PyBuilder 75 | target/ 76 | 77 | # Jupyter Notebook 78 | .ipynb_checkpoints 79 | 80 | # IPython 81 | profile_default/ 82 | ipython_config.py 83 | 84 | # pyenv 85 | .python-version 86 | 87 | # pipenv 88 | # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. 89 | # However, in case of collaboration, if having platform-specific dependencies or dependencies 90 | # having no cross-platform support, pipenv may install dependencies that don't work, or not 91 | # install all needed dependencies. 92 | #Pipfile.lock 93 | 94 | # PEP 582; used by e.g. github.com/David-OConnor/pyflow 95 | __pypackages__/ 96 | 97 | # Celery stuff 98 | celerybeat-schedule 99 | celerybeat.pid 100 | 101 | # SageMath parsed files 102 | *.sage.py 103 | 104 | # Environments 105 | .env 106 | .venv 107 | env/ 108 | venv/ 109 | ENV/ 110 | env.bak/ 111 | venv.bak/ 112 | 113 | # Spyder project settings 114 | .spyderproject 115 | .spyproject 116 | 117 | # Rope project settings 118 | .ropeproject 119 | 120 | # mkdocs documentation 121 | /site 122 | 123 | # mypy 124 | .mypy_cache/ 125 | .dmypy.json 126 | dmypy.json 127 | 128 | # Pyre type checker 129 | .pyre/ 130 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # CLI-Bootcamp 2 | Ideas on how to quickly learn to build command-line tools 3 | 4 | 5 | ## Part 1-Bash 6 | 7 | 8 | ###### Week1: Using Linux 9 | 10 | * [Lesson 1: Using Linux Shell Lab](https://github.com/noahgift/Coursera-DE-C2-Using-Linux) 11 | * [Lesson 2: How shell piping works](https://github.com/noahgift/Coursera-DE-C2-Shell-Piping) 12 | * [Lesson 3: Using SSH](https://github.com/noahgift/ssh-tips-tricks) 13 | 14 | ###### Week2: Using Bash 15 | 16 | * [Lesson 1: Create and Use .bashrc](https://github.com/noahgift/Coursera-DE-C2-configure-shell) 17 | * [Lesson 2: Sourcing shell variables from a script](https://github.com/noahgift/Coursera-DE-C2-shell-variables) 18 | * [Lesson3: Using stdout and stdin](https://github.com/noahgift/Coursera-DE-C2-Standard-Streams) 19 | 20 | ###### Week3: Building Bash Scripts 21 | 22 | * [Lesson 1: Build a for loop in Bash](https://github.com/noahgift/Coursera-DE-C2-Use-Shell-Logic-and-Control-Flow) 23 | * [Lesson 2: Truncate large files with Bash](https://github.com/noahgift/coursera-de-c2-truncate-file) 24 | * [Lesson 3: Building a command-line tool for data processing](https://github.com/noahgift/Coursera-DE-C2-bash-cli-reverse-string) 25 | * [Lesson 4: Build Bash CLI with options ](https://github.com/noahgift/Coursera-DE-C2-Lab3-Building-Bash-Scripts.git) 26 | 27 | ###### Week4: Composing File and Data Management Solutions with Linux 28 | 29 | * [Lesson 1: Understand the search commands](https://github.com/noahgift/Coursera-DE-C2-search-commands) 30 | * [Lesson 2: Setting permissions](https://github.com/noahgift/Coursera-DE-C2-Files-Directories-Permissions) 31 | * [Lesson 3: Using regex to process text from file](https://github.com/noahgift/Coursera-DE-C2-using-regex-search) 32 | * [Lesson 4: Search the filesystem with find](https://github.com/noahgift/Coursera-DE-C2-Lab4-Composing-File-Data-Solutions) 33 | 34 | ## Part 2-Build CLI in Python 35 | 36 | * [python-click-cli-cookbook](https://github.com/noahgift/python-click-cli-cookbook) 37 | 38 | ## Challenges 39 | 40 | ### Bash and ZSH Challenges 41 | * Customize your `~/.bashrc` with at least one alias, function and variable. 42 | * Build a Bash CLI tool that takes options 43 | * Truncate a large file and randomly sample at the same time 44 | * Write a find command and a locate command 45 | * Install [ohmzsh](https://ohmyz.sh), what did you learn? 46 | 47 | ### Python Challenges 48 | * Pick a Python command-line tool library you are not familar with and build a module with a function that is called in a CLI. A few examples: 49 | * [python-fire](https://github.com/google/python-fire), 50 | * [argparse](https://docs.python.org/3/library/argparse.html) 51 | * [python-click](https://click.palletsprojects.com/en/8.0.x/) 52 | * Write a test for your python CLI tool 53 | * Containerize your Python CLI tool and deploy to a public container repo like [Github Container Registry](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry) or [Docker Hub](https://hub.docker.com) or [Amazon Public Container Registry](https://aws.amazon.com/blogs/aws/amazon-ecr-public-a-new-public-container-registry/). Here is an [example project](https://github.com/noahgift/container-from-scratch-python). 54 | 55 | #### Additional Thoughts on Challenge 56 | 57 | * How tiny can you make your container (hint try [alpine python](https://hub.docker.com/layers/python/library/python/3.9.0-alpine3.12/images/sha256-df77433749466a68bb599009753c9e5a8efaa3dd9c16450d442bb32f4c1fad4e?context=explore))? What is the approach that shrinks the size? 58 | * Can you lint your container with a linting tool like: [hadolint?](https://github.com/hadolint/hadolint#how-to-use) What about security scanning? 59 | * Why would it be impressive to have a docker pull command for a CLI on your resume? 60 | * Can you automatically build and push new containers in Github Actions? [Hint...yes](https://github.com/noahgift/Python-MLOps-Cookbook/blob/main/.github/workflows/pythonapp.yml#L25). 61 | * Can you build a GPT-3 CLI tool? https://openai.com/blog/openai-api/ 62 | 63 | *GPT 3*: 64 | * Book: https://learning.oreilly.com/library/view/gpt-3/9781098113612/ 65 | * Interview: https://learning.oreilly.com/videos/52-weeks-of/021822022VIDEOPAIML/ 66 | 67 | 68 | ### Advanced Challenges 69 | 70 | * Write a CLI in a language you don't know: 71 | 72 | * [cli-rosetta](https://github.com/noahgift/cli-rosetta) 73 | * [Bash, Powershell and C# CLI](https://github.com/noahgift/DotNet-AWS/tree/main/chapters/chap1) 74 | 75 | ## References 76 | 77 | * [Python Command Line Tools Course](https://learning.oreilly.com/videos/python-command-line/50131VIDEOPAIML/) 78 | * [Python Command Line Tools-Book](https://learning.oreilly.com/library/view/python-command-line/61619PAIML/) 79 | * [Python CI/CD for the Command-Line](https://learning.oreilly.com/videos/python-ci-cd-for/10092021VIDEOPAIML/) 80 | * [Nuclear Powered CLI Tools](https://github.com/noahgift/nuclear_powered_command_line_tools) 81 | * [mlops cookbook](https://github.com/noahgift/Python-MLOps-Cookbook) 82 | -------------------------------------------------------------------------------- /reduce_ram.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "name": "reduce-ram.ipynb", 7 | "provenance": [], 8 | "authorship_tag": "ABX9TyPlD1Q5CzIAbRj38yRQbl7q", 9 | "include_colab_link": true 10 | }, 11 | "kernelspec": { 12 | "name": "python3", 13 | "display_name": "Python 3" 14 | }, 15 | "language_info": { 16 | "name": "python" 17 | } 18 | }, 19 | "cells": [ 20 | { 21 | "cell_type": "markdown", 22 | "metadata": { 23 | "id": "view-in-github", 24 | "colab_type": "text" 25 | }, 26 | "source": [ 27 | "\"Open" 28 | ] 29 | }, 30 | { 31 | "cell_type": "code", 32 | "execution_count": 1, 33 | "metadata": { 34 | "colab": { 35 | "base_uri": "https://localhost:8080/" 36 | }, 37 | "id": "IPGP-b9i5YMT", 38 | "outputId": "d46ff47a-9e5e-47e9-9eef-c82a63e4fc0f" 39 | }, 40 | "outputs": [ 41 | { 42 | "output_type": "stream", 43 | "name": "stdout", 44 | "text": [ 45 | "--2022-02-26 02:39:31-- https://raw.githubusercontent.com/noahgift/coursera-de-c2-truncate-file/main/nba_2017.csv\n", 46 | "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.110.133, 185.199.111.133, ...\n", 47 | "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n", 48 | "HTTP request sent, awaiting response... 200 OK\n", 49 | "Length: 51688 (50K) [text/plain]\n", 50 | "Saving to: ‘nba_2017.csv’\n", 51 | "\n", 52 | "nba_2017.csv 100%[===================>] 50.48K --.-KB/s in 0.01s \n", 53 | "\n", 54 | "2022-02-26 02:39:31 (3.82 MB/s) - ‘nba_2017.csv’ saved [51688/51688]\n", 55 | "\n" 56 | ] 57 | } 58 | ], 59 | "source": [ 60 | "!wget https://raw.githubusercontent.com/noahgift/coursera-de-c2-truncate-file/main/nba_2017.csv" 61 | ] 62 | }, 63 | { 64 | "cell_type": "code", 65 | "source": [ 66 | "!ls -l nba_2017.csv" 67 | ], 68 | "metadata": { 69 | "colab": { 70 | "base_uri": "https://localhost:8080/" 71 | }, 72 | "id": "zsoUHmn_5apn", 73 | "outputId": "4b839520-fa70-4e4d-b012-f2254f53b5fa" 74 | }, 75 | "execution_count": 2, 76 | "outputs": [ 77 | { 78 | "output_type": "stream", 79 | "name": "stdout", 80 | "text": [ 81 | "-rw-r--r-- 1 root root 51688 Feb 26 02:39 nba_2017.csv\n" 82 | ] 83 | } 84 | ] 85 | }, 86 | { 87 | "cell_type": "code", 88 | "source": [ 89 | "!wc -l nba_2017.csv" 90 | ], 91 | "metadata": { 92 | "colab": { 93 | "base_uri": "https://localhost:8080/" 94 | }, 95 | "id": "MhwzVxHX5h3l", 96 | "outputId": "37d6cf08-6877-4241-dbfb-5588ccb0dd74" 97 | }, 98 | "execution_count": 3, 99 | "outputs": [ 100 | { 101 | "output_type": "stream", 102 | "name": "stdout", 103 | "text": [ 104 | "240 nba_2017.csv\n" 105 | ] 106 | } 107 | ] 108 | }, 109 | { 110 | "cell_type": "code", 111 | "source": [ 112 | "!shuf -n 100 nba_2017.csv > small_nba_2017.csv" 113 | ], 114 | "metadata": { 115 | "id": "HkVCtWlA5kUH" 116 | }, 117 | "execution_count": 4, 118 | "outputs": [] 119 | }, 120 | { 121 | "cell_type": "code", 122 | "source": [ 123 | "!wc -l small_nba_2017.csv" 124 | ], 125 | "metadata": { 126 | "colab": { 127 | "base_uri": "https://localhost:8080/" 128 | }, 129 | "id": "sL11psX65r3b", 130 | "outputId": "5192b472-209e-40e6-9dad-adbea832916f" 131 | }, 132 | "execution_count": 5, 133 | "outputs": [ 134 | { 135 | "output_type": "stream", 136 | "name": "stdout", 137 | "text": [ 138 | "100 small_nba_2017.csv\n" 139 | ] 140 | } 141 | ] 142 | }, 143 | { 144 | "cell_type": "code", 145 | "source": [ 146 | "!shuf --help" 147 | ], 148 | "metadata": { 149 | "colab": { 150 | "base_uri": "https://localhost:8080/" 151 | }, 152 | "id": "T552-4vd5xPe", 153 | "outputId": "a8f0e992-c741-4a6e-82ea-a5893ca5a0b8" 154 | }, 155 | "execution_count": 7, 156 | "outputs": [ 157 | { 158 | "output_type": "stream", 159 | "name": "stdout", 160 | "text": [ 161 | "Usage: shuf [OPTION]... [FILE]\n", 162 | " or: shuf -e [OPTION]... [ARG]...\n", 163 | " or: shuf -i LO-HI [OPTION]...\n", 164 | "Write a random permutation of the input lines to standard output.\n", 165 | "\n", 166 | "With no FILE, or when FILE is -, read standard input.\n", 167 | "\n", 168 | "Mandatory arguments to long options are mandatory for short options too.\n", 169 | " -e, --echo treat each ARG as an input line\n", 170 | " -i, --input-range=LO-HI treat each number LO through HI as an input line\n", 171 | " -n, --head-count=COUNT output at most COUNT lines\n", 172 | " -o, --output=FILE write result to FILE instead of standard output\n", 173 | " --random-source=FILE get random bytes from FILE\n", 174 | " -r, --repeat output lines can be repeated\n", 175 | " -z, --zero-terminated line delimiter is NUL, not newline\n", 176 | " --help display this help and exit\n", 177 | " --version output version information and exit\n", 178 | "\n", 179 | "GNU coreutils online help: \n", 180 | "Full documentation at: \n", 181 | "or available locally via: info '(coreutils) shuf invocation'\n" 182 | ] 183 | } 184 | ] 185 | }, 186 | { 187 | "cell_type": "code", 188 | "source": [ 189 | "" 190 | ], 191 | "metadata": { 192 | "id": "_LzhkwXt5zbO" 193 | }, 194 | "execution_count": null, 195 | "outputs": [] 196 | } 197 | ] 198 | } -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Creative Commons Legal Code 2 | 3 | CC0 1.0 Universal 4 | 5 | CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE 6 | LEGAL SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN 7 | ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS 8 | INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES 9 | REGARDING THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS 10 | PROVIDED HEREUNDER, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM 11 | THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED 12 | HEREUNDER. 13 | 14 | Statement of Purpose 15 | 16 | The laws of most jurisdictions throughout the world automatically confer 17 | exclusive Copyright and Related Rights (defined below) upon the creator 18 | and subsequent owner(s) (each and all, an "owner") of an original work of 19 | authorship and/or a database (each, a "Work"). 20 | 21 | Certain owners wish to permanently relinquish those rights to a Work for 22 | the purpose of contributing to a commons of creative, cultural and 23 | scientific works ("Commons") that the public can reliably and without fear 24 | of later claims of infringement build upon, modify, incorporate in other 25 | works, reuse and redistribute as freely as possible in any form whatsoever 26 | and for any purposes, including without limitation commercial purposes. 27 | These owners may contribute to the Commons to promote the ideal of a free 28 | culture and the further production of creative, cultural and scientific 29 | works, or to gain reputation or greater distribution for their Work in 30 | part through the use and efforts of others. 31 | 32 | For these and/or other purposes and motivations, and without any 33 | expectation of additional consideration or compensation, the person 34 | associating CC0 with a Work (the "Affirmer"), to the extent that he or she 35 | is an owner of Copyright and Related Rights in the Work, voluntarily 36 | elects to apply CC0 to the Work and publicly distribute the Work under its 37 | terms, with knowledge of his or her Copyright and Related Rights in the 38 | Work and the meaning and intended legal effect of CC0 on those rights. 39 | 40 | 1. Copyright and Related Rights. A Work made available under CC0 may be 41 | protected by copyright and related or neighboring rights ("Copyright and 42 | Related Rights"). Copyright and Related Rights include, but are not 43 | limited to, the following: 44 | 45 | i. the right to reproduce, adapt, distribute, perform, display, 46 | communicate, and translate a Work; 47 | ii. moral rights retained by the original author(s) and/or performer(s); 48 | iii. publicity and privacy rights pertaining to a person's image or 49 | likeness depicted in a Work; 50 | iv. rights protecting against unfair competition in regards to a Work, 51 | subject to the limitations in paragraph 4(a), below; 52 | v. rights protecting the extraction, dissemination, use and reuse of data 53 | in a Work; 54 | vi. database rights (such as those arising under Directive 96/9/EC of the 55 | European Parliament and of the Council of 11 March 1996 on the legal 56 | protection of databases, and under any national implementation 57 | thereof, including any amended or successor version of such 58 | directive); and 59 | vii. other similar, equivalent or corresponding rights throughout the 60 | world based on applicable law or treaty, and any national 61 | implementations thereof. 62 | 63 | 2. Waiver. To the greatest extent permitted by, but not in contravention 64 | of, applicable law, Affirmer hereby overtly, fully, permanently, 65 | irrevocably and unconditionally waives, abandons, and surrenders all of 66 | Affirmer's Copyright and Related Rights and associated claims and causes 67 | of action, whether now known or unknown (including existing as well as 68 | future claims and causes of action), in the Work (i) in all territories 69 | worldwide, (ii) for the maximum duration provided by applicable law or 70 | treaty (including future time extensions), (iii) in any current or future 71 | medium and for any number of copies, and (iv) for any purpose whatsoever, 72 | including without limitation commercial, advertising or promotional 73 | purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each 74 | member of the public at large and to the detriment of Affirmer's heirs and 75 | successors, fully intending that such Waiver shall not be subject to 76 | revocation, rescission, cancellation, termination, or any other legal or 77 | equitable action to disrupt the quiet enjoyment of the Work by the public 78 | as contemplated by Affirmer's express Statement of Purpose. 79 | 80 | 3. Public License Fallback. Should any part of the Waiver for any reason 81 | be judged legally invalid or ineffective under applicable law, then the 82 | Waiver shall be preserved to the maximum extent permitted taking into 83 | account Affirmer's express Statement of Purpose. In addition, to the 84 | extent the Waiver is so judged Affirmer hereby grants to each affected 85 | person a royalty-free, non transferable, non sublicensable, non exclusive, 86 | irrevocable and unconditional license to exercise Affirmer's Copyright and 87 | Related Rights in the Work (i) in all territories worldwide, (ii) for the 88 | maximum duration provided by applicable law or treaty (including future 89 | time extensions), (iii) in any current or future medium and for any number 90 | of copies, and (iv) for any purpose whatsoever, including without 91 | limitation commercial, advertising or promotional purposes (the 92 | "License"). The License shall be deemed effective as of the date CC0 was 93 | applied by Affirmer to the Work. Should any part of the License for any 94 | reason be judged legally invalid or ineffective under applicable law, such 95 | partial invalidity or ineffectiveness shall not invalidate the remainder 96 | of the License, and in such case Affirmer hereby affirms that he or she 97 | will not (i) exercise any of his or her remaining Copyright and Related 98 | Rights in the Work or (ii) assert any associated claims and causes of 99 | action with respect to the Work, in either case contrary to Affirmer's 100 | express Statement of Purpose. 101 | 102 | 4. Limitations and Disclaimers. 103 | 104 | a. No trademark or patent rights held by Affirmer are waived, abandoned, 105 | surrendered, licensed or otherwise affected by this document. 106 | b. Affirmer offers the Work as-is and makes no representations or 107 | warranties of any kind concerning the Work, express, implied, 108 | statutory or otherwise, including without limitation warranties of 109 | title, merchantability, fitness for a particular purpose, non 110 | infringement, or the absence of latent or other defects, accuracy, or 111 | the present or absence of errors, whether or not discoverable, all to 112 | the greatest extent permissible under applicable law. 113 | c. Affirmer disclaims responsibility for clearing rights of other persons 114 | that may apply to the Work or any use thereof, including without 115 | limitation any person's Copyright and Related Rights in the Work. 116 | Further, Affirmer disclaims responsibility for obtaining any necessary 117 | consents, permissions or other rights required for any use of the 118 | Work. 119 | d. Affirmer understands and acknowledges that Creative Commons is not a 120 | party to this document and has no duty or obligation with respect to 121 | this CC0 or use of the Work. 122 | --------------------------------------------------------------------------------