├── .gitignore ├── .travis.yml ├── LICENSE.txt ├── MANIFEST.in ├── README.rst ├── clf ├── __init__.py ├── api.py ├── command.py ├── constants.py ├── exceptions.py └── utils.py ├── requirements.txt ├── setup.py ├── test_clf.py └── tox.ini /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | dist 3 | build 4 | *.egg-info 5 | .idea 6 | .tox 7 | 8 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: python 2 | 3 | python: 4 | - "2.7" 5 | - "3.3" 6 | - "3.4" 7 | 8 | install: 9 | - pip install -r requirements.txt 10 | 11 | script: nosetests 12 | -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2015 Nicolas Crocfer 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in 13 | all copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 21 | THE SOFTWARE. 22 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include *.txt -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | clf - Command line tool to search snippets on Commandlinefu.com 2 | =============================================================== 3 | 4 | .. image:: https://travis-ci.org/ncrocfer/clf.svg?branch=master 5 | :target: https://travis-ci.org/ncrocfer/clf 6 | 7 | 8 | `Commandlinefu.com `_ is the place to record awesome command-line snippets. This tool allows you to search and view the results into your terminal. 9 | 10 | **Example 1** 11 | 12 | .. code-block:: shell 13 | 14 | $ clf python server 15 | 16 | # python smtp server 17 | python -m smtpd -n -c DebuggingServer localhost:1025 18 | 19 | # Python version 3: Serve current directory tree at http://$HOSTNAME:8000/ 20 | python -m http.server 21 | 22 | # Start a HTTP server which serves Python docs 23 | pydoc -p 8888 & gnome-open http://localhost:8888 24 | 25 | # put current directory in LAN quickly 26 | python -m SimpleHTTPServer 27 | 28 | **Example 2** 29 | 30 | .. code-block:: shell 31 | 32 | $ clf recursive line count 33 | 34 | # Recursive Line Count 35 | find ./ -not -type d | xargs wc -l | cut -c 1-8 | awk '{total += $1} END {print total}' 36 | 37 | # Recursive Line Count 38 | find * -type f -not -name ".*" | xargs wc -l 39 | 40 | # Get Total Line Count Of All Files In Subdirectory (Recursive) 41 | find . -type f -name "*.*" -exec cat {} > totalLines 2> /dev/null \; && wc -l totalLines && rm totalLines 42 | 43 | # Recursive Line Count 44 | wc -l `find . -name *.php` 45 | 46 | Installation 47 | ------------ 48 | 49 | The tool works with Python 2 and Python 3. It can be installed with `Pip` : 50 | 51 | :: 52 | 53 | pip install clf 54 | 55 | Usage 56 | ----- 57 | 58 | :: 59 | 60 | Command line tool to search snippets on Commandlinefu.com 61 | 62 | Usage: 63 | clf [options] 64 | clf [options] 65 | clf ... [options] 66 | 67 | Options: 68 | -h, --help Show this help. 69 | -v, --version Show version. 70 | -c, --color Enable colorized output. 71 | -i, --id Show the snippets id. 72 | -n NUMBER Show the n first snippets [default: 25]. 73 | --order=ORDER The order output (votes|date) [default: votes]. 74 | --proxy=PROXY The proxy used to perform requests. 75 | 76 | Examples: 77 | clf tar 78 | clf python server 79 | clf tar --proxy=http://127.0.0.1:8080 80 | clf --order=date -n 3 81 | 82 | Notes 83 | ----- 84 | 85 | **Enable the colorized output** 86 | 87 | You can set the :code:`CLF_COLOR` environment variable to enable the colorized output by default : 88 | 89 | :: 90 | 91 | $ export CLF_COLOR=1 92 | 93 | **Use clf in your scripts** 94 | 95 | You can import the :code:`clf` module and use it in your own scripts : 96 | 97 | .. code-block:: python 98 | 99 | #!/usr/bin/env python 100 | 101 | from clf import Clf 102 | 103 | c = Clf() 104 | for cmd in c.browse(): 105 | print("#{}\n{}\n".format( 106 | cmd.summary, 107 | cmd.command 108 | )) 109 | -------------------------------------------------------------------------------- /clf/__init__.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | """ 4 | ________ ______ 5 | / ____/ / / ____/ 6 | / / / / / /_ 7 | / /___/ /___/ __/ 8 | \____/_____/_/ 9 | 10 | Command line tool to search snippets on Commandlinefu.com 11 | 12 | Usage: 13 | clf [options] 14 | clf [options] 15 | clf ... [options] 16 | 17 | Options: 18 | -h, --help Show this help. 19 | -v, --version Show version. 20 | -c, --color Enable colorized output. 21 | -i, --id Show the snippets id. 22 | -l, --local Read the local snippets. 23 | -s ID Save the snippet locally. 24 | -n NUMBER Show the n first snippets [default: 25]. 25 | --order=ORDER The order output (votes|date) [default: votes]. 26 | --proxy=PROXY The proxy used to perform requests. 27 | 28 | Examples: 29 | clf tar 30 | clf python server 31 | clf tar --proxy=http://127.0.0.1:8080 32 | clf --order=date -n 3 33 | """ 34 | 35 | from docopt import docopt 36 | import os 37 | from pygments import highlight 38 | from pygments.lexers.shell import BashLexer 39 | from pygments.formatters import TerminalFormatter 40 | 41 | from clf.constants import VERSION, BLUE, END 42 | from clf.api import Clf 43 | from clf.utils import save_snippet, get_local_snippets 44 | from clf.exceptions import RequestsException, OSException, DuplicateException 45 | 46 | __all__ = ['Clf'] 47 | 48 | 49 | def run(): 50 | arguments = docopt(__doc__, version=VERSION) 51 | 52 | f = Clf(format="json", 53 | order=arguments['--order'], 54 | proxy=arguments['--proxy']) 55 | 56 | # Save the snippet locally 57 | if arguments['-s']: 58 | try: 59 | snippet = save_snippet(arguments['-s'], f._get_proxies()) 60 | except(RequestsException, OSException, DuplicateException) as e: 61 | print(e) 62 | else: 63 | print("The snippet has been successfully saved.") 64 | exit() 65 | 66 | # Retrieve the snippets list 67 | if arguments['--local']: 68 | commands = get_local_snippets() 69 | elif arguments['']: 70 | commands = f.command(arguments['']) 71 | elif arguments['']: 72 | commands = f.search(arguments['']) 73 | else: 74 | commands = f.browse() 75 | 76 | # Show the snippets id 77 | if arguments['--id']: 78 | sid = lambda c: '({})'.format(c.id) 79 | else: 80 | sid = lambda c: '' 81 | 82 | # Display in colors 83 | if (arguments['--color']) or (os.getenv('CLF_COLOR')): 84 | def get_output(command): 85 | detail = highlight(command.command, 86 | BashLexer(), TerminalFormatter(bg="dark")) 87 | return '{}#{} {}{}\n{}'.format(BLUE, sid(command), 88 | command.summary, END, detail) 89 | else: 90 | def get_output(command): 91 | return '#{} {}\n{}\n'.format(sid(command), 92 | command.summary, command.command) 93 | 94 | # Limit the number of snippets 95 | try: 96 | limit = int(arguments['-n']) 97 | except ValueError: 98 | limit = 25 99 | 100 | # Display the snippets 101 | for idx, command in enumerate(commands): 102 | if limit == idx: 103 | break 104 | 105 | print(get_output(command)) 106 | 107 | if __name__ == '__main__': 108 | run() 109 | -------------------------------------------------------------------------------- /clf/api.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | """ 4 | This module implements the Commandlinefu.com API 5 | """ 6 | 7 | import requests 8 | import base64 9 | 10 | try: 11 | from urllib import getproxies 12 | except ImportError: 13 | from urllib.request import getproxies 14 | 15 | try: 16 | from urlparse import urlparse 17 | except ImportError: 18 | from urllib import parse as urlparse 19 | 20 | from clf.command import Command 21 | from clf.constants import URL 22 | from clf.exceptions import (FormatException, OrderException, 23 | QueryException, RequestsException) 24 | 25 | 26 | class Clf(object): 27 | 28 | def __init__(self, format="json", order="votes", proxy=None): 29 | if format not in ['json', 'plaintext', 'rss']: 30 | raise FormatException('The format is invalid') 31 | 32 | self.format = format 33 | self.url = URL 34 | self.proxy = proxy 35 | 36 | if order == 'votes': 37 | self.order = 'sort-by-votes' 38 | elif order == 'date': 39 | self.order = '' 40 | else: 41 | raise OrderException('The order is invalid') 42 | 43 | def _prepare_search_url(self, *args): 44 | if isinstance(args[0], list): 45 | query = ' '.join(args[0]) 46 | elif isinstance(args[0], str): 47 | query = ' '.join(args[0].split()) # remove multiple blanks 48 | else: 49 | raise QueryException('The method expects a string or a list') 50 | 51 | b64_query = base64.b64encode(query.encode('utf-8')).decode('utf-8') 52 | url_query = '-'.join(query.split()) 53 | 54 | return "{}/commands/matching/{}/{}/{}/{}".format(self.url, 55 | url_query, 56 | b64_query, 57 | self.order, 58 | self.format) 59 | 60 | def _prepare_command_url(self, cmd): 61 | return "{}/commands/using/{}/{}/{}".format(self.url, 62 | cmd, 63 | self.order, 64 | self.format) 65 | 66 | def _prepare_browse_url(self): 67 | return "{}/commands/browse/{}/{}".format(self.url, 68 | self.order, 69 | self.format) 70 | 71 | def _get_proxies(self): 72 | proxies = getproxies() 73 | 74 | proxy = {} 75 | if self.proxy: 76 | parsed_proxy = urlparse(self.proxy) 77 | proxy[parsed_proxy.scheme] = parsed_proxy.geturl() 78 | 79 | proxies.update(proxy) 80 | return proxies 81 | 82 | def _get(self, url): 83 | proxies = self._get_proxies() 84 | 85 | try: 86 | r = requests.get(url, proxies=proxies) 87 | except requests.exceptions.ConnectionError: 88 | raise RequestsException("The connection is not available") 89 | 90 | try: 91 | return r.json() 92 | except ValueError: 93 | return [] 94 | 95 | def _cmd_generator(self, commands): 96 | for command in commands: 97 | yield Command( 98 | command['id'], 99 | command['command'], 100 | command['summary'], 101 | command['votes'], 102 | command['url'] 103 | ) 104 | 105 | def browse(self): 106 | commands = self._get(self._prepare_browse_url()) 107 | return self._cmd_generator(commands) 108 | 109 | def command(self, cmd): 110 | commands = self._get(self._prepare_command_url(cmd)) 111 | return self._cmd_generator(commands) 112 | 113 | def search(self, *args): 114 | commands = self._get(self._prepare_search_url(*args)) 115 | return self._cmd_generator(commands) 116 | -------------------------------------------------------------------------------- /clf/command.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | """ 4 | This class contains the class on which all the commands will be based 5 | """ 6 | 7 | 8 | class Command(object): 9 | 10 | def __init__(self, id, command, summary, votes, url): 11 | self.id = id 12 | self.command = command 13 | self.summary = summary 14 | self.votes = votes 15 | self.url = url 16 | 17 | def __repr__(self): 18 | return "{}(id={}, command={}, summary={}, votes={}, url={})".format( 19 | self.__class__.__name__, 20 | self.id, 21 | self.command, 22 | self.summary, 23 | self.votes, 24 | self.url 25 | ) 26 | -------------------------------------------------------------------------------- /clf/constants.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | """ 4 | This module contains the constants used by clf 5 | """ 6 | 7 | URL = "http://www.commandlinefu.com" 8 | VERSION = '0.5.7' 9 | BLUE = '\033[94m' 10 | END = '\033[0m' 11 | -------------------------------------------------------------------------------- /clf/exceptions.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | """ 4 | This module contains the list of clf exceptions 5 | """ 6 | 7 | 8 | class FormatException(Exception): 9 | """ The format is invalid """ 10 | 11 | 12 | class OrderException(Exception): 13 | """ The order is invalid """ 14 | 15 | 16 | class QueryException(Exception): 17 | """ The parameter is invalid """ 18 | 19 | 20 | class RequestsException(Exception): 21 | """ The request is invalid """ 22 | 23 | 24 | class OSException(Exception): 25 | """ An OS error has occurred """ 26 | 27 | 28 | class DuplicateException(Exception): 29 | """ The snippet already exists """ 30 | -------------------------------------------------------------------------------- /clf/utils.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | """ 4 | This module contains functions that are used in CLF. 5 | """ 6 | 7 | import requests 8 | import os 9 | from lxml import html 10 | import yaml 11 | 12 | from clf.constants import URL 13 | from clf.exceptions import RequestsException, OSException, DuplicateException 14 | from clf.command import Command 15 | 16 | 17 | def get_snippet(id, proxies): 18 | url = '{}/commands/view/{}'.format(URL, id) 19 | 20 | try: 21 | r = requests.get(url, proxies=proxies) 22 | except requests.exceptions.ConnectionError: 23 | raise RequestsException("The connection is not available") 24 | 25 | if url not in r.url: 26 | raise RequestsException("The snippet does not seem to exist.") 27 | 28 | tree = html.fromstring(r.text) 29 | 30 | c = Command(id, 31 | str(tree.xpath('//div[@class="command"]/text()')[0]), 32 | str(tree.xpath('//div[@id="terminal-header"]/h1/text()')[0]), 33 | str(tree.xpath('//div[@class="num-votes"]/text()')[0]), 34 | str(r.url) 35 | ) 36 | 37 | return c 38 | 39 | 40 | def save_snippet(id, proxies): 41 | snippets_file = get_snippets_file() 42 | 43 | with open(snippets_file, 'r') as f: 44 | data = yaml.load(f) 45 | if not data: 46 | data = {} 47 | 48 | if id not in data: 49 | snippet = get_snippet(id, proxies) 50 | data[id] = snippet 51 | 52 | with open(snippets_file, 'w') as f: 53 | yaml.dump(data, f) 54 | else: 55 | raise DuplicateException('The snippet is already saved.') 56 | 57 | return snippet 58 | 59 | 60 | def get_snippets_file(): 61 | snippets_file = os.path.join(os.path.expanduser('~'), '.clf.yaml') 62 | 63 | if not os.path.isfile(snippets_file): 64 | try: 65 | f = open(snippets_file, 'w') 66 | f.close() 67 | except OSError: 68 | raise OSException('Could not create {}'.format(snippets_file)) 69 | 70 | return snippets_file 71 | 72 | 73 | def get_local_snippets(): 74 | snippets_file = get_snippets_file() 75 | 76 | with open(snippets_file, 'r') as f: 77 | data = yaml.load(f) 78 | 79 | return data.values() if data != None else [] 80 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | Pygments>=1.5 2 | docopt>=0.6.0 3 | requests>=2.3.0 4 | lxml>=3.4.4 5 | PyYAML>=3.11 -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | from setuptools import setup 4 | 5 | entry_points = { 6 | 'console_scripts': [ 7 | 'clf = clf:run', 8 | ] 9 | } 10 | requirements = open('requirements.txt').read() 11 | readme = open('README.rst').read() 12 | 13 | setup( 14 | name="clf", 15 | version="0.5.7", 16 | url='http://github.com/ncrocfer/clf', 17 | author='Nicolas Crocfer', 18 | author_email='ncrocfer@gmail.com', 19 | description="Command line tool to search snippets on Commandlinefu.com", 20 | long_description=readme, 21 | packages=['clf'], 22 | include_package_data=True, 23 | install_requires=requirements, 24 | entry_points=entry_points, 25 | classifiers=( 26 | 'Development Status :: 5 - Production/Stable', 27 | 'Environment :: Console', 28 | 'Natural Language :: English', 29 | 'Intended Audience :: Developers', 30 | 'License :: OSI Approved :: MIT License', 31 | 'Operating System :: OS Independent', 32 | 'Programming Language :: Python', 33 | 'Programming Language :: Python :: 2', 34 | 'Programming Language :: Python :: 2.7', 35 | 'Programming Language :: Python :: 3', 36 | 'Programming Language :: Python :: 3.3', 37 | 'Programming Language :: Python :: 3.4' 38 | ), 39 | ) 40 | -------------------------------------------------------------------------------- /test_clf.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | """ 4 | All the tests for CLF are here 5 | """ 6 | 7 | 8 | import types 9 | import unittest 10 | try: 11 | from urllib.parse import urljoin 12 | except ImportError: 13 | from urlparse import urljoin 14 | 15 | from clf.api import Clf 16 | from clf.command import Command 17 | from clf.constants import URL 18 | from clf.exceptions import (FormatException, OrderException, 19 | QueryException) 20 | 21 | 22 | class ClfTestCase(unittest.TestCase): 23 | 24 | def setUp(self): 25 | self.urls = { 26 | 'browse_by_votes_in_json': '/commands/browse/sort-by-votes/json', 27 | 'browse_by_votes_in_xml': '/commands/browse/sort-by-votes/xml', 28 | 'browse_by_votes_in_plaintext': '/commands/browse/' 29 | 'sort-by-votes/plaintext', 30 | 'browse_by_date_in_json': '/commands/browse//json', 31 | 'browse_by_date_in_xml': '/commands/browse//xml', 32 | 'browse_by_date_in_plaintext': '/commands/browse//plaintext', 33 | 'command_by_votes_in_json': '/commands/using/tar/' 34 | 'sort-by-votes/json', 35 | 'command_by_votes_in_xml': '/commands/using/tar/sort-by-votes/xml', 36 | 'command_by_votes_in_plaintext': '/commands/using/tar/' 37 | 'sort-by-votes/plaintext', 38 | 'command_by_date_in_json': '/commands/using/tar//json', 39 | 'command_by_date_in_xml': '/commands/using/tar//xml', 40 | 'command_by_date_in_plaintext': '/commands/using/tar//plaintext', 41 | 'search_by_votes_in_json': '/commands/matching/' 42 | 'python-server/cHl0aG9uIHNlcnZlcg==/' 43 | 'sort-by-votes/json', 44 | 'search_by_votes_in_xml': '/commands/matching/' 45 | 'python-server/cHl0aG9uIHNlcnZlcg==/' 46 | 'sort-by-votes/xml', 47 | 'search_by_votes_in_plaintext': '/commands/matching/' 48 | 'python-server/' 49 | 'cHl0aG9uIHNlcnZlcg==/' 50 | 'sort-by-votes/plaintext', 51 | 'search_by_date_in_json': '/commands/matching/python-server/' 52 | 'cHl0aG9uIHNlcnZlcg==//json', 53 | 'search_by_date_in_xml': '/commands/matching/python-server/' 54 | 'cHl0aG9uIHNlcnZlcg==//xml', 55 | 'search_by_date_in_plaintext': '/commands/matching/python-server/' 56 | 'cHl0aG9uIHNlcnZlcg==//plaintext', 57 | 58 | } 59 | self.urls = {k: urljoin(URL, v) for k, v in self.urls.items()} 60 | 61 | def tearDown(self): 62 | pass 63 | 64 | def test_browse_urls(self): 65 | for order in ['votes', 'date']: 66 | clf = Clf(order=order) 67 | 68 | for format in ['json', 'xml', 'plaintext']: 69 | clf.format = format 70 | url = self.urls['browse_by_' + order + '_in_' + format] 71 | self.assertEqual(clf._prepare_browse_url(), 72 | url) 73 | 74 | def test_command_urls(self): 75 | for order in ['votes', 'date']: 76 | clf = Clf(order=order) 77 | 78 | for format in ['json', 'xml', 'plaintext']: 79 | clf.format = format 80 | url = self.urls['command_by_' + order + '_in_' + format] 81 | self.assertEqual(clf._prepare_command_url('tar'), 82 | url) 83 | 84 | def test_search_urls(self): 85 | for order in ['votes', 'date']: 86 | clf = Clf(order=order) 87 | 88 | for format in ['json', 'xml', 'plaintext']: 89 | clf.format = format 90 | 91 | url = self.urls['search_by_' + order + '_in_' + format] 92 | self.assertEqual(clf._prepare_search_url('python server'), 93 | url) 94 | self.assertEqual(clf._prepare_search_url(['python', 'server']), 95 | url) 96 | 97 | def test_browse(self): 98 | commands = Clf().browse() 99 | self.assertIsInstance(commands, types.GeneratorType) 100 | self.assertIsInstance(next(commands), Command) 101 | self.assertEqual(24, sum(1 for _ in commands)) 102 | 103 | def test_command(self): 104 | commands = Clf().command('tar') 105 | self.assertIsInstance(commands, types.GeneratorType) 106 | self.assertIsInstance(next(commands), Command) 107 | self.assertEqual(24, sum(1 for _ in commands)) 108 | 109 | def test_search(self): 110 | commands = Clf().search('python server') 111 | self.assertIsInstance(commands, types.GeneratorType) 112 | self.assertIsInstance(next(commands), Command) 113 | self.assertGreater(sum(1 for _ in commands), 0) 114 | 115 | def test_exceptions(self): 116 | with self.assertRaises(FormatException): 117 | Clf(format='foo') 118 | 119 | with self.assertRaises(OrderException): 120 | Clf(order='foo') 121 | 122 | with self.assertRaises(QueryException): 123 | Clf()._prepare_search_url({'foo': 'bar'}) 124 | 125 | 126 | if __name__ == '__main__': 127 | unittest.main() 128 | -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | [tox] 2 | envlist = py27,py33,py34 3 | 4 | [testenv] 5 | commands = 6 | python test_clf.py 7 | deps = 8 | -rrequirements.txt --------------------------------------------------------------------------------