├── .gitignore ├── Chapter01 ├── price_log.py ├── recipe_cli_step1.py ├── recipe_cli_step2.py ├── recipe_cli_step3.py ├── recipe_format_strings_step1.py └── requirements.txt ├── Chapter02 ├── config.ini ├── config.yaml ├── cron.py ├── email_conf.ini ├── email_task.py ├── prepare_task_step1.py ├── prepare_task_step3.py ├── prepare_task_step6.py ├── prepare_task_yaml.py ├── requirements.txt ├── task_with_error_handling_step1.py └── task_with_error_handling_step4.py ├── Chapter03 ├── crawling_web_step1.py ├── speed_up_step1.py └── test_site │ ├── README │ ├── files │ ├── 33714fc865e02aeda2dabb9a42a787b2-0.html │ ├── 5eabef23f63024c20389c34b94dee593-1.html │ ├── archive-september-2018.html │ ├── b93bec5d9681df87e6e8d5703ed7cd81-2.html │ └── meta.js │ ├── index.html │ ├── rw_common │ ├── images │ │ └── fallback.jpeg │ ├── plugins │ │ └── blog │ │ │ ├── rss.gif │ │ │ ├── smiley_angry.png │ │ │ ├── smiley_embarrassed.png │ │ │ ├── smiley_footinmouth.png │ │ │ ├── smiley_gasp.png │ │ │ ├── smiley_laugh.png │ │ │ ├── smiley_sad.png │ │ │ ├── smiley_smile.png │ │ │ └── smiley_wink.png │ ├── themes │ │ └── offroad │ │ │ ├── assets │ │ │ ├── fonts │ │ │ │ ├── FontAwesome.otf │ │ │ │ ├── fontawesome-webfont.eot │ │ │ │ ├── fontawesome-webfont.ttf │ │ │ │ ├── fontawesome-webfont.woff │ │ │ │ ├── fontawesome-webfont.woff2 │ │ │ │ ├── istok-web-v11-latin-700.eot │ │ │ │ ├── istok-web-v11-latin-700.ttf │ │ │ │ ├── istok-web-v11-latin-700.woff2 │ │ │ │ ├── istok-web-v11-latin-regular.eot │ │ │ │ ├── istok-web-v11-latin-regular.ttf │ │ │ │ ├── istok-web-v11-latin-regular.woff │ │ │ │ ├── istok-web-v11-latin-regular.woff2 │ │ │ │ ├── lora-v12-latin-700.eot │ │ │ │ ├── lora-v12-latin-700.ttf │ │ │ │ ├── lora-v12-latin-700.woff │ │ │ │ ├── lora-v12-latin-700.woff2 │ │ │ │ ├── lora-v12-latin-700italic.eot │ │ │ │ ├── lora-v12-latin-700italic.ttf │ │ │ │ ├── lora-v12-latin-700italic.woff │ │ │ │ ├── lora-v12-latin-700italic.woff2 │ │ │ │ ├── lora-v12-latin-italic.eot │ │ │ │ ├── lora-v12-latin-italic.ttf │ │ │ │ ├── lora-v12-latin-italic.woff │ │ │ │ ├── lora-v12-latin-italic.woff2 │ │ │ │ ├── lora-v12-latin-regular.eot │ │ │ │ ├── lora-v12-latin-regular.ttf │ │ │ │ ├── lora-v12-latin-regular.woff │ │ │ │ └── lora-v12-latin-regular.woff2 │ │ │ ├── images │ │ │ │ └── fallback.jpeg │ │ │ └── javascript │ │ │ │ ├── background-blur.js │ │ │ │ ├── background-no-blur.js │ │ │ │ ├── html5shiv.js │ │ │ │ ├── min │ │ │ │ └── background-dont-blur-min.js │ │ │ │ ├── respond.js │ │ │ │ ├── sidebar-hidden.js │ │ │ │ ├── sidebar-left.js │ │ │ │ └── sidebar-right.js │ │ │ ├── consolidated.css │ │ │ └── javascript.js │ └── version.txt │ ├── simple_delay_server.py │ └── sitemap.xml ├── Chapter04 ├── documents │ ├── dir │ │ ├── file1.txt │ │ ├── file2.txt │ │ ├── file6.pdf │ │ └── subdir │ │ │ ├── file3.txt │ │ │ ├── file4.txt │ │ │ └── file5.pdf │ ├── document-1.docx │ ├── document-1.pdf │ ├── document-2.pdf │ ├── example_iso.txt │ ├── example_logs.log │ ├── example_output_iso.txt │ ├── example_utf8.txt │ ├── top_films.csv │ └── zen_of_python.txt ├── gps_conversion.py ├── images │ ├── photo-dublin-a-text.jpg │ ├── photo-dublin-a1.jpg │ ├── photo-dublin-a2.png │ ├── photo-dublin-b.png │ └── photo-text.jpg └── scan.py ├── Chapter05 ├── jinja_template.html ├── markdown_template.md ├── structuring_pdf.py └── watermarking_pdf.py ├── Chapter06 ├── include_macro.py ├── libreoffice_script.py ├── movies.csv ├── movies.ods └── movies.xlsx ├── Chapter07 ├── aggregate_OH.csv ├── aggregate_by_location.py ├── aggregate_by_location_by_pandas.py ├── aggregate_by_location_parallel.py ├── location_price.py ├── logs_to_csv.py ├── output_1_OH.csv ├── output_2_OH.csv ├── output_3_OH.csv ├── price_log.py ├── sale_logs │ ├── OH │ │ └── logs.txt │ └── ON │ │ └── logs.txt └── standard_date.py ├── Chapter08 ├── adding_legend_and_annotations.py ├── data.png ├── scatter.csv └── visualising_maps.py ├── Chapter09 ├── app.py ├── email_styling.html ├── email_template.md ├── telegram_bot.py └── telegram_bot_custom_keyboard.py ├── Chapter10 ├── config-channel.ini ├── config-opportunity.ini ├── create_personalised_coupons.py ├── email_styling.html ├── email_template.md ├── generate_sales_report.py ├── output.pdf ├── parse_sales_log.py ├── report.xlsx ├── sale_log.py ├── sales │ ├── 345 │ │ └── logs.txt │ ├── 438 │ │ ├── logs_1.txt │ │ ├── logs_2.txt │ │ ├── logs_3.txt │ │ └── logs_4.txt │ └── 656 │ │ └── logs.txt ├── search_keywords.py ├── search_opportunities.py └── send_notifications.py ├── Chapter11 ├── example_shop1.txt ├── example_shop2.txt ├── example_shop3.txt ├── handwrite.jpg ├── image_labels.py ├── image_text.py ├── image_text_box.py ├── images │ ├── handwrite.jpg │ ├── photo-dublin-a-text.jpg │ ├── photo-dublin-a2.png │ ├── photo-dublin-b.png │ └── photo-text.jpg ├── photo-dublin-a-text.jpg ├── photo-dublin-a2.png ├── photo-dublin-b.png ├── photo-text.jpg ├── shop.zip ├── shop_training │ ├── appliances │ │ ├── email1.txt │ │ ├── email10.txt │ │ ├── email2.txt │ │ ├── email3.txt │ │ ├── email4.txt │ │ ├── email5.txt │ │ ├── email6.txt │ │ ├── email7.txt │ │ ├── email8.txt │ │ └── email9.txt │ ├── furniture │ │ ├── email1.txt │ │ ├── email10.txt │ │ ├── email2.txt │ │ ├── email3.txt │ │ ├── email4.txt │ │ ├── email5.txt │ │ ├── email6.txt │ │ ├── email7.txt │ │ ├── email8.txt │ │ └── email9.txt │ └── others │ │ ├── email1.txt │ │ ├── email10.txt │ │ ├── email2.txt │ │ ├── email3.txt │ │ ├── email4.txt │ │ ├── email5.txt │ │ ├── email6.txt │ │ ├── email7.txt │ │ ├── email8.txt │ │ └── email9.txt ├── text_analysis.py ├── text_analysis_categories.py ├── text_predict.py └── texts │ ├── category_example.txt │ ├── crime-and-punishement.txt │ ├── crimen-y-castigo.txt │ ├── pride_and_prejudice.txt │ ├── regenta.txt │ └── tale_two_cities.txt ├── Chapter12 ├── code │ ├── __init__.py │ ├── code_fixtures.py │ ├── code_requests.py │ ├── dependencies.py │ └── external.py ├── conftest.py └── tests │ ├── test_case.py │ ├── test_dependencies.py │ ├── test_external.py │ ├── test_fixtures.py │ ├── test_requests.py │ └── test_requests_time.py ├── Chapter13 ├── debug_algorithm.py ├── debug_logging.py ├── debug_skills.py └── debug_skills_fixed.py ├── LICENSE ├── README.md └── requirements.txt /.gitignore: -------------------------------------------------------------------------------- 1 | .venv 2 | __pycache__ 3 | *.pyc 4 | .DS_Store 5 | -------------------------------------------------------------------------------- /Chapter01/price_log.py: -------------------------------------------------------------------------------- 1 | import parse 2 | from decimal import Decimal 3 | import delorean 4 | 5 | 6 | class PriceLog(object): 7 | 8 | def __init__(self, timestamp, product_id, price): 9 | self.timestamp = timestamp 10 | self.product_id = product_id 11 | self.price = price 12 | 13 | def __repr__(self): 14 | return ''.format(self.timestamp, 15 | self.product_id, 16 | self.price) 17 | 18 | @classmethod 19 | def parse(cls, text_log): 20 | ''' 21 | Parse from a text log with the format 22 | [] - SALE - PRODUCT: - PRICE: $ 23 | to a PriceLog object 24 | ''' 25 | def price(string): 26 | return Decimal(string) 27 | 28 | def isodate(string): 29 | return delorean.parse(string) 30 | 31 | FORMAT = ('[{timestamp:isodate}] - SALE - PRODUCT: {product:d} - ' 32 | 'PRICE: ${price:price}') 33 | 34 | formats = {'price': price, 'isodate': isodate} 35 | result = parse.parse(FORMAT, text_log, formats) 36 | 37 | return cls(timestamp=result['timestamp'], 38 | product_id=result['product'], 39 | price=result['price']) 40 | 41 | -------------------------------------------------------------------------------- /Chapter01/recipe_cli_step1.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | 3 | 4 | def main(number): 5 | print('#' * number) 6 | 7 | 8 | if __name__ == '__main__': 9 | 10 | parser = argparse.ArgumentParser() 11 | parser.add_argument('number', type=int, help='A number') 12 | 13 | args = parser.parse_args() 14 | 15 | main(args.number) 16 | -------------------------------------------------------------------------------- /Chapter01/recipe_cli_step2.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | 3 | 4 | def main(character, number): 5 | print(character * number) 6 | 7 | 8 | if __name__ == '__main__': 9 | 10 | parser = argparse.ArgumentParser() 11 | parser.add_argument('number', type=int, help='A number') 12 | parser.add_argument('-c', type=str, help='Character to print', 13 | default='#') 14 | 15 | args = parser.parse_args() 16 | 17 | main(args.c, args.number) 18 | -------------------------------------------------------------------------------- /Chapter01/recipe_cli_step3.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | 3 | 4 | def main(character, number): 5 | print(character * number) 6 | 7 | 8 | if __name__ == '__main__': 9 | 10 | parser = argparse.ArgumentParser() 11 | parser.add_argument('number', type=int, help='A number') 12 | parser.add_argument('-c', type=str, help='Character to print', 13 | default='#') 14 | parser.add_argument('-U', action='store_true', default=False, 15 | dest='uppercase', 16 | help='Uppercase the character') 17 | 18 | args = parser.parse_args() 19 | 20 | if args.uppercase: 21 | args.c = args.c.upper() 22 | 23 | main(args.c, args.number) 24 | -------------------------------------------------------------------------------- /Chapter01/recipe_format_strings_step1.py: -------------------------------------------------------------------------------- 1 | # INPUT DATA 2 | data = [ 3 | (1000, 10), 4 | (2000, 17), 5 | (2500, 170), 6 | (2500, -170), 7 | ] 8 | 9 | # Print the header for reference 10 | print('REVENUE | PROFIT | PERCENT') 11 | 12 | # This template aligns and displays the data in the proper format 13 | TEMPLATE = '{revenue:>7,} | {profit:>+6} | {percent:>7.2%}' 14 | 15 | # Print the data rows 16 | for revenue, profit in data: 17 | percent = profit / revenue 18 | row = TEMPLATE.format(revenue=revenue, profit=profit, percent=percent) 19 | print(row) 20 | -------------------------------------------------------------------------------- /Chapter01/requirements.txt: -------------------------------------------------------------------------------- 1 | delorean==1.0.0 2 | requests==2.22.0 3 | parse==1.14.0 4 | 5 | -------------------------------------------------------------------------------- /Chapter02/config.ini: -------------------------------------------------------------------------------- 1 | [ARGUMENTS] 2 | n1=5 3 | n2=7 4 | -------------------------------------------------------------------------------- /Chapter02/config.yaml: -------------------------------------------------------------------------------- 1 | ARGUMENTS: 2 | n1: 7 3 | n2: 4 4 | -------------------------------------------------------------------------------- /Chapter02/cron.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import sys 3 | from datetime import datetime 4 | import configparser 5 | 6 | 7 | def main(number, other_number, output): 8 | result = number * other_number 9 | print(f'[{datetime.utcnow().isoformat()}] The result is {result}', 10 | file=output) 11 | 12 | 13 | if __name__ == '__main__': 14 | parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter) 15 | parser.add_argument('-c', dest='config', type=argparse.FileType('r'), 16 | help='config file', 17 | default='/etc/automate.ini') 18 | parser.add_argument('-o', dest='output', type=argparse.FileType('a'), 19 | help='output file', 20 | default=sys.stdout) 21 | 22 | args = parser.parse_args() 23 | if args.config: 24 | config = configparser.ConfigParser() 25 | config.read_file(args.config) 26 | # Transforming values into integers 27 | args.n1 = int(config['ARGUMENTS']['n1']) 28 | args.n2 = int(config['ARGUMENTS']['n2']) 29 | 30 | main(args.n1, args.n2, args.output) 31 | -------------------------------------------------------------------------------- /Chapter02/email_conf.ini: -------------------------------------------------------------------------------- 1 | [DEFAULT] 2 | email = EMAIL@gmail.com 3 | server = smtp.gmail.com 4 | port = 465 5 | password = PASSWORD 6 | -------------------------------------------------------------------------------- /Chapter02/email_task.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import configparser 3 | 4 | import smtplib 5 | from email.message import EmailMessage 6 | 7 | 8 | def main(to_email, server, port, from_email, password): 9 | print(f'With love, from {from_email} to {to_email}') 10 | 11 | # Create the message 12 | subject = 'With love, from ME to YOU' 13 | text = '''This is an example test''' 14 | msg = EmailMessage() 15 | msg.set_content(text) 16 | msg['Subject'] = subject 17 | msg['From'] = from_email 18 | msg['To'] = to_email 19 | 20 | # Open communication and send 21 | server = smtplib.SMTP_SSL(server, port) 22 | server.login(from_email, password) 23 | server.send_message(msg) 24 | server.quit() 25 | 26 | 27 | if __name__ == '__main__': 28 | parser = argparse.ArgumentParser() 29 | parser.add_argument('email', type=str, help='destination email') 30 | parser.add_argument('-c', dest='config', type=argparse.FileType('r'), 31 | help='config file', default=None) 32 | 33 | args = parser.parse_args() 34 | if not args.config: 35 | print('Error, a config file is required') 36 | parser.print_help() 37 | exit(1) 38 | 39 | config = configparser.ConfigParser() 40 | config.read_file(args.config) 41 | 42 | main(args.email, 43 | server=config['DEFAULT']['server'], 44 | port=config['DEFAULT']['port'], 45 | from_email=config['DEFAULT']['email'], 46 | password=config['DEFAULT']['password']) 47 | -------------------------------------------------------------------------------- /Chapter02/prepare_task_step1.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | 3 | 4 | def main(number, other_number): 5 | result = number * other_number 6 | print(f'The result is {result}') 7 | 8 | 9 | if __name__ == '__main__': 10 | parser = argparse.ArgumentParser() 11 | parser.add_argument('-n1', type=int, help='A number', default=1) 12 | parser.add_argument('-n2', type=int, help='Another number', default=1) 13 | 14 | args = parser.parse_args() 15 | 16 | main(args.n1, args.n2) 17 | -------------------------------------------------------------------------------- /Chapter02/prepare_task_step3.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import configparser 3 | 4 | 5 | def main(number, other_number): 6 | result = number * other_number 7 | print(f'The result is {result}') 8 | 9 | 10 | if __name__ == '__main__': 11 | parser = argparse.ArgumentParser() 12 | parser.add_argument('-n1', type=int, help='A number', default=1) 13 | parser.add_argument('-n2', type=int, help='Another number', default=1) 14 | 15 | parser.add_argument('--config', '-c', type=argparse.FileType('r'), 16 | help='config file') 17 | 18 | args = parser.parse_args() 19 | if args.config: 20 | config = configparser.ConfigParser() 21 | config.read_file(args.config) 22 | # Transforming values into integers 23 | args.n1 = int(config['ARGUMENTS']['n1']) 24 | args.n2 = int(config['ARGUMENTS']['n2']) 25 | 26 | main(args.n1, args.n2) 27 | -------------------------------------------------------------------------------- /Chapter02/prepare_task_step6.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import sys 3 | import configparser 4 | 5 | 6 | def main(number, other_number, output): 7 | result = number * other_number 8 | print(f'The result is {result}', file=output) 9 | 10 | 11 | if __name__ == '__main__': 12 | parser = argparse.ArgumentParser() 13 | parser.add_argument('-n1', type=int, help='A number', default=1) 14 | parser.add_argument('-n2', type=int, help='Another number', default=1) 15 | 16 | parser.add_argument('--config', '-c', type=argparse.FileType('r'), 17 | help='config file') 18 | parser.add_argument('-o', dest='output', type=argparse.FileType('w'), 19 | help='output file', 20 | default=sys.stdout) 21 | 22 | args = parser.parse_args() 23 | if args.config: 24 | config = configparser.ConfigParser() 25 | config.read_file(args.config) 26 | # Transforming values into integers 27 | args.n1 = int(config['ARGUMENTS']['n1']) 28 | args.n2 = int(config['ARGUMENTS']['n2']) 29 | 30 | main(args.n1, args.n2, args.output) 31 | -------------------------------------------------------------------------------- /Chapter02/prepare_task_yaml.py: -------------------------------------------------------------------------------- 1 | import yaml 2 | import argparse 3 | import sys 4 | 5 | 6 | def main(number, other_number, output): 7 | result = number * other_number 8 | print(f'The result is {result}', file=output) 9 | 10 | 11 | if __name__ == '__main__': 12 | parser = argparse.ArgumentParser() 13 | parser.add_argument('-n1', type=int, help='A number', default=1) 14 | parser.add_argument('-n2', type=int, help='Another number', default=1) 15 | 16 | parser.add_argument('-c', dest='config', type=argparse.FileType('r'), 17 | help='config file in YAML format', 18 | default=None) 19 | parser.add_argument('-o', dest='output', type=argparse.FileType('w'), 20 | help='output file', 21 | default=sys.stdout) 22 | 23 | args = parser.parse_args() 24 | if args.config: 25 | config = yaml.load(args.config, Loader=yaml.FullLoader) 26 | # Transforming values into integers 27 | args.n1 = config['ARGUMENTS']['n1'] 28 | args.n2 = config['ARGUMENTS']['n2'] 29 | 30 | main(args.n1, args.n2, args.output) 31 | -------------------------------------------------------------------------------- /Chapter02/requirements.txt: -------------------------------------------------------------------------------- 1 | PyYAML==5.3 2 | -------------------------------------------------------------------------------- /Chapter02/task_with_error_handling_step1.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import sys 3 | 4 | 5 | def main(number, other_number, output): 6 | result = number / other_number 7 | print(f'The result is {result}', file=output) 8 | 9 | 10 | if __name__ == '__main__': 11 | parser = argparse.ArgumentParser() 12 | parser.add_argument('-n1', type=int, help='A number', default=1) 13 | parser.add_argument('-n2', type=int, help='Another number', default=1) 14 | 15 | parser.add_argument('-o', dest='output', type=argparse.FileType('w'), 16 | help='output file', default=sys.stdout) 17 | 18 | args = parser.parse_args() 19 | 20 | main(args.n1, args.n2, args.output) 21 | -------------------------------------------------------------------------------- /Chapter02/task_with_error_handling_step4.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import sys 3 | import logging 4 | 5 | LOG_FORMAT = '%(asctime)s %(name)s %(levelname)s %(message)s' 6 | LOG_LEVEL = logging.DEBUG 7 | 8 | 9 | def main(number, other_number, output): 10 | logging.info(f'Dividing {number} between {other_number}') 11 | result = number / other_number 12 | print(f'The result is {result}', file=output) 13 | 14 | 15 | if __name__ == '__main__': 16 | parser = argparse.ArgumentParser() 17 | parser.add_argument('-n1', type=int, help='A number', default=1) 18 | parser.add_argument('-n2', type=int, help='Another number', default=1) 19 | 20 | parser.add_argument('-o', dest='output', type=argparse.FileType('w'), 21 | help='output file', default=sys.stdout) 22 | parser.add_argument('-l', dest='log', type=str, help='log file', 23 | default=None) 24 | 25 | args = parser.parse_args() 26 | if args.log: 27 | logging.basicConfig(format=LOG_FORMAT, filename=args.log, 28 | level=LOG_LEVEL) 29 | else: 30 | logging.basicConfig(format=LOG_FORMAT, level=LOG_LEVEL) 31 | 32 | try: 33 | main(args.n1, args.n2, args.output) 34 | except Exception as exc: 35 | logging.exception("Error running task") 36 | exit(1) 37 | -------------------------------------------------------------------------------- /Chapter03/crawling_web_step1.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import requests 3 | import logging 4 | import http.client 5 | import re 6 | from urllib.parse import urlparse, urljoin 7 | from bs4 import BeautifulSoup 8 | 9 | 10 | DEFAULT_PHRASE = 'python' 11 | 12 | 13 | def process_link(source_link, text): 14 | logging.info(f'Extracting links from {source_link}') 15 | parsed_source = urlparse(source_link) 16 | result = requests.get(source_link) 17 | if result.status_code != http.client.OK: 18 | logging.error(f'Error retrieving {source_link}: {result}') 19 | return [] 20 | 21 | if 'html' not in result.headers['Content-type']: 22 | logging.info(f'Link {source_link} is not an HTML page') 23 | return [] 24 | 25 | page = BeautifulSoup(result.text, 'html.parser') 26 | search_text(source_link, page, text) 27 | 28 | return get_links(parsed_source, page) 29 | 30 | 31 | def get_links(parsed_source, page): 32 | '''Retrieve the links on the page''' 33 | links = [] 34 | for element in page.find_all('a'): 35 | link = element.get('href') 36 | if not link: 37 | continue 38 | 39 | # Avoid internal, same page links 40 | if link.startswith('#'): 41 | continue 42 | 43 | if link.startswith('mailto:'): 44 | # Ignore other links like mailto 45 | # More cases like ftp or similar may be included here 46 | continue 47 | 48 | # Always accept local links 49 | if not link.startswith('http'): 50 | netloc = parsed_source.netloc 51 | scheme = parsed_source.scheme 52 | path = urljoin(parsed_source.path, link) 53 | link = f'{scheme}://{netloc}{path}' 54 | 55 | # Only parse links in the same domain 56 | if parsed_source.netloc not in link: 57 | continue 58 | 59 | links.append(link) 60 | 61 | return links 62 | 63 | 64 | def search_text(source_link, page, text): 65 | '''Search for an element with the searched text and print it''' 66 | for element in page.find_all(text=re.compile(text, flags=re.IGNORECASE)): 67 | print(f'Link {source_link}: --> {element}') 68 | 69 | 70 | def main(base_url, to_search): 71 | checked_links = set() 72 | to_check = [base_url] 73 | max_checks = 10 74 | 75 | while to_check and max_checks: 76 | link = to_check.pop(0) 77 | links = process_link(link, text=to_search) 78 | checked_links.add(link) 79 | for link in links: 80 | if link not in checked_links: 81 | checked_links.add(link) 82 | to_check.append(link) 83 | 84 | max_checks -= 1 85 | 86 | 87 | if __name__ == '__main__': 88 | parser = argparse.ArgumentParser() 89 | parser.add_argument(dest='url', type=str, 90 | help='Base site url. ' 91 | 'Use "http://localhost:8000/" ' 92 | 'for the recipe example') 93 | parser.add_argument('-p', type=str, 94 | help=f'Sentence to search, default: {DEFAULT_PHRASE}', 95 | default=DEFAULT_PHRASE) 96 | args = parser.parse_args() 97 | 98 | main(args.url, args.p) 99 | -------------------------------------------------------------------------------- /Chapter03/speed_up_step1.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import requests 3 | import logging 4 | import http.client 5 | import re 6 | from urllib.parse import urlparse, urljoin 7 | from bs4 import BeautifulSoup 8 | 9 | import concurrent.futures 10 | 11 | URL = 'http://localhost:8000/' 12 | DEFAULT_PHRASE = 'python' 13 | 14 | 15 | def process_link(source_link, text): 16 | logging.info(f'Extracting links from {source_link}') 17 | parsed_source = urlparse(source_link) 18 | result = requests.get(source_link) 19 | if result.status_code != http.client.OK: 20 | logging.error(f'Error retrieving {source_link}: {result}') 21 | return source_link, [] 22 | 23 | if 'html' not in result.headers['Content-type']: 24 | logging.info(f'Link {source_link} is not an HTML page') 25 | return source_link, [] 26 | 27 | page = BeautifulSoup(result.text, 'html.parser') 28 | search_text(source_link, page, text) 29 | 30 | return source_link, get_links(parsed_source, page) 31 | 32 | 33 | def get_links(parsed_source, page): 34 | '''Retrieve the links on the page''' 35 | links = [] 36 | for element in page.find_all('a'): 37 | link = element.get('href') 38 | if not link: 39 | continue 40 | 41 | # Avoid internal, same page links 42 | if link.startswith('#'): 43 | continue 44 | 45 | if link.startswith('mailto:'): 46 | # Ignore other links like mailto 47 | # More cases like ftp or similar may be included here 48 | continue 49 | 50 | # Always accept local links 51 | if not link.startswith('http'): 52 | netloc = parsed_source.netloc 53 | scheme = parsed_source.scheme 54 | path = urljoin(parsed_source.path, link) 55 | link = f'{scheme}://{netloc}{path}' 56 | 57 | # Only parse links in the same domain 58 | if parsed_source.netloc not in link: 59 | continue 60 | 61 | links.append(link) 62 | 63 | return links 64 | 65 | 66 | def search_text(source_link, page, text): 67 | '''Search for an element with the searched text and print it''' 68 | for element in page.find_all(text=re.compile(text, flags=re.IGNORECASE)): 69 | print(f'Link {source_link}: --> {element}') 70 | 71 | 72 | def main(base_url, to_search, workers): 73 | checked_links = set() 74 | to_check = [base_url] 75 | max_checks = 10 76 | 77 | with concurrent.futures.ThreadPoolExecutor(max_workers=workers) as executor: 78 | while to_check: 79 | futures = [executor.submit(process_link, url, to_search) 80 | for url in to_check] 81 | to_check = [] 82 | for data in concurrent.futures.as_completed(futures): 83 | link, new_links = data.result() 84 | 85 | checked_links.add(link) 86 | for link in new_links: 87 | if link not in checked_links and link not in to_check: 88 | to_check.append(link) 89 | 90 | max_checks -= 1 91 | if not max_checks: 92 | return 93 | 94 | 95 | if __name__ == '__main__': 96 | parser = argparse.ArgumentParser() 97 | parser.add_argument('-u', type=str, help='Base site url', 98 | default=URL) 99 | parser.add_argument('-p', type=str, help='Sentence to search', 100 | default=DEFAULT_PHRASE) 101 | parser.add_argument('-w', type=int, help='Number of workers', 102 | default=4) 103 | args = parser.parse_args() 104 | 105 | main(args.u, args.p, args.w) 106 | -------------------------------------------------------------------------------- /Chapter03/test_site/README: -------------------------------------------------------------------------------- 1 | Server this website with 2 | 3 | python3 simple_delay_server.py 4 | 5 | and then access in your browser 6 | 7 | http://localhost:8000/ 8 | -------------------------------------------------------------------------------- /Chapter03/test_site/files/5eabef23f63024c20389c34b94dee593-1.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | Another article | Untitled Page | Crawlable site 14 | 15 | 16 | 17 | 18 | 19 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 |
33 | Skip to main content 34 | 35 | 44 | 45 | 62 | 63 |
64 |
65 |
66 |
67 | 68 |
69 |

Another article

21/09/18 08:44
A smaller article , that contains a reference to Python

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Tincidunt arcu non sodales neque sodales. Quam adipiscing vitae proin sagittis nisl rhoncus. Sit amet nisl purus in mollis nunc sed id. Sit amet dictum sit amet justo donec. Accumsan lacus vel facilisis volutpat est velit egestas dui. Purus ut faucibus pulvinar elementum integer. Fermentum leo vel orci porta. Neque convallis a cras semper auctor neque vitae. Nibh ipsum consequat nisl vel pretium lectus quam id leo. Dictum sit amet justo donec enim diam vulputate ut.
70 |
71 | 72 | 73 |
74 |
75 | 76 | 85 |
86 |
87 | 88 |
89 |
90 |
91 | © 2018 Jaime Buelta 92 |
93 |
94 |
95 |
96 | 97 |
98 | 99 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | 107 | -------------------------------------------------------------------------------- /Chapter03/test_site/files/archive-september-2018.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | Archives for September 2018 | Untitled Page | Crawlable site 14 | 15 | 16 | 17 | 18 | 19 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 |
33 | Skip to main content 34 | 35 | 44 | 45 | 62 | 63 |
64 |
65 |
66 |
67 | 68 |
69 |
September 2018
70 |
Untitled Page
71 |
72 | 73 |
74 |

An uninteresting article

21/09/18 22:45

Lorem ipsum dolor sit amet, harum invenire persequeris sea te. Ne partem causae his, te partiendo consequuntur per. Case vero option mea te, mea oportere complectitur ea, ullum nobis perpetua no mel. Idque scaevola ea nam, nihil iudico virtute ad sit.

Usu ne omnes fabellas definitionem. Ne justo corrumpit vix. Natum saepe sadipscing vim no, omnium discere fabulas no sit. Copiosae lucilius et vis. Mel ad duis verear nominavi.

At ipsum regione noluisse sit, quidam assentior sea cu. Mea velit veniam ut, vero prodesset interesset per in, pro cu suas nostro. Nec persequeris necessitatibus ea, pri at ancillae rationibus, te vero soluta quo. Cu has dolores scripserit neglegentur, id per utinam aperiri salutatus.

Vocent omnesque honestatis an has. Eos ex illum bonorum torquatos. Has eu diam deserunt, usu vocibus luptatum assentior ei. Mei vidit reprehendunt eu, nam ei nobis fabulas. Sit quando posidonium no, duo ad legendos accusamus persequeris. Mei hinc dicta elaboraret ut, vim wisi etiam maluisset ne.

Nothing of interest here. Well, except for the crocodile.

Cibo offendit qui ei. Minim altera ocurreret no sit, lorem delenit vim eu. Vis ex quaeque fabulas incorrupte, ridens noluisse temporibus cu qui, lucilius expetendis elaboraret has ex. Et his vidit sonet, velit augue ignota id eam. Te ferri graece tractatos eam, vim ad tamquam sententiae.

Another article

21/09/18 08:44
A smaller article , that contains a reference to Python

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Tincidunt arcu non sodales neque sodales. Quam adipiscing vitae proin sagittis nisl rhoncus. Sit amet nisl purus in mollis nunc sed id. Sit amet dictum sit amet justo donec. Accumsan lacus vel facilisis volutpat est velit egestas dui. Purus ut faucibus pulvinar elementum integer. Fermentum leo vel orci porta. Neque convallis a cras semper auctor neque vitae. Nibh ipsum consequat nisl vel pretium lectus quam id leo. Dictum sit amet justo donec enim diam vulputate ut.

An article

21/09/18 08:43
This is an article with a lot of text

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Tincidunt arcu non sodales neque sodales. Quam adipiscing vitae proin sagittis nisl rhoncus. Sit amet nisl purus in mollis nunc sed id. Sit amet dictum sit amet justo donec. Accumsan lacus vel facilisis volutpat est velit egestas dui. Purus ut faucibus pulvinar elementum integer. Fermentum leo vel orci porta. Neque convallis a cras semper auctor neque vitae. Nibh ipsum consequat nisl vel pretium lectus quam id leo. Dictum sit amet justo donec enim diam vulputate ut. Read More…
75 |
76 | 77 | 78 |
79 |
80 | 81 | 90 |
91 |
92 | 93 |
94 |
95 |
96 | © 2018 Jaime Buelta 97 |
98 |
99 |
100 |
101 | 102 |
103 | 104 | 105 | 106 | 107 | 108 | 109 | 110 | 111 | 112 | -------------------------------------------------------------------------------- /Chapter03/test_site/files/b93bec5d9681df87e6e8d5703ed7cd81-2.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | An uninteresting article | Untitled Page | Crawlable site 14 | 15 | 16 | 17 | 18 | 19 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 |
33 | Skip to main content 34 | 35 | 44 | 45 | 62 | 63 |
64 |
65 |
66 |
67 | 68 |
69 |

An uninteresting article

21/09/18 22:45

Lorem ipsum dolor sit amet, harum invenire persequeris sea te. Ne partem causae his, te partiendo consequuntur per. Case vero option mea te, mea oportere complectitur ea, ullum nobis perpetua no mel. Idque scaevola ea nam, nihil iudico virtute ad sit.

Usu ne omnes fabellas definitionem. Ne justo corrumpit vix. Natum saepe sadipscing vim no, omnium discere fabulas no sit. Copiosae lucilius et vis. Mel ad duis verear nominavi.

At ipsum regione noluisse sit, quidam assentior sea cu. Mea velit veniam ut, vero prodesset interesset per in, pro cu suas nostro. Nec persequeris necessitatibus ea, pri at ancillae rationibus, te vero soluta quo. Cu has dolores scripserit neglegentur, id per utinam aperiri salutatus.

Vocent omnesque honestatis an has. Eos ex illum bonorum torquatos. Has eu diam deserunt, usu vocibus luptatum assentior ei. Mei vidit reprehendunt eu, nam ei nobis fabulas. Sit quando posidonium no, duo ad legendos accusamus persequeris. Mei hinc dicta elaboraret ut, vim wisi etiam maluisset ne.

Nothing of interest here. Well, except for the crocodile.

Cibo offendit qui ei. Minim altera ocurreret no sit, lorem delenit vim eu. Vis ex quaeque fabulas incorrupte, ridens noluisse temporibus cu qui, lucilius expetendis elaboraret has ex. Et his vidit sonet, velit augue ignota id eam. Te ferri graece tractatos eam, vim ad tamquam sententiae.
70 |
71 | 72 | 73 |
74 |
75 | 76 | 85 |
86 |
87 | 88 |
89 |
90 |
91 | © 2018 Jaime Buelta 92 |
93 |
94 |
95 |
96 | 97 |
98 | 99 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | 107 | -------------------------------------------------------------------------------- /Chapter03/test_site/files/meta.js: -------------------------------------------------------------------------------- 1 | (function() { 2 | var Realmac = Realmac || {}; 3 | 4 | Realmac.meta = { 5 | 6 | // Set the browser title 7 | // 8 | // @var String text 9 | setTitle: function(text) { 10 | return document.title = text; 11 | }, 12 | 13 | // Set the content attribute of a meta tag 14 | // 15 | // @var String name 16 | // @var String content 17 | setTagContent: function(tag, content){ 18 | // If the tag being set is title 19 | // return the result of setTitle 20 | if ( tag === 'title' ) 21 | { 22 | return this.setTitle(content); 23 | } 24 | 25 | // Otherwise try and find the meta tag 26 | var tag = this.getTag(tag); 27 | 28 | // If we have a tag, set the content 29 | if ( tag !== false ) 30 | { 31 | return tag.setAttribute('content', content); 32 | } 33 | 34 | return false; 35 | }, 36 | 37 | // Find a meta tag 38 | // 39 | // @var String name 40 | getTag: function(name) { 41 | var meta = document.querySelectorAll('meta'); 42 | 43 | for ( var i=0; i 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | Untitled Page | Crawlable site 14 | 15 | 16 | 17 | 18 | 19 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 |
33 | Skip to main content 34 | 35 | 44 | 45 | 62 | 63 |
64 |
65 |
66 |
67 |

An uninteresting article

21/09/18 22:45

Lorem ipsum dolor sit amet, harum invenire persequeris sea te. Ne partem causae his, te partiendo consequuntur per. Case vero option mea te, mea oportere complectitur ea, ullum nobis perpetua no mel. Idque scaevola ea nam, nihil iudico virtute ad sit.

Usu ne omnes fabellas definitionem. Ne justo corrumpit vix. Natum saepe sadipscing vim no, omnium discere fabulas no sit. Copiosae lucilius et vis. Mel ad duis verear nominavi.

At ipsum regione noluisse sit, quidam assentior sea cu. Mea velit veniam ut, vero prodesset interesset per in, pro cu suas nostro. Nec persequeris necessitatibus ea, pri at ancillae rationibus, te vero soluta quo. Cu has dolores scripserit neglegentur, id per utinam aperiri salutatus.

Vocent omnesque honestatis an has. Eos ex illum bonorum torquatos. Has eu diam deserunt, usu vocibus luptatum assentior ei. Mei vidit reprehendunt eu, nam ei nobis fabulas. Sit quando posidonium no, duo ad legendos accusamus persequeris. Mei hinc dicta elaboraret ut, vim wisi etiam maluisset ne.

Nothing of interest here. Well, except for the crocodile.

Cibo offendit qui ei. Minim altera ocurreret no sit, lorem delenit vim eu. Vis ex quaeque fabulas incorrupte, ridens noluisse temporibus cu qui, lucilius expetendis elaboraret has ex. Et his vidit sonet, velit augue ignota id eam. Te ferri graece tractatos eam, vim ad tamquam sententiae.

Another article

21/09/18 08:44
A smaller article , that contains a reference to Python

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Tincidunt arcu non sodales neque sodales. Quam adipiscing vitae proin sagittis nisl rhoncus. Sit amet nisl purus in mollis nunc sed id. Sit amet dictum sit amet justo donec. Accumsan lacus vel facilisis volutpat est velit egestas dui. Purus ut faucibus pulvinar elementum integer. Fermentum leo vel orci porta. Neque convallis a cras semper auctor neque vitae. Nibh ipsum consequat nisl vel pretium lectus quam id leo. Dictum sit amet justo donec enim diam vulputate ut.

An article

21/09/18 08:43
This is an article with a lot of text

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Tincidunt arcu non sodales neque sodales. Quam adipiscing vitae proin sagittis nisl rhoncus. Sit amet nisl purus in mollis nunc sed id. Sit amet dictum sit amet justo donec. Accumsan lacus vel facilisis volutpat est velit egestas dui. Purus ut faucibus pulvinar elementum integer. Fermentum leo vel orci porta. Neque convallis a cras semper auctor neque vitae. Nibh ipsum consequat nisl vel pretium lectus quam id leo. Dictum sit amet justo donec enim diam vulputate ut. Read More…
68 |
69 |
70 | 71 | 80 |
81 |
82 | 83 |
84 |
85 |
86 | © 2018 Jaime Buelta 87 |
88 |
89 |
90 |
91 | 92 |
93 | 94 | 95 | 96 | 97 | 98 | 99 | 100 | 101 | 102 | -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/images/fallback.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/images/fallback.jpeg -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/plugins/blog/rss.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/plugins/blog/rss.gif -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/plugins/blog/smiley_angry.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/plugins/blog/smiley_angry.png -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/plugins/blog/smiley_embarrassed.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/plugins/blog/smiley_embarrassed.png -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/plugins/blog/smiley_footinmouth.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/plugins/blog/smiley_footinmouth.png -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/plugins/blog/smiley_gasp.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/plugins/blog/smiley_gasp.png -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/plugins/blog/smiley_laugh.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/plugins/blog/smiley_laugh.png -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/plugins/blog/smiley_sad.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/plugins/blog/smiley_sad.png -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/plugins/blog/smiley_smile.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/plugins/blog/smiley_smile.png -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/plugins/blog/smiley_wink.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/plugins/blog/smiley_wink.png -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/themes/offroad/assets/fonts/FontAwesome.otf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/themes/offroad/assets/fonts/FontAwesome.otf -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/themes/offroad/assets/fonts/fontawesome-webfont.eot: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/themes/offroad/assets/fonts/fontawesome-webfont.eot -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/themes/offroad/assets/fonts/fontawesome-webfont.ttf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/themes/offroad/assets/fonts/fontawesome-webfont.ttf -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/themes/offroad/assets/fonts/fontawesome-webfont.woff: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/themes/offroad/assets/fonts/fontawesome-webfont.woff -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/themes/offroad/assets/fonts/fontawesome-webfont.woff2: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/themes/offroad/assets/fonts/fontawesome-webfont.woff2 -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/themes/offroad/assets/fonts/istok-web-v11-latin-700.eot: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/themes/offroad/assets/fonts/istok-web-v11-latin-700.eot -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/themes/offroad/assets/fonts/istok-web-v11-latin-700.ttf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/themes/offroad/assets/fonts/istok-web-v11-latin-700.ttf -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/themes/offroad/assets/fonts/istok-web-v11-latin-700.woff2: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/themes/offroad/assets/fonts/istok-web-v11-latin-700.woff2 -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/themes/offroad/assets/fonts/istok-web-v11-latin-regular.eot: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/themes/offroad/assets/fonts/istok-web-v11-latin-regular.eot -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/themes/offroad/assets/fonts/istok-web-v11-latin-regular.ttf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/themes/offroad/assets/fonts/istok-web-v11-latin-regular.ttf -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/themes/offroad/assets/fonts/istok-web-v11-latin-regular.woff: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/themes/offroad/assets/fonts/istok-web-v11-latin-regular.woff -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/themes/offroad/assets/fonts/istok-web-v11-latin-regular.woff2: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/themes/offroad/assets/fonts/istok-web-v11-latin-regular.woff2 -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/themes/offroad/assets/fonts/lora-v12-latin-700.eot: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/themes/offroad/assets/fonts/lora-v12-latin-700.eot -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/themes/offroad/assets/fonts/lora-v12-latin-700.ttf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/themes/offroad/assets/fonts/lora-v12-latin-700.ttf -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/themes/offroad/assets/fonts/lora-v12-latin-700.woff: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/themes/offroad/assets/fonts/lora-v12-latin-700.woff -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/themes/offroad/assets/fonts/lora-v12-latin-700.woff2: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/themes/offroad/assets/fonts/lora-v12-latin-700.woff2 -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/themes/offroad/assets/fonts/lora-v12-latin-700italic.eot: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/themes/offroad/assets/fonts/lora-v12-latin-700italic.eot -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/themes/offroad/assets/fonts/lora-v12-latin-700italic.ttf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/themes/offroad/assets/fonts/lora-v12-latin-700italic.ttf -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/themes/offroad/assets/fonts/lora-v12-latin-700italic.woff: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/themes/offroad/assets/fonts/lora-v12-latin-700italic.woff -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/themes/offroad/assets/fonts/lora-v12-latin-700italic.woff2: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/themes/offroad/assets/fonts/lora-v12-latin-700italic.woff2 -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/themes/offroad/assets/fonts/lora-v12-latin-italic.eot: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/themes/offroad/assets/fonts/lora-v12-latin-italic.eot -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/themes/offroad/assets/fonts/lora-v12-latin-italic.ttf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/themes/offroad/assets/fonts/lora-v12-latin-italic.ttf -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/themes/offroad/assets/fonts/lora-v12-latin-italic.woff: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/themes/offroad/assets/fonts/lora-v12-latin-italic.woff -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/themes/offroad/assets/fonts/lora-v12-latin-italic.woff2: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/themes/offroad/assets/fonts/lora-v12-latin-italic.woff2 -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/themes/offroad/assets/fonts/lora-v12-latin-regular.eot: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/themes/offroad/assets/fonts/lora-v12-latin-regular.eot -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/themes/offroad/assets/fonts/lora-v12-latin-regular.ttf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/themes/offroad/assets/fonts/lora-v12-latin-regular.ttf -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/themes/offroad/assets/fonts/lora-v12-latin-regular.woff: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/themes/offroad/assets/fonts/lora-v12-latin-regular.woff -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/themes/offroad/assets/fonts/lora-v12-latin-regular.woff2: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/themes/offroad/assets/fonts/lora-v12-latin-regular.woff2 -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/themes/offroad/assets/images/fallback.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter03/test_site/rw_common/themes/offroad/assets/images/fallback.jpeg -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/themes/offroad/assets/javascript/background-blur.js: -------------------------------------------------------------------------------- 1 | 2 | $(document).ready(function(){$('
').insertAfter('.body-overlay') 3 | $(window).scroll(function(){var opacity=($(window).scrollTop()/300) 4 | $('.blurred').css('opacity',opacity)})}) -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/themes/offroad/assets/javascript/background-no-blur.js: -------------------------------------------------------------------------------- 1 | 2 | $(document).ready(function(){$('.blurred').remove()}) -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/themes/offroad/assets/javascript/html5shiv.js: -------------------------------------------------------------------------------- 1 | !function(e,t){function n(e,t){var n=e.createElement("p"),r=e.getElementsByTagName("head")[0]||e.documentElement;return n.innerHTML="x",r.insertBefore(n.lastChild,r.firstChild)}function r(){var e=y.elements;return"string"==typeof e?e.split(" "):e}function a(e,t){var n=y.elements;"string"!=typeof n&&(n=n.join(" ")),"string"!=typeof e&&(e=e.join(" ")),y.elements=n+" "+e,m(t)}function c(e){var t=E[e[p]];return t||(t={},v++,e[p]=v,E[v]=t),t}function o(e,n,r){if(n||(n=t),u)return n.createElement(e);r||(r=c(n));var a;return a=r.cache[e]?r.cache[e].cloneNode():g.test(e)?(r.cache[e]=r.createElem(e)).cloneNode():r.createElem(e),!a.canHaveChildren||f.test(e)||a.tagUrn?a:r.frag.appendChild(a)}function i(e,n){if(e||(e=t),u)return e.createDocumentFragment();n=n||c(e);for(var a=n.frag.cloneNode(),o=0,i=r(),l=i.length;l>o;o++)a.createElement(i[o]);return a}function l(e,t){t.cache||(t.cache={},t.createElem=e.createElement,t.createFrag=e.createDocumentFragment,t.frag=t.createFrag()),e.createElement=function(n){return y.shivMethods?o(n,e,t):t.createElem(n)},e.createDocumentFragment=Function("h,f","return function(){var n=f.cloneNode(),c=n.createElement;h.shivMethods&&("+r().join().replace(/[\w\-:]+/g,function(e){return t.createElem(e),t.frag.createElement(e),'c("'+e+'")'})+");return n}")(y,t.frag)}function m(e){e||(e=t);var r=c(e);return!y.shivCSS||s||r.hasCSS||(r.hasCSS=!!n(e,"article,aside,dialog,figcaption,figure,footer,header,hgroup,main,nav,section{display:block}mark{background:#FF0;color:#000}template{display:none}")),u||l(e,r),e}var s,u,d="3.7.2",h=e.html5||{},f=/^<|^(?:button|map|select|textarea|object|iframe|option|optgroup)$/i,g=/^(?:a|b|code|div|fieldset|h1|h2|h3|h4|h5|h6|i|label|li|ol|p|q|span|strong|style|table|tbody|td|th|tr|ul)$/i,p="_html5shiv",v=0,E={};!function(){try{var e=t.createElement("a");e.innerHTML="",s="hidden"in e,u=1==e.childNodes.length||function(){t.createElement("a");var e=t.createDocumentFragment();return"undefined"==typeof e.cloneNode||"undefined"==typeof e.createDocumentFragment||"undefined"==typeof e.createElement}()}catch(n){s=!0,u=!0}}();var y={elements:h.elements||"abbr article aside audio bdi canvas data datalist details dialog figcaption figure footer header hgroup main mark meter nav output picture progress section summary template time video",version:d,shivCSS:h.shivCSS!==!1,supportsUnknownElements:u,shivMethods:h.shivMethods!==!1,type:"default",shivDocument:m,createElement:o,createDocumentFragment:i,addElements:a};e.html5=y,m(t)}(this,document); -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/themes/offroad/assets/javascript/min/background-dont-blur-min.js: -------------------------------------------------------------------------------- 1 | 2 | $(document).ready(function(){$(".blurred").remove()}); -------------------------------------------------------------------------------- /Chapter03/test_site/rw_common/themes/offroad/assets/javascript/respond.js: -------------------------------------------------------------------------------- 1 | !function(e){"use strict";e.matchMedia=e.matchMedia||function(e){var t,n=e.documentElement,a=n.firstElementChild||n.firstChild,s=e.createElement("body"),i=e.createElement("div");return i.id="mq-test-1",i.style.cssText="position:absolute;top:-100em",s.style.background="none",s.appendChild(i),function(e){return i.innerHTML='­',n.insertBefore(s,a),t=42===i.offsetWidth,n.removeChild(s),{matches:t,media:e}}}(e.document)}(this),function(e){"use strict";function t(){E(!0)}var n={};e.respond=n,n.update=function(){};var a=[],s=function(){var t=!1;try{t=new e.XMLHttpRequest}catch(n){t=new e.ActiveXObject("Microsoft.XMLHTTP")}return function(){return t}}(),i=function(e,t){var n=s();n&&(n.open("GET",e,!0),n.onreadystatechange=function(){4!==n.readyState||200!==n.status&&304!==n.status||t(n.responseText)},4!==n.readyState&&n.send(null))};if(n.ajax=i,n.queue=a,n.regex={media:/@media[^\{]+\{([^\{\}]*\{[^\}\{]*\})+/gi,keyframes:/@(?:\-(?:o|moz|webkit)\-)?keyframes[^\{]+\{(?:[^\{\}]*\{[^\}\{]*\})+[^\}]*\}/gi,urls:/(url\()['"]?([^\/\)'"][^:\)'"]+)['"]?(\))/g,findStyles:/@media *([^\{]+)\{([\S\s]+?)$/,only:/(only\s+)?([a-zA-Z]+)\s?/,minw:/\([\s]*min\-width\s*:[\s]*([\s]*[0-9\.]+)(px|em)[\s]*\)/,maxw:/\([\s]*max\-width\s*:[\s]*([\s]*[0-9\.]+)(px|em)[\s]*\)/},n.mediaQueriesSupported=e.matchMedia&&null!==e.matchMedia("only all")&&e.matchMedia("only all").matches,!n.mediaQueriesSupported){var r,o,l,m=e.document,d=m.documentElement,h=[],u=[],c=[],f={},p=30,y=m.getElementsByTagName("head")[0]||d,g=m.getElementsByTagName("base")[0],x=y.getElementsByTagName("link"),v=function(){var e,t=m.createElement("div"),n=m.body,a=d.style.fontSize,s=n&&n.style.fontSize,i=!1;return t.style.cssText="position:absolute;font-size:1em;width:1em",n||(n=i=m.createElement("body"),n.style.background="none"),d.style.fontSize="100%",n.style.fontSize="100%",n.appendChild(t),i&&d.insertBefore(n,d.firstChild),e=t.offsetWidth,i?d.removeChild(n):n.removeChild(t),d.style.fontSize=a,s&&(n.style.fontSize=s),e=l=parseFloat(e)},E=function(t){var n="clientWidth",a=d[n],s="CSS1Compat"===m.compatMode&&a||m.body[n]||a,i={},f=x[x.length-1],g=(new Date).getTime();if(t&&r&&p>g-r)return e.clearTimeout(o),void(o=e.setTimeout(E,p));r=g;for(var w in h)if(h.hasOwnProperty(w)){var S=h[w],T=S.minw,C=S.maxw,b=null===T,z=null===C,M="em";T&&(T=parseFloat(T)*(T.indexOf(M)>-1?l||v():1)),C&&(C=parseFloat(C)*(C.indexOf(M)>-1?l||v():1)),S.hasquery&&(b&&z||!(b||s>=T)||!(z||C>=s))||(i[S.media]||(i[S.media]=[]),i[S.media].push(u[S.rules]))}for(var R in c)c.hasOwnProperty(R)&&c[R]&&c[R].parentNode===y&&y.removeChild(c[R]);c.length=0;for(var O in i)if(i.hasOwnProperty(O)){var k=m.createElement("style"),q=i[O].join("\n");k.type="text/css",k.media=O,y.insertBefore(k,f.nextSibling),k.styleSheet?k.styleSheet.cssText=q:k.appendChild(m.createTextNode(q)),c.push(k)}},w=function(e,t,a){var s=e.replace(n.regex.keyframes,"").match(n.regex.media),i=s&&s.length||0;t=t.substring(0,t.lastIndexOf("/"));var r=function(e){return e.replace(n.regex.urls,"$1"+t+"$2$3")},o=!i&&a;t.length&&(t+="/"),o&&(i=1);for(var l=0;i>l;l++){var m,d,c,f;o?(m=a,u.push(r(e))):(m=s[l].match(n.regex.findStyles)&&RegExp.$1,u.push(RegExp.$2&&r(RegExp.$2))),c=m.split(","),f=c.length;for(var p=0;f>p;p++)d=c[p],h.push({media:d.split("(")[0].match(n.regex.only)&&RegExp.$2||"all",rules:u.length-1,hasquery:d.indexOf("(")>-1,minw:d.match(n.regex.minw)&&parseFloat(RegExp.$1)+(RegExp.$2||""),maxw:d.match(n.regex.maxw)&&parseFloat(RegExp.$1)+(RegExp.$2||"")})}E()},S=function(){if(a.length){var t=a.shift();i(t.href,function(n){w(n,t.href,t.media),f[t.href]=!0,e.setTimeout(function(){S()},0)})}},T=function(){for(var t=0;t to stop') 25 | server.serve_forever() 26 | -------------------------------------------------------------------------------- /Chapter03/test_site/sitemap.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | http://localhost:8000/index.html 5 | 2018-09-21 6 | 7 | 8 | http://localhost:8000/files/b93bec5d9681df87e6e8d5703ed7cd81-2.html 9 | 2018-09-21 10 | 11 | 12 | http://localhost:8000/files/5eabef23f63024c20389c34b94dee593-1.html 13 | 2018-09-21 14 | 15 | 16 | http://localhost:8000/files/33714fc865e02aeda2dabb9a42a787b2-0.html 17 | 2018-09-21 18 | 19 | 20 | http://localhost:8000/files/archive-september-2018.html 21 | 2018-09-21 22 | 23 | 24 | -------------------------------------------------------------------------------- /Chapter04/documents/dir/file1.txt: -------------------------------------------------------------------------------- 1 | This is a test file 2 | -------------------------------------------------------------------------------- /Chapter04/documents/dir/file2.txt: -------------------------------------------------------------------------------- 1 | This is a test file 2 | -------------------------------------------------------------------------------- /Chapter04/documents/dir/file6.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter04/documents/dir/file6.pdf -------------------------------------------------------------------------------- /Chapter04/documents/dir/subdir/file3.txt: -------------------------------------------------------------------------------- 1 | This is a test file 2 | -------------------------------------------------------------------------------- /Chapter04/documents/dir/subdir/file4.txt: -------------------------------------------------------------------------------- 1 | This is a test file 2 | -------------------------------------------------------------------------------- /Chapter04/documents/dir/subdir/file5.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter04/documents/dir/subdir/file5.pdf -------------------------------------------------------------------------------- /Chapter04/documents/document-1.docx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter04/documents/document-1.docx -------------------------------------------------------------------------------- /Chapter04/documents/document-1.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter04/documents/document-1.pdf -------------------------------------------------------------------------------- /Chapter04/documents/document-2.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter04/documents/document-2.pdf -------------------------------------------------------------------------------- /Chapter04/documents/example_iso.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter04/documents/example_iso.txt -------------------------------------------------------------------------------- /Chapter04/documents/example_logs.log: -------------------------------------------------------------------------------- 1 | [2018-06-17T22:11:50.268396] - SALE - PRODUCT: 1489 - PRICE: $09.99 2 | [2018-06-17T22:11:50.268442] - SALE - PRODUCT: 4508 - PRICE: $05.30 3 | [2018-06-17T22:11:50.268454] - SALE - PRODUCT: 8597 - PRICE: $15.49 4 | [2018-06-17T22:11:50.268461] - SALE - PRODUCT: 3086 - PRICE: $07.05 5 | [2018-06-17T22:11:50.268468] - SALE - PRODUCT: 1489 - PRICE: $09.99 6 | -------------------------------------------------------------------------------- /Chapter04/documents/example_output_iso.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter04/documents/example_output_iso.txt -------------------------------------------------------------------------------- /Chapter04/documents/example_utf8.txt: -------------------------------------------------------------------------------- 1 | 20£ 2 | -------------------------------------------------------------------------------- /Chapter04/documents/top_films.csv: -------------------------------------------------------------------------------- 1 | Rank,"Admissions 2 | (millions)",Title (year) (studio),Director(s) 3 | 1,225.7,Gone With the Wind (1939) (MGM),"Victor Fleming, George Cukor, Sam Wood" 4 | 2,194.4,Star Wars (Ep. IV: A New Hope) (1977) (Fox),George Lucas 5 | 3,161.0,ET: The Extra-Terrestrial (1982) (Univ),Steven Spielberg 6 | 4,156.4,The Sound of Music (1965) (Fox),Robert Wise 7 | 5,130.0,The Ten Commandments (1956) (Para),Cecil B. DeMille 8 | 6,128.4,Titanic (1997) (Fox),James Cameron 9 | 7,126.3,Snow White and the Seven Dwarfs (1937) (BV),David Hand 10 | 8,120.7,Jaws (1975) (Univ),Steven Spielberg 11 | 9,120.1,Doctor Zhivago (1965) (MGM),David Lean 12 | 10,118.9,The Lion King (1994) (BV),"Roger Allers, Rob Minkoff" -------------------------------------------------------------------------------- /Chapter04/documents/zen_of_python.txt: -------------------------------------------------------------------------------- 1 | The Zen of Python, by Tim Peters 2 | 3 | Beautiful is better than ugly. 4 | Explicit is better than implicit. 5 | Simple is better than complex. 6 | Complex is better than complicated. 7 | Flat is better than nested. 8 | Sparse is better than dense. 9 | Readability counts. 10 | Special cases aren't special enough to break the rules. 11 | Although practicality beats purity. 12 | Errors should never pass silently. 13 | Unless explicitly silenced. 14 | In the face of ambiguity, refuse the temptation to guess. 15 | There should be one-- and preferably only one --obvious way to do it. 16 | Although that way may not be obvious at first unless you're Dutch. 17 | Now is better than never. 18 | Although never is often better than *right* now. 19 | If the implementation is hard to explain, it's a bad idea. 20 | If the implementation is easy to explain, it may be a good idea. 21 | Namespaces are one honking great idea -- let's do more of those! -------------------------------------------------------------------------------- /Chapter04/gps_conversion.py: -------------------------------------------------------------------------------- 1 | import re 2 | 3 | 4 | def tuple_to_decimal(gps_tuple): 5 | ''' 6 | The definition of the tuple is 7 | 8 | degrees, minutes, seconds 9 | 10 | Each one has a number and a scale, e.g. seconds can be 11 | 12 | (3456, 1000) 13 | 14 | meaning it needs to be divided by that number. 15 | ''' 16 | degrees_info, minutes_info, seconds_info = gps_tuple 17 | 18 | degrees = degrees_info[0] / degrees_info[1] 19 | minutes = minutes_info[0] / minutes_info[1] 20 | seconds = seconds_info[0] / seconds_info[1] 21 | 22 | return degrees + minutes / 60 + seconds / 3600 23 | 24 | 25 | def ddm_to_decimal(gps_ddm): 26 | ''' 27 | DDM format is defined as a string, which includes the reference 28 | and divides degrees and minutes by a comma. Minutes are 29 | defined with decimal points. E.g. 30 | 31 | DD,MMM.MMMMR 32 | 33 | Being R the reference. 34 | 35 | No seconds are included, as the minutes are given including 36 | decimal points. 37 | ''' 38 | match = re.match(r'(\d+),([\d.]+)(N|S|E|W)', gps_ddm) 39 | degrees, dminutes, ref = match.groups() 40 | 41 | decimal = float(degrees) + float(dminutes) / 60 42 | return f'{ref}{decimal}' 43 | 44 | 45 | def exif_to_decimal(exif_info): 46 | latitude = tuple_to_decimal(exif_info['GPSLatitude']) 47 | latref = exif_info['GPSLatitudeRef'] 48 | longitude = tuple_to_decimal(exif_info['GPSLongitude']) 49 | longref = exif_info['GPSLongitudeRef'] 50 | 51 | return f'{latref}{latitude}', f'{longref}{longitude}' 52 | 53 | 54 | def rdf_to_decimal(rdf_info): 55 | latitude = ddm_to_decimal(rdf_info['exif:GPSLatitude']) 56 | longitude = ddm_to_decimal(rdf_info['exif:GPSLongitude']) 57 | 58 | return latitude, longitude 59 | -------------------------------------------------------------------------------- /Chapter04/images/photo-dublin-a-text.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter04/images/photo-dublin-a-text.jpg -------------------------------------------------------------------------------- /Chapter04/images/photo-dublin-a1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter04/images/photo-dublin-a1.jpg -------------------------------------------------------------------------------- /Chapter04/images/photo-dublin-a2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter04/images/photo-dublin-a2.png -------------------------------------------------------------------------------- /Chapter04/images/photo-dublin-b.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter04/images/photo-dublin-b.png -------------------------------------------------------------------------------- /Chapter04/images/photo-text.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter04/images/photo-text.jpg -------------------------------------------------------------------------------- /Chapter04/scan.py: -------------------------------------------------------------------------------- 1 | 2 | import os 3 | import argparse 4 | import csv 5 | import docx 6 | from bs4 import UnicodeDammit 7 | from PyPDF2 import PdfFileReader 8 | 9 | 10 | def search_txt(filename, word): 11 | ''' 12 | Search the word in a text file 13 | ''' 14 | # Detect the encoding 15 | with open(filename, 'rb') as file: 16 | content = file.read(1024) 17 | 18 | suggestion = UnicodeDammit(content) 19 | encoding = suggestion.original_encoding 20 | 21 | # Open and read 22 | with open(filename, encoding=encoding) as file: 23 | for line in file: 24 | if word in line.lower(): 25 | return True 26 | 27 | return False 28 | 29 | 30 | def search_csv(filename, word): 31 | ''' 32 | Search the word in a text file 33 | ''' 34 | with open(filename) as file: 35 | for row in csv.reader(file): 36 | for column in row: 37 | if word in column.lower(): 38 | return True 39 | 40 | return False 41 | 42 | 43 | def search_pdf(filename, word): 44 | ''' 45 | Search the word in a PDF file 46 | ''' 47 | with open(filename, 'rb') as file: 48 | document = PdfFileReader(file) 49 | if document.isEncrypted: 50 | return False 51 | for page in document.pages: 52 | text = page.extractText() 53 | if word in text.lower(): 54 | return True 55 | 56 | return False 57 | 58 | 59 | def search_docx(filename, word): 60 | ''' 61 | Search for the word in a Word document 62 | ''' 63 | doc = docx.Document(filename) 64 | for paragraph in doc.paragraphs: 65 | if word in paragraph.text.lower(): 66 | return True 67 | 68 | return False 69 | 70 | 71 | EXTENSIONS = { 72 | 'txt': search_txt, 73 | 'csv': search_csv, 74 | 'pdf': search_pdf, 75 | 'docx': search_docx, 76 | } 77 | 78 | 79 | def main(word): 80 | ''' 81 | Open the current directory and search in all the files 82 | ''' 83 | for root, dirs, files in os.walk('.'): 84 | for file in files: 85 | # Obtain the extension 86 | extension = file.split('.')[-1] 87 | if extension in EXTENSIONS: 88 | search_file = EXTENSIONS.get(extension) 89 | full_file_path = os.path.join(root, file) 90 | if search_file(full_file_path, word): 91 | print(f'>>> Word found in {full_file_path}') 92 | 93 | 94 | if __name__ == '__main__': 95 | parser = argparse.ArgumentParser() 96 | parser.add_argument('-w', type=str, help='Word to search', default='the') 97 | args = parser.parse_args() 98 | 99 | # standarize to lowercase 100 | main(args.w.lower()) 101 | -------------------------------------------------------------------------------- /Chapter05/jinja_template.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | Movies Report 5 | 6 | 7 |

Movies Report

8 |

Date {{date}}

9 |

Movies seen in the last 30 days: {{movies|length}}

10 |
    11 | {% for movie in movies %} 12 |
  1. {{movie}}
    2. 13 | {% endfor %} 14 |
15 |

Total minutes: {{total_minutes}}

16 | 17 | 18 | -------------------------------------------------------------------------------- /Chapter05/markdown_template.md: -------------------------------------------------------------------------------- 1 | Movies Report 2 | ======= 3 | 4 | Date: {date} 5 | 6 | Movies seen in the last 30 days: {num_movies} 7 | 8 | {movies} 9 | 10 | Total minutes: {total_minutes} 11 | -------------------------------------------------------------------------------- /Chapter05/structuring_pdf.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import fpdf 4 | from random import randint 5 | 6 | 7 | class StructuredPDF(fpdf.FPDF): 8 | 9 | LINE_HEIGHT = 5 10 | 11 | def footer(self): 12 | self.set_y(-15) 13 | self.set_font('Times', 'I', 8) 14 | # Page number. Notice the {nb} which will be replaced 15 | # The double curly brackes will be replaced by a single one 16 | page_number = 'Page {number}/{{nb}}'.format(number=self.page_no()) 17 | self.cell(0, self.LINE_HEIGHT, page_number, 0, 0, 'R') 18 | 19 | def chapter(self, title, paragraphs): 20 | self.add_page() 21 | link = self.title_text(title) 22 | page = self.page_no() 23 | 24 | for paragraph in paragraphs: 25 | self.multi_cell(0, self.LINE_HEIGHT, paragraph) 26 | self.ln() 27 | 28 | return link, page 29 | 30 | def title_text(self, title): 31 | self.set_font('Times', 'B', 15) 32 | self.cell(0, self.LINE_HEIGHT, title) 33 | self.set_font('Times', '', 12) 34 | self.line(10, 17, 110, 17) 35 | link = self.add_link() 36 | self.set_link(link) 37 | self.ln() 38 | self.ln() 39 | 40 | return link 41 | 42 | def get_full_line(self, head, tail, fill): 43 | ''' 44 | It returns the line up to the width with the proper number 45 | of fill elements. 46 | ''' 47 | WIDTH = 120 48 | width = 0 49 | number = 1 50 | while width < WIDTH: 51 | number += 1 52 | line = '{} '.format(head) + '.' * number + ' {}'.format(tail) 53 | width = self.get_string_width(line) 54 | 55 | return line 56 | 57 | def toc(self, links): 58 | self.add_page() 59 | self.title_text('Table of contents') 60 | self.set_font('Times', 'I', 12) 61 | 62 | for title, page, link in links: 63 | line = self.get_full_line(title, page, '.') 64 | self.cell(0, self.LINE_HEIGHT, line, link=link) 65 | self.ln() 66 | 67 | 68 | LOREM_IPSUM = ('Lorem ipsum dolor sit amet, consectetur adipiscing elit. ' 69 | 'Donec a diam sem. Sed ac nulla consequat, tempus tortor eget, ' 70 | 'fermentum turpis. Class aptent taciti sociosqu ad litora ' 71 | 'torquent per conubia nostra, per inceptos himenaeos. Fusce ' 72 | 'fermentum nibh ligula, sed dignissim risus hendrerit mollis. ' 73 | 'Fusce aliquam semper odio, in convallis mi sagittis et. Proin ' 74 | 'ac neque non massa lobortis maximus a quis turpis. Vestibulum ' 75 | 'vitae justo elit. Fusce hendrerit, libero in auctor auctor, ' 76 | 'risus velit fermentum dui, sed placerat urna augue vel lorem.' 77 | ' Praesent in enim porta, blandit lorem vulputate, semper ' 78 | 'nulla. Duis placerat neque vitae magna pulvinar elementum. ' 79 | 'Proin in velit pellentesque, tempus dolor vel, tincidunt ' 80 | 'turpis. Quisque vel sem metus. Nullam aliquet risus vel arcu ' 81 | 'tempus elementum.') 82 | 83 | 84 | def main(): 85 | document = StructuredPDF() 86 | document.alias_nb_pages() 87 | links = [] 88 | num_chapters = randint(5, 40) 89 | for index in range(1, num_chapters): 90 | chapter_title = 'Chapter {}'.format(index) 91 | num_paragraphs = randint(10, 15) 92 | link, page = document.chapter(chapter_title, 93 | [LOREM_IPSUM] * num_paragraphs) 94 | links.append((chapter_title, page, link)) 95 | 96 | document.toc(links) 97 | 98 | document.output('report.pdf') 99 | 100 | 101 | if __name__ == '__main__': 102 | main() 103 | -------------------------------------------------------------------------------- /Chapter05/watermarking_pdf.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import os 3 | from datetime import datetime 4 | from PIL import Image, ImageDraw, ImageFont 5 | import PyPDF2 6 | from pdf2image import convert_from_path 7 | 8 | DEFAULT_OUTPUT = 'watermarked.pdf' 9 | DEFAULT_BY = 'default user' 10 | INTERMEDIATE_ENCRYPT_FILE = 'temp.pdf' 11 | 12 | WATERMARK_SIZE = (200, 200) 13 | 14 | 15 | def encrypt(out_pdf, password): 16 | print('Encrypting the document') 17 | 18 | output_pdf = PyPDF2.PdfFileWriter() 19 | 20 | in_file = open(out_pdf, "rb") 21 | input_pdf = PyPDF2.PdfFileReader(in_file) 22 | output_pdf.appendPagesFromReader(input_pdf) 23 | output_pdf.encrypt(password) 24 | 25 | # Intermediate file 26 | with open(INTERMEDIATE_ENCRYPT_FILE, "wb") as out_file: 27 | output_pdf.write(out_file) 28 | 29 | in_file.close() 30 | 31 | # Rename the intermediate file 32 | os.rename(INTERMEDIATE_ENCRYPT_FILE, out_pdf) 33 | 34 | 35 | def create_watermark(watermarked_by): 36 | print('Creating a watermark') 37 | mask = Image.new('L', WATERMARK_SIZE, 0) 38 | draw = ImageDraw.Draw(mask) 39 | font = ImageFont.load_default() 40 | text = 'WATERMARKED BY {}\n{}'.format(watermarked_by, datetime.now()) 41 | draw.multiline_text((0, 100), text, 55, font=font) 42 | 43 | watermark = Image.new('RGB', WATERMARK_SIZE) 44 | watermark.putalpha(mask) 45 | watermark = watermark.resize((1950, 1950)) 46 | watermark = watermark.rotate(45) 47 | # Crop to only the watermark 48 | bbox = watermark.getbbox() 49 | watermark = watermark.crop(bbox) 50 | 51 | return watermark 52 | 53 | 54 | def apply_watermark(watermark, in_pdf, out_pdf): 55 | print('Watermarking the document') 56 | # Transform from PDF to images 57 | images = convert_from_path(in_pdf) 58 | 59 | # Get the location for the watermark 60 | hi, wi = images[0].size 61 | hw, ww = watermark.size 62 | position = ((hi - hw) // 2, (wi - ww) // 2) 63 | 64 | # Paste the watermark in each page 65 | for image in images: 66 | image.paste(watermark, position, watermark) 67 | 68 | # Save the resulting PDF 69 | images[0].save(out_pdf, save_all=True, append_images=images[1:]) 70 | 71 | 72 | def main(in_pdf, out_pdf, watermarked_by, password): 73 | 74 | watermark = create_watermark(watermarked_by) 75 | apply_watermark(watermark, in_pdf, out_pdf) 76 | 77 | if password: 78 | encrypt(out_pdf, password) 79 | 80 | 81 | if __name__ == '__main__': 82 | parser = argparse.ArgumentParser() 83 | parser.add_argument(dest='pdf', type=str, 84 | help='PDF to watermark') 85 | parser.add_argument('-o', type=str, 86 | help=f'Output PDF filename, default: {DEFAULT_OUTPUT}', 87 | default=DEFAULT_OUTPUT) 88 | parser.add_argument('-u', type=str, 89 | help=f'Watermarked by, default: {DEFAULT_BY}', 90 | default=DEFAULT_BY) 91 | parser.add_argument('-p', type=str, 92 | help='Password') 93 | args = parser.parse_args() 94 | main(args.pdf, args.o, args.u, args.p) 95 | -------------------------------------------------------------------------------- /Chapter06/include_macro.py: -------------------------------------------------------------------------------- 1 | import zipfile 2 | import shutil 3 | import os 4 | import argparse 5 | OUTPUT_DIR = 'macro_file' 6 | 7 | 8 | def main(spreadsheet, script): 9 | print("Delete and create directory with_macro") 10 | shutil.rmtree(OUTPUT_DIR, True) 11 | os.mkdir(OUTPUT_DIR) 12 | 13 | filename = OUTPUT_DIR + '/' + spreadsheet 14 | print("Open file " + spreadsheet) 15 | shutil.copyfile(spreadsheet, filename) 16 | 17 | doc = zipfile.ZipFile(filename, 'a') 18 | doc.write(script, 'Scripts/python/' + script) 19 | manifest = [] 20 | for line in doc.open('META-INF/manifest.xml'): 21 | if '' in line.decode('utf-8'): 22 | for path in ['Scripts/', 'Scripts/python/', 23 | 'Scripts/python/' + script]: 24 | man_line = (' ') 27 | manifest.append(man_line) 28 | manifest.append(line.decode('utf-8')) 29 | 30 | doc.writestr('META-INF/manifest.xml', ''.join(manifest)) 31 | doc.close() 32 | print("File created: " + filename) 33 | 34 | 35 | if __name__ == '__main__': 36 | parser = argparse.ArgumentParser('It inserts the macro file "script" ' 37 | 'into the file "spreadsheet" in .ods ' 38 | 'format. The resulting file is located ' 39 | f'in the {OUTPUT_DIR} directory, that ' 40 | 'will be created') 41 | parser.add_argument(dest='spreadsheet', type=str, 42 | help='File to insert the script') 43 | parser.add_argument(dest='script', type=str, 44 | help='Script to insert in the file') 45 | args = parser.parse_args() 46 | main(args.spreadsheet, args.script) 47 | -------------------------------------------------------------------------------- /Chapter06/libreoffice_script.py: -------------------------------------------------------------------------------- 1 | 2 | 3 | def ObtainAggregated(*args): 4 | # get the doc from the scripting context 5 | # which is made available to all scripts 6 | desktop = XSCRIPTCONTEXT.getDesktop() 7 | model = desktop.getCurrentComponent() 8 | 9 | # get the first sheet 10 | sheet = model.Sheets.getByIndex(0) 11 | 12 | # Find the admissions column 13 | MAX_ELEMENT = 20 14 | for column in range(0, MAX_ELEMENT): 15 | cell = sheet.getCellByPosition(column, 0) 16 | if 'Admissions' in cell.String: 17 | break 18 | else: 19 | raise Exception('Admissions not found') 20 | 21 | accumulator = 0.0 22 | for row in range(1, MAX_ELEMENT): 23 | cell = sheet.getCellByPosition(column, row) 24 | value = cell.getValue() 25 | if value: 26 | accumulator += cell.getValue() 27 | else: 28 | break 29 | 30 | cell = sheet.getCellByPosition(column, row) 31 | cell.setValue(accumulator) 32 | 33 | cell = sheet.getCellRangeByName("A15") 34 | cell.String = 'Total' 35 | return None 36 | -------------------------------------------------------------------------------- /Chapter06/movies.csv: -------------------------------------------------------------------------------- 1 | Admissions,Name,Year 2 | 225.7,Gone With the Wind,1939 3 | 194.4,Star Wars,1977 4 | 161.0,ET: The Extra-Terrestrial,1982 5 | -------------------------------------------------------------------------------- /Chapter06/movies.ods: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter06/movies.ods -------------------------------------------------------------------------------- /Chapter06/movies.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter06/movies.xlsx -------------------------------------------------------------------------------- /Chapter07/aggregate_OH.csv: -------------------------------------------------------------------------------- 1 | DATE,TOTAL USD,NUMBER,AVERAGE 2 | 2019-08-27,32.93,7,4.70 3 | 2019-08-28,114.76,24,4.78 4 | 2019-08-29,111.76,24,4.66 5 | 2019-08-30,113.76,24,4.74 6 | 2019-08-31,135.76,24,5.66 7 | 2019-09-01,112.76,24,4.70 8 | 2019-09-02,126.76,24,5.28 9 | 2019-09-03,114.76,24,4.78 10 | 2019-09-04,115.76,24,4.82 11 | 2019-09-05,100.76,24,4.20 12 | 2019-09-06,119.76,24,4.99 13 | 2019-09-07,112.76,24,4.70 14 | 2019-09-08,97.76,24,4.07 15 | 2019-09-09,102.76,24,4.28 16 | 2019-09-10,111.76,24,4.66 17 | 2019-09-11,107.76,24,4.49 18 | 2019-09-12,105.76,24,4.41 19 | 2019-09-13,114.76,24,4.78 20 | 2019-09-14,88.76,24,3.70 21 | 2019-09-15,119.76,24,4.99 22 | 2019-09-16,139.76,24,5.82 23 | 2019-09-17,98.76,24,4.12 24 | 2019-09-18,111.76,24,4.66 25 | 2019-09-19,107.76,24,4.49 26 | 2019-09-20,98.76,24,4.12 27 | 2019-09-21,107.76,24,4.49 28 | 2019-09-22,107.76,24,4.49 29 | 2019-09-23,98.76,24,4.12 30 | 2019-09-24,109.76,24,4.57 31 | 2019-09-25,100.76,24,4.20 32 | 2019-09-26,130.76,24,5.45 33 | 2019-09-27,107.76,24,4.49 34 | 2019-09-28,103.76,24,4.32 35 | 2019-09-29,128.76,24,5.36 36 | 2019-09-30,117.76,24,4.91 37 | 2019-10-01,133.76,24,5.57 38 | 2019-10-02,90.76,24,3.78 39 | 2019-10-03,100.76,24,4.20 40 | 2019-10-04,133.76,24,5.57 41 | 2019-10-05,142.76,24,5.95 42 | 2019-10-06,138.76,24,5.78 43 | 2019-10-07,115.76,24,4.82 44 | 2019-10-08,45.91,9,5.10 45 | -------------------------------------------------------------------------------- /Chapter07/aggregate_by_location.py: -------------------------------------------------------------------------------- 1 | ''' 2 | This file will read a CSV file and produce another with aggregated data 3 | ''' 4 | import csv 5 | import argparse 6 | import delorean 7 | from decimal import Decimal 8 | 9 | 10 | def parse_iso(timestamp): 11 | # Parse the ISO format 12 | total = delorean.parse(timestamp, dayfirst=False) 13 | # Keep only the date 14 | return total.date 15 | 16 | 17 | def line(date, total_usd, number): 18 | data = { 19 | 'DATE': date, 20 | 'TOTAL USD': total_usd, 21 | 'NUMBER': number, 22 | # Round to two decimal places 23 | 'AVERAGE': round(total_usd / number, 2), 24 | } 25 | return data 26 | 27 | 28 | def calculate_results(reader): 29 | result = [] 30 | last_date = None 31 | total_usd = 0 32 | number = 0 33 | 34 | for row in reader: 35 | date = parse_iso(row['STD_TIMESTAMP']) 36 | if not last_date: 37 | last_date = date 38 | 39 | if last_date < date: 40 | # New day! 41 | result.append(line(last_date, total_usd, number)) 42 | total_usd = 0 43 | number = 0 44 | last_date = date 45 | 46 | number += 1 47 | total_usd += Decimal(row['USD']) 48 | 49 | # Final results 50 | result.append(line(date, total_usd, number)) 51 | return result 52 | 53 | 54 | def main(input_file, output_file): 55 | reader = csv.DictReader(input_file) 56 | result = calculate_results(reader) 57 | 58 | # Save into csv format 59 | header = result[0].keys() 60 | writer = csv.DictWriter(output_file, fieldnames=header) 61 | writer.writeheader() 62 | writer.writerows(result) 63 | 64 | 65 | if __name__ == '__main__': 66 | parser = argparse.ArgumentParser() 67 | parser.add_argument(dest='input', type=argparse.FileType('r'), 68 | help='input file') 69 | parser.add_argument(dest='output', type=argparse.FileType('w'), 70 | help='output file') 71 | args = parser.parse_args() 72 | main(args.input, args.output) 73 | -------------------------------------------------------------------------------- /Chapter07/aggregate_by_location_by_pandas.py: -------------------------------------------------------------------------------- 1 | ''' 2 | This file will read a CSV file and produce another with aggregated data 3 | ''' 4 | import csv 5 | import argparse 6 | import pandas as pd 7 | 8 | 9 | def pandas_format(row): 10 | row['DATE'] = pd.to_datetime(row['STD_TIMESTAMP']) 11 | row['USD'] = pd.to_numeric(row['USD']) 12 | 13 | return row 14 | 15 | 16 | def calculate_results(reader): 17 | # Load the data, formatting 18 | data = pd.DataFrame(pandas_format(r) for r in reader) 19 | by_usd = data.groupby(data['DATE'].dt.date)['USD'] 20 | result = by_usd.agg(['sum', 'count', 'mean']) 21 | 22 | # Round to 2 digital places 23 | result = result.round(2) 24 | 25 | # Rename colums 26 | result = result.rename(columns={ 27 | 'sum': 'TOTAL USD', 28 | 'count': 'NUMBER', 29 | 'mean': 'AVERAGE', 30 | }) 31 | 32 | return result 33 | 34 | 35 | def main(input_file, output_file): 36 | reader = csv.DictReader(input_file) 37 | result = calculate_results(reader) 38 | 39 | # Save into csv format 40 | output_file.write(result.to_csv()) 41 | 42 | 43 | if __name__ == '__main__': 44 | parser = argparse.ArgumentParser() 45 | parser.add_argument(dest='input', type=argparse.FileType('r'), 46 | help='input file') 47 | parser.add_argument(dest='output', type=argparse.FileType('w'), 48 | help='output file') 49 | args = parser.parse_args() 50 | main(args.input, args.output) 51 | -------------------------------------------------------------------------------- /Chapter07/aggregate_by_location_parallel.py: -------------------------------------------------------------------------------- 1 | ''' 2 | This file will read a list of matched CSV file 3 | and produce one with aggregated data for each 4 | ''' 5 | import glob 6 | import re 7 | import argparse 8 | import concurrent.futures 9 | from aggregate_by_location import main as main_by_file 10 | 11 | 12 | def aggregate_filename(filename): 13 | try: 14 | print(f'Processing {filename}') 15 | # Obtain the location 16 | match = re.match(r'output_3_(.*).csv', filename) 17 | location = match.group(1) 18 | output_file = f'aggregate_{location}.csv' 19 | 20 | with open(filename) as in_file, open(output_file, 'w') as out_file: 21 | main_by_file(in_file, out_file) 22 | 23 | print(f'Done with {filename} => {output_file}', flush=True) 24 | except Exception as exc: 25 | print(f'Unexpected exception {exc}') 26 | 27 | 28 | def main(input_glob): 29 | input_files = [filename for filename in glob.glob(input_glob)] 30 | 31 | with concurrent.futures.ProcessPoolExecutor(max_workers=4) as executor: 32 | futures = [executor.submit(aggregate_filename, filename) 33 | for filename in input_files] 34 | concurrent.futures.wait(futures) 35 | 36 | 37 | if __name__ == '__main__': 38 | parser = argparse.ArgumentParser() 39 | parser.add_argument(dest='input', type=str, help='input glob') 40 | args = parser.parse_args() 41 | main(args.input) 42 | -------------------------------------------------------------------------------- /Chapter07/location_price.py: -------------------------------------------------------------------------------- 1 | ''' 2 | This file will read a log file and produce an CSV file with the data 3 | ''' 4 | import csv 5 | import argparse 6 | from decimal import Decimal, getcontext 7 | 8 | # Set precission to two digital positions 9 | getcontext().prec = 2 10 | 11 | US_LOCATIONS = ['AL', 'AK', 'AZ', 'AR', 'CA', 'CO', 'CT', 'DE', 'FL', 'GA', 12 | 'HI', 'ID', 'IL', 'IN', 'IA', 'KS', 'KY', 'LA', 'ME', 'MD', 13 | 'MA', 'MI', 'MN', 'MS', 'MO', 'MT', 'NE', 'NV', 'NH', 'NJ', 14 | 'NM', 'NY', 'NC', 'ND', 'OH', 'OK', 'OR', 'PA', 'RI', 'SC', 15 | 'SD', 'TN', 'TX', 'UT', 'VT', 'VA', 'WA', 'WV', 'WI', 'WY', 16 | 'DC'] 17 | CAD_LOCATIONS = ['AB', 'BC', 'MB', 'NB', 'NL', 'NS', 'ON', 'PE', 'QC', 'SK', 18 | 'NT', 'NU', 'YT'] 19 | CAD_TO_USD = Decimal(0.76) 20 | 21 | 22 | def add_price_by_location(row): 23 | location = row['LOCATION'] 24 | if location in US_LOCATIONS: 25 | row['COUNTRY'] = 'USA' 26 | row['CURRENCY'] = 'USD' 27 | row['USD'] = Decimal(row['PRICE']) 28 | elif location in CAD_LOCATIONS: 29 | row['COUNTRY'] = 'CANADA' 30 | row['CURRENCY'] = 'CAD' 31 | row['USD'] = Decimal(row['PRICE']) * CAD_TO_USD 32 | else: 33 | raise Exception('Location not found') 34 | 35 | return row 36 | 37 | 38 | def main(input_file, output_file): 39 | reader = csv.DictReader(input_file) 40 | result = [add_price_by_location(row) for row in reader] 41 | 42 | # Save into csv format 43 | header = result[0].keys() 44 | writer = csv.DictWriter(output_file, fieldnames=header) 45 | writer.writeheader() 46 | writer.writerows(result) 47 | 48 | 49 | if __name__ == '__main__': 50 | parser = argparse.ArgumentParser() 51 | parser.add_argument(dest='input', type=argparse.FileType('r'), 52 | help='input file') 53 | parser.add_argument(dest='output', type=argparse.FileType('w'), 54 | help='output file') 55 | args = parser.parse_args() 56 | main(args.input, args.output) 57 | -------------------------------------------------------------------------------- /Chapter07/logs_to_csv.py: -------------------------------------------------------------------------------- 1 | ''' 2 | This file will read a log file and produce an CSV file with the data 3 | ''' 4 | import csv 5 | import argparse 6 | from price_log import PriceLog 7 | 8 | 9 | def log_to_csv(input_file, output_file, location): 10 | logs = [PriceLog.parse(location, line) for line in input_file] 11 | 12 | # Save into csv format 13 | writer = csv.writer(output_file) 14 | writer.writerow(PriceLog.header()) 15 | writer.writerows(l.row() for l in logs) 16 | 17 | 18 | if __name__ == '__main__': 19 | parser = argparse.ArgumentParser() 20 | parser.add_argument(dest='input', type=argparse.FileType('r'), 21 | help='input file') 22 | parser.add_argument(dest='output', type=argparse.FileType('w'), 23 | help='output file') 24 | parser.add_argument('-l', dest='location', type=str, 25 | help='Location of the logs. Default US', default='US') 26 | 27 | args = parser.parse_args() 28 | log_to_csv(args.input, args.output, args.location) 29 | -------------------------------------------------------------------------------- /Chapter07/price_log.py: -------------------------------------------------------------------------------- 1 | import parse 2 | from decimal import Decimal 3 | 4 | 5 | class PriceLog(object): 6 | 7 | def __init__(self, location, timestamp, product_id, price): 8 | self.timestamp = timestamp 9 | self.product_id = product_id 10 | self.price = price 11 | self.location = location 12 | 13 | @classmethod 14 | def parse(cls, location, text_log): 15 | ''' 16 | Parse from a text log with the format 17 | [] - SALE - PRODUCT: - PRICE: 18 | to a PriceLog object 19 | 20 | It requires a location 21 | ''' 22 | def price(string): 23 | return Decimal(string) 24 | 25 | FORMAT = ('[{timestamp}] - SALE - PRODUCT: {product:d} - ' 26 | 'PRICE: {price:price}') 27 | 28 | formats = {'price': price} 29 | result = parse.parse(FORMAT, text_log, formats) 30 | 31 | return cls(location=location, timestamp=result['timestamp'], 32 | product_id=result['product'], price=result['price']) 33 | 34 | @classmethod 35 | def header(cls): 36 | return ['LOCATION', 'TIMESTAMP', 'PRODUCT', 'PRICE'] 37 | 38 | def row(self): 39 | return [self.location, self.timestamp, self.product_id, self.price] 40 | -------------------------------------------------------------------------------- /Chapter07/standard_date.py: -------------------------------------------------------------------------------- 1 | ''' 2 | This file will read a log file and produce an CSV file with the data 3 | ''' 4 | import csv 5 | import argparse 6 | from datetime import datetime, timezone 7 | 8 | 9 | def american_format(timestamp): 10 | ''' 11 | Transform from MM-DD-YYYY HH:MM:SS to iso 8601 12 | ''' 13 | FORMAT = '%m-%d-%Y %H:%M:%S' 14 | 15 | parsed_tmp = datetime.strptime(timestamp, FORMAT) 16 | time_with_tz = parsed_tmp.astimezone(timezone.utc) 17 | isotimestamp = time_with_tz.isoformat() 18 | 19 | return isotimestamp 20 | 21 | 22 | def add_std_timestamp(row): 23 | country = row['COUNTRY'] 24 | if country == 'USA': 25 | # No change 26 | row['STD_TIMESTAMP'] = american_format(row['TIMESTAMP']) 27 | elif country == 'CANADA': 28 | # No change 29 | row['STD_TIMESTAMP'] = row['TIMESTAMP'] 30 | else: 31 | raise Exception('Country not found') 32 | 33 | return row 34 | 35 | 36 | def main(input_file, output_file): 37 | reader = csv.DictReader(input_file) 38 | result = [add_std_timestamp(row) for row in reader] 39 | 40 | # Save into csv format 41 | header = result[0].keys() 42 | writer = csv.DictWriter(output_file, fieldnames=header) 43 | writer.writeheader() 44 | writer.writerows(result) 45 | 46 | 47 | if __name__ == '__main__': 48 | parser = argparse.ArgumentParser() 49 | parser.add_argument(dest='input', type=argparse.FileType('r'), 50 | help='input file') 51 | parser.add_argument(dest='output', type=argparse.FileType('w'), 52 | help='output file') 53 | args = parser.parse_args() 54 | main(args.input, args.output) 55 | -------------------------------------------------------------------------------- /Chapter08/adding_legend_and_annotations.py: -------------------------------------------------------------------------------- 1 | import matplotlib.pyplot as plt 2 | 3 | # STEP 2 4 | LEGEND = ('ProductA', 'ProductB', 'ProductC') 5 | DATA = ( 6 | ('Q1 2017', 100, 30, 3), 7 | ('Q2 2017', 105, 32, 15), 8 | ('Q3 2017', 125, 29, 40), 9 | ('Q4 2017', 115, 31, 80), 10 | ) 11 | 12 | # STEP 3 13 | POS = list(range(len(DATA))) 14 | VALUESA = [valueA for label, valueA, valueB, valueC in DATA] 15 | VALUESB = [valueB for label, valueA, valueB, valueC in DATA] 16 | VALUESC = [valueC for label, valueA, valueB, valueC in DATA] 17 | LABELS = [label for label, valueA, valueB, valueC in DATA] 18 | 19 | # STEP 4 20 | WIDTH = 0.2 21 | valueA = plt.bar([p - WIDTH for p in POS], VALUESA, width=WIDTH) 22 | valueB = plt.bar([p for p in POS], VALUESB, width=WIDTH) 23 | valueC = plt.bar([p + WIDTH for p in POS], VALUESC, width=WIDTH) 24 | plt.ylabel('Sales') 25 | plt.xticks(POS, LABELS) 26 | 27 | # STEP 5 28 | plt.annotate('400% growth', xy=(1.2, 18), xytext=(1.3, 40), 29 | horizontalalignment='center', 30 | fontsize=9, 31 | arrowprops={'facecolor': 'black', 32 | 'arrowstyle': "fancy", 33 | 'connectionstyle': "angle3", 34 | }) 35 | 36 | # STEP 6 37 | # Draw the legend outside the plot 38 | plt.legend(LEGEND, title='Products', bbox_to_anchor=(1, 0.8)) 39 | plt.subplots_adjust(right=0.80) 40 | 41 | # STEP 6 42 | plt.show() 43 | -------------------------------------------------------------------------------- /Chapter08/data.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter08/data.png -------------------------------------------------------------------------------- /Chapter08/scatter.csv: -------------------------------------------------------------------------------- 1 | 4,3.46 2 | 25,98.73 3 | 18,48.78 4 | 11,5.08 5 | 15,36.3 6 | 3,7.31 7 | 28,92.07 8 | 21,57.91 9 | 23,54.49 10 | 17,94.56 11 | 7,1.06 12 | 24,99.36 13 | 11,3.64 14 | 30,6.52 15 | 8,4.12 16 | 17,7.63 17 | 27,87.13 18 | 13,7.52 19 | 14,69.53 20 | 9,2.03 21 | 7,6.29 22 | 15,98.66 23 | 17,37.96 24 | 20,98.25 25 | 15,8.87 26 | 14,55.77 27 | 11,7.07 28 | 13,3.74 29 | 26,56.62 30 | 15,6.06 31 | 20,5.59 32 | 5,5.43 33 | 11,4.0 34 | 22,56.17 35 | 18,4.74 36 | 20,5.83 37 | 15,95.68 38 | 2,3.32 39 | 8,3.65 40 | 26,33.46 41 | 30,97.73 42 | 28,61.42 43 | 20,38.34 44 | 2,1.03 45 | 17,9.69 46 | 15,8.26 47 | 18,68.66 48 | 23,21.62 49 | 13,74.91 50 | 11,68.28 51 | 2,8.39 52 | 27,78.32 53 | 16,64.9 54 | 28,38.97 55 | 8,4.73 56 | 7,4.37 57 | 14,68.04 58 | 10,6.59 59 | 20,8.38 60 | 10,70.38 61 | 11,5.98 62 | 19,81.12 63 | 25,95.13 64 | 11,9.75 65 | 24,38.97 66 | 1,6.68 67 | 29,16.29 68 | 29,69.58 69 | 22,84.33 70 | 23,11.55 71 | 20,8.89 72 | 19,5.65 73 | 10,7.26 74 | 29,55.0 75 | 21,72.17 76 | 13,34.15 77 | 12,7.89 78 | 8,3.73 79 | 30,51.94 80 | 20,32.15 81 | 23,7.24 82 | 15,66.65 83 | 26,21.7 84 | 27,6.15 85 | 18,4.54 86 | 11,8.6 87 | 7,8.78 88 | 2,1.88 89 | 11,8.73 90 | 16,12.1 91 | 19,9.62 92 | 4,5.61 93 | 19,53.45 94 | 17,44.76 95 | 15,84.19 96 | 22,97.54 97 | 12,1.19 98 | 19,4.99 99 | 14,3.18 100 | 19,97.6 101 | -------------------------------------------------------------------------------- /Chapter08/visualising_maps.py: -------------------------------------------------------------------------------- 1 | import matplotlib.pyplot as plt 2 | import matplotlib.cm as cm 3 | import fiona 4 | 5 | # STEP 2 6 | COUNTRIES_POPULATION = { 7 | 'Spain': 47.2, 8 | 'Portugal': 10.6, 9 | 'United Kingdom': 63.8, 10 | 'Ireland': 4.7, 11 | 'France': 64.9, 12 | 'Italy': 61.1, 13 | 'Germany': 82.6, 14 | 'Netherlands': 16.8, 15 | 'Belgium': 11.1, 16 | 'Denmark': 5.6, 17 | 'Slovenia': 2, 18 | 'Austria': 8.5, 19 | 'Luxembourg': 0.5, 20 | 'Andorra': 0.077, 21 | 'Switzerland': 8.2, 22 | 'Liechtenstein': 0.038, 23 | } 24 | MAX_POPULATION = max(COUNTRIES_POPULATION.values()) 25 | MIN_POPULATION = min(COUNTRIES_POPULATION.values()) 26 | 27 | # STEP 3 28 | colormap = cm.get_cmap('Greens') 29 | COUNTRY_COLOUR = { 30 | country_name: colormap( 31 | (population - MIN_POPULATION) / (MAX_POPULATION - MIN_POPULATION) 32 | ) 33 | for country_name, population in COUNTRIES_POPULATION.items() 34 | } 35 | 36 | # STEP 4 37 | with fiona.open('europe.geojson') as fd: 38 | full_data = [data for data in fd] 39 | 40 | # STEP 5 41 | full_data = [data for data in full_data 42 | if data['properties']['NAME'] in COUNTRIES_POPULATION] 43 | 44 | for data in full_data: 45 | country_name = data['properties']['NAME'] 46 | colour = COUNTRY_COLOUR[country_name] 47 | 48 | # Draw the ISO3 code of each country 49 | long, lat = data['properties']['LON'], data['properties']['LAT'] 50 | iso3 = data['properties']['ISO3'] 51 | plt.text(long, lat, iso3, horizontalalignment='center') 52 | 53 | geo_type = data['geometry']['type'] 54 | if geo_type == 'Polygon': 55 | data_x = [x for x, y in data['geometry']['coordinates'][0]] 56 | data_y = [y for x, y in data['geometry']['coordinates'][0]] 57 | plt.fill(data_x, data_y, c=colour) 58 | # Draw a line surrounding the area 59 | plt.plot(data_x, data_y, c='black', linewidth=0.2) 60 | elif geo_type == 'MultiPolygon': 61 | for coordinates in data['geometry']['coordinates']: 62 | data_x = [x for x, y in coordinates[0]] 63 | data_y = [y for x, y in coordinates[0]] 64 | plt.fill(data_x, data_y, c=colour) 65 | # Draw a line surrounding the area 66 | plt.plot(data_x, data_y, c='black', linewidth=0.2) 67 | 68 | # Set the background to light blue 69 | axes = plt.gca() 70 | axes.set_facecolor('xkcd:light blue') 71 | # Set the proper aspect to avoid distorsions 72 | axes.set_aspect('equal', adjustable='box') 73 | # Remove labels from axes 74 | plt.xticks([]) 75 | plt.yticks([]) 76 | 77 | # STEP 6 78 | plt.show() 79 | -------------------------------------------------------------------------------- /Chapter09/app.py: -------------------------------------------------------------------------------- 1 | ''' 2 | This app.oy file should be used with the Heroku template in 3 | https://github.com/datademofun/heroku-basic-flask 4 | 5 | Add twilio on requirements.txt and replace app.py with this file 6 | 7 | See chapter 9 for further detail 8 | ''' 9 | from flask import Flask, request 10 | from twilio.twiml.messaging_response import MessagingResponse 11 | 12 | app = Flask(__name__) 13 | 14 | 15 | @app.route('/') 16 | def homepage(): 17 | return 'All working!' 18 | 19 | 20 | @app.route("/sms", methods=['GET', 'POST']) 21 | def sms_reply(): 22 | 23 | from_number = request.form['From'] 24 | body = request.form['Body'] 25 | resp = MessagingResponse() 26 | 27 | msg = (f'Awwwww! Thanks so much for your message {from_number}, ' 28 | f'"{body}" to you too. ') 29 | 30 | resp.message(msg) 31 | return str(resp) 32 | 33 | 34 | if __name__ == '__main__': 35 | app.run() 36 | -------------------------------------------------------------------------------- /Chapter09/email_styling.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 221 | 222 | 223 |
224 | {{content}} 225 |
226 | 227 | 228 | -------------------------------------------------------------------------------- /Chapter09/email_template.md: -------------------------------------------------------------------------------- 1 | Hi {name}: 2 | 3 | This is an email talking about **things** 4 | 5 | ### Very important info 6 | 7 | 1. One thing 8 | 2. Other thing 9 | 3. Some extra detail 10 | 11 | Best regards, 12 | 13 | *The email team* 14 | -------------------------------------------------------------------------------- /Chapter09/telegram_bot.py: -------------------------------------------------------------------------------- 1 | import time 2 | import telepot 3 | from telepot.loop import MessageLoop 4 | from telepot.delegate import per_chat_id, create_open, pave_event_space 5 | 6 | 7 | TOKEN = '' 8 | 9 | 10 | # Define the information to return per command 11 | def get_help(): 12 | msg = ''' 13 | Use one of the following commands: 14 | help: To show this help 15 | offers: To see this week offers 16 | events: To see this week events 17 | ''' 18 | return msg 19 | 20 | 21 | def get_offers(): 22 | msg = ''' 23 | This week enjoy these amazing offers! 24 | 20% discount in beach products 25 | 15% discount if you spend more than €50 26 | ''' 27 | return msg 28 | 29 | 30 | def get_events(): 31 | msg = ''' 32 | Join us for an incredible party the Thursday in our Sun City shop! 33 | ''' 34 | return msg 35 | 36 | 37 | COMMANDS = { 38 | 'help': get_help, 39 | 'offers': get_offers, 40 | 'events': get_events, 41 | } 42 | 43 | 44 | class MarketingBot(telepot.helper.ChatHandler): 45 | 46 | def open(self, initial_msg, seed): 47 | self.sender.sendMessage(get_help()) 48 | # prevent on_message() from being called on the initial message 49 | return True 50 | 51 | def on_chat_message(self, msg): 52 | # If the data sent is not test, return an error 53 | content_type, chat_type, chat_id = telepot.glance(msg) 54 | 55 | if content_type != 'text': 56 | self.sender.sendMessage("I don't understand you. " 57 | "Please type 'help' for options") 58 | return 59 | 60 | # Make the commands case insensitive 61 | command = msg['text'].lower() 62 | if command not in COMMANDS: 63 | self.sender.sendMessage("I don't understand you. " 64 | "Please type 'help' for options") 65 | return 66 | 67 | message = COMMANDS[command]() 68 | self.sender.sendMessage(message) 69 | 70 | def on_idle(self, event): 71 | self.close() 72 | 73 | def on_close(self, event): 74 | # Add any required cleanup here 75 | pass 76 | 77 | 78 | # Create and start the bot 79 | bot = telepot.DelegatorBot(TOKEN, [ 80 | pave_event_space()( 81 | per_chat_id(), create_open, MarketingBot, timeout=10), 82 | ]) 83 | MessageLoop(bot).run_as_thread() 84 | print('Listening ...') 85 | 86 | while 1: 87 | time.sleep(10) 88 | -------------------------------------------------------------------------------- /Chapter09/telegram_bot_custom_keyboard.py: -------------------------------------------------------------------------------- 1 | import time 2 | import telepot 3 | from telepot.loop import MessageLoop 4 | from telepot.delegate import per_chat_id, create_open, pave_event_space 5 | 6 | from telepot.namedtuple import ReplyKeyboardMarkup, KeyboardButton 7 | 8 | TOKEN = '' 9 | 10 | 11 | # Define the information to return per command 12 | def get_help(): 13 | msg = ''' 14 | Use one of the following commands: 15 | help: To show this help 16 | offers: To see this week offers 17 | events: To see this week events 18 | ''' 19 | return msg 20 | 21 | 22 | def get_offers(): 23 | msg = ''' 24 | This week enjoy these amazing offers! 25 | 20% discount in beach products 26 | 15% discount if you spend more than €50 27 | ''' 28 | return msg 29 | 30 | 31 | def get_events(): 32 | msg = ''' 33 | Join us for an incredible party the Thursday in our Sun City shop! 34 | ''' 35 | return msg 36 | 37 | 38 | COMMANDS = { 39 | 'help': get_help, 40 | 'offers': get_offers, 41 | 'events': get_events, 42 | } 43 | 44 | # Create a custom keyboard with only the valid responses 45 | keys = [[KeyboardButton(text=text)] for text in COMMANDS] 46 | KEYBOARD = ReplyKeyboardMarkup(keyboard=keys) 47 | 48 | 49 | class MarketingBot(telepot.helper.ChatHandler): 50 | 51 | def open(self, initial_msg, seed): 52 | self.sender.sendMessage(get_help(), reply_markup=KEYBOARD) 53 | # prevent on_message() from being called on the initial message 54 | return True 55 | 56 | def on_chat_message(self, msg): 57 | # If the data sent is not test, return an error 58 | content_type, chat_type, chat_id = telepot.glance(msg) 59 | 60 | if content_type != 'text': 61 | self.sender.sendMessage("I don't understand you. " 62 | "Please type 'help' for options", 63 | reply_markup=KEYBOARD) 64 | return 65 | 66 | # Make the commands case insensitive 67 | command = msg['text'].lower() 68 | if command not in COMMANDS: 69 | self.sender.sendMessage("I don't understand you. " 70 | "Please type 'help' for options", 71 | reply_markup=KEYBOARD) 72 | return 73 | 74 | message = COMMANDS[command]() 75 | self.sender.sendMessage(message, reply_markup=KEYBOARD) 76 | 77 | def on__idle(self, event): 78 | self.close() 79 | 80 | 81 | # Create and start the bot 82 | bot = telepot.DelegatorBot(TOKEN, [ 83 | pave_event_space()( 84 | per_chat_id(), create_open, MarketingBot, timeout=10), 85 | ]) 86 | MessageLoop(bot).run_as_thread() 87 | print('Listening ...') 88 | 89 | while 1: 90 | time.sleep(10) 91 | -------------------------------------------------------------------------------- /Chapter10/config-channel.ini: -------------------------------------------------------------------------------- 1 | [MAILGUN] 2 | KEY = 3 | DOMAIN = 4 | FROM = 5 | 6 | [TWILIO] 7 | ACCOUNT_SID = 8 | AUTH_TOKEN = 9 | FROM = 10 | -------------------------------------------------------------------------------- /Chapter10/config-opportunity.ini: -------------------------------------------------------------------------------- 1 | [SEARCH] 2 | keywords = cpu 3 | feeds = http://feeds.reuters.com/reuters/technologyNews, 4 | https://rss.nytimes.com/services/xml/rss/nyt/Technology.xml, 5 | https://feeds.bbci.co.uk/news/science_and_environment/rss.xml 6 | 7 | [EMAIL] 8 | user = 9 | password = 10 | from = 11 | to = 12 | -------------------------------------------------------------------------------- /Chapter10/create_personalised_coupons.py: -------------------------------------------------------------------------------- 1 | # IMPORTS 2 | import hashlib 3 | import re 4 | import csv 5 | from random import choice 6 | 7 | CHARACTERS = 'ACEFGHJKLMNPRTUVWXY379' 8 | 9 | 10 | # FUNCTIONS 11 | def random_code(digits): 12 | # All possibilities, except the letters and numbers that can 13 | # be confusing ( 0 and O, etc ) 14 | digits = [choice(CHARACTERS) for _ in range(digits)] 15 | return ''.join(digits) 16 | 17 | 18 | def checksum(code1, code2): 19 | m = hashlib.sha256() 20 | m.update(code1.encode()) 21 | m.update(code2.encode()) 22 | checksum = int(m.hexdigest()[:2], base=16) 23 | digit = CHARACTERS[checksum % len(CHARACTERS)] 24 | return digit 25 | 26 | 27 | def check_code(code): 28 | # Divide the code in its parts 29 | CODE = r'(\w{4})-(\w{5})-(\w)(\w)$' 30 | match = re.match(CODE, code) 31 | if not match: 32 | return False 33 | 34 | # Check the checksum 35 | code1, code2, check1, check2 = match.groups() 36 | expected_check1 = checksum(code1, code2) 37 | expected_check2 = checksum(code2, code1) 38 | 39 | if expected_check1 == check1 and expected_check2 == check2: 40 | return True 41 | 42 | return False 43 | 44 | 45 | def generate_code(): 46 | code1 = random_code(4) 47 | code2 = random_code(5) 48 | check1 = checksum(code1, code2) 49 | check2 = checksum(code2, code1) 50 | 51 | return f'{code1}-{code2}-{check1}{check2}' 52 | 53 | 54 | # SET UP TASK 55 | BATCHES = 500_000, 300_000, 200_000 56 | TOTAL_NUM_CODES = sum(BATCHES) 57 | NUM_TRIES = 3 58 | 59 | 60 | # GENERATE CODES 61 | 62 | codes = set() 63 | for _ in range(TOTAL_NUM_CODES): 64 | code = generate_code() 65 | for _ in range(NUM_TRIES): 66 | code = generate_code() 67 | if code not in codes: 68 | break 69 | 70 | else: 71 | raise Exception('Run out of tries generating a code') 72 | 73 | codes.add(code) 74 | 75 | assert check_code(code) 76 | print(f'Code: {code}') 77 | 78 | 79 | # CREATE AND SAVE BATCHES 80 | for index, batch_size in enumerate(BATCHES, 1): 81 | batch = [(codes.pop(),) for _ in range(batch_size)] 82 | filename = f'codes_batch_{index}.csv' 83 | with open(filename, 'w') as fp: 84 | writer = csv.writer(fp) 85 | writer.writerows(batch) 86 | 87 | assert 0 == len(codes) 88 | -------------------------------------------------------------------------------- /Chapter10/email_styling.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 221 | 222 | 223 |
224 | {{content}} 225 |
226 | 227 | 228 | -------------------------------------------------------------------------------- /Chapter10/email_template.md: -------------------------------------------------------------------------------- 1 | Hi! 2 | 3 | This is an automated email checking articles published last week containing the *keywords*: {keywords} 4 | in the following feeds: 5 | {feed_list} 6 | 7 | 8 | List of articles: 9 | ===== 10 | {article_list} 11 | 12 | 13 | 14 | Enjoy the read! 15 | -------------------------------------------------------------------------------- /Chapter10/generate_sales_report.py: -------------------------------------------------------------------------------- 1 | import os 2 | import openpyxl 3 | import fpdf 4 | import argparse 5 | import delorean 6 | import PyPDF2 7 | from collections import defaultdict 8 | from sale_log import SaleLog 9 | import matplotlib.pyplot as plt 10 | from itertools import islice 11 | 12 | A4_INCHES = (11.69, 8.27) 13 | 14 | 15 | def generate_summary(logs): 16 | ''' 17 | Generate a summary, according to the received logs 18 | 19 | The summary has: 20 | 21 | { 22 | 'start_time': start_time, 23 | 'end_time': end_time, 24 | 'total_income': total_income, 25 | 'average_discount': average_discount, 26 | 'units': units, 27 | 'by_product': 29 | } 30 | 31 | Note that the summary of a product won't contain the 'by_product' field 32 | ''' 33 | total_income = sum(log.price for log in logs) 34 | start_time = min(log.timestamp for log in logs) 35 | end_time = max(log.timestamp for log in logs) 36 | average_discount = sum(log.discount for log in logs) / len(logs) 37 | units = len(logs) 38 | 39 | # Group by product and get summaries 40 | products = defaultdict(list) 41 | for log in logs: 42 | products[log.name].append(log) 43 | 44 | summary = { 45 | 'start_time': start_time, 46 | 'end_time': end_time, 47 | 'total_income': total_income, 48 | 'average_discount': average_discount, 49 | 'units': units, 50 | } 51 | 52 | if len(products) > 1: 53 | by_product = {name: generate_summary(logs) 54 | for name, logs in products.items()} 55 | summary['by_product'] = by_product 56 | 57 | return summary 58 | 59 | 60 | def aggregate_by_day(logs): 61 | ''' 62 | Aggregate logs by day 63 | 64 | returns a list of: 65 | (day, summary) 66 | ''' 67 | days = [] 68 | day = [logs[0]] 69 | for log in logs[1:]: 70 | end_of_day = day[0].timestamp.end_of_day 71 | if log.timestamp.datetime > end_of_day: 72 | # A new day 73 | days.append(day) 74 | day = [] 75 | day.append(log) 76 | 77 | # Generate a summary by day 78 | def date_string(log): 79 | return log.timestamp.truncate('day').datetime.strftime('%d %b') 80 | 81 | summaries = [ 82 | (date_string(day[0]), generate_summary(day)) 83 | for day in days 84 | ] 85 | return summaries 86 | 87 | 88 | def aggregate_by_shop(logs): 89 | ''' 90 | Aggregate logs by shop 91 | 92 | returns a list of: 93 | (shop, summary) 94 | ''' 95 | # Aggregate the results by shop 96 | by_shop = defaultdict(list) 97 | for log in logs: 98 | by_shop[log.shop].append(log) 99 | 100 | # Generate a summary by day 101 | summaries = [(shop, generate_summary(logs)) 102 | for shop, logs in by_shop.items()] 103 | return summaries 104 | 105 | 106 | def graph(full_summary, products, temp_file, skip_labels=1): 107 | ''' 108 | Generate a page with two graph rows from a summary: 109 | - Top row with Total income by product as an stacked bar graph 110 | - A row with X graphs by units, one for each product 111 | 112 | The X axis will be the tag of the summary. 113 | 114 | full_summary: a list of [(tags, summary)] 115 | products: All the available products 116 | temp_file: Temporal file name to strore a PDF with the graphs 117 | skip_labels: An optional number to skip labels, to improve 118 | readability. Default 1 (show all) 119 | ''' 120 | pos = list(range(len(full_summary))) 121 | units = [summary['units'] for tag, summary in full_summary] 122 | # Calculate the average discount 123 | # discount = [summary['average_discount'] for tag, summary in full_summary] 124 | 125 | income_by_product = [] 126 | units_per_product = [] 127 | # Store the aggregated income to display the bars 128 | baselevel = None 129 | default = { 130 | 'total_income': 0, 131 | 'units': 0, 132 | } 133 | max_units = 0 134 | for product in products: 135 | product_income = [ 136 | summary['by_product'].get(product, default)['total_income'] 137 | for day, summary in full_summary 138 | ] 139 | product_units = [summary['by_product'].get(product, default)['units'] 140 | for day, summary in full_summary] 141 | if not baselevel: 142 | baselevel = [0 for _ in range(len(full_summary))] 143 | 144 | income_by_product.append((product, product_income, baselevel)) 145 | units_per_product.append((product, product_units)) 146 | max_units = max(max(product_units), max_units) 147 | 148 | baselevel = [product + bottom 149 | for product, bottom in zip(product_income, baselevel)] 150 | 151 | labels = [day for day, summary in full_summary] 152 | 153 | plt.figure(figsize=A4_INCHES) 154 | 155 | plt.subplot(2, 1, 1) 156 | plt.ylabel('Income by product') 157 | for name, product, baseline in income_by_product: 158 | plt.bar(pos, product, bottom=baseline) 159 | 160 | plt.legend([name for name, _, _ in income_by_product]) 161 | 162 | # Display a line with the average discount 163 | # plt.twinx() 164 | # plt.plot(pos, discount, 'o-', color='green') 165 | # plt.ylabel('Average Discount') 166 | 167 | plt.xticks(pos[::skip_labels], labels[::skip_labels]) 168 | 169 | max_units += 1 170 | 171 | num_products = len(units_per_product) 172 | for index, (product, units) in enumerate(units_per_product): 173 | plt.subplot(2, num_products, num_products + index + 1) 174 | plt.ylabel('Total units {} sold'.format(product)) 175 | plt.ylim(ymax=max_units) 176 | plt.bar(pos, units) 177 | # Display only on on each skip labels 178 | plt.xticks(pos[::skip_labels], labels[::skip_labels]) 179 | 180 | plt.savefig(temp_file) 181 | return temp_file 182 | 183 | 184 | def create_summary_brief(summary, temp_file): 185 | ''' 186 | Write a PDF page with the summary information, in the specified temp_file 187 | ''' 188 | document = fpdf.FPDF() 189 | document.set_font('Times', '', 12) 190 | document.add_page() 191 | TEMPLATE = ''' 192 | Report generated at {now} 193 | Covering data from {start_time} to {end_time} 194 | 195 | 196 | Summary 197 | ------- 198 | TOTAL INCOME: $ {income} 199 | TOTAL UNIT: {units} units 200 | AVERAGE DISCOUNT: {discount}% 201 | ''' 202 | 203 | def format_full_tmp(timestamp): 204 | return timestamp.datetime.isoformat() 205 | 206 | def format_brief_tmp(timestamp): 207 | return timestamp.datetime.strftime('%d %b') 208 | 209 | text = TEMPLATE.format(now=format_full_tmp(delorean.utcnow()), 210 | start_time=format_brief_tmp(summary['start_time']), 211 | end_time=format_brief_tmp(summary['end_time']), 212 | income=summary['total_income'], 213 | units=summary['units'], 214 | discount=summary['average_discount']) 215 | 216 | document.multi_cell(0, 6, text) 217 | document.ln() 218 | document.output(temp_file) 219 | return temp_file 220 | 221 | 222 | def main(input_file, output_file): 223 | xlsfile = openpyxl.load_workbook(input_file) 224 | sheet = xlsfile['Sheet'] 225 | 226 | def row_to_dict(header, row): 227 | return {header: cell.value for cell, header in zip(row, header)} 228 | 229 | # islice skips the first row, the header 230 | data = [SaleLog.from_row([cell.value for cell in row]) 231 | for row in islice(sheet, 1, None)] 232 | 233 | # Generate each of the pages: a full summary, graph by day, and by shop 234 | total_summary = generate_summary(data) 235 | products = total_summary['by_product'].keys() 236 | summary_by_day = aggregate_by_day(data) 237 | summary_by_shop = aggregate_by_shop(data) 238 | 239 | # Compose the PDF with a brief summary and all the graphs 240 | summary_file = create_summary_brief(total_summary, 'summary.pdf') 241 | by_day_file = graph(summary_by_day, products, 'by_day.pdf', 7) 242 | by_shop_file = graph(summary_by_shop, products, 'by_shop.pdf') 243 | 244 | # Group all the pdfs into a single file 245 | pdfs = [summary_file, by_day_file, by_shop_file] 246 | pdf_files = [open(filename, 'rb') for filename in pdfs] 247 | output_pdf = PyPDF2.PdfFileWriter() 248 | for pdf in pdf_files: 249 | reader = PyPDF2.PdfFileReader(pdf) 250 | output_pdf.appendPagesFromReader(reader) 251 | 252 | # Write the resulting PDF 253 | with open(output_file, "wb") as out_file: 254 | output_pdf.write(out_file) 255 | 256 | # Close the files 257 | for pdf in pdf_files: 258 | pdf.close() 259 | 260 | # clean the temp files 261 | for pdf_filename in pdfs: 262 | os.remove(pdf_filename) 263 | 264 | 265 | if __name__ == '__main__': 266 | # Compile the input and output files from the command line 267 | parser = argparse.ArgumentParser() 268 | parser.add_argument(type=str, dest='input_file') 269 | parser.add_argument(type=str, dest='output_file') 270 | args = parser.parse_args() 271 | 272 | # Call the main function 273 | main(args.input_file, args.output_file) 274 | -------------------------------------------------------------------------------- /Chapter10/output.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter10/output.pdf -------------------------------------------------------------------------------- /Chapter10/parse_sales_log.py: -------------------------------------------------------------------------------- 1 | import os 2 | import openpyxl 3 | import argparse 4 | from sale_log import SaleLog 5 | 6 | 7 | def get_logs_from_file(shop, log_filename): 8 | ''' 9 | Based on a file, obtain and parse all logs 10 | Return a list of SaleLog objects 11 | ''' 12 | with open(log_filename) as logfile: 13 | logs = [SaleLog.parse(shop=shop, text_log=log) 14 | for log in logfile] 15 | 16 | return logs 17 | 18 | 19 | def main(log_dir, output_filename): 20 | logs = [] 21 | for dirpath, dirnames, filenames in os.walk(log_dir): 22 | for filename in filenames: 23 | # The shop is the last directory 24 | shop = os.path.basename(dirpath) 25 | fullpath = os.path.join(dirpath, filename) 26 | logs.extend(get_logs_from_file(shop, fullpath)) 27 | 28 | # Create and save the Excel sheet 29 | xlsfile = openpyxl.Workbook() 30 | sheet = xlsfile['Sheet'] 31 | 32 | # Write the first row 33 | sheet.append(SaleLog.row_header()) 34 | for log in logs: 35 | sheet.append(log.row()) 36 | 37 | xlsfile.save(output_filename) 38 | 39 | 40 | if __name__ == '__main__': 41 | parser = argparse.ArgumentParser() 42 | parser.add_argument(type=str, dest='log_directory') 43 | parser.add_argument('-o', type=str, dest='output_file', 44 | default='result.xlsx') 45 | 46 | args = parser.parse_args() 47 | 48 | main(args.log_directory, args.output_file) 49 | -------------------------------------------------------------------------------- /Chapter10/report.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter10/report.xlsx -------------------------------------------------------------------------------- /Chapter10/sale_log.py: -------------------------------------------------------------------------------- 1 | import parse 2 | from decimal import Decimal 3 | import delorean 4 | 5 | 6 | class SaleLog(object): 7 | 8 | def __init__(self, timestamp, product_id, price, name, discount, 9 | shop=None): 10 | self.timestamp = timestamp 11 | self.product_id = product_id 12 | self.price = price 13 | self.name = name 14 | self.discount = discount 15 | self.shop = shop 16 | 17 | def __repr__(self): 18 | return ''.format(self.timestamp, 19 | self.product_id, 20 | self.price) 21 | 22 | @classmethod 23 | def row_header(cls): 24 | HEADER = ('Timestamp', 'Shop', 'Product Id', 'Name', 'Price', 25 | 'Discount') 26 | return HEADER 27 | 28 | def row(self): 29 | return (self.timestamp.datetime.isoformat(), self.shop, 30 | self.product_id, self.name, self.price, 31 | '{}%'.format(self.discount)) 32 | 33 | @classmethod 34 | def from_row(cls, row): 35 | timestamp_str, shop, product_id, name, raw_price, discount_str = row 36 | timestamp = delorean.parse(timestamp_str) 37 | discount = parse.parse('{:d}%', discount_str)[0] 38 | # Round to remove possible rounding errors in Excel 39 | price = round(Decimal(raw_price), 2) 40 | 41 | return cls(timestamp=timestamp, product_id=product_id, 42 | price=price, name=name, discount=discount, 43 | shop=shop) 44 | 45 | @classmethod 46 | def parse(cls, shop, text_log): 47 | ''' 48 | Parse from a text log with the format 49 | [] - SALE - PRODUCT: - PRICE: $ - NAME: - DISCOUNT: % 50 | to a SaleLog object 51 | ''' 52 | def price(string): 53 | return Decimal(string) 54 | 55 | def isodate(string): 56 | return delorean.parse(string) 57 | 58 | FORMAT = ('[{timestamp:isodate}] - SALE - PRODUCT: {product:d} ' 59 | '- PRICE: ${price:price} - NAME: {name:D} ' 60 | '- DISCOUNT: {discount:d}%') 61 | 62 | formats = {'price': price, 'isodate': isodate} 63 | result = parse.parse(FORMAT, text_log, formats) 64 | 65 | return cls(timestamp=result['timestamp'], 66 | product_id=result['product'], 67 | price=result['price'], 68 | name=result['name'], 69 | discount=result['discount'], 70 | shop=shop) 71 | -------------------------------------------------------------------------------- /Chapter10/search_keywords.py: -------------------------------------------------------------------------------- 1 | # IMPORTS # 2 | import argparse 3 | import sys 4 | import configparser 5 | 6 | import feedparser 7 | import datetime 8 | import delorean 9 | import requests 10 | from bs4 import BeautifulSoup 11 | import mistune 12 | import jinja2 13 | from collections import namedtuple 14 | 15 | import smtplib 16 | from email.mime.multipart import MIMEMultipart 17 | from email.mime.text import MIMEText 18 | 19 | # READ TEMPLATES # 20 | 21 | # Group the email configuration parameters 22 | # Note the 'from_' to avoid using a reserved Python keyword (from) 23 | EmailConfig = namedtuple('EmailConfig', ['user', 'password', 'from_', 'to']) 24 | 25 | 26 | # Get the email templates from hard disk 27 | EMAIL_TEMPLATE_FILE = 'email_template.md' 28 | EMAIL_STYLING_FILE = 'email_styling.html' 29 | 30 | with open(EMAIL_TEMPLATE_FILE) as md_file: 31 | EMAIL_TEMPLATE = md_file.read() 32 | 33 | with open(EMAIL_STYLING_FILE) as html_file: 34 | EMAIL_STYLING = html_file.read() 35 | 36 | 37 | def get_articles(keywords, feeds): 38 | ''' 39 | Retrieve a list of articles from the feeds that contain the keywords 40 | 41 | Each article is returned in the format: 42 | 43 | (title, summary, link) 44 | ''' 45 | articles = [] 46 | 47 | for feed in feeds: 48 | rss = feedparser.parse(feed) 49 | updated_time = rss.get('updated', str(datetime.datetime.utcnow())) 50 | 51 | time_limit = delorean.parse(updated_time) - datetime.timedelta(days=7) 52 | for entry in rss.entries: 53 | # Normalise the time 54 | entry_time = delorean.parse(entry.published) 55 | entry_time.shift('UTC') 56 | if entry_time < time_limit: 57 | # Skip this entry 58 | continue 59 | 60 | # Get the article 61 | response = requests.get(entry.link) 62 | article = BeautifulSoup(response.text, 'html.parser') 63 | article_reference = (article.title.string.strip(), 64 | entry.summary.strip(), 65 | entry.link) 66 | article_text = article.get_text() 67 | 68 | for keyword in keywords: 69 | if keyword.lower() in article_text.lower(): 70 | articles.append(article_reference) 71 | break 72 | 73 | return articles 74 | 75 | 76 | def compose_email_body(articles, keywords, feed_list): 77 | ''' 78 | From the list of articles, keywords and feeds, fill the email template 79 | 80 | Set the list in the adequate format for the template 81 | ''' 82 | # Compose the list of articles 83 | ARTICLE_TEMPLATE = '* **{title}** {summary}: {link}' 84 | article_list = [ARTICLE_TEMPLATE.format(title=title, summary=summary, 85 | link=link) 86 | for title, summary, link in articles] 87 | 88 | data = { 89 | 'article_list': '\n'.join(article_list), 90 | 'keywords': ', '.join(keywords), 91 | 'feed_list': ', '.join(feed_list), 92 | } 93 | text = EMAIL_TEMPLATE.format(**data) 94 | 95 | html_content = mistune.markdown(text) 96 | html = jinja2.Template(EMAIL_STYLING).render(content=html_content) 97 | 98 | return text, html 99 | 100 | 101 | def send_email(email_config, text_body, html_body): 102 | ''' 103 | Send an email with the text and html body, using the parameters 104 | configured in email_config 105 | ''' 106 | msg = MIMEMultipart('alternative') 107 | msg['Subject'] = 'Weekly report' 108 | msg['From'] = email_config.from_ 109 | msg['To'] = email_config.to 110 | 111 | part_plain = MIMEText(text_body, 'plain') 112 | part_html = MIMEText(html_body, 'html') 113 | 114 | msg.attach(part_plain) 115 | msg.attach(part_html) 116 | 117 | with smtplib.SMTP('smtp.gmail.com', 587) as server: 118 | server.starttls() 119 | server.login(email_config.user, email_config.password) 120 | server.sendmail(email_config.from_, [email_config.to], msg.as_string()) 121 | 122 | 123 | def main(keywords, feeds, email_config): 124 | articles = get_articles(keywords, feeds) 125 | text_body, html_body = compose_email_body(articles, keywords, feeds) 126 | send_email(email_config, text_body, html_body) 127 | 128 | 129 | if __name__ == '__main__': 130 | parser = argparse.ArgumentParser() 131 | parser.add_argument(type=argparse.FileType('r'), dest='config', 132 | help='config file') 133 | parser.add_argument('-o', dest='output', type=argparse.FileType('w'), 134 | help='output file', 135 | default=sys.stdout) 136 | 137 | args = parser.parse_args() 138 | config = configparser.ConfigParser() 139 | config.read_file(args.config) 140 | keywords = config['SEARCH']['keywords'].split(',') 141 | feeds = [feed.strip() for feed in config['SEARCH']['feeds'].split(',')] 142 | 143 | email_user = config['EMAIL']['user'] 144 | email_password = config['EMAIL']['password'] 145 | email_from = config['EMAIL']['from'] 146 | email_to = config['EMAIL']['to'] 147 | email_config = EmailConfig(email_user, email_password, email_from, 148 | email_to) 149 | 150 | main(keywords, feeds, email_config) 151 | -------------------------------------------------------------------------------- /Chapter10/search_opportunities.py: -------------------------------------------------------------------------------- 1 | ############ 2 | # IMPORTS 3 | ############ 4 | import argparse 5 | import configparser 6 | import feedparser 7 | import datetime 8 | import delorean 9 | import requests 10 | from bs4 import BeautifulSoup 11 | import mistune 12 | import jinja2 13 | from collections import namedtuple 14 | import smtplib 15 | from email.mime.multipart import MIMEMultipart 16 | from email.mime.text import MIMEText 17 | 18 | # Group the email configuration parameters 19 | # Note the 'from_' to avoid using a reserved Python keyword (from) 20 | EmailConfig = namedtuple('EmailConfig', ['user', 'password', 'from_', 'to']) 21 | 22 | ############ 23 | # READ TEMPLATES INTO MEMORY 24 | ############ 25 | 26 | # Get the email templates from hard disk 27 | EMAIL_TEMPLATE_FILE = 'email_template.md' 28 | EMAIL_STYLING_FILE = 'email_styling.html' 29 | 30 | with open(EMAIL_TEMPLATE_FILE) as md_file: 31 | EMAIL_TEMPLATE = md_file.read() 32 | 33 | with open(EMAIL_STYLING_FILE) as html_file: 34 | EMAIL_STYLING = html_file.read() 35 | 36 | 37 | def get_articles(keywords, feeds): 38 | ''' 39 | Retrieve a list of articles from the feeds that contain the keywords 40 | 41 | Each article is returned in the format: 42 | 43 | (title, summary, link) 44 | ''' 45 | articles = [] 46 | 47 | for feed in feeds: 48 | rss = feedparser.parse(feed) 49 | updated_time = rss.get('updated', str(datetime.datetime.utcnow())) 50 | 51 | # Only get the articles published in the last 7 days 52 | time_limit = delorean.parse(updated_time) - datetime.timedelta(days=7) 53 | for entry in rss.entries: 54 | # Normalise the time 55 | entry_time = delorean.parse(entry.published) 56 | entry_time.shift('UTC') 57 | if entry_time < time_limit: 58 | # Skip this entry 59 | continue 60 | 61 | # Get the article 62 | response = requests.get(entry.link) 63 | article = BeautifulSoup(response.text, 'html.parser') 64 | article_reference = (article.title.string.strip(), 65 | entry.summary.strip(), 66 | entry.link) 67 | article_text = article.get_text() 68 | 69 | for keyword in keywords: 70 | # match with the keyword. Notice the lower on both to 71 | # make it case-insensitive 72 | if keyword.lower() in article_text.lower(): 73 | articles.append(article_reference) 74 | break 75 | 76 | return articles 77 | 78 | 79 | def compose_email_body(articles, keywords, feed_list): 80 | ''' 81 | From the list of articles, keywords and feeds, fill the email template 82 | 83 | Set the list in the adequate format for the template 84 | ''' 85 | # Compose the list of articles 86 | ARTICLE_TEMPLATE = '* **{title}** {summary}: {link}' 87 | article_list = [ARTICLE_TEMPLATE.format(title=title, summary=summary, 88 | link=link) 89 | for title, summary, link in articles] 90 | 91 | data = { 92 | 'article_list': '\n'.join(article_list), 93 | 'keywords': ', '.join(keywords), 94 | 'feed_list': ', '.join(feed_list), 95 | } 96 | text = EMAIL_TEMPLATE.format(**data) 97 | 98 | html_content = mistune.markdown(text) 99 | html = jinja2.Template(EMAIL_STYLING).render(content=html_content) 100 | 101 | return text, html 102 | 103 | 104 | def send_email(email_config, text_body, html_body): 105 | ''' 106 | Send an email with the text and html body, using the parameters 107 | configured in email_config 108 | ''' 109 | msg = MIMEMultipart('alternative') 110 | msg['Subject'] = 'Weekly report' 111 | msg['From'] = email_config.from_ 112 | msg['To'] = email_config.to 113 | 114 | part_plain = MIMEText(text_body, 'plain') 115 | part_html = MIMEText(html_body, 'html') 116 | 117 | msg.attach(part_plain) 118 | msg.attach(part_html) 119 | 120 | with smtplib.SMTP('smtp.gmail.com', 587) as server: 121 | server.starttls() 122 | server.login(email_config.user, email_config.password) 123 | server.sendmail(email_config.from_, [email_config.to], msg.as_string()) 124 | 125 | 126 | def main(keywords, feeds, email_config): 127 | articles = get_articles(keywords, feeds) 128 | text_body, html_body = compose_email_body(articles, keywords, feeds) 129 | send_email(email_config, text_body, html_body) 130 | 131 | 132 | if __name__ == '__main__': 133 | parser = argparse.ArgumentParser() 134 | parser.add_argument(type=argparse.FileType('r'), dest='config', 135 | help='config file') 136 | 137 | args = parser.parse_args() 138 | config = configparser.ConfigParser() 139 | config.read_file(args.config) 140 | keywords = config['SEARCH']['keywords'].split(',') 141 | feeds = [feed.strip() for feed in config['SEARCH']['feeds'].split(',')] 142 | 143 | email_user = config['EMAIL']['user'] 144 | email_password = config['EMAIL']['password'] 145 | email_from = config['EMAIL']['from'] 146 | email_to = config['EMAIL']['to'] 147 | email_config = EmailConfig(email_user, email_password, email_from, 148 | email_to) 149 | 150 | main(keywords, feeds, email_config) 151 | -------------------------------------------------------------------------------- /Chapter10/send_notifications.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import configparser 3 | import os.path 4 | import csv 5 | import delorean 6 | import requests 7 | from twilio.rest import Client 8 | 9 | 10 | def send_phone_notification(entry, config): 11 | ACCOUNT_SID = config['TWILIO']['ACCOUNT_SID'] 12 | AUTH_TOKEN = config['TWILIO']['AUTH_TOKEN'] 13 | FROM = config['TWILIO']['FROM'] 14 | coupon = entry['Code'] 15 | TO = entry['Target'] 16 | text = f'Congrats! Here is a redeemable coupon! {coupon}' 17 | 18 | try: 19 | client = Client(ACCOUNT_SID, AUTH_TOKEN) 20 | client.messages.create(body=text, from_=FROM, to=TO) 21 | except Exception as err: 22 | return 'ERROR' 23 | 24 | return 'SENT' 25 | 26 | 27 | def send_email_notification(entry, config): 28 | KEY = config['MAILGUN']['KEY'] 29 | DOMAIN = config['MAILGUN']['DOMAIN'] 30 | FROM = config['MAILGUN']['FROM'] 31 | TO = entry['Target'] 32 | name = entry['Name'] 33 | auth = ('api', KEY) 34 | coupon = entry['Code'] 35 | text = f'Congrats! Here is a redeemable coupon! {coupon}' 36 | 37 | data = { 38 | 'from': f'Sender <{FROM}>', 39 | 'to': f'{name} <{TO}>', 40 | 'subject': 'You have a coupon!', 41 | 'text': text, 42 | } 43 | response = requests.post(f"https://api.mailgun.net/v3/{DOMAIN}/messages", 44 | auth=auth, data=data) 45 | if response.status_code == 200: 46 | return 'SENT' 47 | 48 | return 'ERROR' 49 | 50 | 51 | def send_notification(entry, send, config): 52 | if not send: 53 | return entry 54 | 55 | # Route each of the notifications 56 | METHOD = { 57 | 'PHONE': send_phone_notification, 58 | 'EMAIL': send_email_notification, 59 | } 60 | try: 61 | method = METHOD[entry['Contact Method']] 62 | result = method(entry, config) 63 | except KeyError: 64 | result = 'INVALID_METHOD' 65 | 66 | entry['Timestamp'] = delorean.utcnow().datetime.isoformat() 67 | entry['Status'] = result 68 | return entry 69 | 70 | 71 | def save_file(notif_file, data): 72 | ''' 73 | Overwrite the file with the new information 74 | ''' 75 | 76 | # Start at the start of the file 77 | notif_file.seek(0) 78 | 79 | header = data[0].keys() 80 | writer = csv.DictWriter(notif_file, fieldnames=header) 81 | writer.writeheader() 82 | writer.writerows(data) 83 | 84 | # Be sure to write to disk 85 | notif_file.flush() 86 | 87 | 88 | def main(data, codes, notif_file, config, send): 89 | # Go through each line that is not sent 90 | for index, entry in enumerate(data): 91 | if entry['Status'] == 'SENT': 92 | continue 93 | 94 | if not entry['Code']: 95 | if not codes: 96 | msg = ('The file is missing codes, and no code file ' 97 | 'has been defined') 98 | raise Exception(msg) 99 | entry['Code'] = codes.pop() 100 | 101 | entry = send_notification(entry, send, config) 102 | data[index] = entry 103 | 104 | # Save the data into the file 105 | save_file(notif_file, data) 106 | 107 | 108 | if __name__ == '__main__': 109 | parser = argparse.ArgumentParser() 110 | parser.add_argument(type=argparse.FileType('r+'), dest='notif_file', 111 | help='notifications file') 112 | parser.add_argument('-c', '--codes', type=argparse.FileType('r'), 113 | help='Optional file with codes. If present, the ' 114 | 'file will be populated with codes. ' 115 | 'No codes will be sent') 116 | parser.add_argument('--config', type=str, dest='config_file', 117 | default='config.ini', 118 | help='config file (detaulf config.ini)') 119 | args = parser.parse_args() 120 | 121 | # Read configuration 122 | if not os.path.isfile(args.config_file): 123 | print(f'Config file {args.config_file} is missing. Aborting') 124 | exit(1) 125 | with open(args.config_file) as fp: 126 | config = configparser.ConfigParser() 127 | config.read_file(fp) 128 | 129 | # Read data 130 | reader = csv.DictReader(args.notif_file) 131 | data = list(reader) 132 | 133 | codes = None 134 | send = True 135 | if args.codes: 136 | codes = [code_line[0] for code_line in csv.reader(args.codes)] 137 | send = False 138 | 139 | main(data=data, codes=codes, notif_file=args.notif_file, 140 | config=config, send=send) 141 | -------------------------------------------------------------------------------- /Chapter11/example_shop1.txt: -------------------------------------------------------------------------------- 1 | Hello: 2 | 3 | Are there any offers in fridges? I'm searching to replace mine. I live in a fifth floor and the lift is broken, would that be a problem? I'll be fine with paying an extra. 4 | 5 | Thanks a lot, 6 | Carrie 7 | -------------------------------------------------------------------------------- /Chapter11/example_shop2.txt: -------------------------------------------------------------------------------- 1 | Hello: 2 | 3 | Are there any offers in fridges? I'm looking to replace mine that is old. I live in a fifth floor and the lift is broken, would that be a problem? I'll be fine with paying an extra. 4 | I think you also have a furniture department, right? What are the prices for mattresses? 5 | 6 | Thanks a lot, 7 | Carrie 8 | -------------------------------------------------------------------------------- /Chapter11/example_shop3.txt: -------------------------------------------------------------------------------- 1 | Hello: 2 | 3 | I need your full details including your address and phone for an invoice. Can you please send them to me? 4 | 5 | Thanks a lot, 6 | Carrie 7 | -------------------------------------------------------------------------------- /Chapter11/handwrite.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter11/handwrite.jpg -------------------------------------------------------------------------------- /Chapter11/image_labels.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | from google.cloud import vision 3 | 4 | 5 | def landmark(client, image): 6 | print('Landmark detected') 7 | response = client.landmark_detection(image=image) 8 | landmarks = response.landmark_annotations 9 | for landmark in landmarks: 10 | print(f' {landmark.description}') 11 | for location in landmark.locations: 12 | coord = location.lat_lng 13 | print(f' Latitude {coord.latitude}') 14 | print(f' Longitude {coord.longitude}') 15 | 16 | if response.error.message: 17 | raise Exception( 18 | '{}\nFor more info on error messages, check: ' 19 | 'https://cloud.google.com/apis/design/errors'.format( 20 | response.error.message)) 21 | 22 | 23 | def main(image_file): 24 | content = image_file.read() 25 | 26 | client = vision.ImageAnnotatorClient() 27 | 28 | image = vision.types.Image(content=content) 29 | 30 | response = client.label_detection(image=image) 31 | labels = response.label_annotations 32 | print('Labels for the image and score:') 33 | 34 | for label in labels: 35 | print(label.description, label.score) 36 | if(label.description == 'Landmark'): 37 | landmark(client, image) 38 | 39 | if response.error.message: 40 | raise Exception( 41 | '{}\nFor more info on error messages, check: ' 42 | 'https://cloud.google.com/apis/design/errors'.format( 43 | response.error.message)) 44 | 45 | 46 | if __name__ == '__main__': 47 | parser = argparse.ArgumentParser() 48 | parser.add_argument(dest='input', type=argparse.FileType('rb'), 49 | help='input image') 50 | args = parser.parse_args() 51 | main(args.input) 52 | -------------------------------------------------------------------------------- /Chapter11/image_text.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | from google.cloud import vision 3 | 4 | 5 | def main(image_file, verbose): 6 | content = image_file.read() 7 | 8 | client = vision.ImageAnnotatorClient() 9 | image = vision.types.Image(content=content) 10 | response = client.document_text_detection(image=image) 11 | 12 | for page in response.full_text_annotation.pages: 13 | 14 | for block in page.blocks: 15 | 16 | if verbose: 17 | print('\nBlock confidence: {}\n'.format(block.confidence)) 18 | 19 | if block.confidence < 0.8: 20 | if verbose: 21 | print('Skipping block due to low confidence') 22 | continue 23 | 24 | for paragraph in block.paragraphs: 25 | paragraph_text = [] 26 | for word in paragraph.words: 27 | word_text = ''.join([ 28 | symbol.text for symbol in word.symbols 29 | ]) 30 | paragraph_text.append(word_text) 31 | if verbose: 32 | print(f'Word text: {word_text} ' 33 | f'(confidence: {word.confidence})') 34 | for symbol in word.symbols: 35 | print(f'\tSymbol: {symbol.text} ' 36 | f'(confidence: {symbol.confidence})') 37 | 38 | print(' '.join(paragraph_text)) 39 | 40 | if response.error.message: 41 | raise Exception( 42 | '{}\nFor more info on error messages, check: ' 43 | 'https://cloud.google.com/apis/design/errors'.format( 44 | response.error.message)) 45 | 46 | 47 | if __name__ == '__main__': 48 | parser = argparse.ArgumentParser() 49 | parser.add_argument(dest='input', type=argparse.FileType('rb'), 50 | help='input image') 51 | parser.add_argument('-v', dest='verbose', help='Print more data', 52 | action='store_true') 53 | args = parser.parse_args() 54 | main(args.input, args.verbose) 55 | -------------------------------------------------------------------------------- /Chapter11/image_text_box.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | from google.cloud import vision 3 | 4 | 5 | def main(image_file, verbose): 6 | content = image_file.read() 7 | 8 | client = vision.ImageAnnotatorClient() 9 | image = vision.types.Image(content=content) 10 | response = client.text_detection(image=image) 11 | 12 | print('Texts:') 13 | for text in response.text_annotations: 14 | print('"{}"'.format(text.description)) 15 | if verbose: 16 | points = ['({},{})'.format(p.x, p.y) 17 | for p in text.bounding_poly.vertices] 18 | print('box: {}'.format(','.join(points))) 19 | 20 | if response.error.message: 21 | raise Exception( 22 | '{}\nFor more info on error messages, check: ' 23 | 'https://cloud.google.com/apis/design/errors'.format( 24 | response.error.message)) 25 | 26 | 27 | if __name__ == '__main__': 28 | parser = argparse.ArgumentParser() 29 | parser.add_argument(dest='input', type=argparse.FileType('rb'), 30 | help='input image') 31 | parser.add_argument('-v', dest='verbose', help='Print more data', 32 | action='store_true') 33 | args = parser.parse_args() 34 | main(args.input, args.verbose) 35 | -------------------------------------------------------------------------------- /Chapter11/images/handwrite.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter11/images/handwrite.jpg -------------------------------------------------------------------------------- /Chapter11/images/photo-dublin-a-text.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter11/images/photo-dublin-a-text.jpg -------------------------------------------------------------------------------- /Chapter11/images/photo-dublin-a2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter11/images/photo-dublin-a2.png -------------------------------------------------------------------------------- /Chapter11/images/photo-dublin-b.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter11/images/photo-dublin-b.png -------------------------------------------------------------------------------- /Chapter11/images/photo-text.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter11/images/photo-text.jpg -------------------------------------------------------------------------------- /Chapter11/photo-dublin-a-text.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter11/photo-dublin-a-text.jpg -------------------------------------------------------------------------------- /Chapter11/photo-dublin-a2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter11/photo-dublin-a2.png -------------------------------------------------------------------------------- /Chapter11/photo-dublin-b.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter11/photo-dublin-b.png -------------------------------------------------------------------------------- /Chapter11/photo-text.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter11/photo-text.jpg -------------------------------------------------------------------------------- /Chapter11/shop.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter11/shop.zip -------------------------------------------------------------------------------- /Chapter11/shop_training/appliances/email1.txt: -------------------------------------------------------------------------------- 1 | Sorry, my washing machine broke today, and I am in desperate need of one. What are the available models you have in stock that I can buy today? I'll try to get to the shop later this afternoon 2 | -------------------------------------------------------------------------------- /Chapter11/shop_training/appliances/email10.txt: -------------------------------------------------------------------------------- 1 | I have a delivery that should come today for the order of a washing machine and a refrigerator during today. The order id is Y345654, I bought it last weekend. Any aproximate time is going to be delivered? I need to go out for grocery shopping, but I can adjust based on the delivery. 2 | 3 | Thank you, 4 | Anna 5 | -------------------------------------------------------------------------------- /Chapter11/shop_training/appliances/email2.txt: -------------------------------------------------------------------------------- 1 | Hello: 2 | 3 | What is the proper size for a washing machine for a family of three. Any recomendations? 4 | 5 | Thanks, 6 | Peter 7 | -------------------------------------------------------------------------------- /Chapter11/shop_training/appliances/email3.txt: -------------------------------------------------------------------------------- 1 | Good morning: 2 | 3 | I'm looking to replace my fridge. What are the available options for single door ones? It could include freezer, but it's not a requirement. 4 | 5 | Thanks, 6 | Greg 7 | -------------------------------------------------------------------------------- /Chapter11/shop_training/appliances/email4.txt: -------------------------------------------------------------------------------- 1 | My old refrigerator is not working properly. I'm thinkin on what are the current offers? Any fast delivery available? Thanks. 2 | -------------------------------------------------------------------------------- /Chapter11/shop_training/appliances/email5.txt: -------------------------------------------------------------------------------- 1 | I need a washing drier that is not too big. Not sure about the brand, I saw some good models in your website, can you tell me about options? 2 | -------------------------------------------------------------------------------- /Chapter11/shop_training/appliances/email6.txt: -------------------------------------------------------------------------------- 1 | Hello, I have a problem with a washing machine that I bought from you last week. It's leaking and leaving the floor totally flooded. Can you get someone to take a look? The guarantee should cover this. Please be quick, this is annoying. Thanks. 2 | -------------------------------------------------------------------------------- /Chapter11/shop_training/appliances/email7.txt: -------------------------------------------------------------------------------- 1 | Good morning, 2 | 3 | Are there any good fridges with double doors? I am changing the kitchen and was looking to get a bigger one. Ideally it should be a freezer in one and regular refrigerator in the other. And with an ice dispensator in front. If possible, in aluminum or chrome front. 4 | 5 | Thank you, 6 | Amanda 7 | -------------------------------------------------------------------------------- /Chapter11/shop_training/appliances/email8.txt: -------------------------------------------------------------------------------- 1 | I can't distinguish between the different fridges in your website. What are the differences between the model FDM4564 and the WOMDFX? They seem to have the same capacity and both have a small freezer in the bottom part, are they just different brands? The money difference between both models seems to be high if they are the same. Thanks. 2 | -------------------------------------------------------------------------------- /Chapter11/shop_training/appliances/email9.txt: -------------------------------------------------------------------------------- 1 | Good evening, 2 | 3 | I bought recently a washing machine model MD4550A and I don't know what program is suitable for delicate wool. The instruction manual is not very clear on that. Can you give advice? 4 | 5 | Thanks, 6 | Bob 7 | -------------------------------------------------------------------------------- /Chapter11/shop_training/furniture/email1.txt: -------------------------------------------------------------------------------- 1 | Hi: 2 | 3 | I am looking for a double bed. Do you have any offer in matresses? 4 | 5 | Thanks, 6 | Rob 7 | -------------------------------------------------------------------------------- /Chapter11/shop_training/furniture/email10.txt: -------------------------------------------------------------------------------- 1 | Hi, I bought a year ago a size table for the bedroom, it was mahogany colour, with three drawers and a dar top. Do you know the name of the model? Do you still have it available? Thanks a lot 2 | -------------------------------------------------------------------------------- /Chapter11/shop_training/furniture/email2.txt: -------------------------------------------------------------------------------- 1 | Good evening: 2 | 3 | I saw an offer on wardrobes. Is the shop open next sunday? 4 | 5 | Cheers 6 | -------------------------------------------------------------------------------- /Chapter11/shop_training/furniture/email3.txt: -------------------------------------------------------------------------------- 1 | I ordered recently a bed and some pieces of furniture, and haven't receive them yet. I got the order on the 3rd of December, to Alice. The order is number Y34234. Thanks 2 | -------------------------------------------------------------------------------- /Chapter11/shop_training/furniture/email4.txt: -------------------------------------------------------------------------------- 1 | Hello: 2 | 3 | Are the bed frames still on sale? 4 | 5 | Thanks, 6 | Karen 7 | -------------------------------------------------------------------------------- /Chapter11/shop_training/furniture/email5.txt: -------------------------------------------------------------------------------- 1 | Do you have tables of the Mara model in shop? I saw them on the website last week, but I can't find them any more. 2 | 3 | Thanks, 4 | Lacey 5 | -------------------------------------------------------------------------------- /Chapter11/shop_training/furniture/email6.txt: -------------------------------------------------------------------------------- 1 | Good morning. I'm wondering what is the size and weight of the king size bed model Florida, the one with the drawers below. Can you tell me, please? 2 | -------------------------------------------------------------------------------- /Chapter11/shop_training/furniture/email7.txt: -------------------------------------------------------------------------------- 1 | I have a wardrobe closet that I want to return. Any chance that you could pick it up? Will that have an extra cost? 2 | -------------------------------------------------------------------------------- /Chapter11/shop_training/furniture/email8.txt: -------------------------------------------------------------------------------- 1 | I bought a double size mattress and it's not the correct size for my frame. I think is the small double size instead of the big double size, or something. It's a bit bigger on the sizes. I'd like to swap it for one of the proper size. How fast could be done? I'm not able to sleep properly. 2 | -------------------------------------------------------------------------------- /Chapter11/shop_training/furniture/email9.txt: -------------------------------------------------------------------------------- 1 | Hi: 2 | 3 | I saw your advertising on TV and I'd like to get more information about your new mattresses. How much is the model X250 for a Queen Size? Please let me know as well the delivery cost if not included. Would it be possible to test it? 4 | 5 | Thank you, 6 | Mara 7 | -------------------------------------------------------------------------------- /Chapter11/shop_training/others/email1.txt: -------------------------------------------------------------------------------- 1 | Where are you located in Jamestown? I'm trying to find your address, but I can't find it in the web page. Thanks 2 | -------------------------------------------------------------------------------- /Chapter11/shop_training/others/email10.txt: -------------------------------------------------------------------------------- 1 | Hi: 2 | 3 | Sorry, but I've lost my keys and I think is possible it was in your shop on Tuesday. They have a red keyring with a lightling symbol on them. Any chance that you saw them? 4 | 5 | Thanks a lot, 6 | Hank 7 | -------------------------------------------------------------------------------- /Chapter11/shop_training/others/email2.txt: -------------------------------------------------------------------------------- 1 | Good morning: 2 | 3 | Are you open on next Saturday? What times? 4 | 5 | Thanks, 6 | James 7 | -------------------------------------------------------------------------------- /Chapter11/shop_training/others/email3.txt: -------------------------------------------------------------------------------- 1 | Good morning: 2 | 3 | I'm selling some industrial cleaning products that could be of your interest for keeping the comercial spaces in pristine condition. 4 | 5 | Who can I contact to make a free demonstration in your place? 6 | 7 | Thanks, 8 | The Superclean Team 9 | -------------------------------------------------------------------------------- /Chapter11/shop_training/others/email4.txt: -------------------------------------------------------------------------------- 1 | Hi! Do you sell lamps? Thanks 2 | -------------------------------------------------------------------------------- /Chapter11/shop_training/others/email5.txt: -------------------------------------------------------------------------------- 1 | Good evening: 2 | 3 | What are the opening hours on weekends? 4 | 5 | Thanks, 6 | Jim 7 | -------------------------------------------------------------------------------- /Chapter11/shop_training/others/email6.txt: -------------------------------------------------------------------------------- 1 | Hello: 2 | 3 | I was looking for an employee called Christian. I talked to him about some sales. Is he around? Can I talk to him? 4 | 5 | Deborah 6 | -------------------------------------------------------------------------------- /Chapter11/shop_training/others/email7.txt: -------------------------------------------------------------------------------- 1 | I am selling this fine leather jackets. 2 | -------------------------------------------------------------------------------- /Chapter11/shop_training/others/email8.txt: -------------------------------------------------------------------------------- 1 | Is this the car sales? Do you know their contact? I'm sure they are in the same location... 2 | -------------------------------------------------------------------------------- /Chapter11/shop_training/others/email9.txt: -------------------------------------------------------------------------------- 1 | Good morning: 2 | 3 | I'm contacting you on behalf of the electrical company. There is going to be some works for the whole commercial area during May. We don't expect any interruption of the supply, but won't be able to guarantee it. Please contact us as soon as possible if you think this will call an inconvenience. 4 | 5 | Thank you, 6 | The Electrical Company 7 | -------------------------------------------------------------------------------- /Chapter11/text_analysis.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | from google.cloud import language 3 | from google.cloud import translate_v2 as translate 4 | from google.cloud.language import enums 5 | from google.cloud.language import types 6 | 7 | 8 | def main(image_file): 9 | content = image_file.read() 10 | print(f'Text: {content}') 11 | document = types.Document(content=content, 12 | type=enums.Document.Type.PLAIN_TEXT) 13 | 14 | client = language.LanguageServiceClient() 15 | 16 | response = client.analyze_sentiment(document=document) 17 | lang = response.language 18 | print(f'Language: {lang}') 19 | sentiment = response.document_sentiment 20 | score = sentiment.score 21 | magnitude = sentiment.magnitude 22 | print(f'Sentiment Score (how positive the sentiment is): {score}') 23 | print(f'Sentiment Magnitude (how strong it is): {magnitude}') 24 | if lang != 'en': 25 | # Translate into English 26 | translate_client = translate.Client() 27 | response = translate_client.translate(content, target_language='en') 28 | print('IN ENGLISH') 29 | print(response['translatedText']) 30 | 31 | 32 | if __name__ == '__main__': 33 | parser = argparse.ArgumentParser() 34 | parser.add_argument(dest='input', type=argparse.FileType('r'), 35 | help='input text') 36 | args = parser.parse_args() 37 | main(args.input) 38 | -------------------------------------------------------------------------------- /Chapter11/text_analysis_categories.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | from google.cloud import language 3 | from google.cloud.language import enums 4 | from google.cloud.language import types 5 | 6 | 7 | def main(image_file): 8 | content = image_file.read() 9 | print(f'Text: {content}') 10 | document = types.Document(content=content, 11 | type=enums.Document.Type.PLAIN_TEXT) 12 | 13 | client = language.LanguageServiceClient() 14 | 15 | print('Categories') 16 | response = client.classify_text(document=document) 17 | if not response.categories: 18 | print('No categories detected') 19 | 20 | for category in response.categories: 21 | print(f'Category: {category.name}') 22 | print(f'Confidence: {category.confidence}') 23 | 24 | 25 | if __name__ == '__main__': 26 | parser = argparse.ArgumentParser() 27 | parser.add_argument(dest='input', type=argparse.FileType('r'), 28 | help='input text') 29 | args = parser.parse_args() 30 | main(args.input) 31 | -------------------------------------------------------------------------------- /Chapter11/text_predict.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | 3 | from google.api_core.client_options import ClientOptions 4 | from google.cloud import automl_v1 5 | 6 | 7 | def main(input_file, model_name): 8 | content = input_file.read() 9 | options = ClientOptions(api_endpoint='automl.googleapis.com') 10 | prediction_client = automl_v1.PredictionServiceClient( 11 | client_options=options 12 | ) 13 | payload = {'text_snippet': {'content': content, 'mime_type': 'text/plain'}} 14 | params = {} 15 | request = prediction_client.predict(model_name, payload, params) 16 | for result in request.payload: 17 | label = result.display_name 18 | match = result.classification.score 19 | print(f'Label: {label} : {match:.5f}') 20 | 21 | 22 | if __name__ == '__main__': 23 | parser = argparse.ArgumentParser() 24 | parser.add_argument(dest='input', type=argparse.FileType('r'), 25 | help='input text') 26 | parser.add_argument('-m', dest='model', type=str, help='model ref') 27 | args = parser.parse_args() 28 | 29 | main(args.input, args.model) 30 | -------------------------------------------------------------------------------- /Chapter11/texts/category_example.txt: -------------------------------------------------------------------------------- 1 | This text talks about literature and different authors from the XIX century. It discusses the different styles from different authors in different languages, analysing and comparing them with their historical context. 2 | -------------------------------------------------------------------------------- /Chapter11/texts/crime-and-punishement.txt: -------------------------------------------------------------------------------- 1 | On an exceptionally hot evening early in July a young man came out of the garret in which he lodged in S. Place and walked slowly, as though in hesitation, towards K. bridge. 2 | 3 | He had successfully avoided meeting his landlady on the staircase. His garret was under the roof of a high, five-storied house and was more like a cupboard than a room. The landlady who provided him with garret, dinners, and attendance, lived on the floor below, and every time he went out he was obliged to pass her kitchen, the door of which invariably stood open. And each time he passed, the young man had a sick, frightened feeling, which made him scowl and feel ashamed. He was hopelessly in debt to his landlady, and was afraid of meeting her. 4 | 5 | This was not because he was cowardly and abject, quite the contrary; but for some time past he had been in an overstrained irritable condition, verging on hypochondria. He had become so completely absorbed in himself, and isolated from his fellows that he dreaded meeting, not only his landlady, but anyone at all. He was crushed by poverty, but the anxieties of his position had of late ceased to weigh upon him. He had given up attending to matters of practical importance; he had lost all desire to do so. Nothing that any landlady could do had a real terror for him. But to be stopped on the stairs, to be forced to listen to her trivial, irrelevant gossip, to pestering demands for payment, threats and complaints, and to rack his brains for excuses, to prevaricate, to lie—no, rather than that, he would creep down the stairs like a cat and slip out unseen. 6 | 7 | This evening, however, on coming out into the street, he became acutely aware of his fears. 8 | -------------------------------------------------------------------------------- /Chapter11/texts/crimen-y-castigo.txt: -------------------------------------------------------------------------------- 1 | Una tarde extremadamente calurosa de principios de julio, un joven salió de la reducida habitación que tenía alquilada en la callejuela de S... y, con paso lento e indeciso, se dirigió 2 | al puente K... 3 | 4 | Había tenido la suerte de no encontrarse con su patrona en la escalera. Su cuartucho se hallaba bajo el tejado de un gran edificio de cinco pisos y, más que una habitación, parecía una alacena. En cuanto a la patrona, que le había alquilado el cuarto con servicio y pensión, ocupaba un departamento del piso de abajo; de modo que nuestro joven, cada vez que salía, se veía obligado a pasar por delante de la puerta de la cocina, que daba a la escalera y estaba casi siempre abierta de par en par. En esos momentos experimentaba invariablemente una sensación ingrata de vago temor, que le humillaba y daba a su semblante una expresión sombría. Debía una cantidad considerable a la patrona y por eso temía encontrarse con ella. 5 | 6 | No es que fuera un cobarde ni un hombre abatido por la vida. Por el contrario, se hallaba desde hacía algún tiempo en un estado de irritación, de tensión incesante, que rayaba en la hipocondría. Se había habituado a vivir tan encerrado en sí mismo, tan aislado, que no sólo temía encontrarse con su patrona, sino que rehuía toda relación con sus semejantes. La pobreza le abrumaba. Sin embargo, últimamente esta miseria había dejado de ser para él un sufrimiento. El joven había renunciado a todas sus ocupaciones diarias, a todo trabajo. 7 | -------------------------------------------------------------------------------- /Chapter11/texts/pride_and_prejudice.txt: -------------------------------------------------------------------------------- 1 | It is a truth universally acknowledged, that a single man in possession of a good fortune, must be in want of a wife. However little known the feelings or views of such a man may be on his first entering a neighbourhood, this truth is so well fixed in the minds of the surrounding families, that he is considered the rightful property of some one or other of their daughters. 2 | -------------------------------------------------------------------------------- /Chapter11/texts/regenta.txt: -------------------------------------------------------------------------------- 1 | La heroica ciudad dormía la siesta. El viento Sur, caliente y perezoso, empujaba las nubes blanquecinas que se rasgaban al correr hacia el Norte. En las calles no había más ruido que el rumor estridente de los remolinos de polvo, trapos, pajas y papeles que iban de arroyo en arroyo, de acera en acera, de esquina en esquina revolando y persiguiéndose, como mariposas que se buscan y huyen y que el aire envuelve en sus pliegues invisibles. Cual turbas de pilluelos, aquellas migajas de la basura, aquellas sobras de todo se juntaban en un montón, parábanse como dormidas un momento y brincaban de nuevo sobresaltadas, dispersándose, trepando unas por las paredes hasta los cristales temblorosos de los faroles, otras hasta los carteles de papel mal pegado a las esquinas, y había pluma que llegaba a un tercer piso, y arenilla que se incrustaba para días, o para años, en la vidriera de un escaparate, agarrada a un plomo. 2 | -------------------------------------------------------------------------------- /Chapter11/texts/tale_two_cities.txt: -------------------------------------------------------------------------------- 1 | It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way – in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only. 2 | -------------------------------------------------------------------------------- /Chapter12/code/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter12/code/__init__.py -------------------------------------------------------------------------------- /Chapter12/code/code_fixtures.py: -------------------------------------------------------------------------------- 1 | from zipfile import ZipFile 2 | 3 | INTERNAL_FILE = 'internal.txt' 4 | 5 | 6 | def write_zipfile(filename, content): 7 | 8 | with ZipFile(filename, 'w') as zipfile: 9 | zipfile.writestr(INTERNAL_FILE, content) 10 | 11 | 12 | def read_zipfile(filename): 13 | with ZipFile(filename, 'r') as zipfile: 14 | with zipfile.open(INTERNAL_FILE) as intfile: 15 | content = intfile.read() 16 | 17 | return content.decode('utf8') 18 | -------------------------------------------------------------------------------- /Chapter12/code/code_requests.py: -------------------------------------------------------------------------------- 1 | import requests 2 | from datetime import datetime, timedelta 3 | 4 | 5 | RECIPES = { 6 | 'DEFAULT': { 7 | 'size': 'small', 8 | 'topping': ['bacon', 'onion'], 9 | }, 10 | 'SPECIAL': { 11 | 'size': 'large', 12 | 'topping': ['bacon', 'mushroom', 'onion'], 13 | } 14 | } 15 | 16 | 17 | def order_pizza(recipe='DEFAULT'): 18 | 19 | delivery_time = datetime.now() + timedelta(hours=1) 20 | delivery = delivery_time.strftime('%H:%M') 21 | 22 | data = { 23 | 'custname': "Sean O'Connell", 24 | 'custtel': '123-456-789', 25 | 'custemail': 'sean@oconnell.ie', 26 | # Indicate the time 27 | 'delivery': delivery, 28 | 'comments': '' 29 | } 30 | 31 | extra_info = RECIPES[recipe] 32 | data.update(extra_info) 33 | resp = requests.post('https://httpbin.org/post', data) 34 | return resp.json()['form'] 35 | -------------------------------------------------------------------------------- /Chapter12/code/dependencies.py: -------------------------------------------------------------------------------- 1 | PI = 3.14159 2 | 3 | 4 | def rectangle(sideA, sideB): 5 | return sideA * sideB 6 | 7 | 8 | def circle(radius): 9 | return 2 * PI * radius 10 | 11 | 12 | def calculate_area(shape, sizeA, sizeB=0): 13 | if sizeA <= 0: 14 | raise ValueError('sizeA needs to be positive') 15 | 16 | if sizeB < 0: 17 | raise ValueError('sizeB needs to be positive') 18 | 19 | if shape == 'SQUARE': 20 | return rectangle(sizeA, sizeA) 21 | 22 | if shape == 'RECTANGLE': 23 | return rectangle(sizeA, sizeB) 24 | 25 | if shape == 'CIRCLE': 26 | return circle(sizeA) 27 | 28 | raise Exception(f'Shape {shape} not defined') 29 | -------------------------------------------------------------------------------- /Chapter12/code/external.py: -------------------------------------------------------------------------------- 1 | def division(a, b): 2 | return a / b 3 | -------------------------------------------------------------------------------- /Chapter12/conftest.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/0a6302820ce0d6d1fc038ccc5d4b698b098bae4c/Chapter12/conftest.py -------------------------------------------------------------------------------- /Chapter12/tests/test_case.py: -------------------------------------------------------------------------------- 1 | LIST = [1, 2, 3] 2 | 3 | 4 | def test_one(): 5 | # Nothing happens 6 | pass 7 | 8 | 9 | def test_two(): 10 | assert 2 == 1 + 1 11 | 12 | 13 | def test_three(): 14 | assert 3 in LIST 15 | 16 | 17 | def test_fail(): 18 | assert 4 in LIST 19 | -------------------------------------------------------------------------------- /Chapter12/tests/test_dependencies.py: -------------------------------------------------------------------------------- 1 | from unittest import mock 2 | from code.dependencies import calculate_area 3 | 4 | 5 | def test_square(): 6 | result = calculate_area('SQUARE', 2) 7 | 8 | assert result == 4 9 | 10 | 11 | def test_rectangle(): 12 | result = calculate_area('RECTANGLE', 2, 3) 13 | 14 | assert result == 6 15 | 16 | 17 | def test_circle_with_proper_pi(): 18 | result = calculate_area('CIRCLE', 2) 19 | 20 | assert result == 12.56636 21 | 22 | 23 | @mock.patch('code.dependencies.PI', 3) 24 | def test_circle_with_mocked_pi(): 25 | result = calculate_area('CIRCLE', 2) 26 | 27 | assert result == 12 28 | 29 | 30 | @mock.patch('code.dependencies.rectangle') 31 | def test_circle_with_mocked_rectangle(mocked_rectangle): 32 | mocked_rectangle.return_value = 12 33 | 34 | result = calculate_area('SQUARE', 2) 35 | 36 | assert result == 12 37 | mocked_rectangle.assert_called() 38 | -------------------------------------------------------------------------------- /Chapter12/tests/test_external.py: -------------------------------------------------------------------------------- 1 | import pytest 2 | from code.external import division 3 | 4 | 5 | def test_int_division(): 6 | assert 4 == division(8, 2) 7 | 8 | 9 | def test_float_division(): 10 | assert 3.5 == division(7, 2) 11 | 12 | 13 | def test_division_by_zero(): 14 | with pytest.raises(ZeroDivisionError): 15 | division(1, 0) 16 | -------------------------------------------------------------------------------- /Chapter12/tests/test_fixtures.py: -------------------------------------------------------------------------------- 1 | import os 2 | import random 3 | import string 4 | from pytest import fixture 5 | from zipfile import ZipFile 6 | from code.code_fixtures import write_zipfile, read_zipfile 7 | 8 | 9 | @fixture 10 | def fzipfile(): 11 | content_length = 50 12 | content = ''.join(random.choices(string.ascii_lowercase, k=content_length)) 13 | fnumber = ''.join(random.choices(string.digits, k=3)) 14 | 15 | filename = f'file{fnumber}.zip' 16 | 17 | write_zipfile(filename, content) 18 | yield filename, content 19 | 20 | os.remove(filename) 21 | 22 | 23 | def test_writeread_zipfile(): 24 | TESTFILE = 'test.zip' 25 | TESTCONTENT = 'This is a test' 26 | write_zipfile(TESTFILE, TESTCONTENT) 27 | content = read_zipfile(TESTFILE) 28 | 29 | assert TESTCONTENT == content 30 | 31 | 32 | def test_readwrite_zipfile(fzipfile): 33 | filename, expected_content = fzipfile 34 | content = read_zipfile(filename) 35 | 36 | assert content == expected_content 37 | 38 | 39 | def test_internal_zipfile(fzipfile): 40 | filename, expected_content = fzipfile 41 | EXPECTED_LIST = ['internal.txt'] 42 | 43 | # Verify only a single file exist in the zipfile 44 | with ZipFile(filename, 'r') as zipfile: 45 | assert zipfile.namelist() == EXPECTED_LIST 46 | -------------------------------------------------------------------------------- /Chapter12/tests/test_requests.py: -------------------------------------------------------------------------------- 1 | import pytest 2 | import requests 3 | import responses 4 | import urllib.parse 5 | from code.code_requests import order_pizza 6 | 7 | 8 | @responses.activate 9 | def test_order_pizza(): 10 | body = { 11 | 'form': { 12 | 'size': 'small', 13 | 'topping': ['bacon', 'onion'] 14 | } 15 | } 16 | responses.add(responses.POST, 'https://httpbin.org/post', 17 | json=body, status=200) 18 | 19 | result = order_pizza() 20 | assert result['size'] == 'small' 21 | # Decode the sent data 22 | encoded_body = responses.calls[0].request.body 23 | sent_data = urllib.parse.parse_qs(encoded_body) 24 | assert sent_data['size'] == ['small'] 25 | 26 | 27 | @responses.activate 28 | def test_order_pizza_timeout(): 29 | responses.add(responses.POST, 'https://httpbin.org/post', 30 | body=requests.exceptions.Timeout()) 31 | 32 | with pytest.raises(requests.exceptions.Timeout): 33 | order_pizza() 34 | -------------------------------------------------------------------------------- /Chapter12/tests/test_requests_time.py: -------------------------------------------------------------------------------- 1 | import responses 2 | import urllib.parse 3 | from freezegun import freeze_time 4 | from code.code_requests import order_pizza 5 | 6 | 7 | @responses.activate 8 | @freeze_time("2020-03-17T19:34") 9 | def test_order_time(): 10 | body = { 11 | 'form': { 12 | 'size': 'small', 13 | 'topping': ['bacon', 'onion'] 14 | } 15 | } 16 | responses.add(responses.POST, 'https://httpbin.org/post', 17 | json=body, status=200) 18 | 19 | order_pizza() 20 | # Decode the sent data 21 | encoded_body = responses.calls[0].request.body 22 | sent_data = urllib.parse.parse_qs(encoded_body) 23 | assert sent_data['delivery'] == ['20:34'] 24 | -------------------------------------------------------------------------------- /Chapter13/debug_algorithm.py: -------------------------------------------------------------------------------- 1 | def valid(candidate): 2 | if candidate <= 1: 3 | return False 4 | 5 | lower = candidate - 1 6 | while lower > 1: 7 | if candidate / lower == candidate // lower: 8 | return False 9 | lower -= 1 10 | 11 | return True 12 | 13 | 14 | assert not valid(1) 15 | assert valid(3) 16 | assert not valid(15) 17 | assert not valid(18) 18 | assert not valid(50) 19 | assert valid(53) 20 | -------------------------------------------------------------------------------- /Chapter13/debug_logging.py: -------------------------------------------------------------------------------- 1 | import logging 2 | logging.basicConfig(format='%(levelname)s:%(message)s', level=logging.INFO) 3 | 4 | 5 | def bubble_sort(alist): 6 | logging.info(f'Sorting the list: {alist}') 7 | for passnum in reversed(range(len(alist) - 1)): 8 | for i in range(passnum): 9 | if alist[i] > alist[i + 1]: 10 | alist[i], alist[i + 1] = alist[i + 1], alist[i] 11 | logging.debug(f'alist: {alist}') 12 | 13 | logging.info(f'Sorted list : {alist}') 14 | return alist 15 | 16 | 17 | assert [1, 2, 3, 4, 7, 10] == bubble_sort([3, 7, 10, 2, 4, 1]) 18 | -------------------------------------------------------------------------------- /Chapter13/debug_skills.py: -------------------------------------------------------------------------------- 1 | import requests 2 | import parse 3 | 4 | NAMES = [ 5 | 'John Smyth', 6 | 'Michael Craig', 7 | 'Poppy-Mae Pate', 8 | 'Vivienne Rennie', 9 | 'Fathima Mccabe', 10 | 'Mai Cordova', 11 | 'Rocío García', 12 | 'Roman Sullivan', 13 | 'John Paul Smith', 14 | "Séamus O'Carroll", 15 | 'Keagan Berg', 16 | ] 17 | 18 | sorted_names = [] 19 | for name in NAMES: 20 | data = { 21 | 'custname': name, 22 | } 23 | # Request the name from the server 24 | result = requests.get('http://httpbin.org/post', json=data) 25 | if result.status_code != 200: 26 | raise Exception(f'Error accessing server: {result}') 27 | # Obtain a raw name 28 | raw_result = result.json()['data'] 29 | # Extract the name from the result 30 | full_name = parse.search('"custname": "{name}"', raw_result)['name'] 31 | # Split it into first name and last name 32 | first_name, last_name = full_name.split() 33 | ready_name = f'{last_name}, {first_name}' 34 | # Add the name in last_name, first_name format to the list 35 | sorted_names.append(ready_name) 36 | 37 | # Properly sort the list and display the result 38 | sorted_names.sort() 39 | print(sorted_names) 40 | -------------------------------------------------------------------------------- /Chapter13/debug_skills_fixed.py: -------------------------------------------------------------------------------- 1 | import requests 2 | import parse 3 | 4 | NAMES = [ 5 | 'John Smyth', 6 | 'Michael Craig', 7 | 'Poppy-Mae Pate', 8 | 'Vivienne Rennie', 9 | 'Fathima Mccabe', 10 | 'Mai Cordova', 11 | 'Rocío García', 12 | 'Roman Sullivan', 13 | 'John Paul Smith', 14 | "Séamus O'Carroll", 15 | 'Keagan Berg', 16 | ] 17 | 18 | sorted_names = [] 19 | for name in NAMES: 20 | data = { 21 | 'custname': name, 22 | } 23 | # Request the name from the server 24 | # ERROR step 2. Using .get when it should be .post 25 | # (old) result = requests.get('http://httpbin.org/post', json=data) 26 | result = requests.post('http://httpbin.org/post', json=data) 27 | if result.status_code != 200: 28 | raise Exception(f'Error accessing server: {result}') 29 | # Obtain a raw name 30 | # ERROR Step 11. Obtain the value from a raw value. Use 31 | # the decoded JSON instead 32 | # raw_result = result.json()['data'] 33 | # Extract the name from the result 34 | # full_name = parse.search('"custname": "{name}"', raw_result)['name'] 35 | raw_result = result.json()['json'] 36 | full_name = raw_result['custname'] 37 | # Split it into first name and last name 38 | # ERROR step 6. split only two words. Some names has middle names 39 | # (old) first_name, last_name = full_name.split() 40 | first_name, last_name = full_name.rsplit(maxsplit=1) 41 | ready_name = f'{last_name}, {first_name}' 42 | # Add the name in last_name, first_name format to the list 43 | sorted_names.append(ready_name) 44 | 45 | # Properly sort the list and display the result 46 | sorted_names.sort() 47 | print(sorted_names) 48 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2020 Packt 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Python Automation Cookbook 2 | 3 | Python Automation Cookbook 4 | 5 | This is the code repository for [Python Automation Cookbook](https://www.packtpub.com/application-development/python-automation-cookbook?utm_source=github&utm_medium=repository&utm_campaign=9781789133806), published by Packt. 6 | 7 | **Explore the world of automation using Python recipes that will enhance your skills** 8 | 9 | ## What is this book about? 10 | Using a problem-solution-based approach, we'll show how you can automate all the boring stuff and let your computer do it for you instead of using the Python programming language. By the end of the book, you will have learned to identify problems and correct them to produce superior and reliable systems. 11 | 12 | This book covers the following exciting features: 13 | * Get to grips with scraping a website to detect changes 14 | * Search and process raw sales files to aggregate information in spreadsheets 15 | * Explore techniques to extract information from an Excel spreadsheet and generate exciting reports with graphs 16 | * Discover the techniques required to generate random, print-friendly codes to be used as single-use coupons 17 | * Automatically generate a marketing campaign, contacting the recipients over different channels 18 | * Identify and implement precise solutions 19 | If you feel this book is for you, get your [copy](https://www.amazon.com/dp/B07F2L2CDC) today! 20 | 21 | https://www.packtpub.com/ 23 | 24 | ## Instructions and Navigations 25 | All of the code is organized into folders. For example, Chapter02. 26 | 27 | The code will look like the following: 28 | ``` 29 | # IMPORTS 30 | from sale_log import SaleLog 31 | def get_logs_from_file(shop, log_filename): 32 | def main(log_dir, output_filename): 33 | ... 34 | if __name__ == '__main__': 35 | # PARSE COMMAND LINE ARGUMENTS AND CALL main() 36 | ``` 37 | 38 | **Following is what you need for this book:** 39 | Before reading this book, readers need to know the basics of the Python language. We do not assume that the reader is an expert in the language. 40 | The reader needs to know how to input commands in the command line (Terminal, Bash, or equivalent). 41 | To understand the code in this book, you need a text editor, which will enable you to read and edit the code. You can use an IDE that supports the Python language, such as PyCharm and PyDev—which you choose is up to you. Check out this link for ideas about 42 | IDEs: https://realpython.com/python-ides-code-editors-guide/. 43 | 44 | With the following software and hardware list you can run all code files present in the book. 45 | ### Software and Hardware List 46 | | Chapter | Software required | OS required | 47 | | -------- | ------------------------------------| -----------------------------------| 48 | | 1-10 | Python 3.7 https://www.python.org | Windows, macOS, and Linux (Any) | 49 | 50 | 51 | We also provide a PDF file that has color images of the screenshots/diagrams used in this book. [Click here to download it](https://www.packtpub.com/sites/default/files/downloads/9781789133806_ColorImages.pdf). 52 | 53 | ### Related products 54 | * Learn Python Programming - Second Edition [[Packt]](https://www.packtpub.com/application-development/learn-python-programming-second-edition?utm_source=github&utm_medium=repository&utm_campaign=9781788116662 ) [[Amazon]](https://www.amazon.com/dp/1788996666) 55 | 56 | * Mastering Python Design Patterns - Second Edition [[Packt]](https://www.packtpub.com/application-development/mastering-python-design-patterns-second-edition?utm_source=github&utm_medium=repository&utm_campaign=9781788837484 ) [[Amazon]](https://www.amazon.com/dp/B07FNXNXY7) 57 | 58 | 59 | ## Get to Know the Author 60 | **Jaime Buelta** 61 | has been a professional programmer and a full-time Python developer and has been exposed to a lot of different technologies over his career. He has developed software for a variety of fields and industries, including aerospace, networking and communications, industrial SCADA systems, video game online services, and finance services. As part of these companies, he worked closely with various areas, such as marketing, management, sales, and game design, helping the companies achieve to their goals. He is a strong proponent of automating everything and making computers do most of the heavy lifting so users can focus on the important stuff. He is currently living in Dublin, Ireland, and has been a regular speaker at PyCon Ireland. 62 | 63 | 64 | ### Suggestions and Feedback 65 | [Click here](https://docs.google.com/forms/d/e/1FAIpQLSdy7dATC6QmEL81FIUuymZ0Wy9vH1jHkvpY57OiMeKGqib_Ow/viewform) if you have any feedback or suggestions. 66 | 67 | 68 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | delorean==1.0.0 2 | requests==2.22.3 3 | parse==1.8.2 4 | beautifulsoup4==4.6.0 5 | feedparser==5.2.1 6 | Pillow==7.0.0 7 | xmltodict==0.11.0 8 | PyPDF2==1.26.0 9 | python-docx==0.8.6 10 | jinja2==2.11.1 11 | mistune==0.8.3 12 | fpdf==1.7.2 13 | pdf2image==0.1.14 14 | openpyxl==2.5.4 15 | matplotlib==2.2.2 16 | Fiona==1.7.13 17 | Shapely==1.6.4.post2 18 | twilio==6.16.3 19 | telepot==12.7 20 | 21 | 22 | 23 | Certain files will be created only after the steps are followed from the recipes. 24 | These files are not added in the repository. 25 | --------------------------------------------------------------------------------