├── LICENSE ├── README.md ├── app.py ├── extract_image.png └── requirements.txt /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2020 Artem Bugara 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # extract-news-api 2 | Flask code to deploy an API that pulls structured data from online news articles. 3 | 4 | This is a source code of an API that you can find here (free plan available): https://rapidapi.com/kotartemiy/api/extract-news 5 | 6 | 7 | 8 | 9 | ![](extract_image.png) 10 | 11 | 12 | ## Quick Start 13 | 1. Clone the repository to your local folder 14 | 15 | `git clone https://github.com/kotartemiy/extract-news-api.git` 16 | 17 | 2. Create a Python virtual environment (3.6+) 18 | 19 | `python -m venv env` 20 | 21 | 3. Activate the environment 22 | 23 | `source env/bin/activate` 24 | 25 | 4. Run `pip install -r extract-news-ap/requirements.txt` 26 | 5. Run `python extract-news-ap/app.py` in your terminal 27 | 28 | If everything is OK then you should be able to check your API on `http://127.0.0.1:5000/v0/article` 29 | 30 | Example of request: `http://127.0.0.1:5000/v0/article?url=https://www.nytimes.com/2020/03/21/arts/d-nice-instagram.html` 31 | 32 | ## Example of Response Body 33 | ``` javascript 34 | { 35 | "status": "ok", 36 | "article": { 37 | "source_url": "https://www.nytimes.com", 38 | "published": "Sat, 21 Mar 2020 23:31:12 GMT", 39 | "published_method_found": "Extracted from tag:\n", 40 | "published_guess_accuracy": "datetime", 41 | "title": "The Hottest Parties in Town Are Now Online", 42 | "text": "On Friday night there was one place in the country where you could take part in a social gathering and not be afraid of spreading or contracting the coronavirus.\n\nOver 4,000 people were in attendance, including headliners like Jennifer Lopez, Drake, Naomi Campbell, Diddy, Mary J. Blige, DJ Khaled, T.I., Queen Latifah and Tracee Ellis Ross.\n\nThere was no charge at the door, no security, no drink minimum and you could attend in your pajamas from the comfort of your own home.\n\nThe party, named Homeschoolin’, was easy to find: It was on D.J. D-Nice’s Instagram live.\n\nSince Wednesday, Derrick Jones, 49, more popularly known as D-Nice, has held streams of hourslong jam sessions from his home in Los Angeles. He plays all of the hits, new and old, but you never hear the same song twice.", 43 | "authors": [ 44 | "Sandra E. Garcia" 45 | ], 46 | "images": [ 47 | "https://static01.nyt.com/images/2020/03/21/us/21xp-virus-digitalparties-image/21xp-virus-digitalparties-image-articleLarge.jpg?quality=75&auto=webp&disable=upscale", 48 | "https://static01.nyt.com/images/2020/03/21/us/21xp-virus-digitalparties-image/21xp-virus-digitalparties-image-facebookJumbo.jpg" 49 | ], 50 | "top_image": "https://static01.nyt.com/images/2020/03/21/us/21xp-virus-digitalparties-image/21xp-virus-digitalparties-image-facebookJumbo.jpg", 51 | "meta_image": "https://static01.nyt.com/images/2020/03/21/us/21xp-virus-digitalparties-image/21xp-virus-digitalparties-image-facebookJumbo.jpg", 52 | "movies": [], 53 | "meta_keywords": [ 54 | "" 55 | ], 56 | "tags": [], 57 | "meta_description": "As many are quarantining or socially distancing themselves to curb the spread of the coronavirus, the party has moved to social media.", 58 | "meta_lang": "en", 59 | "title_lang": "en", 60 | "text_lang": "en", 61 | "meta_favicon": "/vi-assets/static-assets/favicon-4bf96cb6a1093748bf5b3c429accb9b4.ico" 62 | } 63 | } 64 | ``` 65 | 66 | ## Good to use in combination with 67 | [newscatcher](https://github.com/kotartemiy/newscatcher) 68 | 69 | Newscatcher Python library allows you to collect normalized news from (almost) any website. 70 | 71 | [Newscatcher API](https://newscatcherapi.com/) 72 | 73 | Newscatcher API is a news API that allows you to find the most relevant news artiles by searching for any keywod. 74 | 75 | 76 | ## Built with 77 | [Flask](https://github.com/pallets/flask) Copyright 2010 Pallets 78 | 79 | [newspaper](https://github.com/codelucas/newspaper) Copyright (c) 2013 Lucas Ou-Yang 80 | 81 | [date_guesser](https://github.com/mitmedialab/date_guesser) Copyright (c) 2018 MIT Center for Civic Media 82 | 83 | -------------------------------------------------------------------------------- /app.py: -------------------------------------------------------------------------------- 1 | from flask import Flask, jsonify, request 2 | from newspaper import Article 3 | from date_guesser import guess_date, Accuracy 4 | from langdetect import detect, detect_langs 5 | 6 | 7 | app = Flask(__name__) 8 | app.config['JSON_SORT_KEYS'] = False 9 | 10 | 11 | @app.route('/v0/article', methods=['GET']) 12 | def get_article(): 13 | url = None 14 | 15 | url = request.args.get('url', type=str) 16 | 17 | if url == None: 18 | return 'url parameter is required', 400 19 | 20 | article = Article(url) 21 | article.download() 22 | 23 | if (article.download_state == 2): 24 | article.parse() 25 | article_dict = {} 26 | article_dict['status'] = 'ok' 27 | 28 | article_dict['article'] = {} 29 | article_dict['article']['source_url'] = article.source_url 30 | 31 | 32 | try: 33 | guess = guess_date(url = url, html = article.html) 34 | article_dict['article']['published'] = guess.date 35 | article_dict['article']['published_method_found'] = guess.method 36 | article_dict['article']['published_guess_accuracy'] = None 37 | if guess.accuracy is Accuracy.PARTIAL: 38 | article_dict['article']['published_guess_accuracy'] = 'partial' 39 | if guess.accuracy is Accuracy.DATE: 40 | article_dict['article']['published_guess_accuracy'] = 'date' 41 | if guess.accuracy is Accuracy.DATETIME: 42 | article_dict['article']['published_guess_accuracy'] = 'datetime' 43 | if guess.accuracy is Accuracy.NONE: 44 | article_dict['article']['published_guess_accuracy'] = None 45 | except: 46 | article_dict['article']['published'] = article.publish_date 47 | article_dict['article']['published_method_found'] = None 48 | article_dict['article']['published_guess_accuracy'] = None 49 | 50 | article_dict['article']['title'] = article.title 51 | article_dict['article']['text'] = article.text 52 | article_dict['article']['authors'] = list(article.authors) 53 | 54 | try: 55 | title_lang = detect(article.title) 56 | except: 57 | title_lang = None 58 | 59 | 60 | try: 61 | text_lang = detect(article.text) 62 | except: 63 | text_lang = None 64 | 65 | article_dict['article']['images'] = list(article.images) 66 | article_dict['article']['top_image'] = article.top_image 67 | article_dict['article']['meta_image'] = article.meta_img 68 | article_dict['article']['movies'] = list(article.movies) 69 | article_dict['article']['meta_keywords'] = list(article.meta_keywords) 70 | article_dict['article']['tags'] = list(article.tags) 71 | article_dict['article']['meta_description'] = article.meta_description 72 | article_dict['article']['meta_lang'] = article.meta_lang 73 | article_dict['article']['title_lang'] = str(title_lang) 74 | article_dict['article']['text_lang'] = str(text_lang) 75 | article_dict['article']['meta_favicon'] = article.meta_favicon 76 | return jsonify(article_dict) 77 | 78 | else: 79 | article_dict = {} 80 | article_dict['status'] = 'error' 81 | article_dict['article'] = article.download_exception_msg 82 | return jsonify(article_dict) 83 | 84 | 85 | if __name__ == '__main__': 86 | app.run() -------------------------------------------------------------------------------- /extract_image.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kotartemiy/extract-news-api/3628a5d18fce9bb9b69f0426657be974d0c57e21/extract_image.png -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | argcomplete==1.11.1 2 | arrow==0.15.5 3 | beautifulsoup4==4.9.0 4 | boto3==1.12.39 5 | botocore==1.15.39 6 | certifi==2020.4.5.1 7 | cfn-flip==1.2.2 8 | chardet==3.0.4 9 | click==7.1.1 10 | cssselect==1.1.0 11 | date-guesser==2.1.4 12 | DateTime==4.3 13 | docutils==0.15.2 14 | durationpy==0.5 15 | feedfinder2==0.0.4 16 | feedparser==5.2.1 17 | Flask==1.1.2 18 | future==0.18.2 19 | hjson==3.0.1 20 | idna==2.9 21 | importlib-metadata==1.6.0 22 | itsdangerous==1.1.0 23 | jieba3k==0.35.1 24 | Jinja2==2.11.1 25 | jmespath==0.9.5 26 | kappa==0.6.0 27 | langdetect==1.0.8 28 | lxml==4.5.0 29 | MarkupSafe==1.1.1 30 | newspaper3k==0.2.8 31 | nltk==3.4.5 32 | Pillow==7.1.1 33 | pip-tools==4.5.1 34 | placebo==0.9.0 35 | python-dateutil==2.6.1 36 | python-slugify==4.0.0 37 | pytz==2019.3 38 | PyYAML==5.3.1 39 | requests==2.23.0 40 | requests-file==1.4.3 41 | s3transfer==0.3.3 42 | six==1.14.0 43 | soupsieve==2.0 44 | text-unidecode==1.3 45 | tinysegmenter==0.3 46 | tldextract==2.2.2 47 | toml==0.10.0 48 | tqdm==4.45.0 49 | troposphere==2.6.0 50 | urllib3==1.25.8 51 | Werkzeug==0.16.1 52 | wsgi-request-logger==0.4.6 53 | zappa==0.51.0 54 | zipp==3.1.0 55 | zope.interface==5.1.0 56 | --------------------------------------------------------------------------------