├── requirements.txt
├── README.md
└── noderank.py


/requirements.txt:
--------------------------------------------------------------------------------
1 | requests
2 | bs4
3 | numpy
4 | nltk
5 | networkx
6 | lxml
7 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
 1 | # NodeRank
 2 | Web Content Extraction using the PageRank algorithm to find the element containing the best content.
 3 | 
 4 | This project was primarily started as a way to extract quality content from web pages with reasonable accuracy, omitting chrome and other superfluous code, on serverless infrastructures like AWS Lambda. There are still a few hurdles on Lambda like making the Punkt (NLTK) library available and fitting within size restrictions of stored code, but it is doable, where in the case of Dragnet and Boilerpipe, dependancy and size issues make this a very difficult, perhaps impossible, task.
 5 | 
 6 | ## Running:
 7 | ``` python
 8 | # install requirements `pip install -r requirements.txt`
 9 | # Put the noderank.py file in your working directory
10 | import noderank
11 | url="https://salience.co.uk/insight/magazine/basic-log-file-analysis/"
12 | content - extract_content_noderank(url=url, timeout=10)
13 | ```
14 | 
15 | ## Idea
16 | Since there is a structural relationship between the nodes in the DOM of a webpage, can we use that structure to help identify the parent which is most likely to contain the quality content on the page.
17 | 
18 | This algorithm does the following.
19 | 1. The `body` element of the page is exracted using BeautifulSoup.
20 | 1. Unneccesary elements are stripped out of the `body` node. eg. `<script>`, `<style>`, etc.  This can be adjusted to suit your needs.
21 | 1. A matrix (`sim_mat`) is initialized with zeros and for each pairwise node combintation across the x and y axis. A "proability" is established based on the number of sentences of the child and the distance between the nodes.
22 | 1. This matrix is then passed to the networkx PageRank agorithm and the resulting nodes are ranked by relative strength.
23 | 1. The top node is returned and the text is gotten using the BeautifulSoup `get_text()` method.
24 | 1. We then apply some light NLP post-processing to clean things up a bit more.
25 | 
26 | ## Notes
27 | * To run the Jupyter notebook, you will need to install `dragnet` and `boilerpipe3` via pip.  Tested on Ubuntu.  Had issues installing dragnet on windows due to GNU depencencies.
28 | * If you can think of a way to improve or want to contribute, email me at `jroakes@gmail.com`.
29 | * Hamlet Batista supplied a Colaboraory link [here](https://colab.research.google.com/github/jroakes/NodeRank/blob/master/noderank.ipynb).
30 | 
31 | ## Acknowledgements
32 | * Ideas and code taken from [An Introduction to Text Summarization using the TextRank Algorithm with Python implementation](https://www.analyticsvidhya.com/blog/2018/11/introduction-text-summarization-textrank-python/).
33 | 
34 | 
35 | 
36 | 
37 | 
38 | 


--------------------------------------------------------------------------------
/noderank.py:
--------------------------------------------------------------------------------
  1 | import requests
  2 | from bs4 import BeautifulSoup
  3 | import random
  4 | import math
  5 | import string
  6 | import re
  7 | import numpy as np
  8 | import nltk
  9 | import networkx as nx
 10 | 
 11 | def getUA():
 12 |     uastrings = ["Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.111 Safari/537.36",\
 13 |                 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.72 Safari/537.36",\
 14 |                 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10) AppleWebKit/600.1.25 (KHTML, like Gecko) Version/8.0 Safari/600.1.25",\
 15 |                 "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:33.0) Gecko/20100101 Firefox/33.0",\
 16 |                 "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.111 Safari/537.36",\
 17 |                 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.111 Safari/537.36",\
 18 |                 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/600.1.17 (KHTML, like Gecko) Version/7.1 Safari/537.85.10",\
 19 |                 "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko",\
 20 |                 "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:33.0) Gecko/20100101 Firefox/33.0",\
 21 |                 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.104 Safari/537.36"\
 22 |                 ]
 23 | 
 24 |     return random.choice(uastrings)
 25 | 
 26 | def prepare_sentences(text):
 27 |     
 28 |     # Clean characters
 29 |     text = "".join([t for t in text if t.isalnum() or t in string.punctuation or t == ' '])
 30 |     
 31 |     # Get rid of whitespace characters
 32 |     sentences = nltk.sent_tokenize(text)
 33 |     text = " ".join([s.strip() for s in sentences])
 34 |     
 35 |     # Fix puctuation spacing
 36 |     text = re.sub(r"(\w{2,})([\.\!\?]+)(\w)", r"\1\2 \3", text)
 37 |     
 38 |     return text
 39 | 
 40 | 
 41 | def find_children(n1, n2):
 42 |     
 43 |     children1 = [c for c in n1.children]
 44 |     children2 = [c for c in n2.children]
 45 |     
 46 |     for i, children in enumerate([children1,children2]):
 47 |         level = 1
 48 | 
 49 |         while children:
 50 |             lchildren = children
 51 |             children = []
 52 |             for ch in lchildren:
 53 |                 if not ch.name:
 54 |                     continue
 55 |                 if ch == n1 or ch == n2:
 56 |                     if i == 0:
 57 |                         return len( nltk.sent_tokenize(n2.get_text()) ) * ( 1/math.exp(level) )
 58 |                     else:
 59 |                         return len( nltk.sent_tokenize(n1.get_text()) ) * ( 1/math.exp(level) )
 60 |                                         
 61 |                 children.extend(ch.children)
 62 | 
 63 |             level += 1
 64 |             
 65 |     return 0
 66 | 
 67 | 
 68 | def find_content(html):
 69 |     
 70 |     # Get body and extract out non-content nodes
 71 |     soup = BeautifulSoup(html, 'lxml')
 72 |     body = soup.find('body')
 73 |     remove = [s.extract() for s in body(["script", "style","iframe","noscript","nav","footer","header", "svg", "h1","h2", "h3", "h4", "h5", "xml"])]
 74 |     text = ""
 75 |     
 76 |     # Further clean nodes and build a lookup list
 77 |     nodes = body.findAll()
 78 |     nodes = [n for n in nodes if n.name and n.get_text()]
 79 |     nodeix = [i for i,v in enumerate(nodes)]    
 80 | 
 81 |     # Build similarity matrix and use number of sentences child resursion depth as metric
 82 |     sim_mat = np.zeros([len(nodes), len(nodes)])
 83 |     for i in range(len(nodes)):
 84 |         for j in range(len(nodes)):
 85 |             if i != j:
 86 |                 sim_mat[i][j] = find_children(nodes[i], nodes[j])
 87 | 
 88 |     # Run pagerank algorithm on matrix
 89 |     nx_graph = nx.from_numpy_array(sim_mat)
 90 |     scores = nx.pagerank(nx_graph)
 91 |     
 92 |     # Sort nodex by best and get text of best node
 93 |     ranked_nodes = sorted(((scores[i],s) for i,s in enumerate(nodeix)), reverse=True)
 94 |     text = nodes[ranked_nodes[0][1]].get_text()
 95 |     
 96 |     return prepare_sentences(text)
 97 | 
 98 | 
 99 | def extract_content_noderank(url=None, html=None, timeout=10):
100 |     
101 |     if url:
102 |         headers = {'user-agent': getUA()}
103 |         r = requests.get(url, headers = headers, verify=False, timeout=timeout)
104 |         html = r.content
105 |     elif html:
106 |         pass
107 |     else:
108 |         raise Exception('You must supply either `url` or `html`')
109 |         
110 |     return find_content(html)
111 |     
112 | 
113 | # Test URLS
114 | url="https://www.nytimes.com/2019/02/05/technology/artificial-intelligence-drug-research-deepmind.html"
115 | #url="https://www.houstonchronicle.com/news/investigations/article/Southern-Baptist-sexual-abuse-spreads-as-leaders-13588038.php"
116 | #url="https://www.clearwaycommunitysolar.com/"
117 | #url="https://thenewstack.io/dr-michael-stonebraker-a-short-history-of-database-systems/"
118 | 
119 | print(extract_content_noderank(url=url, timeout=10))
120 | 


--------------------------------------------------------------------------------