├── .gitignore ├── README.md ├── const.py ├── motion.py └── requirements.txt /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | site/ 3 | .netlify 4 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # 🚀 Static Motion [DISCONTINUED] 2 | 3 | Make your static site from [Notion.so](https://notion.so) with the help of Netlify and Travis CI. 4 | 5 | [![Build Status](https://travis-ci.org/alanzchen/static-motion.svg?branch=master)](https://travis-ci.org/alanzchen/static-motion) 6 | 7 | **Demo:** 8 | 9 | - [Project Homepage](https://staticmotion.zenan.ch) 10 | - [Disqus Integration](https://staticmotion.zenan.ch/disqus-integration-demo) 11 | 12 | ## Installation 13 | 14 | How do I install this… **No! You do not need to install it on your machine.** 15 | 16 | All you need is a Github account. 17 | 18 | ## Get Started 19 | 20 | 1. Fork this project. 21 | 2. Turn on Travis CI for your forked repository. 22 | 3. Get your access token at Netlify and set the following environment variables on Travis CI's setting page: 23 | See `conf.py` for example. 24 | - `ACCESS_TOKEN`: fill your Netlify access token. 25 | (Remember to secure it!) 26 | - `SITE_ID`: The "API ID" of your Netlify site. 27 | - `index`: The URL of your index page on [Notion.so](https://notion.so). 28 | - `title_sep`: Your title separator. No space around! 29 | - `description`: Your site description. "Remember to quote your string!" 30 | - `base_url`: Your site URL. 31 | - `twitter`: Your twitter ID. 32 | Declare an empty string if you do not want it. 33 | - `build_mobile`: Will build a mobile version of your entire site at /m/ if this option presents. 34 | - `apple_touch_icon`: the URL of your `apple-touch-icon.png` hosted externally. 35 | - `favicon`: the URL of your `favicon.ico` hosted externally. 36 | - `anchor`: Enable anchor detection if this option presents. All urls starting with `https://www.notion.so` with `#` somewhere in the url will be rewritten to your relative path. 37 | 4. Bang! Wait a few minutes and your site will be up! 38 | 5. Want to have Google Analytics and Disqus? See [customization](https://staticmotion.zenan.ch/customization). 39 | 40 | ## Updating Your Site 41 | 42 | You can set up [cron jobs](https://docs.travis-ci.com/user/cron-jobs/) in your Travis CI project. However, if you wish to update your site immediately, please manually rebuild your project in Travis CI. 43 | 44 | Committing directly to your Github repository to trigger Travis CI rebuild **is highly discouraged**. 45 | 46 | ## Upgrading Your Static Motion 47 | 48 | Simply merge this repository with your repository. 49 | 50 | ## Debugging 51 | 52 | As [notion.so](https://notion.so) make changes to their template, this utility may break at any time. 53 | 54 | Before you go, make sure you have: 55 | 56 | - Python 3.6, 57 | - the latest Google Chrome, 58 | - a working `chromedriver` . 59 | 60 | Once you are ready, clone this repo and cd into it. Install the dependency: 61 | 62 | ``` 63 | pip install -r requirements.txt 64 | ``` 65 | 66 | Then run the static site generator. 67 | 68 | ``` 69 | python motion.py 70 | ``` 71 | 72 | ## Known Issue 73 | 74 | - Embedded elements might not work. 75 | - Embedded tweets are not working. 76 | - Fonts in code blocks may flicker slightly. 77 | - Pages with the same base url but with different hash cannot coexist in the same site. 78 | -------------------------------------------------------------------------------- /const.py: -------------------------------------------------------------------------------- 1 | assets = ["index.appcache", 2 | "all.css", 3 | "css-print/print.css", 4 | "fonts/lyon-text-regular-italic.woff", 5 | "fonts/lyon-text-regular.woff", 6 | "fonts/lyon-text-semibold-italic.woff", 7 | "fonts/lyon-text-semibold.woff", 8 | "fonts/nitti-medium-italic.woff", 9 | "fonts/nitti-medium.woff", 10 | "fonts/nitti-normal-italic.woff", 11 | "fonts/nitti-normal.woff"] 12 | -------------------------------------------------------------------------------- /motion.py: -------------------------------------------------------------------------------- 1 | from selenium import webdriver 2 | from selenium.webdriver.chrome.options import Options 3 | from bs4 import BeautifulSoup 4 | from os.path import exists 5 | from os import mkdir 6 | import requests 7 | import time 8 | from const import assets 9 | from os import environ as options 10 | 11 | 12 | blacklist = ["inter", "segment", "facebook", "fullstory", "loggly.js", "app-"] 13 | notions = {} 14 | user_agent = 'Mozilla/5.0 (iPhone; CPU iPhone OS 10_0 like Mac OS X) AppleWebKit/602.1.38 (KHTML, like Gecko) Version/10.0 Mobile/14A300 Safari/602.1' 15 | visited = set() 16 | notions = {} 17 | 18 | 19 | def motion(is_mobile=False): 20 | global visited, notions 21 | chrome_options = Options() 22 | chrome_options.add_argument("--headless") 23 | chrome_options.add_argument("--disable-gpu") 24 | visited = set() 25 | notions = {} 26 | print("Parsing index...") 27 | if is_mobile: 28 | print("Building mobile version...") 29 | chrome_options.add_argument('--user-agent=' + user_agent) 30 | driver = webdriver.Chrome(chrome_options=chrome_options) 31 | n = Notion(options['index'], driver, options=options, is_index=True, is_mobile=is_mobile) 32 | n.mod() 33 | print("Index page looks good.") 34 | n.walk() 35 | for notion in notions.values(): 36 | notion.save() 37 | n.save() 38 | print("Site generated successfully.") 39 | driver.quit() 40 | 41 | 42 | def download_file(url, local_filename, overwrite=False): 43 | # https://stackoverflow.com/questions/16694907/how-to-download-large-file-in-python-with-requests-py#16696317 44 | if local_filename.startswith("/"): 45 | local_filename = local_filename[1:] 46 | local_filename = "site/" + local_filename 47 | if exists(local_filename) and not overwrite: 48 | print("File " + local_filename + " found. Skipping.") 49 | return 50 | md(local_filename) 51 | print("Downloading: " + url + " to " + local_filename) 52 | r = requests.get(url, stream=True) 53 | with open(local_filename, 'wb') as f: 54 | for chunk in r.iter_content(chunk_size=1024): 55 | if chunk: # filter out keep-alive new chunks 56 | f.write(chunk) 57 | # f.flush() commented by recommendation from J.F.Sebastian 58 | return local_filename 59 | 60 | 61 | def md(local_filename): 62 | folders = [i for i in local_filename.split("/") if i] 63 | if folders: 64 | tmp = folders[0] 65 | for folder in folders[1:]: 66 | if not exists(tmp): 67 | print("Making missing directory: " + tmp) 68 | mkdir(tmp) 69 | tmp += "/" + folder 70 | 71 | 72 | class Notion: 73 | def __init__(self, url, driver, options=None, is_index=False, wait=0, is_mobile=False): 74 | self.driver = driver 75 | print("Visiting " + url) 76 | self.is_index = is_index 77 | self.is_mobile = is_mobile 78 | self.url = url 79 | if url.startswith("https://"): 80 | self.driver.get(url) 81 | else: 82 | self.driver.get("https://notion.so" + url) 83 | time.sleep(wait) 84 | self.dom = BeautifulSoup(driver.page_source, "html.parser") 85 | self.source = self.driver.page_source.replace('', '!(notion)!') 86 | self.wait_spinner() 87 | self.divs = [d for d in self.dom.find_all("div") if d.has_attr("data-block-id")] 88 | self.links = set() 89 | if not options: 90 | options = {} 91 | self.options = options 92 | if is_index: 93 | self.filename = "index.html" 94 | self.init_site() 95 | else: 96 | url_list = url.split("/")[-1].split('-') 97 | if len(url_list) > 1: 98 | self.filename = '-'.join(url_list[:-1]) + ".html" 99 | else: 100 | self.filename = url_list[0] + ".html" 101 | 102 | def wait_spinner(self): 103 | i = 0 104 | self.dom = BeautifulSoup(self.driver.page_source.replace('', '!(notion)!'), "html.parser") 105 | while (self.dom.find(class_="loading-spinner")): 106 | i += 1 107 | print("Waiting for spinner... " + str(i)) 108 | time.sleep(1) 109 | self.dom = BeautifulSoup(self.driver.page_source.replace('', '!(notion)!'), "html.parser") 110 | self.source = self.driver.page_source 111 | 112 | def init_site(self): 113 | for f in assets: 114 | download_file("https://notion.so/" + f, f) 115 | if 'favicon' in self.options: 116 | download_file(self.options['favicon'], "images/favicon.ico") 117 | if 'apple_touch_icon' in self.options: 118 | download_file(self.options['apple_touch_icon'], 'images/logo-ios.png') 119 | if 'atom' in self.options: 120 | download_file(self.options['atom'], 'feed', overwrite=True) 121 | 122 | def mod(self, tries=0): 123 | try: 124 | self.save_assets() 125 | self.meta() 126 | # self.remove_overlay() 127 | self.clean() 128 | self.parse_links() 129 | self.remove_scripts() 130 | self.iframe() 131 | self.div() 132 | self.disqus() 133 | self.gen_html() 134 | except Exception as e: 135 | print(e) 136 | if tries > 3: 137 | print("Unhandled Error, aborting...") 138 | print('--------------') 139 | print(self.source) 140 | raise 141 | else: 142 | sec = tries * 5 143 | print("Exception occurred, sleep for,", sec, "secs and retry...") 144 | time.sleep(sec) 145 | self.dom = BeautifulSoup(self.driver.page_source.replace('', '!(notion)!'), "html.parser") # Reset the DOM 146 | self.source = self.driver.page_source.replace('', '!(notion)!') 147 | self.mod(tries + 1) 148 | 149 | def save(self): 150 | if self.is_mobile: 151 | local_filename = "site/m/" + self.filename 152 | else: 153 | local_filename = "site/" + self.filename 154 | md(local_filename) 155 | with open(local_filename, "w") as f: 156 | f.write(self.html) 157 | 158 | def gen_html(self): 159 | s = str(self.dom) 160 | self.html = s.replace('!(notion)!', '') 161 | 162 | def clean(self): 163 | cursor_div = self.dom.find(class_='notion-cursor-listener') 164 | if cursor_div: 165 | css = cursor_div["style"] 166 | cursor_div["style"] = ";".join([i for i in css.strip().split(";") 167 | if 'cursor' not in i]) 168 | wrapper_div = [i for i in self.dom.find_all("div") 169 | if i.has_attr('style') and 'padding-bottom: 30vh;' in i['style']] 170 | if wrapper_div: 171 | wrapper_div = wrapper_div[0] 172 | css = wrapper_div['style'] 173 | wrapper_div['style'] = ";".join([i for i in css.strip().split(";") 174 | if '30vh' not in i]) 175 | intercom_css = self.dom.find('style', id_="intercom-stylesheet") 176 | if intercom_css: 177 | intercom_css.decompose() 178 | 179 | 180 | def disqus(self): 181 | for div in self.divs: 182 | if div.text.strip() == "[comment]": 183 | div.string = "" 184 | div["id"] = "disqus_thread" 185 | 186 | def div(self): 187 | in_comment = False 188 | in_html = False 189 | in_attr = False 190 | attr = None 191 | # Redo 192 | self.divs = [d for d in self.dom.find_all("div") if d.has_attr("data-block-id")] 193 | for div in self.divs: 194 | try: 195 | div["id"] = div["data-block-id"] 196 | div["class"] = ["content-block"] 197 | except TypeError: 198 | continue 199 | text = div.text.strip() 200 | # Comments 201 | if text == '/*': 202 | in_comment = True 203 | div.decompose() 204 | continue 205 | if text == '*/': 206 | in_comment = False 207 | div.decompose() 208 | continue 209 | if in_comment: 210 | div.decompose() 211 | continue 212 | # HTML 213 | if text == '[html]': 214 | in_html = True 215 | div.decompose() 216 | continue 217 | if text == '[/html]': 218 | in_html = False 219 | div.decompose() 220 | continue 221 | if in_html: 222 | inner_html = BeautifulSoup(BeautifulSoup(str(div).replace('!(notion)!', ''), "html.parser").find('div').text.strip('HTML'), 'html.parser') 223 | div.replace_with(inner_html) 224 | print('Custom HTML inserted: ') 225 | print('----------------------') 226 | print(inner_html) 227 | print('----------------------') 228 | print(div) 229 | continue 230 | # Attr 231 | if text.startswith('[attr'): 232 | in_attr = True 233 | attr = [a.split('=') for a in text[:-1].split(" ")[1:]] 234 | div.decompose() 235 | continue 236 | if text == '[/attr]': 237 | in_attr = False 238 | div.decompose() 239 | continue 240 | if in_attr: 241 | for a in attr: 242 | div[a[0]] = a[1] 243 | continue 244 | # For lightGallery.js 245 | img = div.find('img') 246 | if img and img.has_attr('src') and not div.find('a'): 247 | img['data-src'] = img['src'] 248 | div['class'].append('lg') 249 | 250 | def iframe(self): 251 | for iframe in self.dom.find_all('iframe'): 252 | if iframe.has_attr('style'): 253 | css = iframe['style'].split(';') 254 | new_css = [i for i in css if 'pointer' not in i] 255 | iframe['style'] = ';'.join(new_css) 256 | 257 | def parse_links(self): 258 | for a in self.dom.find_all("a"): 259 | href = a['href'] 260 | if href.startswith('/'): 261 | if href == '/login' or 'file/' in href: 262 | a.decompose() 263 | continue 264 | elif href[1:] == self.options["index"].split("/")[-1]: 265 | a['href'] = '/' 266 | else: 267 | self.links.add(href) 268 | href_list = href.split("/")[-1].split('-') 269 | if len(href_list) > 1: 270 | a['href'] = "/" + '-'.join(href_list[:-1]) 271 | else: 272 | a['href'] = "/" + href_list[0] 273 | if 'anchor' in self.options and href.startswith("https://www.notion.so/") and "#" in href: 274 | url_ = href.split('/')[-1].split("#") 275 | page_url = "/" + url_[0] 276 | anchor = url_[1] 277 | if self.url == page_url: 278 | a["target"] = "" 279 | a["href"] = '#' + anchor 280 | else: 281 | a["href"] = '/' + '-'.join(url_[0].split('-')[:-1]) + '#' + anchor 282 | print("Internal link with anchor detected: " + a['href']) 283 | 284 | def meta(self): 285 | if self.dom.find('html').has_attr("manifest"): 286 | self.dom.find('html')["manifest"] = '' 287 | titles = [i for i in self.dom.find_all( 288 | "div") if (i.has_attr("placeholder") and i["placeholder"] == 'Untitled')] 289 | title = titles[0].text.strip() 290 | titles[0]["id"] = 'title' 291 | if self.is_index: 292 | self.title = title 293 | self.options["site_title"] = title 294 | else: 295 | self.title = title + ' ' + \ 296 | self.options["title_sep"] + ' ' + self.options["site_title"] 297 | self.dom.find("title").string = self.title 298 | self.dom.find("meta", attrs={"name": "twitter:site"})[ 299 | "content"] = self.options["twitter"] 300 | page_path = '-'.join(self.url.split('/')[-1].split('-')[:-1]) 301 | self.dom.find("meta", attrs={"name": "twitter:url"})[ 302 | "content"] = self.options["base_url"] + page_path 303 | self.dom.find("meta", attrs={"property": "og:url"})[ 304 | "content"] = self.options["base_url"] + page_path 305 | self.dom.find("meta", attrs={"property": "og:title"})[ 306 | "content"] = self.title 307 | self.dom.find("meta", attrs={"name": "twitter:title"})[ 308 | "content"] = self.title 309 | self.dom.find("meta", attrs={"property": "og:site_name"})[ 310 | "content"] = self.options["site_title"] 311 | self.dom.find("meta", attrs={"name": "description"})[ 312 | "content"] = self.options["description"] 313 | self.dom.find("meta", attrs={"name": "twitter:description"})[ 314 | "content"] = self.options["description"] 315 | self.dom.find("meta", attrs={"property": "og:description"})[ 316 | "content"] = self.options["description"] 317 | app_store = self.dom.find("meta", attrs={"name": "apple-itunes-app"}) 318 | if app_store: 319 | app_store.decompose() 320 | # Add Canonical URL for SEO 321 | if self.is_mobile: 322 | new_tag = self.dom.new_tag("link", rel='canonical', 323 | href=self.options["base_url"] + page_path) 324 | else: 325 | new_tag = self.dom.new_tag("link", rel='alternate', media='only screen and (max-width: 768px)', 326 | href=self.options["base_url"] + 'm/' + page_path) 327 | self.dom.find('head').append(new_tag) 328 | if 'atom' in self.options: 329 | atom = self.dom.new_tag('link', rel='feed', type="application/atom+xml", href="/feed") 330 | self.dom.find('head').append(atom) 331 | print("Title: " + self.dom.find("title").string) 332 | imgs = [i for i in self.dom.find_all('img') if i.has_attr( 333 | "style") and "30vh" in i["style"]] 334 | if imgs: 335 | img_src = imgs[0]["src"] 336 | if img_src.startswith('/'): 337 | img_url = self.options["base_url"] + img_src[1:] 338 | else: 339 | img_url = img_src 340 | self.dom.find("meta", attrs={"property": "og:image"})[ 341 | "content"] = img_url 342 | self.dom.find("meta", attrs={"name": "twitter:image"})[ 343 | "content"] = img_url 344 | else: 345 | self.dom.find("meta", attrs={"property": "og:image"}).decompose() 346 | self.dom.find("meta", attrs={"name": "twitter:image"}).decompose() 347 | 348 | def remove_scripts(self): 349 | for s in self.dom.find_all("script"): 350 | if s.has_attr("src"): 351 | if any([bool(b in s["src"]) for b in blacklist]): 352 | pass 353 | else: 354 | continue 355 | s.decompose() 356 | for s in self.dom.find_all("noscript"): 357 | s.decompose() 358 | 359 | def save_assets(self): 360 | for css in self.dom.find_all("link"): 361 | if css["href"].startswith("/") and ("stylesheet" in css["rel"]): 362 | download_file("https://notion.so" + 363 | css["href"], css["href"][1:]) 364 | for img in self.dom.find_all("img"): 365 | if img["src"].startswith("/"): 366 | download_file("https://notion.so" + img["src"], img["src"][1:]) 367 | # elif img["src"].startswith("https://notion.imgix.net/"): 368 | # download_file(img["src"], img) 369 | for script in self.dom.find_all("script"): 370 | if script.has_attr("src") and script["src"].startswith("/"): 371 | download_file("https://notion.so" + 372 | script["src"], script["src"][1:]) 373 | 374 | def remove_overlay(self): 375 | overlay = self.dom.find(class_="notion-overlay-container") 376 | if overlay: 377 | overlay.decompose() 378 | 379 | def walk(self): 380 | global visited, notions 381 | for link in self.links: 382 | if link not in visited: 383 | page = Notion(link, self.driver, options=options, is_mobile=self.is_mobile) 384 | notions[link] = page 385 | page.mod() 386 | visited.add(link) 387 | page.walk() 388 | 389 | 390 | if __name__ == "__main__": 391 | motion() 392 | if "build_mobile" in options: 393 | motion(is_mobile=True) 394 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | requests>=2.20.0 2 | selenium==2.48.0 3 | beautifulsoup4==4.6.0 4 | --------------------------------------------------------------------------------