├── 20171006_python_03.ipynb ├── CNAME ├── README.md ├── _config.yml ├── bb101 ├── bb101-20170420.ipynb ├── bb101-20170425.ipynb ├── bb101-20170427.ipynb ├── bb101-20170502.ipynb └── underground_water.ipynb ├── bb102 ├── cai_speech.txt ├── donald_trump.txt ├── python-etl-20170707.ipynb ├── python_etl_20170627.ipynb ├── python_etl_20170629.ipynb ├── python_etl_20170705.ipynb ├── selenium.ipynb └── yes123 │ ├── detail_html │ └── .gitkeep │ ├── job_detail.py │ ├── job_list.py │ └── list_html │ └── .gitkeep ├── bb103 ├── bb103_pythonetl_20170815.ipynb ├── bb103_pythonetl_20170817.ipynb ├── bb103_pythonetl_20170822.ipynb ├── bb103_pythonetl_20170824.ipynb ├── bb103_pythonetl_20170829.ipynb ├── pm2.png └── trump_interview.txt ├── bb104 ├── amazon.ipynb ├── appledaily.py ├── bb104_20171026.ipynb ├── bb104_20171031.ipynb ├── bb104_20171102.ipynb ├── bb104_20171107.ipynb └── bb104_20171109.ipynb ├── bb105 ├── .ipynb_checkpoints │ └── bb105_20171225-checkpoint.ipynb ├── bb105_20171225.ipynb ├── bb105_20171227.ipynb ├── bb105_20180102.ipynb ├── bb105_20180108.ipynb ├── bb105_20180131.ipynb ├── ptt_crawler.py └── yes123 │ ├── README.md │ ├── requirements.txt │ ├── tasks.py │ └── yes123.py ├── bb106 ├── bb106_20180320.ipynb ├── bb106_20180322.ipynb ├── bb106_20180327.ipynb ├── bb106_20180330.ipynb ├── bb106_20180402.ipynb └── google_trends.py ├── cb101 ├── CB101_20180529.ipynb ├── CB101_20180530.ipynb ├── CB101_20180605.ipynb ├── CB101_20180607.ipynb ├── flippingmed_crawler.ipynb ├── pandas_demo.ipynb ├── right-pet.cc 爬蟲.ipynb ├── 吱吱復健.ipynb └── 時間日期轉換.ipynb ├── cb102 ├── cb102_20180720.ipynb └── cb102_20180726.ipynb ├── datasets ├── A_LVR_LAND_A_BUILD.CSV ├── eng_stop_words.txt ├── gym.csv └── trump_speech.txt ├── draft ├── Untitled.ipynb ├── Untitled2.ipynb └── coolpc-ssd.ipynb ├── install_gcin.sh ├── lib ├── __init__.py └── toolbox.py ├── ptt ├── crawler.py ├── data │ └── .gitkeep └── requirements.txt ├── python_crawler_20171013.ipynb └── utils ├── chromedriver ├── chromedriver_linux64.zip └── install_chromedriver.sh /CNAME: -------------------------------------------------------------------------------- 1 | pythonetl.ianchenhq.com -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Python ETL Jupyter Notebooks 2 | 3 | ## 如何安裝環境 4 | 5 | 以下是如何在Linux/Mac底下安裝Python環境 6 | 7 | 參照 https://github.com/pyenv/pyenv/wiki 8 | 9 | ### Linux 10 | 11 | ``` 12 | # 更新系統套件版本列表 13 | sudo apt-get update 14 | 15 | # 安裝C語言build tools 16 | apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev xz-utils tk-dev 17 | ``` 18 | 19 | ### MacOS 20 | 21 | ``` 22 | # 安裝homebrew 23 | /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" 24 | 25 | # 安裝一些系統套件 26 | brew install openssl readline xz 27 | ``` 28 | 29 | ### Linux/MacOS 30 | 31 | ``` 32 | # 安裝pyenv 33 | curl -L https://raw.githubusercontent.com/pyenv/pyenv-installer/master/bin/pyenv-installer | bash 34 | ``` -------------------------------------------------------------------------------- /_config.yml: -------------------------------------------------------------------------------- 1 | theme: jekyll-theme-minimal -------------------------------------------------------------------------------- /bb102/cai_speech.txt: -------------------------------------------------------------------------------- 1 | 各位友邦的元首與貴賓、各國駐台使節及代表、現場的好朋友，全體國人同胞，大家好 2 | 3 | 感謝與承擔 4 | 5 | 就在剛剛，我和陳建仁已經在總統府裡面，正式宣誓就任中華民國第十四任總統與副總統。我們要感謝這塊土地對我們的栽培，感謝人民對我們的信任，以及，最重要的，感謝這個國家的民主機制，讓我們透過和平的選舉過程，實現第三次政黨輪替，並且克服種種不確定因素，順利渡過長達四個月的交接期，完成政權和平移轉。 6 | 7 | 台灣，再一次用行動告訴世界，作為一群民主人與自由人，我們有堅定的信念，去捍衛民主自由的生活方式。這段旅程，我們每一個人都參與其中。親愛的台灣人民，我們做到了。 8 | 9 | 我要告訴大家，對於一月十六日的選舉結果，我從來沒有其他的解讀方式。人民選擇了新總統、新政府，所期待的就是四個字：解決問題。此時此刻，台灣的處境很困難，迫切需要執政者義無反顧的承擔。這一點，我不會忘記。 10 | 11 | 我也要告訴大家，眼前的種種難關，需要我們誠實面對，需要我們共同承擔。所以，這個演說是一個邀請，我要邀請全體國人同胞一起來，扛起這個國家的未來。 12 | 13 | 國家不會因為領導人而偉大；全體國民的共同奮鬥，才讓這個國家偉大。總統該團結的不只是支持者，總統該團結的是整個國家。團結是為了改變，這是我對這個國家最深切的期待。在這裡，我要誠懇地呼籲，請給這個國家一個機會，讓我們拋下成見，拋下過去的對立，我們一起來完成新時代交給我們的使命。 14 | 15 | 在我們共同奮鬥的過程中，身為總統，我要向全國人民宣示，未來我和新政府，將領導這個國家的改革，展現決心，絕不退縮。 16 | 17 | 為年輕人打造一個更好的國家 18 | 19 | 未來的路並不好走，台灣需要一個正面迎向一切挑戰的新政府，我的責任就是領導這個新政府。 20 | 21 | 我們的年金制度，如果不改，就會破產。 22 | 我們僵化的教育制度，已經逐漸與社會脈動脫節。 23 | 我們的能源與資源十分有限，我們的經濟缺乏動能，舊的代工模式已經面臨瓶頸，整個國家極需要新的經濟發展模式。 24 | 我們的人口結構急速老化，長照體系卻尚未健全。 25 | 我們的人口出生率持續低落，完善的托育制度卻始終遙遙無期。 26 | 我們環境汙染問題仍然嚴重。 27 | 我們國家的財政並不樂觀。 28 | 我們的司法已經失去人民的信任。 29 | 我們的食品安全問題，困擾著所有家庭。 30 | 我們的貧富差距越來越嚴重。 31 | 我們的社會安全網還是有很多破洞。 32 | 最重要的，我要特別強調，我們的年輕人處於低薪的處境，他們的人生，動彈不得，對於未來，充滿無奈與茫然。 33 | 34 | 年輕人的未來是政府的責任。如果不友善的結構沒有改變，再多個人菁英的出現，都不足以讓整體年輕人的處境變好。我期許自己，在未來的任期之內，要一步一步，從根本的結構來解決這個國家的問題。 35 | 36 | 這就是我想為台灣的年輕人做的事。雖然我沒有辦法立刻幫所有的年輕人加薪，但是我願意承諾，新政府會立刻展開行動。請給我們一點時間，也請跟我們一起走上改革的這一條路。 37 | 38 | 改變年輕人的處境，就是改變國家的處境。一個國家的年輕人沒有未來，這個國家必定沒有未來。幫助年輕人突破困境，實現世代正義，把一個更好的國家交到下一代手上，就是新政府重大的責任。 39 | 40 | 第一、經濟結構的轉型 41 | 42 | 要打造一個更好的國家，未來，新政府要做到以下幾件事情。 43 | 44 | 首先，就是讓台灣的經濟結構轉型。這是新政府所必須承擔的最艱鉅使命。我們不要妄自菲薄，更不要失去信心。台灣有很多別的國家沒有的優勢，我們有海洋經濟的活力和靭性，高素質的人力資源、務實可靠的工程師文化、完整的產業鏈、敏捷靈活的中小企業，以及，永不屈服的創業精神。 45 | 46 | 我們要讓台灣經濟脫胎換骨，就必須從現在起就下定決心，勇敢地走出另外一條路。這一條路，就是打造台灣經濟發展的新模式。 47 | 48 | 新政府將打造一個以創新、就業、分配為核心價值，追求永續發展的新經濟模式。改革的第一步，就是強化經濟的活力與自主性，加強和全球及區域的連結，積極參與多邊及雙邊經濟合作及自由貿易談判，包括TPP、RCEP等，並且，推動新南向政策，提升對外經濟的格局及多元性，告別以往過於依賴單一市場的現象。 49 | 50 | 除此之外，新政府相信，唯有激發新的成長動能，我們才能突破當前經濟的停滯不前。我們會以出口和內需作為雙引擎，讓企業生產和人民生活互為表裡，讓對外貿易和在地經濟緊密連結。 51 | 52 | 我們會優先推動五大創新研發計畫，藉著這些產業來重新塑造台灣的全球競爭力。我們也要積極提升勞動生產力，保障勞工權益，讓薪資和經濟成長能同步提升。 53 | 54 | 這是台灣經濟發展的關鍵時刻。我們有決心，也有溝通能力。我們已經有系統性的規劃，未來，會以跨部會聯手的模式，把整個國家的力量集結起來，一起來催生這個新模式。 55 | 56 | 在經濟發展的同時，我們不要忘記對環境的責任。經濟發展的新模式會和國土規劃、區域發展及環境永續，相互結合。產業的佈局和國土的利用，應該拋棄零碎的規畫，和短視近利的眼光。我們必須追求區域的均衡發展，這需要中央來規畫、整合，也需要地方政府充分發揮區域聯合治理的精神。 57 | 58 | 我們也不能再像過去，無止盡地揮霍自然資源及國民健康。所以，對各種汙染的控制，我們會嚴格把關，更要讓台灣走向循環經濟的時代，把廢棄物轉換為再生資源。對於能源的選擇，我們會以永續的觀念去逐步調整。新政府會嚴肅看待氣候變遷、國土保育、災害防治的相關議題，因為，我們只有一個地球，我們也只有一個台灣。 59 | 60 | 第二、強化社會安全網 61 | 62 | 新政府必須要承擔的第二件事情，就是強化台灣的社會安全網。這些年，幾件關於兒少安全及隨機殺人的事件，都讓整個社會震驚。不過，一個政府不能永遠在震驚，它必須要有同理心。沒有人可以替受害者家屬承受傷痛，但是，一個政府，尤其是第一線處理問題的人，必須要讓受害者以及家屬覺得，不幸事件發生的時候，政府是站在他們這一邊。 63 | 64 | 除了同理心之外，政府更應該要提出解決的方法。全力防止悲劇一再發生，從治安、教育、心理健康、社會工作等各個面向，積極把破洞補起來。尤其是治安與反毒的工作，這些事情，新政府會用最嚴肅的態度和行動來面對。 65 | 66 | 在年金的改革方面，這是攸關台灣生存發展的關鍵改革，我們不應該遲疑，也不可以躁進。由陳建仁副總統擔任召集人的年金改革委員會，已經緊鑼密鼓在籌備之中。過去的政府在這個議題上，曾經有過一些努力。但是，缺乏社會的參與。新政府的做法，是發動一個集體協商，因為年金改革必須是一個透過協商來團結所有人的過程。 67 | 68 | 這就是為什麼，我們要召開年金改革國是會議，由不同階層、不同職業代表，在社會團結的基礎上，共同協商。一年之內，我們會提出可行的改革方案。無論是勞工還是公務員，每一個國民的退休生活都應該得到公平的保障。 69 | 70 | 另外，在長期照顧的議題上，我們將會把優質、平價、普及的長期照顧系統建立起來。和年金改革一樣，長照體系也是一個社會總動員的過程。新政府的做法是由政府主導和規劃，鼓勵民間發揮社區主義的精神，透過社會集體互助的力量，來建立一套妥善而完整的體系。每一個老年人都可以在自己熟悉的社區，安心享受老年生活，每一個家庭的照顧壓力將會減輕。照顧老人的工作不能完全讓它變成自由市場。我們會把責任扛起來，按部就班來規劃與執行，為超高齡社會的來臨，做好準備。 71 | 72 | 第三、社會的公平與正義 73 | 74 | 新政府要承擔的第三件事情，就是社會的公平與正義。在這個議題上，新政府會持續和公民社會一起合作，讓台灣的政策更符合多元、平等、開放、透明、人權的價值，讓台灣的民主機制更加深化與進化。 75 | 76 | 新的民主制度要能夠上路，我們必須先找出面對過去的共同方法。未來，我會在總統府成立真相與和解委員會，用最誠懇與謹慎的態度，來處理過去的歷史。追求轉型正義的目標是在追求社會的真正和解，讓所有台灣人都記取那個時代的錯誤。 77 | 78 | 我們將從真相的調查與整理出發，預計在三年之內，完成台灣自己的轉型正義調查報告書。我們將會依據調查報告所揭示的真相，來進行後續的轉型正義工作。挖掘真相、彌平傷痕、釐清責任。從此以後，過去的歷史不再是台灣分裂的原因，而是台灣一起往前走的動力。 79 | 80 | 同樣在公平正義的議題上，我會秉持相同的原則，來面對原住民族的議題。今天的就職典禮，原住民族的小朋友在唱國歌之前，先唱了他們部落傳統的古調。這象徵了，我們不敢忘記，這個島上先來後到的順序。 81 | 82 | 新政府會用道歉的態度，來面對原住民族相關議題，重建原民史觀，逐步推動自治，復育語言文化，提升生活照顧，這就是我要領導新政府推動的改變。 83 | 84 | 接下來，新政府也會積極推動司法改革。這是現階段台灣人民最關心的議題。司法無法親近人民、不被人民信任、司法無法有效打擊犯罪，以及，司法失去作為正義最後一道防線的功能，是人民普遍的感受。 85 | 86 | 為了展現新政府的決心，我們會在今年十月召開司法國是會議，透過人民實際的參與，讓社會力進來，一起推動司法改革。司法必須回應人民的需求，不再只是法律人的司法，而是全民的司法。司法改革也不只是司法人的家務事，而是全民參與的改革。這就是我對司法改革的期待。 87 | 88 | 第四、區域的和平穩定發展及兩岸關係 89 | 90 | 新政府要承擔的第四件事情，是區域的和平穩定與發展，以及妥善處理兩岸關係。過去三十年，無論是對亞洲或是全球，都是變動最劇烈的時期；而全球及區域的經濟穩定和集體安全，也是各國政府越來越關切的課題。 91 | 92 | 台灣在區域發展當中，一直是不可或缺的關鍵角色。但是近年來，區域的情勢快速變動，如果台灣不善用自己的實力和籌碼，積極參與區域事務，不但將會變得無足輕重，甚至可能被邊緣化，喪失對於未來的自主權。 93 | 94 | 我們有危機，但也有轉機。台灣現階段的經濟發展，和區域中許多國家高度關聯和互補。如果將打造經濟發展新模式的努力，透過和亞洲、乃至亞太區域的國家合作，共同形塑未來的發展策略，不但可以為區域的經濟創新、結構調整和永續發展，做出積極的貢獻，更可以和區域內的成員，建立緊密的「經濟共同體」意識。 95 | 96 | 我們要和其他國家共享資源、人才與市場，擴大經濟規模，讓資源有效利用。「新南向政策」就是基於這樣的精神。我們會在科技、文化與經貿等各層面，和區域成員廣泛交流合作，尤其是增進與東協、印度的多元關係。為此，我們也願意和對岸，就共同參與區域發展的相關議題，坦誠交換意見，尋求各種合作與協力的可能性。 97 | 98 | 在積極發展經濟的同時，亞太地區的安全情勢也變得越來越複雜，而兩岸關係，也成為建構區域和平與集體安全的重要一環。這個建構的進程，台灣會做一個「和平的堅定維護者」，積極參與，絕不缺席；我們也將致力維持兩岸關係的和平穩定；我們更會努力促成內部和解，強化民主機制，凝聚共識，形成一致對外的立場。 99 | 100 | 對話和溝通，是我們達成目標最重要的關鍵。台灣也要成為一個「和平的積極溝通者」，我們將和相關的各方，建立常態、緊密的溝通機制，隨時交換意見，防止誤判，建立互信，有效解決爭議。我們將謹守和平原則、利益共享原則，來處理相關的爭議。 101 | 102 | 我依照中華民國憲法當選總統，我有責任捍衛中華民國的主權和領土；對於東海及南海問題，我們主張應擱置爭議，共同開發。 103 | 104 | 兩岸之間的對話與溝通，我們也將努力維持現有的機制。1992年兩岸兩會秉持相互諒解、求同存異的政治思維，進行溝通協商，達成若干的共同認知與諒解，我尊重這個歷史事實。92年之後，20多年來雙方交流、協商所累積形成的現狀與成果，兩岸都應該共同珍惜與維護，並在這個既有的事實與政治基礎上，持續推動兩岸關係和平穩定發展；新政府會依據中華民國憲法、兩岸人民關係條例及其他相關法律，處理兩岸事務。兩岸的兩個執政黨應該要放下歷史包袱，展開良性對話，造福兩岸人民。 105 | 106 | 我所講的既有政治基礎，包含幾個關鍵元素，第一，1992年兩岸兩會會談的歷史事實與求同存異的共同認知，這是歷史事實；第二，中華民國現行憲政體制；第三，兩岸過去20多年來協商和交流互動的成果；第四，台灣民主原則及普遍民意。 107 | 108 | 第五、外交與全球性議題 109 | 110 | 新政府要承擔的第五件事情，是善盡地球公民的責任，在外交與全球性的議題上做出貢獻。讓台灣走向世界，也要讓世界走進台灣。 111 | 112 | 現場有許多來自各國的元首與使節團，我要特別謝謝他們，長久以來一直幫助台灣，讓我們有機會參與國際社會。未來，我們會持續透過官方互動、企業投資與民間合作各種方式，分享台灣發展的經驗，與友邦建立永續的夥伴關係。 113 | 114 | 台灣是全球公民社會的模範生，民主化以來，我們始終堅持和平、自由、民主及人權的普世價值。我們會秉持這個精神，加入全球議題的價值同盟。我們會繼續深化與包括美國、日本、歐洲在內的友好民主國家的關係，在共同的價值基礎上，推動全方位的合作。 115 | 116 | 我們會積極參與國際經貿合作及規則制定，堅定維護全球的經濟秩序，並且融入重要的區域經貿體系。我們也不會在防制全球暖化、氣候變遷的議題上缺席。我們將會在行政院設立專責的能源和減碳辦公室，並且根據COP21巴黎協議的規定，定期檢討溫室氣體的減量目標，與友好國家攜手，共同維護永續的地球。 117 | 118 | 同時，新政府會支持並參與，全球性新興議題的國際合作，包括人道救援、醫療援助、疾病的防治與研究、反恐合作，以及共同打擊跨國犯罪，讓台灣成為國際社會不可或缺的夥伴。 119 | 120 | 結語 121 | 122 | 1996年台灣第一次總統直選，到今天剛好20年。過去20年，在幾任政府以及公民社會的努力之下，我們成功渡過了許多新興民主國家必須面對的難關。在這個過程中，我們曾經有過許多感動人心的時刻和故事，不過，正如同世界上其他國家一樣，我們也曾經有過焦慮、不安、矛盾、與對立。 123 | 124 | 我們看到了社會的對立，進步與保守的對立，環境與開發的對立，以及，政治意識之間的對立。這些對立，曾經激發出選舉時的動員能量，不過也因為這些對立，我們的民主逐漸失去了解決問題的能力。 125 | 126 | 民主是一個進程，每一個時代的政治工作者，都要清楚認識他身上所肩負的責任。民主會前進，民主也有可能倒退。今天，我站在這裡，就是要告訴大家，倒退不會是我們的選項。新政府的責任就是把台灣的民主推向下一個階段：以前的民主是選舉的輸贏，現在的民主則是關於人民的幸福；以前的民主是兩個價值觀的對決，現在的民主則是不同價值觀的對話。 127 | 128 | 打造一個沒有被意識形態綁架的「團結的民主」，打造一個可以回應社會與經濟問題的「有效率的民主」，打造一個能夠實質照料人民的「務實的民主」，這就是新時代的意義。 129 | 130 | 只要我們相信，新時代就會來臨。只要這個國家的主人，有堅定的信念，新時代一定會在我們這一代人的手上誕生。 131 | 132 | 各位親愛的台灣人民，演講要結束了，改革要開始了。從這一刻起，這個國家的擔子交在新政府身上。我會讓大家看見這個國家的改變。 133 | 134 | 歷史會記得我們這個勇敢的世代，這個國家的繁榮、尊嚴、團結、自信和公義，都有我們努力的痕跡。歷史會記得我們的勇敢，我們在2016年一起把國家帶向新的方向。這塊土地上的每一個人，都因為參與台灣的改變，而感到驕傲。 135 | 136 | 剛才表演節目中的一首歌曲當中，有一句讓我很感動的歌詞： 137 | （台語）現在是彼一天，勇敢ㄟ台灣人。 138 | 139 | 各位國人同胞，兩千三百萬的台灣人民，等待已經結束，現在就是那一天。今天，明天，未來的每一天，我們都要做一個守護民主、守護自由、守護這個國家的台灣人。 140 | 141 | 謝謝大家。 142 | -------------------------------------------------------------------------------- /bb102/donald_trump.txt: -------------------------------------------------------------------------------- 1 | Chief Justice Roberts, President Carter, President Clinton, President Bush, President Obama, fellow Americans, and people of the world: thank you. 2 | 3 | We, the citizens of America, are now joined in a great national effort to rebuild our country and to restore its promise for all of our people. 4 | 5 | Together, we will determine the course of America and the world for years to come. 6 | 7 | We will face challenges. We will confront hardships. But we will get the job done. 8 | 9 | Every four years, we gather on these steps to carry out the orderly and peaceful transfer of power, and we are grateful to President Obama and First Lady Michelle Obama for their gracious aid throughout this transition. They have been magnificent. 10 | 11 | Today’s ceremony, however, has very special meaning. Because today we are not merely transferring power from one Administration to another, or from one party to another – but we are transferring power from Washington, D.C. and giving it back to you, the American People. 12 | 13 | For too long, a small group in our nation’s Capital has reaped the rewards of government while the people have borne the cost. 14 | 15 | Washington flourished – but the people did not share in its wealth. 16 | 17 | Politicians prospered – but the jobs left, and the factories closed. 18 | 19 | The establishment protected itself, but not the citizens of our country. 20 | 21 | Their victories have not been your victories; their triumphs have not been your triumphs; and while they celebrated in our nation’s Capital, there was little to celebrate for struggling families all across our land. 22 | 23 | That all changes – starting right here, and right now, because this moment is your moment: it belongs to you. 24 | 25 | It belongs to everyone gathered here today and everyone watching all across America. 26 | 27 | This is your day. This is your celebration. 28 | 29 | And this, the United States of America, is your country. 30 | 31 | What truly matters is not which party controls our government, but whether our government is controlled by the people. 32 | 33 | January 20th 2017, will be remembered as the day the people became the rulers of this nation again. 34 | 35 | The forgotten men and women of our country will be forgotten no longer. 36 | 37 | Everyone is listening to you now. 38 | 39 | You came by the tens of millions to become part of a historic movement the likes of which the world has never seen before. 40 | 41 | At the center of this movement is a crucial conviction: that a nation exists to serve its citizens. 42 | 43 | Americans want great schools for their children, safe neighborhoods for their families, and good jobs for themselves. 44 | 45 | These are the just and reasonable demands of a righteous public. 46 | 47 | But for too many of our citizens, a different reality exists: Mothers and children trapped in poverty in our inner cities; rusted-out factories scattered like tombstones across the landscape of our nation; an education system, flush with cash, but which leaves our young and beautiful students deprived of knowledge; and the crime and gangs and drugs that have stolen too many lives and robbed our country of so much unrealized potential. 48 | 49 | This American carnage stops right here and stops right now. 50 | 51 | We are one nation – and their pain is our pain. Their dreams are our dreams; and their success will be our success. We share one heart, one home, and one glorious destiny. 52 | 53 | The oath of office I take today is an oath of allegiance to all Americans. 54 | 55 | For many decades, we’ve enriched foreign industry at the expense of American industry; 56 | 57 | Subsidized the armies of other countries while allowing for the very sad depletion of our military; 58 | 59 | We've defended other nation’s borders while refusing to defend our own; 60 | 61 | And spent trillions of dollars overseas while America's infrastructure has fallen into disrepair and decay. 62 | 63 | We’ve made other countries rich while the wealth, strength, and confidence of our country has disappeared over the horizon. 64 | 65 | One by one, the factories shuttered and left our shores, with not even a thought about the millions upon millions of American workers left behind. 66 | 67 | The wealth of our middle class has been ripped from their homes and then redistributed across the entire world. 68 | 69 | But that is the past. And now we are looking only to the future. 70 | 71 | We assembled here today are issuing a new decree to be heard in every city, in every foreign capital, and in every hall of power. 72 | 73 | From this day forward, a new vision will govern our land. 74 | 75 | From this moment on, it’s going to be America First. 76 | 77 | Every decision on trade, on taxes, on immigration, on foreign affairs, will be made to benefit American workers and American families. 78 | 79 | We must protect our borders from the ravages of other countries making our products, stealing our companies, and destroying our jobs. Protection will lead to great prosperity and strength. 80 | 81 | I will fight for you with every breath in my body – and I will never, ever let you down. 82 | 83 | America will start winning again, winning like never before. 84 | 85 | We will bring back our jobs. We will bring back our borders. We will bring back our wealth. And we will bring back our dreams. 86 | 87 | We will build new roads, and highways, and bridges, and airports, and tunnels, and railways all across our wonderful nation. 88 | 89 | We will get our people off of welfare and back to work – rebuilding our country with American hands and American labor. 90 | 91 | We will follow two simple rules: Buy American and Hire American. 92 | 93 | We will seek friendship and goodwill with the nations of the world – but we do so with the understanding that it is the right of all nations to put their own interests first. 94 | 95 | We do not seek to impose our way of life on anyone, but rather to let it shine as an example for everyone to follow. 96 | 97 | We will reinforce old alliances and form new ones – and unite the civilized world against Radical Islamic Terrorism, which we will eradicate completely from the face of the Earth. 98 | 99 | At the bedrock of our politics will be a total allegiance to the United States of America, and through our loyalty to our country, we will rediscover our loyalty to each other. 100 | 101 | When you open your heart to patriotism, there is no room for prejudice. 102 | 103 | The Bible tells us, “how good and pleasant it is when God’s people live together in unity.” 104 | 105 | We must speak our minds openly, debate our disagreements honestly, but always pursue solidarity. 106 | 107 | When America is united, America is totally unstoppable. 108 | 109 | There should be no fear – we are protected, and we will always be protected. 110 | 111 | We will be protected by the great men and women of our military and law enforcement and, most importantly, we are protected by God. 112 | 113 | Finally, we must think big and dream even bigger. 114 | 115 | In America, we understand that a nation is only living as long as it is striving. 116 | 117 | We will no longer accept politicians who are all talk and no action – constantly complaining but never doing anything about it. 118 | 119 | The time for empty talk is over. 120 | 121 | Now arrives the hour of action. 122 | 123 | Do not let anyone tell you it cannot be done. No challenge can match the heart and fight and spirit of America. 124 | 125 | We will not fail. Our country will thrive and prosper again. 126 | 127 | We stand at the birth of a new millennium, ready to unlock the mysteries of space, to free the Earth from the miseries of disease, and to harness the energies, industries and technologies of tomorrow. 128 | 129 | A new national pride will stir our souls, lift our sights, and heal our divisions. 130 | 131 | It is time to remember that old wisdom our soldiers will never forget: that whether we are black or brown or white, we all bleed the same red blood of patriots, we all enjoy the same glorious freedoms, and we all salute the same great American Flag. 132 | 133 | And whether a child is born in the urban sprawl of Detroit or the windswept plains of Nebraska, they look up at the same night sky, they fill their heart with the same dreams, and they are infused with the breath of life by the same almighty Creator. 134 | 135 | So to all Americans, in every city near and far, small and large, from mountain to mountain, and from ocean to ocean, hear these words: 136 | 137 | You will never be ignored again. 138 | 139 | Your voice, your hopes, and your dreams, will define our American destiny. And your courage and goodness and love will forever guide us along the way. 140 | 141 | Together, We Will Make America Strong Again. 142 | 143 | We Will Make America Wealthy Again. 144 | 145 | We Will Make America Proud Again. 146 | 147 | We Will Make America Safe Again. 148 | 149 | And, Yes, Together, We Will Make America Great Again. Thank you, God Bless You, And God Bless America. 150 | -------------------------------------------------------------------------------- /bb102/yes123/detail_html/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ianchen06/pythonetl/6b4fb75bab09cb40bc774bd0c57cddb6c837c13c/bb102/yes123/detail_html/.gitkeep -------------------------------------------------------------------------------- /bb102/yes123/job_detail.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # coding: utf-8 3 | import requests as r 4 | 5 | HOST = 'https://www.yes123.com.tw/admin/' 6 | 7 | def get_detail(): 8 | with open('./urls_uniq.txt') as f: 9 | for line in f: 10 | URL = HOST + line.strip() 11 | print("[INFO] crawling %s"%URL) 12 | res = r.get(URL, headers={'User-Agent': ''}) 13 | job_id = line.split("=")[-1].strip() 14 | print(res.status_code) 15 | with open('./detail_html/java_job_%s.html'%job_id, 'w') as f: 16 | f.write(res.text) 17 | -------------------------------------------------------------------------------- /bb102/yes123/job_list.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # coding: utf-8 3 | import os 4 | 5 | import requests as r 6 | 7 | import job_detail 8 | 9 | URL = "https://www.yes123.com.tw/admin/job_refer_list.asp" 10 | 11 | data = {'_mu_chkbox_2': '', 12 | '_mu_chkbox_3': '', 13 | '_mu_edu_1': '', 14 | '_mu_edu_2': '', 15 | '_mu_edu_3': '', 16 | '_mu_edu_4': '', 17 | '_mu_edu_5': '', 18 | '_mu_edu_6': '', 19 | '_mu_edu_7': '', 20 | '_mu_job_1': '', 21 | '_mu_lang_1': '', 22 | '_mu_lang_2': '', 23 | '_mu_lang_3': '', 24 | '_mu_psn_1': '', 25 | '_mu_sc_1': '', 26 | '_mu_sc_2': '', 27 | '_mu_se_1': '', 28 | '_mu_sf_1': '', 29 | '_mu_vc_1': '', 30 | '_mu_vc_2': '', 31 | '_mu_vc_3': '', 32 | '_mu_vc_4': '', 33 | '_mu_wk_1': '', 34 | '_mu_wk_2': '', 35 | '_mu_wk_3': '', 36 | '_mu_wk_4': '', 37 | '_mu_wk_5': '', 38 | '_mu_year_1': '', 39 | '_mu_year_2': '', 40 | '_rdo_1': '1', 41 | '_rdo_2': '1', 42 | 'find_cert_mode1': '', 43 | 'find_cert_mode2': '', 44 | 'find_cert_mode3': '', 45 | 'find_cert_mode4': '', 46 | 'find_cert_mode5': '', 47 | 'find_indy_mode1': '', 48 | 'find_indy_mode10': '', 49 | 'find_indy_mode2': '', 50 | 'find_indy_mode3': '', 51 | 'find_indy_mode4': '', 52 | 'find_indy_mode5': '', 53 | 'find_indy_mode6': '', 54 | 'find_indy_mode7': '', 55 | 'find_indy_mode8': '', 56 | 'find_indy_mode9': '', 57 | 'find_job_mode1': '', 58 | 'find_job_mode10': '', 59 | 'find_job_mode2': '', 60 | 'find_job_mode3': '', 61 | 'find_job_mode4': '', 62 | 'find_job_mode5': '', 63 | 'find_job_mode6': '', 64 | 'find_job_mode7': '', 65 | 'find_job_mode8': '', 66 | 'find_job_mode9': '', 67 | 'find_key1': 'java', 68 | 'find_key2': '', 69 | 'find_key3': '', 70 | 'find_map_mode1': '', 71 | 'find_map_mode2': '', 72 | 'find_map_mode3': '', 73 | 'find_map_mode4': '', 74 | 'find_metro_mode1': '', 75 | 'find_metro_mode10': '', 76 | 'find_metro_mode2': '', 77 | 'find_metro_mode3': '', 78 | 'find_metro_mode4': '', 79 | 'find_metro_mode5': '', 80 | 'find_metro_mode6': '', 81 | 'find_metro_mode7': '', 82 | 'find_metro_mode8': '', 83 | 'find_metro_mode9': '', 84 | 'find_sche_mode1': '', 85 | 'find_sche_mode2': '', 86 | 'find_sche_mode3': '', 87 | 'find_sche_mode4': '', 88 | 'find_sche_mode5': '', 89 | 'find_scl_mode1': '', 90 | 'find_scl_mode10': '', 91 | 'find_scl_mode2': '', 92 | 'find_scl_mode3': '', 93 | 'find_scl_mode4': '', 94 | 'find_scl_mode5': '', 95 | 'find_scl_mode6': '', 96 | 'find_scl_mode7': '', 97 | 'find_scl_mode8': '', 98 | 'find_scl_mode9': '', 99 | 'find_se_work_mode1': '', 100 | 'find_sf_subj_mode1': '', 101 | 'find_subj_mode1': '', 102 | 'find_subj_mode2': '', 103 | 'find_subj_mode3': '', 104 | 'find_sw_mode1': '', 105 | 'find_sw_mode2': '', 106 | 'find_sw_mode3': '', 107 | 'find_sw_mode4': '', 108 | 'find_sw_mode5': '', 109 | 'find_work_mode1': '', 110 | 'find_work_mode10': '', 111 | 'find_work_mode2': '', 112 | 'find_work_mode3': '', 113 | 'find_work_mode4': '', 114 | 'find_work_mode5': '', 115 | 'find_work_mode6': '', 116 | 'find_work_mode7': '', 117 | 'find_work_mode8': '', 118 | 'find_work_mode9': '', 119 | 'find_zone_mode1': '', 120 | 'find_zone_mode10': '', 121 | 'find_zone_mode2': '', 122 | 'find_zone_mode3': '', 123 | 'find_zone_mode4': '', 124 | 'find_zone_mode5': '', 125 | 'find_zone_mode6': '', 126 | 'find_zone_mode7': '', 127 | 'find_zone_mode8': '', 128 | 'find_zone_mode9': '', 129 | 'job_show_type': 'L', 130 | 'order_ascend': 'desc', 131 | 'order_by': 'neworder', 132 | 's_find_cert_mode1': '', 133 | 's_find_cert_mode2': '', 134 | 's_find_cert_mode3': '', 135 | 's_find_cert_mode4': '', 136 | 's_find_cert_mode5': '', 137 | 's_find_indy_mode1': '', 138 | 's_find_indy_mode10': '', 139 | 's_find_indy_mode2': '', 140 | 's_find_indy_mode3': '', 141 | 's_find_indy_mode4': '', 142 | 's_find_indy_mode5': '', 143 | 's_find_indy_mode6': '', 144 | 's_find_indy_mode7': '', 145 | 's_find_indy_mode8': '', 146 | 's_find_indy_mode9': '', 147 | 's_find_job_mode1': '', 148 | 's_find_job_mode10': '', 149 | 's_find_job_mode2': '', 150 | 's_find_job_mode3': '', 151 | 's_find_job_mode4': '', 152 | 's_find_job_mode5': '', 153 | 's_find_job_mode6': '', 154 | 's_find_job_mode7': '', 155 | 's_find_job_mode8': '', 156 | 's_find_job_mode9': '', 157 | 's_find_metro_mode1': '', 158 | 's_find_metro_mode10': '', 159 | 's_find_metro_mode2': '', 160 | 's_find_metro_mode3': '', 161 | 's_find_metro_mode4': '', 162 | 's_find_metro_mode5': '', 163 | 's_find_metro_mode6': '', 164 | 's_find_metro_mode7': '', 165 | 's_find_metro_mode8': '', 166 | 's_find_metro_mode9': '', 167 | 's_find_sche_mode1': '', 168 | 's_find_sche_mode2': '', 169 | 's_find_sche_mode3': '', 170 | 's_find_sche_mode4': '', 171 | 's_find_sche_mode5': '', 172 | 's_find_scl_mode1': '', 173 | 's_find_scl_mode10': '', 174 | 's_find_scl_mode2': '', 175 | 's_find_scl_mode3': '', 176 | 's_find_scl_mode4': '', 177 | 's_find_scl_mode5': '', 178 | 's_find_scl_mode6': '', 179 | 's_find_scl_mode7': '', 180 | 's_find_scl_mode8': '', 181 | 's_find_scl_mode9': '', 182 | 's_find_se_work_mode1': '', 183 | 's_find_sf_subj_mode1': '', 184 | 's_find_subj_mode1': '', 185 | 's_find_subj_mode2': '', 186 | 's_find_subj_mode3': '', 187 | 's_find_sw_mode1': '', 188 | 's_find_sw_mode2': '', 189 | 's_find_sw_mode3': '', 190 | 's_find_sw_mode4': '', 191 | 's_find_sw_mode5': '', 192 | 's_find_work_mode1': '', 193 | 's_find_work_mode10': '', 194 | 's_find_work_mode2': '', 195 | 's_find_work_mode3': '', 196 | 's_find_work_mode4': '', 197 | 's_find_work_mode5': '', 198 | 's_find_work_mode6': '', 199 | 's_find_work_mode7': '', 200 | 's_find_work_mode8': '', 201 | 's_find_work_mode9': '', 202 | 's_find_zone_mode1': '', 203 | 's_find_zone_mode10': '', 204 | 's_find_zone_mode2': '', 205 | 's_find_zone_mode3': '', 206 | 's_find_zone_mode4': '', 207 | 's_find_zone_mode5': '', 208 | 's_find_zone_mode6': '', 209 | 's_find_zone_mode7': '', 210 | 's_find_zone_mode8': '', 211 | 's_find_zone_mode9': '', 212 | 'search_feature': '', 213 | 'search_from': 'joblist', 214 | 'search_item': '1', 215 | 'search_job2': '請選擇行業', 216 | 'search_key_word': 'java', 217 | 'search_multi_loc': '請選擇地區', 218 | 'search_multi_loc2': '請選擇地區', 219 | 'search_multi_loc3': '請選擇地區', 220 | 'search_multi_loc4': '請選擇地區', 221 | 'search_multi_loc5': '請選擇地區', 222 | 'search_multi_loc6': '請選擇地區', 223 | 'search_multi_loc7': '請選擇地區', 224 | 'search_subj': '', 225 | 'search_type': 'job', 226 | 'search_work': '請選擇職務', 227 | 'search_work2': '請選擇職務', 228 | 'search_work3': '請選擇職務', 229 | 'search_work4': '請選擇職務', 230 | 'search_work6': '', 231 | 'search_work7': '請選擇職務', 232 | 'search_work8': '請選擇職務', 233 | 'us_menu': ''} 234 | 235 | strrec = 0 236 | data['strrec'] = strrec 237 | 238 | while True: 239 | print("[INFO] crawling strrec %s"%strrec) 240 | res = r.post(URL, data=data, headers={'User-Agent': ''}) 241 | res.encoding = 'utf-8' 242 | if '查無資料' in res.text: 243 | break 244 | with open('./list_html/java_{}.html'.format(strrec), 'w') as f: 245 | f.write(res.text) 246 | strrec += 20 247 | data['strrec'] = strrec 248 | 249 | os.system("cat ./list_html/java_*.html | grep -Po 'job_refer_comp_job_detail2\.asp\?p_id=.*&job_id=\d*_\d*' | uniq > urls_uniq.txt") 250 | 251 | job_detail.get_detail() 252 | -------------------------------------------------------------------------------- /bb102/yes123/list_html/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ianchen06/pythonetl/6b4fb75bab09cb40bc774bd0c57cddb6c837c13c/bb102/yes123/list_html/.gitkeep -------------------------------------------------------------------------------- /bb103/bb103_pythonetl_20170829.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 5, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "# Mizuno crawler\n", 12 | "import requests as r\n", 13 | "from bs4 import BeautifulSoup" 14 | ] 15 | }, 16 | { 17 | "cell_type": "code", 18 | "execution_count": 3, 19 | "metadata": { 20 | "collapsed": true 21 | }, 22 | "outputs": [], 23 | "source": [ 24 | "res = r.get(\"http://www.mizuno.com.tw/04product/product_series_all.aspx?fid=6&tid=35\")" 25 | ] 26 | }, 27 | { 28 | "cell_type": "code", 29 | "execution_count": 6, 30 | "metadata": { 31 | "collapsed": true 32 | }, 33 | "outputs": [], 34 | "source": [ 35 | "soup = BeautifulSoup(res.text, 'html5lib')" 36 | ] 37 | }, 38 | { 39 | "cell_type": "code", 40 | "execution_count": 48, 41 | "metadata": { 42 | "collapsed": true 43 | }, 44 | "outputs": [], 45 | "source": [ 46 | "product_list = soup.find_all(lambda x: 'product_detail' in x.get('href', ''))" 47 | ] 48 | }, 49 | { 50 | "cell_type": "code", 51 | "execution_count": 52, 52 | "metadata": { 53 | "scrolled": true 54 | }, 55 | "outputs": [ 56 | { 57 | "data": { 58 | "text/plain": [ 59 | "{'product_detail.aspx?pid=17852',\n", 60 | " 'product_detail.aspx?pid=17853',\n", 61 | " 'product_detail.aspx?pid=19849',\n", 62 | " 'product_detail.aspx?pid=19850',\n", 63 | " 'product_detail.aspx?pid=19881',\n", 64 | " 'product_detail.aspx?pid=19882',\n", 65 | " 'product_detail.aspx?pid=19883',\n", 66 | " 'product_detail.aspx?pid=19884',\n", 67 | " 'product_detail.aspx?pid=19885',\n", 68 | " 'product_detail.aspx?pid=20216',\n", 69 | " 'product_detail.aspx?pid=20217'}" 70 | ] 71 | }, 72 | "execution_count": 52, 73 | "metadata": {}, 74 | "output_type": "execute_result" 75 | } 76 | ], 77 | "source": [ 78 | "products = {list['href'] for list in product_list}\n", 79 | "products" 80 | ] 81 | }, 82 | { 83 | "cell_type": "code", 84 | "execution_count": 54, 85 | "metadata": {}, 86 | "outputs": [ 87 | { 88 | "name": "stdout", 89 | "output_type": "stream", 90 | "text": [ 91 | "http://www.mizuno.com.tw/04product/product_detail.aspx?pid=20216\n", 92 | "http://www.mizuno.com.tw/04product/product_detail.aspx?pid=20217\n", 93 | "http://www.mizuno.com.tw/04product/product_detail.aspx?pid=17853\n", 94 | "http://www.mizuno.com.tw/04product/product_detail.aspx?pid=19884\n", 95 | "http://www.mizuno.com.tw/04product/product_detail.aspx?pid=19882\n", 96 | "http://www.mizuno.com.tw/04product/product_detail.aspx?pid=19885\n", 97 | "http://www.mizuno.com.tw/04product/product_detail.aspx?pid=19883\n", 98 | "http://www.mizuno.com.tw/04product/product_detail.aspx?pid=19850\n", 99 | "http://www.mizuno.com.tw/04product/product_detail.aspx?pid=17852\n", 100 | "http://www.mizuno.com.tw/04product/product_detail.aspx?pid=19849\n", 101 | "http://www.mizuno.com.tw/04product/product_detail.aspx?pid=19881\n" 102 | ] 103 | } 104 | ], 105 | "source": [ 106 | "HOST = \"http://www.mizuno.com.tw/04product/\"\n", 107 | "\n", 108 | "for link in products:\n", 109 | " url = HOST + link\n", 110 | " print(url)" 111 | ] 112 | }, 113 | { 114 | "cell_type": "code", 115 | "execution_count": null, 116 | "metadata": { 117 | "collapsed": true 118 | }, 119 | "outputs": [], 120 | "source": [ 121 | "# Requests and proxy" 122 | ] 123 | }, 124 | { 125 | "cell_type": "code", 126 | "execution_count": null, 127 | "metadata": { 128 | "collapsed": true 129 | }, 130 | "outputs": [], 131 | "source": [ 132 | "# HTTP proxy\n", 133 | "# found online, https://www.sslproxies.org/" 134 | ] 135 | }, 136 | { 137 | "cell_type": "code", 138 | "execution_count": 40, 139 | "metadata": {}, 140 | "outputs": [ 141 | { 142 | "data": { 143 | "text/plain": [ 144 | "'201.218.63.171'" 145 | ] 146 | }, 147 | "execution_count": 40, 148 | "metadata": {}, 149 | "output_type": "execute_result" 150 | } 151 | ], 152 | "source": [ 153 | "import requests as r\n", 154 | "\n", 155 | "proxies = {\n", 156 | " 'http': 'http://201.218.63.171:65301',\n", 157 | " 'https': 'http://201.218.63.171:65301',\n", 158 | "}\n", 159 | "\n", 160 | "url = 'http://api.ipify.org'\n", 161 | "r.get(url, proxies=proxies).text" 162 | ] 163 | }, 164 | { 165 | "cell_type": "code", 166 | "execution_count": 41, 167 | "metadata": {}, 168 | "outputs": [ 169 | { 170 | "data": { 171 | "text/plain": [ 172 | "'111.248.77.73'" 173 | ] 174 | }, 175 | "execution_count": 41, 176 | "metadata": {}, 177 | "output_type": "execute_result" 178 | } 179 | ], 180 | "source": [ 181 | "import requests as r\n", 182 | "\n", 183 | "proxies = {\n", 184 | " 'http': 'http://201.218.63.171:65301',\n", 185 | " 'https': 'http://201.218.63.171:65301',\n", 186 | "}\n", 187 | "\n", 188 | "url = 'http://api.ipify.org'\n", 189 | "r.get(url).text" 190 | ] 191 | }, 192 | { 193 | "cell_type": "code", 194 | "execution_count": null, 195 | "metadata": { 196 | "collapsed": true 197 | }, 198 | "outputs": [], 199 | "source": [ 200 | "# SOCKS proxy,\n", 201 | "# can be any server with SSH\n", 202 | "# example:\n", 203 | "# ssh -D [PORT] -N [username]@[HOST(ip)]\n", 204 | "# In this example, PORT is 8123" 205 | ] 206 | }, 207 | { 208 | "cell_type": "code", 209 | "execution_count": 44, 210 | "metadata": {}, 211 | "outputs": [ 212 | { 213 | "data": { 214 | "text/plain": [ 215 | "'35.185.238.206'" 216 | ] 217 | }, 218 | "execution_count": 44, 219 | "metadata": {}, 220 | "output_type": "execute_result" 221 | } 222 | ], 223 | "source": [ 224 | "import requests as r\n", 225 | "\n", 226 | "proxies = {\n", 227 | " 'http': 'socks5://localhost:8123',\n", 228 | " 'https': 'socks5://localhost:8123',\n", 229 | "}\n", 230 | "\n", 231 | "url = 'http://api.ipify.org'\n", 232 | "r.get(url, proxies=proxies).text" 233 | ] 234 | }, 235 | { 236 | "cell_type": "code", 237 | "execution_count": 45, 238 | "metadata": {}, 239 | "outputs": [ 240 | { 241 | "data": { 242 | "text/plain": [ 243 | "'111.248.77.73'" 244 | ] 245 | }, 246 | "execution_count": 45, 247 | "metadata": {}, 248 | "output_type": "execute_result" 249 | } 250 | ], 251 | "source": [ 252 | "import requests as r\n", 253 | "\n", 254 | "proxies = {\n", 255 | " 'http': 'socks5://localhost:8123',\n", 256 | " 'https': 'socks5://localhost:8123',\n", 257 | "}\n", 258 | "\n", 259 | "url = 'http://api.ipify.org'\n", 260 | "r.get(url).text" 261 | ] 262 | }, 263 | { 264 | "cell_type": "code", 265 | "execution_count": 46, 266 | "metadata": { 267 | "collapsed": true 268 | }, 269 | "outputs": [], 270 | "source": [ 271 | "# Python and MySQL" 272 | ] 273 | }, 274 | { 275 | "cell_type": "code", 276 | "execution_count": 55, 277 | "metadata": { 278 | "collapsed": true 279 | }, 280 | "outputs": [], 281 | "source": [ 282 | "# Install MySQL on Ubuntu\n", 283 | "# sudo apt-get update && sudo apt-get intall -y mariadb-server" 284 | ] 285 | }, 286 | { 287 | "cell_type": "code", 288 | "execution_count": 56, 289 | "metadata": {}, 290 | "outputs": [ 291 | { 292 | "name": "stdout", 293 | "output_type": "stream", 294 | "text": [ 295 | "Collecting peewee\n", 296 | " Downloading peewee-2.10.1.tar.gz (515kB)\n", 297 | "\u001b[K 100% |████████████████████████████████| 522kB 2.0MB/s ta 0:00:01\n", 298 | "\u001b[?25hInstalling collected packages: peewee\n", 299 | " Running setup.py install for peewee ... \u001b[?25ldone\n", 300 | "\u001b[?25hSuccessfully installed peewee-2.10.1\n" 301 | ] 302 | } 303 | ], 304 | "source": [ 305 | "!pip install peewee" 306 | ] 307 | }, 308 | { 309 | "cell_type": "code", 310 | "execution_count": 57, 311 | "metadata": {}, 312 | "outputs": [ 313 | { 314 | "name": "stdout", 315 | "output_type": "stream", 316 | "text": [ 317 | "Requirement already satisfied: PyMySQL in /Users/ian/.pyenv/versions/3.6.1/lib/python3.6/site-packages\r\n" 318 | ] 319 | } 320 | ], 321 | "source": [ 322 | "!pip install PyMySQL" 323 | ] 324 | }, 325 | { 326 | "cell_type": "code", 327 | "execution_count": 58, 328 | "metadata": { 329 | "collapsed": true 330 | }, 331 | "outputs": [], 332 | "source": [ 333 | "import peewee" 334 | ] 335 | }, 336 | { 337 | "cell_type": "code", 338 | "execution_count": 111, 339 | "metadata": { 340 | "collapsed": true 341 | }, 342 | "outputs": [], 343 | "source": [ 344 | "db = peewee.MySQLDatabase('news',\n", 345 | " user='admin',\n", 346 | " host='localhost',\n", 347 | " password='')" 348 | ] 349 | }, 350 | { 351 | "cell_type": "code", 352 | "execution_count": 61, 353 | "metadata": {}, 354 | "outputs": [ 355 | { 356 | "data": { 357 | "text/plain": [ 358 | "{'email': 'ianchen06@gmail.com',\n", 359 | " 'id': 1,\n", 360 | " 'name': 'ian',\n", 361 | " 'website': 'ianchenhq.com',\n", 362 | " 'year': 2017}" 363 | ] 364 | }, 365 | "execution_count": 61, 366 | "metadata": {}, 367 | "output_type": "execute_result" 368 | } 369 | ], 370 | "source": [ 371 | "{\n", 372 | " 'id': 1,\n", 373 | " 'name': 'ian',\n", 374 | " 'email': 'ianchen06@gmail.com',\n", 375 | " 'website': 'ianchenhq.com',\n", 376 | " 'year': 2017\n", 377 | "}" 378 | ] 379 | }, 380 | { 381 | "cell_type": "code", 382 | "execution_count": 113, 383 | "metadata": { 384 | "collapsed": true 385 | }, 386 | "outputs": [], 387 | "source": [ 388 | "customers = [\n", 389 | " {\n", 390 | " 'name': 'ian',\n", 391 | " 'email': 'ianchen06@gmail.com',\n", 392 | " 'eth': 2017\n", 393 | " },\n", 394 | " {\n", 395 | " 'name': 'ian',\n", 396 | " 'email': 'ianchen06@gmail.com',\n", 397 | " 'eth': 2018\n", 398 | " }\n", 399 | "]" 400 | ] 401 | }, 402 | { 403 | "cell_type": "code", 404 | "execution_count": 87, 405 | "metadata": { 406 | "collapsed": true 407 | }, 408 | "outputs": [], 409 | "source": [ 410 | "class Customer(peewee.Model):\n", 411 | " name = peewee.CharField()\n", 412 | " email = peewee.CharField()\n", 413 | " eth = peewee.IntegerField()\n", 414 | " \n", 415 | " class Meta:\n", 416 | " database = db" 417 | ] 418 | }, 419 | { 420 | "cell_type": "code", 421 | "execution_count": 65, 422 | "metadata": {}, 423 | "outputs": [ 424 | { 425 | "data": { 426 | "text/plain": [ 427 | "" 428 | ] 429 | }, 430 | "execution_count": 65, 431 | "metadata": {}, 432 | "output_type": "execute_result" 433 | } 434 | ], 435 | "source": [ 436 | "db.create_table(Customer)" 437 | ] 438 | }, 439 | { 440 | "cell_type": "code", 441 | "execution_count": 66, 442 | "metadata": { 443 | "collapsed": true 444 | }, 445 | "outputs": [], 446 | "source": [ 447 | "user = Customer(name='Ian',\n", 448 | " email='ianchen06@gmail.com',\n", 449 | " eth=1)" 450 | ] 451 | }, 452 | { 453 | "cell_type": "code", 454 | "execution_count": 67, 455 | "metadata": {}, 456 | "outputs": [ 457 | { 458 | "data": { 459 | "text/plain": [ 460 | "<__main__.Customer at 0x10e6c4518>" 461 | ] 462 | }, 463 | "execution_count": 67, 464 | "metadata": {}, 465 | "output_type": "execute_result" 466 | } 467 | ], 468 | "source": [ 469 | "user" 470 | ] 471 | }, 472 | { 473 | "cell_type": "code", 474 | "execution_count": 68, 475 | "metadata": {}, 476 | "outputs": [ 477 | { 478 | "data": { 479 | "text/plain": [ 480 | "1" 481 | ] 482 | }, 483 | "execution_count": 68, 484 | "metadata": {}, 485 | "output_type": "execute_result" 486 | } 487 | ], 488 | "source": [ 489 | "user.save()" 490 | ] 491 | }, 492 | { 493 | "cell_type": "code", 494 | "execution_count": 88, 495 | "metadata": { 496 | "collapsed": true 497 | }, 498 | "outputs": [], 499 | "source": [ 500 | "user = Customer(name='David',\n", 501 | " email='david@gmail.com',\n", 502 | " eth=2)" 503 | ] 504 | }, 505 | { 506 | "cell_type": "code", 507 | "execution_count": 89, 508 | "metadata": {}, 509 | "outputs": [ 510 | { 511 | "data": { 512 | "text/plain": [ 513 | "1" 514 | ] 515 | }, 516 | "execution_count": 89, 517 | "metadata": {}, 518 | "output_type": "execute_result" 519 | } 520 | ], 521 | "source": [ 522 | "user.save()" 523 | ] 524 | }, 525 | { 526 | "cell_type": "code", 527 | "execution_count": 91, 528 | "metadata": {}, 529 | "outputs": [ 530 | { 531 | "data": { 532 | "text/plain": [ 533 | "'ianchen06@gmail.com'" 534 | ] 535 | }, 536 | "execution_count": 91, 537 | "metadata": {}, 538 | "output_type": "execute_result" 539 | } 540 | ], 541 | "source": [ 542 | "Customer.get(Customer.name == 'Ian').email" 543 | ] 544 | }, 545 | { 546 | "cell_type": "code", 547 | "execution_count": 114, 548 | "metadata": {}, 549 | "outputs": [], 550 | "source": [ 551 | "# Bulk insert\n", 552 | "\n", 553 | "with db.atomic():\n", 554 | " Customer.insert_many(customers).execute()" 555 | ] 556 | }, 557 | { 558 | "cell_type": "code", 559 | "execution_count": null, 560 | "metadata": { 561 | "collapsed": true 562 | }, 563 | "outputs": [], 564 | "source": [ 565 | "# High level query API,\n", 566 | "# peewee.Dataset\n", 567 | "# very freee API to interact with RDBMS" 568 | ] 569 | }, 570 | { 571 | "cell_type": "code", 572 | "execution_count": 93, 573 | "metadata": {}, 574 | "outputs": [], 575 | "source": [ 576 | "my_dict = {}" 577 | ] 578 | }, 579 | { 580 | "cell_type": "code", 581 | "execution_count": 94, 582 | "metadata": {}, 583 | "outputs": [ 584 | { 585 | "data": { 586 | "text/plain": [ 587 | "dict_keys([])" 588 | ] 589 | }, 590 | "execution_count": 94, 591 | "metadata": {}, 592 | "output_type": "execute_result" 593 | } 594 | ], 595 | "source": [ 596 | "my_dict.keys()" 597 | ] 598 | }, 599 | { 600 | "cell_type": "code", 601 | "execution_count": 117, 602 | "metadata": {}, 603 | "outputs": [], 604 | "source": [ 605 | "from playhouse.dataset import DataSet\n", 606 | "\n", 607 | "db = DataSet('mysql://admin@localhost/news')" 608 | ] 609 | }, 610 | { 611 | "cell_type": "code", 612 | "execution_count": 118, 613 | "metadata": {}, 614 | "outputs": [ 615 | { 616 | "data": { 617 | "text/plain": [ 618 | "['cnyes', 'customer']" 619 | ] 620 | }, 621 | "execution_count": 118, 622 | "metadata": {}, 623 | "output_type": "execute_result" 624 | } 625 | ], 626 | "source": [ 627 | "db.tables" 628 | ] 629 | }, 630 | { 631 | "cell_type": "code", 632 | "execution_count": 119, 633 | "metadata": {}, 634 | "outputs": [], 635 | "source": [ 636 | "customer = db['customer']" 637 | ] 638 | }, 639 | { 640 | "cell_type": "code", 641 | "execution_count": 121, 642 | "metadata": {}, 643 | "outputs": [ 644 | { 645 | "data": { 646 | "text/plain": [ 647 | "6" 648 | ] 649 | }, 650 | "execution_count": 121, 651 | "metadata": {}, 652 | "output_type": "execute_result" 653 | } 654 | ], 655 | "source": [ 656 | "customer.insert(\n", 657 | " **{\n", 658 | " 'name': 'eason',\n", 659 | " 'email': 'eason@gmail.com',\n", 660 | " 'eth': 100\n", 661 | " }\n", 662 | ")" 663 | ] 664 | }, 665 | { 666 | "cell_type": "code", 667 | "execution_count": 122, 668 | "metadata": {}, 669 | "outputs": [ 670 | { 671 | "name": "stdout", 672 | "output_type": "stream", 673 | "text": [ 674 | "{'id': 6, 'name': 'eason', 'email': 'eason@gmail.com', 'eth': 100}\n" 675 | ] 676 | } 677 | ], 678 | "source": [ 679 | "for row in customer.find(name='eason'):\n", 680 | " print(row)" 681 | ] 682 | }, 683 | { 684 | "cell_type": "code", 685 | "execution_count": 107, 686 | "metadata": {}, 687 | "outputs": [ 688 | { 689 | "data": { 690 | "text/plain": [ 691 | "{'email': 'ianchen06@gmail.com', 'eth': 1, 'id': 1, 'name': 'Ian'}" 692 | ] 693 | }, 694 | "execution_count": 107, 695 | "metadata": {}, 696 | "output_type": "execute_result" 697 | } 698 | ], 699 | "source": [ 700 | "customer.find_one(name='Ian')" 701 | ] 702 | }, 703 | { 704 | "cell_type": "code", 705 | "execution_count": 116, 706 | "metadata": { 707 | "collapsed": true 708 | }, 709 | "outputs": [], 710 | "source": [ 711 | "# Real example\n", 712 | "# https://github.com/ianchen06/cnyes_cralwer" 713 | ] 714 | }, 715 | { 716 | "cell_type": "code", 717 | "execution_count": null, 718 | "metadata": { 719 | "collapsed": true 720 | }, 721 | "outputs": [], 722 | "source": [] 723 | } 724 | ], 725 | "metadata": { 726 | "kernelspec": { 727 | "display_name": "Python 3", 728 | "language": "python", 729 | "name": "python3" 730 | }, 731 | "language_info": { 732 | "codemirror_mode": { 733 | "name": "ipython", 734 | "version": 3 735 | }, 736 | "file_extension": ".py", 737 | "mimetype": "text/x-python", 738 | "name": "python", 739 | "nbconvert_exporter": "python", 740 | "pygments_lexer": "ipython3", 741 | "version": "3.6.1" 742 | } 743 | }, 744 | "nbformat": 4, 745 | "nbformat_minor": 2 746 | } 747 | -------------------------------------------------------------------------------- /bb103/pm2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ianchen06/pythonetl/6b4fb75bab09cb40bc774bd0c57cddb6c837c13c/bb103/pm2.png -------------------------------------------------------------------------------- /bb103/trump_interview.txt: -------------------------------------------------------------------------------- 1 | PRESIDENT TRUMP: Hello, everybody. Great to be back in New York with all of our friends and some great friends outside the building, I must tell you. 2 | 3 | I want to thank all of our distinguished guests who are with us today, including members of our cabinet: Treasury secretary Stephen Mnuchin, OMB Director Mick Mulvaney and, of course, our Transportation Secretary who is doing a fabulous job, Elaine Chao. 4 | 5 | Thank you all for doing a really incredible and creative job on what we're going to be discussing today, which is infrastructure. We've just had a great set of briefings upstairs on our infrastructure agenda. My administration is working every day to deliver the world class infrastructure that our people deserve, and frankly our country deserves. That’s why I just signed a new Executive Order to dramatically reform the nation's badly broken infrastructure permitting process. Just blocks away is the Empire State Building. It took 11 months to build the Empire State Building. But today it can take as long as a decade, and much more than that. Many, many stories where it takes 20 and 25 years just to get approvals to start construction of a fairly routine highway. Highway builders must get up to 16 different approvals involving nine different federal agencies, governed by 29 different statutes. One agency alone can stall a project for many, many years and even decades. 6 | 7 | 8 | Not only does this cost our economy billions of dollars, but it also denies our citizens the safe and modern infrastructure they deserve. This overregulated permitting process is a massive self-inflicted wound on our country. It’s disgraceful. Denying our people much needed investments in their community, and I just want to show you this, because it was just shown to me, and I said, I think I'm going to show it to the media – both real and fake media by the way. 9 | 10 | This is what it takes to get something approved today. Elaine, you see that? So this is what it takes. Permitting process flow chart, that's a flow chart. So that can go out to about 20 years, this shows about 10. But that can go out to about 20 years to get something approved. This is for a highway. I’ve seen a highway recently in a certain state – I won’t mention its name, it’s 17 years. I could have built it for $4 million, $5 million without the permitting process. It costs hundreds of millions of dollars, but it took 17 years to get it approved, and many, many, many, many pages of environmental impact studies. 11 | 12 | This is what we will bring it down to. This is less than two years. This is going to happen quickly, that's what I'm signing today. This will be less than two years for a highway. So it's going to be quick, it's going to be a very streamlined process, and by the way, if it doesn't meet environmental safeguards, we are not going to approve it. Very simple, we’re not going to approve it. So this is – maybe this one will say, let's throw the other one away. Would anybody like it from the media? Would anybody like that long, beautiful chart, you can have it. 13 | 14 | 15 | So my executive order also requires agencies to work together efficiently by requiring one lead agency for each major infrastructure project. It also holds agencies accountable if they fail to streamline their review process, so each agency is accountable. We're going to get infrastructure built quickly, inexpensively, relatively speaking, and the permitting process will go very, very quickly. No longer will we tolerate one job killing delay after another. No longer will we accept a broken system that benefits consultants and lobbyists at the expense of hardworking Americans. 16 | 17 | Now, I knew the process very well, probably better than anybody. I had to get permits for this building and many of the buildings I built. All of the buildings I built in Manhattan and many other places, and I will tell you that the consultants are rich people. They go around making it very difficult. They lobby congress, they lobby state government, city governments to make it very difficult so that you have to hire consultants and that you have to take years and pay them a fortune, so we’re streamlining the process, and we won't be having so much of that anymore. 18 | 19 | No longer will we allow the infrastructure of our magnificent country to crumble and decay, while protecting the environment we will build gleaming new roads, bridges, railways, waterways, tunnels and highways. We will rebuild our country with American workers, American iron, American aluminum, American steel. We will create millions of new jobs and make millions of American dreams come true. 20 | 21 | 22 | Our infrastructure will again be the best in the world. We used to have the greatest infrastructure anywhere in the world, and today, we are like a third-world country. We are literally like a third-world country. Our infrastructure will again be the best, and we will restore the pride in our communities, our nation, and all over the United States, we will be proud again. 23 | 24 | So I want to thank everybody for being here. God bless you, God bless the United States. If you have any questions, Mick, you could come up here please, come on up. Mick Mulvaney. If you have any questions, please feel free to ask. 25 | 26 | REPORTER: Mr. President, why is it that are CEOs are leaving your manufacturing council? 27 | 28 | TRUMP: Because they're not taking their jobs seriously as it pertains to this country. We want jobs, manufacturing in this country. If you look at some of those people that you’re talking about, they’re outside of the country, they’re having a lot of their product made outside, if you look at Merck as an example. 29 | 30 | (chatter) 31 | 32 | Excuse me, excuse me. Take a look at where their product is made. It’s made outside of our country. We want products made in the country. Now I have to tell you, some of the folks that will leave, they’re leaving out of embarrassment, because they make their products outside, and I’ve been lecturing them, including the gentleman you’ve been referring to, about, you have to bring it back to this country. You can’t do it, necessarily, in Ireland, and all of these other places. You have to bring this work back to this country. That’s what I want – I want manufacturing to be back into the United States so that American workers can benefit. 33 | 34 | (chatter) 35 | 36 | Reporter: Mr. President, why did you wait so long to blast neo-Nazis? 37 | 38 | TRUMP: I didn’t wait long. I didn’t wait long. I wanted to make sure, unlike most politicians, that what I said was correct – not make a quick statement. The statement I made on Saturday, the first statement, was a fine statement. But you don’t make statements that direct unless you know the facts. It takes a little while to get the facts. You still don’t know the facts. And it’s a very, very important process to me and it’s an important statement. So I don’t want to go quickly, and just make a statement for the sake of making a political statement. I want to know the facts. If you go back to my – in fact, I brought it. I brought it. I brought it. 39 | 40 | As I said on, remember, Saturday, “We condemn in the strongest possible terms this egregious display of hatred, bigotry and violence. It has no place in America.” And then I went on from there. 41 | 42 | Now, here’s the thing. 43 | 44 | (chatter) 45 | 46 | Excuse me, excuse me. Take it nice and easy. 47 | 48 | Here’s the thing. When I make a statement, I like to be correct. I want the facts. This event just happened. In fact, a lot of it didn’t even happen yet, as we were speaking. This event just happened. Before I make a statement, I need the facts. I don’t want to rush into a statement, so making the statement when I made it was excellent. 49 | 50 | In fact, the young woman, who I hear is a fantastic young woman, her mother wrote me, and said, through I guess Twitter, social media, the nicest things, and I very much appreciated that. I hear she was a fine, really actually an incredible young woman. But her mother, on Twitter, thanked me for what I said, and honestly, if the press were not fake, and it was honest, the press would have said what I said was very nice. 51 | 52 | (chatter) 53 | 54 | Excuse me. But unlike you, and unlike the media – before I make a statement, I like to know the facts. 55 | 56 | (chatter) 57 | 58 | They don’t. They don’t. 59 | 60 | How about a couple of infrastructure questions. 61 | 62 | Reporter: The CEO of Walmart said you missed a critical moment to bring the country together. Did you? 63 | 64 | TRUMP: Not at all. Look, you take a look. I’ve created over a million jobs since I’m president. The country is booming, the stock market is setting records, we have the highest unemployment we’ve ever had in the history of our country, we’re doing record business, we have the highest levels of enthusiasm, so the head of Walmart, who I know – he’s a very nice guy – was making a political statement. I mean, ask him how his – they do it the same way, and you know why? Because I want to make sure, when I make the statement, that the statement was correct. There’s no way, there was no way of making a correct statement that early. I had to see the facts, unlike a lot of reporters, unlike a lot of reporters. 65 | 66 | I didn’t know David Duke was there. I wanted to see the facts, and the facts, as they started coming out, were very well stated. In fact, everybody said, “his statement was beautiful. If he would have made it sooner, that would have been good.” I couldn’t have made it sooner because I didn’t know all of the facts. Frankly, people still don’t know all of the facts. It was very important – 67 | 68 | (chatter) 69 | 70 | Excuse me, excuse me. It was very important to me to get the facts out, and correctly. Because if I would have made a fast statement – and the first statement was made without knowing much other than what we were seeing. The second statement was made after – with knowledge, with great knowledge. There’s still things – excuse me – there’s still things that people don’t know. I want to make a statement with knowledge, I wanted to know the facts. Okay. 71 | 72 | Reporter: Two questions – was this terrorism? And can you tell us how you’re feeling about your chief strategist, Stephen Bannon? 73 | 74 | TRUMP: Well, I think the driver of the car is a disgrace to himself, his family and this country, and that is – you can call it terrorism, you can call it murder, you can call it whatever you want. I would just call it as the fastest one to come up with a good verdict, that’s what I’d call it, because there is a question, is it murder? Is it terrorism? And then you get into legal semantics. 75 | 76 | The driver of the car is a murderer, and what he did is a horrible, horrible, inexcusable thing. 77 | 78 | Reporter: Can you tell us how you feel about your chief strategist, Mr. Bannon? 79 | 80 | TRUMP: I never spoke to Mr. Bannon about it. 81 | 82 | Reporter: But can you tell us broadly what your, do you still have confidence in Steve? 83 | 84 | TRUMP: Well, let’s see. Look, look, I like Mr. Bannon, he’s a friend of mine. But Mr. Bannon came on very late, you know that. I went through 17 senators, governors, and I won all the primaries. Mr. Bannon came on very much later than that, and I like him. He’s a good man. He’s not a racist, I can tell you that. He’s a good person. He actually gets a very unfair press in that regard. But, we’ll see what happens with Mr. Bannon. But he’s a good person, and I think the press treats him, frankly, very unfairly. 85 | 86 | Reporter: McCain has called on you to defend your national security adviser, H.R. McMaster, against attacks. 87 | 88 | TRUMP: I’ve already done that, I did it the last time. 89 | 90 | Reporter: And he called on it again, 91 | 92 | TRUMP: Senator McCain, you mean the one who voted against Obamacare? You mean Senator McCain, who voted against us getting good health care? 93 | 94 | Reporter: Senator McCain said that the alt-right is behind these attacks, and he linked that same group to those who perpetrated the attack against Charlottesville. 95 | 96 | TRUMP: Well, I don’t know, I can’t tell you. I’m sure Senator McCain must know what he’s talking about. But when you say the alt-right, define alt-right to me. You define it to me. Define it for me, go ahead. 97 | 98 | Reporter: Senator McCain defined it as the same group – 99 | 100 | Trump: Okay, what about the alt-left that came charging at ‘em. Excuse me, what about the alt-left that came charging at, as you say the alt-right? Do they have any semblance of guilt? 101 | 102 | (chatter) 103 | 104 | TRUMP: Let me ask you this – what about the fact they came charging, that they came charging with clubs in their hands, swinging clubs? Do they have any problem? I think they do. So, you know, as far as I’m concerned, that was a horrible, horrible day. Wait a minute – I’m not finished, fake news. That was a horrible day. 105 | 106 | (chatter) 107 | 108 | TRUMP: I will tell you something. I watched those very closely, much more closely than you people watched it, and you have, you had a group on one side that was bad, and you had a group on the other side that was also very violent. And nobody wants to say that, but I’ll say it right now. You had a group on the other side that came charging in, without a permit, and they were very, very violent. 109 | 110 | (chatter) 111 | 112 | Reporter: Do you think that what you call the alt-left is the same as neo-Nazis? 113 | 114 | TRUMP: Those people, all of those people, excuse me. I’ve condemned neo-Nazis. I’ve condemned many different groups. But not all of those people were neo-Nazis, believe me. Not all of those people were white supremacists, by any stretch. Those people were also there because they wanted to protest the taking down of a statue of Robert E. Lee. 115 | 116 | Q Should that statue be taken down? 117 | 118 | THE PRESIDENT: Excuse me. If you take a look at some of the groups, and you see – and you'd know it if you were honest reporters, which in many cases you're not – but many of those people were there to protest the taking down of the statue of Robert E. Lee. So this week it's Robert E. Lee. I noticed that Stonewall Jackson is coming down. I wonder, is it George Washington next week? And is it Thomas Jefferson the week after? You know, you really do have to ask yourself, where does it stop? 119 | 120 | But they were there to protest -- excuse me, if you take a look, the night before they were there to protest the taking down of the statue of Robert E. Lee. Infrastructure question. Go ahead. 121 | 122 | Reporter: Should the statues of Robert E. Lee stay up? 123 | 124 | TRUMP: I would say that's up to a local town, community, or the federal government, depending on where it is located. 125 | 126 | Reporter: How concerned are you about race relations in America, and do you think things have gotten worse or better since you took office? 127 | 128 | TRUMP: I think they've gotten better or the same. Look, they've been frayed for a long time. And you can ask President Obama about that, because he'd make speeches about it. But I believe that the fact that I brought in -- it will be soon -- millions of jobs -- you see where companies are moving back into our country -- I think that's going to have a tremendous, positive impact on race relations. We have companies coming back into our country. We have two car companies that just announced. We have Foxconn in Wisconsin just announced. We have many companies, I say, pouring back into the country. I think that's going to have a huge, positive impact on race relations. 129 | 130 | You know why? It's jobs. What people want now, they want jobs. They want great jobs with good pay, and when they have that, you watch how race relations will be. And I’ll tell you, we’re spending a lot of money on the inner cities. We’re fixing the inner cities. We’re doing far more than anybody has done with respect to the inner cities. It’s a priority for me, and it’s very important. 131 | 132 | Reporter: Mr. President, are you putting what you’re calling the alt-left and white supremacists on the same moral plane? 133 | 134 | TRUMP: I’m not putting anybody on a moral plane. What I’m saying is this: You had a group on one side and you had a group on the other, and they came at each other with clubs -- and it was vicious and it was horrible. And it was a horrible thing to watch. But there is another side. There was a group on this side. You can call them the left -- you just called them the left -- that came violently attacking the other group. So you can say what you want, but that’s the way it is. 135 | 136 | Reporter: You said there was hatred, there was violence on both sides. Are the – 137 | 138 | THE PRESIDENT: Yes, I think there’s blame on both sides. If you look at both sides -- I think there’s blame on both sides. And I have no doubt about it, and you don’t have any doubt about it either. And if you reported it accurately, you would say. 139 | 140 | Reporter: The neo-Nazis started this. They showed up in Charlottesville to protest – 141 | 142 | TRUMP: Excuse me, excuse me. They didn’t put themselves -- and you had some very bad people in that group, but you also had people that were very fine people, on both sides. You had people in that group. 143 | 144 | (chatter) 145 | 146 | PRESIDENT: Excuse me, excuse me. I saw the same pictures as you did. You had people in that group that were there to protest the taking down of, to them, a very, very important statue and the renaming of a park from Robert E. Lee to another name. 147 | 148 | Reporter: George Washington and Robert E. Lee are not the same. 149 | 150 | TRUMP: George Washington was a slave owner. Was George Washington a slave owner? So will George Washington now lose his status? Are we going to take down -- Excuse me, are we going to take down statues to George Washington? How about Thomas Jefferson? What do you think of Thomas Jefferson? You like him? 151 | 152 | Reporter: I do love Thomas Jefferson. 153 | 154 | TRUMP: Okay, good. Are we going to take down the statue? Because he was a major slave owner. Now, are we going to take down his statue? So you know what, it’s fine. You’re changing history. You’re changing culture. And you had people -- and I’m not talking about the neo-Nazis and the white nationalists -- because they should be condemned totally. But you had many people in that group other than neo-Nazis and white nationalists. Okay? And the press has treated them absolutely unfairly. Now, in the other group also, you had some fine people. But you also had troublemakers, and you see them come with the black outfits and with the helmets, and with the baseball bats. You had a lot of bad people in the other group. 155 | 156 | Reporter: Who are the good people? Sir, I just didn’t understand what you were saying. You were saying the press has treated white nationalists unfairly? I just don’t understand what you were saying. 157 | 158 | TRUMP: No, no. There were people in that rally -- and I looked the night before -- if you look, there were people protesting very quietly the taking down of the statue of Robert E. Lee. I’m sure in that group there were some bad ones. The following day it looked like they had some rough, bad people -- neo-Nazis, white nationalists, whatever you want to call them. But you had a lot of people in that group that were there to innocently protest, and very legally protest -- because I don’t know if you know, they had a permit. The other group didn’t have a permit. So I only tell you this: There are two sides to a story. I thought what took place was a horrible moment for our country -- a horrible moment. But there are two sides to the country. Does anybody have a final – 159 | 160 | Reporter: I have an infrastructure question. 161 | 162 | TRUMP: You have an infrastructure – 163 | 164 | Reporter: What makes you think you can get an infrastructure bill? You didn’t get healthcare – 165 | 166 | TRUMP: Well, you know, I’ll tell you. We came very close with healthcare. Unfortunately, John McCain decided to vote against it at the last minute. You’ll have to ask John McCain why he did that. But we came very close to healthcare. We will end up getting healthcare. But we’ll get the infrastructure. And actually, infrastructure is something that I think we’ll have bipartisan support on. I actually think Democrats will go along with the infrastructure. 167 | 168 | Reporter: Mr. President, have you spoken to the family of the victim of the car attack? 169 | 170 | TRUMP: No, I’ll be reaching out. I’ll be reaching out. 171 | 172 | Reporter: When will you be reaching out? 173 | 174 | TRUMP: I thought that the statement put out -- the mother’s statement I thought was a beautiful statement. I will tell you, it was something that I really appreciated. I thought it was terrific. And, really, under the kind of stress that she’s under and the heartache that she’s under, I thought putting out that statement, to me, was really something. I won’t forget it. Thank you, all, very much. Thank you. Thank you. 175 | 176 | Reporter: Will you go to Charlottesville? Will you go to check out what happened? 177 | 178 | TRUMP: I own a house in Charlottesville. Does anyone know I own a house in Charlottesville? 179 | 180 | Reporter: Where is it? 181 | 182 | TRUMP: Oh boy, it’s going to be – 183 | 184 | Reporter: Where is it? 185 | 186 | TRUMP: It's in Charlottesville. You'll see. 187 | 188 | Reporter: Is it a winery or something? 189 | 190 | TRUMP: It is the winery. I mean, I know a lot about Charlottesville. Charlottesville is a great place that's been very badly hurt over the last couple of days. 191 | 192 | (chatter) 193 | 194 | TRUMP: I own, actually, one of the largest wineries in the United States. It's in Charlottesville. 195 | 196 | Reporter: Do you believe your words are helping to heal this country right now? What do you think needs to be done to overcome the racial divides in this country? 197 | 198 | subscribe 199 | The story must be told. 200 | Your subscription supports journalism that matters. 201 | Try 1 month for 99¢ 202 | TRUMP: Well, I think jobs can have a big impact. I think if we continue to create jobs -- over a million, substantially more than a million. And you see just the other day, the car companies coming in with Foxconn. I think if we continue to create jobs at levels that I’m creating jobs, I think that’s going to have a tremendous impact -- positive impact on race relations. 203 | 204 | Reporter: Your remarks today, how do you think that will impact the racial, sort of conflict, today? 205 | 206 | TRUMP: The people are going to be working, they’re going to be making a lot of money – much more money than they ever thought possible. But that’s going to happen. 207 | 208 | Reporter: Your remarks today– 209 | 210 | TRUMP: And the other thing – very important – I believe wages will start going up. They haven’t gone up for a long time. I believe wages now – because the economy is doing so well with respect to employment and unemployment, I believe wages will start to go up. I think that will have a tremendously positive impact on race relations. 211 | -------------------------------------------------------------------------------- /bb104/amazon.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 27, 6 | "metadata": {}, 7 | "outputs": [ 8 | { 9 | "data": { 10 | "text/plain": [ 11 | "'\\n\\n\\n