├── .github └── FUNDING.yml ├── Machine Learning Assessment 12.pdf ├── README.md ├── appliedml.md ├── company_guides.md ├── course.md ├── design.md ├── extra.md ├── faqs.md ├── how.md ├── images ├── LC.png ├── ad_bidding.png ├── ads.png ├── air.png ├── delivery.png ├── feed.png ├── ml.png ├── nce.png ├── score.png ├── score1.png ├── sponsor.png ├── stat_cheatsheet.png ├── system_design.png └── uc3.png ├── interview_experiences.md ├── leetcode.md ├── questions.md ├── quiz.md ├── sample ├── backprop.py ├── kmeans.ipynb └── logistic_regression.ipynb └── terstimonials.png /.github/FUNDING.yml: -------------------------------------------------------------------------------- 1 | # These are supported funding model platforms 2 | 3 | github: khangich # Replace with up to 4 GitHub Sponsors-enabled usernames e.g., [user1, user2] 4 | patreon: # Replace with a single Patreon username 5 | open_collective: # Replace with a single Open Collective username 6 | ko_fi: # Replace with a single Ko-fi username 7 | tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel 8 | community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry 9 | liberapay: # Replace with a single Liberapay username 10 | issuehunt: # Replace with a single IssueHunt username 11 | otechie: # Replace with a single Otechie username 12 | custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2'] 13 | -------------------------------------------------------------------------------- /Machine Learning Assessment 12.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/khangich/machine-learning-interview/968a286650c964f8583b73b3e0864475b86f22b6/Machine Learning Assessment 12.pdf -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | # Minimum Viable Study Plan for Machine Learning Interviews 3 |

4 | Machine Learning System Design - Early Preview - Buy on Amazon
5 | 6 | 7 |

8 |

9 | Machine Learning interviews book on Amazon.
10 | 11 | 12 |

13 | 14 | Follow [News about AI projects](https://news.llmlab.io/) 15 | 16 | - Most popular post: [One lesson I learned after solving 500 leetcode questions](https://mlengineer.io/from-semiconductor-to-software-engineer-8c3126dde65b) 17 | - Oct 10th: Machine Learning System Design course became the [number 1 ML course](https://www.linkedin.com/posts/activity-6853724396188790784-tWxj) on educative. 18 | - June 8th: launch [interview stories series](https://rebrand.ly/interviewstory). 19 | - April 29th: I launched [mlengineer.io](https://mlengineer.io/from-google-rejection-to-40-offers-71337a224ebe?sk=1408513db21536d25c23f67ce898b37d) blog so you can get latest machine learning interview experience. 20 | - April 15th 2021: Machine Learning System Design is launched on [interviewquery.com](https://rebrand.ly/mldesigninterview). 21 | - Feb 9th 2021: [Machine Learning System design](https://rebrand.ly/mlsd_launch) is now available on [educative.io](https://rebrand.ly/mlsd_launch). 22 | - I'm a SWE, ML with 10 years of experience ([Linkedin profile](https://www.linkedin.com/in/phamkhang/)). I had offers from Google, LinkedIn, Coupang, Snap and StichFix. Read my [blog](https://rebrand.ly/mleio). 23 | 24 | 25 | 26 | ## Machine Learning Design 27 | 28 | | Section | | 29 | | ------------- | ------------- | 30 | | 1. [Youtube Recommendation](https://rebrand.ly/mldesign) |

| 31 | | 2. [The main components in MLSD](https://rebrand.ly/mldesign) |

| 32 | | 3. [LinkedIn Feed Ranking](https://rebrand.ly/mldesign) |

| 33 | | 4. [Ad Click Prediction](https://rebrand.ly/mldesign)|

| 34 | | 5. [Estimate Delivery time](https://rebrand.ly/mldesign)|

| 35 | | 6. [Airbnb Search ranking](https://rebrand.ly/mldesign)|

| 36 | 37 | 38 | ## Getting Started 39 | 40 | | How to | Resources | 41 | | ------------- | ------------- | 42 | | List of promising companies | [WealthFront 2021 list](https://blog.wealthfront.com/career-launching-companies-list/#companies-list). | 43 | | Prepare for interview | [Common questions about Machine Learning Interview process](faqs.md). | 44 | | Study guide | [Study guide](README.md) contained minimum set of focus area to aces your interview. | 45 | | Design ML system | [ML system design](https://mlengineer.io/machine-learning-design-interview-d08be9f44260?source=friends_link&sk=97fe3a510957d65b6311d5d38b30c639) includes actual ML system design usecases. | 46 | | ML usecases | [ML usecases](appliedml.md) from top companies | 47 | | Test your ML knowledge | [Machine Learning quiz](https://mlengineer.io/machine-learning-assessment-db935aa9fafd?source=friends_link&sk=1062e407bea5d842b7684668b005d08c) are designed based on actual interview questions from dozen of big companies. | 48 | | One week before onsite interview | Read [one week check list](https://mlengineer.io/machine-learning-engineer-onsite-interview-one-week-checklist-cfd19d57fa02?source=friends_link&sk=80d2bb43c590156a7fa72260dfb4972c) | 49 | | How to get offer? | Read [success stories](https://mlengineer.io/from-google-rejection-to-40-offers-71337a224ebe?source=friends_link&sk=1408513db21536d25c23f67ce898b37d) | 50 | | FAANG companies actual MLE interviews | Read [interview stories](https://mlengineer.io/mlengineer-io-interview/home) | 51 | | Practice coding | [Leetcode questions by categories for MLE](https://mlengineer.io/common-leetcode-questions-by-categories-532b301130b) | 52 | | Advance topics | Read [advance topics](extra.md) | 53 | 54 | 55 | 56 | ## Study guide 57 | ### LeetCode (not all companies ask Leetcode questions) 58 | [

](https://docs.google.com/spreadsheets/d/1RCb1dVQCLmtOGlJ5J-NJ5pIC7Tda-N2U/edit#gid=274831950) 59 | 60 | - NOTE: there are a lot of companies that do **NOT** ask leetcode questions. There are many paths to become an MLE, you can create your own path if you feel like leetcoding is a waste of time. 61 | 62 | - I use [LC time tracking](https://docs.google.com/spreadsheets/d/1RCb1dVQCLmtOGlJ5J-NJ5pIC7Tda-N2U/edit#gid=274831950) to keep track of how many times I solves a question and how long I spent each time. Once I finish non-trivial medium LC questions 3 times, I have absolutely no issues solving them in actual interviews (sometimes within 8-10 minutes). It makes a big difference. A better way is to use **LeetPlug** chrome extension [here](https://leetplug.azurewebsites.net/static/pages/howto.html) 63 | 64 | [Leetcode questions by categories](https://mlengineer.io/common-leetcode-questions-by-categories-532b301130b?sk=cf77975462cb0c96e6a6daebaa3ab7b9) 65 | ### SQL 66 | * Know SQL join: [self join](https://www.sqlservertutorial.net/sql-server-basics/sql-server-self-join/), inner, left, right etc. 67 | * Use [hackerrank](https://www.hackerrank.com/domains/sql) to practice SQL. 68 | * Revise/Learn SQL Window Functions: [window functions](https://www.windowfunctions.com/questions/intro/) 69 | 70 | ### Programming 71 | * [Java garbage collection](https://stackify.com/what-is-java-garbage-collection/#:~:text=Java%20garbage%20collection%20is%20the,Machine%2C%20or%20JVM%20for%20short.&text=The%20garbage%20collector%20finds%20these,them%20to%20free%20up%20memory.) 72 | * [Python pass-by-object-reference](https://robertheaton.com/2014/02/09/pythons-pass-by-object-reference-as-explained-by-philip-k-dick/) 73 | * [Python GIL, Fluent Python, chapter 17](http://index-of.es/Varios-2/Fluent%20Python%20Clear%20Concise%20and%20Effective%20Programming.pdf) 74 | * [Python multithread](https://realpython.com/intro-to-python-threading/) 75 | * [Python concurrency, Fluent Python, chapter 18](http://index-of.es/Varios-2/Fluent%20Python%20Clear%20Concise%20and%20Effective%20Programming.pdf) 76 | 77 | ### Statistics and probability 78 | 79 | * The only cheatsheet that you''ll ever need 80 | 81 | [

](http://www.wzchen.com/probability-cheatsheet) 82 | 83 | 84 | 85 | * Learn Bayesian and practice [problems in Bayesian](https://blogs.kent.ac.uk/jonw/files/2015/04/Puza2005.pdf) 86 | * Let A and B be events on the same sample space, with P (A) = 0.6 and P (B) = 0.7. Can these two events be disjoint? 87 | * Given that Alice has 2 kids, at least one of which is a girl, what is the probability that both kids are girls? (credit [swierdo](https://www.reddit.com/user/swierdo/)) 88 | * A group of 60 students is randomly split into 3 classes of equal size. All partitions are equally likely. Jack and Jill are two students belonging to that group. What is the probability that Jack and Jill will end up in the same class? 89 | * Given an unfair coin with the probability of heads not equal to .5. What algorithm could you use to create a list of random 1s and 0s. 90 | 91 | 92 | ### Big data (NOT required for Google, Facebook interview) 93 | * Spark [architecture](http://datastrophic.io/core-concepts-architecture-and-internals-of-apache-spark/) and Spark [lessons learned](https://databricks.com/blog/2016/08/31/apache-spark-scale-a-60-tb-production-use-case.html) (outdated since Spark 3.0 release) 94 | * Spark [OOM](https://stackoverflow.com/questions/21138751/spark-java-lang-outofmemoryerror-java-heap-space) 95 | * Cassandra [best practice](https://tech.ebayinc.com/engineering/cassandra-data-modeling-best-practices-part-1/) and [here](https://cassandra.apache.org/doc/latest/data_modeling/intro.html), [link](https://towardsdatascience.com/when-to-use-cassandra-and-when-to-steer-clear-72b7f2cede76![image](https://user-images.githubusercontent.com/1975237/109818900-5a25fc00-7be8-11eb-8715-ca7d279e84b6.png) 96 | ), [cassandra performance](https://www.scnsoft.com/blog/cassandra-performance) 97 | * Practice problem [finding friends with MapReduce](http://stevekrenzel.com/finding-friends-with-mapreduce) 98 | * Everything in [one page](https://mlengineer.io/big-data-knowledge-for-machine-learning-engineer-interview-2020-148d7c335e12?source=friends_link&sk=604c593c522db5195d3bda33dc4662d7). 99 | 100 | 101 | 102 | 103 | 104 | ### ML fundamentals 105 | * [Collinearity](https://statisticsbyjim.com/regression/multicollinearity-in-regression-analysis/) and [read more](https://www.youtube.com/watch?v=Cba9LJ9lS8s) 106 | * [Features scaling](https://sebastianraschka.com/Articles/2014_about_feature_scaling.html) 107 | * [Random forest vs GBDT](https://medium.com/@aravanshad/gradient-boosting-versus-random-forest-cfa3fa8f0d80) 108 | * [SMOTE synthetic minority over-sampling technique](https://arxiv.org/pdf/1106.1813.pdf) 109 | * [Compare discriminative vs generative model](https://medium.com/@mlengineer/generative-and-discriminative-models-af5637a66a3) and [extra read](http://ai.stanford.edu/~ang/papers/nips01-discriminativegenerative.pdf) 110 | * [Logistic regression](https://www.youtube.com/watch?v=-la3q9d7AKQ). Try to implement logistic regression from scratch. Bonus point for vectorized version in numpy + completed in 20 minutes [sample code from martinpella](sample/logistic_regression.ipynb). Followup with MapReduce version. 111 | * [Quantile regression](https://www.youtube.com/watch?v=s203ScTy4xQ&t=954s) 112 | * [L1/L2 intuition](https://www.linkedin.com/pulse/intuitive-visual-explanation-differences-between-l1-l2-xiaoli-chen/) 113 | * [Decision tree and Random Forest fundamental](https://people.csail.mit.edu/dsontag/courses/ml16/slides/lecture11.pdf) 114 | * [Explain boosting](https://web.stanford.edu/~hastie/TALKS/boost.pdf) 115 | * [Least Square as Maximum Likelihood Estimator](https://www.youtube.com/watch?v=_-Gnu498s3o) 116 | * [Maximum Likelihood Estimator introduction](https://www.youtube.com/watch?v=WflqTUOvdik&t=15s) 117 | * [Kmeans](https://stanford.edu/~cpiech/cs221/handouts/kmeans.html). Try to implement Kmeans from scratch [sample code from flothesof.github.io](sample/kmeans.ipynb). Bonus point for vectorized version in numpy + completed in 20 minutes. Follow-up with worst case time complexity and improvement for [initialization](extra.md). 118 | * Fundamentals about [PCA](http://alexhwilliams.info/itsneuronalblog/2016/03/27/pca/) 119 | * I didn't use [flashcard](https://machinelearningflashcards.com/) but I'm sure it helps up to certain extend. 120 | 121 | ### AB testing 122 | * [Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing](https://www.researchgate.net/publication/339914315_Trustworthy_Online_Controlled_Experiments_A_Practical_Guide_to_AB_Testing) 123 | 124 | 125 | ### DL fundamentals 126 | * [The deep learning book](https://www.deeplearningbook.org/). Read [Part ii](https://www.deeplearningbook.org/contents/part_practical.html) 127 | * [Machine Learning Yearning](https://d2wvfoqc9gyqzf.cloudfront.net/content/uploads/2018/09/Ng-MLY01-13.pdf). Read from section 5 to section 27. 128 | * [Neural network and backpropagation](http://cs231n.stanford.edu/slides/2020/lecture_4.pdf) 129 | * [Activation functions](https://missinglink.ai/guides/neural-network-concepts/7-types-neural-network-activation-functions-right/) 130 | * [Loss and optimization](http://cs231n.stanford.edu/slides/2020/lecture_3.pdf) 131 | * [Convolution Neural network notes](https://cs231n.github.io/convolutional-networks/) 132 | * [Recurrent Neural Networks](http://cs231n.stanford.edu/slides/2020/lecture_10.pdf) 133 | 134 | 135 | 136 | ### ML system design 137 | #### ML classic paper 138 | * [Technical debt in ML](https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf) 139 | * [Rules of ML](https://developers.google.com/machine-learning/guides/rules-of-ml) 140 | * [An Opinionated Guide to ML Research](http://joschu.net/blog/opinionated-guide-ml-research.html). There is valuable advice in the Personal development section at the bottom. 141 | 142 | #### ML productions 143 | * [Scaling ML at Uber](https://eng.uber.com/scaling-michelangelo/) 144 | * [DL in production](https://github.com/alirezadir/Production-Level-Deep-Learning) 145 | #### Food delivery 146 | * [Uber eats trip optimization](https://eng.uber.com/uber-eats-trip-optimization/) 147 | * [Uber food discovery](https://eng.uber.com/uber-eats-query-understanding/) 148 | * [Personalized store feed](https://blog.doordash.com/personalized-store-feed-with-vector-embeddings-251ad7a2c09a) 149 | * [Doordash dispatch optimization](https://doordash.engineering/2020/02/28/next-generation-optimization-for-dasher-dispatch-at-doordash/) 150 | 151 | #### ML design common usecases 152 | * [ML system design primer](https://interview.mlengineer.io/) 153 | * [Video recommendation](https://interview.mlengineer.io/) 154 | * [Feed ranking](https://interview.mlengineer.io/) 155 | 156 | 157 | #### Fraud detection (TBD) 158 | 159 | #### Adtech 160 | * [Ad click prediction trend](https://storage.googleapis.com/pub-tools-public-publication-data/pdf/41159.pdf) 161 | * [Ad Clicks CTR](https://research.fb.com/wp-content/uploads/2016/11/practical-lessons-from-predicting-clicks-on-ads-at-facebook.pdf) 162 | * [Delayed feedbacks](https://blog.twitter.com/engineering/en_us/topics/insights/2019/improving-engagement-on-digital-ads-with-delayed-feedback.html) 163 | * [Entity embedding](https://blog.twitter.com/engineering/en_us/topics/insights/2018/embeddingsattwitter.html) 164 | * [Star space, embedding all the things](https://arxiv.org/pdf/1709.03856.pdf) 165 | * [Twitter timeline ranking](https://blog.twitter.com/engineering/en_us/topics/insights/2017/using-deep-learning-at-scale-in-twitters-timelines.html) 166 | 167 | #### Recommendations: 168 | * [Instagram explore](https://ai.facebook.com/blog/powered-by-ai-instagrams-explore-recommender-system/) 169 | * [TikTok recommendation](https://newsroom.tiktok.com/en-us/how-tiktok-recommends-videos-for-you) 170 | * [Deep Neural Networks for YouTube Recommendations](https://storage.googleapis.com/pub-tools-public-publication-data/pdf/45530.pdf) 171 | * [Wide & Deep Learning for Recommender Systems](https://arxiv.org/pdf/1606.07792.pdf%29/) 172 | 173 | ## Testimonials 174 | - V, Amazon L5 DS 175 | > I really found the quizzes very helpful for testing my ML understanding. Also, the resources shared helped me a lot for revising concepts for my interview preparation. This course will definitely help engineers crack Machine Learning Engineering and Data Science interviews. 176 | 177 | 178 | - K, Facebook MLE 179 | > I really like what you've built, it'll help a lot of engineers. 180 | 181 | - D, NVIDIA DS 182 | > I have been using your github repo to prep for my interviews and got an offer with NVIDIA with their data science team. Thanks again for your help! 183 | 184 | - A, Booking 185 | > Woow this is very useful summaries, so nice. 186 | 187 | - H, Microsoft 188 | > That's incredible! 189 | 190 | - V, Intel 191 | > The repo is extremely cohesive! Thanks again. 192 | 193 | 194 | ## Intro 195 | 196 | * This repo is written based on REAL interview questions from big companies and the study materials are based on legit experts i.e Andrew Ng, Yoshua Bengio etc. 197 | 198 | * I have 6 YOE in Machine Learning and have interviewed more than dozen big companies. This is the **minimum** viable study plan that covers all actual interview questions from Facebook, Amazon, Apple, Google, MS, SnapChat, Linkedin etc. 199 | 200 | * If you're interested to learn more about paid ML system design course, [click here](course.md). This course will provide 6-7 practical usecases with proven solutions. After this course you will be able to solve new problem with systematic approach. 201 | 202 | 203 | # Acknowledgements and contributing 204 | 1. Thanks for early feedbacks and contributions from [Vivian](https://github.com/liuvivian11), [aragorn87](https://github.com/aragorn87) and others. You can create an Issue or Pull Request on this repo. You can also help upvote on [ProductHunt](https://www.producthunt.com/posts/machine-learning-interview-guideline) 205 | 206 | 2. If you find this helpful, you can Sponsor this project. It's cool if you don't. 207 | 208 | 3. Thanks to this community, we have donated about $200 to [HopeForPaws](https://www.hopeforpaws.org/). If you want to support, you can contribute too on their website. 209 | 210 | 211 | 212 | -------------------------------------------------------------------------------- /appliedml.md: -------------------------------------------------------------------------------- 1 | 2 | # ML use cases by company ([source](https://github.com/eugeneyan/applied-ml)) 3 | 4 | Read more about ML system design on [ML interview](https://rebrand.ly/interviewml), [educative.io](https://rebrand.ly/mlsd_launch) or [interviewquery.com](https://rebrand.ly/mldesigninterview). For mock interview, click [here](https://rebrand.ly/mlsdmock) 5 | 6 | **Companies** 7 | 8 | 1. [Alibaba](#Alibaba) 9 | 2. [AirBnB](#AirBnb) 10 | 3. [Amazon](#Amazon) 11 | 4. [Apple](#Apple) 12 | 5. [Booking](#Booking) 13 | 6. [ByteDance](#ByteDance) 14 | 7. [Doordash](#Doordash) 15 | 8. [Dropbox](#Dropbox) 16 | 9. [Facebook](#Facebook) 17 | 10. [GoJek](#Gojek) 18 | 11. [Google](#Google) 19 | 12. [Instagram](#Instagram) 20 | 13. [Linkedin](#Linkedin) 21 | 14. [Lyft](#Lyft) 22 | 15. [Microsoft](#Microsoft) 23 | 16. [Netflix](#Netflix) 24 | 17. [Spotify](#Spotify) 25 | 18. [Shopify](#Shopify) 26 | 19. [StitchFix](#StitchFix) 27 | 20. [Pinterest](#Pinterest) 28 | 21. [Uber](#Uber) 29 | 22. [OpenAI](#Openai) 30 | 23. [Tesla](#Tesla) 31 | 24. [Twitter](#Twitter) 32 | 25. [Yahoo](#yahoo) 33 | 26. [Youtube](#youtube) 34 | 27. [Other companies](#Others) 35 | 36 | ## Alibaba 37 | 1. [Recommending Complementary Products in E-Commerce Push Notifications](https://arxiv.org/abs/1707.08113) ([Paper](https://arxiv.org/pdf/1707.08113.pdf)) 38 | 2. [Behavior Sequence Transformer for E-commerce Recommendation in Alibaba](https://arxiv.org/abs/1905.06874) ([Paper](https://arxiv.org/pdf/1905.06874.pdf)) 39 | 3. [TPG-DNN: A Method for User Intent Prediction with Multi-task Learning](https://arxiv.org/abs/2008.02122) ([Paper](https://arxiv.org/pdf/2008.02122.pdf)) 40 | 4. [COLD: Towards the Next Generation of Pre-Ranking System](https://arxiv.org/abs/2007.16122) ([Paper](https://arxiv.org/pdf/2007.16122.pdf)) 41 | 5. [Billion-scale Commodity Embedding for E-commerce Recommendation in Alibaba](https://arxiv.org/abs/1803.02349) ([Paper](https://arxiv.org/pdf/1803.02349.pdf)) 42 | 6. [Practice on Long Sequential User Behavior Modeling for Click-Through Rate Prediction](https://arxiv.org/abs/1905.09248) ([Paper](https://arxiv.org/pdf/1905.09248.pdf)) 43 | 7. [Search-based User Interest Modeling with Sequential Behavior Data for CTR Prediction](https://arxiv.org/abs/2006.05639) ([Paper](https://arxiv.org/pdf/2006.05639.pdf)) 44 | 8. [Deep Reinforcement Learning for Sponsored Search Real-time Bidding](https://arxiv.org/abs/1803.00259) ([Paper](https://arxiv.org/pdf/1803.00259.pdf)) 45 | 9. [Dynamic Pricing on E-commerce Platform with Deep Reinforcement Learning](https://arxiv.org/abs/1912.02572) ([Paper](https://arxiv.org/pdf/1912.02572.pdf)) 46 | 20. [Budget Constrained Bidding by Model-free Reinforcement Learning in Display Advertising](https://arxiv.org/abs/1802.08365) ([Paper](https://arxiv.org/pdf/1802.08365.pdf)) 47 | 11. [AliGraph: A Comprehensive Graph Neural Network Platform](https://arxiv.org/abs/1902.08730) ([Paper](https://arxiv.org/pdf/1902.08730.pdf)) 48 | 12. [One-shot Text Labeling using Attention and Belief Propagation for Information Extraction](https://arxiv.org/abs/2009.04153) ([Paper](https://arxiv.org/pdf/2009.04153.pdf)) 49 | 50 | ## AirBnb 51 | 1. [Discovering and Classifying In-app Message Intent at Airbnb](https://medium.com/airbnb-engineering/discovering-and-classifying-in-app-message-intent-at-airbnb-6a55f5400a0c) 52 | 2. [Using Machine Learning to Predict Value of Homes On Airbnb](https://medium.com/airbnb-engineering/using-machine-learning-to-predict-value-of-homes-on-airbnb-9272d3d4739d) 53 | 3. [Applying Deep Learning To Airbnb Search](https://arxiv.org/abs/1810.09591) ([Paper](https://arxiv.org/pdf/1810.09591.pdf)) 54 | 4. [Managing Diversity in Airbnb Search](https://arxiv.org/abs/2004.02621) ([Paper](https://arxiv.org/pdf/2004.02621.pdf)) 55 | 5. [Machine Learning-Powered Search Ranking of Airbnb Experiences](https://medium.com/airbnb-engineering/machine-learning-powered-search-ranking-of-airbnb-experiences-110b4b1a0789) 56 | 7. [Categorizing Listing Photos at Airbnb](https://medium.com/airbnb-engineering/categorizing-listing-photos-at-airbnb-f9483f3ab7e3) 57 | 8. [Amenity Detection and Beyond — New Frontiers of Computer Vision at Airbnb](https://medium.com/airbnb-engineering/amenity-detection-and-beyond-new-frontiers-of-computer-vision-at-airbnb-144a4441b72e) 58 | 9. [Scaling Knowledge Access and Retrieval at Airbnb](https://medium.com/airbnb-engineering/scaling-knowledge-access-and-retrieval-at-airbnb-665b6ba21e95) 59 | 10. [Optimal Pricing](https://medium.com/airbnb-engineering/learning-market-dynamics-for-optimal-pricing-97cffbcc53e3) 60 | 11. [Apply DL to airbnb search](https://medium.com/airbnb-engineering/applying-deep-learning-to-airbnb-search-7ebd7230891f) 61 | 12. [Forecasting platform](https://medium.com/@jerry.chu/airbnbs-data-platform-of-revenue-forecasting-2e95a01122e6) 62 | 63 | 64 | ## Amazon 65 | 1. [Amazon.com Recommendations: Item-to-Item Collaborative Filtering](https://ieeexplore.ieee.org/document/1167344) ([Paper](https://www.cs.umd.edu/~samir/498/Amazon-Recommendations.pdf)) 66 | 2. [Temporal-Contextual Recommendation in Real-Time](https://www.amazon.science/publications/temporal-contextual-recommendation-in-real-time) ([Paper](https://assets.amazon.science/96/71/d1f25754497681133c7aa2b7eb05/temporal-contextual-recommendation-in-real-time.pdf)) 67 | 3. [Amazon Search: The Joy of Ranking Products](https://www.amazon.science/publications/amazon-search-the-joy-of-ranking-products) ([Paper](https://assets.amazon.science/89/cd/34289f1f4d25b5857d776bdf04d5/amazon-search-the-joy-of-ranking-products.pdf), [Video](https://www.youtube.com/watch?v=NLrhmn-EZ88), [Code](https://github.com/dariasor/TreeExtra)) 68 | 4. [Why Do People Buy Seemingly Irrelevant Items in Voice Product Search?](https://www.amazon.science/publications/why-do-people-buy-irrelevant-items-in-voice-product-search) ([Paper](https://assets.amazon.science/f7/48/0562b2c14338a0b76ccf4f523fa5/why-do-people-buy-irrelevant-items-in-voice-product-search.pdf)) 69 | 5. [Goal-Oriented End-to-End Conversational Models with Profile Features in a Real-World Setting](https://www.amazon.science/publications/goal-oriented-end-to-end-chatbots-with-profile-features-in-a-real-world-setting) ([Paper](https://assets.amazon.science/47/03/e0d14dc34d3eb6e0d4ec282067bd/goal-oriented-end-to-end-chatbots-with-profile-features-in-a-real-world-setting.pdf)) 70 | 6. [Converting Text to Images for Product Discovery](https://www.amazon.science/blog/converting-text-to-images-for-product-discovery) ([Paper](https://assets.amazon.science/4c/76/5830542547b7a11089ce3af943b4/scipub-972.pdf)) 71 | 7. [AutoKnow: self-driving knowledge collection for products of thousands of types](https://www.amazon.science/publications/autoknow-self-driving-knowledge-collection-for-products-of-thousands-of-types) ([Paper](https://arxiv.org/pdf/2006.13473.pdf), [Video](https://crossminds.ai/video/5f3369730576dd25aef288a6/)) 72 | 8. [On Challenges in Machine Learning Model Management](http://sites.computer.org/debull/A18dec/p5.pdf) 73 | 74 | 75 | ## Apple 76 | 1. [Overton: A Data System for Monitoring and Improving Machine-Learned Products](https://arxiv.org/abs/1909.05372) ([Paper](https://arxiv.org/pdf/1909.05372.pdf)) 77 | 78 | ## Booking 79 | 1. [Machine Learning in Production: The Booking.com Approach](https://booking.ai/https-booking-ai-machine-learning-production-3ee8fe943c70) 80 | 2. [150 Successful Machine Learning Models: 6 Lessons Learned at Booking.com](https://www.kdd.org/kdd2019/accepted-papers/view/150-successful-machine-learning-models-6-lessons-learned-at-booking.com) ([Paper](https://dl.acm.org/doi/pdf/10.1145/3292500.3330744)) 81 | 82 | ## ByteDance 83 | 1. [How TikTok recommends videos #ForYou](https://newsroom.tiktok.com/en-us/how-tiktok-recommends-videos-for-you) 84 | 2. [Deep Retrieval: End-to-End Learnable Structure Model for Large-Scale Recommendations](https://arxiv.org/abs/2007.07203) ([Paper](https://arxiv.org/pdf/2007.07203.pdf)) 85 | 86 | ## Doordash 87 | 1. [Next-Generation Optimization for Dasher Dispatch at DoorDash](https://doordash.engineering/2020/02/28/next-generation-optimization-for-dasher-dispatch-at-doordash/) 88 | 2. [Retraining Machine Learning Models in the Wake of COVID-19](https://doordash.engineering/2020/09/15/retraining-ml-models-covid-19/) 89 | 3. [Supporting Rapid Product Iteration with an Experimentation Analysis Platform](https://doordash.engineering/2020/09/09/experimentation-analysis-platform-mvp/) 90 | 4. https://doordash.engineering/2021/04/28/improving-eta-prediction-accuracy-for-long-tail-events/ 91 | 5. https://heartbeat.comet.ml/5-regression-loss-functions-all-machine-learners-should-know-4fb140e9d4b0 92 | 93 | ## Dropbox 94 | 1. [Using Machine Learning to Predict what File you Need Next (Part 1)](https://dropbox.tech/machine-learning/content-suggestions-machine-learning) 95 | 2. [Using Machine Learning to Predict what File you Need Next (Part 2)](https://dropbox.tech/machine-learning/using-machine-learning-to-predict-what-file-you-need-next-part-2) 96 | 3. [Creating a Modern OCR Pipeline Using Computer Vision and Deep Learning](https://dropbox.tech/machine-learning/creating-a-modern-ocr-pipeline-using-computer-vision-and-deep-learning) 97 | 4. [Using Machine Learning to Index Text from Billions of Images](https://dropbox.tech/machine-learning/using-machine-learning-to-index-text-from-billions-of-images) 98 | 99 | ## eBay 100 | 1. [Large-scale Item Categorization for e-Commerce](https://dl.acm.org/doi/10.1145/2396761.2396838) ([Paper](https://www.researchgate.net/profile/Jean_David_Ruvini/publication/262270957_Large-scale_item_categorization_for_e-commerce/links/5512dc3d0cf270fd7e33a0d5/Large-scale-item-categorization-for-e-commerce.pdf)) 101 | 102 | 103 | ## Facebook 104 | 1. [Powered by AI: Instagram’s Explore recommender system](https://ai.facebook.com/blog/powered-by-ai-instagrams-explore-recommender-system/) 105 | 2. [Neural Code Search: ML-based Code Search Using Natural Language Queries](https://ai.facebook.com/blog/neural-code-search-ml-based-code-search-using-natural-language-queries/) 106 | 3. [AI Advances to Better Detect Hate Speech](https://ai.facebook.com/blog/ai-advances-to-better-detect-hate-speech/) 107 | 4. [A State-of-the-Art Open Source Chatbot](https://ai.facebook.com/blog/state-of-the-art-open-source-chatbot) ([Paper](https://arxiv.org/pdf/2004.13637.pdf)) 108 | 5. [A Highly Efficient, Real-Time Text-to-Speech System Deployed on CPUs](https://ai.facebook.com/blog/a-highly-efficient-real-time-text-to-speech-system-deployed-on-cpus/) 109 | 6. [Deep Learning to Translate Between Programming Languages](https://ai.facebook.com/blog/deep-learning-to-translate-between-programming-languages/) ([Paper](https://arxiv.org/abs/2006.03511), [Code](https://github.com/facebookresearch/TransCoder)) 110 | 7. [Deploying Lifelong Open-Domain Dialogue Learning](https://arxiv.org/abs/2008.08076) ([Paper](https://arxiv.org/pdf/2008.08076.pdf)) 111 | 8. [Leveraging Online Social Interactions For Enhancing Integrity at Facebook](https://research.fb.com/blog/2020/08/leveraging-online-social-interactions-for-enhancing-integrity-at-facebook/) ([Paper](https://research.fb.com/wp-content/uploads/2020/08/TIES-Temporal-Interaction-Embeddings-For-Enhancing-Social-Media-Integrity-At-Facebook.pdf), [Video](https://crossminds.ai/video/5f3369780576dd25aef288cf/)) 112 | 9. [Scalable Data Classification for Security and Privacy](https://engineering.fb.com/security/data-classification-system/) ([Paper](https://arxiv.org/pdf/2006.14109.pdf)) 113 | 10. [Powered by AI: Advancing product understanding and building new shopping experiences](https://ai.facebook.com/blog/powered-by-ai-advancing-product-understanding-and-building-new-shopping-experiences/) 114 | 11. [GrokNet: Unified Computer Vision Model Trunk and Embeddings For Commerce](https://ai.facebook.com/research/publications/groknet-unified-computer-vision-model-trunk-and-embeddings-for-commerce/) ([Paper](https://scontent-sea1-1.xx.fbcdn.net/v/t39.8562-6/99353320_565175057533429_3886205100842024960_n.pdf?_nc_cat=110&_nc_sid=ae5e01&_nc_ohc=WQBaZy1gnmUAX8Ecqtt&_nc_ht=scontent-sea1-1.xx&oh=cab2f11dd9154d817149cb73e8b692a8&oe=5F5A3778)) 115 | 12. [Constrained Bayesian Optimization with Noisy Experiments](https://research.fb.com/publications/constrained-bayesian-optimization-with-noisy-experiments/) ([Paper](https://arxiv.org/pdf/1706.07094.pdf)) 116 | 13. [Practical lesson in Ad Click Prediction](https://research.fb.com/wp-content/uploads/2016/11/practical-lessons-from-predicting-clicks-on-ads-at-facebook.pdf) 117 | 14. [Roberta](https://arxiv.org/pdf/1907.11692.pdf) 118 | 15. [StarSpace embedding](https://arxiv.org/pdf/1709.03856.pdf) 119 | 16. [XML paper](https://arxiv.org/pdf/1911.01464.pdf) 120 | 17. [Unsupervised XLM](https://arxiv.org/pdf/1911.02116.pdf) 121 | 18. [Hate Speech](https://ai.facebook.com/blog/ai-advances-to-better-detect-hate-speech/) 122 | 19. ([DL recommendation model](https://ai.facebook.com/blog/dlrm-an-advanced-open-source-deep-learning-recommendation-model/) and https://arxiv.org/pdf/1906.00091.pdf 123 | ## GoJek 124 | 1. [The Secret Sauce Behind Search Personalisation](https://blog.gojekengineering.com/the-secret-sauce-behind-search-personalisation-a856fb83c2f) 125 | 2. [How Gojek Uses NLP to Name Pickup Locations at Scale](https://blog.gojekengineering.com/how-gojek-uses-nlp-to-name-pickup-locations-at-scale-ffdb249d1433) 126 | 3. [Under the Hood of Gojek’s Automated Forecasting Tool](https://blog.gojekengineering.com/under-the-hood-of-gojeks-automated-forecasting-tool-3ddb24ad29f1) 127 | 128 | ## Google 129 | 1. [Learning to Diagnose with LSTM Recurrent Neural Networks](https://arxiv.org/abs/1511.03677) ([Paper](https://arxiv.org/pdf/1511.03677.pdf)) 130 | 2. [Prediction of Advertiser Churn for Google AdWords](https://research.google/pubs/pub36678/) ([Paper](https://storage.googleapis.com/pub-tools-public-publication-data/pdf/36678.pdf)) 131 | 3. [BusTr: Predicting Bus Travel Times from Real-Time Traffic](https://dl.acm.org/doi/abs/10.1145/3394486.3403376) ([Paper](https://dl.acm.org/doi/pdf/10.1145/3394486.3403376), [Video](https://crossminds.ai/video/5f3369790576dd25aef288db/)) 132 | 4. [Zero-Shot Heterogeneous Transfer Learning from RecSys to Cold-Start Search Retrieval](https://arxiv.org/abs/2008.02930) ([Paper](https://arxiv.org/pdf/2008.02930.pdf)) 133 | 5. [Improved Deep & Cross Network for Feature Cross Learning in Web-scale LTR Systems](https://arxiv.org/abs/2008.13535) ([Paper](https://arxiv.org/pdf/2008.13535.pdf)) 134 | 6. [Understanding Searches Better Than Ever Before](https://www.blog.google/products/search/search-language-understanding-bert/) ([Paper](https://arxiv.org/pdf/1810.04805.pdf)) 135 | 7. [Announcing ScaNN: Efficient Vector Similarity Search](https://ai.googleblog.com/2020/07/announcing-scann-efficient-vector.html) ([Paper](https://arxiv.org/pdf/1908.10396.pdf), [Code](https://github.com/google-research/google-research/tree/master/scann)) 136 | 8. [Smart Reply: Automated Response Suggestion for Email](https://research.google/pubs/pub45189/) ([Paper](https://storage.googleapis.com/pub-tools-public-publication-data/pdf/45189.pdf)) 137 | 9. [Gmail Smart Compose: Real-Time Assisted Writing](https://arxiv.org/abs/1906.00080) ([Paper](https://arxiv.org/pdf/1906.00080.pdf)) 138 | 10. [SmartReply for YouTube Creators](https://ai.googleblog.com/2020/07/smartreply-for-youtube-creators.html) 139 | 12. [Using Neural Networks to Find Answers in Tables](https://ai.googleblog.com/2020/04/using-neural-networks-to-find-answers.html) ([Paper](https://arxiv.org/pdf/2004.02349.pdf)) 140 | 13. [A Scalable Approach to Reducing Gender Bias in Google Translate](https://ai.googleblog.com/2020/04/a-scalable-approach-to-reducing-gender.html) 141 | 14. [PEGASUS: A State-of-the-Art Model for Abstractive Text Summarization](https://ai.googleblog.com/2020/06/pegasus-state-of-art-model-for.html) ([Paper](https://arxiv.org/pdf/1912.08777.pdf), [Code](https://github.com/google-research/pegasus)) 142 | 15. [Deep Learning for Electronic Health Records](https://ai.googleblog.com/2018/05/deep-learning-for-electronic-health.html) ([Paper](https://www.nature.com/articles/s41746-018-0029-1.pdf)) 143 | 16. [On-device Supermarket Product Recognition](https://ai.googleblog.com/2020/07/on-device-supermarket-product.html) 144 | 17. [Using Machine Learning to Detect Deficient Coverage in Colonoscopy Screenings](https://ai.googleblog.com/2020/08/using-machine-learning-to-detect.html) ([Paper](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9097918)) 145 | 18. [A Neural Weather Model for Eight-Hour Precipitation Forecasting](https://ai.googleblog.com/2020/03/a-neural-weather-model-for-eight-hour.html) ([Paper](https://arxiv.org/pdf/2003.12140.pdf)) 146 | 19. [Machine Learning-based Damage Assessment for Disaster Relief](https://ai.googleblog.com/2020/06/machine-learning-based-damage.html) ([Paper](https://arxiv.org/pdf/1910.06444.pdf)) 147 | 20. [RepNet: Counting Repetitions in Videos](https://ai.googleblog.com/2020/06/repnet-counting-repetitions-in-videos.html) ([Paper](https://openaccess.thecvf.com/content_CVPR_2020/papers/Dwibedi_Counting_Out_Time_Class_Agnostic_Video_Repetition_Counting_in_the_CVPR_2020_paper.pdf)) 148 | 21. [The Reusable Holdout: Preserving Validity in Adaptive Data Analysis](https://ai.googleblog.com/2015/08/the-reusable-holdout-preserving.html) ([Paper](https://science.sciencemag.org/content/sci/349/6248/636.full.pdf)) 149 | 22. [Extracting Structured Data from Templatic Documents](https://ai.googleblog.com/2020/06/extracting-structured-data-from.html) ([Paper](https://www.aclweb.org/anthology/I13-1190.pdf)) 150 | 23. [Machine Learning: The High Interest Credit Card of Technical Debt](https://research.google/pubs/pub43146/) ([Paper](https://storage.googleapis.com/pub-tools-public-publication-data/pdf/43146.pdf)) ([Paper](https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf)) 151 | 24. [Rules of Machine Learning: Best Practices for ML Engineering](https://developers.google.com/machine-learning/guides/rules-of-ml) 152 | 25. [When It Comes to Gorillas, Google Photos Remains Blind](https://www.wired.com/story/when-it-comes-to-gorillas-google-photos-remains-blind/) 153 | 26. [Snorkel DryBell: A Case Study in Deploying Weak Supervision at Industrial Scale](https://dl.acm.org/doi/abs/10.1145/3299869.3314036) ([Paper](https://dl.acm.org/doi/pdf/10.1145/3299869.3314036)) 154 | 27. [Traffic Prediction with Advanced Graph Neural Networks](https://deepmind.com/blog/article/traffic-prediction-with-advanced-graph-neural-networks) 155 | 28. [DL recommendation model](https://ai.facebook.com/blog/dlrm-an-advanced-open-source-deep-learning-recommendation-model/ 156 | 157 | ## Instagram 158 | 1. [Lessons learned](https://instagram-engineering.com/lessons-learned-at-instagram-stories-and-feed-machine-learning-54f3aaa09e56) 159 | 2. [Feeds ranking](https://ai.facebook.com/blog/powered-by-ai-instagrams-explore-recommender-system/) 160 | 161 | ## Linkedin 162 | 1. [High-Precision Phrase-Based Document Classification on a Modern Scale](https://engineering.linkedin.com/research/2011/high-precision-phrase-based-document-classification-on-a-modern-scale) ([Paper](http://web.stanford.edu/~gavish/documents/phrase_based.pdf)) 163 | 2. [Personalized Recommendations in LinkedIn Learning](https://engineering.linkedin.com/blog/2016/12/personalized-recommendations-in-linkedin-learning) 164 | 3. [A Closer Look at the AI Behind Course Recommendations on LinkedIn Learning (Part 1)](https://engineering.linkedin.com/blog/2020/course-recommendations-ai-part-one) 165 | 4. [A Closer Look at the AI Behind Course Recommendations on LinkedIn Learning (Part 2)](https://engineering.linkedin.com/blog/2020/course-recommendations-ai-part-two) 166 | 5. [Learning to be Relevant: Evolution of a Course Recommendation System](https://dl.acm.org/doi/pdf/10.1145/3357384.3357817) 167 | 6. [Learning to Rank Personalized Search Results in Professional Networks](https://arxiv.org/abs/1605.04624) ([Paper](https://arxiv.org/pdf/1605.04624.pdf)) 168 | 7. [Entity Personalized Talent Search Models with Tree Interaction Features](https://arxiv.org/abs/1902.09041) ([Paper](https://arxiv.org/pdf/1902.09041.pdf)) 169 | 8. [In-session Personalization for Talent Search](https://arxiv.org/abs/1809.06488) ([Paper](https://arxiv.org/pdf/1809.06488.pdf)) 170 | 9. [The AI Behind LinkedIn Recruiter search and recommendation systems](https://engineering.linkedin.com/blog/2019/04/ai-behind-linkedin-recruiter-search-and-recommendation-systems) 171 | 10. [Quality Matches Via Personalized AI for Hirer and Seeker Preferences](https://engineering.linkedin.com/blog/2020/quality-matches-via-personalized-ai) 172 | 11. [Understanding Dwell Time to Improve LinkedIn Feed Ranking](https://engineering.linkedin.com/blog/2020/understanding-feed-dwell-time) 173 | 12. [Ads Allocation in Feed via Constrained Optimization](https://dl.acm.org/doi/abs/10.1145/3394486.3403391) ([Paper](https://dl.acm.org/doi/pdf/10.1145/3394486.3403391), [Video](https://crossminds.ai/video/5f33697a0576dd25aef288ea/)) 174 | 13. [Towards Deep and Representation Learning for Talent Search at LinkedIn](https://arxiv.org/abs/1809.06473) ([Paper](https://arxiv.org/pdf/1809.06473.pdf)) 175 | 14. [How Natural Language Processing Helps LinkedIn Members Get Support Easily](https://engineering.linkedin.com/blog/2019/04/how-natural-language-processing-help-support) 176 | 15. [Building Smart Replies for Member Messages](https://engineering.linkedin.com/blog/2017/10/building-smart-replies-for-member-messages) 177 | 16. [DeText: A deep NLP Framework for Intelligent Text Understanding](https://engineering.linkedin.com/blog/2020/open-sourcing-detext) ([Code](https://github.com/linkedin/detext)) 178 | 17. [Detecting and Preventing Abuse on LinkedIn using Isolation Forests](https://engineering.linkedin.com/blog/2019/isolation-forest) ([Code](https://github.com/linkedin/isolation-forest)) 179 | 18. [Preventing Abuse Using Unsupervised Learning](https://databricks.com/session_na20/preventing-abuse-using-unsupervised-learning) 180 | 19. [The Technology Behind Fighting Harassment on LinkedIn](https://engineering.linkedin.com/blog/2020/fighting-harassment) 181 | 20. [Building The LinkedIn Knowledge Graph](https://engineering.linkedin.com/blog/2016/10/building-the-linkedin-knowledge-graph) 182 | 21. [Detecting Interference: An A/B Test of A/B Tests](https://engineering.linkedin.com/blog/2019/06/detecting-interference--an-a-b-test-of-a-b-tests) 183 | 22. [Our evolution towards T-REX: The prehistory of experimentation infrastructure at LinkedIn](https://engineering.linkedin.com/blog/2020/our-evolution-towards-t-rex--the-prehistory-of-experimentation-i) 184 | 23. [Building Inclusive Products Through A/B Testing](https://engineering.linkedin.com/blog/2020/building-inclusive-products-through-a-b-testing) ([Paper](https://arxiv.org/pdf/2002.05819.pdf)) 185 | 24. [LiFT: A Scalable Framework for Measuring Fairness in ML Applications](https://engineering.linkedin.com/blog/2020/lift-addressing-bias-in-large-scale-ai-applications) ([Paper](https://arxiv.org/pdf/2008.07433.pdf)) 186 | 25. https://www.slideshare.net/QiGuo19/talent-search-and-recommendation-systems-at-linkedin-practical-challenges-and-lessons-learned-127365935 187 | 26. https://www.linkedin.com/in/souvix/detail/treasury/position:287530671/?entityUrn=urn%3Ali%3Afsd_profileTreasuryMedia%3A(ACoAAAYyI0MBDXEAGeQTiwqWQZyI48Yk682t5wE%2C50828921)§ion=position%3A287530671&treasuryCount=2 188 | 27. [Personalization](https://engineering.linkedin.com/blog/2020/quality-matches-via-personalized-ai) 189 | 28. [Course recommendation](https://engineering.linkedin.com/blog/2020/course-recommendations-ai-part-one) 190 | 29. [Fairness toolkit](https://engineering.linkedin.com/blog/2021/using-the-linkedin-fairness-toolkit-large-scale-ai) 191 | 30. [Embedding feature platform](https://engineering.linkedin.com/blog/2020/pensieve) 192 | 31. [recrutier search and recommendation](https://engineering.linkedin.com/blog/2019/04/ai-behind-linkedin-recruiter-search-and-recommendation-systems) 193 | 194 | ## Lyft 195 | 1. [Matchmaking in Lyft Line (Part 1)](https://eng.lyft.com/matchmaking-in-lyft-line-9c2635fe62c4) [(Part 2)](https://eng.lyft.com/matchmaking-in-lyft-line-691a1a32a008) [(Part 3)](https://eng.lyft.com/matchmaking-in-lyft-line-part-3-d8f9497c0e51) 196 | 197 | ## Microsoft 198 | 1. [Using ML to Subtype Patients Receiving Digital Mental Health Interventions](https://www.microsoft.com/en-us/research/blog/a-path-to-personalization-using-ml-to-subtype-patients-receiving-digital-mental-health-interventions/) ([Paper](https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2768347)) 199 | 2. [AI at Scale in Bing](https://blogs.bing.com/search/2020_05/AI-at-Scale-in-Bing) 200 | 3. [Assistive AI Makes Replying Easier](https://www.microsoft.com/en-us/research/group/msai/articles/assistive-ai-makes-replying-easier-2/) 201 | 4. [Unit Test Case Generation with Transformers](https://arxiv.org/pdf/2009.05617.pdf) 202 | 203 | 204 | ## Netflix 205 | 1. [Netflix Recommendations: Beyond the 5 stars (Part 1](https://netflixtechblog.com/netflix-recommendations-beyond-the-5-stars-part-1-55838468f429) ([Part 2](https://netflixtechblog.com/netflix-recommendations-beyond-the-5-stars-part-2-d9b96aa399f5)) 206 | 2. [Learning a Personalized Homepage](https://netflixtechblog.com/learning-a-personalized-homepage-aa8ec670359a) 207 | 3. [Artwork Personalization at Netflix](https://netflixtechblog.com/artwork-personalization-c589f074ad76) 208 | 4. [To Be Continued: Helping you find shows to continue watching on Netflix](https://netflixtechblog.com/to-be-continued-helping-you-find-shows-to-continue-watching-on-7c0d8ee4dab6) 209 | 5. [Calibrated Recommendations](https://dl.acm.org/doi/10.1145/3240323.3240372) ([Paper](https://dl.acm.org/doi/pdf/10.1145/3240323.3240372)) 210 | 6. [Open-Sourcing Riskquant, a Library for Quantifying Risk](https://netflixtechblog.com/open-sourcing-riskquant-a-library-for-quantifying-risk-6720cc1e4968) ([Code](https://github.com/Netflix-Skunkworks/riskquant)) 211 | 7. [Computational Causal Inference at Netflix](https://netflixtechblog.com/computational-causal-inference-at-netflix-293591691c62) ([Paper](https://arxiv.org/pdf/2007.10979.pdf)) 212 | 8. [Key Challenges with Quasi Experiments at Netflix](https://netflixtechblog.com/key-challenges-with-quasi-experiments-at-netflix-89b4f234b852) 213 | 9. [Machine Learning for a Better Developer Experience](https://netflixtechblog.com/machine-learning-for-a-better-developer-experience-1e600c69f36c) 214 | 10. [Detecting Performance Anomalies in External Firmware Deployments](https://netflixtechblog.com/detecting-performance-anomalies-in-external-firmware-deployments-ed41b1bfcf46) 215 | 11. [Runway - Model Lifecycle Management at Netflix](https://www.usenix.org/conference/opml20/presentation/cepoi) 216 | 217 | ## Spotify 218 | 1. [How Music Recommendation Works — And Doesn’t Work](https://notes.variogr.am/2012/12/11/how-music-recommendation-works-and-doesnt-work/) 219 | 2. [Music recommendation at Spotify](http://sigir.org/afirm2019/slides/16.%20Friday%20-%20Music%20Recommendation%20at%20Spotify%20-%20Ben%20Carterette.pdf) 220 | 3. [Recommending Music on Spotify with Deep Learning](https://benanne.github.io/2014/08/05/spotify-cnns.html) 221 | 4. [For Your Ears Only: Personalizing Spotify Home with Machine Learning](https://engineering.atspotify.com/2020/01/16/for-your-ears-only-personalizing-spotify-home-with-machine-learning/) 222 | 5. [Reach for the Top: How Spotify Built Shortcuts in Just Six Months](https://engineering.atspotify.com/2020/04/15/reach-for-the-top-how-spotify-built-shortcuts-in-just-six-months/) 223 | 6. [Explore, Exploit, and Explain: Personalizing Explainable Recommendations with Bandits](https://dl.acm.org/doi/10.1145/3240323.3240354) ([Paper](https://static1.squarespace.com/static/5ae0d0b48ab7227d232c2bea/t/5ba849e3c83025fa56814f45/1537755637453/BartRecSys.pdf)) 224 | 225 | 226 | ## Shopify 227 | 1. [The Evolution of Kit: Automating Marketing Using Machine Learning](https://engineering.shopify.com/blogs/engineering/evolution-kit-automating-marketing-machine-learning) 228 | 2. [Categorizing Products at Scale](https://engineering.shopify.com/blogs/engineering/categorizing-products-at-scale) 229 | 3. [How to Use Quasi-experiments and Counterfactuals to Build Great Products](https://engineering.shopify.com/blogs/engineering/using-quasi-experiments-counterfactuals) 230 | 231 | ## StitchFix 232 | 1. [Large Scale Experimentation at Stitch Fix](https://multithreaded.stitchfix.com/blog/2020/07/07/large-scale-experimentation/) ([Paper](http://proceedings.mlr.press/v89/schmit19a/schmit19a.pdf)) 233 | 2. [Multi-Armed Bandits and the Stitch Fix Experimentation Platform](https://multithreaded.stitchfix.com/blog/2020/08/05/bandits/) 234 | 3. [Understanding Latent Style](https://multithreaded.stitchfix.com/blog/2018/06/28/latent-style/) 235 | 4. [Give Me Jeans not Shoes: How BERT Helps Us Deliver What Clients Want](https://multithreaded.stitchfix.com/blog/2019/07/15/give-me-jeans/) 236 | 5. [Engineers Shouldn’t Write ETL: A Guide to Building a High Functioning Data Science Department](https://multithreaded.stitchfix.com/blog/2016/03/16/engineers-shouldnt-write-etl/) 237 | 6. [Beware the Data Science Pin Factory: The Power of the Full-Stack Data Science Generalist](https://multithreaded.stitchfix.com/blog/2019/03/11/FullStackDS-Generalists/) 238 | 239 | ## Pinterest 240 | 1. [Shop The Look: Building a Large Scale Visual Shopping System at Pinterest](https://dl.acm.org/doi/abs/10.1145/3394486.3403372) ([Paper](https://dl.acm.org/doi/pdf/10.1145/3394486.3403372), [Video](https://crossminds.ai/video/5f3369790576dd25aef288d7/)) 241 | 2. [Driving upsells from search](https://medium.com/pinterest-engineering/driving-shopping-upsells-from-pinterest-search-d06329255402) 242 | 3. [Pre-Submit Integration Tests For Ads-Serving](https://medium.com/pinterest-engineering/pre-submit-integration-tests-for-ads-serving-a07b5a12ea3f) 243 | 244 | 245 | ## Uber 246 | 1. [Forecasting at Uber: An Introduction](https://eng.uber.com/forecasting-introduction/) 247 | 2. [Engineering Extreme Event Forecasting at Uber with RNN](https://eng.uber.com/neural-networks/) 248 | 3. [Transforming Financial Forecasting with Data Science and Machine Learning at Uber](https://eng.uber.com/transforming-financial-forecasting-machine-learning/) 249 | 4. [Food Discovery with Uber Eats: Recommending for the Marketplace](https://eng.uber.com/uber-eats-recommending-marketplace/) 250 | 5. [Food Discovery with Uber Eats: Building a Query Understanding Engine](https://eng.uber.com/uber-eats-query-understanding/) 251 | 6. [Food Discovery with Uber Eats: Using Graph Learning to Power Recommendations](https://eng.uber.com/uber-eats-graph-learning/) 252 | 7. [How Trip Inferences and Machine Learning Optimize Delivery Times on Uber Eats](https://eng.uber.com/uber-eats-trip-optimization/) 253 | 8. [Announcing a New Framework for Designing Optimal Experiments with Pyro](https://eng.uber.com/oed-pyro-release/) ([Paper](https://papers.nips.cc/paper/9553-variational-bayesian-optimal-experimental-design.pdf)) ([Paper](https://arxiv.org/pdf/1911.00294.pdf)) 254 | 9. https://heartbeat.comet.ml/5-regression-loss-functions-all-machine-learners-should-know-4fb140e9d4b0 255 | 256 | ## OpenAI 257 | 1. [Better Language Models and Their Implications](https://openai.com/blog/better-language-models/) ([Paper](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)) 258 | 2. [Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165) ([Paper](https://arxiv.org/pdf/2005.14165.pdf)) ([GPT-3 Blog post](https://openai.com/blog/openai-api/)) 259 | 3. [Image GPT](https://openai.com/blog/image-gpt/) ([Paper](https://cdn.openai.com/papers/Generative_Pretraining_from_Pixels_V2.pdf), [Code](https://github.com/openai/image-gpt)) 260 | 4. [It's Hard to Generate Neural Text From GPT-3 About Muslims](https://twitter.com/abidlabs/status/1291165311329341440) 261 | 262 | ## Tesla 263 | 1. [AI for Full-Self Driving at Tesla](https://youtu.be/hx7BXih7zx8?t=513) 264 | 265 | 266 | ## Twitter 267 | 1. [Using Deep Learning at Scale in Twitter’s Timelines](https://blog.twitter.com/engineering/en_us/topics/insights/2017/using-deep-learning-at-scale-in-twitters-timelines.html) 268 | 2. [Using Machine Learning to Predict the Value of Ad Requests](https://blog.twitter.com/engineering/en_us/topics/insights/2020/using-machine-learning-to-predict-the-value-of-ad-requests.html) 269 | 3. [A Meta-Learning Perspective on Cold-Start Recommendations for Items](https://papers.nips.cc/paper/7266-a-meta-learning-perspective-on-cold-start-recommendations-for-items) ([Paper](https://papers.nips.cc/paper/7266-a-meta-learning-perspective-on-cold-start-recommendations-for-items.pdf)) 270 | 4. [Embeddings@Twitter](https://blog.twitter.com/engineering/en_us/topics/insights/2018/embeddingsattwitter.html) 271 | 5. [Experimenting to Solve Cramming](https://blog.twitter.com/engineering/en_us/topics/insights/2017/Experimenting-To-Solve-Cramming.html) 272 | 6. [SimClusters: Community-Based Representations for Heterogeneous Recommendations at Twitter](https://dl.acm.org/doi/10.1145/3394486.3403370) ([Paper](https://dl.acm.org/doi/pdf/10.1145/3394486.3403370), [Video](https://crossminds.ai/video/5f3369790576dd25aef288d5/)) 273 | 7. [Delayed Feedback](https://blog.twitter.com/engineering/en_us/topics/insights/2019/improving-engagement-on-digital-ads-with-delayed-feedback.html) 274 | 8. [ML Workflow](https://blog.twitter.com/engineering/en_us/topics/insights/2018/ml-workflows.html) 275 | 9. [SplitNet](https://blog.twitter.com/engineering/en_us/topics/insights/2020/a-new-approach-metric-learning-for-splitnet.html) 276 | 10. [Protect User Identity](https://blog.twitter.com/engineering/en_us/topics/insights/2018/twitter_silhouette.html) 277 | 11. [Twitter meet Tensorflow](https://blog.twitter.com/engineering/en_us/topics/insights/2018/twitter_silhouette.html) 278 | 279 | 280 | ## WalmartLabs 281 | 1. [Chimera: Large-scale Classification using Machine Learning, Rules, and Crowdsourcing](https://dl.acm.org/doi/10.14778/2733004.2733024) ([Paper](http://pages.cs.wisc.edu/%7Eanhai/papers/chimera-vldb14.pdf)) 282 | 2. [Retail Graph — Walmart’s Product Knowledge Graph](https://medium.com/walmartlabs/retail-graph-walmarts-product-knowledge-graph-6ef7357963bc) 283 | 284 | 285 | 286 | 287 | 288 | ## Yahoo 289 | 1. [E-commerce in Your Inbox: Product Recommendations at Scale](https://arxiv.org/abs/1606.07154) 290 | 2. [Product Recommendations at Scale](https://arxiv.org/abs/1606.07154) ([Paper](https://arxiv.org/pdf/1606.07154.pdf)) 291 | 3. [Ranking Relevance in Yahoo Search](https://www.kdd.org/kdd2016/subtopic/view/ranking-relevance-in-yahoo-search) ([Paper](https://www.kdd.org/kdd2016/papers/files/adf0361-yinA.pdf)) 292 | 3. [Abusive Language Detection in Online User Content](https://dl.acm.org/doi/10.1145/2872427.2883062) ([Paper](http://www.yichang-cs.com/yahoo/WWW16_Abusivedetection.pdf)) 293 | 294 | ## YouTube 295 | 1. [Deep Neural Networks for YouTube Recommendations](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45530.pdf) 296 | 297 | ## Others 298 | 1. [Large-scale Item Categorization in e-Commerce Using Multiple Recurrent Neural Networks](https://www.kdd.org/kdd2016/subtopic/view/large-scale-item-categorization-in-e-commerce-using-multiple-recurrent-neur/) ([Paper](https://www.kdd.org/kdd2016/papers/files/adf0392-haAemb.pdf)) `NAVER` 299 | 2. [How We Built the Good First Issues Feature](https://github.blog/2020-01-22-how-we-built-good-first-issues/) `GitHub` 300 | 3. [Teaching Machines to Triage Firefox Bugs](https://hacks.mozilla.org/2019/04/teaching-machines-to-triage-firefox-bugs/) `Mozilla` 301 | 4. [Testing Firefox More Efficiently with Machine Learning](https://hacks.mozilla.org/2020/07/testing-firefox-more-efficiently-with-machine-learning/) `Mozilla` 302 | 5. [Enabling 10x More Experiments with Traveloka Experiment Platform](https://medium.com/traveloka-engineering/enabling-10x-more-experiments-with-traveloka-experiment-platform-8cea13e952c) `Traveloka` 303 | 6. [Modeling Conversion Rates and Saving Millions Using Kaplan-Meier and Gamma Distributions](https://better.engineering/modeling-conversion-rates-and-saving-millions-of-dollars-using-kaplan-meier-and-gamma-distributions/) ([Code](https://github.com/better/convoys)) `Better` 304 | 7. [Successes and Challenges in Adopting Machine Learning at Scale at a Global Bank](https://www.youtube.com/watch?v=QYQKG5OcwEI) `Rabobank` 305 | 8. [160k+ High School Students Will Graduate Only If a Model Allows Them to](http://positivelysemidefinite.com/2020/06/160k-students.html) `International Baccalaureate` 306 | 9. [A British AI Tool to Predict Violent Crime Is Too Flawed to Use](https://www.wired.co.uk/article/police-violence-prediction-ndas) `United Kingdom` 307 | 10. More in [awful-ai](https://github.com/daviddao/awful-ai) 308 | 11. [Deep Learned Super Resolution for Feature Film Production](https://graphics.pixar.com/library/SuperResolution/) ([Paper](https://graphics.pixar.com/library/SuperResolution/paper.pdf)) `Pixar` 309 | 12. [Osprey: Weak Supervision of Imbalanced Extraction Problems without Code](https://dl.acm.org/doi/abs/10.1145/3329486.3329492) ([Paper](https://ajratner.github.io/assets/papers/Osprey_DEEM.pdf)) `Intel` 310 | 13. [Bootstrapping Conversational Agents with Weak Supervision](https://www.aaai.org/ojs/index.php/AAAI/article/view/5011) ([Paper](https://arxiv.org/pdf/1812.06176.pdf)) `IBM` 311 | 14. [Unsupervised Extraction of Attributes and Their Values from Product Description](https://www.aclweb.org/anthology/I13-1190/) ([Paper](https://www.aclweb.org/anthology/I13-1190.pdf)) `Rakuten` 312 | 15. [Information Extraction from Receipts with Graph Convolutional Networks](https://nanonets.com/blog/information-extraction-graph-convolutional-networks/) `Nanonets` 313 | 16. [The Data and Science behind GrabShare Carpooling](https://ieeexplore.ieee.org/document/8259801) (**PAPER NEEDED**) `Grab` 314 | 17. [Optimization of Passengers Waiting Time in Elevators Using Machine Learning](https://www.youtube.com/watch?v=vXndCC89BCw&t=4s) `Thyssen Krupp AG` 315 | 19. [Automatic Forecasting using Prophet, Databricks, Delta Lake and MLflow](https://www.youtube.com/watch?v=TkcpjnLh690) ([Paper](https://peerj.com/preprints/3190.pdf), [Code](https://github.com/facebook/prophet)) `Atlassian` 316 | 20. [Session-based Recommendations with Recurrent Neural Networks](https://arxiv.org/abs/1511.06939) ([Paper](https://arxiv.org/pdf/1511.06939.pdf)) `Telefonica` 317 | 21. [Personalized Recommendations for Experiences Using Deep Learning](https://www.tripadvisor.com/engineering/personalized-recommendations-for-experiences-using-deep-learning/) `TripAdvisor` 318 | 22. [Uncovering Insurance Fraud Conspiracy with Network Learning](https://arxiv.org/abs/2002.12789) ([Paper](https://arxiv.org/pdf/2002.12789.pdf)) `Ant Financial` 319 | 23. [How Does Spam Protection Work on Stack Exchange?](https://stackoverflow.blog/2020/06/25/how-does-spam-protection-work-on-stack-exchange/) `Stack Exchange` 320 | 24. [Auto Content Moderation in C2C e-Commerce](https://www.usenix.org/conference/opml20/presentation/ueta) `Mercari` 321 | 25. [Personalized Channel Recommendations in Slack](https://slack.engineering/personalized-channel-recommendations-in-slack/) `Slack` 322 | 26. [Blocking Slack Invite Spam With Machine Learning](https://slack.engineering/blocking-slack-invite-spam-with-machine-learning/) `Slack` 323 | 27. [Cloudflare Bot Management: Machine Learning and More](https://blog.cloudflare.com/cloudflare-bot-management-machine-learning-and-more/) `Cloudflare` 324 | 28. [Anomalies in Oil Temperature Variations in a Tunnel Boring Machine](https://www.youtube.com/watch?v=YV_uLLhPRAk) `SENER` 325 | 29. [Using Anomaly Detection to Monitor Low-Risk Bank Customers](https://www.youtube.com/watch?v=MExokMM_Bp4&t=3s) `Rabobank` 326 | 30. [Deep Reinforcement Learning in Production Part1](https://towardsdatascience.com/deep-reinforcement-learning-in-production-7e1e63471e2) [Part 2](https://towardsdatascience.com/deep-reinforcement-learning-in-production-part-2-personalizing-user-notifications-812a68ce2355) `Zynga` 327 | 31. [How we Improved Computer Vision Metrics by More Than 5% Only by Cleaning Labelling Errors](https://deepomatic.com/en/how-we-improved-computer-vision-metrics-by-more-than-5-percent-only-by-cleaning-labelling-errors/) `Deepomatic` 328 | 32. [How Disney Uses PyTorch for Animated Character Recognition](https://medium.com/pytorch/how-disney-uses-pytorch-for-animated-character-recognition-a1722a182627) `Disney` 329 | 33. [Image Captioning as an Assistive Technology](https://www.ibm.com/blogs/research/2020/07/image-captioning-assistive-technology/) ([Video](https://ivc.ischool.utexas.edu/~yz9244/VizWiz_workshop/videos/MMTeam-oral.mp4)) `IBM` 330 | 34. [AI for AG: Production machine learning for agriculture](https://medium.com/pytorch/ai-for-ag-production-machine-learning-for-agriculture-e8cfdb9849a1) `Blue River` 331 | 35. [Building AI Trading Systems](https://dennybritz.com/blog/ai-trading/) `Denny Britz` 332 | 36. [How Lazada Ranks Products to Improve Customer Experience and Conversion](https://www.slideshare.net/eugeneyan/how-lazada-ranks-products-to-improve-customer-experience-and-conversion) `Lazada` 333 | 37. [An Ensemble-based Approach to Click-Through Rate Prediction for Promoted Listings at Etsy](https://arxiv.org/abs/1711.01377) ([Paper](https://arxiv.org/pdf/1711.01377.pdf)) `Etsy` 334 | 38. [Query Understanding Engine in Traveloka Universal Search](https://medium.com/traveloka-engineering/query-understanding-engine-in-traveloka-universal-search-410ad3895db7) `Traveloka` 335 | 39. [Bayesian Product Ranking at Wayfair](https://tech.wayfair.com/data-science/2020/01/bayesian-product-ranking-at-wayfair) `Wayfair` 336 | 40. [Vector Representation Of Items, Customer And Cart To Build A Recommendation System](https://arxiv.org/abs/1705.06338) ([Paper](https://arxiv.org/pdf/1705.06338.pdf)) `Sears` 337 | 42. [The State-of-the-art Open-Domain Chatbot in Chinese and English](http://research.baidu.com/Blog/index-view?id=142) ([Paper](https://arxiv.org/pdf/2006.16779.pdf)) `Baidu` 338 | 43. [Photon: A Robust Cross-Domain Text-to-SQL System](https://www.aclweb.org/anthology/2020.acl-demos.24/) ([Paper](https://www.aclweb.org/anthology/2020.acl-demos.24.pdf)) ([Demo](http://naturalsql.com)) `Salesforce` 339 | 44. [GeDi: A Powerful New Method for Controlling Language Models](https://blog.einstein.ai/gedi/) ([Paper](https://arxiv.org/abs/2009.06367), [Code](https://github.com/salesforce/GeDi)) `Salesforce` 340 | 45. [Applying Topic Modeling to Improve Call Center Operations](https://www.youtube.com/watch?v=kzRR8OjF_eI&t=2s) `RICOH` 341 | 46. [Deep Learning for Understanding Consumer Histories](https://engineering.zalando.com/posts/2016/10/deep-learning-for-understanding-consumer-histories.html) ([Paper](https://doogkong.github.io/2017/papers/paper2.pdf)) `Zalando` 342 | 47. [Continual Prediction of Notification Attendance with Classical and Deep Networks](https://arxiv.org/abs/1712.07120) ([Paper](https://arxiv.org/pdf/1712.07120.pdf)) `Telefonica` 343 | 48. [Using Recurrent Neural Network Models for Early Detection of Heart Failure Onset](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5391725/) ([Paper](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5391725/pdf/ocw112.pdf)) `Sutter Health` 344 | 49. [Doctor AI: Predicting Clinical Events via Recurrent Neural Networks](https://arxiv.org/abs/1511.05942) ([Paper](https://arxiv.org/pdf/1511.05942.pdf)) `Sutter Health` 345 | 50. [How Duolingo uses AI in every part of its app](https://venturebeat.com/2020/08/18/how-duolingo-uses-ai-in-every-part-of-its-app/) `Duolingo` 346 | 51. [Future Data Helps Training: Modeling Future Contexts for Session-based Recommendation]() ([Paper](https://arxiv.org/pdf/1906.04473.pdf)) `Tencent` 347 | 348 | 349 | Survey Recommendation papers 350 | 1. https://arxiv.org/pdf/2104.13030.pdf 351 | -------------------------------------------------------------------------------- /company_guides.md: -------------------------------------------------------------------------------- 1 | # Interview preparation guides from top companies 2 | 1. [Facebook MLE](https://research.fb.com/the-facebook-field-guide-to-machine-learning-video-series/) 3 | 2. [TripleByte](https://triplebyte.com/blog/triplebyte-s-way-too-long-technical-interview-prep-guide) 4 | -------------------------------------------------------------------------------- /course.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # Machine Learning System Design course 4 | ## Course content 5 | | Section | | 6 | | ------------- | ------------- | 7 | | 1. [Youtube Recommendation](https://rebrand.ly/mldesign) |

| 8 | | 2. [The main components in MLSD](https://rebrand.ly/mldesign) |

| 9 | | 3. [LinkedIn Feed Ranking](https://rebrand.ly/mldesign) |

| 10 | | 4. [Ad Click Prediction](https://rebrand.ly/mldesign)|

| 11 | | 5. [Estimate Delivery time](https://rebrand.ly/mldesign)|

| 12 | | 6. [Airbnb Search ranking](https://rebrand.ly/mldesign)|

| 13 | 14 | 15 | 16 | 17 | - All course contents are subjected to changed based on actual demands. If you want to subscribe the course [click here](https://github.com/sponsors/khangich) and become sponsor or pay one time fee $50 [here](https://www.paypal.com/paypalme/MLinterviews). Please follow instructions below. 18 | 19 | 20 | ## System design sample 21 | - You can see the sample of ML system design [here](https://docs.google.com/document/d/12KrfFYxiqCVmjmHiOfIoqBWuOuZEBXcL1RjWFkY7hLk/edit#). 22 | - Another [sample](https://github.com/khangich/machine-learning-interview/blob/master/design.md) 23 | 24 | 25 | ## Frequently asked questions (MUST read before subscribe) 26 | - Who should NOT subscribe to this course? New grad, people applied for data scientist, research scientist, software engineer positions. People will have onsite interview tomorrow or 1 year from now. 27 | - Do you guarantee the sytem design is good enough for me to get a job? No. 28 | - Did you copy the solution from the internet? No. 29 | - How do I know the solutions are good? Solutions are based on actual production and well-known published papers. 30 | - Can I get a refund if I'm not satisfied? No, I can't do it because it's too complicated. 31 | - Can I stop subscription? You can stop your subscription any time. You won't be able to access the course. 32 | - Do you have working solution with code and data? No, it's impossible. 33 | - Will you in the future release the whole course for free? No 34 | - Can I request system design usecase? Yes, you can [request here](https://forms.gle/ALsKMik5ocwptUZh7) 35 | - For further questions, please send email helppreparemle@gmail.com. 36 | 37 | ## How to start learning ML system design? (available Now) 38 | - You need to become the sponsor to be able to access the course. Your sponsorship is crucial for me to make high quality content. In general I spend 40 hours to finish one MLSD. 39 | 40 | - Step 1: click the Sponsor button then follow instruction. 41 | 42 | [![screen](images/sponsor.png)](https://github.com/sponsors/khangich) 43 | 44 | - Step 2: Go to [online course](https://docs.google.com/document/d/15XIFNS23oWEttgot1ILL3yxWQqGYHnRn8Jh3YyCer3s/edit). You will receive system design in google docs. 45 | - Check your email for the online course link (within 2 hours) 46 | 47 | 48 | ## Notes 49 | * If you find this helpful, you can Sponsor this project. It's cool if you don't. 50 | -------------------------------------------------------------------------------- /design.md: -------------------------------------------------------------------------------- 1 | # ML system design usecases 2 | * The goal of this section is to provide a structural way to approach ML system design questions. If you're interested to learn more about paid ML system design course with more examples, [click here](https://rebrand.ly/interviewml). 3 | 4 | ## Ad Click Prediction for Social Networks. 5 | * Build a machine learning model to predict if an ad will be clicked. For simplicity reasons, 6 | we will not focus on the cascade of classifiers that's commonly used in adtech. 7 | 8 | * Background on how Adtech bidding works (source: [ML in AdTech](https://www.slideshare.net/databricks/machine-learning-for-adtech-in-action-with-cyrille-dubarry-and-han-ju)) 9 | 10 | ![Score distribution](images/ad_bidding.png) 11 | 12 | 1. Requirements 13 | * ML model with good performance. 14 | * System can scale to larger number of users with low latency. 15 | * Imbalance data: you can assume Click Through Rate (CTR) is very small in practice (1%-2%). 16 | * Serving: from the RTB worfklow diagram, it's important to have low latency (150 ms) for ad prediction. 17 | 18 | 19 | 2. Calculate and estimation 20 | * Assumptions: 4K ads requests per second which is 10 billions ads requests per month. 21 | * Data: historical ad clicks data includes [user, ads, click_or_not]. With estimated 1% CTR, it has 100 millions clicked ads. We can start with 1 month of data for training and validation. 22 | 23 | * Train/validation data split: We split train/validation to simulate the actual online system for example: split by time. 24 | 25 | * Features: it's natural that the model need to have enough capacity to learn patterns from big training data. In practice it's common to have hundreds even thousands of features. 26 | 27 | * Training: ability to retrain many times within one day to increase model performance in online manner. 28 | 29 | * Serving: latency within 150ms per request and 4K request per second. Test 30 | 31 | 32 | 3. Metrics evaluation 33 | * During the training phase, we can focus on machine learning metrics instead of revenue metrics or CTR metrics. Regarding revenue-related metrics, we usually monitor during deployment and it's out of scope. See section 7 further readings to understand about offline metrics and online metrics. 34 | * Normalized Cross Entropy: predictive log loss divided by the cross entropy of the background CTR. This way NCE is insensitive to background CTR. This is the NCE formula (source [facebook](https://research.fb.com/wp-content/uploads/2016/11/practical-lessons-from-predicting-clicks-on-ads-at-facebook.pdf)) 35 | 36 | ![Score distribution](images/nce.png) 37 | 38 | * Calibration metrics measured by the expected clicks vs the actual observed clicks. Read more about calibration [here](https://arxiv.org/pdf/1706.04599.pdf) 39 | 40 | 4. Modeling: features scaling, major sub-sambling. 41 | * Model: We can use probabilistic sparse linear classifier (logistic regression). It's popular because of the computation efficiency and sparsity features. 42 | * Feature engineering: 43 | * AdvertiserID: it's easy to have millions of advertisers. One common way is to use embedding as a distributed representation for advitiserID. If you're not familiar with [embedding](https://blog.twitter.com/engineering/en_us/topics/insights/2018/embeddingsattwitter.html). Another way is to use [feature hashing by Kilian Weinberger](https://arxiv.org/pdf/0902.2206.pdf) to handle large sparse column. Feature hashing helps increase speed and reduce memory usage. See video [here](https://www.coursera.org/lecture/machine-learning-applications-big-data/hashing-trick-GswXH). 44 | * Numeric columns: pay attention to [feature scaling](https://www.datacamp.com/community/tutorials/preprocessing-in-data-science-part-2-centering-scaling-and-logistic-regression). 45 | 46 | * Data processing: 47 | * One way is subsampling majority negative class at different sub-sampling ratio. The key here is ensuring that the validation dataset has the same distribution as the test data set. We also need to pay attention how this sampling affect predictions. 48 | 49 | 5. Model deployment and testing 50 | * During deployment phase, it's crucial to monitor the actual CTR and other revenue-related metrics. 51 | * Related to this topic, read more about A/B testing and multi-arms bandit. 52 | 53 | 6. High level system design 54 | ![system desgin](images/system_design.png) 55 | 56 | 57 | 7. Further readings 58 | * [Common ML designs](https://rebrand.ly/interviewml) 59 | * [Machine Learning in Adtech](https://www.slideshare.net/databricks/machine-learning-for-adtech-in-action-with-cyrille-dubarry-and-han-ju) 60 | * [Ad Click Prediction: a View from the trenches](https://storage.googleapis.com/pub-tools-public-publication-data/pdf/41159.pdf) 61 | * [Practical lessons from Predicting Clicks on Ads at Facebook](https://research.fb.com/wp-content/uploads/2016/11/practical-lessons-from-predicting-clicks-on-ads-at-facebook.pdf) 62 | * [Predictive Model Performance: Offline and Online Evaluation](http://chbrown.github.io/kdd-2013-usb/kdd/p1294.pdf) 63 | * [Kaggle CTR prediction](https://www.kaggle.com/c/avazu-ctr-prediction/overview) 64 | 65 | # Notes 66 | * If you're interested to learn more about paid ML system design course with more examples, [click here](https://rebrand.ly/mlsd_launch). 67 | * If you find this helpful, you can Sponsor this project. It's cool if you don't. 68 | -------------------------------------------------------------------------------- /extra.md: -------------------------------------------------------------------------------- 1 | # Extra reads 2 | ## ML advance topics 3 | | Paper | Why | 4 | | ------------- | ------------- | 5 | | [Sampling techniques](https://towardsdatascience.com/sampling-techniques-a4e34111d808) | Stratifield sampling is popular. | 6 | | [A Comparative Study of Efficient Initialization Methods for the k-means Clustering Algorithm](https://arxiv.org/pdf/1209.1960.pdf) and [kmeans initialization, Coursera](https://www.coursera.org/lecture/cluster-analysis/3-3-initialization-of-k-means-clustering-bPyBl)| Fundamentals about kemans, favourite topics in interviews i.e: LinkedIn| 7 | | [A Comprehensive Survey of Clustering Algorithms](https://link.springer.com/content/pdf/10.1007/s40745-015-0040-1.pdf) | Clustering fundamentals | 8 | | [A Tutorial on Spectral Clustering](https://arxiv.org/pdf/0711.0189.pdf)| Sepctral clustering is intuitive and quite popular. | 9 | | [Partial residual plot](https://en.wikipedia.org/wiki/Partial_residual_plot) | Useful for model diagnosis. | 10 | | [Compare GINI index and Information Gain](https://www.unine.ch/files/live/sites/imi/files/shared/documents/papers/Gini_index_fulltext.pdf) | Intuition behind Decision Tree, RandomForest | 11 | | [Explain tf-idf](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.97.7340&rep=rep1&type=pdf) | Fundamentals about tf-idf. | 12 | | [Understanding L-BFGS](https://aria42.com/blog/2014/12/understanding-lbfgs) | Advance about optimization, rarely asked in interview | 13 | | [Optimizer Quasi newton method](http://www.seas.ucla.edu/~vandenbe/236C/lectures/qnewton.pdf) | Advance about optimization. | 14 | 15 | 16 | ## DL classic papers 17 | | Paper | Why | 18 | | ------------- | ------------- | 19 | | [Understanding the Difficulty of Training Deep Feedforward Neural Networks](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf) | Classic paper (2010) about initialization, sigmoid etc | 20 | | [Delving Deep into Rectifiers - Surpassing Human-Level Performance on ImageNet Classification](https://www.cv-foundation.org/openaccess/content_iccv_2015/papers/He_Delving_Deep_into_ICCV_2015_paper.pdf) | Classic paper (2015) about ReLU, PReLU| 21 | | [Batch Normalization - Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://arxiv.org/pdf/1502.03167.pdf%20http://arxiv.org/abs/1502.03167.pdf) | Classic paper about BatchNorm | 22 | | [Dropout - A Simple Way to Prevent Neural Networks from Overfitting](https://www.jmlr.org/papers/volume15/srivastava14a/srivastava14a.pdf) | Classic paper about Dropout | 23 | | [Deep Residual Learning for Image Recognition](https://openaccess.thecvf.com/content_cvpr_2016/papers/He_Deep_Residual_Learning_CVPR_2016_paper.pdf) | Classic ResNet | 24 | | [On Large-Batch Training for Deep Learning - Generalization Gap and Sharp Minima](https://arxiv.org/pdf/1609.04836.pdf,) | Practical technique for large batch training | 25 | 26 | 27 | ## DL advance topics 28 | | Paper | Why | 29 | | ------------- | ------------- | 30 | | [Calibration in modern neural network](https://arxiv.org/pdf/1706.04599.pdf) | Important topics in ML system design i.e: facebook | 31 | | [Attention model](https://www.youtube.com/watch?v=quoGRI-1l0A&list=RDCMUCcIXc5mJsHVYTZR1maL5l9w&index=2) | Fundamentals in Attention, powerful architecture in NLP | 32 | | [Ilya's thesis](https://www.cs.utoronto.ca/~ilya/pubs/ilya_sutskever_phd_thesis.pdf) | Network in network | 33 | 34 | 35 | ## NLP 36 | | Paper | Why | 37 | | ------------- | ------------- | 38 | | [Adaptive Importance Sampling to Accelerate Training of a Neural Probabilistic Language Model](https://wiki.inf.ed.ac.uk/twiki/pub/CSTR/ListenSemester2_2009_10/4443871.pdf) | Classic paper (2008) in NLP | 39 | | [Natural Language Processing (Almost) from Scratch](https://www.jmlr.org/papers/volume12/collobert11a/collobert11a.pdf) | Classic paper (2011) in NLP | 40 | | [Word2vec](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf) | The classic paper in NLP, still popular in industry: Uber, DoorDash, Twitter etc | 41 | | [GloVe - Global Vectors for Word Representation](https://www.aclweb.org/anthology/D14-1162.pdf) | Classic paper (2014) in NLP | 42 | | [Bag of Tricks for Efficient Text Classification](https://arxiv.org/pdf/1607.01759.pdf) | Cool tricks in NLP tasks | 43 | | [BERT - Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/pdf/1810.04805.pdf) | The famous BERT | 44 | | [Smart Reply - Automated Response Suggestion for Email](https://arxiv.org/pdf/1606.04870.pdf) | NLP application, useful for ML system design | 45 | | [Enriching Word Vectors with Subword Information](https://www.mitpressjournals.org/doi/pdfplus/10.1162/tacl_a_00051) | Simple and fast method to train NLP task in Facebook| 46 | | [Neural Approaches to Conversational AI](https://arxiv.org/pdf/1809.08267.pdf) | Comprehensive survey (2018) about chatbots| 47 | 48 | ## Recent breakthrough 49 | - [Feb 2021, recent breakthrough ClubHouse chat](https://victordibia.com/blog/ai-breakthroughs-feb21/) 50 | 51 | # Notes 52 | * If you're interested to learn more about paid ML system design course on educative.io with more examples, [click here](https://rebrand.ly/mlsd_launch). 53 | * If you find this helpful, you can Sponsor this project. It's cool if you don't. 54 | -------------------------------------------------------------------------------- /faqs.md: -------------------------------------------------------------------------------- 1 | # List of common questions 2 | 3 | ## Question for people have no working experience in DS/ML 4 | 5 | * What is the strategy/study plan to get a job in DS/ML? I don't know. There are many ways to make up for knowledge gap. For a competitive field like ML/DS, it's common for people to do the following: 1) get a Master in DS/ML 1) Try career-focused program like [insight fellows](https://insightfellows.com/) 2) [interview query](https://www.interviewquery.com/). 6 | 7 | * Common questions about [my interview experience](https://mlengineer.io/first-ama-with-me-ml-system-design-author-9d33840b27d1?source=friends_link&sk=197d2de7d04235d4f63ded4d2a42dec5). 8 | 9 | 10 | ## Questions about focus areas 11 | 12 | * What is the MLE interview process? This [document](https://s3.amazonaws.com/external_clips/3406172/1568840998_The_Ultimate_Guide_to_AI_and_Machine_Learning_Job_Interviews_1_1_.pdf?1587691917) described the process very well. 13 | * Did you get questions around PyTorch, TensorFlow and other frameworks? No. 14 | 15 | * Which language should I learn for MLE? Python is useful for both interview coding questions and actual work (tensorflow, pytorch, scikitlearn, pandas etc). 16 | 17 | * Do I need to implement ML algorithm from scratch? Yes, see details in [study guide](README.md) 18 | 19 | * Do I need to solve LC hard? Yes. I got about 5%-10% questions are LC hard. 20 | 21 | * Were you asked about cloud technologies, containers, orchestration, etc? No, but it's a MUST have when you present system design interview. 22 | 23 | * For ML theory, was it more focused on classical ML or DL? Both. 24 | 25 | * Were you expected to be up to speed with recent cutting-edge research in ML? No, nice to have. At the minimum please read about Embedding. 26 | 27 | * Is it ok to be a generalist or are you expected to have a speacialisation in Vision, NLP, etc? It depends on roles and company. In general, it's not required. 28 | 29 | * Were there any questions around various Data Engineering tools? In general, No. But a lot of companies expect you to know how to handle big data with Spark etc. 30 | 31 | * Did you get probability and statistics questions? If yes, what kind? Yes, Bayesian. And nobody care if you know Gamma and Poisson distribution. 32 | 33 | * Were there any ML System Design or general system design questions? Both. 34 | 35 | * Do the interviewers ask indepth questions in ML like explain Adagrad or XGBoost in detail? Yes, but do not try to memorize the details, see [how to study](how.md) 36 | 37 | * Do I need to read any research papers for the interviews in top companies? No, nice to have. 38 | 39 | ## Questions about before/during on-site interview 40 | * What to do one day before the onsite? Relax, do not study. Go to sleep early. If you have to fly, please do not fly on the onsite day. 41 | 42 | * Do I need to customize my prepraration for company? I do that and found it's extremely helpful. I spent one hour or two on glassdoor to get the overview of the interview. I also review interviewer's Linkedin thoroughly. 43 | 44 | * Do interviews need to like me? Yes, it helps. 45 | 46 | * What to do when you black-out and have no idea how to answer? Take one step back, breath deeply, do not think about anything, just give 30 seconds or so to calm down. If you know the answer, it will come to you. 47 | 48 | * What is the right way to answer technical questions? Think about interview's intentions then answer to the point. Try to stay precise, concise and to the point. 49 | 50 | ## Questions about logistics 51 | 52 | * How long does it take to prepare for interviewing? It depends, 3-4 months is a stretch but doable. 53 | 54 | * Do you recommend doing a few interviews are startups as practice before moving onto the big tech companies? Yes, after 2-3 startup companies there is diminish of returns. 55 | 56 | * Do you have any negotiation tips you can share? Two simple rules: 1) Do not specify the TC number first. 2) Have competing offers for negotiation. 57 | 58 | * Did you get many questions about your past experience? if yes, how did you navigate not being able to share sensitive information? Yes, tell interviews, they understand that. 59 | 60 | * Do you think non-CS background affects my chances in any way? Not to my knowledge. 61 | 62 | * How should I go about preparing? See [plan](README.md) 63 | 64 | * Can you help in how to get through the in-depth ML questions asked in the interviews? What sources did you follow for that? See [plan](README.md) 65 | 66 | * Can you provide me mentorship on what to prepare and how to maximise my chances? Yes. Send me an email to helppreparemle@gmail.com. 67 | 68 | # Notes 69 | * If you're interested to learn more about paid ML system design course with more examples, [click here](course.md). 70 | * If you find this helpful, you can Sponsor this project. It's cool if you don't. 71 | -------------------------------------------------------------------------------- /how.md: -------------------------------------------------------------------------------- 1 | # How to study (TBD) 2 | * Richard Feynman on [thinking](https://www.youtube.com/watch?v=lr8sVailoLw) 3 | * How to deepen your [understanding](http://cognitivemedium.com/srs-mathematics). Tl;dr: Ask many why questions and put them in Anki card. 4 | * How to [understand things](https://nabeelqu.co/understanding) 5 | 6 | # Notes 7 | * If you're interested to learn more about paid ML system design course with more examples, [click here](course.md). 8 | * If you find this helpful, you can Sponsor this project. It's cool if you don't. -------------------------------------------------------------------------------- /images/LC.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/khangich/machine-learning-interview/968a286650c964f8583b73b3e0864475b86f22b6/images/LC.png -------------------------------------------------------------------------------- /images/ad_bidding.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/khangich/machine-learning-interview/968a286650c964f8583b73b3e0864475b86f22b6/images/ad_bidding.png -------------------------------------------------------------------------------- /images/ads.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/khangich/machine-learning-interview/968a286650c964f8583b73b3e0864475b86f22b6/images/ads.png -------------------------------------------------------------------------------- /images/air.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/khangich/machine-learning-interview/968a286650c964f8583b73b3e0864475b86f22b6/images/air.png -------------------------------------------------------------------------------- /images/delivery.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/khangich/machine-learning-interview/968a286650c964f8583b73b3e0864475b86f22b6/images/delivery.png -------------------------------------------------------------------------------- /images/feed.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/khangich/machine-learning-interview/968a286650c964f8583b73b3e0864475b86f22b6/images/feed.png -------------------------------------------------------------------------------- /images/ml.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/khangich/machine-learning-interview/968a286650c964f8583b73b3e0864475b86f22b6/images/ml.png -------------------------------------------------------------------------------- /images/nce.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/khangich/machine-learning-interview/968a286650c964f8583b73b3e0864475b86f22b6/images/nce.png -------------------------------------------------------------------------------- /images/score.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/khangich/machine-learning-interview/968a286650c964f8583b73b3e0864475b86f22b6/images/score.png -------------------------------------------------------------------------------- /images/score1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/khangich/machine-learning-interview/968a286650c964f8583b73b3e0864475b86f22b6/images/score1.png -------------------------------------------------------------------------------- /images/sponsor.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/khangich/machine-learning-interview/968a286650c964f8583b73b3e0864475b86f22b6/images/sponsor.png -------------------------------------------------------------------------------- /images/stat_cheatsheet.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/khangich/machine-learning-interview/968a286650c964f8583b73b3e0864475b86f22b6/images/stat_cheatsheet.png -------------------------------------------------------------------------------- /images/system_design.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/khangich/machine-learning-interview/968a286650c964f8583b73b3e0864475b86f22b6/images/system_design.png -------------------------------------------------------------------------------- /images/uc3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/khangich/machine-learning-interview/968a286650c964f8583b73b3e0864475b86f22b6/images/uc3.png -------------------------------------------------------------------------------- /interview_experiences.md: -------------------------------------------------------------------------------- 1 | # Interview experiences 2 | 3 | 1. [Facebook | E5 MLE | USA | Apr 2020 [Offer]](https://leetcode.com/discuss/interview-experience/556993/Facebook-or-E5-MLE-or-USA-or-Apr-2020-Offer) 4 | 2. [Facebook/LinkedIn | E5 MLE [Rejected] | USA | Apr 2020 | Google, Coupang [Offer]](https://leetcode.com/discuss/interview-experience/1148901/MLE%3A-Offers-(Google-Coupang)-Rejected%3A-FB-LinkedIn) 5 | 3. [Amazon | Seattle | MLE2 | Alexa AI | Virtual Onsite 15th Apr](https://leetcode.com/discuss/interview-experience/591430/Amazon-or-Seattle-or-MLE2-or-Alexa-AI-or-Virtual-Onsite-15th-Apr) 6 | 4. [Google | PayPal | SWE | ML | Nov 2020 | [Offer]](https://leetcode.com/discuss/interview-experience/939742/Google-or-PayPal-or-SWE-or-ML-or-Nov-2020-or-Offer) 7 | 5. [Hiring freeze during COVID-19 | Google L3 | Microsoft L59 | Zurich](https://leetcode.com/discuss/interview-experience/557684/Hiring-freeze-during-COVID-19-or-Google-L3-or-Microsoft-L59-or-Zurich) 8 | 6. [Facebook | E6 ML | NYC | Apr 2020](https://leetcode.com/discuss/interview-experience/577087/Facebook-or-E6-ML-or-NYC-or-Apr-2020) 9 | 7. [Facebook | E5 | July 2020 | Reject](https://leetcode.com/discuss/interview-experience/737679/Facebook-or-E5-or-July-2020-or-Reject) 10 | 8. [Facebook | Research Scientist | Seattle | December 2019 [Reject]](https://leetcode.com/discuss/interview-experience/541073/Facebook-or-Research-Scientist-or-Seattle-or-December-2019-Reject) 11 | 9. [Facebook | ML Engineer | Santa Monica | Apr 2020 [Reject]](https://leetcode.com/discuss/interview-experience/563659/Facebook-or-ML-Engineer-or-Santa-Monica-or-Apr-2020-Reject) 12 | 10. [ByteDance | Machine Learning Engineer | New Grad | Singapore | 2020 [Reject]](https://leetcode.com/discuss/interview-experience/956142/ByteDance-or-Machine-Learning-Engineer-or-New-Grad-or-Singapore-or-2020-Reject) 13 | 11. [Facebook E5 | Google L5 | Microsoft AS 2 (62) | Seattle | May 2020 [Offers]](https://leetcode.com/discuss/interview-experience/665619/Facebook-E5-or-Google-L5-or-Microsoft-AS-2-(62)-or-Seattle-or-May-2020-Offers) 14 | 15 | 16 | -------------------------------------------------------------------------------- /leetcode.md: -------------------------------------------------------------------------------- 1 | 2 | # Leetcode compact list 3 | [

](https://docs.google.com/spreadsheets/d/1RCb1dVQCLmtOGlJ5J-NJ5pIC7Tda-N2U/edit#gid=274831950) 4 | 5 | * I use [LC time tracking](https://docs.google.com/spreadsheets/d/1RCb1dVQCLmtOGlJ5J-NJ5pIC7Tda-N2U/edit#gid=274831950) to keep track of how many times I solves a question and how long I spent each time. Once I finish non-trivial medium LC questions 3 times, I have absolutely no issues solving them in actual interviews (sometimes within 8-10 minutes). It makes a big difference. A better way is to use LeetPlug chrome extension [here](https://leetplug.azurewebsites.net/static/pages/howto.html) 6 | 7 | * This [Leetcode post](https://leetcode.com/discuss/general-discussion/665604/important-and-useful-links-from-all-over-the-leetcode/659097) combines 36 tips and tricks in improving Leetcode problem solving skills. 8 | 9 | 10 | 11 | ## Leetcode readiness 12 | * You can to solve Medium question within 20 minutes with little to no bug and clean code. 13 | * You can solve medium question with 2,3 different solutions (recursive, top-down DP with memorization, bottom-up DP, bfs/dfs). 14 | * You can estimate time complexity for your algorithm correctly. 15 | 16 | * Leetcode by topics for beginnier: https://leetcode.com/discuss/career/448024/Topic-wise-problems-for-Beginners 17 | * Leetcode list by topics (more comprehensive): https://protegejj.gitbook.io/oj-practices/chapter1/dynamic-programming 18 | ## Two Pointers 19 | * 75 Sort Colors 20 | * 80 Remove Duplicates from Sorted Array II 21 | * 88 Merge Sorted Array 22 | * 457 Circular Array Loop 23 | * 713 Subarray Product Less Than K 24 | * 845 Longest Mountain in Array 25 | * 904 Fruit into baskets 26 | * 925 Long Pressed name 27 | * 986 Interval list intersections 28 | ## Binary Search 29 | * 33 Search in rotated sorted array 30 | * 69 Sqrt(x) 31 | * 153 Find min in rotated sorted array 32 | * 349 Intersection of two arrays 33 | * 441 Arranging coins 34 | * 475 Heaters 35 | * 658 Find K closest elements 36 | * 911 Online Election 37 | * [Divide two integers](https://leetcode.com/problems/divide-two-integers/submissions/) 38 | * [Find in sorted array](https://leetcode.com/problems/find-first-and-last-position-of-element-in-sorted-array) 39 | * [Search in rotated array](https://leetcode.com/problems/search-in-rotated-sorted-array/) 40 | 41 | ## Hashmap 42 | * 463 Island Perimeter 43 | * 535 Encode and decode TinyURL 44 | * 575 Distribute Candies 45 | * 739 Daily Temperatures 46 | * 748 Shortest Completing Word 47 | * 811 Subdomain visit count 48 | * 884 Uncommon words from Twos sentences 49 | * 953 Verifying an Alien Dictionary 50 | * 961 N-Repeated Element in Size 2N Array 51 | * 1160 Find words that can be formed by characters 52 | * 1078 Occurences after bigram 53 | ## Stack 54 | * 496 Next Greater Element I 55 | * 682 Baseball Game 56 | * 856 Score of Parentheses 57 | * 946 Validate Stack Sequences 58 | * 1190 Reverse Substrings Between Each Pair of Parentheses 59 | * 1209 Remove All Adjacent Duplicates in String II 60 | ## String 61 | * 468 Validate IP Address 62 | * 522 Longest Uncommon Subsequence II 63 | * 539 Minimum Time Difference 64 | * 553 Optimal Division 65 | * 791 Custom Sort String 66 | * 833 Find and replace in String 67 | * 835 Image Overlap 68 | * 893 Groups of Special-Equivalent Strings 69 | * 1156 Swap For Longest Repeated Character Substring 70 | ## Sliding Window 71 | * 424 Longest Repeating Character Replacement 72 | * 1040 Moving Stones Until Consecutive II 73 | ## Tree 74 | * 101 Symmetric Tree 75 | * 108 Convert Sorted Array to Binary Search Tree 76 | * 226 Invert Binary Tree 77 | * 538 Convert BST to Greater Tree 78 | * 543 Diameter of a Binary Tree 79 | * 637 Average of Levels in Binary Tree 80 | * 654 Maximum Binary Tree 81 | * 669 Trim a Binary Search Tree 82 | * 589 N-ary Tree Preorder Traversal 83 | * 894 All Possible Full Binary Trees 84 | * 979 Distribute Coins in Binary Tree 85 | * 993 Cousins in a Binary Tree 86 | * 1022 Sum of Root To Leaf Binary Numbers 87 | * 1104 Path In Zigzag Labelled Binary Tree 88 | * 1110 Delete Nodes And Return Forest 89 | * [Ser/deser tree](https://leetcode.com/problems/serialize-and-deserialize-n-ary-tree/) 90 | * [Distributed coins in tree](https://leetcode.com/problems/distribute-coins-in-binary-tree/) 91 | 92 | ## Recursive/Graph/DP 93 | * [Course schedule ii](https://leetcode.com/problems/course-schedule-ii/) 94 | * [Rotting oranges](https://leetcode.com/problems/rotting-oranges/) 95 | * [Maximum depth of tree](https://leetcode.com/problems/maximum-depth-of-n-ary-tree/) 96 | * [Maximum differences between node and ancestor](https://leetcode.com/problems/maximum-difference-between-node-and-ancestor/) 97 | * [Lowest common ancestor in binary tree](https://leetcode.com/problems/lowest-common-ancestor-of-a-binary-tree) 98 | * [Critical connections](https://leetcode.com/problems/critical-connections-in-a-network) 99 | * [Task scheduler](https://leetcode.com/problems/task-scheduler/) 100 | * [Coin change](https://leetcode.com/problems/coin-change/submissions/) 101 | * [Word break ii](https://leetcode.com/problems/word-break-ii/) 102 | * [Word search ii](https://leetcode.com/problems/word-search-ii/) 103 | * [Combination sum iv](https://leetcode.com/problems/combination-sum-iv/) 104 | * [Frog jump](https://leetcode.com/problems/frog-jump/) 105 | * [Number of islands](https://leetcode.com/problems/number-of-distinct-islands) 106 | * [Permutations](https://leetcode.com/problems/permutations) 107 | * [Subset ii](https://leetcode.com/problems/subsets-ii/submissions/) 108 | 109 | 110 | ## LinkedList 111 | * Contrary to popular belief, no body really asked LinkedList questions during the interview. There is one exception where you need to use LinkedList to implement [LRU](https://leetcode.com/problems/lru-cache/). [Copy list with random pointer](https://leetcode.com/problems/copy-list-with-random-pointer/) 112 | 113 | ## Design 114 | * [LRU cache](https://leetcode.com/problems/lru-cache/solution/) 115 | 116 | 117 | 118 | # Notes 119 | * If you're interested to learn more about paid ML system design course with more examples, [click here](course.md). 120 | * If you find this helpful, you can Sponsor this project. It's cool if you don't. 121 | -------------------------------------------------------------------------------- /questions.md: -------------------------------------------------------------------------------- 1 | # Yet another questions list 2 | 3 | * Why do we need the 8209th questions list on github? The questions here sorted by different area and focus on testing your actual knowledge instead of checking your memorization. 4 | * How do you pick questions? A good question can distinguish candidate that **understand** ve candidates that **knows** the concept. There are two level of questions: fundamental and advance, depend on your timeline you might want to priortize your learning accordingly. 5 | 6 | 7 | 8 | | Question | Level | 9 | | ------------- | ------------- | 10 | | Underlying decision tree algorithm, what's the difference between regression and classification? | Fundamental | 11 | | What is the difference between logistic regression vs multilayer perceptron? | Advance | 12 | | [Why L1 sparse?](https://stats.stackexchange.com/questions/45643/why-l1-norm-for-sparse-models) | Advance | 13 | | Why do RNNs have a tendency to suffer from exploding/vanishing gradient? | Fundamental | 14 | | What is the problem with sigmoid during backprop? | Advance | 15 | | What is the problem with Relu? Why do we need Leaky Relu? | Fundamental | 16 | | What are the assumptions of linear/logistic regression? | Fundamental | 17 | | If some features have strong colinearity does it make model performance worse? | Fundamental | 18 | | What does the decision boundary of logistic regression look like? | Fundamental | 19 | 20 | 21 | # List 99 ML questions 22 | ## Basic stats questions 23 | 1. What is the normal distribution and its functions? 24 | 2. What is the variance? 25 | 3. What is stddev? 26 | 4. What is exponential distribution? 27 | 5. What is the binomial distribution? 28 | 6. What is Bernoulli distribution? 29 | 7. What is the multinomial distribution? 30 | 8. What is the lognormal distribution? 31 | 9. What is logistic distribution? 32 | 10. P value and test hypothesis. 33 | 11. Bias coin, unbiased coin posterior probability. Observe the head, what is the likelihood of 34 | coin bias. 35 | 12. What is hypothesis testing? chi-squared test: determine if any statistically significant 36 | difference between expected frequencies and observed frequencies in one or more 37 | categories. More popularly used in medicine. 38 | 13. What is the gamma distribution and its functions? 39 | 14. Poisson distribution and its function. 40 | 15. What is the Central Limit Theorem and why is it important? 41 | 16. What is sampling? How many sampling methods do you know? 42 | 17. Basic Bayes Theorem. P(A|B) = P(B|A) * P(A) / P(B). Conditional prob and marginal prob 43 | 18. What is an example of a data set with a non-Gaussian distribution? 44 | 19. Given a generator of unbiased Bernoulli numbers (0 or 1 with p=0.5), create a biased Bernoulli trial generator (generate 0 or 1 with the specified 0 < p < 1) 45 | ## Machine Learning fundamentals 46 | 20. Explain collinearity: how is it possible to have negative coefficient in glm? 47 | 21. Assumptions about linear regression: 3 residual errors follow a normal distribution and 48 | independent with each other, output and input have linear distribution, low collinearity 49 | between input features. 50 | 22. Explain the concept of bias vs variance 51 | 23. Assumptions about logistic regression? 52 | 24. Overfitting how to avoid 53 | 25. Type of regularization, which one easier to use: L1 and L2. L1 is easier. 54 | 26. What are the different metrics to classify the dataset? 55 | 27. explain bagging vs boosting 56 | 28. Is random forest bagging or boosting?. 57 | 29. how to handle unbalanced data. 58 | 30. how do you inspect missing data and when are they important 59 | 31. how to compare two regressions. 60 | 32. What is k-mean? What is kmean loss function? what kind of distance metric would you choose, what if different features have a different dynamic range Kmeans is an algorithm to group similar data points into the same groups. The loss function used L2 norm: sum 61 | of the square of the distance to the centroid. Similar to the Expectation-Maximization algorithm. 62 | 33. general ML questions like generative v.s. 63 | 34. What is the difference between type I vs type II error? 64 | 35. What is the linear regression? What do the terms p-value, coefficient, and r-squared 65 | value mean? What is the significance of each of these components? 66 | 36. What is selection bias? It is the bias introduced by the selection from groups that is not 67 | achieved by randomness. 68 | 37. What is regularization? How does it solve bias, variance problems? 69 | 38. What is the learning curve tool? 70 | 39. What is the meaning of the contour visualization of the cost function in linear regression? 71 | 40. Explain logistic regression? 72 | a. Why is the name logistic function? 73 | b. Interpretation of logistic function 74 | i. Why do we need continuous output of the logistics function? 75 | c. What does the decision boundary look like? 76 | d. What kind of normalization is needed? 77 | e. Relationship with cross-entropy 78 | f. What are the other logit families? 79 | 41. How does the optimization algorithm work i.e to minimize the cost function? 80 | a. Gradient descent, Conjugate gradient, BFGS, L-BFGS 81 | 42. Explain SVM 82 | a. What is the decision boundary? 83 | b. What is a kernel trick? 84 | c. What is the cost function? 85 | d. How to tune hyperparameters to avoid underfit, overfit? 86 | i. kernel choice 87 | ii. c parameter 88 | iii. Math formula 89 | 43. Explain the decision tree 90 | a. Why do we need a decision tree? 91 | b. How does it work? How do we split? How do we select which feature to split? 92 | c. What is the meaning of the parameters? 93 | d. What is the cost function? Why do we need that cost function? And in which case 94 | do we need which 95 | e. What does tree pruning mean? 96 | 44. explain random forest 97 | a. How does it work? 98 | b. How to tune parameters to avoid underfit/overfit? 99 | c. Is it an ensemble of bagging? 100 | 45. Explain boosting 101 | a. Basic intuition 102 | b. When to do it? 103 | c. What problem does it solve? 104 | d. What are the parameters? 105 | e. How to tune? 106 | 46. Explain PCA. What is physical intuition? 107 | 47. Explain MLP: kind of fully connected feedforward neural network that use sigmoid or tanh as activation functions, widely used for classification task similar to logistic regression. 108 | 48. Explain MLE 109 | 49. Explain MAP 110 | 50. what is cross-validation: the purpose is to estimate how a model performs on unseen data by partition data set into many sets and pick a model with higher 111 | accuracy on the validation set. 112 | 51. SVM: how to choose the model and how to determine if a model is better than another 113 | 52. Explain how you select the best model? 114 | 53. Explain TPE hyperparameter optimization. explain hyper optimization: BayesOptimization 115 | 54. pros and cons of random forest and why. Pros: effective, better than tree, effective feature importance. Cons: not easy to interpret, prone to out of range input issue, less 116 | effective with big data or nonsmooth assumption. 117 | 55. How GMM works (EM algorithm) 118 | 56. When using the Gaussian mixture model, how do you know it is applicable? 119 | 57. Describe some criteria for model selection? Why is dimension reduction important? 120 | 58. How do you choose the best model among all possible models? explain neural network assumptions of linear regression models if you have a large number of predictors how do you handle them? 121 | 59. What is the bias-variance tradeoff? How is XGBoost handling bias-variance tradeoff? 122 | 60. How do you find an anomaly in a distribution? How do you investigate that a certain trend in distribution is due to anomaly? 123 | ## Deep learning fundamentals 124 | 61. Explain all standard activation functions? tanh, relu, leakyRelu, sigmoid, softmax, softplus 125 | 62. How does a neural network with one layer and one input and output compare to logistic regression? It’s the same with logistic regression if the NN is using sigmoid function and logistic regression. 126 | 63. Explain RNN Problem. How does LSTM help solve it? 127 | ## Companies Interview questions: 128 | 64. Netflix question – When you split a population for A/B testing, what are some reasons you could see a significant difference in the control and variant groups? 129 | 65. Google question – Find the width of the confidence interval 130 | 66. Facebook question – If you draw 2 cards from a shuffled 52 card deck, what is the probability that you’ll have a pair? 131 | 67. Microsoft question – Generate 7 integers with equal probability from a function which returns 1/0 with probability p and (1-p). 132 | 68. Given an unfair coin with the probability of heads not equal to .5, what algorithm could 133 | you use to create a list of random 1s and 0s? 134 | 69. Groupon question – You are on a number line and you can jump to one of the neighboring points with equal probability, with the exception of n=0 where you can’t go to negative numbers but have to come back to n=1. If you start at n=44, what is the expected number of steps to reach n=4444? 135 | 70. CA Technologies question – How do you design an algorithm for fraud detection? 136 | 71. LinkedIn question – Come up with some of the factors that could be used to produce certain algorithms (‘people you may know,’ and an algorithm to discover when a person is starting to search for new job). PYMK: uses many features to calculate a connection probability among two people. Some examples are: companies position overlap (How 137 | many months/years both individuals work together at same company, similar/related 138 | field), school education overlap, strong unconnected people in LinkedIn connection graph. (both individuals have many common friends) etc. 139 | 72. An important metric goes down, how would you dig into the causes? 140 | 73. Amazon question – Estimate the cumulative sum of the top 10 most profitable products 141 | of the last 6 months for customers in Seattle. 142 | 74. Booking question – How can we automatically propose ‘good value deals’ to customers, 143 | including hotels that don’t have a rating yet? 144 | 75. If you have a customer and want to decide whether they will “buy today” or “not buy 145 | today” and you know 1. where they live, 2. their income, 3. their gender, 4. their 146 | profession, how would you define a machine-learning algorithm to figure this out? 147 | 76. LinkedIn question – How would you design an A/B test for the homepage? 148 | 77. Netflix question – Given a month’s worth of login data from Netflix such as account_id, 149 | device_id, and metadata concerning payments, how would you detect fraud? (identity 150 | theft, payment fraud, etc.) 151 | 78. Booking question – How would you tag a listing as value for money? How would you 152 | measure the “value”? What features would you select to explain the “value”? 153 | 79. Apple question – How do you take millions of users with 100s of transactions each, 154 | amongst 10ks of products and group the users together in a meaningful segment? 155 | 80. Facebook question – How many high schools that people have listed on their profiles are 156 | real? How do we find out, and deploy at scale, a way of finding invalid schools? 157 | 81. We have a product that is getting used differently by two different groups. What is your 158 | hypothesis about why and how would you go about testing it? 159 | 82. Intuit question – How would you design a ranking system? 160 | 83. Uber question – Explain how network effects might influence your choice of how to 161 | assign experimental/control units and measure your main outcome metrics 162 | 84. LinkedIn question – What product metrics do you construct? How do you tell if your experiment is successful? 163 | 85. What trends in the data indicate that a given market is healthy? What does price tell you? 164 | 86. Facebook question – Given a series of tables; write the SQL code you would need to count subpopulations through joins. 165 | 87. Pinterest question – Write a SQL query to count the number of unique users per day who logged in from both an iPhone and the web, where iPhone logs and weblogs are in 166 | distinct relations. 167 | 88. Spotify question – Given a sample set of tables, write a SQL query to get a summary metric from those tables. 168 | 89. Twitter question – How can you illustrate a tree-based system with a SQL query? 169 | 90. If you have a table with a billion rows, how would you add a column inserting data from 170 | the original source without affecting the user experience? 171 | 91. Facebook question – There is a table that tracks every time a user turns a feature on or off, with columns for user_id, action (“on” or “off), date, and time. How many users turned the feature on today? How many users have ever turned the feature on? In a table that tracks the status of every user every day, how would you add today’s data to it? 172 | 92. Check if an integer is a palindrome (do not convert the integer to string) 173 | 93. Given 2 sorted arrays of integers, code to find a number from each array such that their 174 | sum is closest to some integer K 175 | 94. Amazon question – Write a Python function that displays the first n Fibonacci numbers. 176 | 95. Cisco question – Merge 2 sorted linked list 177 | 96. eBay question – Given a function roll() that uniformly returns a double between 0 and 1 and an array/list of numbers of length N (no duplicates), create a function shuffle() that returns a permutation of equal probability. 178 | 97. How would you create/design/implement a certain algorithm from start to end? 179 | 98. LinkedIn question – Given a random generator that produces a number 1 to 5 uniformly, 180 | write a function that produces a number from 1 to 7 uniformly 181 | 99. Generate a sorted vector from two sorted vectors. 182 | 183 | # Other question list 184 | * [List of ML questions](https://github.com/Sroy20/machine-learning-interview-questions/blob/master/list_of_questions_machine_learning.md) 185 | * [List of DL questions](https://github.com/Sroy20/machine-learning-interview-questions/blob/master/list_of_questions_deep_learning.md) 186 | * [Other interview questions](https://github.com/Feynman687/Interviews/blob/master/StatML.md) 187 | 188 | * If you find this helpful, you can Sponsor this project. It's cool if you don't. 189 | 190 | 191 | -------------------------------------------------------------------------------- /quiz.md: -------------------------------------------------------------------------------- 1 | # Machine Learning Quiz 2 | 3 | ## List of quizzes 4 | 5 | | Quiz | Release date | Number of people took quiz | Top 5% score | Solution | 6 | | ------------- | ------------- |------------- |------------- |------------- | 7 | |ML quiz, upcoming | TBD | | | | 8 | |[ML quiz 531 (difficult level)](https://forms.gle/dLK8TE6NLoWNU1LV6)| 5/30/2020 | 34 | 9 | | 9 | |[ML quiz 12 (medium level)](https://forms.gle/aieiHULBTHRYpNH59)| April 2020 | 173 | 9 | [solution](https://rebrand.ly/ml_solution) 10 | 11 | If you like these quizzes and want to have more in the future, make [one time donation](https://www.paypal.com/paypalme/MLinterviews) to support me making more quizzes. 12 | 13 | ## Quiz statistics 14 | * If you can score 9/10 corrected questions, you are quite ready for ML interview. I have prepared ML tests based on real questions at big companies i.e: FAAG, Linkedin, Snapchat etc... Send an email to helppreparemle@gmail.com for solution. 15 | 16 | * ML quiz 12 statistics 17 | ![Score distribution](images/score1.png) 18 | 19 | # Notes 20 | * If you're interested to learn more about paid ML system design course with more examples, [click here](course.md). 21 | * If you find this helpful, you can [Sponsor](https://github.com/sponsors/khangich) this project. It's cool if you don't. 22 | 23 | -------------------------------------------------------------------------------- /sample/backprop.py: -------------------------------------------------------------------------------- 1 | #source https://towardsdatascience.com/lets-code-a-neural-network-in-plain-numpy-ae7e74410795 2 | def full_backward_propagation(Y_hat, Y, memory, params_values, nn_architecture): 3 | grads_values = {} 4 | m = Y.shape[1] 5 | Y = Y.reshape(Y_hat.shape) 6 | 7 | dA_prev = - (np.divide(Y, Y_hat) - np.divide(1 - Y, 1 - Y_hat)); 8 | 9 | for layer_idx_prev, layer in reversed(list(enumerate(nn_architecture))): 10 | layer_idx_curr = layer_idx_prev + 1 11 | activ_function_curr = layer["activation"] 12 | 13 | dA_curr = dA_prev 14 | 15 | A_prev = memory["A" + str(layer_idx_prev)] 16 | Z_curr = memory["Z" + str(layer_idx_curr)] 17 | W_curr = params_values["W" + str(layer_idx_curr)] 18 | b_curr = params_values["b" + str(layer_idx_curr)] 19 | 20 | dA_prev, dW_curr, db_curr = single_layer_backward_propagation( 21 | dA_curr, W_curr, b_curr, Z_curr, A_prev, activ_function_curr) 22 | 23 | grads_values["dW" + str(layer_idx_curr)] = dW_curr 24 | grads_values["db" + str(layer_idx_curr)] = db_curr 25 | 26 | return grads_values -------------------------------------------------------------------------------- /sample/logistic_regression.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "%matplotlib inline\n", 12 | "import numpy as np\n", 13 | "import matplotlib.pyplot as plt\n", 14 | "import seaborn as sns\n", 15 | "from sklearn import datasets" 16 | ] 17 | }, 18 | { 19 | "cell_type": "code", 20 | "execution_count": 2, 21 | "metadata": { 22 | "collapsed": true 23 | }, 24 | "outputs": [], 25 | "source": [ 26 | "iris = datasets.load_iris()" 27 | ] 28 | }, 29 | { 30 | "cell_type": "code", 31 | "execution_count": 3, 32 | "metadata": { 33 | "collapsed": true 34 | }, 35 | "outputs": [], 36 | "source": [ 37 | "X = iris.data[:, :2]\n", 38 | "y = (iris.target != 0) * 1" 39 | ] 40 | }, 41 | { 42 | "cell_type": "code", 43 | "execution_count": 4, 44 | "metadata": {}, 45 | "outputs": [ 46 | { 47 | "data": { 48 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlYAAAFpCAYAAABeYWb6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X+MJOWd3/HPd7vXWo/PDrEZx2SHmcECIS/Ix7FjDv8Q\nIuw6NmuE/7GsReNYtv+Y2x5y4ZKzfHFWQrqTkBwpOvkwN+tMjBJZOwe6+PAPIUAJe1zi/GFzs4Ax\nC0bmzjvLbMh5byVDyJizd/abP6qHme2tnu6ufrqqnqr3Syr11NNVXU899XTpO91PP19zdwEAAGB4\nO4quAAAAQFUQWAEAAARCYAUAABAIgRUAAEAgBFYAAACBEFgBAAAEQmAFAAAQCIEVAABAIARWAAAA\ngfQdWJlZw8yeNrOHU5672cxeNbNn2svdYasJAABQfs0Btr1L0guS3tHl+e+7+239vtill17q09PT\nAxweAACgGMePH/97dx/vtV1fgZWZTUj6hKR7JP2bIesmSZqentby8nKIlwIAABgpM1vpZ7t+vwr8\nqqQvSTq/zTYfMrNnzexRM7umz9cFAACojJ6BlZndJunn7n58m82ekjTp7u+X9DVJ3+nyWnNmtmxm\ny2fOnMlUYQAAgLLq5xOrD0u63cxOSnpQ0i1mdnTrBu7+mru/3v77EUk7zezSzhdy90V3n3H3mfHx\nnl9TAgAARKXnGCt3/7KkL0vJr/8kfdHdP7N1GzN7j6S/c3c3sxuUBGxnw1cXAADE6te//rVWV1f1\nxhtvFF2Vrnbt2qWJiQnt3Lkz0/6D/CrwAmZ2SJLc/euSPiWpZWbnJP1S0kF396yvDQAAqmd1dVVv\nf/vbNT09LTMrujoXcXedPXtWq6uruuKKKzK9xkCBlbv/laS/av/99S3l90m6L1MNAABALbzxxhul\nDaokycz0rne9S8OMA2fmdQAAkJuyBlUbhq0fgRUAAKiNxx57TFdffbWuvPJKfeUrXwn++gRWAACg\nFtbX13XnnXfq0Ucf1fPPP68HHnhAzz//fNBjEFgBAIBSWlqSpqelHTuSx6Wl4V7vySef1JVXXqn3\nvve9estb3qKDBw/qu9/9boiqvonACgAAlM7SkjQ3J62sSO7J49zccMHV6dOndfnll7+5PjExodOn\nTweo7SYCK6AmQv/nBwCjdPiwtLZ2YdnaWlJeZpnnsQIQj43//DZuUhv/+UnS7Gxx9QKAbk6dGqy8\nH7t379bLL7/85vrq6qp2796d/QVT8IkVUAOx/ucHoL4mJwcr78cHPvAB/fSnP9XPfvYz/epXv9KD\nDz6o22+/PfsLpiCwAmpgFP/5AcAo3XOPNDZ2YdnYWFKeVbPZ1H333aePfexjet/73qdPf/rTuuaa\na4araOcxgr4agFKanEy+/ksrB4Ay2himcPhw8k/g5GQSVA07fOHAgQM6cODA8BXsgk+sgBoYxX9+\nADBqs7PSyZPS+fPJYwxjQgmsgBqYnZUWF6WpKckseVxcjOMmBQAx4atAoCZmZwmkAGDU+MQKAAAg\nEAIrAACAQAisAAAAAiGwAgAAtfGFL3xB7373u3XttdeO5PUJrAAAQG187nOf02OPPTay1yewAgAA\n5TSC7PE33XST3vnOdw79Ot0w3QIAACifSLPH84kVAAAon0izxxNYAQCA8ok0ezyBFQAAKJ9uWeJL\nnj2ewAoAAJTPiLLH33HHHfrgBz+oF198URMTE7r//vuHer1ODF4HAADlszFA/fDh5Ou/yckkqBpy\n4PoDDzwQoHLdEVgBAIByijB7PF8FAgAABEJgBQAAEAiBFQAAyI27F12FbQ1bPwIroCAjyNQAAKW2\na9cunT17trTBlbvr7Nmz2rVrV+bXYPA6UIBIMzUAwFAmJia0urqqM2fOFF2Vrnbt2qWJiYnM+1tR\nUePMzIwvLy8XcmygaNPTSTDVaWpKOnky79oAAHoxs+PuPtNrO74KBAoQaaYGAEAPBFZAASLN1AAA\n6IHACijAiDI1AAAKRmAFFGB2VlpcTMZUmSWPi4sMXAeA2PGrQKAgEWZqAAD0wCdWAAAAgRBYAQAA\nBEJgBQAAEAiBFQAAQCAEVqgccvABAIrCrwJRKeTgAwAUiU+sUCmHD28GVRvW1pJyAABGjcAKlUIO\nPgBAkQisUCnk4AMAFInACpVCDj4AQJEIrFAp5OADABSJXwWicsjBBwAoCp9YAQAABEJgBQAAEAiB\nFQAAQCB9B1Zm1jCzp83s4ZTnzMzuNbOXzOxZM7s+bDUBAADKb5BPrO6S9EKX526VdFV7mZN0ZMh6\nASgJci8CQP/6CqzMbELSJyR9o8smn5T0TU/8QNIlZnZZoDoCKMhG7sWVFcl9M/ciwRUApOv3E6uv\nSvqSpPNdnt8t6eUt66vtMgARI/ciAAymZ2BlZrdJ+rm7Hx/2YGY2Z2bLZrZ85syZYV8OwIiRexEA\nBtPPJ1YflnS7mZ2U9KCkW8zsaMc2pyVdvmV9ol12AXdfdPcZd58ZHx/PWGUAeSH3IgAMpmdg5e5f\ndvcJd5+WdFDSX7r7Zzo2+56kz7Z/HXijpFfd/ZXw1QWQJ3IvAsBgMs9jZWaHzOxQe/URSX8r6SVJ\n/0nSfIC6ASgYuRcBYDDm7oUceGZmxpeXlws5NgAAwCDM7Li7z/TajpnXAQAAAiGwAgAACITACgAA\nIBACKwAAgEAIrAAAAAIhsAKGND8vNZvJdATNZrIOAKinZtEVAGI2Py8dObK5vr6+ub6wUEydAADF\n4RMrYAiLi4OVAwCqjcAKGML6+mDlAIBqI7AChtBoDFYOAKg2AitgCHNzg5UDAKqNwevAEDYGqC8u\nJl//NRpJUMXAdQCoJwIrYEgLCwRSAIAEXwUCAAAEQmAFAAAQCIEVAABAIARWAAAAgRBYobT270/y\n720s+/cXXaO4LS1J09PSjh3J49JS0TUCUCl532RKelMjsEIp7d8vHTt2YdmxYwRXWS0tJdNArKxI\n7snj3Fxp7kMAYpf3TabENzVz90IOPDMz48vLy4UcG+Vn1v25grps1Kank/tOp6kp6eTJvGsDoHLy\nvskUcFMzs+PuPtNrOz6xAmrg1KnBygFgIHnfZEp8UyOwAmpgcnKwcgAYSN43mRLf1AisUEr79g1W\nju3dc480NnZh2dhYUg4AQ8v7JlPimxqBFUrp8ccvDqL27UvKMbjZ2SSf4dRUMn5taipZn50tumYA\nKiHvm0yJb2oMXgcAAOiBwesAAAA5I7ACAAAIhMAKAAAgEAIrAACAQAisUFqxpJ0qaboqAEABmkVX\nAEizkQZqbS1Z30gDJY3m17RZj5d3PQEA5cZ0CyilWNJOkYMPAOqB6RYQtVjSTpU4XRUAoAAEViil\nWNJOlThdFQCgAARWKKVY0k6VOF0VAKAABFYopVjSTpU4XRUAoAAMXgcAAOiBwesAAAA5I7ACAAAI\nhMAKAAAgEAIrAACAQAisClaHPHN1OEcANcENDT2QK7BAdcgzV4dzBFAT3NDQB6ZbKFAd8szV4RwB\n1AQ3tFpjuoUI1CHPXB3OEUBNcENDHwisClSHPHN1OEcANcENDX0gsCpQHfLM1eEcAdQENzT0gcCq\nQHXIM1eHcwRQE9zQ0AcGrwMAAPTA4HUAAICcEVgBAAAEQmAFAAAQSM/Aysx2mdmTZvYjMzthZn+Y\nss3NZvaqmT3TXu4eTXUBAADKq59PrP5B0i3u/puSrpP0cTO7MWW777v7de3lj4LWElGbn5eazeRH\nNM1msj7K/WJJ5RVLPQEA/euZK9CTnw2+3l7d2V6K+SkhojM/Lx05srm+vr65vrAQfr9YUnnFUk8A\nwGD6mm7BzBqSjku6UtKfuvsfdDx/s6SHJK1KOi3pi+5+YrvXZLqFemg2k6CoU6MhnTsXfr9YUnnF\nUk8AQCLodAvuvu7u10makHSDmV3bsclTkibd/f2SvibpO10qNWdmy2a2fObMmX4OjcilBUfblQ+7\nXyypvGKpJwBgMAP9KtDdfyHpCUkf7yh/zd1fb//9iKSdZnZpyv6L7j7j7jPj4+NDVBuxaDQGKx92\nv1hSecVSTwDAYPr5VeC4mV3S/vutkj4q6Scd27zHzKz99w3t1z0bvrqIzca4oX7Lh90vllResdQT\nADCYfj6xukzSE2b2rKS/lvTf3f1hMztkZofa23xK0nNm9iNJ90o66EXlykGpLCxIrdbmJ02NRrK+\n3QD0YfaLJZVXLPUEAAyGXIEAAAA9kCsQAAAgZwRWAAAAgRBYAQAABEJgBQAAEAiBFQAAQCAEVgXL\nOxFv1sTGRRwza9vUoU0rLe8LGEtHAxAHdy9k2bt3r9fd0aPuY2Pu0uYyNpaUj0KrdeGxNpZWazTH\nG+aYWdumDm1aaXlfwFg6GoDCSVr2PuIb5rEqUN6JeLMmNi7imFnbpg5tWml5X8BYOhqAwvU7jxWB\nVYF27Ej+1e1kJp0/H/54SdKhdKPqBlmPmbVt6tCmlZb3BYylowEoHBOERiDvRLxZExsXccysbVOH\nNq20vC9gLB0NQDQIrAqUdyLerImNizhm1rapQ5tWWt4XMJaOBiAe/QzEGsXC4PXE0aPuU1PuZsnj\nqMe+tlrujUYy1rbRyGeQddZjZm2bOrRppeV9AWPpaAAKJQavAwAAhMEYKwAAgJwRWAEAAARCYAUA\nABAIgRUAAEAgBFYYOVKxoZKqniSy6ucHjEiz6Aqg2paWkjmd1taS9ZWVzTmeZmfD7wfkYn5eOnJk\nc319fXN9YaGYOoVU9fMDRojpFjBSpGJDJVU9SWTVzw/IgOkWUAqnTg1WPux+QC7Sgo7tymNT9fMD\nRojACiNFKjZUUtWTRFb9/IARIrDCSJGKDZVU9SSRVT8/YIQIrDBSs7PS4mIyNsoseVxc7D0APet+\nQC4WFqRWa/MTnEYjWa/KwO6qnx8wQgxeBwAA6IHB6wAAADkjsAIAAAiEwAoAACAQAisAAIBACKw6\n5J2fLuvxYkrjRc6/muLCp8v65o2lPfNODhrLTRv14e6FLHv37vWyOXrUfWzMXdpcxsaS8jIdr9W6\ncJ+NpdUaTT2HkXeboiS48Omyvnljac+s9cx7v6xiuQ4YCUnL3kd8w3QLW+Sdny7r8WJK40XOv5ri\nwqfL+uaNpT3zTg4ay00bldDvdAsEVlvs2JH8C9LJTDp/vjzHM+v+XEGXs6u82xQlwYVPl/XNG0t7\nZq1n3vtlFct1wEgwj1UGeeeny3q8mNJ4kfOvprjw6bK+eWNpz7yTg8Zy00atEFhtkXd+uqzHiymN\nFzn/aooLny7rmzeW9sw7OWgsN23USz8DsUaxlHHwunsyBnFqyt0seRz1mMSsx2u13BuNZOxko1HO\ngesb8m5TlAQXPl3WN28s7Zm1nnnvl1Us1wHBicHrAAAAYTDGCgAAIGcEVgAAAIEQWAEAAARCYAUA\nABAIgVXBYkpztX9/Mg/exrJ//6hqCVRYLLnm8n7Dc4NBRRBYFWhpKZm+ZmUlmcx3ZSVZH9V9dpjj\n7d8vHTt2YdmxY9z7gIHk/abPKu83PDcYVAjTLRQopjRXMaXRAUorllxzeb/hucEgAky3EIFTpwYr\nj+14ADrwJgQqj8CqQKS5AmqGNyFQeQRWBYopzdW+fYOVA0gRS665vN/w3GBQIQRWBZqdlRYXk+EV\nZsnj4mJSXrbjPf74xfe4ffuScgB9yvtNn1Xeb3huMKgQBq8DAAD0wOB1AACAnBFYAQAABEJgBQAA\nEEjPwMrMdpnZk2b2IzM7YWZ/mLKNmdm9ZvaSmT1rZtePproAAADl1c8nVv8g6RZ3/01J10n6uJnd\n2LHNrZKuai9zko4ErWUGWdNxxZLGK6v5eanZTH6Q1Gwm6/2iTbvI+wSzXsSq13OYzh2DrLn08m7P\nWN7w3NDSDXN+VW+bfrl734ukMUlPSfrtjvL/KOmOLesvSrpsu9fau3evj8rRo+5jY+5JLoRkGRtL\nykexXyxarQvPbWNptXrvS5t2kfcJZr2IVa/nMJ07Bvv2pZ/fvn3b75d3e8byhueGlm6Y86t627i7\npGXvJ1bqayOpIekZSa9L+vcpzz8s6SNb1o9JmtnuNUcZWE1Npd8TpqZGs18sGo3082s0eu9Lm3aR\n9wlmvYhVr+cwnTsGaee2sWwn7/aM5Q3PDS3dMOdX9bZx7zuwGmgeKzO7RNK3Jf2uuz+3pfxhSV9x\n9//VXj8m6Q/cfblj/zklXxVqcnJy70paMtIAduxIrujF9ZfOnw+/XyyGyXNKm3aR9wlmvYhVr2fV\nk/jG0p6xvOG5oaUb5vyq3jYa0TxW7v4LSU9I+njHU6clXb5lfaJd1rn/orvPuPvM+Pj4IIceSNZ0\nXFVP49VoDFa+FW3aRd4nmPUiVr2ew3TuKsu7PWN5w3NDSzfM+VW9bQbQz68Cx9ufVMnM3irpo5J+\n0rHZ9yR9tv3rwBslverurwSvbZ+ypuOKJY1XVnNzg5VvRZt2kfcJZr2IVa/nMJ07Bllz6eXdnrG8\n4bmhpRvm/KreNoPo9V2hpPdLelrSs5Kek3R3u/yQpEPtv03Sn0r6G0k/Vo/xVe6jHWPlnoyXm5py\nN0se+x0/l3W/WLRam8MnGo3BxvbSpl3kfYJZL2LV6zlM545B5wD2XgPXN+TdnrG84bmhpRvm/Cre\nNhrFGKuQyBUIAABiQa5AAACAnBFYAQAABEJgBQAAEAiBFQAAQCAEVgAAAIEQWHUghyRKr+rJY/M+\nv1j2yyqW6151XIf66GdOhlEso57HKosa5JBE7KqePDbv84tlv6xiue5Vx3WoBDGP1eCmp6W09IVT\nU9LJk3nXBkiRtZPG0rnzPr9Y9ssqlutedVyHSuh3HisCqy1qkEMSsat68ti8zy+W/bKK5bpXHdeh\nEpggNANySKL0qp48Nu/zi2W/rGK57lXHdagVAqstyCGJ0qt68ti8zy+W/bKK5bpXHdehXvoZiDWK\npYyD190rn0MSVVD15LF5n18s+2UVy3WvOq5D9MTgdQAAgDAYYwUAAJAzAisAAIBACKwAAAACIbAC\nAAAIhMAKiE3WnGPz81KzmUxK2Gwm66M8Xt7yPj+uQ7pY6hmLmNozprqOUj8/HRzFUtbpFoBSy5pz\nrNW6cJ+NpdUazfHylvf5cR3SxVLPWMTUnjHVNSMx3QJQQVlzjjWb0vr6xeWNhnTuXPjj5S3v8+M6\npIulnrGIqT1jqmtG5AoEqihrzjGz7s9tdw+IJcdZ3ufHdUgXSz1jEVN7xlTXjJjHCqiirDnHGo3B\nyoc9Xt7yPj+uQ7pY6hmLmNozprqOGIEVEJOsOcfm5gYrH/Z4ecv7/LgO6WKpZyxias+Y6jpq/QzE\nGsXC4HUgo6w5x1ot90YjGVTaaPQeMD3s8fKW9/lxHdLFUs9YxNSeMdU1AzF4HQAAIAzGWAEAAOSM\nwAoAACAQAisAAIBACKwAAAACIbBC9VQ9X1Xe53fNNckkfxvLNdeM9nixXL+8cwwCiEM/Px0cxcJ0\nCxiJqueryvv89uxJz223Z89ojhfL9cs7xyCAwonpFlBLVc9Xlff5ZU3BklUs1y/vHIMACkeuQNRT\n1fNV5X1+eQdWsVy/vHMMAigc81ihnqqer4rzK4e8cwwCiAaBFaql6vmq8j6/PXsGKx9WLNcv7xyD\nAKJBYIVqmZ2VFheTMStmyePiYlJeBXmf34kTFwdRe/Yk5aMQy/XLWs9Yzg9AZoyxAgAA6IExVgAA\nADkjsAIAAAiEwAoAACAQAisAAIBACKyAouSdM67que1iqSdQVbwHJUnNoisA1NLSkjQ3J62tJesr\nK8m6NJqf3mc9Xt71zCqWegJVxXvwTUy3ABQh75xxVc9tF0s9gaqqwXuQXIFAmeWdM67que1iqSdQ\nVTV4DzKPFVBmeeeMq3puu1jqCVQV78E3EVgBRcg7Z1zVc9vFUk+gqngPvonACihC3jnjqp7bLpZ6\nAlXFe/BNjLECAADogTFWAAAAOSOwAgAACITACgAAIJCegZWZXW5mT5jZ82Z2wszuStnmZjN71cye\naS93j6a6AAAA5dXPJ1bnJP2+u++RdKOkO81sT8p233f369rLHwWtJcohltx2scg7d1/V2zNv8/NS\ns5n8AqrZTNarhP4CZOPuAy2Svivpox1lN0t6eJDX2bt3ryMiR4+6j425J3PrJsvYWFJehePlLev5\n5b0f0rVaF7blxtJqFV2zMOgvwEUkLXsf8c1A0y2Y2bSk/ynpWnd/bUv5zZIekrQq6bSkL7r7ie1e\ni+kWIhNLbrtY5J27r+rtmbdmU1pfv7i80ZDOncu/PqHRX4CLBM8VaGa/Iel/SLrH3R/qeO4dks67\n++tmdkDSn7j7VSmvMSdpTpImJyf3rqS9cVFOseS2i0Xeufuq3p55M+v+XEFzAwZFfwEuEnQeKzPb\nKekvJC11BlWS5O6vufvr7b8fkbTTzC5N2W7R3WfcfWZ8fLyfQ6MsYsltF4u8c/dVvT3z1mgMVh4b\n+guQWT+/CjRJ90t6wd3/uMs272lvJzO7of26Z0NWFAWLJbddLPLO3Vf19szb3Nxg5bGhvwDZ9RqE\nJekjklzSs5KeaS8HJB2SdKi9zb+UdELSjyT9QNKHer0ug9cjdPSo+9SUu1nyOOqBrHkfL29Zzy/v\n/ZCu1XJvNJKB3Y1GdQaub6C/ABfQKAavh8TgdQAAEAtyBQIAAOSMwAoAACAQAisAAIBACKwAAAAC\nIbACAAAIhMAK/SMpa1hVT+ILADXULLoCiMTSUjL54dpasr6ysjkZ4uxscfWK1fy8dOTI5vr6+ub6\nwkIxdQIADI15rNAfkrKGVfUkvgBQMcxjhbBOnRqsHNtLC6q2KwcARIHACv0hKWtYVU/iCwA1RWCF\n/pCUNayqJ/EFgJoisEJ/ZmelxcVkTJVZ8ri4yMD1rBYWpFZr8xOqRiNZZ+A6AESNwesAAAA9MHgd\nAAAgZwRWAAAAgRBYAQAABEJgBQAAEAiBVSCk0dsGjZOOdokb1w9ACnIFBkAavW3QOOlol7hx/QB0\nwXQLAZBGbxs0TjraJW5cP6B2+p1ugcAqgB07pLRmNJPOn8+/PqVC46SjXeLG9QNqh3msckQavW3Q\nOOlol7hx/QB0QWAVAGn0tkHjpKNd4sb1A9AFgVUApNHbBo2TjnaJG9cPQBeMsQIAAOiBMVYAAAA5\nI7ACAAAIhMAKAAAgEAIrAACAQAisAJTL/LzUbCa/tms2k/VRIucfgIDIFQigPObnpSNHNtfX1zfX\nFxbCH4+cfwACY7oFAOXRbCbBVKdGQzp3LvzxyPkHoE9MtwAgPmlB1Xblwzp1arByAOiBwApAeTQa\ng5UPi5x/AAIjsAJQHhvjm/otHxY5/wAERmAFoDwWFqRWa/MTqkYjWR/FwHWJnH8AgmPwOgAAQA8M\nXgcAAMgZgRUAAEAgBFYAAACBEFgBAAAEQmAFAAAQCIEVAABAIARWAAAAgRBYAQAABEJgBQAAEAiB\nFQAAQCAEVgAAAIEQWAEAAARCYAUAABAIgRUAAEAgBFYAAACB9AyszOxyM3vCzJ43sxNmdlfKNmZm\n95rZS2b2rJldP5rqAgAAlFc/n1idk/T77r5H0o2S7jSzPR3b3CrpqvYyJ+lI0FoibktL0vS0tGNH\n8ri0VHSNAAAYiZ6Blbu/4u5Ptf/+v5JekLS7Y7NPSvqmJ34g6RIzuyx4bRGfpSVpbk5aWZHck8e5\nOYIrAEAlDTTGysymJf2WpB92PLVb0stb1ld1cfCFOjp8WFpbu7BsbS0pBwCgYvoOrMzsNyT9haTf\nc/fXshzMzObMbNnMls+cOZPlJRCbU6cGKwcAIGJ9BVZmtlNJULXk7g+lbHJa0uVb1ifaZRdw90V3\nn3H3mfHx8Sz1RWwmJwcrBwAgYv38KtAk3S/pBXf/4y6bfU/SZ9u/DrxR0qvu/krAeiJW99wjjY1d\nWDY2lpQDAFAxzT62+bCkfyHpx2b2TLvs30malCR3/7qkRyQdkPSSpDVJnw9fVURpdjZ5PHw4+fpv\ncjIJqjbKAQCoEHP3Qg48MzPjy8vLhRwbAABgEGZ23N1nem3HzOsAAACBEFgBAAAEQmAFAAAQCIEV\nAABAIARWAAAAgRBYAQAABEJgBQAAEAiBFQAAQCAEVgAAAIEQWAEAAARCYAUAABAIgRUAAEAgBFYA\nAACBmLsXc2CzM5JWCjn4aF0q6e+LrkRJ0TbpaJd0tEs62iUd7ZKOdulu0LaZcvfxXhsVFlhVlZkt\nu/tM0fUoI9omHe2SjnZJR7uko13S0S7djapt+CoQAAAgEAIrAACAQAiswlssugIlRtuko13S0S7p\naJd0tEs62qW7kbQNY6wAAAAC4RMrAACAQAishmBmDTN72sweTnnuZjN71cyeaS93F1HHvJnZSTP7\ncfucl1OeNzO718xeMrNnzez6IupZhD7apq595hIz+5aZ/cTMXjCzD3Y8X8s+00e71K6/mNnVW873\nGTN7zcx+r2Ob2vWXPtuldv1FkszsX5vZCTN7zsweMLNdHc8H7y/NYV+g5u6S9IKkd3R5/vvufluO\n9SmLf+bu3eYGuVXSVe3ltyUdaT/WxXZtI9Wzz/yJpMfc/VNm9hZJYx3P17XP9GoXqWb9xd1flHSd\nlPxjK+m0pG93bFa7/tJnu0g16y9mtlvSv5K0x91/aWZ/LumgpP+yZbPg/YVPrDIyswlJn5D0jaLr\nEplPSvqmJ34g6RIzu6zoSqEYZvaPJN0k6X5JcvdfufsvOjarXZ/ps13qbp+kv3H3zomma9dfOnRr\nl7pqSnqrmTWV/HPyvzueD95fCKyy+6qkL0k6v802H2p/tPiomV2TU72K5pIeN7PjZjaX8vxuSS9v\nWV9tl9VBr7aR6tdnrpB0RtJ/bn+t/g0ze1vHNnXsM/20i1S//rLVQUkPpJTXsb9s1a1dpJr1F3c/\nLek/SDol6RVJr7r7f+vYLHh/IbDKwMxuk/Rzdz++zWZPSZp09/dL+pqk7+RSueJ9xN2vU/Lx6p1m\ndlPRFSqRXm1Txz7TlHS9pCPu/luS/p+kf1tslUqhn3apY3+RJLW/Gr1d0n8tui5l0qNdatdfzOwf\nK/lE6gqVBrdnAAABxUlEQVRJ/1TS28zsM6M+LoFVNh+WdLuZnZT0oKRbzOzo1g3c/TV3f7399yOS\ndprZpbnXNGft/xDk7j9X8h3/DR2bnJZ0+Zb1iXZZ5fVqm5r2mVVJq+7+w/b6t5QEFFvVsc/0bJea\n9pcNt0p6yt3/LuW5OvaXDV3bpab9Zb+kn7n7GXf/taSHJH2oY5vg/YXAKgN3/7K7T7j7tJKPXf/S\n3S+Igs3sPWZm7b9vUNLWZ3OvbI7M7G1m9vaNvyX9c0nPdWz2PUmfbf8S40YlH82+knNVc9dP29Sx\nz7j7/5H0spld3S7aJ+n5js1q12f6aZc69pct7lD3r7tq11+26NouNe0vpyTdaGZj7XPfp+QHZ1sF\n7y/8KjAgMzskSe7+dUmfktQys3OSfinpoFd/NtZ/Iunb7fduU9KfuftjHe3yiKQDkl6StCbp8wXV\nNW/9tE0d+4wk/a6kpfbXGH8r6fP0GUm926WW/aX9j8lHJf3OlrLa95c+2qV2/cXdf2hm31LyNeg5\nSU9LWhx1f2HmdQAAgED4KhAAACAQAisAAIBACKwAAAACIbACAAAIhMAKAAAgEAIrAACAQAisAAAA\nAiGwAgAACOT/A1flvRpe7BsWAAAAAElFTkSuQmCC\n", 49 | "text/plain": [ 50 | "" 51 | ] 52 | }, 53 | "metadata": {}, 54 | "output_type": "display_data" 55 | } 56 | ], 57 | "source": [ 58 | "plt.figure(figsize=(10, 6))\n", 59 | "plt.scatter(X[y == 0][:, 0], X[y == 0][:, 1], color='b', label='0')\n", 60 | "plt.scatter(X[y == 1][:, 0], X[y == 1][:, 1], color='r', label='1')\n", 61 | "plt.legend();" 62 | ] 63 | }, 64 | { 65 | "cell_type": "code", 66 | "execution_count": 5, 67 | "metadata": { 68 | "collapsed": true 69 | }, 70 | "outputs": [], 71 | "source": [ 72 | "class LogisticRegression:\n", 73 | " def __init__(self, lr=0.01, num_iter=100000, fit_intercept=True, verbose=False):\n", 74 | " self.lr = lr\n", 75 | " self.num_iter = num_iter\n", 76 | " self.fit_intercept = fit_intercept\n", 77 | " self.verbose = verbose\n", 78 | " \n", 79 | " def __add_intercept(self, X):\n", 80 | " intercept = np.ones((X.shape[0], 1))\n", 81 | " return np.concatenate((intercept, X), axis=1)\n", 82 | " \n", 83 | " def __sigmoid(self, z):\n", 84 | " return 1 / (1 + np.exp(-z))\n", 85 | " def __loss(self, h, y):\n", 86 | " return (-y * np.log(h) - (1 - y) * np.log(1 - h)).mean()\n", 87 | " \n", 88 | " def fit(self, X, y):\n", 89 | " if self.fit_intercept:\n", 90 | " X = self.__add_intercept(X)\n", 91 | " \n", 92 | " # weights initialization\n", 93 | " self.theta = np.zeros(X.shape[1])\n", 94 | " \n", 95 | " for i in range(self.num_iter):\n", 96 | " z = np.dot(X, self.theta)\n", 97 | " h = self.__sigmoid(z)\n", 98 | " gradient = np.dot(X.T, (h - y)) / y.size\n", 99 | " self.theta -= self.lr * gradient\n", 100 | " \n", 101 | " z = np.dot(X, self.theta)\n", 102 | " h = self.__sigmoid(z)\n", 103 | " loss = self.__loss(h, y)\n", 104 | " \n", 105 | " if(self.verbose ==True and i % 10000 == 0):\n", 106 | " print(f'loss: {loss} \\t')\n", 107 | " \n", 108 | " def predict_prob(self, X):\n", 109 | " if self.fit_intercept:\n", 110 | " X = self.__add_intercept(X)\n", 111 | " \n", 112 | " return self.__sigmoid(np.dot(X, self.theta))\n", 113 | " \n", 114 | " def predict(self, X):\n", 115 | " return self.predict_prob(X).round()" 116 | ] 117 | }, 118 | { 119 | "cell_type": "code", 120 | "execution_count": 6, 121 | "metadata": { 122 | "collapsed": true 123 | }, 124 | "outputs": [], 125 | "source": [ 126 | "model = LogisticRegression(lr=0.1, num_iter=300000)" 127 | ] 128 | }, 129 | { 130 | "cell_type": "code", 131 | "execution_count": 7, 132 | "metadata": {}, 133 | "outputs": [ 134 | { 135 | "name": "stdout", 136 | "output_type": "stream", 137 | "text": [ 138 | "CPU times: user 14.3 s, sys: 12 ms, total: 14.3 s\n", 139 | "Wall time: 14.3 s\n" 140 | ] 141 | } 142 | ], 143 | "source": [ 144 | "%time model.fit(X, y)" 145 | ] 146 | }, 147 | { 148 | "cell_type": "code", 149 | "execution_count": 8, 150 | "metadata": {}, 151 | "outputs": [ 152 | { 153 | "data": { 154 | "text/plain": [ 155 | "1.0" 156 | ] 157 | }, 158 | "execution_count": 8, 159 | "metadata": {}, 160 | "output_type": "execute_result" 161 | } 162 | ], 163 | "source": [ 164 | "preds = model.predict(X)\n", 165 | "(preds == y).mean()" 166 | ] 167 | }, 168 | { 169 | "cell_type": "code", 170 | "execution_count": 9, 171 | "metadata": {}, 172 | "outputs": [ 173 | { 174 | "data": { 175 | "text/plain": [ 176 | "array([-25.96818124, 12.56179068, -13.44549335])" 177 | ] 178 | }, 179 | "execution_count": 9, 180 | "metadata": {}, 181 | "output_type": "execute_result" 182 | } 183 | ], 184 | "source": [ 185 | "model.theta" 186 | ] 187 | }, 188 | { 189 | "cell_type": "code", 190 | "execution_count": 11, 191 | "metadata": {}, 192 | "outputs": [ 193 | { 194 | "data": { 195 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlYAAAFpCAYAAABeYWb6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xt8FNX9//H3IQGRapXWQJWrln75orblpxDCRcpNqmCR\nCuIFUBEbSdBaL9QqFrHfolXxAipiREHlqoDcxFRAiggk4aJVUesVEIuCIBeJAsme3x+bSIgbstnL\nzM7M6/l4zCPZ2dmdz5yZnXwyc/Z8jLVWAAAAiF8ttwMAAADwCxIrAACABCGxAgAASBASKwAAgAQh\nsQIAAEgQEisAAIAEIbECAABIEBIrAACABCGxAgAASJCoEytjTJox5g1jzKIIz3UxxuwxxrxZNo1K\nbJgAAACpL70Gy94g6T1JP67i+ZXW2guifbOTTjrJNm/evAarBwAAcMf69eu/stZmVLdcVImVMaax\npN6Sxki6Kc7YJEnNmzfXunXrEvFWAAAASWWM2RzNctHeCnxY0p8lhY6yTAdjzFvGmJeNMWdE+b4A\nAAC+UW1iZYy5QNJ2a+36oyy2QVJTa+2vJD0iaV4V75VtjFlnjFm3Y8eOmAIGAABIVdFcseooqY8x\nZpOkmZK6GWOmVlzAWrvXWvtN2e+LJdU2xpxU+Y2stXnW2jbW2jYZGdXepgQAAPCUavtYWWtvk3Sb\nFP72n6RbrLWDKi5jjPmZpC+ttdYYk6lwwrYz8eECAACvOnTokLZu3arvvvvO7VCqVLduXTVu3Fi1\na9eO6fU1+VbgEYwxwyTJWjtRUn9JOcaYEknfSrrUWmtjfW8AAOA/W7du1fHHH6/mzZvLGON2OD9g\nrdXOnTu1detWnXrqqTG9R40SK2vtvyT9q+z3iRXmPyrp0ZgiAAAAgfDdd9+lbFIlScYY/fSnP1U8\n/cAZeR0AADgmVZOqcvHGR2IFAAACIz8/Xy1btlSLFi30j3/8I+HvT2IFAAACobS0VMOHD9fLL7+s\nd999VzNmzNC7776b0HWQWAEAgJQ0bZrUvLlUq1b457Rp8b1fUVGRWrRoodNOO0116tTRpZdeqvnz\n5yci1O+RWAEAgJQzbZqUnS1t3ixZG/6ZnR1fcvX555+rSZMm3z9u3LixPv/88wREexiJFRAQif7P\nDwCSaeRIqbj4yHnFxeH5qSzmcawAeEf5f37lJ6ny//wkaeBA9+ICgKps2VKz+dFo1KiRPvvss+8f\nb926VY0aNYr9DSPgihUQAF79zw/esHPnTt144406cOCA26HAR5o2rdn8aLRt21YffvihPv30Ux08\neFAzZ85Unz59Yn/DCEisgABIxn9+gCS98847yszMVJ06dZSezk0QJM6YMVK9ekfOq1cvPD9W6enp\nevTRR/Xb3/5WrVq10oABA3TGGWfEF2jldST03QCkpKZNw7f/Is0HYjV//nz94Q9/0IMPPqhBgwZV\n/wKgBsq7KYwcGf4nsGnTcFIVb/eFXr16qVevXvEHWAUSKyAAxow5so+VFP9/fggua63uvvtuTZw4\nUYsWLVJmZqbbIcGnBg70Xj9QEisgAJL1nx+Cp7i4WEOGDNGmTZtUWFioU045xe2QgJRCYgUEhBf/\n80Nq2bJli/r27atf/vKXWrFiherWret2SEDKofM6AKBar7/+urKysjRw4EBNmTKFpAqoAlesAABH\nNWnSJN1+++165plndP7557sdDpDSSKwAABGVlJTopptu0j//+U+tXLlSLVu2dDskIOVxKxAA8AO7\ndu3Seeedpw8++ECFhYUkVfCNq6++Wg0aNNCZZ56ZlPcnsQIAHGHjxo3KzMxU69at9dJLL+nEE090\nOyQgYa666irl5+cn7f1JrAAA31u4cKG6du2qUaNGaezYsUpLS3M7JARZEqrHd+7cWT/5yU/ifp+q\n0McKAKBQKKR77rlHEyZM0IIFC5SVleV2SAg6j1aPJ7ECgIDbt2+frrzySm3btk1r165l0E+khqNV\nj0/hxIpbgQAQYB988IHatWunjIwM/etf/yKpQurwaPV4EisACKjFixerU6dOuuGGG/TEE0/omGOO\ncTsk4LCqqsSnePV4EisACBhrre655x794Q9/0Isvvqhrr73W7ZCAHxozJlwtvqIEVI+/7LLL1L59\ne/3nP/9R48aN9dRTT8X1fpXRxwoAAmT//v0aMmSINm/erKKiIjVq1MjtkIDIklQ9fsaMGQkIrmok\nVgAQEJ9++qn69u2rs846iyLK8AYPVo/nViAABMDy5cvVvn17DR06VE8//TRJFZAkXLECAB+z1urR\nRx/VmDFjNH36dHXr1s3tkABfI7ECAJ86cOCAcnNztXbtWq1evVqnnXaa2yEBstbKGON2GFWy1sb1\nem4FAi5JQqUG4Hvbtm1T165dtWfPHpIqpIy6detq586dcScvyWKt1c6dO+O6Vc4VK8AFHq3UAI9Y\nvXq1BgwYoGuvvVYjR45UrVr8D43U0LhxY23dulU7duxwO5Qq1a1bV40bN4759catrLFNmzZ23bp1\nrqwbcFvz5uFkqrJmzaRNm5yOBn4yadIk3X777Zo8ebJ69+7tdjiAbxhj1ltr21S3HFesABd4tFID\nUlhJSYluueUWvfzyy1q5cqVatmzpdkhAIJFYAS5o2jTyFasUr9SAFPX111/rkksukTFGBQUFql+/\nvtshAYHFjXfABUmq1IAA+uCDD5SVlaXTTz9dL730EkkV4DISK8AFAwdKeXnhPlXGhH/m5dFxHTXz\nyiuv6JxzztGIESP08MMPKz2dmxCA2/gUAi7xYKUGpAhrrcaPH69//OMfeuGFF9S5c2e3QwJQhsQK\nADzku++++37QzzVr1qh58+ZuhwSgAm4FAoBHbN26Veecc472799PUgWkKBIrAPCAVatWKTMzU/37\n99fMmTN13HHHuR0SgAi4FQgAKe7JJ5/UyJEjNWXKFPXq1cvtcAAcBVes4DvU4INfHDp0SMOHD9eD\nDz6o119/naQK8ACuWMFXqMEHv9i+fbsuvvhiHX/88SooKNAJJ5zgdkgAosAVK/jKyJGHk6pyxcXh\n+YBXvPHGG8rMzFSnTp00f/58kirAQ7hiBV+hBh+8bvr06brhhhs0YcIEXXzxxW6HA6CGSKzgK9Tg\ng1eVlJTotttu09y5c7Vs2TL96le/cjskADHgViB8hRp88KJdu3apV69eevPNN1VUVERSBXgYiRV8\nhRp88Jq33npLbdu21S9/+Uu9/PLL+ulPf+p2SADiwK1A+A41+OAVM2fO1PXXX6/x48frsssuczsc\nAAlAYgUADistLdXIkSM1a9YsLVmyRK1bt3Y7JAAJQmIFAA7au3evLr/8cn3zzTdau3atTjrpJLdD\nApBA9LECAId89NFHysrKUrNmzbRkyRKSKsCHok6sjDFpxpg3jDGLIjxnjDHjjTEfGWPeMsacldgw\nAcDbli5dqo4dO+qPf/yjHnvsMdWuXdvtkAAkQU2uWN0g6b0qnjtf0i/KpmxJj8cZF4AUQe3F+Fhr\nNX78eA0aNEjPP/+8hg0b5nZIAJIoqj5WxpjGknpLGiPppgiLXCjpWWutlVRgjDnRGHOytXZb4kIF\n4DRqL8bnwIEDys3N1dq1a7VmzRqdeuqpbocEIMmivWL1sKQ/SwpV8XwjSZ9VeLy1bB4AD6P2Yuy+\n+OILdevWTV9//bVWr15NUgUERLWJlTHmAknbrbXr412ZMSbbGLPOGLNux44d8b4dgCSj9mJsNmzY\noMzMTJ177rmaPXu2jjvuOLdDAuCQaK5YdZTUxxizSdJMSd2MMVMrLfO5pCYVHjcum3cEa22etbaN\ntbZNRkZGjCEDcEpVNRapvVi1WbNm6be//a0efPBBjR49WrVq8eVrIEiq/cRba2+z1ja21jaXdKmk\nV621gyottkDSFWXfDsyStIf+VYD3UXsxeqFQSCNHjtStt96qJUuWqH///m6HBMAFMQ8QaowZJknW\n2omSFkvqJekjScWShiQkOgCuKu+gPnJk+PZf06bhpIqO60fau3evBg0apN27d6uoqEgNGjRwOyQA\nLjHhL/I5r02bNnbdunWurBsAEuXjjz9Wnz59dM4552j8+PGqU6eO2yEBSAJjzHprbZvqluPmPwDE\n6JVXXlGHDh00fPhwTZw4kaQKALUCAaCmrLW6//779fDDD+uFF15Q586d3Q4JQIogsQKAGti/f7+u\nvvpqffLJJyosLFSTJk2qfxGAwOBWIABE6dNPP1WHDh107LHHauXKlSRVAH6AxAoAorB06VK1b99e\n11xzjSZPnqy6deu6HRKAFERiBcQpN1dKT5eMCf/MzXU7IiSStVYPPvigBg8erJkzZ+r666+XMcbt\nsACkKPpYAXHIzZUef/zw49LSw48nTHAnJiTOt99+q+zsbL3zzjsqKChQs2bN3A4JQIrjihUQh7y8\nms2Hd2zdulWdO3dWSUmJVq1aRVIFICokVkAcSktrNh/esHr1arVr1079+/fX9OnTVa9yXR8AqAK3\nAoE4pKVFTqLS0pyPBYnx1FNP6bbbbtOUKVPUq1cvt8MB4DEkVkAcsrOP7GNVcT685dChQ7rpppu0\nZMkSrVy5Ui1btnQ7JAAeRGIFxKG8g3peXvjKVVpaOKmi47q3fPHFFxowYIB+/OMfq7CwUCeccILb\nIQHwKPpYAXGaMEEqKZGsDf8kqfKWDRs2KDMzU127dtWCBQtIqgDEhStWAALrhRdeUG5uriZOnKh+\n/fq5HQ4AHyCxAhA4oVBId911l6ZMmaIlS5aodevWbocEwCdIrAAEyv79+3XllVfqv//9r4qKitSw\nYUO3QwLgI/SxAhAYW7ZsUadOnXTcccdp+fLlJFUAEo7ECimrR49w/b3yqUcPtyPytmnTpObNpVq1\nwj+nTXM7ImetXr1aWVlZGjRokCZPnqxjjjnG7ZAAf3H6JJOiJzUSK6SkHj2kZcuOnLdsGclVrKZN\nCw8DsXlz+NuLmzeHH6fIeSjp8vLy1LdvXz355JO6+eabKaIMJJrTJ5kUPqkZa60rK27Tpo1dt26d\nK+tG6jva3z2XDllPa948fN6prFkzadMmp6NxzsGDB3Xddddp1apVmjdvnn7xi1+4HRLgT06fZFw4\nqRlj1ltr21S3HJ3XgQDYsqVm8/3giy++UL9+/dSwYUMVFBTo+OOPdzskwL+cPsmk8EmNW4FAADRt\nWrP5Xrdu3TplZmaqZ8+emj17NkkVkGxOn2RS+KRGYoWU1L17zebj6MaMkerVO3JevXrh+X4zdepU\nnX/++Ro3bpzuvPNO1arFaQ5IOqdPMil8UuOMg5S0dOkPk6ju3cPzUXMDB4brGTZrFu6/1qxZ+PHA\ngW5HljilpaUaMWKE7rzzTr366qv6/e9/73ZIQHA4fZJJ4ZMandcBeN6uXbt02WWXqbS0VLNmzdJP\nf/pTt0MC4DPRdl7nihUAT3v77beVmZmpM888U/n5+SRVAFzFtwIBeNacOXM0bNgwPfTQQxo0aJDb\n4QAAiRUA7yktLdWdd96p5557Tvn5+Tr77LPdDgkAJJFYAfCY3bt3a+DAgdq/f7/Wrl2rBg0auB0S\nAHyPPlZIWV4pO5Wi5ap8aePGjWrbtq1atGihJUuWkFQBSDlcsUJKKi8DVVwcflxeBkpKzrdpY12f\n03EG2fz58/WHP/xBY8eO1RVXXOF2OAAQEcMtICV5pexUUGvwOclaq7vvvluPP/645s6dq8zMTLdD\nAhBA1AqEp3ml7FQKl6vyheLiYl199dX65JNPVFRUpFNOOcXtkADgqOhjhZTklbJTKVyuyvM+++wz\nnXPOOapdu7ZWrFhBUgXAE0iskJK8UnYqhctVedrq1avVrl07XXrppXr22Wd17LHHuh0SAESFxAop\nyStlp1K4XJVnPf300+rbt68mTZqkESNGyBjjdkgAEDU6rwNICSUlJbrlllu0ePFizZ8/X61atXI7\nJAD4Hp3XAXjGrl27dMkllygtLU2FhYWqX7++2yEBQEy4FQjAVe+++67atWunX//611q0aBFJFQBP\nI7EC4JqFCxfqN7/5je644w6NHTtW6elcRAfgbZzFADjOWqt7771XjzzyiBYuXKisrCy3QwKAhOCK\nlcuCUGcuCNuI6BUXF2vgwIGaM2eOioqKSKrgLZzQUA0SKxeV15nbvFmy9nCdOT99ToOwjYjepk2b\n1LFjR9WqVUuvvfaaGjVq5HZIQPQ4oSEKDLfgoiDUmQvCNiI6r776qi6//HL95S9/0Q033MD4VPAe\nTmiBxnALHhCEOnNB2EYcnbVWDz/8sO677z7NmDFDXbt2dTskIDac0BAFEisXNW0a+Z8fP9WZC8I2\nomrffvutsrOztXHjRhUUFKhZs2ZuhwTEjhMaokAfKxcFoc5cELYRkW3ZskWdOnVSaWmpXn/9dZIq\neB8nNESBxMpFQagzF4RtxA+99tprateunS6//HJNmzZN9Sr/MQK8iBMaokDndQAJY63VxIkTNXr0\naD333HPq2bOn2yEBQELQeR2Aow4ePKjrr79er7/+ulatWqUWLVq4HRIAOI7ECkDcvvzyS/Xr108Z\nGRkqKCjQ8ccf73ZIAOAK+lgBiMv69euVmZmpHj16aM6cOSRVAAKt2sTKGFPXGFNkjPm3MWajMeau\nCMt0McbsMca8WTaNSk64AFLJ9OnTdd555+mhhx7S6NGjVasW/6sBCLZozoIHJHWz1v5aUmtJ5xlj\nIhX3WmmtbV02/S2hUcLTcnOl9PTwl2jS08OPk/k6r5Ty8kqckZSUlOhPf/qTRo0apVdffVUXXXSR\n2yEBQEqoto+VDX9t8Juyh7XLJne+SgjPyc2VHn/88OPS0sOPJ0xI/OvKS3kVF4cfl5fyklLrG9Fe\niTOSXbt2acCAAUpPT9fatWtVv359t0MCgJQR1XALxpg0SesltZD0mLX21krPd5E0V9JWSZ9LusVa\nu/Fo78lwC8GQnh5OiipLS5NKShL/Oq+U8vJKnJW9//77+t3vfqc+ffrovvvuU1pamtshAYAjoh1u\nIaoOEdbaUmtta0mNJWUaY86stMgGSU2ttb+S9IikeVUElW2MWWeMWbdjx45oVg2Pi5QcHW1+vK/z\nSikvr8RZUX5+vjp37qzbb79dDzzwAEkVAERQo56m1trdkpZLOq/S/L3W2m/Kfl8sqbYx5qQIr8+z\n1rax1rbJyMiII2x4RVV/e6v7mxzr66oq2ZVqpby8EqcUHvTzoYce0pAhQ/Tiiy9qyJAhbocEACkr\nmm8FZhhjTiz7/VhJ50p6v9IyPzPGmLLfM8ved2fiw4XXlPcbinZ+vK/zSikvr8R54MABDR06VM88\n84wKCgrUsWNHt0MCgJQWzRWrkyUtN8a8JWmtpCXW2kXGmGHGmGFly/SX9I4x5t+Sxku61LpVKwcp\nZcIEKSfn8JWmtLTw46N1QI/ndV4p5eWFOLdv367u3btr9+7dFFEGgChRKxDADxQVFal///666qqr\nGJ8KAEStQAAxmjx5sm699VY9+eSTuvDCC90OBwA8hcQKgKTwoJ+33HKLFi9erBUrVqhVq1ZuhwQA\nnkNiBUA7d+7UgAEDVKdOHRUVFenEE090OyQA8CQ6TgAB9/bbb6tt27Zq06aNFi1aRFIFAHHgihUQ\nYHPnztW1116rcePG6fLLL3c7HADwPK5YuczpQryxFjZ2Y52xtk0Q2jReoVBIf/3rX3XjjTcqPz8/\ntZIqp3egVw40AN5grXVlOvvss23QTZ1qbb161kqHp3r1wvOTISfnyHWVTzk5yVlfPOuMtW2C0Kbx\n2r17t73gggts586d7Zdfful2OEdyegd65UAD4DpJ62wU+Q3jWLnI6UK8sRY2dmOdsbZNENo0Hu+/\n/7769u2rHj166KGHHlLt2rXdDulITu9ArxxoAFwX7ThWJFYuqlUr/K9uZcZIoVDi1xcuOhRZsg6D\nWNcZa9sEoU1jtXDhQg0dOlR33323rrnmGrfDiczpHeiVAw2A66JNrOhj5SKnC/HGWtjYjXXG2jZB\naNOaCoVCuuuuu5STk6MFCxakblIlOb8DvXKgAfAMEisXOV2IN9bCxm6sM9a2CUKb1sTevXv1+9//\nXkuWLNG6deuUlZXldkhH5/QO9MqBBsA7oumIlYyJzuthU6da26yZtcaEfya772tOjrVpaeG+tmlp\nznSyjnWdsbZNENo0Gh9++KFt1aqVHTZsmD1w4IDb4UTP6R3olQMNgKtE53UguJYuXaqBAwdq9OjR\nysnJcTscAPA8ijADAWSt1SOPPKK7775bs2bNUpcuXdwOCQAChcQK8ImDBw9q+PDhKiws1Jo1a3Tq\nqae6HRIABA6JFeAD27dvV79+/ZSRkaHVq1fruOOOczskAAgkvhUIeNybb76pzMxMdevWTbNnzyap\nAgAXkVgh6SjFljzPP/+8evbsqfvvv1933XWXatXiI+0YLxaJrAm/bx+QJNwKRFJNmxYe06m4OPx4\n8+bDYzwNHJj41wVFKBTSqFGjNHXqVL3yyitq3bq12yEFS26u9Pjjhx+Xlh5+PGGCOzElkt+3D0gi\nhltAUlGKLfH27dunwYMHa9euXZo9e7YaNGjgdkjB47UikTXl9+0DYkBJG6SELVtqNj/e1/ndxx9/\nrPbt26thw4ZaunQpSZVbIiUdR5vvNX7fPiCJSKyQVJRiS5xly5apQ4cOys3N1cSJE1WnTh23Qwou\nLxSJjIfftw9IIhIrJBWl2OJXPujnwIEDNXPmTOXm5soY43ZYwZbqRSLj5fftA5KIzutIqvKO5iNH\nhm/jNW0aTo6q64Ae6+v85rvvvtOwYcO0YcMGrV69WqeddprbIUE63IE7Ly98eywtLZx0+KVjt9+3\nD0giOq8DKeqzzz7TRRddpNNOO01PP/20fvSjH7kdEgAEFp3XAQ977bXXlJmZqQEDBmjmzJkkVQDg\nEdwKBFKItVYTJkzQ3/72Nz333HPq2bOn2yEBAGqAxApIEQcOHFBubq6Kioq0evVq/fznP3c7JABA\nDXErEEgB//3vf/Wb3/xGe/bs0Zo1a0iqAMCjSKwqcbo+Xazr81IZL2r+HV1BQYEyMzP1u9/9Ti+8\n8IJ/iiiz4yOL9cPrlfZ0ujioV07aCA5rrSvT2WefbVPN1KnW1qtnrXR4qlcvPD+V1peTc+Rryqec\nnOTEGQ+n29RrJk+ebDMyMuyCBQvcDiWx2PGRxfrh9Up7xhqn06+LlVf2A5JC0jobRX7DcAsVOF2f\nLtb1eamMFzX/IispKdEtt9yixYsXa/78+WrVqpXbISUWOz6yWD+8XmlPp4uDeuWkDV+IdrgFEqsK\natUK/wtSmTFSKJQ66zvaoNsu7c4qOd2mXrBz505dcsklSk9P14wZM1S/fn23Q0o8dnxksX54vdKe\nscbp9Oti5ZX9gKRgHKsYOF2fLtb1eamMFzX/jvT2228rMzNTZ511ll566SV/JlUSO74qsX54vdKe\nThcH9cpJG4FCYlWB0/XpYl2fl8p4UfPvsOeff17dunXT3/72N913331KS8VMOFHY8ZHF+uH1Sns6\nXRzUKydtBEs0HbGSMaVi53Vrw30QmzWz1pjwz2T3SYx1fTk51qalhftOpqWlZsf1ck63aaopLS21\nd9xxh23WrJl944033A7HOUHf8VWJ9cPrlfaMNU6nXxcrr+wHJJzovA6475tvvtEVV1yhHTt2aM6c\nOWrQoIHbIQEAYkAfK8BlmzdvVseOHVW/fn0tXbqUpAoAAoDECkiC119/XVlZWRoyZIgmTZqkY445\nxu2QAAAOoFYgkGBPPfWUbrvtNj333HP67W9/63Y4AAAHkVgBCVJSUqIRI0bopZde0sqVK9WyZUu3\nQwIAOIxbgS7zUpmrHj3C4+CVTz16JCtK79mxY4d69uyp9957T4WFhSRVqJpXas05/YHnBAOfILFy\n0bRp4eFrNm8OD+a7eXP4cbLOs/Gsr0cPadmyI+ctW8a5T5LeeOMNtW3bVllZWf4e9BPxc/pDHyun\nP/CcYOAjDLfgIi+VufJSGR0nzZo1S9ddd50mTJigiy++2O1wkOq8UmvO6Q88Jxh4QLTDLdDHykVb\nttRsvtfW52elpaW64447NHPmTC1dulS//vWv3Q4JXsCHEPA9bgW6iDJX3rRnzx716dNHBQUFWrt2\nLUkVoseHEPA9EisXeanMVffuNZvvV++//77atWunn//853rllVd00kknuR0SvMQrteac/sBzgoGf\nRFP3JhlTqtYKdJqXylx17x4ub1Y+de+erChT04IFC2xGRoadNGmS26HAy7xSa87pD3zQTzBIeaJW\nIJAYoVBId999tyZOnKjZs2crKyvL7ZAAAA6j8zqQAPv27dNVV12lbdu2ae3atTr55JPdDgkAkMLo\nYwVU4aOPPlL79u31k5/8RMuXLyepAgBUi8QKiODll19Wx44dNXz4cOXl5VFEGQAQlWpvBRpj6kp6\nTdIxZcvPttbeWWkZI2mcpF6SiiVdZa3dkPhwgeSy1uqee+7RY489prlz56pjx45uhwQA8JBorlgd\nkNTNWvtrSa0lnWeMqdx793xJvyibsiU9ntAoYxBrOS6vlPGKVW6ulJ4eHug4PT38OFp+b9Pi4mJd\ndtllmjdvnoqKiqJPqpzewFh3ot/jjOfg9oJYa+k53Z5e+cD7/YQWq3i2z+9tE61ovjpYPkmqJ2mD\npHaV5j8h6bIKj/8j6eSjvVcyh1uYOtXaevWO/OZuvXrVf6s51td5RU7OkdtWPuXkVP9av7fpli1b\n7FlnnWUHDRpki4uLo3+h0xsY6070e5zxHNxeUHkogmiHJHC6Pb3ygff7CS1W8Wyf39vGRj/cQrQJ\nVZqkNyV9I+neCM8vktSpwuNlktoc7T2TmVg1axb5nNCsWXJe5xVpaZG3Ly2t+tf6uU1Xr15tTznl\nFHvffffZUChUsxc7vYGx7kS/xxnPwe0FkbatfDoap9vTCx94a/19QotHPNvn97axNurEqkbjWBlj\nTpT0oqTrrbXvVJi/SNI/rLWvlz1eJulWa+26Sq/PVvhWoZo2bXr25kjFSBOgVq3wHv1h/FIolPjX\neUU8dU7Rxa5tAAAZDklEQVT92qZTpkzRn//8Z02ZMkW9evWq+Rs4vYGx7kS/x+n3Ir5eac9U/8CX\n8+sJLV7xbJ/f20bRj2NVo28FWmt3S1ou6bxKT30uqUmFx43L5lV+fZ61to21tk1GRkZNVl0jsZbj\n8nsZr7S0ms2vyG9tWlJSoptuukljxozRihUrYkuqJOc3MNad6Pc44zm4/czp9kzVD3xlfjuhJUo8\n2+f3tqmBahMrY0xG2ZUqGWOOlXSupPcrLbZA0hUmLEvSHmvttoRHG6VYy3F5pYxXrLKzaza/Ij+1\n6ddff63evXvrnXfeUVFRkVq1ahX7mzm9gbHuRL/HGc/B7QWx1tJzuj1T8QMfiZ9OaIkUz/b5vW1q\norp7hZJ+JekNSW9JekfSqLL5wyQNK/vdSHpM0seS3lY1/ausTX6twFjLcXmljFescnIOd59IS6tZ\n314/tOm7775rW7RoYf/0pz/ZQ4cOJeZNnd7AWHei3+OM5+D2glhr6Tndnqn0gT8aP5zQkiGe7fN5\n24hagcCRFi1apKuvvlr333+/rrzySrfDAQB4CLUCgTLWWt1777165JFHtGDBAoooAwCShsQKvlZc\nXKxrrrlGH374oYqKitSoUSO3QwIA+Bi1AuFbW7duVefOnWWM0WuvvUZSBQBIOhIr+NKqVavUrl07\nDRgwQFOnTtWxxx7rdkgAgADgViB8xVqrJ554QqNGjdLkyZPVu3dvt0MCAAQIV6wqoYakdx04cEDZ\n2dl65JFHtGrVKv8mVX4vHuv09nnldbHyyn73O/ZDcEQzJkMypmSPYxWLANSQ9K3PP//cZmVl2d//\n/vd27969boeTPH4vHuv09nnldbHyyn73O/aDL4hxrGqueXMpUvnCZs2kTZucjgbRWrNmjfr376/c\n3FzddtttqlXLxxdiYz1IvXJwO719XnldrLyy3/2O/eAL0Y5jRWJVQQBqSPrOpEmTdPvtt+vpp5/W\nBRdc4HY4yef34rFOb59XXhcrr+x3v2M/+EJSijD7HTUkvePgwYPKzc3VAw88oJUrVwYjqZL8XzzW\n6e3zyuti5ZX97nfsh0AhsaqAGpLesH37dp177rnaunWrCgsL1bJlS7dDco7fi8c6vX1eeV2svLLf\n/Y79ECzRdMRKxpSKndet9X0NSc/bsGGDbdasmR05cqQtLS11Oxx3+L14rNPb55XXxcor+93v2A+e\nJzqvw29mzZql6667To899pgGDBjgdjgAgAChCDN8IxQK6a9//aumT5+uJUuWqHXr1m6HBABARCRW\nSGm7d+/W4MGDtW/fPhUVFSkjI8PtkAAAqBKd15Gy3nvvPWVmZurUU0/VkiVLSKoAACmPxAopafHi\nxfrNb36j22+/XePHj1ft2rXdDgkAgGqRWCGlWGs1duxYXXPNNZo3b56uuuoqt0NKPbHWHMvNldLT\nw4MSpqeHHydzfU5zevvYD5F5JU6v8FJ7einWZIrmq4PJmFJ1uAW457vvvrNXXnmlbd26td28ebPb\n4aSmWGuO5eQc+ZryKScnOetzmtPbx36IzCtxeoWX2tNLscZIDLcAL/niiy900UUX6ZRTTtEzzzyj\nH/3oR26HlJpirTmWni6Vlv5wflqaVFKS+PU5zentYz9E5pU4vcJL7emlWGNErUB4xhtvvKG+fftq\nyJAhGjVqlL+LKMcr1ppjxlT93NHOAV6pceb09rEfIvNKnF7hpfb0UqwxolYgPOH5559Xz549NXbs\nWI0ePZqkqjqx1hxLS6vZ/HjX5zSnt4/9EJlX4vQKL7Wnl2JNMv6KwRUlJSUaMWKEbr31Vr3yyiu6\n+OKL3Q7JG2KtOZadXbP58a7PaU5vH/shMq/E6RVeak8vxZps0XTESsZE5/Xg2rlzpz333HNtjx49\n7FdffeV2ON4Ta82xnBxr09LCnUrT0qrvMB3v+pzm9PaxHyLzSpxe4aX29FKsMRCd15GKNm7cqAsv\nvFAXXnih7r33XqWnM/g/ACD10ccKKWfevHnq0qWLRo0apQceeICkCgDgO/xlQ9KFQiH9/e9/15NP\nPqnFixerbdu2bocEAEBSkFghqfbt26errrpK27ZtU1FRkU4++WS3QwIAIGm4FYik+eijj9S+fXv9\n5Cc/0fLly0mqAAC+R2KFpPjnP/+pjh07avjw4crLy9Mxxxzj3Mr9Xq/K6e0744zwIH/l0xlnJHd9\nXtl/TtcYBOAN0Xx1MBkTwy34UygUsvfdd589+eST7WuvveZ8AH6vV+X09p1+euTadqefnpz1eWX/\nOV1jEIDrxHALcNr+/fs1dOhQffTRR3rxxRfVpEkT54Pwe70qp7cv1hIssfLK/nO6xiAA1zHcAhz1\nySefqH379jrmmGO0cuVKd5IqSdqypWbzvYbtSw2xxumV7QMQMxIrxG3p0qVq3769srOzNWXKFB17\n7LHuBeP3elVsX2pwusYgAM8gsULMrLUaP368Bg8erOeff17XXXedzNFuHTnB7/WqnN6+00+v2fx4\neWX/OV1jEIB3RNMRKxkTnde97cCBA3bo0KH2l7/8pf3kk0/cDudIPq9X5fj2Ve7AnqyO6+W8sv+c\nrjEIwFWi8zqSZfv27erXr59OOukkPfvsszr++OPdDgkAgKSi8zqS4s0331RmZqa6du2qOXPmkFQB\nAFABJW0QtTlz5ignJ0ePPvqoBgwY4HY4AACkHBIrVCsUCmn06NF65plnlJ+fr7POOsvtkAAASEkk\nVjiqffv2afDgwfrqq69UVFSkhg0buh0SAAApiz5WqFL5oJ8ZGRl69dVXSaoSzemacX6vbeeVOAG/\n4jMYFs1XB5MxMdxCalu2bJlt2LChfeSRR2woFHI7HP9xumac32vbeSVOwK8C8BkUwy0gFtZaPfbY\nY/q///s/zZgxQ926dXM7JH9yumac32vbeSVOwK8C8BmMdrgFEit87+DBgxo+fLjWrFmjBQsW6LTT\nTnM7JP+qVStyEWNjpFAoddbndJyx8kqcgF8F4DPIOFaokS+++ELdu3fXjh07tGbNGpKqZHO6Zpzf\na9t5JU7Ar/gMfo/ECiosLFTbtm3Vo0cPzZ07l0E/neB0zTi/17bzSpyAX/EZPCyajljJmOi8nhqe\neuopm5GRYefPn+92KMHjdM04v9e280qcgF/5/DMoOq/jaA4ePKgbb7xRy5Yt07x58/S///u/bocE\nAEDKiraPFQOEBtCXX36piy++WCeccIIKCwt1wgknuB0SAAC+QB+rgFm3bp3atm2rrl27av78+SRV\nAAAkEFesAmTq1Km66aablJeXp759+7odDgAAvlNtYmWMaSLpWUkNJVlJedbacZWW6SJpvqRPy2bN\ntdb+LbGhIlYlJSX6y1/+onnz5mn58uU644wz3A4JAABfiuZWYImkm621p0vKkjTcGHN6hOVWWmtb\nl00kVSni66+/Vu/evfXWW2+pqKgovqTKK7XtvMLp2n1+b0+n5eZK6enhARDT08OP/YTjBYhNNF8d\nrDgpfGXq3ErzukhaVJP3YbiF5Nu4caNt0aKFvfHGG+2hQ4fiezOv1LbzCqdr9/m9PZ2Wk3NkW5ZP\nOTluR5YYHC/ADygZwy0YY5pLek3SmdbavRXmd5E0V9JWSZ9LusVau/Fo78VwC8m1cOFCDR06VPff\nf7+uvPLK+N/QK7XtvMLp2n1+b0+npadLpaU/nJ+WJpWUOB9PonG8AD+Q8FqBxpjjJK2QNMZaO7fS\ncz+WFLLWfmOM6SVpnLX2FxHeI1tStiQ1bdr07M2RPriISygU0pgxY/TEE09ozpw5ateuXWLe2Cu1\n7bzC6dp9fm9PpxlT9XMujQ2YUBwvwA8ktFagMaa2pDmSplVOqiTJWrvXWvtN2e+LJdU2xpwUYbk8\na20ba22bjIyMaFaNGti3b58uuugi5efna+3atYlLqiTv1LbzCqdr9/m9PZ2Wllaz+V7D8QLErNrE\nyhhjJD0l6T1r7YNVLPOzsuVkjMkse9+diQwUR/fpp5+qQ4cOysjI0PLly3XyyScndgVeqW3nFU7X\n7vN7ezotO7tm872G4wWIXXWdsCR1UniYhbckvVk29ZI0TNKwsmWuk7RR0r8lFUjqUN370nk9cVas\nWGF/9rOf2XHjxtlQKJS8FXmltp1XOF27z+/t6bScHGvT0sIdu9PS/NNxvRzHC3AEUSswGJ588knd\ncccdmjp1qs4991y3wwEAwJeoFehzJSUluvnmm5Wfn6+VK1fqf/7nf9wOCQCAwCOx8qCvv/5al1xy\niWrVqqXCwkKdeOKJbocEAABEEWbP2bhxo9q1a6czzjhDixYtIqkCACCFkFh5yJw5c9SlSxeNHDlS\nDz30kNLTueAIAEAq4S+zB4RCId1555169tlnlZ+fr7PPPtvtkAAAQARcsUpxe/fuVd++fbVixQqt\nXbvW3aSKoqyJ5fcivgAQQCRWKeyDDz5QVlaWGjdurKVLl6pBgwbuBTNtWnjww82bw6UuNm8OPya5\nik1urvT444frzZWWhh+TXAGApzGOVYrKz8/XFVdcob///e/KToXRnCnKmlh+L+ILAD7DOFYeZa3V\n/fffr4cfflhz585Vp06d3A4pbMuWms3H0UVKqo42HwDgCSRWKaS4uFhDhw7Vhx9+qMLCQjVp0sTt\nkA5r2jTyFSuKssYmLa3qK1YAAM+ij1WK2LRpkzp27KjatWtr5cqVqZVUSRRlTTS/F/EFgIAisUoB\ny5cvV1ZWlq688ko988wzOvbYY90O6YcGDpTy8sJ9qowJ/8zLC89HzU2YIOXkHL5ClZYWfjxhgrtx\nAQDiQud1F1lrNW7cON1zzz2aPn26unfv7nZIAAAgAjqvp7ji4mJde+21euedd1RQUKBTTz3V7ZAA\nAECcuBXogs8++0znnHOOQqGQVq1aRVIFAIBPkFg5bM2aNWrXrp0uvfRSTZ06VfUqdwgHAACexa1A\nB02ZMkV//vOfNWXKFPXq1cvtcAAAQIJxxSpBjlZGr6SkRDfddJPGjBmjFStWBC+posZgZLSLt7H/\nAETAFasEKC+jV1wcflxeRk+SevferUsvvVShUEhFRUWqX7++e4G64WiNE+ShGmgXb2P/AagCwy0k\nQFVl9E455T867rg+Ov/88zV27Filpwcwj6XGYGS0i7ex/4DAiXa4BRKrBKhVS/phMy6WdJUmTbpH\nQ4cOdSGqFBG5ccKDjIZCzseTKmgXb2P/AYETbWJFH6sEOLJcnpV0r6Rr1LDhvGAnVVLVtQSDXmOQ\ndvE29h+AKpBYJcDhMnrfShok6QXVrVuoBx7o4G5gqYAag5HRLt7G/gNQBRKrBBg4ULrnnq2qU6ez\npJCaNHlNkyY1oQ+rRI3BqtAu3sb+A1AF+lglQEFBgfr166frr79et956q4wxbocEAAASiFqBDikf\n9PPpp5/WBRdc4HY4AADARSRWMTp06JBGjBihl156SStWrFCrVq3cDgkAALiMxCoG27dv14ABA1Sv\nXr1gDvoJAAAiovN6Da1fv15t27ZVx44dtXDhQpIqAADwPRKrGnjuued03nnn6cEHH9SYMWOUlpbm\ndkiA/+TmSunp4W/bpaeHHycTNf8AJBC3AqNQUlKiESNGaOHChVq+fLnOPPNMt0MC/Ck3V3r88cOP\nS0sPP54wIfHro+YfgARjuIVqfPXVV7rkkktUu3ZtzZgxg1t/QDKlp4eTqcrS0qSSksSvj5p/AKJE\nSZsE+Pe//622bduqTZs2eumll0iqgGSLlFQdbX68tmyp2XwAqAaJVRVmz56tHj166O6779a9995L\nfyrACVV9zpL1+aPmH4AEI7GqJBQKadSoUbr55pv1z3/+U5dddpnbIQHBUd6/Kdr58aLmH4AEo/N6\nBXv37tXgwYO1a9curV27Vg0aNHA7JCBYyjuo5+WFb/+lpYWTqmR0XJcOd1AfOTJ8+69p03BSRcd1\nADGi83qZjz/+WH369NE555yj8ePHq06dOm6HBAAAUgSd12sgPz9fHTp00PXXX6+JEyeSVAEAgJgE\n+lagtVb333+/xo0bpzlz5qhTp05uhwQAADwssInVd999p+zsbG3cuFEFBQVq0qSJ2yEBAACPC+St\nwG3btqlLly46cOCAVq5cSVIFAAASInCJ1fr165WZmanevXtr5syZqlf5q9YAAAAxCtStwFmzZum6\n667TxIkT1a9fP7fDAQAAPhOIxCoUCmn06NF65plntGTJErVu3drtkAAAgA/5PrHas2ePrrjiCn31\n1VcqKipSw4YN3Q4JAAD4lO/7WF100UVq1KiRli9fTlIFAACSyvdXrGbOnKmMjAy3wwAAAAHg+ytW\nJFUAAMApvk+sAAAAnEJiBQAAkCAkVgAAAAlCYgUAAJAg1SZWxpgmxpjlxph3jTEbjTE3RFjGGGPG\nG2M+Msa8ZYw5KznhAgAApK5orliVSLrZWnu6pCxJw40xp1da5nxJvyibsiU9ntAo4W3TpknNm0u1\naoV/TpvmdkQAACRFtYmVtXabtXZD2e/7JL0nqVGlxS6U9KwNK5B0ojHm5IRHC++ZNk3KzpY2b5as\nDf/Mzia5AgD4Uo36WBljmkv6f5IKKz3VSNJnFR5v1Q+TLwTRyJFScfGR84qLw/MBAPCZqBMrY8xx\nkuZI+pO1dm8sKzPGZBtj1hlj1u3YsSOWt4DXbNlSs/kAAHhYVImVMaa2wknVNGvt3AiLfC6pSYXH\njcvmHcFam2etbWOtbcOI6AHRtGnN5gMA4GHRfCvQSHpK0nvW2gerWGyBpCvKvh2YJWmPtXZbAuOE\nV40ZI9Wrd+S8evXC8wEA8JloijB3lDRY0tvGmDfL5t0uqakkWWsnSlosqZekjyQVSxqS+FDhSQMH\nhn+OHBm+/de0aTipKp8PAICPGGutKytu06aNXbdunSvrBgAAqAljzHprbZvqlmPkdQAAgAQhsQIA\nAEgQEisAAIAEIbECAABIEBIrAACABCGxAgAASBASKwAAgAQhsQIAAEgQEisAAIAEIbECAABIEBIr\nAACABCGxAgAASBASKwAAgAQx1lp3VmzMDkmbXVl5cp0k6Su3g0hRtE1ktEtktEtktEtktEtktEvV\nato2zay1GdUt5Fpi5VfGmHXW2jZux5GKaJvIaJfIaJfIaJfIaJfIaJeqJattuBUIAACQICRWAAAA\nCUJilXh5bgeQwmibyGiXyGiXyGiXyGiXyGiXqiWlbehjBQAAkCBcsQIAAEgQEqs4GGPSjDFvGGMW\nRXiuizFmjzHmzbJplBsxOs0Ys8kY83bZNq+L8Lwxxow3xnxkjHnLGHOWG3G6IYq2Ceoxc6IxZrYx\n5n1jzHvGmPaVng/kMRNFuwTueDHGtKywvW8aY/YaY/5UaZnAHS9RtkvgjhdJMsbcaIzZaIx5xxgz\nwxhTt9LzCT9e0uN9g4C7QdJ7kn5cxfMrrbUXOBhPquhqra1qbJDzJf2ibGon6fGyn0FxtLaRgnnM\njJOUb63tb4ypI6lepeeDesxU1y5SwI4Xa+1/JLWWwv/YSvpc0ouVFgvc8RJlu0gBO16MMY0k/VHS\n6dbab40xz0u6VNKUCosl/HjhilWMjDGNJfWWNMntWDzmQknP2rACSScaY052Oyi4wxhzgqTOkp6S\nJGvtQWvt7kqLBe6YibJdgq67pI+ttZUHmg7c8VJJVe0SVOmSjjXGpCv8z8l/Kz2f8OOFxCp2D0v6\ns6TQUZbpUHZp8WVjzBkOxeU2K2mpMWa9MSY7wvONJH1W4fHWsnlBUF3bSME7Zk6VtEPS5LLb6pOM\nMT+qtEwQj5lo2kUK3vFS0aWSZkSYH8TjpaKq2kUK2PFirf1c0lhJWyRtk7THWvtKpcUSfryQWMXA\nGHOBpO3W2vVHWWyDpKbW2l9JekTSPEeCc18na21rhS+vDjfGdHY7oBRSXdsE8ZhJl3SWpMettf9P\n0n5Jf3E3pJQQTbsE8XiRJJXdGu0j6QW3Y0kl1bRL4I4XY0x9ha9InSrpFEk/MsYMSvZ6Saxi01FS\nH2PMJkkzJXUzxkytuIC1dq+19puy3xdLqm2MOcnxSB1W9h+CrLXbFb7Hn1lpkc8lNanwuHHZPN+r\nrm0CesxslbTVWltY9ni2wglFRUE8Zqptl4AeL+XOl7TBWvtlhOeCeLyUq7JdAnq89JD0qbV2h7X2\nkKS5kjpUWibhxwuJVQystbdZaxtba5srfNn1VWvtEVmwMeZnxhhT9numwm290/FgHWSM+ZEx5vjy\n3yX1lPROpcUWSLqi7JsYWQpfmt3mcKiOi6ZtgnjMWGu/kPSZMaZl2azukt6ttFjgjplo2iWIx0sF\nl6nq212BO14qqLJdAnq8bJGUZYypV7bt3RX+wllFCT9e+FZgAhljhkmStXaipP6ScowxJZK+lXSp\n9f9orA0lvVj22U2XNN1am1+pXRZL6iXpI0nFkoa4FKvTommbIB4zknS9pGlltzE+kTSEY0ZS9e0S\nyOOl7B+TcyVdW2Fe4I+XKNolcMeLtbbQGDNb4dugJZLekJSX7OOFkdcBAAAShFuBAAAACUJiBQAA\nkCAkVgAAAAlCYgUAAJAgJFYAAAAJQmIFAACQICRWAAAACUJiBQAAkCD/HzC1lcWJj3kTAAAAAElF\nTkSuQmCC\n", 196 | "text/plain": [ 197 | "" 198 | ] 199 | }, 200 | "metadata": {}, 201 | "output_type": "display_data" 202 | } 203 | ], 204 | "source": [ 205 | "plt.figure(figsize=(10, 6))\n", 206 | "plt.scatter(X[y == 0][:, 0], X[y == 0][:, 1], color='b', label='0')\n", 207 | "plt.scatter(X[y == 1][:, 0], X[y == 1][:, 1], color='r', label='1')\n", 208 | "plt.legend()\n", 209 | "x1_min, x1_max = X[:,0].min(), X[:,0].max(),\n", 210 | "x2_min, x2_max = X[:,1].min(), X[:,1].max(),\n", 211 | "xx1, xx2 = np.meshgrid(np.linspace(x1_min, x1_max), np.linspace(x2_min, x2_max))\n", 212 | "grid = np.c_[xx1.ravel(), xx2.ravel()]\n", 213 | "probs = model.predict_prob(grid).reshape(xx1.shape)\n", 214 | "plt.contour(xx1, xx2, probs, [0.5], linewidths=1, colors='black');" 215 | ] 216 | }, 217 | { 218 | "cell_type": "code", 219 | "execution_count": 19, 220 | "metadata": { 221 | "collapsed": true 222 | }, 223 | "outputs": [], 224 | "source": [ 225 | "from sklearn.linear_model import LogisticRegression" 226 | ] 227 | }, 228 | { 229 | "cell_type": "code", 230 | "execution_count": 20, 231 | "metadata": {}, 232 | "outputs": [], 233 | "source": [ 234 | "model = LogisticRegression(C=1e20)" 235 | ] 236 | }, 237 | { 238 | "cell_type": "code", 239 | "execution_count": 22, 240 | "metadata": {}, 241 | "outputs": [ 242 | { 243 | "name": "stdout", 244 | "output_type": "stream", 245 | "text": [ 246 | "CPU times: user 0 ns, sys: 0 ns, total: 0 ns\n", 247 | "Wall time: 855 µs\n" 248 | ] 249 | }, 250 | { 251 | "data": { 252 | "text/plain": [ 253 | "LogisticRegression(C=1e+20, class_weight=None, dual=False, fit_intercept=True,\n", 254 | " intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n", 255 | " penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n", 256 | " verbose=0, warm_start=False)" 257 | ] 258 | }, 259 | "execution_count": 22, 260 | "metadata": {}, 261 | "output_type": "execute_result" 262 | } 263 | ], 264 | "source": [ 265 | "%time model.fit(X, y)" 266 | ] 267 | }, 268 | { 269 | "cell_type": "code", 270 | "execution_count": 23, 271 | "metadata": {}, 272 | "outputs": [ 273 | { 274 | "data": { 275 | "text/plain": [ 276 | "1.0" 277 | ] 278 | }, 279 | "execution_count": 23, 280 | "metadata": {}, 281 | "output_type": "execute_result" 282 | } 283 | ], 284 | "source": [ 285 | "preds = model.predict(X)\n", 286 | "(preds == y).mean()" 287 | ] 288 | }, 289 | { 290 | "cell_type": "code", 291 | "execution_count": 24, 292 | "metadata": {}, 293 | "outputs": [ 294 | { 295 | "data": { 296 | "text/plain": [ 297 | "(array([-80.62725491]), array([[ 31.61988897, -28.31500665]]))" 298 | ] 299 | }, 300 | "execution_count": 24, 301 | "metadata": {}, 302 | "output_type": "execute_result" 303 | } 304 | ], 305 | "source": [ 306 | "model.intercept_, model.coef_" 307 | ] 308 | }, 309 | { 310 | "cell_type": "code", 311 | "execution_count": null, 312 | "metadata": { 313 | "collapsed": true 314 | }, 315 | "outputs": [], 316 | "source": [] 317 | } 318 | ], 319 | "metadata": { 320 | "kernelspec": { 321 | "display_name": "Python 3", 322 | "language": "python", 323 | "name": "python3" 324 | }, 325 | "language_info": { 326 | "codemirror_mode": { 327 | "name": "ipython", 328 | "version": 3 329 | }, 330 | "file_extension": ".py", 331 | "mimetype": "text/x-python", 332 | "name": "python", 333 | "nbconvert_exporter": "python", 334 | "pygments_lexer": "ipython3", 335 | "version": "3.6.2" 336 | } 337 | }, 338 | "nbformat": 4, 339 | "nbformat_minor": 2 340 | } -------------------------------------------------------------------------------- /terstimonials.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/khangich/machine-learning-interview/968a286650c964f8583b73b3e0864475b86f22b6/terstimonials.png --------------------------------------------------------------------------------