├── EVALML Automating NLP.ipynb ├── EVALML With Machine Learning.ipynb ├── LICENSE └── README.md /EVALML Automating NLP.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "attachments": { 5 | "image.png": { 6 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAm0AAACjCAYAAADYdilTAAAgAElEQVR4Ae2d6ZMUR5rm55/Zj2trtmtrtrNfdm3XpnVw3yB1y2Z7x2w0PdPdazPW0vR0tzRCCCQhDiEOcUpq1N0z3a0bEPd930UBVVDFIW4KCgQCAQVV79rjHh7hERmRmeEReUTm88HN884I9yfcf/H46+5/MTg4KEwsA2qAGqAGqAFqgBqgBppbA3/BCmruCmL9sH6oAWqAGqAGqAFqABogtNFppNNKDVAD1AA1QA1QAwXQAKGtAJXEOyzeYVED1AA1QA1QA9QAoY3QxrsraoAaoAaoAWqAGiiABghtBagk3l3x7ooaoAaoAWqAGqAGCG2ENt5dUQPUADVADVAD1EABNEBoK0Al8e6Kd1fUADVADVAD1AA1QGgjtPHuihqgBqgBaoAaoAYKoAFCWwEqiXdXvLuiBqgBaoAaoAaoAUIboY13V9QANUANUAPUADVQAA0Q2gpQSby74t0VNUANUAPUADVADRDaCG28u6IGqAFqgBqgBqiBAmiA0FaASuLdFe+uqAFqgBqgBqgBaoDQRmjj3RU1QA1QA9QANUANFEADhLYCVBLvrnh3RQ1QA9QANUANUAOENkIb766oAWqAGqAGqAFqoAAaaGloe/JkUJ48Lnga8I6/XI73mBLLYPAJ7854h04NUAPUADVQfA20HLQN3B+S764NyrffDMmtM4OxqV+9Hrzf34vPDQlyk271Ws978PqQ9CPvsXM8jnl+2vvc6SHpPz0kN/Hce6yen/JePzUkN73Ubz02r/l5t/c55JVS15Dc6BqSm1Yyz5GrdFK/f+PkkCQnCd47gc+J3EDuJ+v5cbwucgP5cZE+P8dr4ed9eN7pfa4Tj/VznXvPj1k5HldI1zu8zyDvEFHPvRyPkfpPiXx3ZUgePyz+RcuGl3VIDVAD1EB7aqBloA2u2nfXPUg7OyQKzKI5wMwDNhvUDLD5uQdsCtx6BiUpDwGbD2oa2BSo9QTAZsDt5ikNcLG5DXDdg6JArnvIzxXEJQCcArNuD9iieZcGsJuVcoAZgM4AWlJ+QqRPwZsHZHjugVsot4HNBzUDbKW5ArdjQxrkIvl1PD8m4ucd3nMP2Ay4hXIb4I4GQHe/b4jDAAUYBmCn1J6dEuud9U4NJGugJaANQ6B3Lw/JrbODOgHM8DjBaTOvh8DNc9kCh804bVae6LhFHDjLVQs5bL7zFjhtxnnzXbU4xy0B1EpcN7hr+KznstkOG17zn8Nhw/OyTpvnqhlwK3HYjNNm5YmOW8SB8x0247Ql5NU6bja42aBmOW/KcTuqXbfrR7XrxoYhuWFg2bBsqAFqgBpoPg20BLRhOFQDm3HYIo5btQ6bN0RqwC3RYTNDpcgTHDbzuu+weUOksQ4bhlD9IVPPYcNzAFz3oB5C9R23wdAQqXHYFKh1iwYz32nznvvAZhw3DWy+85bgsMU5bsZh08OkCQ6b7bxV67Dhc4C5OIdNva7BznfavCHT6yWO21B4iNQ4bFZuAO7BLTpubJSbr1FmnbBOqAFqIEkDhYe2R/cMsFl5jNN2ZvOAmNSrHj8ueY7XTTqz+bH/uHcTXn8sKt9k53gc83zjgH5942PpVWlArh8L4toCQCsf03atY1B6NzxOl9Y/lp71j6XXSuY5ciQ1BFrRaQvHtF3c/0Q6Pv1ODv3+jhz8/bdy0M+/lYMf35aDv/tWDnx8W6X9H9/ycv1838f9gtf2rUTql30rb8r+lf2yV6WbsnflDdm7EvlN2bPyhuz56Ibs+a2Xf9Qnu0Ppuuz+SKddH16XXR9ek50fXFP5LuQfXJWdH12V46vuqFg2BWgJjlt/l3CYlMOk1AA1QA1QA4XRQOGh7e5VwJo3qcDKTUzbgQ8eyCcv3pV//+s7DU97Fj6Qax1P9OSEuNg2K6btwLIHNTveT/72ruxf+kB8p83EusU4blePDMqGmTdkyeRelRZP6pHFk3vk/UmnVFo0sVsWTeqWhRO7VFow4YQsmHhC5o8/LvMnHJf3xh+T98Z3yrxxHTJvfIe8O+6oSnPHHpa54w7LnLGHVJo95qDMHntQZo3ZL7PH7Jd3Ru+TWaP3yczRe+Wd0Xvl7VG7Zeao3fLWyF3y9qhd8ubIHfLWyB3y5ojt8ubI7TJjxDaVpg/fItNHbJE3hm+WBT/eJ/v+0KedN8tpU5MVvKHSB7fptiXd0fF13u1TA9QANdBcGig8tN0+ZzlskZi2LTPu1Qx8XCFwzcv35JrnusU5bnht9Uv1Oe7Nb9wvG9d29fAT+fM/XQ6AbbIBttMBsE00wHZSFkxAOqFgTQNbp4I2wFoI2MYdEUCbAbY5ADaVDihoA6wB2gBrgLaZo/YoaAOsIb01cqcGtpFRYNsqgDYAG9K0YZtU2rr0knbdrJg2xLUh3b1IaGOj3FyNMuuD9UENUANJGig0tA08iAJbENN2cEXgVG2Zfl/O7xrwl/XwZ4maGLZoXmH2qL/8R5UxbZcOPJHdC773AXL3ggdSGtumJyfYDhug6vy2gVxj2s5tfSz4XQOd+5c+1LNFY2aNrn8ryWE7LaUO20lZMPGk5bB1ynsTwg7bvHFHZe7YIzJ33BEFbHPHHlKwNmcsoO2ASsZhA7DZDlsY2HZ6DtsOy2HbKtNHbA0D2/BN8vqzG+T1YRula913vuNmYtrguN3qZeOQ1DjwdWqDGqAGqIHm0kChoe3R/Si0ec/PDPpDolum3yuZRWqW/YibRVrLddoAawaWEOMGV80ks04bhi7xGYCVeU/NCFWTEvJbp82AG/4vbhbpN7sey5LJZ/SQaJzDFhoSBbDBYdMuG4ZDdToWctjehcMWctkChw3QhqFR7bJpYIPDhiHREmDDsGhoSNQAm3bZjMMGWHt92AaZ+ux6+XRaVxDjZjlut3rotLFRbq5GmfXB+qAGqIEkDbQItIVj2jCZwMCRctjMem3VziKt4LSZWaVp12mD42aOq2fDQElsW+8G67i3Dfjrs5n12hTEJSz/YWaRxuYmZs3KT68N/qtnnTc5wYppO/SHOwraEL+mY9gwJGoctlNWDJvnsE0wMWydMh8Om4plO6bi1+aN1w4boG0OYtkwNDrmoNgOG2Bt1hg9JKqGRQFsozEsag+J7lSxbIhnC2LYbGDboodE4bApYNuogG3qsA3y7gs7fKfNjmkjtLFxTGoc+Tq1QQ1QA82mgRaBtrDjZkObcdPi8kas02ag7cjHD5WTZq/TBpAz7/sum3HYTA5oS0op12kz/wWAi7pte1bcksWTe8PANumUmnSwyEw6mKiBzY9jCzlsx/TkAzXxIHDYgli2YEg03mEL4th0DJsNbHriAYZD9ZCo57Ahji0EbNplg9P25tjNKobNDI2amDZCGxvlZmuUeTzUJDVADSRpoMWgTce02dBmhkL93NqyKjG2rYLTljamTYGZt06bASXErmlgMzskDInttAHaar1OmzmWnrWW04ZFdE+KANowYxTumnLYvNmiwSzRk7IQMWwTjuvJB+PDDpuZLYoYNjUsamaLljhs+9WwqIlhU0OiymHDsChmi+6Ut0bZwObNEh2xVWaoGDYAGxw2PfkgcNgQy7ZBXnt2nUx9dp28OXaT77QZcFMxbRweLcxU96RGjK+zg6MGqIF20UDDoW3J9M/kx/9rqiye9qkMDDxO1YEkxbTZ0BbrsHlbWZn3bMetljFtgDQDSgeWa6fNOGoAtMxOm7cbghoirWLvUXMs8U7bbcHyHhra4LCdUpMPNLRpYAscNgyN6pmiWOJDAds4s7yHnima7LDp5T3UkOhoxLAZh223vOXNFMVwqEojwg6bnilqXDY9U1RBm5p8oF02QNtrz6xV0GbWbFMuG3ZOOCpCp42Nfbs09jxPap0aKL4GGgpti6Z+ImP/00t++vnYWdK5v7dqcAugLTmmzazX5jttHrDZoFbiuFVw2lxj2kLQVuK0YSFda3i0DnuPGmiLi2nbvbxfFk8+rWANa7ItxHpsWN5jQgBsCtpC67F5wKbWY4s4bP56bHqW6KzRxmHTy3vEO2zBemwzvPXYMBxqHDZAm+2wTUMcmwVsGBYFsL327FqZMWaD77Qxpq34DRc7H9YhNUANtKMGGgZtT5488WHNBjc8/uPijVWBWwBtxYtpM06bHiLVs0gzO205xrTtXtEfXkDXi2PDkChgzXfZlMPmrccWWkA36rDp5T38WaLeArpY4kOtxWYNiYbXYtML6ALaMPlAxbGptdi0wxasx6ZniprZohrY1ilg+9dnvlbQZjtsjGljg9+ODX4znPPz/+cn8h/+41+mSvhOMxw7j4HtRqM10JTQBnB77cWlcvl8X9kLtRTaGNMWnj1afu9R32mLiWmD0waHrXQ9NhvYzI4HtsOWvOMBZovaDhtmi9oOm73jAXY7QDybcdjMbFHEr+lh0fACuoC1UmBbJwA2pOmW08aYNja8jW542/n/CW28/tpZ/1nPvWmhDeD23H/7tWz6/EAiuJVCm3bcGNM2JGoDeS+uzcS4IVcJC+l2BfF1cTFtu5ff8maKmh0Puqrc8QAOG2aLBjse6N0O9GxRzBTVux2YHQ/0WmxmPTYTx6a2qFJrsVlbVGG2aMmOB3DYgvXYlMOGOLZnAWxrFbC9+vQaBW1Fjmk7fqJb0NkVPb0+Y3bi9Zy1MeP3iwEDjYK2tNfOiz97Sd5dsFQuXLzU8pr95PNV8tKvpqZuX/782VeZyiZLu4bvtuM139TQZoZN57/6R3nw/cOSCgqgjTFtYYdNwxmWBtHApjd/j+416jttMeu07V5+M35PUWs9tvg9RYMdD7BNFYBNOWxqT9EDek9ROGzejgeAtZmjd8vb3p6i/vIesXuKxux4YK/HpmLYDLBphw3A9uoza+SN0esKHdO2Z9/BVMNJaYef6vV5dJzt2NDynAOgbBS0uWr8v/z3v5JWBgTAqWvZAGqzaDtLu4bvZvnvon63ENAGeJv98u9KKiiANsa0qbXb8oxpW34zvIBuyZ6iegFdta+o2QQ+uuNBzJ6iTjse+A6bN1PU21M0cNismaK2w/aMBrZXn16toK3IMW1ZGjfXBrkW3yO0BfBS1E4j63EXDdpwHYwc/0JJ/5O1HJrh+yt++wdnYEO5ENrqfz0XBtoAbtElQUqhjTFtYcfNPaZtlwdtahN4fz02xLDpLarsHQ/exZ6i4/RsUbhr2FPU7Hgwy9tTVO94gKFRa4sqTD6wHTasx1bisNkL6CbseACHzVuPTQ+JrhU4bP/6zBp55enVoqFtre+0FTGmjdBW/8axGTrVVjyGIkIbACXrUGCz1eXtb+8IXMQsN2eEtvq3S4WCtsePn4TudkqhjTFtiFXTQ6JBXJt5jlylKmLadi3vV4vn6lmiek/R+TE7HgDYsHiuWUAX0Ibh0PAWVVjeI7JFlb8e2y69gO5Is4BuJIbNXkAXOx4MC7aowvIedgwblvcwEw8wJApYA7S98tQqeWP02kLvPUpoq3/j2GydbKscT1Gh7X8+NVYAOq1SDwCuLMBGp60xbVJhoO33C9aVXCwBtDGmTYGZiWGL5taeo9iuysS2lYtp27X8hrfjgZ4tCoctbk9RG9ji9xQ167FFHDa1p2jSjgfbZfrwZIcN67FN9dZjsx02rMf2qjdbFMCmoO2pVfLK06tk2qivfaetiOu0Edoa00C2SgfdTOdRVGjLA1KapR7ycNnyKI8s7Rq+2yzlWc/jaHpo+7//e5rs2XAstnICaGNMW94xbYC2YC02Myx6TIIdD8IOW+KOB/5s0b2h5T2CtdiMw4ZN4K212CyHTa3FNhw7HtgzRS2XTTlslsuGyQfKZdPA9punvpJpo78u9N6jWRq3rHfTeX4fHXY9Gzj+V/PBbpGhDcOJreC2YRZ3Htc1h0frf301NbTN/KeV0t/3bWIjXwptjGkLO25ZYto0tBmHDbDmA1t0x4Ox9vIe0R0Pog5bsKcowM1sUWXWY7Mdtukjgh0PAGy2w4ZhUSTEsMFh84dFjcP2dABsgLbXR63xnTbGtKVb2DSPxt38BqGt/o18s4FrkaENOs4KKo2uDyxhYq7HrHnWsshyM0qnbbC+jUm5HREw6eCrj3ckwpoRfSm0MaYtv5i2m2ozeB3H5gHb+A61r6iJY4O7FjhsSTse7JWZ2FNU7StqgG2X2gQey3sA2rTDZrls1gK605TDZlw2uGvaYfNj2ew4thiHDcD266e+VNBW5HXasjRuWRvmPL9PaKtvO2vaymbKiw5tcNuKvHbbL/5lKqGtzryT5/XXMKcNJzHpv/5LyVZWv3xhgfQcv1AR2PD9ANoY0xZ22LKv07ZzWZ+CtlKHLWnHAzhsBwWL59oL6GLHg3e8LarsHQ8Aa1hA13fYRmzT21QB2EZskTeGRXY8SNhTNHDY9PIeiF9DAqwZYAO0TR212nfaGNNGpy3PRpS/lQ5Eiw5tuIkB+BSx3vN02VAOdNrSaT8PzTQU2ua/8scQtH04a1WqCyGANsa05R3TBmjTy3scE7UWmxoS9YAN67GFhkSD3Q4UtI3Wm8BjeQ/bYVNxbKP0bFE1LBpa3gO7HaTZ8cAeEkUMm17eIw7YfvWDL+T1UasZ05Zyv8c8HTbzW3Ta6t/I59FR5PkbrQBt0HMR3bYsC+maa9jOCW31v54bCm1oCLo7zsvWrw5J99FzqYAN3y2FNsa0hR0395g2BW0TOtVwKKBND4keVbBmtqjCemxqiyrsKWrveKCgTceyvY0dD0ZhWHSXStphK7cem+WwDd8kWNbD31N02Hp57Zn4HQ/U0h5Pr9YO29N6SPQ3T30pv/rB5ypNHRk4bYxpo9OWJ4Twt9J1XK0CbUVz22oRYkFoS6f9PNqKhkNblpMohTbGtOUV07Zz2Q3BAro2sIXXYtPrsel9RTE0ul/0bgfh9dgAbQbY9BZVO+TNkdtVMpvA68kH2O0gaccDPekAy3uE9hRVa7EZhw1rsulhUQyHqvSDL3xoU05bh+i12o4G+a2eodQ3C1k06/rdLA1u1obV9Zj5vfo36EUo81aBNjhORQqGdyl321WLe5y1bcnSrhWp7PO8LlsE2pJj2s7vGpBbZ7UDd+uMlfcOinoel/cOST9eR96TnPf36PdVfnpQ7Pym91zlp4cE+aUDT8RfG23DgPTj9VODft67YcB///y2Aek/NSQY9jT5Te+5n3fr9/GZsMMWeR6zTtu5rY/9/+qJ2Xt0/+/6fWCbhx0PxoZ3PDB7is4O7XgQBjYMjYaBzThsiGUzMWxmPbaIw2bvKRpy2DAsqnc8wAK6ZscDxK+9EnLYvpBfP6Wh7dc/+FxmPrehbWPasjaseTY4/C2CnAs84DtZtRMHHVlfy+O4sp5XNd/PAkflyihr25LluAhtBZxNkeS03TozKJ/87V0FJVtm3BM8j0sKzAByCtAGA1DrxWP9eijv8V5HbgNb9PnpIQVigDI77V7wwAel68cGBfClwQ3wNiTXOgb993fPf6Be8wHNAJsFaiqOzX6eYu/RzW/cV/+FcsKCu9F0ZvsDAaz5M0VDcWwxOx5YQ6KYfKCHRLXL5m8CP1K7bD6wYU9Rfz02vduBveOBmSkadth0LFtoxwN/4sGXgiHRX//gC5XM0CjyP7xytG1j2rI2rNV0CvwMYaxaDbQStAFoigAP2Du1HHy5vpe1bSG0pW83Cu20DTwIT0DwHbWzg3JwRQBIO+Z8L5cOPU521gy4mbyC06ZAzgBcgsPWbzltALQ9C7/3gWz7rO9DDpsBN+QHlgXHvWfBA7m477GGN99xG1TumwE247BhWPRmtxfD5jtvpTFtF/c8kT3zg//Yv/Sh3DwpCtpUfgIAJ3LjxJB8MfUb32Hzl/cYEwW2fd5s0eT12MzSHnp5j6jDpodFAWtvIIbNd9h0LFsQw6bXYzN7ipodD37j7XgAWFOzRQFsxmHzckBb56rbvtMWimnrTX/RVNs55fm5LI1b1oY1z/PgbxVDb7Wsp1aDtmZ327BnqiuUVfpe1rYlS7tWBFiuxXVUaGhDgdw+FwU3z1U7OyhrXr7ng5IZlmx0vuale8pRM6CmnDQ4bnDSPLdt9Uv1Oe7Vv7hX4rAFjpvIxQMD8uGLXXrygT9bVM8UxbBoMFPUAza1FhuGRI3DZs0U9ddj26bcNThs01UMm15AV6/Hpnc8UA7bsPCOB2ZfUeOwRXc8UA7bU6UOG4Bt3Xu9GthiYtruXGRMWy0aFv4m4SxJA60GbQCbZt5MHnumVoIv1/cJbfW/zgsPbXevPlExa2r482wQ29Z/ZlD6Tg7KbsvhajSwwWG71vHEj2HT4BbEtBlwu94xKLvnB85cLY5713sP/D1IzV6kcY7bxf0D8vlr52SOctgwWzRux4N9kS2q4vYUxeK52xWwzfCGRAFt07z12OCw2TsemD1FAWslOx48ozeBNxMPShw2b8bomxPXybYVF3yHzazPZnI4bg9uEdqSOle+Xv8GuR3KvBWhDWDUjHVXS5cNoEdoq38bUXhoe/TdkNw6G3HbEMOG17xYtr4TT6R384AaMsWwqUkH1OOHovMHXq6fH1gRky/HZx7KAeTLy+V4L0jdqwZExbB5MW43TR6JaTOum8kR49azYUAOLHtYXVrqfQ750oeCoc9o3rNuQK4eGVQTFwJXrTSmTb1nDZWe3fZA9n18Q3YsvSbbl16V7Sq/oh5vW3pFti1Buuzll9TjLUsuydYll2XLkouydfEl2bz4omxZfFE2v2/SBdn0vpUWfSObFn0jG1U6LxsXmXRONiw6JxsWmnRWNiw8K+tNWnBG1i88I+vm96qE9w7+6bqaKaoALcZhu35UDwE3Y0Mbd0xZhhGyNqxxx8PX6t9Yt0qZtyK0NaPbhj1Sa+myEdoa0wYUHtrQkN29aqDNmx1qgC3trNEaxbQpRw0xbpHZouHnZlLCoD9UqmeN6gkLwSzSbDFtgDG1LIjKrVg2PPdi2UxMW/Q5Xu8DyJ3QwIO877h+nph3DsmN4yJ9ZfK+Tu995MeGRD338uv+c3xGRD0/hhyfxdIdQ15uPfcgzThq0dzEtH3fXwyXDRontDWmgWwVUGqm82hVaAMgNdNm8rhZcx32rPZ7WW8Is7RrjGkr4OxR0xA9GRiUO5csxy3itBnHzeQYOsVyH/5za9kPzCT1l/vwHmPZD3/yQWjWKJb8sJb9UI8BX2b5Dw/EPGADpGlQC3LjqpmhUfM8lGOGaDWzR/G5Lm+5D0xM8JKarOC9jsc2uCW7bRrooo4bJijoZMANUOaBHHIAmp9LyfO+EMBpEAPQaXDzngPIFMBpEAOclUuAMPU+chvYjMNmvW6A7e7l4gAboY3AZtq6VshbFdrycJ7yql/AI/ZIrRa+XD9HaKt/29QSThuEDnBTjpsCNuO4RXI4aR6whZb7MA6bySvMHjXrt4WAzQc1DWxx67SFnbUY582bjGAcNp3XZp02DW6e02bWcUuYRWqcNz+Hw+Y5bokOmw9ulZ0248TZDlus46YcNuOsmdw4bDG5DXBHPaA7KnLvWrGADfrOckeatWHNqyPh79S/gW/GMm9laAMoNYPbVg+XLQ9IzdKu0WkrsNNmN0yP7g3J3StP5Pa5wEkzjlo0D4FbicOW/zptfixbGcct5LB5EKeW96jGaUuxTptx3JKdNs9RM0OmJQ6bcdqsPNFxizhwajjUuGxl8modN89pU46bDWqWw4bJFncvDamtz2y9FOVxlsaN0NZesHT8RLeC/PWbtqlAcdS/SR+s/Df1XiP3zSwKtGGbKhcHqtHXm6vL5lIvWc81S7tGaGsRaLM7Ybhvjx+1WHronU+5HO8x+WUAHdi6KOLjLI1b1oa1XuVlYAPnWk1qJHigTNA5VnOc5jO1OF78JkDspV9NTR10DlcIG4h/8vmqurpDLnCA72TVYVoAe/GnvxAXcEO51qKuqz1/l2P+z3/5V7J+49bUkJq1bcG1kbZezOfx3WrLpJU+1zLDo61UKTyX4kNW3nWYpXHL2rDmfS5Jv+fS2QBWGjEchTJNGzMEKE069zSvG1DLe5V7lGU9YKMo0IbjhLYANAYUqs2h5TR1mtdnUX/VHqP9OeiZ0FaMfofQ1oIOY14NAH+neS7idoA2QI3dkVT7uN4dpMvaVyPG/yhzJw4NwBmrtlxcP1dryC8StKENRHm4lGU9ADjaRrvc+PyPH4xR2nQ5z6xaydKu4bvR82+H520BbfdvPZHLRwak87MHLZUuHhyQO1cfpxZuX+8jObPje+nZqtPprfcllLbck9Nb7smpUPpOTm3RqXvLd9K95a50b7bTHenabKdvpWvzt3Jq2x251vN96mNsh4svzTlmadyyNqxpjjPrZ9GBNHsH6bL2VZYV8wGzLqDjUo7mO3Dx8nIGo5pwORd8J/o7aZ+bc6s2N/9ZFLfN1WUz2iS0Nc9Nejlttzy09W5+2HRbWeW9w8Gu+feratCudD6Sr1+9LR+/cENWvtAnH7/QJ7/90TVZ+aPr8tsfXVWPP/zhFfnoh1fkwx9eVumD5y/KB89fkhXPX5AVz1+U5c99o9KyKedl+XPnZemUc7JsyjlZOvmMLJ1yVpZM7lVp8aQeWTy5R96fdEren3Ra/vzL83Lh6L2qjrOcYNv1vXaBthW//YMTtL0+Y3ZdtOVSDxhecxnCxXemvTnHqTyqBZNyn8Pwby3ArWjQhjbHVZf1dINcytW4bDhHQhuhrS6NaLlOfNus+uzhmTeEuf5eubI4t+eBgjUDbIA2DWvXPGC76sGaDWyXBNCmge1CAGzPnZdlSIC1KWc9YDsTA2ynFbQtmtgtiyZ1y8KJXXJ657cN10W5cmrW91xgwXTIRXLaACrmuNPk9dpGyGX4yWX4FvXt4uilKbNqPlsLcHOBC+N6Zbk+qzlf+zPR/3RxgaO/keX4y33XtX0wLht+m9BGaGto52w7bLsW3pfr3QMNPZ5yF5zrexgaPbk6cBK71jyMPUcMD//b39zUDvtyZJgAACAASURBVNuP+pTLBocN6aMfwmG7Kr7D9rztsF2UFc/BYbOAbYoGNsDaMuWswWE7I4tjHbZTYgMboO3957vk7s1HscfpWg7t8D3XRhmdUJGgDXXpAkY4z1q4Qra2AJRpJx+4HJdL52nDRt6Pcc55xmcVFdpcYhlRF7h2bR3V4nEeZeqiu6xtS5Z2rR7lWou6yvqbLTs8ummadtn2L69u6DBrQTby+ydXPfCHgAFo0WM59vk9f0h05QsYCkW6Jh/96KpOajjUOGxw18o5bHDXAodtyRTtsAHa9JAo3DXLYZuoHTYA24IJJ2XBhBOy74/XS44xesx8Hr7ry9K4ZW1Y610Xrufq4milOTeXTjvtBATM4MwbuvL4PcS4pSmrcp/NAzDK/X7Se2nLIc4la0a3zfV6iUIPoS3c5ibpqNGvtyS0Pbr/xIeY2xfTB+o3ulJc/v/zn95V54wJF9Hvr59+W1ZmdtjOqSFRNRzq4LApYJt4UuZPOC6fvXqu5Bijx8zn4QbEtWFGR1U0aEPdu3SOcIRqqRus25W247eHnyodG+Ly0v5+PT+fl46KDG2u12EaHVTSSfR9l2H0OCAltIXb3Gg5N8vzloS2aycHfGhrloKu9XFsnqGdRcyQjf7XV7+8JdphC4ZElctmJhw8f0k+/KF211QMW2hIVMeuLXvOc9c8YAscth5ZPOm0SnrSgTck6jts2l2DwwZgQ/r456dLjjF6zHwebkBcO4uiQptr4DfWmqqFdlxn5qWZgIDPuqwJVi9wAxSnOZ+keigytOGcXI6/VjGXLu4v9BJ12XBehLZwm5uk30a/TmhrkXXaykNbvxoONTFsADY9O/SKGgoFsGFmaBjYMDv0GzXZIA7YMDMUM0UxFLp48mlZpGaJ2jFsJ9XEA8DagokBsL03vlM+/vmpmnSsjb6Yavn/7QZtgAMXGKnVEKkLRLoci0vHacoJ7iT+Ex059GInwCycPBcH0/w+8jzcNhfoiXOG0l5v9nlU8zjpP12vxVq4bS4uGxzjuLJz0V5WPbiWJeoP3407j1Z/jdDWBtD25T/f9CcdYOIBJh0A2nTsmolf8yYdPGcmHXgOm4lfMw7bZD3pQC/nYcWvTarssAHYkFb+LJ+V4Vv94rTPL0vjVk0Hledn8mpMASAux5WHG2SXPR67dI4u5YBjTwtWKKc0kzAAD66OXh5D0EWHNujB5RygoTy16eqyJU0qIbTRaWsYBXN4NCy+L395U80SDRw2DWxxDptZfw3LeSiHbfJZb/21M4Ih0WD9Nc9hm4h12AKHDbFrCyd6Q6Jw2MbrIVEFbBM6Zd64Dln5c0JbFAoqPW9HaHPZVgeQl7ejASBKC48Ar0p1mvR+tecNWEvqgJN+27yOc8IkibTnlUf5ugBPkutlzqeaPO25lvtP1+HyrM6UOU/AHwA67TlBM+Y3ojmhLdxvRsunWZ7TaWsDp+2Lf74RWoMt7LBd0Mt6qEVzg/XX/DXY1OxQa4aoWjDXdti69bIeE7v0cKgBNj+GTbtr740/JvPGd6j02591JTYczXJhNNtxtCO0oQ7Suk7oxMp1ti716jJBAMOpLv9lvlPOZQRsubh45rdN7gKjKN9yHb/57XJ5K0Abzq9cHSXBVF5xgS6AhWMqB/kuv5kVQrO0a3lcA+V02qzvEdraBNqCHQ4uyQdeDJvZ4WDF84hd0zsclDps0R0ONLAtmnhKxbFhKY9F3nIevsMGYPMdtmOigG2cBrZ3xx0VQlv6O7osjVtSB1Kr1/NsTF06kkqdU9rG2MXRyGMYDOduD2MC1vJ2EV2ANOsQaatAW6PcNleXDXVdTvsu1xqhLX1bXq4OqnmP0NYG0Pb5y32hGDbAmr1gLrajMjschB02D9gmmy2pPGDDcKi3wwGgTcFarMPmAZvnsAHYFLT99GTZxqMa4bbbZ9oV2lw7xqxOl9FXtUOVNgBndaLMf9cjdy3fco5NpeNuFWjDebpCb5byc4GrarZSc/ldQhuhLZfOnDFtYSF9/s99etIBHDZrOY84h22J2kM0aUuq06EdDuIcNsSuzZ+AIdFShw3ANnfsYfnoZ4S2Sh1b9P12hTaUg8v6aHktBuvy33k6jVEd1OK5S2wbYNb1WFoJ2uB62W6oDe/lHruCPWDPxfmtBq4IbeF+01Xftf4enbY2cNo+e/m6t4doZYfNrL+G5TwwQxQTD/wdDrwJB9pd8xw2b5eDBd4abHqGqOWwjetQ7tq7447IXCRA209PODf4tb4gmvX32xnaXNwudJhpZlTG1Ts65HIdb9x7WSYgxB1DPV5zic2qBgKSjr2VoA3n6AI70I6L2+ZSV9W4bK7nkUUH+M8s7VrRbo6Sroe0rxPa2gDaPn/5eshh0+uv2XuInpNkhw2zQ8MOm78llbfDAdZiS3bYjihoA6zNHXdY5ow9RGhz0FyWxi0OLmr5Wi0aU5cJCZVieCo1li5LKmTtxCodUy3ed4GOLOfZatBWL7cNkOdy3VZbV/XWAbScpV2rRTtTi+sr798ktDl0oHlXQh6/V25x3c9evhaKYVPxa2qHA7PLAYZDrSFR5bCVzhA1DpvZQ3S+mSHqrb8WGhL14teUwwZgG6uBDdD24T8cp9OWUndZGjeXhj7Ld2rRmLp0KFlXoccQa9pycHFP8rj+s/yGi7aSFmit5jhaDdpwzi6LL0Nbaa4VF5ctjfPrco1VC4RJunDRnrkm05Rd0v8X8XVCW8rOs1kruTy0XVW7G0R3OFhmFszF+muT9aQDvcMBhkTD668FwBbscIAhUX/9NS+GDXFr88ZjwgEcNj0cagPb7DEH5MOfdhLaUuouS+NmGrl65bVoTF1dBtdjcfm/LCDTyHbFRVsAL9djbkVoQ1m4uMHVlqPr8ixpZhsT2hjT5nxRuzYG5nuciBAW36cvXdFbUmGWqLXDwVJvDTYAm05B/Jq/YK6/h2iXBA6bXjB3vuewzVNrsB1TC+fqGaJJwHZQAG0f/MOxhmnDaKRouUvHWi9Ii/6PKyhVqhOXSQGuAd8uswKzBOdXOvdavu+irWphI+64WxXaXIbTce1Uc724lFkalw31RGgL95tx2m2G1+i0pXQ8mqHS4o6hnNMGaPPXX5sS2eEg5LBhSLSyw4ZN32MdNjUkmgRsBxSwzRq9n9DmoDmXjtXAFBp8NMj1SrUaInTpFF3XFEu7bVXaDjLuGm7Uay7aIrTFd/Aublulmc4u9YNrP43LBu0R2uLrtFHXZdL/EtocOtCkwmzk6+Wg7ZOXLoccNh2/FuwhunhSEL9W6rB5W1KZ+DUDbMZdwxpsVvxa6ZCodtfgsM0as1+lFX/fQactpe5cG2403miMG6nNPP/bZXmFtJ2XS1kXuYxdzpfQFt/Bu5RlJcBycdmwjEva647QFl+nacux1p8ntKXsPGtdIa6/XwnalvoOG2LXAGw9gmU99HIep9XuBgA2xK4tUkOi2EMUQ6InVIK7ph02LOfRqbajwsQDPRx6VC3nAWCbYyYdjDmoZooC1hSwjdbA9s7ovbLiHwhtaevZtTNoNWhzGbZMG2vmEvBdK3cxrU6SPo8ZjtAQ0vpN20Ku60u/mpp6wgWhLbmDd4GspEkzrtc9vpekhaTXCW3JdZpUZo14ndDWBtD2519cCjZ9N+uvWTNEF6kdDvSm73rCgbXpe6zDprekMg6bWn/Nm3SA2aFzxgLYLIfNB7Z9Amhb/vdHUzcojbg4muk/XRvvVoM2lwkCKANASzX1ic+lXbw0LRRWcxyun4FO0PlOe3OO2oM17TAvyqqaRGhL7uBdr9U4R9gFAF3rhtCWXKeu12MtvkdoaxNog6vmL5hrbfqOPUQDh61LFk6IAFuZPUT1Dgc6hg2wNhfAphy2OGDbq4Bt5qg9hDYHzbl2BK0GbWgEXTqyuA4xrkHF56qBFvsz1f523P9lfQ2uGQDNZXkS+xzSPnYFA9f6y/J/pozreY4uk2aisW2u1zy+Z845TU5oI7Q5CSeNyJI+y9mjYfH96RcX44HN20NUD4fqHQ5Cm76rIVFsSaW3pZoX2UNUO2zWGmyxwAZ3TTtsM0fvEaRlPznSMG0kaabZX3dtwNFRoTFu9vNLc3wuYBXtEJP+L21nixi7pN+q1esANZchzbTQUu7zWSDKBbqz/J+ph3LnE/delv90dYRt4EqrRZxDlmMmtIX7TaObZstb0ml7dP+J/Ptf31Hp1vnHdW9UG1HJX/7jXXW+l48MlJzvV1Mvq+2oEMfmx7BNxKbvOoZNx65phw0L5iKObX4VDhugTQ+HHpLZHrDNiolhw5AoHDYA29ujdsvvf8nFddNqhNAWblBdJiRUijtz6Wiz7rpQrQ4wbItONe3QbRyM5PFaFjjAd9MeQ5b/M2Vc7/90iY00S9S4aBHnV0njpizickJbuI2JK6NmeK0loQ0Fu2naPQUxu+bfL4GYZij4PI/hzLZHPqTev/Wk5HwP/PGmmniggU3Hri2a1K0mGhhgWzBRzxJVEw7KbvqOBXOxj2iwwwGAbbYXw6ZmiI7eJ7PGaHctCmxvj9ol2z8+X3KMeZZHK/4WoS3coLpMSMCq9eW04bKqfZZOstyxmPeaDdYM+GSBqHaBNlfwMnVuyrra3ACf0U7anNAWbmPSll+9Pt+y0Na7+aEPMp2fPZD+c6UOVL0KuVb/c/viYzn67w/88+xa8zC2U7rb90gWT9HLemAoNApsGBIFrKXdQxQuW+Cw7fdmiSYB2y7lsr09epfcvvZ97HHWqpxa4XcJbeEG1aVDTJqhZ/SRNi4sC7iY/yyXY7HeZnHWouCQ5dzbBdpQty4ghJsHlwkkWW8gXI4V3ymn4UrvZWnX8N1Kv9+K77cstKGyts3SbpsZKm31vJxAu7d+q5byiAJbaJeDlHuIKocNLptZgw0Omxe/5jtso/SQKBy2t0bulGMbr7XlhVaubqp5L0vjlrVhreb4GvEZl84f2wHFHavLNkG1moAAp6XRMWtRSIs+J7SFbyLiNIXXUJcuQ/nR8q70PKvLhmMltFVXp0l1Xa/XWxraUIi249aq0FbtEPDZ/Xfl3/7xrLf+2knBkKgaDvUWzJ0/AeuvYR02vSUVJh6E9hDFkOgYLOlhOWzWch4YEp05KpglqmLYRsJh2yXL//6wnNpzM7bDrJfYi/w/hLbSBtVlQkJSDFra4dZaTUBAJ5/W8avUoVd6H+eSdiV/QlupHpPaFxcYqlRn9vuoP+gm6f+rfd3lOLPeEGZp1+i0OSxDUK0YGv05xHkhQB/DpK2ULh4ckDtX00+0uHj8O+lcf0uOrr4pR1ffCKUjq/vkyKo+0fl1ObLquhz20zU5vCpIh766KuF0RQ5+ZafLcnjNFTnfcStzg9JoDTX6/7M0blkb1kafe9L/u7gYSUOkaYchk+Av6Vireb0WwAa4ggsDDQByoaO4zi5th01oqx7aXHRqQ1mlx3ld32k1gOPK+t9Z2rU4HVdznRX9My3vtBW9gnj81TeOrVxWWRq3rA1rM5crgKRSpxZ9PzpEitix6GcqPY/+Rh5llHVIFI4LygNwlvb40nbYhLZ07ZKLK1xJg3g/L5cN+k2rAfx/1rYlS7tGaGthpy2PBpW/ka6RYnnlW15ZGresDWsz16VLLFrUJUsLfi77OlYqQxdwNJ06AArfr/Qf5d5P22ET2tJf32mHoE39lsvzvLbTaoDQll4D5a7Bat+j00ZYzdTYVys0fi7bBU5oSy4/QFS5ji36XnSINO1MPbgmees57THgnAABebkNaTtsQluyHpO0kbfbhvrPI5bNHG9aDRDa0mvAlHWWnNBGaMu9A8oiSH43viEgtMWXC/Ti0hma5RHSOnV5DkcZrbvULUC1kR02oS1Zj6Ze4/K0NxjRGw77ed43D4Q2tzqNq+davkZoI7QR2gqgAZeO3TTwaIxr2Yg0+rddAr1Nh5d2QV0MpeZ9vmmHZ/N2WHA+aTtsQptbB5/lOjbXM3JoIG8dptUAjiNr25KlPPDdvMugCL9HaCtAh10EIfEY3RrxasstS+OWtWGt9hgb+bm04GPgK+3+jmkD/Kspk7QzVw1wVvPb1X4mbYdNaHO/3lF2NoC5PG4GDRDa3DVQ7XUZ9zlCG6GtLe9W4i6GZn6N0Fa+gUw7zGni2tIAUy0mIEBzaTttM7Sbp14JbX9ZUg9ZwLRc3WS5lqGVWukwrQYIbeXbpHIayPJeW0Dbw++eyLWTA3Jq3cNc1mnD7+D38LtpC//y8e/l1JbvpHvTXelS6Y50bdLp5KY7cnLTt3Jyo0m35eTG23LCT7fkxMZbcmLDLTmuUr8c3xCkzg03JUg3pHO9TsfW94mdOtZfF5NO7e6Xuzfjt79Ke278fO0u4iwNfTs4bdBe2nihtKBXC3cj7TGgo6zFdZa2w84CNC5OU5b/M+WVFo7z+E/z39E8rcNrHzvagujv5fE8rQYIbbVr78vVZ8tDW+enwd6ctdgR4fDvqttHs2f7PfnT/7siy5/7RqVlU87JsufOy9IpZ1VaMvmMLJlyRm3svnhyr+jN3U/Loomn5P1Jp9QuBosmdgm2nYrbK9TsZPDuuKMyb/xRtak7djOYO/awSnPGHJQ53qbuatupyKbu6xedkYGH6SG0nLj4Xn4XNaGtclm6xKfZnWGlx3kG/ptrw6VezXfzyuHcpd2FIQvQENoGBWVeSW9x72cp90p6IbRVbmMqlWE93m9paNs8oz57j6575V7ZO5+OL+/IiucvBMD23HlZ9ty5MLBNBrD1amib5G3uPumULEKa2K2gDbCGraewsbvafsraKxTQNm9ch8wbd1RtPTXXBza97VQI2Mbsj9kjdLf87uVOefQw/Q4L9RBqu/+HS+duGv12cdoAVeac885NDFzeOnSp1zzj6vBbaYaITblmgQdCm4aDtHGYKHvoJW8Nmt8jtBHaaiYuI7Jyue2w7Zh7T+5ezxdG8Ht7l9wX497h/+KO52rXgzCwTQGwaYdt2ZSzEjhsvbLEdtgmlXfY5o/vlPBeoXDYOsIOm9ortLzDNnPUHlF7hI7Se4SuW9gTex5x58bX6neRu3TupoNtF2iDHl06QlNO5fJadpbl/jfuvbzqE8O9LsCGYyK0Zb/20856rtWNg2nHiwZt2EEEx9yIVItQCVMPlfKWddo+/bu7CqgAbJUKIcv7+H2A26c/uRv7P1vm3/AdtuVw2KYEDtvSKWcUtAUOW48aFlXuWsRhUy6b5bAB2NTG7okO22G1sXtZh230Hpk5are8rdIueWvkTpXu3IgH0CzlxO9ma+SzQFtcx9+o17J09tVoqBblVIvlFexzSVsXAK0sbhuG5l782UuZXMks9UinLWgLUI9Y+6+SBhCvWYvheVuHRYO2SmVWy/ez6N8uc5fHLQltmCRgHLD+cwOxMOVSWHHfwe+b/8L/Rj/z+xcveTFspQ4boA3AZhy2xZNPq+HQ+Bi2E3pYtMRh61AOG+LXysew7ZdZY/bLO6P3qhR22Hb7wPbmyB3SvetGyXlEz4vPg4a3HmVRCxipZaOW9Nv1aOwAWUn/7/I6YuVqWccuEANwS+v+Aday7m9qyi9LPbqcb5b/M3Vnjr3aPI//NP9dLge4JTnEADrAVLnv5/Ueoa10BnGSVuqljbi6bXloizvpvF8rB2164oEBNuOyaYcNsLZ4co8snqQdNsBaNIYt7LAdV0OicNiUy2bFsIWAbSzi2A4KJhyoSQexMWx7Shw2ABvS4bVX6tJI5F0Prfx7hLbqITnthISkhtm8XmuHI8vxovP45PNVsc4bIA26QWecdqKBOfekPEunhe8m/W7S61n+z7QLSb+d9Hoe/2n+u9ocAIc6Q6q17qLHRGgjtDWs47edtqgwa/G8HLQt82PYzqmh0bgYNjhsADbbYQOsIc2fUMFhG2s5bLExbEkOmx4WNUOiBthmjNgmh9debljd1aJ+WuE3CW3VQxs6u6SOOO3rtY4jgjbRUac9rkZ/PgvQENqq13I92y5CG6GtYR1/U0GbmnQQ57D1hhw25bJZs0QrO2wdguU9UjlsKobNdth0HFsAbNtFQdvXhLZ6NpbV/BehLV1HlzTclBZ2UO7V1E/Wz7iATNpzKff5auKq7O8T2tLpMas+6vF9QhuhrS6NXZyYmwnalmI9tinn1ISDuBi2wGHr1kt7eOuwYVkPtbSHimE7roZD/SHR8QC2qMOml/bwh0RHV+GwjdqphkMBbYA1pOnDt8phQlvDtBunZ7xGaEvXSeZRXrWegGDXteu6XTZIuT5GkDvcvjSLExPa0unRrutmfUxoI7Q1rONrLmjD4rl64VwdwxbvsGHhXDMk6gPbhLgYtiodNjXpYJ+edFClwzZ9xFZBIrQ1X4OcB4S4dup5fi9LZ5+2s8s6IQGdWNr/zPL512fMrvswKerDxE6lia3LUo/4blpNZfk/UyeN+E/z30XICW2Etro2ePZF0UzQhhg2vbRHeJZoicPmxbD5wDYewJbSYRt7QGb5DhuAbZ8Es0StGLY4h23EVpkxYqu8MXyLHP76UsPqzq5HPg7gkdAWlEW1unDphOyOHe5Xtf+V1+eybG9kH3s1j6NQmia2LgtEEdrSazkvfZX7HZfrJaqhcr8f915R27Us+o8rhzSvcfZoDhvGl5uIoCYeeMt6YKao3p5KTzrATgfaYdMumw9svsPmrcWGWaJqSLSKGLZEh223vDUqHMP25ggdw2YctunDtyhoO0Roq3tnXemiLWrjFoWHejZ2WYYcAU+V6qQW78P1coGaaDmXew4HEnqKO/5q3cks9ehyfln+z5xnuTKJey+P/zT/XYSc0EanLbZRqId4m8tp0w4blvXQ0GYB2yRsT6VnifrAVq3Dppb1OBQs65HaYdsuM0ZuV8OhxmEDtE0btlkIbc13J0xoc6sTV+dq/catDWu/0Ea6dKBx4GG/Vs2aX9UOkWYBGkKbm5Zr3Xe6aI5OW/3rkk5bjZ02tduBD2x6aQ/tsFn7iZpJB77D5q3DNh57idoxbHoD+Dk+sOm12LBorl4414phG21miWLh3F3y1ijsdqDXYZuhHLbAZcOQqE6bZdrwTXLo64sN7bBq3TgV8fcJbW6NI+DLBpdqHtdzAkI5LWK4Mo9ZsDgfdK4mdq3cf+K9akCX0Oamx0pl38j3CW102hrW8TeT01beYetSM0Thss1PdNgwJHpU5o49LHOxDpsPbN7CuRUdNr2nqL2sx5sJDtsbwzfL68M2EtpyAPm8G1904Ogoi54QbJ932VT6PUBImnKD21TpN+v5PoZ5cUw4j2qX5sBMUJR10jBopeOH3tCJJ6Usey/it3FcaRK+U+mYK72fRgP4bCO0Wukcavk+6jRtGWXRAc6lqO1aI7VBpy2HDrpcTJsdx1beYcOkA7OfqKvDttfb/N1MOgg7bDqGLXDYTAwbYA0O27RhGzW0raHTVsvGkb/dei5FPevUhh50mjb8NGLyRD3Pnf/Fa6fdNUBoqzG0mVmiCtj8GDbPYZsYddg61XBoMOmgjMMWGhLdp2DtnciQ6NujdlnrsG0X5bAND2aJTh+hY9gUtA3bqKBt6rMb6LTloIl2b1h4/uxcqQFqgBrIXwOEthw66HJOm9npYJECtrhZokkO21GZO658DNus0SaGTTtsM0fZDps1UzQSwwaHTbtsgcOmXbYNAmg7uOZC5qEIXqz5X6wsU5YpNUANUAPtrQFCW42hLXDYAGxxDhuGRKtx2A7K7LEH1YSDYNKBXoftndF71ebvgDZMOjAOGyYeYNJBEMO2TU04iHPYXn92g7w+DNC2ntCWgybYsLZ3w8r6Z/1TA9RALTRAaMuhgy7ntAHaAGt6L9GTemsqNUs04rD567BV67AZl22PGIcNsBZai62sw7ZZpg0zcWwBsE19dh2hLQdN1OJi5W+yE6AGqAFqoL01QGjLoYMuB20a2IzDdtKaJdop701I57DN9hfORQzbXrVF1dujditoizpsmHRgO2xqSDQUwwZg2yS2wwaX7bVn1hLactAEG9b2blhZ/6x/aoAaqIUGCG05dNDloQ3uWpzD1inzxh+zdjooddjmjAmvw6Zj2GyHzazFpodE3xrpbQDvAZvaAB77iYZi2JIctvXy2rPrPGj7hjFtOeiiFhcsf5MdATVADVAD7asBQlsOnXM5aFM7HUw8qfYR1XuJJjlsR9TEg2AdtoMyx4thMw4boK1ahw3ApqANwDYCi+duVkkPicY7bK89u1b+9Zmv5eAaQhsbxfZtFFn3rHtqgBpoVg0Q2uoAbYA1H9gw6SDOYbMWztUOm+WyqVmi6Rw2bE01fbh22cLAhrXY7Bi2wGEDsCEdILTRaczhumjWRo/HxQ6ZGqAGiqqBloe2/nMDNe2A8fvlnLZlP+4KgC0SwzZP7XQQOGxzx2IvUeOw6R0P3hm9T8zSHvYsUUw6sGeJmhi2sMO2Vd4YZjlsw8s7bAC2V59eI8d3XKlpmRX1YuFxs6GnBqgBaoAaaKQGWhLaUKCf/t1dBVM73r1XUwDZu+S+D21xFbl2zgVvp4NgtwNsS6VTeB02DIcC2maPAbDt92BNO2wAtpmjKsew6SHROIdtk9rtwCzroSYdeDFsxmF79Zk18urTq+X29fs1LbO4cuJrbAipAWqAGqAGqIHyGmhZaOv89IEPU3uX3pe71x/nCiL4vcO/+97/D/xfnNgudN4tmSVqHLZ3xx1Re4kmOmxjbGDTs0Tf9tZhw6QDNfHAmnQQxLBtDcewxTps69TEAx/Ynl4jgLYv3j0Wex5x58bXyl9cLB+WDzVADVAD1ECeGmhZaEMhrfvNPR+qzBBmLXL8T7lK2f+na2qWKGDNd9hCux1Udtj0WmzBLFEsnIslPZCCIdEUDps36UANiSqHbY0s/vlOeXCvtsPJ5cqJ77FxowaoAWqAGqAGkjXQ0tCGircdt1oA26GPv5eH3z0pC204juMbbsqyvzkhc8ceLK2AgwAAA6RJREFUEe2wHZa5Yw/7MWx6SPSAqBi2OIfNi2FLdti2yvQRFRy2YViHLd5h+2z2Ufn+u0cVz4MXU/LFxLJh2VAD1AA1QA3UUgMtD20oPEDVtZMDCuAAcXkk/F41sBatvHNHvpWOtX1yZPV1L12Tw6tNuiqHV1+VQ6tMuiKHVpl0WQ6u8tJXl+Sglw58dUkOfHUxlPZ/dUFU+vKC7P/yG9nnp/Oy78sg7f3inHRsviS3rpZ3CqPnwOdslKgBaoAaoAaogfproC2gjcKqv7BY5ixzaoAaoAaoAWogXw0Q2rgeFYdEqQFqgBqgBqgBaqAAGiC0FaCSeKeS750Ky5PlSQ1QA9QANVBEDRDaCG28u6IGqAFqgBqgBqiBAmiA0FaASiri3QCPmXex1AA1QA1QA9RAvhogtBHaeHdFDVAD1AA1QA1QAwXQAKGtAJXEO5V871RYnixPaoAaoAaogSJqgNBGaOPdFTVADVAD1AA1QA0UQAOEtgJUUhHvBnjMvIulBqgBaoAaoAby1QChjdDGuytqgBqgBqgBaoAaKIAGCG0FqCTeqeR7p8LyZHlSA9QANUANFFEDhDZCG++uqAFqgBqgBqgBaqAAGiC0FaCSing3wGPmXSw1QA1QA9QANZCvBghthDbeXVED1AA1QA1QA9RAATRAaCtAJfFOJd87FZYny5MaoAaoAWqgiBogtBHaeHdFDVAD1AA1QA1QAwXQAKGtAJVUxLsBHjPvYqkBaoAaoAaogXw1QGgjtPHuihqgBqgBaoAaoAYKoAFCWwEqiXcq+d6psDxZntQANUANUANF1AChjdDGuytqgBqgBqgBaoAaKIAGCG0FqKQi3g3wmHkXSw1QA9QANUAN5KsBQhuhjXdX1AA1QA1QA9QANVAADRDaClBJvFPJ906F5cnypAaoAWqAGiiiBghthDbeXVED1AA1QA1QA9RAATRAaCtAJRXxboDHzLtYaoAaoAaoAWogXw0Q2ghtvLuiBqgBaoAaoAaogQJogNBWgErinUq+dyosT5YnNUANUAPUQBE1QGgjtPHuihqgBqgBaoAaoAYKoAFCWwEqqYh3Azxm3sVSA9QANUANUAP5aoDQRmjj3RU1QA1QA9QANUANFEADhLYCVBLvVPK9U2F5sjypAWqAGqAGiqgBQhuhjXdX1AA1QA1QA9QANVAADRDaClBJRbwb4DHzLpYaoAaoAWqAGshXA4Q2QhvvrqgBaoAaoAaoAWqgABr4/wwsY6aj8DKpAAAAAElFTkSuQmCC" 7 | } 8 | }, 9 | "cell_type": "markdown", 10 | "id": "1dd280bc", 11 | "metadata": {}, 12 | "source": [ 13 | "![image.png](attachment:image.png)" 14 | ] 15 | }, 16 | { 17 | "cell_type": "code", 18 | "execution_count": 1, 19 | "id": "6a1ed9ad", 20 | "metadata": {}, 21 | "outputs": [ 22 | { 23 | "name": "stdout", 24 | "output_type": "stream", 25 | "text": [ 26 | "Requirement already satisfied: evalml in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (0.22.0)\n", 27 | "Requirement already satisfied: graphviz>=0.13 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from evalml) (0.16)\n", 28 | "Requirement already satisfied: pyzmq<22.0.0 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from evalml) (21.0.2)\n", 29 | "Requirement already satisfied: featuretools>=0.20.0 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from evalml) (0.23.3)\n", 30 | "Requirement already satisfied: lightgbm<3.1.0,>=2.3.1 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from evalml) (3.0.0)\n", 31 | "Requirement already satisfied: matplotlib>=3.3.3 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from evalml) (3.4.1)\n", 32 | "Requirement already satisfied: catboost>=0.20 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from evalml) (0.25.1)\n", 33 | "Requirement already satisfied: kaleido>=0.1.0 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from evalml) (0.2.1)\n", 34 | "Requirement already satisfied: click>=7.0.0 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from evalml) (7.1.2)\n", 35 | "Requirement already satisfied: numpy>=1.19.1 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from evalml) (1.20.2)\n", 36 | "Requirement already satisfied: scipy>=1.2.1 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from evalml) (1.4.1)\n", 37 | "Requirement already satisfied: shap>=0.35.0 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from evalml) (0.39.0)\n", 38 | "Requirement already satisfied: ipywidgets>=7.5 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from evalml) (7.6.3)\n", 39 | "Requirement already satisfied: networkx>=2.5 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from evalml) (2.5.1)\n", 40 | "Requirement already satisfied: seaborn>=0.11.1 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from evalml) (0.11.1)\n", 41 | "Requirement already satisfied: colorama in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from evalml) (0.4.4)\n", 42 | "Requirement already satisfied: pandas>=1.1.0 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from evalml) (1.2.4)\n", 43 | "Requirement already satisfied: sktime>=0.5.3 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from evalml) (0.5.3)\n", 44 | "Requirement already satisfied: plotly>=4.14.0 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from evalml) (4.14.3)\n", 45 | "Requirement already satisfied: category-encoders>=2.0.0 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from evalml) (2.2.2)\n", 46 | "Requirement already satisfied: imbalanced-learn>=0.7.0 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from evalml) (0.8.0)\n", 47 | "Requirement already satisfied: statsmodels>=0.12.2 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from evalml) (0.12.2)\n", 48 | "Requirement already satisfied: nlp-primitives>=1.1.0 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from evalml) (1.1.0)\n", 49 | "Requirement already satisfied: scikit-optimize>=0.8.1 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from evalml) (0.8.1)\n", 50 | "Requirement already satisfied: cloudpickle>=0.2.2 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from evalml) (1.6.0)\n", 51 | "Requirement already satisfied: psutil>=5.6.3 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from evalml) (5.8.0)\n", 52 | "Requirement already satisfied: requirements-parser>=0.2.0 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from evalml) (0.2.0)\n", 53 | "Requirement already satisfied: xgboost<1.3.0,>=0.82 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from evalml) (1.2.1)\n", 54 | "Requirement already satisfied: woodwork==0.0.11 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from evalml) (0.0.11)\n", 55 | "Requirement already satisfied: texttable>=1.6.2 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from evalml) (1.6.3)\n", 56 | "Requirement already satisfied: scikit-learn>=0.23.1 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from evalml) (0.24.1)\n", 57 | "Requirement already satisfied: six in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from catboost>=0.20->evalml) (1.15.0)\n", 58 | "Requirement already satisfied: patsy>=0.5.1 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from category-encoders>=2.0.0->evalml) (0.5.1)\n", 59 | "Requirement already satisfied: distributed>=2.12.0 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from featuretools>=0.20.0->evalml) (2021.4.0)\n", 60 | "Requirement already satisfied: tqdm>=4.32.0 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from featuretools>=0.20.0->evalml) (4.60.0)\n", 61 | "Requirement already satisfied: pyyaml>=3.12 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from featuretools>=0.20.0->evalml) (5.4.1)\n", 62 | "Requirement already satisfied: dask[dataframe]>=2.12.0 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from featuretools>=0.20.0->evalml) (2021.4.0)\n", 63 | "Requirement already satisfied: fsspec>=0.6.0 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from dask[dataframe]>=2.12.0->featuretools>=0.20.0->evalml) (0.9.0)\n", 64 | "Requirement already satisfied: partd>=0.3.10 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from dask[dataframe]>=2.12.0->featuretools>=0.20.0->evalml) (1.2.0)\n", 65 | "Requirement already satisfied: toolz>=0.8.2 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from dask[dataframe]>=2.12.0->featuretools>=0.20.0->evalml) (0.11.1)\n", 66 | "Requirement already satisfied: setuptools in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from distributed>=2.12.0->featuretools>=0.20.0->evalml) (52.0.0.post20210125)\n", 67 | "Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from distributed>=2.12.0->featuretools>=0.20.0->evalml) (2.3.0)\n", 68 | "Requirement already satisfied: msgpack>=0.6.0 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from distributed>=2.12.0->featuretools>=0.20.0->evalml) (1.0.2)\n", 69 | "Requirement already satisfied: tornado>=5 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from distributed>=2.12.0->featuretools>=0.20.0->evalml) (6.1)\n", 70 | "Requirement already satisfied: tblib>=1.6.0 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from distributed>=2.12.0->featuretools>=0.20.0->evalml) (1.7.0)\n", 71 | "Requirement already satisfied: zict>=0.1.3 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from distributed>=2.12.0->featuretools>=0.20.0->evalml) (2.0.0)\n", 72 | "Requirement already satisfied: importlib-metadata in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from fsspec>=0.6.0->dask[dataframe]>=2.12.0->featuretools>=0.20.0->evalml) (3.10.1)\n", 73 | "Requirement already satisfied: joblib>=0.11 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from imbalanced-learn>=0.7.0->evalml) (1.0.1)\n", 74 | "Requirement already satisfied: jupyterlab-widgets>=1.0.0 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from ipywidgets>=7.5->evalml) (1.0.0)\n", 75 | "Requirement already satisfied: ipython>=4.0.0 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from ipywidgets>=7.5->evalml) (7.22.0)\n", 76 | "Requirement already satisfied: nbformat>=4.2.0 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from ipywidgets>=7.5->evalml) (5.1.3)\n", 77 | "Requirement already satisfied: ipykernel>=4.5.1 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from ipywidgets>=7.5->evalml) (5.5.3)\n", 78 | "Requirement already satisfied: widgetsnbextension~=3.5.0 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from ipywidgets>=7.5->evalml) (3.5.1)\n", 79 | "Requirement already satisfied: traitlets>=4.3.1 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from ipywidgets>=7.5->evalml) (5.0.5)\n", 80 | "Requirement already satisfied: jupyter-client in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from ipykernel>=4.5.1->ipywidgets>=7.5->evalml) (6.1.12)\n", 81 | "Requirement already satisfied: decorator in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from ipython>=4.0.0->ipywidgets>=7.5->evalml) (4.4.2)\n", 82 | "Requirement already satisfied: pygments in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from ipython>=4.0.0->ipywidgets>=7.5->evalml) (2.8.1)\n", 83 | "Requirement already satisfied: pickleshare in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from ipython>=4.0.0->ipywidgets>=7.5->evalml) (0.7.5)\n", 84 | "Requirement already satisfied: backcall in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from ipython>=4.0.0->ipywidgets>=7.5->evalml) (0.2.0)\n", 85 | "Requirement already satisfied: jedi>=0.16 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from ipython>=4.0.0->ipywidgets>=7.5->evalml) (0.18.0)\n", 86 | "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from ipython>=4.0.0->ipywidgets>=7.5->evalml) (3.0.18)\n", 87 | "Requirement already satisfied: parso<0.9.0,>=0.8.0 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from jedi>=0.16->ipython>=4.0.0->ipywidgets>=7.5->evalml) (0.8.2)\n", 88 | "Requirement already satisfied: pyparsing>=2.2.1 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from matplotlib>=3.3.3->evalml) (2.4.7)\n", 89 | "Requirement already satisfied: python-dateutil>=2.7 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from matplotlib>=3.3.3->evalml) (2.8.1)\n", 90 | "Requirement already satisfied: cycler>=0.10 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from matplotlib>=3.3.3->evalml) (0.10.0)\n", 91 | "Requirement already satisfied: pillow>=6.2.0 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from matplotlib>=3.3.3->evalml) (8.2.0)\n", 92 | "Requirement already satisfied: kiwisolver>=1.0.1 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from matplotlib>=3.3.3->evalml) (1.3.1)\n", 93 | "Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from nbformat>=4.2.0->ipywidgets>=7.5->evalml) (3.2.0)\n", 94 | "Requirement already satisfied: jupyter-core in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from nbformat>=4.2.0->ipywidgets>=7.5->evalml) (4.7.1)\n", 95 | "Requirement already satisfied: ipython-genutils in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from nbformat>=4.2.0->ipywidgets>=7.5->evalml) (0.2.0)\n", 96 | "Requirement already satisfied: pyrsistent>=0.14.0 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets>=7.5->evalml) (0.17.3)\n", 97 | "Requirement already satisfied: attrs>=17.4.0 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets>=7.5->evalml) (20.3.0)\n", 98 | "Requirement already satisfied: nltk>=3.4.5 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from nlp-primitives>=1.1.0->evalml) (3.6.1)\n", 99 | "Requirement already satisfied: regex in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from nltk>=3.4.5->nlp-primitives>=1.1.0->evalml) (2021.4.4)\n", 100 | "Requirement already satisfied: pytz>=2017.3 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from pandas>=1.1.0->evalml) (2021.1)\n", 101 | "Requirement already satisfied: locket in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from partd>=0.3.10->dask[dataframe]>=2.12.0->featuretools>=0.20.0->evalml) (0.2.1)\n", 102 | "Requirement already satisfied: retrying>=1.3.3 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from plotly>=4.14.0->evalml) (1.3.3)\n", 103 | "Requirement already satisfied: wcwidth in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=4.0.0->ipywidgets>=7.5->evalml) (0.2.5)\n", 104 | "Requirement already satisfied: threadpoolctl>=2.0.0 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from scikit-learn>=0.23.1->evalml) (2.1.0)\n", 105 | "Requirement already satisfied: pyaml>=16.9 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from scikit-optimize>=0.8.1->evalml) (20.4.0)\n", 106 | "Requirement already satisfied: numba in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from shap>=0.35.0->evalml) (0.53.1)\n", 107 | "Requirement already satisfied: slicer==0.0.7 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from shap>=0.35.0->evalml) (0.0.7)\n", 108 | "Requirement already satisfied: wheel in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from sktime>=0.5.3->evalml) (0.36.2)\n", 109 | "Requirement already satisfied: llvmlite<0.37,>=0.36.0rc1 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from numba->shap>=0.35.0->evalml) (0.36.0)\n", 110 | "Requirement already satisfied: notebook>=4.4.1 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (6.3.0)\n", 111 | "Requirement already satisfied: terminado>=0.8.3 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (0.9.4)\n", 112 | "Requirement already satisfied: argon2-cffi in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (20.1.0)\n", 113 | "Requirement already satisfied: Send2Trash>=1.5.0 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (1.5.0)\n", 114 | "Requirement already satisfied: prometheus-client in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (0.10.1)\n", 115 | "Requirement already satisfied: nbconvert in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (6.0.7)\n", 116 | "Requirement already satisfied: jinja2 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (2.11.3)\n", 117 | "Requirement already satisfied: pywin32>=1.0 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from jupyter-core->nbformat>=4.2.0->ipywidgets>=7.5->evalml) (300)\n", 118 | "Requirement already satisfied: pywinpty>=0.5 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from terminado>=0.8.3->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (0.5.7)\n", 119 | "Requirement already satisfied: heapdict in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from zict>=0.1.3->distributed>=2.12.0->featuretools>=0.20.0->evalml) (1.0.1)\n", 120 | "Requirement already satisfied: cffi>=1.0.0 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (1.14.5)\n", 121 | "Requirement already satisfied: pycparser in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from cffi>=1.0.0->argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (2.20)\n", 122 | "Requirement already satisfied: zipp>=0.5 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from importlib-metadata->fsspec>=0.6.0->dask[dataframe]>=2.12.0->featuretools>=0.20.0->evalml) (3.4.1)\n", 123 | "Requirement already satisfied: typing-extensions>=3.6.4 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from importlib-metadata->fsspec>=0.6.0->dask[dataframe]>=2.12.0->featuretools>=0.20.0->evalml) (3.7.4.3)\n", 124 | "Requirement already satisfied: MarkupSafe>=0.23 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from jinja2->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (1.1.1)\n", 125 | "Requirement already satisfied: defusedxml in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (0.7.1)\n", 126 | "Requirement already satisfied: bleach in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (3.3.0)\n", 127 | "Requirement already satisfied: pandocfilters>=1.4.1 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (1.4.3)\n", 128 | "Requirement already satisfied: mistune<2,>=0.8.1 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (0.8.4)\n", 129 | "Requirement already satisfied: jupyterlab-pygments in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (0.1.2)\n", 130 | "Requirement already satisfied: entrypoints>=0.2.2 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (0.3)\n", 131 | "Requirement already satisfied: testpath in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (0.4.4)\n", 132 | "Requirement already satisfied: nbclient<0.6.0,>=0.5.0 in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (0.5.3)\n", 133 | "Requirement already satisfied: async-generator in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from nbclient<0.6.0,>=0.5.0->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (1.10)\n", 134 | "Requirement already satisfied: nest-asyncio in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from nbclient<0.6.0,>=0.5.0->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (1.5.1)\n", 135 | "Requirement already satisfied: webencodings in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (0.5.1)\n", 136 | "Requirement already satisfied: packaging in c:\\users\\win10\\anaconda3\\envs\\test\\lib\\site-packages (from bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (20.9)\n" 137 | ] 138 | } 139 | ], 140 | "source": [ 141 | "!pip install evalml" 142 | ] 143 | }, 144 | { 145 | "cell_type": "code", 146 | "execution_count": 2, 147 | "id": "d5343fc1", 148 | "metadata": {}, 149 | "outputs": [ 150 | { 151 | "data": { 152 | "text/html": [ 153 | "
\n", 154 | "\n", 167 | "\n", 168 | " \n", 169 | " \n", 170 | " \n", 171 | " \n", 172 | " \n", 173 | " \n", 174 | " \n", 175 | " \n", 176 | " \n", 177 | " \n", 178 | " \n", 179 | " \n", 180 | " \n", 181 | " \n", 182 | " \n", 183 | " \n", 184 | " \n", 185 | " \n", 186 | " \n", 187 | " \n", 188 | " \n", 189 | " \n", 190 | " \n", 191 | " \n", 192 | " \n", 193 | " \n", 194 | " \n", 195 | " \n", 196 | " \n", 197 | " \n", 198 | " \n", 199 | " \n", 200 | " \n", 201 | " \n", 202 | "
CategoryMessage
0spamFree entry in 2 a wkly comp to win FA Cup fina...
1spamFreeMsg Hey there darling it's been 3 week's n...
2spamWINNER!! As a valued network customer you have...
3spamHad your mobile 11 months or more? U R entitle...
4spamSIX chances to win CASH! From 100 to 20,000 po...
\n", 203 | "
" 204 | ], 205 | "text/plain": [ 206 | " Category Message\n", 207 | "0 spam Free entry in 2 a wkly comp to win FA Cup fina...\n", 208 | "1 spam FreeMsg Hey there darling it's been 3 week's n...\n", 209 | "2 spam WINNER!! As a valued network customer you have...\n", 210 | "3 spam Had your mobile 11 months or more? U R entitle...\n", 211 | "4 spam SIX chances to win CASH! From 100 to 20,000 po..." 212 | ] 213 | }, 214 | "execution_count": 2, 215 | "metadata": {}, 216 | "output_type": "execute_result" 217 | } 218 | ], 219 | "source": [ 220 | "from urllib.request import urlopen\n", 221 | "import pandas as pd\n", 222 | "\n", 223 | "input_data = urlopen('https://featurelabs-static.s3.amazonaws.com/spam_text_messages_modified.csv')\n", 224 | "data = pd.read_csv(input_data)\n", 225 | "data.head()" 226 | ] 227 | }, 228 | { 229 | "cell_type": "code", 230 | "execution_count": 3, 231 | "id": "d586d491", 232 | "metadata": {}, 233 | "outputs": [], 234 | "source": [ 235 | "### Independent And Dependent Features\n", 236 | "X=data.drop('Category',axis=1)\n", 237 | "y=data['Category']" 238 | ] 239 | }, 240 | { 241 | "cell_type": "code", 242 | "execution_count": 4, 243 | "id": "9ac07d78", 244 | "metadata": {}, 245 | "outputs": [ 246 | { 247 | "data": { 248 | "text/html": [ 249 | "
\n", 250 | "\n", 263 | "\n", 264 | " \n", 265 | " \n", 266 | " \n", 267 | " \n", 268 | " \n", 269 | " \n", 270 | " \n", 271 | " \n", 272 | " \n", 273 | " \n", 274 | " \n", 275 | " \n", 276 | " \n", 277 | " \n", 278 | " \n", 279 | " \n", 280 | " \n", 281 | " \n", 282 | " \n", 283 | " \n", 284 | " \n", 285 | " \n", 286 | " \n", 287 | " \n", 288 | " \n", 289 | " \n", 290 | " \n", 291 | " \n", 292 | "
Message
0Free entry in 2 a wkly comp to win FA Cup fina...
1FreeMsg Hey there darling it's been 3 week's n...
2WINNER!! As a valued network customer you have...
3Had your mobile 11 months or more? U R entitle...
4SIX chances to win CASH! From 100 to 20,000 po...
\n", 293 | "
" 294 | ], 295 | "text/plain": [ 296 | " Message\n", 297 | "0 Free entry in 2 a wkly comp to win FA Cup fina...\n", 298 | "1 FreeMsg Hey there darling it's been 3 week's n...\n", 299 | "2 WINNER!! As a valued network customer you have...\n", 300 | "3 Had your mobile 11 months or more? U R entitle...\n", 301 | "4 SIX chances to win CASH! From 100 to 20,000 po..." 302 | ] 303 | }, 304 | "execution_count": 4, 305 | "metadata": {}, 306 | "output_type": "execute_result" 307 | } 308 | ], 309 | "source": [ 310 | "X.head()" 311 | ] 312 | }, 313 | { 314 | "cell_type": "code", 315 | "execution_count": 8, 316 | "id": "62f23e2d", 317 | "metadata": {}, 318 | "outputs": [ 319 | { 320 | "data": { 321 | "text/plain": [ 322 | "ham 0.750084\n", 323 | "spam 0.249916\n", 324 | "Name: Category, dtype: float64" 325 | ] 326 | }, 327 | "execution_count": 8, 328 | "metadata": {}, 329 | "output_type": "execute_result" 330 | } 331 | ], 332 | "source": [ 333 | "y.value_counts(normalize=True)" 334 | ] 335 | }, 336 | { 337 | "cell_type": "code", 338 | "execution_count": 9, 339 | "id": "177656ff", 340 | "metadata": {}, 341 | "outputs": [], 342 | "source": [ 343 | "import evalml" 344 | ] 345 | }, 346 | { 347 | "cell_type": "code", 348 | "execution_count": 10, 349 | "id": "f2b2c06d", 350 | "metadata": {}, 351 | "outputs": [], 352 | "source": [ 353 | "#### TRain A\\nd test data split\n", 354 | "X_train,X_test,y_train,y_test=evalml.preprocessing.split_data(X,y,problem_type='binary')" 355 | ] 356 | }, 357 | { 358 | "cell_type": "code", 359 | "execution_count": 12, 360 | "id": "fdcbb5f2", 361 | "metadata": {}, 362 | "outputs": [ 363 | { 364 | "data": { 365 | "text/plain": [ 366 | "[,\n", 367 | " ,\n", 368 | " ,\n", 369 | " ,\n", 370 | " ,\n", 371 | " ]" 372 | ] 373 | }, 374 | "execution_count": 12, 375 | "metadata": {}, 376 | "output_type": "execute_result" 377 | } 378 | ], 379 | "source": [ 380 | "evalml.problem_types.ProblemTypes.all_problem_types" 381 | ] 382 | }, 383 | { 384 | "cell_type": "code", 385 | "execution_count": 14, 386 | "id": "322016fb", 387 | "metadata": {}, 388 | "outputs": [ 389 | { 390 | "data": { 391 | "text/html": [ 392 | "
\n", 393 | "\n", 406 | "\n", 407 | " \n", 408 | " \n", 409 | " \n", 410 | " \n", 411 | " \n", 412 | " \n", 413 | " \n", 414 | " \n", 415 | " \n", 416 | " \n", 417 | " \n", 418 | " \n", 419 | " \n", 420 | " \n", 421 | " \n", 422 | " \n", 423 | " \n", 424 | " \n", 425 | " \n", 426 | " \n", 427 | " \n", 428 | " \n", 429 | " \n", 430 | " \n", 431 | " \n", 432 | " \n", 433 | " \n", 434 | " \n", 435 | " \n", 436 | " \n", 437 | " \n", 438 | " \n", 439 | " \n", 440 | " \n", 441 | " \n", 442 | " \n", 443 | " \n", 444 | " \n", 445 | " \n", 446 | " \n", 447 | "
Data ColumnMessage
Physical Typestring
Logical TypeNaturalLanguage
Semantic Tag(s)[]
562Haha I heard that, text me when you're around
1253I'm thinking that chennai forgot to come for a...
1816Can you tell Shola to please go to college of ...
2054K k pa Had your lunch aha.
511staff.science.nus.edu.sg/~phyhcmk/teaching/pc1323
\n", 448 | "
" 449 | ], 450 | "text/plain": [ 451 | "Data Column Message\n", 452 | "Physical Type string\n", 453 | "Logical Type NaturalLanguage\n", 454 | "Semantic Tag(s) []\n", 455 | "562 Haha I heard that, text me when you're around\n", 456 | "1253 I'm thinking that chennai forgot to come for a...\n", 457 | "1816 Can you tell Shola to please go to college of ...\n", 458 | "2054 K k pa Had your lunch aha.\n", 459 | "511 staff.science.nus.edu.sg/~phyhcmk/teaching/pc1323" 460 | ] 461 | }, 462 | "execution_count": 14, 463 | "metadata": {}, 464 | "output_type": "execute_result" 465 | } 466 | ], 467 | "source": [ 468 | "X_train.head()" 469 | ] 470 | }, 471 | { 472 | "cell_type": "code", 473 | "execution_count": 15, 474 | "id": "83516bd9", 475 | "metadata": {}, 476 | "outputs": [], 477 | "source": [ 478 | "from evalml import AutoMLSearch" 479 | ] 480 | }, 481 | { 482 | "cell_type": "code", 483 | "execution_count": 19, 484 | "id": "f6a32174", 485 | "metadata": {}, 486 | "outputs": [ 487 | { 488 | "name": "stdout", 489 | "output_type": "stream", 490 | "text": [ 491 | "Generating pipelines to search over...\n" 492 | ] 493 | } 494 | ], 495 | "source": [ 496 | "automl=AutoMLSearch(X_train=X_train,y_train=y_train,problem_type='binary',max_batches=1,optimize_thresholds=True)" 497 | ] 498 | }, 499 | { 500 | "cell_type": "code", 501 | "execution_count": 20, 502 | "id": "bf7f8259", 503 | "metadata": {}, 504 | "outputs": [ 505 | { 506 | "name": "stdout", 507 | "output_type": "stream", 508 | "text": [ 509 | "*****************************\n", 510 | "* Beginning pipeline search *\n", 511 | "*****************************\n", 512 | "\n", 513 | "Optimizing for Log Loss Binary. \n", 514 | "Lower score is better.\n", 515 | "\n", 516 | "Using SequentialEngine to train and score pipelines.\n", 517 | "Searching up to 1 batches for a total of 9 pipelines. \n", 518 | "Allowed model families: lightgbm, catboost, decision_tree, random_forest, linear_model, extra_trees, xgboost\n", 519 | "\n" 520 | ] 521 | }, 522 | { 523 | "data": { 524 | "application/vnd.jupyter.widget-view+json": { 525 | "model_id": "a59e0fa5afe046b186d1c00f8ecb2d8f", 526 | "version_major": 2, 527 | "version_minor": 0 528 | }, 529 | "text/plain": [ 530 | "FigureWidget({\n", 531 | " 'data': [{'mode': 'lines+markers',\n", 532 | " 'name': 'Best Score',\n", 533 | " 'type'…" 534 | ] 535 | }, 536 | "metadata": {}, 537 | "output_type": "display_data" 538 | }, 539 | { 540 | "name": "stdout", 541 | "output_type": "stream", 542 | "text": [ 543 | "Batch 1: (1/9) Mode Baseline Binary Classification P... Elapsed:00:00\n", 544 | "\tStarting cross validation\n", 545 | "\tFinished cross validation - mean Log Loss Binary: 8.638\n", 546 | "Batch 1: (2/9) Decision Tree Classifier w/ Text Feat... Elapsed:00:00\n", 547 | "\tStarting cross validation\n", 548 | "\tFinished cross validation - mean Log Loss Binary: 0.802\n", 549 | "High coefficient of variation (cv >= 0.2) within cross validation scores. Decision Tree Classifier w/ Text Featurization Component may not perform as estimated on unseen data.\n", 550 | "Batch 1: (3/9) LightGBM Classifier w/ Text Featuriza... Elapsed:00:13\n", 551 | "\tStarting cross validation\n", 552 | "\tFinished cross validation - mean Log Loss Binary: 0.215\n", 553 | "High coefficient of variation (cv >= 0.2) within cross validation scores. LightGBM Classifier w/ Text Featurization Component may not perform as estimated on unseen data.\n", 554 | "Batch 1: (4/9) Extra Trees Classifier w/ Text Featur... Elapsed:00:26\n", 555 | "\tStarting cross validation\n", 556 | "\tFinished cross validation - mean Log Loss Binary: 0.252\n", 557 | "High coefficient of variation (cv >= 0.2) within cross validation scores. Extra Trees Classifier w/ Text Featurization Component may not perform as estimated on unseen data.\n", 558 | "Batch 1: (5/9) Elastic Net Classifier w/ Text Featur... Elapsed:00:39\n", 559 | "\tStarting cross validation\n", 560 | "\tFinished cross validation - mean Log Loss Binary: 0.543\n", 561 | "Batch 1: (6/9) CatBoost Classifier w/ Text Featuriza... Elapsed:00:51\n", 562 | "\tStarting cross validation\n", 563 | "\tFinished cross validation - mean Log Loss Binary: 0.526\n", 564 | "Batch 1: (7/9) XGBoost Classifier w/ Text Featurizat... Elapsed:01:03\n", 565 | "\tStarting cross validation\n", 566 | "\tFinished cross validation - mean Log Loss Binary: 0.179\n", 567 | "High coefficient of variation (cv >= 0.2) within cross validation scores. XGBoost Classifier w/ Text Featurization Component may not perform as estimated on unseen data.\n", 568 | "Batch 1: (8/9) Random Forest Classifier w/ Text Feat... Elapsed:01:16\n", 569 | "\tStarting cross validation\n", 570 | "\tFinished cross validation - mean Log Loss Binary: 0.155\n", 571 | "High coefficient of variation (cv >= 0.2) within cross validation scores. Random Forest Classifier w/ Text Featurization Component may not perform as estimated on unseen data.\n", 572 | "Batch 1: (9/9) Logistic Regression Classifier w/ Tex... Elapsed:01:29\n", 573 | "\tStarting cross validation\n", 574 | "\tFinished cross validation - mean Log Loss Binary: 0.214\n", 575 | "High coefficient of variation (cv >= 0.2) within cross validation scores. Logistic Regression Classifier w/ Text Featurization Component + Standard Scaler may not perform as estimated on unseen data.\n", 576 | "\n", 577 | "Search finished after 01:43 \n", 578 | "Best pipeline: Random Forest Classifier w/ Text Featurization Component\n", 579 | "Best pipeline Log Loss Binary: 0.154849\n" 580 | ] 581 | } 582 | ], 583 | "source": [ 584 | "automl.search()" 585 | ] 586 | }, 587 | { 588 | "cell_type": "code", 589 | "execution_count": 21, 590 | "id": "2484cf11", 591 | "metadata": {}, 592 | "outputs": [ 593 | { 594 | "data": { 595 | "text/html": [ 596 | "
\n", 597 | "\n", 610 | "\n", 611 | " \n", 612 | " \n", 613 | " \n", 614 | " \n", 615 | " \n", 616 | " \n", 617 | " \n", 618 | " \n", 619 | " \n", 620 | " \n", 621 | " \n", 622 | " \n", 623 | " \n", 624 | " \n", 625 | " \n", 626 | " \n", 627 | " \n", 628 | " \n", 629 | " \n", 630 | " \n", 631 | " \n", 632 | " \n", 633 | " \n", 634 | " \n", 635 | " \n", 636 | " \n", 637 | " \n", 638 | " \n", 639 | " \n", 640 | " \n", 641 | " \n", 642 | " \n", 643 | " \n", 644 | " \n", 645 | " \n", 646 | " \n", 647 | " \n", 648 | " \n", 649 | " \n", 650 | " \n", 651 | " \n", 652 | " \n", 653 | " \n", 654 | " \n", 655 | " \n", 656 | " \n", 657 | " \n", 658 | " \n", 659 | " \n", 660 | " \n", 661 | " \n", 662 | " \n", 663 | " \n", 664 | " \n", 665 | " \n", 666 | " \n", 667 | " \n", 668 | " \n", 669 | " \n", 670 | " \n", 671 | " \n", 672 | " \n", 673 | " \n", 674 | " \n", 675 | " \n", 676 | " \n", 677 | " \n", 678 | " \n", 679 | " \n", 680 | " \n", 681 | " \n", 682 | " \n", 683 | " \n", 684 | " \n", 685 | " \n", 686 | " \n", 687 | " \n", 688 | " \n", 689 | " \n", 690 | " \n", 691 | " \n", 692 | " \n", 693 | " \n", 694 | " \n", 695 | " \n", 696 | " \n", 697 | " \n", 698 | " \n", 699 | " \n", 700 | " \n", 701 | " \n", 702 | " \n", 703 | " \n", 704 | " \n", 705 | " \n", 706 | " \n", 707 | " \n", 708 | " \n", 709 | " \n", 710 | " \n", 711 | " \n", 712 | " \n", 713 | " \n", 714 | " \n", 715 | "
idpipeline_namescorevalidation_scorepercent_better_than_baselinehigh_variance_cvparameters
07Random Forest Classifier w/ Text Featurization...0.1548490.11030298.207418True{'Random Forest Classifier': {'n_estimators': ...
16XGBoost Classifier w/ Text Featurization Compo...0.1786390.11325497.932010True{'XGBoost Classifier': {'eta': 0.1, 'max_depth...
28Logistic Regression Classifier w/ Text Featuri...0.2140110.16562497.522538True{'Logistic Regression Classifier': {'penalty':...
32LightGBM Classifier w/ Text Featurization Comp...0.2145800.13626097.515944True{'LightGBM Classifier': {'boosting_type': 'gbd...
43Extra Trees Classifier w/ Text Featurization C...0.2522060.21619897.080377True{'Extra Trees Classifier': {'n_estimators': 10...
55CatBoost Classifier w/ Text Featurization Comp...0.5264030.51271793.906174False{'CatBoost Classifier': {'n_estimators': 10, '...
64Elastic Net Classifier w/ Text Featurization C...0.5428030.52915293.716325False{'Elastic Net Classifier': {'alpha': 0.5, 'l1_...
71Decision Tree Classifier w/ Text Featurization...0.8017660.55517990.718481True{'Decision Tree Classifier': {'criterion': 'gi...
80Mode Baseline Binary Classification Pipeline8.6383058.6238600.000000False{'Baseline Classifier': {'strategy': 'mode'}}
\n", 716 | "
" 717 | ], 718 | "text/plain": [ 719 | " id pipeline_name score \\\n", 720 | "0 7 Random Forest Classifier w/ Text Featurization... 0.154849 \n", 721 | "1 6 XGBoost Classifier w/ Text Featurization Compo... 0.178639 \n", 722 | "2 8 Logistic Regression Classifier w/ Text Featuri... 0.214011 \n", 723 | "3 2 LightGBM Classifier w/ Text Featurization Comp... 0.214580 \n", 724 | "4 3 Extra Trees Classifier w/ Text Featurization C... 0.252206 \n", 725 | "5 5 CatBoost Classifier w/ Text Featurization Comp... 0.526403 \n", 726 | "6 4 Elastic Net Classifier w/ Text Featurization C... 0.542803 \n", 727 | "7 1 Decision Tree Classifier w/ Text Featurization... 0.801766 \n", 728 | "8 0 Mode Baseline Binary Classification Pipeline 8.638305 \n", 729 | "\n", 730 | " validation_score percent_better_than_baseline high_variance_cv \\\n", 731 | "0 0.110302 98.207418 True \n", 732 | "1 0.113254 97.932010 True \n", 733 | "2 0.165624 97.522538 True \n", 734 | "3 0.136260 97.515944 True \n", 735 | "4 0.216198 97.080377 True \n", 736 | "5 0.512717 93.906174 False \n", 737 | "6 0.529152 93.716325 False \n", 738 | "7 0.555179 90.718481 True \n", 739 | "8 8.623860 0.000000 False \n", 740 | "\n", 741 | " parameters \n", 742 | "0 {'Random Forest Classifier': {'n_estimators': ... \n", 743 | "1 {'XGBoost Classifier': {'eta': 0.1, 'max_depth... \n", 744 | "2 {'Logistic Regression Classifier': {'penalty':... \n", 745 | "3 {'LightGBM Classifier': {'boosting_type': 'gbd... \n", 746 | "4 {'Extra Trees Classifier': {'n_estimators': 10... \n", 747 | "5 {'CatBoost Classifier': {'n_estimators': 10, '... \n", 748 | "6 {'Elastic Net Classifier': {'alpha': 0.5, 'l1_... \n", 749 | "7 {'Decision Tree Classifier': {'criterion': 'gi... \n", 750 | "8 {'Baseline Classifier': {'strategy': 'mode'}} " 751 | ] 752 | }, 753 | "execution_count": 21, 754 | "metadata": {}, 755 | "output_type": "execute_result" 756 | } 757 | ], 758 | "source": [ 759 | "automl.rankings" 760 | ] 761 | }, 762 | { 763 | "cell_type": "code", 764 | "execution_count": 22, 765 | "id": "15c8818a", 766 | "metadata": {}, 767 | "outputs": [ 768 | { 769 | "data": { 770 | "text/plain": [ 771 | "GeneratedPipeline(parameters={'Random Forest Classifier':{'n_estimators': 100, 'max_depth': 6, 'n_jobs': -1},})" 772 | ] 773 | }, 774 | "execution_count": 22, 775 | "metadata": {}, 776 | "output_type": "execute_result" 777 | } 778 | ], 779 | "source": [ 780 | "automl.best_pipeline" 781 | ] 782 | }, 783 | { 784 | "cell_type": "code", 785 | "execution_count": 23, 786 | "id": "1806b9b8", 787 | "metadata": {}, 788 | "outputs": [], 789 | "source": [ 790 | "best_pipeline = automl.best_pipeline" 791 | ] 792 | }, 793 | { 794 | "cell_type": "code", 795 | "execution_count": 24, 796 | "id": "d5bc8ec7", 797 | "metadata": {}, 798 | "outputs": [ 799 | { 800 | "name": "stdout", 801 | "output_type": "stream", 802 | "text": [ 803 | "************************************************************\n", 804 | "* Random Forest Classifier w/ Text Featurization Component *\n", 805 | "************************************************************\n", 806 | "\n", 807 | "Problem Type: binary\n", 808 | "Model Family: Random Forest\n", 809 | "\n", 810 | "Pipeline Steps\n", 811 | "==============\n", 812 | "1. Text Featurization Component\n", 813 | "2. Random Forest Classifier\n", 814 | "\t * n_estimators : 100\n", 815 | "\t * max_depth : 6\n", 816 | "\t * n_jobs : -1\n", 817 | "\n", 818 | "Training\n", 819 | "========\n", 820 | "Training for binary problems.\n", 821 | "Total training time (including CV): 13.2 seconds\n", 822 | "\n", 823 | "Cross Validation\n", 824 | "----------------\n", 825 | " Log Loss Binary MCC Binary AUC Precision F1 Balanced Accuracy Binary Accuracy Binary Sensitivity at Low Alert Rates # Training # Validation\n", 826 | "0 0.110 0.895 0.987 0.938 0.921 0.942 0.961 0.246 1594.000 797.000\n", 827 | "1 0.144 0.854 0.980 0.919 0.888 0.917 0.946 0.246 1594.000 797.000\n", 828 | "2 0.210 0.783 0.962 0.839 0.837 0.891 0.918 0.266 1594.000 797.000\n", 829 | "mean 0.155 0.844 0.977 0.899 0.882 0.917 0.942 0.252 - -\n", 830 | "std 0.051 0.057 0.013 0.052 0.042 0.026 0.022 0.011 - -\n", 831 | "coef of var 0.326 0.067 0.013 0.058 0.048 0.028 0.023 0.045 - -\n" 832 | ] 833 | } 834 | ], 835 | "source": [ 836 | "automl.describe_pipeline(automl.rankings.iloc[0][\"id\"])" 837 | ] 838 | }, 839 | { 840 | "cell_type": "code", 841 | "execution_count": null, 842 | "id": "524be0ad", 843 | "metadata": {}, 844 | "outputs": [], 845 | "source": [ 846 | "### Evaluate on the test data" 847 | ] 848 | }, 849 | { 850 | "cell_type": "code", 851 | "execution_count": 27, 852 | "id": "dbd8d23c", 853 | "metadata": {}, 854 | "outputs": [ 855 | { 856 | "name": "stdout", 857 | "output_type": "stream", 858 | "text": [ 859 | "Accuracy Binary: 0.9732441471571907\n" 860 | ] 861 | } 862 | ], 863 | "source": [ 864 | "scores = best_pipeline.score(X_test, y_test, objectives=evalml.objectives.get_core_objectives('binary'))\n", 865 | "print(f'Accuracy Binary: {scores[\"Accuracy Binary\"]}')" 866 | ] 867 | }, 868 | { 869 | "cell_type": "code", 870 | "execution_count": null, 871 | "id": "b20c9d68", 872 | "metadata": {}, 873 | "outputs": [], 874 | "source": [] 875 | }, 876 | { 877 | "cell_type": "code", 878 | "execution_count": null, 879 | "id": "4068564e", 880 | "metadata": {}, 881 | "outputs": [], 882 | "source": [] 883 | }, 884 | { 885 | "cell_type": "code", 886 | "execution_count": null, 887 | "id": "6c7d3fbb", 888 | "metadata": {}, 889 | "outputs": [], 890 | "source": [] 891 | } 892 | ], 893 | "metadata": { 894 | "kernelspec": { 895 | "display_name": "Python 3", 896 | "language": "python", 897 | "name": "python3" 898 | }, 899 | "language_info": { 900 | "codemirror_mode": { 901 | "name": "ipython", 902 | "version": 3 903 | }, 904 | "file_extension": ".py", 905 | "mimetype": "text/x-python", 906 | "name": "python", 907 | "nbconvert_exporter": "python", 908 | "pygments_lexer": "ipython3", 909 | "version": "3.7.10" 910 | } 911 | }, 912 | "nbformat": 4, 913 | "nbformat_minor": 5 914 | } 915 | -------------------------------------------------------------------------------- /EVALML With Machine Learning.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "attachments": { 5 | "image.png": { 6 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAuYAAADUCAYAAADQtObcAAAgAElEQVR4Ae2d53McR5rm95+5jxcXcRdxF3Nf9r6sLA1Agk5mbvf2dle7GzMrjWY0e9LIURQlDUUjWtGIMpRIaUfeUCIpUvTegwYkAHqJDvQgCYIE3osns7IqyzW6squB6u7nQ0V2VVd3V2U+VfnLp99686/6+vqEC+uAGqAGqAFqgBqgBqgBaoAaGFoN/BUbYGgbgPXP+qcGqAFqgBqgBqgBaoAagAYI5vzHgP+YUAPUADVADVAD1AA1QA0UQAME8wI0AkfJHCVTA9QANUANUAPUADVADRDMCeYcIVMD1AA1QA1QA9QANUANFEADBPMCNAJHyBwhUwPUADVADVAD1AA1QA0QzAnmHCFTA9QANUANUAPUADVADRRAAwTzAjQCR8gcIVMD1AA1QA1QA9QANUANEMwJ5hwhUwPUADVADVAD1AA1QA0UQAME8wI0AkfIHCFTA9QANUANUAPUADVADRDMCeYcIVMD1AA1QA1QA9QANUANFEADBPMCNAJHyBwhUwPUADVADVAD1AA1QA0QzAnmHCFTA9QANUANUAPUADVADRRAAwTzAjQCR8gcIVMD1AA1QA1QA9QANUANEMwJ5hwhUwPUADVADVAD1AA1QA0UQAME8wI0AkfIHCFTA9QANUANUAPUADVADRDMCeYcIVMD1AA1QA1QA9QANUANFEADBPMCNAJHyBwhUwPUADVADVAD1AA1QA0QzAnmHCFTA9QANUANUAPUADVADRRAAwTzAjQCR8gcIVMD1AA1QA1QA9QANUANEMwJ5hwhUwPUADVADVAD1AA1QA0UQAME8wI0AkfIHCFTA9QANUANUAPUADVADRDMCeYcIVMD1AA1QA1QA9QANUANFEADBPMCNAJHyBwhUwPUADVADVAD1AA1QA0QzAnmHCFTA9QANUANUAPUADVADRRAAwTzAjQCR8gcIVMD1AA1QA1QA9QANUANEMwJ5hwhUwPUADVADVAD1AA1QA0UQAME8wI0AkfIHCFTA9QANUANUAPUADVADRDMCeYcIVMD1AA1QA1QA9QANUANFEADBPMCNAJHyBwhUwPUADVADVAD1AA1QA0QzAnmHCFTA9QANUANUAPUADVADRRAAwTzAjQCR8gcIVMD1AA1QA1QA9QANUANEMwJ5hwhUwPUADVADVAD1AA1QA0UQAME8wI0AkfIHCFTA9QANUANUAPUADVADRDMCeYcIVMD1AA1QA1QA9QANUANFEADBPMCNAJHyBwhUwPUADVADVAD1AA1QA0QzAnmHCFTA9QANUANUAPUADVADRRAAwTzAjQCR8gcIVMD1AA1QA1QA9QANUANEMwJ5hwhUwPUADVADVAD1AA1QA0UQAME8wI0AkfIHCFTA9QANUANUAPUADVADRDMCeYcIVMD1AA1QA1QA9QANUANFEADBPMCNAJHyBwhUwPUADVADVAD1AA1QA0QzAnmHCFTA9QANUANUAPUADVADRRAAwTzAjQCR8gcIVMD1AA1QA1QA9QANUANEMwJ5hwhUwPUADVADVAD1AA1QA0UQAME8wI0AkfIHCFTA9QANUANUAPUADVADRDMCeYcIVMD1AA1QA1QA9QANUANFEADBPMCNAJHyBwhUwPUADVADVAD1AA1QA0QzAnmHCFTA9QANUANUAPUADVADRRAAwTzAjQCR8gcIVMD1AA1QA1QA9QANUANEMwJ5hwhUwPUADVADVAD1AA1QA0UQAME8wI0AkfIHCFTA9QANUANUAPUADVADRDMCeYcIVMD1AA1QA1QA9QANUANFEADBPMCNAJHyBwhUwPUADVADVAD1AA1QA0QzAnmHCFTA9QANUANUAPUADVADRRAAwTzAjQCR8gcIVMD1AA1QA1QA9QANUANEMwJ5qER8p3ufrlxoU+unemTq6f65epJqzzprZ/slyvYrtaxrV+unNDrV05Y6yewvV+uqrLPK7314/HyyvF+wXLVK826Ko/p96545WWU/iJi1nUpcrlTv6/LyHqnt26XHf1yBetWebnDW/dKs67K9n6xy0tYbxe57JVm3S+P6vcv2eVREbWeVB7pl0vYfkTksir71Wus6yWy3uatt+n3u1C2iaSWh7337fKwSBfWU8ornf3Sfa5f7vbwxsnOkxqgBqgBaoAaqIYGCOYEcwXmd27dk+u/9MvlY31yuTN5uaS29/vvX+rAfv2C0iwAW/Nal2a9Xy61Yz+vbLdLvAaI9nklXnvrqtTrXdZrvN+l4FWXeK3WvdKs+2Vbv3QNtBz29kF5uF8ueiVe2+vYrpZD3vZD/XLRXyR4fRDbRS6i9Je0dZELB7CfyEWUB7x1VeJ1wnprv97eKnJBLf1e6a3vt0q8HmA5v8/bB+U+EbXulXh9fq+3fa9I99n+0ICuGjcnfic7PWqAGqAGqIFG0wDBnGAuPTc8ED/mQbdVGhj3SwvGDZTHSg/OFaS39ykXGlBu1hPh3IPyrqQSEI7tCsbLKAHnykFG2aeAXUG5t13BugfpCr7bPAiPleLDuYZ0Dd1wlQHisRIQju0ejJvSh3Mf0g2cBzBuoDyxtKG81YP2pNKG8/0epEfK81jfL3Leg/Tz+7x1D8oNnNtQriDdgnKzjn9CGu2GyfMlJFAD1AA1QA1UUwME8wYHc4QlKJccTrlZ4IwP6JxrlxzuOpzxwCm3HHLlpJdyyj2HHE66ctDTnHPjoIedc+WQG+c8zSlPgPGynHPLLddQHjjoGsptl3yA1yWd8wDSlVN+0HbGU5zzmFNuHPOUcgCn3HfSE5zyNOfcOOjXThPOq3mD5ncTAKgBaoAaaCwNEMwbHMyvn72nYrVV+IqB8WP9ohzyiHPeueaudKzplbLK1d5+VtmBz9vrq/V6B8ofeyWpPLGhV0G775jHnHMP1o/0ySUPzlVpOeVntt+TjlW90rHqrr+0W6+xvX2lfk+X9/T6Sm/7ynvSvuKudKy8K2f39FlOuQZy3zmPOObGKbed85ObeuX4hl45tu6OHF+vy2PreqVzXY/allquvS3H1vVIp1d2rL2tPmOXHWt7RK2jXHNLvVblT7elHeuqvC3tP92Wo6tvqm1HV9/yyuh6t5zeejcIZ0lwzA2c37lOOGfH2VgdJ9ub7U0NUAPV0gDBvIHBHHHlvkvuQ3kQX37h4D3Z8e5t+frJG/LJ314b0mX5H7ul9bM7Xuy5gfGgNLHkBs5PrL8raybfrMoxf/ZP12Xz7NsepEfd8niM+bl9fbJu1iV57++Py4LxHTJ/fLvMH3dU3h53RC3zxrYJlrljD8vcsYdkzhgsB2X2mANqmdXSKrNa9svMln1qeWv0XsEyY9QemTF6t8wYtVumj9ol05p3yrRRO2Va8w6Z2rxdpjZtU8ubTVtlCpaRW2TKyM3y55Gb1PLGiI2C5fURG+T1EevlteHr1DJ5+FqZPOwneXXYT/Lub3bL9o8vKkC3Y8zx2ix4WLdaNyh+Lzs/aoAaoAaogUbSAMG8gcH81uVo+EoQYw4oX/7v3VUB20ogf/20mwPGmh/4y51BOe6v/+2GHF97N3DQI445nHI45J/89rQCcgXl49oVmIegfJyB8sMKyOeMjUJ5q8wcHYfyt0bvUUA+w4Py6aN2ytTmHRrMAeXN2wRQrsAcUN60JQ7lw9crMPehfNhaAZgDyrFMeni1TBq2Wr6fdkw/EJrgnCMUppFumjxXQgI1QA1QA9RAtTRAMG9gML9xLuKYWzHmNpT/NLlb2pb3qjAWhLL4y2rvNcpSiwpT6dXhKng90IKwE+yjwk965cCnd2T5M8EgYcc7PR6cB4657ZQb8DfOdvvKXmlf1SuqxOuBlhW90m4tR73XKA9/3Svr37zlg/93z3QHWVj8zCyBi/7ZMz8HUA6nfHy7vO255fPG2U45oDzJKW+VmcYt95xyAPkMLJ5TDiDHAqdcueVN2i0PQflIQLl2y+NOuXbLlVNuQ/mwNQrMX3n4R8Gy6f2zvnNuwliMa957izfpat2k+b3UFjVADVADjaMBgnkDg/n1s9oxRz5wO8b8wOeB47xnaY/KEe5nZfFSJqoHPr1UiX5WljKzsfjZWayUiSYbi0mZaNbtcuULGs4/e+K6nNt3LwLnOsZ8zas6fAVQjnCWtOwsFyOpE8PZWdKysQQx5a2fBHW0e8mdIBuLlZVlzyc3NJRP6JT5vlOOEJajKnQFYK7DV4xTfkiFrswZc0B0+IqGcoSxIHRlJsJXRu+RwCnfLdObE6C8eXtpp3zkRnl9OMJXNshrw9fLayPWy2TLKUcYi3HKAeSTHv5RJj60Smb/3eZQykQD53hAFJl92HGwDqgBaoAaoAaogco0QDBvaDCPOOZeNhY45HCdEddt5zQ3cG622XCuc5bb2VlMbnMrS0uFecxP77jnO9Vw0e2sLCbG3Ljluz/s0WkSTbaWCIgnZmZBJhbsF8lbHl0HxGP57g+6nlY8b7vmQYz58lfOBzHlvlN+VOaNO2LFlJu4coSveCEsKqbcxJXvV2EsOq5cQ3lJp1xBuQ5hUTHlKnwlySn3oNyKKzfhK6/CKR+2WgE5oPyVh1cpMJ/40Eo59MN1P7bcuOUoCeaV3YjZkbH+qAFqgBqgBqABgnlDg3lyjPnKFzVwbpl3W2dngTNeyil3dM51ikQzqZAubYcc76t1K485nHDA945Ft2Ox5gh9MWCOcJVodhYF7wBvk0Ixlrc8ms88zTnX8L1jYY/6PcSa+/nLLcf806fPpDjlR5RTPm+sDl/BA5+AcjzwCad89piIU96Chz0DKFdgrpxyPPBpha8kOuWA8s3qIc834JSrBz01lOOBT4SvvOaFr2infI0oMPec8lceWuWB+UoBmG9bej4Wa04wZ2dCoKAGqAFqgBrIRwME84YG84hj7sWYGzDfufh2yDH3nXIF6UOTx9yEsyDO3HbMAeGII/fBfFVv4JhHYDzRLbcddSuHeXTGT7WuJhfql+0emOM3AebR5T9+d9qKKddZWMJx5YdkjsrCEn3Yc7/OwoIHPlVceQTKR+3yYsqtDCy2U95kHvQ0UL5JAOVveFD+OsJX1LJOgTmAHAuAXEO5dsvhlGu3XEP5yw+tkG3LzmvH3JsJ1LjmdMzzuSGzY2M9UgPUADXQ2BogmNcwmLcfOCVzXvqLrPh0q1N8b1qMuQ/m7/Z4seeWY17KOa9yjDlm/lzhxZkHjrl5ALRPPSzqg/lKD8xTZgCtNMYcEG4ccwPmKl+55ZgDzBFPPn88Ysp1asS5SI04rk096KlTI4adcsSTq9SIfhYWQDlSIwZpEXVqRM8pR1pEH8q3qZSIb/rhK3DKdWpEOOUKzFUWFuOUr1NAPnk4YsotKB+2WqJO+csP/iAGzM3Mn4wxb+zOg/DA9qcGqAFqIH8NEMxrFMxbt3fIqP/yjL+8/MRCOXP8fCZAx+RCSXnMfTCPOOYmnMV3zjHjpxfGMhgx5l1H+yXNMYd7nuqYW6ErJd3yjDHmyY55EGP+yZOnrAwsOq4c4St44NM45X6+ci9XuZ+vfPQ+BeQqAwvylfsZWLx85SZXuZev/M2mbTpXOdxylX1Fx5UjZ7mfq9yDcqRGNLnK4075apWBJeqUv/zQD/LSg9/LduOYW3nMGcqS/42ZnR3rlBqgBqiBxtQAwbxGwfylf1zgQ7kB9An/4zn58YvtZcO5ccwVnKsHP3V2Fh/M3+0JYstLOeWDGGMed8z7/FjzoYoxT3fMT6kHPZGzXE0ihCwsYw55kwhpp3x2i5lEKOqUm0mEdqtJhOCS25MIwSWf5jnlmEgIEwghPSLiyQHmYaccWViQr9yaRAjhKyq2POKUq/AVZGHxwlceXKGcckD5Sw/+wBjzGr1fsIOv3w7+wME2efTv/iXz8ukX35TdV1A/9asftm3x2pZgXqMd7dPjZsTA3AD6rOc/kVs3ewa86cYcc8aYh7KyJGVjyRJj/slTJ1Nm9vSgfKCZPUO5ynep1Ihqdk/jljch+4qXgcWb2dPkKjdOuR9XHoLyYGbPIKY8zSlfoYAcYP7ig8sZY16j9wt2vsXrfPNqky3bdsp/+s+/yry8NWfhgH1EXsfI76lf/bFt829bgnmNdrSlwByA/q/D/yy7N7aVvPEaxzyax9x2zFXYyjHGmCOmPIDycFaWNMccYO475WPhlJt85QclcMpbZRaysPgx5bZTnj6zJ4AcTrmZRMg45QDyAMrtfOXrRIWw+PnKo065nkRIOeUPr5KXfaccseU/yIsPLFdgvnXpOT8rSy3GmF+5ek0AMvWwnDp9puT1zQ4z/w6ziHU6lGDuch3B4S9iPVbjmFzvN3nUEb7DpX3wmWrUBb+z/PsRwbxOwdy45x/N+iH1Ios55iqcpU98MGeMuZ/TXEG5ld8cecwHijH/+MkT6kFPPYkQZvUsPbOnzlWuUyPaMeV6Vs/wzJ7GKQeYq3zl3qyeIShXWVisDCw+lOsMLPYkQphICJMI6RAWE76CuHIdWw63/MUHvpNtS8/VdB5zV4hxcSSr/Rk6nuV3dPUMBa6azkM/rhof0fJrAbTWc7sgVOi//c+/yfxPBuoUoUmV1g2+w7V9Kv1tfr6yexPBvM7BHID+wj/MT7zIjWPOGPNo/nKzXl4e8zTH/OOnTlgze+rUiLPt8JWIU65m9hylUyMinhxwjtAVDeYmX/k2mdocOOXBJEIRp9zPVx51ygHl4Zk9DZTjgU9kXpn4kA5f0U45Ysu/V1AOMLcdcz87Sw1NMOQKMa4dXDU/lwdYsQOtrAMtQv25ajoP/VSib8B5EeqvGsfg2iamPgnmtX9dVqIrgnkDgDngfMe6Q7GbYMwxZ4x5rjHmHz913AtfMU65zleOSYT0sl9mtphc5Z5T7sWVz/Ae9gxBuZ+BRT/oqZ3yLeqBz7hTHszsqTKweJMIKSi3ZvYElCM1onHLAeYqNaLnlL+knHLtlr/wwLcKzFXu8hrNY15ph2k6ziKUeYBVJZ0HP1sMeHDVdB76qfQ6qNcHUDHoqKRuCObFuLaG6h5HMG8QMP/+k80JYK5n/mSMuXHITZnmlOtJhLoOlxdjDjBH+Iqe2dNkXwnP7AmXHHA+w3PKVb7y0bvVg54hKG8y+cotKA/lK9+oJhEKZvaEU27yla9VLrl2ynVsOYB8kgflgVO+UkO5lxpRO+U6tvyF+78VA+a+U+7BOdZrZYIhV4ippJOt1mfzAKuh6nj4u/mBh6um89BPpdr+X/ePqruQFgw2Kq0Xgnl+10ct3msI5g0C5q07OhLAnHnMQ3nNc85jvuzfjvn5ylUIS4uGcp2rfL9+4LPkzJ7WJEJN260MLMhVbpzyzfKGnascISz2rJ7DwxlYTBYW45QH+co9pzwK5Q8slxce+E5B+fP3f8MYc4fsF5V20mmfzwOsarHT4jGHoaWWwRzarjcdY7CRds2Wu51gHtZ4o13zBPM6B/N/fvh12bn+cAzKIXTGmBuHPK1Mc87Lc8yXPXlcZo8x+coDKI/P7KknEJqhnHKTr9zElBunXKdGDGLKMYEQ8pXDKd8kZmZPuOTIV47wFWRhgUsenkQITnn6zJ46Xzliyj2n/AHtlAPKn3/gG9m69KyflcV3zhljXnFHXG6Hbe9Xb0DTaJ1vXudb62COByTr5UHQPNxyXOMEc4J5IrTlddPg91RHYAOlS0Rc+VvPfSzd12+lti9jzPt1THlbpPSyr1Sax3zZk8fEn9lzDOLKrUmEQjN77i45s6fOwBLM7KlzleuZPRWYj9iowFw75WYSITjlxi23UiMifEVNImQysKxUD3yamT398JWIUw4w/9P9XyswZ4x59pzRNlDn9ZpgXp17a631WbUO5rge6kHLGFzk4ZYTzHld0zGvQ8d83H9/VlZ+ujUVyE3HYxxzxphHHfM0pzxbjDnAPMhXbkF5yz55a7TOvuLHlEdn9vRjynUWFtspt2f2NGBunHK45DpfOWb21GkRTfiKcspLzuxpO+UIX/lOAOQvPKChHGC+xXLMazGPuSvE5AXTeX5PPcCMuRexdIcRV03noZ889VzreflRn3nVBx1z9+uhHu4lBPMaBfMnR09NnPnzxX+YL6c7zw0I5RBvzDFnHvNQVhbkKjeuuUse86W/7RTElvsZWEKTCFnhKzaUN+t85VObTUy5mdkTceWb1aIysHhx5f7Mnl5cuYJyP67cdsrLmNnTC19BWkQ86Pn8/d+q8JXn7/9aueV/uu+rwDHfK6F85nz4c/Bd9DzAqh46sUY/h3oB8z88O7GsfquI7Q233DVneRLME8wJ5jV7MRTxAh2sY5ry+yUxMP943spMbWkcc+YxjzrmZj3NOS8vxnzpk53h8BXllMdn9kTOcpOvHEA+zYNy5Cuf0mSysGyWKU2b5c8jIvnKvZjywClfq+LL4ZL7TvkwA+WYRGilWlRaRC9fOSYRQkw5UiPCJQeYA8oB53DJFZjf95X86f6vQo45Y8wHH8btTpxg3tidt+lr6gXMoe1adc3zdMtRDwTzxr626ZjXqGN+9fINWTD5C3mqZZr86f/Mk/3b2jNBOW7qMcececxDjrlxy41z7peHtJOePPOnDne5eKhfPvq3DgXmKle5DeXIVT7aiitXLnnglE9t2iZYMKtneTN7euEryik3qRHXqIc8dfgK4sqDmT0NlCNfuf+wpzezp8nAoh3zbwIov+8ree6+LxWYM8Z8aIHcwDnBvLE773oE8yd++0zmfszUw1CVGEzk6ZYTzHldE8xrFMzzuAkZx5wx5sYhN2WaU54txhxgDijHQ59vqbSIe/185Xpmz11+vvKpzSYLS8rMnrZTPnKjSomITCxxpzw+syfylSunPDqzJ5zyB/XMnsYpN/nK4ZLbTjmg3IC575Q3WB7zv76vWT2kBiAuwgKnNI/7AL+jtkGgnhxzQGmt6RohOGawnFdJx7y2r8lK76kE84YG89J5zDdMuymXEXfuLZdU2R+sd+C9frnU0ecvlzvC65f89X651I79vLLdLvG6Xy4d7fNKvPbWVanXu472y2dPXJdP/vaa7HinR7qOeNuP9KvXJ9bfVe/h/fZVvWob9lFLNPNK0nrGPOabZt1Wv/fdH7oFDrledJgLXi99sj11Zk+A+fTmnRKfRCjqlJt85Zt0akSTgWWEyb6yToWuID0iUiPqRbvlKle5l4VFh7Cs0BlYlFNuueWILffiymNO+f3aKTdgvu3js6HYcuWeN0i6xDw6y0pv2Px8Y3fYSe1fb2BeS9cZ3PK8YNz+njzqAN9hf2eW10k647bBu/cQzBsazPXMn9EY8w3Tbyng/PrJG2Jg3C8tGDdQHis9GFeQ3t4n0TIRzj0o70oqj/YLtnes6vXB+/A3vQreu470hUpAuQL3RT1yCVDe1heAuVoPg7oKT2kzTnm0THPONXx/90y3+q01r95UUN51SG9X5cF+WT75tHLKMbunmdlz+igvhCURyvHApxW+omb2REy5yVfupUU0TjnylSMlogflyFeO0BXElpuZPSc+tEqlR0T4CuAc8eTh1Ig6phxAHkD5N4IHPRFTboAc5Z/u+1LaVt1o2DzmeXSW7NwGr3NrlLquNzAHQNaKa14Ntxznn8e9hmBeu/cagnlDg3nEMfdizDvWBAC8YXrYNYd7riFdu+RqHbBuO+PKQTfOeSmn3HPI4aQrBz3NOe+X8/v7ZPkfNQh/9k/XPRgPHHMF4Uf6Zf1UPagIueZwxw2UJznl0W0lsrGo7CyH+mXHwh5/kND6yR3LMTfOeb/s//KaF8JipUYctVuUWz5Kx5RPa94heOAzyMKyVc3qOUVBuclVrt1yhK7o8JWIW2475cNWC0JXtFuOXOUmX7k3s6cdV57olJvUiGEoB5jP/sf1YlIk+qWXnaURsrLk0Vk2CizyPAcPCuoRzJEPvOgacq33cpzrPO41BPPBuwbz1irBvIHB/MZ57ZhHY8wvH+uXNZM1BANw4ZzvePd2eFnsrdslXjsu29/xPuuVZh3llrm3/RAWHE/btwhTMU65gfM+5ZCf3n5PAO7YDwtAfcc7t2X7otuC7wqV2Ja49ES298j2hd6+C2/LyheCukEYS9dhyynHg6Fwzg8C0EW+ePG4ylmu8pX7UI4sLCam3EB51Ck3M3vqLCw2lKuZPYdhZk8dvoJ85ZMeNllY9Myexin3s7DEnPKEmT29SYSQiSXklHvO+c6/XFBhLEkx5r23auMmWElnmkdnmfcNnN9XG7qrZju5ajqPh4fLgUzXfTCLZjXrrdLvrgR8B6qTPO41lRxfpXXDz1d2XyKYNzCY375qhbLALffymKO8cPCe/GTBuQHdoS4PfHrHiz/XLjhi0f04csScH+kXxJrbcF6tYwaUn9vbF3HLgxhzwPn5/f3yl//XWcbMnlsFLrlaRgZQ/sbIjSq2XIG5ylWu3XKEryB0BYtKizjMZGFJmdnTdsojGVgwiZCZ2RNQjjAWG8zx+qdFJ8NQbuUxv9gqhe5A7U7CFWLQkebRWdrHwteVdV6sP11/rpouOpgX2TV3rfOBgNy8n8e9hmBeu/cXgnkDg/ndHgvMFZR7D3YeC8oDn9+Rnybf9B3oakFuqe8FZK+felNObLhrOeXGMU8oj/QrYN6xqEdMHHip78/63ornuwXhK75TnuKY2875lg8uyNKnj8qMMdotn2rN7Plm0zYVvvKmF75iz+wJIMckQmZmz8mlZvZEvnIvpjxwyvUDn35aRG8SIZ0WUc/s+bw3syeysADCDZi/MW6lLHt+jxxecS0E5cYxN+W1k/0E8wa+jxCQhxYAXCGx6GAOSF38wbJC3lsqgV4D36VKgvnQXlNDfU8jmDd4h9p9wYJzL8bcds7Vay8ri3k9FDHmOktLkJ3FOOV+abKvJJVViDE3seZBNpYgtjy2zQtrUeEteK0WL9zloFUewHsiF1AeSCqxTeRCa79X4nUZy36RC+Us+0Q/2Fmi9GPLvVSJyMpyp7t2bqKuEJrYl9wAACAASURBVINONI/Ocqhv+Pz92tFquW3lqulaAHPkB8esmuXWxWDshxCbUlCdx3t53GsqGTwMRj3yN9LvRQTzBgfzu7335Npp45CbcJZ+/YCn5Zz7WVm8lIlIkeiajcVkadEPfOoHP002FpMy0ayHSi87i4Zx45SHY8wR1hLNxhJdxz4XIw98hrOzpGVj0fAdOOWR9UhWFsSYA8Jt51xB+SGRCxE4VzBuoNyUNpy3erBeotSQ3i8ILbmwv19Du1cipEa9D0BvFTm/r1/BOkAb0J64Dvg2kL7Xg3arxHs3ztaOW46OwBViCObpnQg72KGtG1dN1wKY47rL4zjz1ChCbPKA71LfQTAf2msqT724fBfBvMHBHKLpvd0n105F4DziksMtN3DuO+cWnJtc5kF2FpPbvJzsLGnZWDR8G7cceczNa98pT4gxt2PO/dcREO9KWs+YxxzOeNw5D8eYGzgP3HLjkMedcwPnF2OOueeU+9vTnHMPvo2L7kF4WW65gnPPWTcwbpXGKTfhK3DKr5+pLSiH1l0hhmDe2B2lS+c6WJ9x1XQewFsKLvN6r0iu+WC45Xnda+iY1+49i2BOMFd/E97r7RMV1hKJNTcw7pcWjMccc2+yIQPnxhmPlq55zMNOuXHME0ovNWKSU66A3EqdGHbKs+UxD5zz5KwsSU55NIzFwHjJslznXAG5ccaTS+WcA8K90Ja4U24c9EhYi+WU47xvXao9KCeY125HNViQW4u/U0tg/l9/9TdObjPyhRehbQbDLSeY8z5FMCeYh254vT335NaVPrlxrk+u/5K8XPO2X/tZv6/Kn/sEZVnLGW8/lC7Lae9zKMtYrnr7XD2l91flqT5BWdZy0tsPZdlLf7DvCXyuX66iPBEvr3jbY+Xxfrlyol+uHO+TK3hd7nLM2xely9LpfQ5lJ/5J0evXf+6X21dqE8hNp+4KMXl1luY4WBa/84VW3l3ysQqlgPtoLyNafh1ah/v82ZffCmaCHOy2ddX0UDjmpg5d3PShqFu7LVFfLsft8jnUk/3bLq/xHS7Hi8+4/B4/k989jWBOMOdFSA00jAZcIQadVR6d5WB1Xk/85g8hcDRAlFQ+8dtnFHwOJfigXSa9Pr3sY65GWxw42KaOAdDtCjT4HFxVAP1gPbToqumhAnPUs0v9DqVrjrZESE3W44ZOXdonD33jO7Ier9l/sO5T/J1kmCeYE8oaBsp4E0i+CTRSvbh0kqazyqOzHKy6dpkqHOABaBqsYzS/4xK3mxekYTCCAUG1QhSeeW5i1QHdVdNDAea4ltDuLvrEZ4dCnzheF9cbx4u2cdF3Hvcagnnt9ncEc4L5oHfEpkNmWbs3jlptO1eIQSebR2c5WPXm6koCUAfL6UVdAIxRt1kXtGMldYn6ATRn/V2X/THggYNeyfGW+qyrpocSzF3bfSiuwUrccrSbC9TncZ4E89rtXwnmBPOSHcblk73S9kOPbJ57U9a81l1Xy7aFt6R99R3p6uwtWQfRTrF97W3ZMPe6rHz1ilpWvHpZ9HJJfnj1kvwwqUst30+6KHq5IN9PuiDLXzmvlu9eOS/fvXJOvpuI5ax8q5Zf5JuJWH6Wb17Wy9cvnxEs300+I1uXXpQLnbcyHWf0uLneWFlZhrc87gSecPgGSysuzulf39fsfHwAwsEC8ijEV8s9r0Uwh75c2h51WumgLKu2XY8TWsNvEcxrF5CzaiWv/QnmBPPUTu6X/b3y2RPXh3TWz6yzcrruv2vJwNB77vAd+fqZy/LR/74oSx6/IEt+fUE+ePy8LPn1eXn/sbPyweNn5f3HfpH3H/9F3nv0Z3nvsZ/l3UfPyLuPnZHFj5ySxY+elnceOSmLHz0piyackHceOSGLJhyXRY8cl4Xjj8nCCcdkwfgOWTChU+aPa5f549vl7XFHZf74ozJv7BGZN+6IbFl6PrW98rop1PP3uEIMgCAPF2sw69blL3ScJ+LTB+s4XeJ2XZ1euNYuvxcF7ErWEb+e9z8Srpp2rUdbGy51YT6PenDJ0jKY16Grsw+YN+dJMCeYGy2UWxLMCeb+DcQWzZFVPSEgXze1W1q/uF1Xy873b8mXvwkGHnuWpcP5pRO9suzvu+TDX1+UD38dQDnA/IPHz8n7Bsof+0UBuQ/lj55RQL740VMKygHmCsgNlE84roB84YRODeXjOxSQGygHmL89DlDeJnPHHlbL5g8J57ZWs7x2hRgAyGACQZZzStsX4OMCTvhM3vCYdIwrf1zrdHzGiUz6zqRtOJdK/tZ3rcO0zwHOk47TdZurpocazHG+LtCKeoV2XOsry+cqdctdzzGPe00lms9SR9w3/4EHwZxgHrvBdR3rlb/832sKzBG+kjXUo5Yu1Cun78oPz3X7g5ATW+7E6gPns+bNqwrKQ0754xGnHFBuO+WAcuWUB1BunPKFcMoB5eM7JQTlllMOKJ83tk055YDyeWMPy5wxh2Tu2EPyS1t34nHWUt0PxbG6QgxgII/OcrDP2RUsFn+wrOr6gjOfBq9p27O6+YglH2qXPOlcENaSlxZcNV0EMHd1zfEsRF71l/Y9rs9p2G45vttl8JHHvYZgnj8wp2kl7+0Ec4J57AbX8VPglp87lC3+Om+BDtb3rXxJwzli6aO/efWXXg3lyimHQ65dchW6kuqUnxa45LZTvgguueeUI2xFL2lOuQ5dsZ1yAPkcLGMOyrp3fokdZ/S4uR6/MbtCTK2Cuev55u3qRrXo6uZncUoRylNEKDegnuVcovVnr7u2cRHAHOeBQaCpkyxltZ+FcAFbhOZE/20imMfvw7Z++TpePwRzgnkM8BDigVjtTbPjkFqvF9HWBfqcl//7jVh9nNzeI0tULDliys/JEoSuPHZWha+8h5jyx35RseQ6fOW0ep3klAPKjVO+oCynXIevGKccUD57zEEF5n95tjN2nPXaNnmelyvE1CqYo+7wsGQW4DH7Zg0ZydJOLjCW9aFPF0fenPtglBg0RCEuSx2afV01XRQwd9VoNTMI5VmnBPM4eBrtskyuG4I5wTwGeAhfAZgjprxRLhycq3lwNHrOnRtvKTAHlOsFD3meFUC5XryHPB8FlHtO+SNW+EqiSx5xylUseRBPjhAWE1OunPIxBxWUzx5zQLAs+317w7RNtD0qWXftcAFqefy9XMmxu37WBYJxvq+8Nq1qGnOZxCcrSLo+uFcKyl0eViz1fVnPKUkDrprO47dLnVvae0nn4Pqgch7nkHQ8ebnl+G4cY1pdpG3P417jcg7meJLqhNuSIboa9UIwJ5jHOl+CefgCBJgbKE92ys8oIFdQHospR9YVOOUm64qOKZ+P7Ct40FNlXUH2lSP6IU8VUx5AOWLKtVN+QOaMOSCzWlpl9phWWfY0wdzlhugKMeiwAJP4/GAtiHF1OcfoZ1zDRqoVx+sau+vi4LtAEdoa7jwGJgg3SXO1cR7YpxJYz8M1d9V0HlBrQC5LGdWnWXf5ZyeP+jO/b8q869NFgwTzcB9s2qZRSoI5wTzW+RPMwzcFA+Y684pxyq10iMYpB5T7TjlSIep0iDaUI4RFAbmC8nYF5iEot5xyBeXKKdcu+eyWVgXms1r2y7LfH421W6PctCo5T9dONwt45LVvHp2zqSvXh0DzGhyY40AJmM1aR1kf+jS/B6jOAs6o86yx3/gN1/pFPVQaK+2q6aKB+VCeh9ELSgxIs+oTg4q0ARzBPNyf2nXN18l1QzAnmMcAj2AevlgA5oFT/rMfU4785HDJsbyT6JSbVIjIT96p85N7TrnJTx6CcislIh7wnDMW4SuBUz5rTKvMbNkvAPOlTx+JtRtvcuF2S6oP184/a0edx/55grnreUczTCTVadZtLuCTFZbtYyonlAdgVSkgu8J5pe3s2rZFA3O0Geoi67UD19zl3xRbI+a1a0hNKe0QzAe+L5v6Z6nrimBOMI8BHsE8fCPp2HBTwTge7lQLJg1SixdTbrnk/qRBfn5yD8pVfnIvR/k4PXFQCMp9pxxZV3TmFRNPjvAVLIBytYzeRzB3vG5dISYrLOSxf6XAFu3kXEMFot9TyToAO2vdwPGu5Dfx2TRoxncDnNLczqy/6zrbaiW/76rpIoK567nkNYB0GTTiuiqlE4J5uD8tVVd8j2Be8mJqZIEQzMM3EoC5gXJ/Jk/fKcdMnnjQMzKTpxVTrmbyHI+ZPDuCmTxjMeXIT4485TrrShzK9ymn/K3Re2Xm6L3y0e/yiT9uNJ27dvxZYTKP/fMG83Kc46TjrsStjuorDZCTftdsy+shVLiaqFOzAJjyclrNeboMPHCeldSxq6aLCOaoR9dsOpW2pev1Ucotx/kQzMP9qblWWKbXCx1zR+etnkVFMA9fMAbMtUtuZvI8HYonL+2UA8qNS24yrxzRkwd5s3mazCsA8zCUey55yz4F5ADzt0bvkY+eJpi7XIOuEGMgcTDLvMEcrqzL8bvGd0fbx/X3KwWu6HFUe93ln4lKINlV05X8pqlDFz2Zz6aVaG+X733it884m23QJkJisv5uOdcowTzcn6a1O7cH9UQwJ5jHbmYE8+ACwc0CYG5DuY4pPymLHznpO+UmP3nwoKeXdUU55QGUz/OccjOTJ4A8GcoRS47wlX1q0UC+V2aM3qPA/MPfHY61G29s4XZLqg9XiMnaYeexfzmdftI5ltrm4ljjXCoJtTDH4xK/W406MMdTrdLl4dZKztNV00UFc7SLq05RFy7t6gLPuC7K+T2X765ED+b88R2u9yHzHSwH7lOqUUcEc4J57EZGMA9fjO0bur385NEc5UH4CsA8mMnTZF4JO+WAcrVY8eTJUI6YcjjlcSgHmM8YtVsI5uE2Kvfm6Aoxrh1cJZ/Lo3OO1ovr+Q/0d330d5LWXUAhj99NOpZqbnMZgFQy06prmxYZzF1dc5drpppuOXRGMHe7V1fzGi36dxPMCeYE874+NZlS2gRDAPOkmTwRvuI75WomTzzoafKTR6Bc5Sc/ouLIzUyegHIzk+fsFp0SEUCuoHx0BMpHaad8+qhdMmPULlny1KFYuxX9ZlOE43OFmEoA2/WzLpBRTh27hFpUeiwuoJXHQ5/l1Efe+7hqzPU4XH+vyGCOunABWlxrqI8sdenyD0eW33E5j0qvN5w/vsP13pOl/rhv/gMPgjnBPHYTo2MevtAUmKsHPBG6clJNGIRJgxb5LjkmD4rM5OnHlHsuuZUKUbnkY03mFRNTrl3ydKd8t3LKAebTm3fKkt8RzF06BFeIce3gKvlcHp1zUh25gALOo5JYb5ffzOuhz6Q6qOY2V425HpPr76FNXH/TfM5F3+azA5VwsrPkoDfHkuXfB5cBI34ny3MXLtrP49onmIf78YH0VqT3CeYE89jNmWAevqAB5gByBeUTdPgKwlYWIfOKcso7VcYVPZOnccoxk+dR/YCnBeVIhaidcuQnPyiYNAgPe5ZyyhG+ooDcg/Lpo3bKkqcOxtqtSDeWoh6LK8SYTn8wyzw656R2cIURZK1I+r5ytrmkoatkIFDOMVVzHxeduB6Pq6aLDuaoDxeoRd2XGwLlGsueRZsu55DHtU8wD/fjrtfXUHyOYE4wj3W2BPPwBd2+/oYP5YseOR445R6Uh2bytJ1yFb7SpsJXdCrEQzLHd8oPeFBuOeWh8BUdSz5j9G4fyqc17xRA+bTmHbLkqQOxdhuKG0it/aYrxLiAVqWfyaNzTmsfl5R0gOu07yu1HbOHZq2Lap57qWPN672s54v9XX/bVdO1AOZwzV1Cr8rRqusAFTCfpa0I5uH+NEvdNeq+BHOCeewmQzAP30jaN9xQMI7wlSDrSqcKX0Fu8rhT7qVCjDjlgHK443OUQw6nPJjJc6YP5TqWHA94JkH51ObtCsw/eKo11m6NehPLct6uEANwwuQx+PxgLQDaLOeWZV/XfNsux+TiSpbreGY558Hcl2D+q9TBWNZ2cHmYFvU/0D88SK/o0k5Z3HKcK8E83J9mbf9G3J9gTjCPdf4E8/CN5Oh6DeY2lC8YbzKvdMj8ce0yf5zOT47ZPOfBKfczr2DSoEg8uTeTp44n3y8KylV+cp2jXEH5qLBTDrccTjnAfGrTNiGYh9uo3Jt3JWBe6y5utI5cnEiXuO+s+aFr9aFPu35dgM/+fJbXrpquBcfc1IOLVqG7tDSfrnXmon+Cudu92rR9I5YEc4I5wXyArCwAcxvK8aAnJgzSTvlR9drkJ1dQHnLKD6rwFXvSoESn3Mu64jvlzbtUCIsJX5natF1B+ZtN22Rq8zZ5/8n9sXZrxBtY1nN27ZABWvUG5i7AUE6IgN0mLm5n1lAB+/cG4zX+NYCOsKAO7QUuLHRCMM/PMUebuugIbYC2SdKESxthwJgG+km/Yba5XGd53GtcztHo1hw7y6EZVBDMCeaxGxcd8/DFeHT9dQXm2iU3TrmeNMg45SY/eeCUa5c8PpOnlaN89F5vNk8rfMV3yuGQa5fcOOWAcr1sJZg7XrcE80DbrjG2WaaPd4lldwmXqQZA4Dg++/JbmfT6dAXbLg+wGtApp3Q9B1dNp0FrluMo57yi+2T5fntfF9BMcs0Hu74I5sE9x25Pvk6vF4K5Ywdfz6IimIcvGIC5gnLPKYdbjowr88cfVRMG+eEryilH1hWEr8ApR9YVk5+8VWYhpnz0PpWBBTN5zmzZKzNSnfIdMm3UTok65VNGbpU3m7bKe0/uiw2o6lmTeZ2ba6cMuMjDxcrrPPL6HhdwLtfRdgF/xPHndW5ZvwfHCxCH6501/CYKny7rWY/X7O+q6VoDc9fzjD6v4PLMg6tbjjYimIf7U6Nblun1QjAnmMc6QoJ5+ILRYN6hUiIaKAeYKyD3Y8oPe9lX0p3ymWY2Tzue3JvJEyEsOiVi2CVHTLntlE9p2iJTRm4hmDtet66de72CuctDoEkuZFIni4fvsgJqFKKSvjfPbYDxd5d8LMh9nfVY897f9bxcNV1rYI76cXHN7fArl8Ei2nmgB0lLtR3BPNyflqorvqfrimDu2MHXs4Bav7gtmAVz0+ybMWiv1/PGueKcf5zUHTvnU3u79QOetlM+Vk8cNNdLiWjyk5dyyme27JN0pzxIhahCV+yY8qZtCsYB5X8euVmmNG2W/3iRecxdtOgKMeic69ExRx26PFhXDkBnhd1KXMmsWoAOnnlu4pDDuA33Wc/B7O+q6VoEc4QX2XVW7mvUEerLZZZPXB+mrl1KgjnBPKtuCOYE89hN5/TOXgWpANUL7b2x97OKrOj7dx3rlU//8bo6590f3Yqdb8/Nu7LgER2+EjjlyE9ucpQH+cmDmHLEkpt48v1ioBxg/hZcct8p1w95mvzkJp48ySkHlOtlk2z74kzsOItez0U4PleIqWcwdwGHgWY+dAGockNkKtERHFMX17VcAKxkP9fzctV0LYI56sglFMXo1SVEqZxBaKm2c7m+8jABKtF5qfPhe9UfaBDMCeaJgLfypW4fzjvX98j183cT96vli7S7654cWdUjX/9OQzkGIjev3Es8z61LL1ozeR5RYSvzVCy5nslTOeVqJk8TU75fxZIH+ckB5EFMOcJWVPiKNWlQ2ClHLHnEKR+5Wd4YsUlmPb5Nbt2o/wFTNbTlCjH1DOauf++XylDh4kxW+6FPPMRZCThX+7OuenfVdK2CuateXQC5UrccberyuwTz6sOv6/U2GJ8jmBPME0H02tm7PpgDWBthObntTmJdmAvxu9dPyzwrFeJczOLpzeQZdso9KG/ZF3LKlUs+ercG8lFwyiPhK16OcpN5RcWTe+Erxil/Y+RGad/WVfI4zfGyjN/cXSGmnsEcOnF5CLRU3G1WZ7KaD30C+LOG1VQbwpO+3/V6ddV0rYI56sll4JdVk2ijLBmI0tqPYB6/D6fVFbfruiKYE8xTIe/m5XuyfrqOva5nMMe/Az/vLc+B3vLheTFOuZ7J86DKwIKwFeQnx6RBauIgfyZP45QHM3nOGLVLpUIMha+o/OR40FNnXcEDnn5MuXLKN8qyZ1vl1IErqe3Fm9rAHYArxNQ7mOPv+iRYLLUNsJukOZcHSisNF0g6DmwDlLsAWanzHug9uJ0YaAy0X/T9tHMYaLurpmsZzPFvDZ5JiNZhnut5uNZoO4L5wPflgTTeaO8TzAnmiZ2rfSF0dfZKx5oewUOh9bQgROfyiewhOrdv3JXjO6/LwdWXveWSHFxtL11yYLW1/NglB368KK2h5YK0/hgs+388L/tXhZd9q84JlgNrz8u5zusDtpPdZnyd3Bm4Qgw6/Lw66qK2jQvoIKwgej5ZY4Cr9dBn3lCO44QG4NYCtqAlLEl14KKzaD2Wu+7yW9BzLYM56sYFeLOAO+q13DYotZ/LceZxr8F3ZDlfe99S58P3kvuWPOuFYE4wz+Xmk6co+V3Vv/AbtY5dIQadVh6dZZHr3SU8ICmcJatDXY2HPuGoVjohEEAcxwY3Pwm+S7Wli85KfV+p91x+C3qudTCvpmue57VOMGd/Vur6TXqPYE4wJ5hTAw2jAVeIaQQwd3moLhrO4hISgzZJ6pwq2eYyyDCOIeLtKz0mF525nq/Lb9UDmKO+XHLlm3YuVVba/nZbEswJ5rYeynlNMCeU5d4pliM87sOb1VBowBVi0Inn6aINxbmX85suf3/b2VmyhrHkkfUiel4uAwzTvnkBmYvOoudR7rrLb+F8a90xN/Xjkoe/FJTn/Q8OwZx9ndFquSXBnGBOMKcGGkYDrhBjwK3cG2ut7ufieNsPbmYNY0kKham07rIODtC2ecOYi85cz9vlt+oJzF00WwrMs4YtDdRuBHOC+UAaib5PMCeUNQyURcXP9ca7YbpCTKOAOa6JrA+BGqjFw5algCfpPdttz+t6zDo4qMY/IS46cz1/l99CW9SLY456c8mCk6RHo2XXtkj6HMG88fqZJB1k2UYwJ5gTzKmBhtGAK8SgE68GwGW5WQ/Wvlnjs/GQJY4ta7xvNSDIpX3xmbzr1uU4XI/B5bfqDcxd68CGcwxI83bL0aYEc4J51mubYE4oy71TyipC7s8b12BpoJIOvFHA3CVGG5/JOkkR2iLvds86OACY5X0M+D4Xnbkeh8tv1RuYo+5cno+wwTyPfxCS2pBgzv4tSRelthHMCeYDdkzXz9+Vc4d6c19KCTPtvQudt6Vjc7e0b7wRLBtuSPuG6/5ydMN1ObrhWmg5suGaHFl/NbS0rb8i4eWytK3Xy+F1l+Xwukuh5dC6Lgktay/K4Q1d8ssR5hhPa6+ibXeFGHTgjQLmaLOskINJhbKkJ6zGQ5847qwQVK3jcNGZ67Xi8lv1COau9YC6gFtejbAqF03mda/Jeg3jd83iqkV+Lp9BCMGcYJ4K5peO98qa125ItWb9/PxfrsuRFT2pv29f5Cd23JSvnj0n7zxyUi2LJpyQRY+ckIUTjqllwfhOWTihU+aP75AF4zvk7XFHZf74ozJv7BF5e9wRmTv2sMwb2yZzxhySuWMPqdk654w5KLNTZ+zcI2+N3iMzRu1Wy/RRuwTLtOYdapnatF2mNgczdS7+zV5p29hV1rnY58XX+dzIyq3HSjrvRgLzrA/UZQ1/qcZDn9BAVjAHiJSrnXL3A+A989xEH3IM7AxUlvv90f1cNZ2HQzzQOSW9Hz3+PNez/mtjji+Pukg7DxdN5nGvIZgPbt+S1v4u2wnmBPPEjglQDnCuFpTb37t14c3EYzCCPrTyuix+9FQA5Y8YKD/uQznAHEA+f3y7zB93VIE5gHwelrFtCswVkNtQPuaAAvNZLfsFy8zR+2Tm6L0KyONQvlOmNWPZoYBcQ/k2ebNpq0xp2qKWP4/cLDu++rnkuZhzYjk0N01XiEEHnkdnWUvtnvUhUAM55ZRFcifzPBZ8F3K7l1MH0X1cteGq6TxgNHoO5ay7nmc5n3MJw6qmW45jJpgPzb2+HL0UdR+COcE8BpI9N+7Jt7/XTvnn/3xd2n7oEYSz5ClihMasea3bB/9j6+8kfj9CVxY/ejqA8gkn5J0MTjmgfN64ajnl22TKyADKAeZvjNgkpw5eTTyXPOuP3+V2s3eFGABHo4F5Vhe8HCjDPtV46NNcDwipKfc4zH52ukfzPS4lstJkCecxv29Kl9/EZ1w1XY9gjvrICsLQjGvdl/O5rMcDPeRxr6nEMYemhmrBdVROvdbzPgRzgnnsIgCIG0cbAF3NC2DF8xrOMRBI+p0NCy7JYhO+4jnliyZYTvmEsFOu3fLAKZ839rByy+eUcspbjFOu3fJ4+MrATrmB8jdGbpSvphxOPJek8+M2N8B2rTdXiDHwVKQyD7AqVY8u6Q/LqZ9qgpBL+wKmK3HN8VkX+IrWVam2KPWeyznjt/PQT/QcylkvdS55vVculFZzkGjOxUUbQw3m5bRjtfbJ49xN3ddqSTAnmMcgcsMMDctwtKst7NM7e/1BAMJnor+37F9/9mPKjVMOME+NKR+nY8qNUz43Mab8gApd8cNXWvZ54St7dUz56N0qnjw5phzhK8lO+Z9HbpLXR2yQ6RO2xM4jel5cH1wgN/XtCjHV6oQq+d48wMrUS1qZV35oc57VetjSHD8g2fxWlhLhJ1lT5eG3PvvyW8maNz3tuMw5ZC1dNZ2HftLOpdT2rOfnuj/OLy0cCzqs5gDRPmaCefBQaSldmPcI5n1CMCeYxyDShJjs+vBW7D37hpPHa4TNlHLn9cOeOqbcOOV4yHOB75R3yPxxJq7cgnI7rnzMQfWw52zElI9pDaBcOeWA8qxOuY4rh0tuO+VvjNigwPz14eul52a+oT951DW/w/1vf9NpFKnMA6wG0kTWh0AHqp/BOOZKBhN4aHPl6nWpDjr+RQCMY7+8gNzU2UBtkfY+wbz0IB9thjoyy2CHShDMCeZp127adoI5wTwG3wbMWz+/HXsvTUiVbC8F5si8UsopV1A+vl095Kke9vRiykNOiibVUwAADrtJREFU+VhkXwGUJznlgPISTnmzyb7iOeVNW+TNpi0KyA2UG6ccYP7a8PXy+giCeSV6qOZnXSHGwFORysGAXLjCaa6jS11kdaVdtOCSyzztXBDmkjeAp/2Wy7niM66azkM/aedSarvredbq5wjmBPOs2iWYE8xj8F00MPedci8los6+4jnl45GBxY4ptzKwlOuUe2kRg5SI4ZjyqU0m+4qXgSXklG+SN0Zs9J1yuOWvDV9Hx7yg15UrxJQCjaF6Lw+wKqfDQBxuHueIVHbl/F6l++Q9mHA996wDGtfzdtV0HvpxqRvX86zVzxHMCeZZtUswLyhAZG3IPPcvEpgvnHBckmLKjVMOKA855WOjecpdnPKdkZSIOqYcqRHhkk/xsq8kOeWTh68jmBf4mnKFGBcAqfZn8gCrcu4beT0EOlgxvTgnFxjKs70QApR1QFNOWyTt46rpPPTjUmdJ51DP21y0mEecNb7DpX2G+jN5nHut64lgXmCIGCpxFQvM9QRCKq7cmzwoW0w5wld0XPlMxJSrBz1NTLmeQKi0U75Npphc5Z5TDiB/A0uCUz55+FqZPOwnOuYFva5cIWaoO6uk388DrMq9x1QSt41jr/ZDn9HzgGuO30yqt2pug0tuYpgxEMnyW9FzKHfdVdN56CfL+Zl9yz2vetmPYE7HPKuWCeYFBYisDZnn/oUCcxW+csyf0TPVKbezr8Riylv15EE+lCfFlOsJhDBx0DQ1o6eJKd+qJhFKc8oRuoKYcuOUA8onDweYxzPM5NlG/K7SD3yl1Y8rxBioKFKZB1il1VN0e6UPgQ7msZpjByBnDSeppH0xeMGAwPw+yizfZ38uy2tXTefRJlnOz+yb5dzqYV+COcE8q44J5gTzUEcCARUKzL3sKyquHNlXxrcHs3riQc/U7CsRp3y0yb6yV2ZEYsqnR2b01DHltlNuHvbcJMott53yETqm3DjlAPNXh60hmBf0unKFGAMVRSrzAKtyO4xK47YH46HPpHMZLDhPa4ssU8QnHX8521w1nXbM5fym2cflejCfbZSSYE4wz6p1gnlBASJrQ+a5f5HAHEAehvJoTLk3gRAe9Aw55QhfKdMpH6Vjyo1TDjBH+EqpmHLjlONBTyzGKQeUE8zd3Ow8NZz2Xa4Q4wIg1f5MHmCVVk9J27PGTJvzH6yHPpOOGdswKKg0FMecS7REPGypQUcWvaUd/0Dbs/yGffx56Mf+vnJfD3Q+9fY+wZxgnlXTBHOCeaEdc52BpQKn3A5fSXTKrQwsKvuKycCyRaaMjDvlOle5Dl8JoFzHlRson/TwajrmBb2uXCGmXOgYzP3yAKssHYbrQ6CD+dBnqfNBfeUVd47BBrRU6vfMe+UOaMz+WUtXTeehHxe9Zz2/Wt+fYE4wz6phgnlBASJrQ+a5f5Ec89SY8rElnPIxpZ3yGaN2yTSEr/hO+Q41m6dJi2g75QhdCbKvbJS4U75WEMZiQ/mkYQTzPPWY53e5QowLgFT7M3mAVda6zeo8I8Y7629Ue3/EywOss8afwx1HjvRSDnnasWNwgvbCd6QtaZ8daDuOB9+ddSl3YFHq9100Xur76vE9tEvWeoJGKq0LfEfW3y3C/nmce6V1N9SfJ5gTzGM3gEKBeTSmfFxannITU94qM1v2W9lXTAaW3WKyr0wfpR/0nNa8Q6dFtJxyQHmqU+496BlyyocjplzHlcMpx/LKwz/SMed1Fbuuhvpmz9+Ph1jhXwCAehrUAqjzAFjWfbzuWSesE2ogWQMEcwJEDCCKBOY6T/lR9ZDnPB/KD8scE1M+xuQpT4sp36Me9gSUG6ccYD61eYcAzN9s2iZTm3X4SsgpHxE86KnSIiZCedwpf+WhVQRzXlOxa4odUHIHxHphvVAD1AA1ENYAwZwQEYOIooE5Mq+EofyQAvPZUSiPOOUq+8poyylX2VeSY8qNU47Jg0z4CoBcx5RvUCkRQ065csnjTvkrD6+SiQ+tpGPO6yp2XbHzCXc+rA/WBzVADVADcQ0QzAkQMYAoEpgnQvnYQzK7xXLKIzHlM0cjT3myUw6X3HfKVQhLOPtKGMrLjyk3TjmgfOLDBHPebOM3W9YJ64QaoAaoAWpgIA0QzAnmxQbzUPhKslM+K+KUvxXNvmLFlKu48qbtKoQFLnlmp1w96JnulAPMX35oBR1zXlex62qgmzHfZ4dNDVAD1AA1QDAnQMQAokiO+Vw/+8ohmQOn3A5fSXHKAebRmHLjlCPzih1TjtCVKU1Ii5gUU75B5SjXecqTs6/gQU8sxil/+cEVBHNeU7Frip0tO1tqgBqgBqiBcjRAMCdExCCiaGA+Z0xppxyhK2+pRYevzBhl4sqD7CvGKbdTIiL7SjymfKO8PiISUz7cQHmaU67jyuGUY3npwR/omPO6il1X5dyQuQ87bmqAGqAGGlsDBHMCRAwgigTmgPK5Ead8dtQpb9Ex5dop3y0Ac+Qp12kRdUx53Ck3EwhFnHIF5Vmd8lVinHJA+csPEczZsTR2x8L2Z/tTA9QANeCmAYI5wbzYYD42cMtnteiUiIkx5V5cOVIi2nnKlVPevF2yOeXrVQjL5GEJTvkwnadcZ1/RTvlEyykHmL/04Pd0zHldxa4rdlJunRTrjfVGDVADjaQBgjkBIgYQxjHfMKM79l7eF8el473yyd9eU8u5Q72x33v7ER1XDihXTnnLfgGYI3RlZoJTPr0cpzwxphzhK0lOOcJXflITB2FGz3BM+SoVumLCV15+8Ad58YHl8tKDy+Vu773YueRdd/w+dlbUADVADVAD1EB9aYBgTjCPAeSuD28pUP78X65Lz43qAmbr57d9ME/6rS9ePC6zI055Uky5ccpD4SuJTjke9NS5ynWecjum3HPKS8aU40HPdKf8xQeXy/zfborVKW+c9XXjZHuyPakBaoAaoAaqoQGCOcE8BpHXz9/1YXnNazeqBufH1t/xf2frgpux44DgD/54KeSUA8ptpxzx5Ophz0xO+WYJQ3kWpxwze8Zjyo1T/uID38nWr44nnks1LmB+JzsGaoAaoAaoAWqgfjRAMCeYJ0Kkcc0RZgLnfMNb3dL6xe1clq0Lb8qKF7p9KP/8n0s789++flxmtuyTmaOxIANLkH0lnIHFe9Az6pQ3mQc9k5xyQHkJp3zYapnkp0Q0TrnOVa7jyXVMOZxyQPkHz25LrE/eNOvnpsm2ZFtSA9QANUANVEsDBHOCeSpI2mEmJg4873LF892COPNSAu/tuSfLpwLOg+wr6U75dpnajAmErMmDVEw5oNxzykea8BUHp/yhFYKHPU32Fe2Uf6+gfNnEnXKr+07Jcyl1nnyPN3pqgBqgBqgBaqCxNUAwJ5iXBEmEtcA9xwOheS54sBShLFluQMd2XZVVc0/KJ88ekY//2CbL/nhYLUv/eEjU8sxB+UgtB+SjZw7Ih39oVcuSP+wXtfx+n3yglr3ywe/3yvu/3yPvP71b3lPLLnnv6V3y7tM75d3fYdkhi9WyXd55apu3bJVFT22VRU9uUcvCJzfLO09vlS+n7ZO2recynUuW8+a+jX2TZvuz/akBaoAaaBwNEMwJ5gRKaoAaoAaoAWqAGqAGqIECaIBgXoBG4Ei4cUbCbGu2NTVADVAD1AA1QA2kaYBgTjDnCJkaoAaoAWqAGqAGqAFqoAAaIJgXoBHSRk3czhE1NUANUAPUADVADVADjaMBgjnBnCNkaoAaoAaoAWqAGqAGqIECaIBgXoBG4Ei4cUbCbGu2NTVADVAD1AA1QA2kaYBgTjDnCJkaoAaoAWqAGqAGqAFqoAAaIJgXoBHSRk3czhE1NUANUAPUADVADVADjaMBgjnBnCNkaoAaoAaoAWqAGqAGqIECaIBgXoBG4Ei4cUbCbGu2NTVADVAD1AA1QA2kaYBgTjDnCJkaoAaoAWqAGqAGqAFqoAAaIJgXoBHSRk3czhE1NUANUAPUADVADVADjaMBgjnBnCNkaoAaoAaoAWqAGqAGqIECaIBgXoBG4Ei4cUbCbGu2NTVADVAD1AA1QA2kaYBgTjDnCJkaoAaoAWqAGqAGqAFqoAAaIJgXoBHSRk3czhE1NUANUAPUADVADVADjaMBgjnBnCNkaoAaoAaoAWqAGqAGqIECaIBgXoBG4Ei4cUbCbGu2NTVADVAD1AA1QA2kaYBgTjDnCJkaoAaoAWqAGqAGqAFqoAAaIJgXoBHSRk3czhE1NUANUAPUADVADVADjaMBgjnBnCNkaoAaoAaoAWqAGqAGqIECaIBgXoBG4Ei4cUbCbGu2NTVADVAD1AA1QA2kaYBgTjDnCJkaoAaoAWqAGqAGqAFqoAAaIJgXoBHSRk3czhE1NUANUAPUADVADVADjaMBgjnBnCNkaoAaoAaoAWqAGqAGqIECaIBgXoBG4Ei4cUbCbGu2NTVADVAD1AA1QA2kaYBgTjDnCJkaoAaoAWqAGqAGqAFqoAAaIJgXoBHSRk3czhE1NUANUAPUADVADVADjaMBgjnBnCNkaoAaoAaoAWqAGqAGqIECaIBgXoBG4Ei4cUbCbGu2NTVADVAD1AA1QA2kaYBgTjDnCJkaoAaoAWqAGqAGqAFqoAAaIJgXoBHSRk3czhE1NUANUAPUADVADVADjaMBgjnBnCNkaoAaoAaoAWqAGqAGqIECaIBgXoBG4Ei4cUbCbGu2NTVADVAD1AA1QA2kaYBgTjDnCJkaoAaoAWqAGqAGqAFqoAAaIJgXoBHSRk3czhE1NUANUAPUADVADVADjaMBgjnBnCNkaoAaoAaoAWqAGqAGqIECaIBgXoBG4Ei4cUbCbGu2NTVADVAD1AA1QA2kaYBgTjDnCJkaoAaoAWqAGqAGqAFqoAAaIJgXoBHSRk3czhE1NUANUAPUADVADVADjaMBgjnBnCNkaoAaoAaoAWqAGqAGqIECaIBgXoBG4Ei4cUbCbGu2NTVADVAD1AA1QA2kaYBgTjDnCJkaoAaoAWqAGqAGqAFqoAAaIJgXoBHSRk3czhE1NUANUAPUADVADVADjaOB/w/MfeuNbZk6zwAAAABJRU5ErkJggg==" 7 | } 8 | }, 9 | "cell_type": "markdown", 10 | "metadata": {}, 11 | "source": [ 12 | "![image.png](attachment:image.png)" 13 | ] 14 | }, 15 | { 16 | "cell_type": "code", 17 | "execution_count": 52, 18 | "metadata": {}, 19 | "outputs": [ 20 | { 21 | "name": "stdout", 22 | "output_type": "stream", 23 | "text": [ 24 | "Requirement already satisfied: evalml in c:\\users\\win10\\anaconda3\\lib\\site-packages (0.22.0)\n", 25 | "Requirement already satisfied: statsmodels>=0.12.2 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from evalml) (0.12.2)\n", 26 | "Requirement already satisfied: nlp-primitives>=1.1.0 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from evalml) (1.1.0)\n", 27 | "Requirement already satisfied: ipywidgets>=7.5 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from evalml) (7.5.1)\n", 28 | "Requirement already satisfied: lightgbm<3.1.0,>=2.3.1 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from evalml) (3.0.0)\n", 29 | "Requirement already satisfied: scipy>=1.2.1 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from evalml) (1.5.2)\n", 30 | "Requirement already satisfied: imbalanced-learn>=0.7.0 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from evalml) (0.8.0)\n", 31 | "Requirement already satisfied: catboost>=0.20 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from evalml) (0.25.1)\n", 32 | "Requirement already satisfied: scikit-optimize>=0.8.1 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from evalml) (0.8.1)\n", 33 | "Requirement already satisfied: click>=7.0.0 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from evalml) (7.1.2)\n", 34 | "Requirement already satisfied: pyzmq<22.0.0 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from evalml) (19.0.2)\n", 35 | "Requirement already satisfied: colorama in c:\\users\\win10\\anaconda3\\lib\\site-packages (from evalml) (0.4.4)\n", 36 | "Requirement already satisfied: kaleido>=0.1.0 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from evalml) (0.2.1)\n", 37 | "Requirement already satisfied: category-encoders>=2.0.0 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from evalml) (2.2.2)\n", 38 | "Requirement already satisfied: sktime>=0.5.3; python_version < \"3.9\" in c:\\users\\win10\\anaconda3\\lib\\site-packages (from evalml) (0.5.3)\n", 39 | "Requirement already satisfied: numpy>=1.19.1 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from evalml) (1.20.2)\n", 40 | "Requirement already satisfied: matplotlib>=3.3.3 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from evalml) (3.4.1)\n", 41 | "Requirement already satisfied: graphviz>=0.13 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from evalml) (0.16)\n", 42 | "Requirement already satisfied: requirements-parser>=0.2.0 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from evalml) (0.2.0)\n", 43 | "Requirement already satisfied: woodwork==0.0.11 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from evalml) (0.0.11)\n", 44 | "Requirement already satisfied: shap>=0.35.0 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from evalml) (0.39.0)\n", 45 | "Requirement already satisfied: texttable>=1.6.2 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from evalml) (1.6.3)\n", 46 | "Requirement already satisfied: psutil>=5.6.3 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from evalml) (5.7.2)\n", 47 | "Requirement already satisfied: networkx>=2.5 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from evalml) (2.5)\n", 48 | "Requirement already satisfied: plotly>=4.14.0 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from evalml) (4.14.3)\n", 49 | "Requirement already satisfied: seaborn>=0.11.1 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from evalml) (0.11.1)\n", 50 | "Requirement already satisfied: featuretools>=0.20.0 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from evalml) (0.23.3)\n", 51 | "Requirement already satisfied: cloudpickle>=0.2.2 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from evalml) (1.6.0)\n", 52 | "Requirement already satisfied: scikit-learn>=0.23.1 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from evalml) (0.23.2)\n", 53 | "Requirement already satisfied: xgboost<1.3.0,>=0.82 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from evalml) (1.2.1)\n", 54 | "Requirement already satisfied: pandas>=1.1.0 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from evalml) (1.1.3)\n", 55 | "Requirement already satisfied: patsy>=0.5 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from statsmodels>=0.12.2->evalml) (0.5.1)\n", 56 | "Requirement already satisfied: nltk>=3.4.5 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from nlp-primitives>=1.1.0->evalml) (3.5)\n", 57 | "Requirement already satisfied: widgetsnbextension~=3.5.0 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from ipywidgets>=7.5->evalml) (3.5.1)\n", 58 | "Requirement already satisfied: ipython>=4.0.0; python_version >= \"3.3\" in c:\\users\\win10\\anaconda3\\lib\\site-packages (from ipywidgets>=7.5->evalml) (7.19.0)\n", 59 | "Requirement already satisfied: ipykernel>=4.5.1 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from ipywidgets>=7.5->evalml) (5.3.4)\n", 60 | "Requirement already satisfied: traitlets>=4.3.1 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from ipywidgets>=7.5->evalml) (5.0.5)\n", 61 | "Requirement already satisfied: nbformat>=4.2.0 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from ipywidgets>=7.5->evalml) (5.0.8)\n", 62 | "Requirement already satisfied: joblib>=0.11 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from imbalanced-learn>=0.7.0->evalml) (0.17.0)\n", 63 | "Requirement already satisfied: six in c:\\users\\win10\\anaconda3\\lib\\site-packages (from catboost>=0.20->evalml) (1.15.0)\n", 64 | "Requirement already satisfied: pyaml>=16.9 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from scikit-optimize>=0.8.1->evalml) (20.4.0)\n", 65 | "Requirement already satisfied: numba>=0.50 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from sktime>=0.5.3; python_version < \"3.9\"->evalml) (0.51.2)\n", 66 | "Requirement already satisfied: wheel in c:\\users\\win10\\anaconda3\\lib\\site-packages (from sktime>=0.5.3; python_version < \"3.9\"->evalml) (0.36.2)\n", 67 | "Requirement already satisfied: pillow>=6.2.0 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from matplotlib>=3.3.3->evalml) (8.0.1)\n", 68 | "Requirement already satisfied: pyparsing>=2.2.1 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from matplotlib>=3.3.3->evalml) (2.4.7)\n", 69 | "Requirement already satisfied: kiwisolver>=1.0.1 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from matplotlib>=3.3.3->evalml) (1.3.0)\n", 70 | "Requirement already satisfied: python-dateutil>=2.7 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from matplotlib>=3.3.3->evalml) (2.8.1)\n", 71 | "Requirement already satisfied: cycler>=0.10 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from matplotlib>=3.3.3->evalml) (0.10.0)\n", 72 | "Requirement already satisfied: tqdm>4.25.0 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from shap>=0.35.0->evalml) (4.50.2)\n", 73 | "Requirement already satisfied: slicer==0.0.7 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from shap>=0.35.0->evalml) (0.0.7)\n", 74 | "Requirement already satisfied: decorator>=4.3.0 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from networkx>=2.5->evalml) (4.4.2)\n", 75 | "Requirement already satisfied: retrying>=1.3.3 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from plotly>=4.14.0->evalml) (1.3.3)\n", 76 | "Requirement already satisfied: dask[dataframe]>=2.12.0 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from featuretools>=0.20.0->evalml) (2.30.0)\n", 77 | "Requirement already satisfied: distributed>=2.12.0 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from featuretools>=0.20.0->evalml) (2.30.1)\n", 78 | "Requirement already satisfied: pyyaml>=3.12 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from featuretools>=0.20.0->evalml) (5.3.1)\n", 79 | "Requirement already satisfied: threadpoolctl>=2.0.0 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from scikit-learn>=0.23.1->evalml) (2.1.0)\n", 80 | "Requirement already satisfied: pytz>=2017.2 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from pandas>=1.1.0->evalml) (2020.1)\n", 81 | "Requirement already satisfied: regex in c:\\users\\win10\\anaconda3\\lib\\site-packages (from nltk>=3.4.5->nlp-primitives>=1.1.0->evalml) (2020.10.15)\n", 82 | "Requirement already satisfied: notebook>=4.4.1 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (6.1.4)\n", 83 | "Requirement already satisfied: setuptools>=18.5 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from ipython>=4.0.0; python_version >= \"3.3\"->ipywidgets>=7.5->evalml) (54.2.0)\n", 84 | "Requirement already satisfied: pickleshare in c:\\users\\win10\\anaconda3\\lib\\site-packages (from ipython>=4.0.0; python_version >= \"3.3\"->ipywidgets>=7.5->evalml) (0.7.5)\n", 85 | "Requirement already satisfied: backcall in c:\\users\\win10\\anaconda3\\lib\\site-packages (from ipython>=4.0.0; python_version >= \"3.3\"->ipywidgets>=7.5->evalml) (0.2.0)\n", 86 | "Requirement already satisfied: pygments in c:\\users\\win10\\anaconda3\\lib\\site-packages (from ipython>=4.0.0; python_version >= \"3.3\"->ipywidgets>=7.5->evalml) (2.7.2)\n", 87 | "Requirement already satisfied: jedi>=0.10 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from ipython>=4.0.0; python_version >= \"3.3\"->ipywidgets>=7.5->evalml) (0.17.1)\n", 88 | "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from ipython>=4.0.0; python_version >= \"3.3\"->ipywidgets>=7.5->evalml) (3.0.8)\n", 89 | "Requirement already satisfied: jupyter-client in c:\\users\\win10\\anaconda3\\lib\\site-packages (from ipykernel>=4.5.1->ipywidgets>=7.5->evalml) (6.1.7)\n", 90 | "Requirement already satisfied: tornado>=4.2 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from ipykernel>=4.5.1->ipywidgets>=7.5->evalml) (6.0.4)\n", 91 | "Requirement already satisfied: ipython-genutils in c:\\users\\win10\\anaconda3\\lib\\site-packages (from traitlets>=4.3.1->ipywidgets>=7.5->evalml) (0.2.0)\n", 92 | "Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from nbformat>=4.2.0->ipywidgets>=7.5->evalml) (3.2.0)\n", 93 | "Requirement already satisfied: jupyter-core in c:\\users\\win10\\anaconda3\\lib\\site-packages (from nbformat>=4.2.0->ipywidgets>=7.5->evalml) (4.6.3)\n", 94 | "Requirement already satisfied: llvmlite<0.35,>=0.34.0.dev0 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from numba>=0.50->sktime>=0.5.3; python_version < \"3.9\"->evalml) (0.34.0)\n", 95 | "Requirement already satisfied: partd>=0.3.10; extra == \"dataframe\" in c:\\users\\win10\\anaconda3\\lib\\site-packages (from dask[dataframe]>=2.12.0->featuretools>=0.20.0->evalml) (1.1.0)\n", 96 | "Requirement already satisfied: toolz>=0.8.2; extra == \"dataframe\" in c:\\users\\win10\\anaconda3\\lib\\site-packages (from dask[dataframe]>=2.12.0->featuretools>=0.20.0->evalml) (0.11.1)\n", 97 | "Requirement already satisfied: fsspec>=0.6.0; extra == \"dataframe\" in c:\\users\\win10\\anaconda3\\lib\\site-packages (from dask[dataframe]>=2.12.0->featuretools>=0.20.0->evalml) (0.8.3)\n", 98 | "Requirement already satisfied: tblib>=1.6.0 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from distributed>=2.12.0->featuretools>=0.20.0->evalml) (1.7.0)\n", 99 | "Requirement already satisfied: msgpack>=0.6.0 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from distributed>=2.12.0->featuretools>=0.20.0->evalml) (1.0.0)\n", 100 | "Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from distributed>=2.12.0->featuretools>=0.20.0->evalml) (2.2.2)\n", 101 | "Requirement already satisfied: zict>=0.1.3 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from distributed>=2.12.0->featuretools>=0.20.0->evalml) (2.0.0)\n", 102 | "Requirement already satisfied: argon2-cffi in c:\\users\\win10\\anaconda3\\lib\\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (20.1.0)\n", 103 | "Requirement already satisfied: Send2Trash in c:\\users\\win10\\anaconda3\\lib\\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (1.5.0)\n", 104 | "Requirement already satisfied: terminado>=0.8.3 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (0.9.1)\n", 105 | "Requirement already satisfied: jinja2 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (2.11.2)\n", 106 | "Requirement already satisfied: nbconvert in c:\\users\\win10\\anaconda3\\lib\\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (6.0.7)\n", 107 | "Requirement already satisfied: prometheus-client in c:\\users\\win10\\anaconda3\\lib\\site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (0.8.0)\n", 108 | "Requirement already satisfied: parso<0.8.0,>=0.7.0 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from jedi>=0.10->ipython>=4.0.0; python_version >= \"3.3\"->ipywidgets>=7.5->evalml) (0.7.0)\n", 109 | "Requirement already satisfied: wcwidth in c:\\users\\win10\\anaconda3\\lib\\site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=4.0.0; python_version >= \"3.3\"->ipywidgets>=7.5->evalml) (0.2.5)\n", 110 | "Requirement already satisfied: pyrsistent>=0.14.0 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets>=7.5->evalml) (0.17.3)\n", 111 | "Requirement already satisfied: attrs>=17.4.0 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets>=7.5->evalml) (20.3.0)\n", 112 | "Requirement already satisfied: pywin32>=1.0; sys_platform == \"win32\" in c:\\users\\win10\\anaconda3\\lib\\site-packages (from jupyter-core->nbformat>=4.2.0->ipywidgets>=7.5->evalml) (227)\n", 113 | "Requirement already satisfied: locket in c:\\users\\win10\\anaconda3\\lib\\site-packages (from partd>=0.3.10; extra == \"dataframe\"->dask[dataframe]>=2.12.0->featuretools>=0.20.0->evalml) (0.2.0)\n", 114 | "Requirement already satisfied: heapdict in c:\\users\\win10\\anaconda3\\lib\\site-packages (from zict>=0.1.3->distributed>=2.12.0->featuretools>=0.20.0->evalml) (1.0.1)\n", 115 | "Requirement already satisfied: cffi>=1.0.0 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (1.14.3)\n", 116 | "Requirement already satisfied: pywinpty>=0.5 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from terminado>=0.8.3->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (0.5.7)\n", 117 | "Requirement already satisfied: MarkupSafe>=0.23 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from jinja2->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (1.1.1)\n", 118 | "Requirement already satisfied: testpath in c:\\users\\win10\\anaconda3\\lib\\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (0.4.4)\n", 119 | "Requirement already satisfied: nbclient<0.6.0,>=0.5.0 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (0.5.1)\n", 120 | "Requirement already satisfied: defusedxml in c:\\users\\win10\\anaconda3\\lib\\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (0.6.0)\n", 121 | "Requirement already satisfied: pandocfilters>=1.4.1 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (1.4.3)\n", 122 | "Requirement already satisfied: jupyterlab-pygments in c:\\users\\win10\\anaconda3\\lib\\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (0.1.2)\n", 123 | "Requirement already satisfied: entrypoints>=0.2.2 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (0.3)\n", 124 | "Requirement already satisfied: mistune<2,>=0.8.1 in c:\\users\\win10\\anaconda3\\lib\\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (0.8.4)\n", 125 | "Requirement already satisfied: bleach in c:\\users\\win10\\anaconda3\\lib\\site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (3.2.1)\n", 126 | "Requirement already satisfied: pycparser in c:\\users\\win10\\anaconda3\\lib\\site-packages (from cffi>=1.0.0->argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (2.20)\n", 127 | "Requirement already satisfied: nest-asyncio in c:\\users\\win10\\anaconda3\\lib\\site-packages (from nbclient<0.6.0,>=0.5.0->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (1.4.2)\n", 128 | "Requirement already satisfied: async-generator in c:\\users\\win10\\anaconda3\\lib\\site-packages (from nbclient<0.6.0,>=0.5.0->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (1.10)\n", 129 | "Requirement already satisfied: packaging in c:\\users\\win10\\anaconda3\\lib\\site-packages (from bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (20.4)\n", 130 | "Requirement already satisfied: webencodings in c:\\users\\win10\\anaconda3\\lib\\site-packages (from bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5->evalml) (0.5.1)\n" 131 | ] 132 | } 133 | ], 134 | "source": [ 135 | "!pip install evalml" 136 | ] 137 | }, 138 | { 139 | "cell_type": "markdown", 140 | "metadata": {}, 141 | "source": [ 142 | "### Loading The Dataset\n", 143 | "- We can also read the dataset from csv\n", 144 | "- then convert to datatable" 145 | ] 146 | }, 147 | { 148 | "cell_type": "code", 149 | "execution_count": 54, 150 | "metadata": {}, 151 | "outputs": [], 152 | "source": [ 153 | "import evalml\n", 154 | "X, y = evalml.demos.load_breast_cancer()\n", 155 | "X_train, X_test, y_train, y_test = evalml.preprocessing.split_data(X, y, problem_type='binary')" 156 | ] 157 | }, 158 | { 159 | "cell_type": "code", 160 | "execution_count": null, 161 | "metadata": {}, 162 | "outputs": [], 163 | "source": [] 164 | }, 165 | { 166 | "cell_type": "code", 167 | "execution_count": 55, 168 | "metadata": {}, 169 | "outputs": [ 170 | { 171 | "data": { 172 | "text/html": [ 173 | "
\n", 174 | "\n", 187 | "\n", 188 | " \n", 189 | " \n", 190 | " \n", 191 | " \n", 192 | " \n", 193 | " \n", 194 | " \n", 195 | " \n", 196 | " \n", 197 | " \n", 198 | " \n", 199 | " \n", 200 | " \n", 201 | " \n", 202 | " \n", 203 | " \n", 204 | " \n", 205 | " \n", 206 | " \n", 207 | " \n", 208 | " \n", 209 | " \n", 210 | " \n", 211 | " \n", 212 | " \n", 213 | " \n", 214 | " \n", 215 | " \n", 216 | " \n", 217 | " \n", 218 | " \n", 219 | " \n", 220 | " \n", 221 | " \n", 222 | " \n", 223 | " \n", 224 | " \n", 225 | " \n", 226 | " \n", 227 | " \n", 228 | " \n", 229 | " \n", 230 | " \n", 231 | " \n", 232 | " \n", 233 | " \n", 234 | " \n", 235 | " \n", 236 | " \n", 237 | " \n", 238 | " \n", 239 | " \n", 240 | " \n", 241 | " \n", 242 | " \n", 243 | " \n", 244 | " \n", 245 | " \n", 246 | " \n", 247 | " \n", 248 | " \n", 249 | " \n", 250 | " \n", 251 | " \n", 252 | " \n", 253 | " \n", 254 | " \n", 255 | " \n", 256 | " \n", 257 | " \n", 258 | " \n", 259 | " \n", 260 | " \n", 261 | " \n", 262 | " \n", 263 | " \n", 264 | " \n", 265 | " \n", 266 | " \n", 267 | " \n", 268 | " \n", 269 | " \n", 270 | " \n", 271 | " \n", 272 | " \n", 273 | " \n", 274 | " \n", 275 | " \n", 276 | " \n", 277 | " \n", 278 | " \n", 279 | " \n", 280 | " \n", 281 | " \n", 282 | " \n", 283 | " \n", 284 | " \n", 285 | " \n", 286 | " \n", 287 | " \n", 288 | " \n", 289 | " \n", 290 | " \n", 291 | " \n", 292 | " \n", 293 | " \n", 294 | " \n", 295 | " \n", 296 | " \n", 297 | " \n", 298 | " \n", 299 | " \n", 300 | " \n", 301 | " \n", 302 | " \n", 303 | " \n", 304 | " \n", 305 | " \n", 306 | " \n", 307 | " \n", 308 | " \n", 309 | " \n", 310 | " \n", 311 | " \n", 312 | " \n", 313 | " \n", 314 | " \n", 315 | " \n", 316 | " \n", 317 | " \n", 318 | " \n", 319 | " \n", 320 | " \n", 321 | " \n", 322 | " \n", 323 | " \n", 324 | " \n", 325 | " \n", 326 | " \n", 327 | " \n", 328 | " \n", 329 | " \n", 330 | " \n", 331 | " \n", 332 | " \n", 333 | " \n", 334 | " \n", 335 | " \n", 336 | " \n", 337 | " \n", 338 | " \n", 339 | " \n", 340 | " \n", 341 | " \n", 342 | " \n", 343 | " \n", 344 | " \n", 345 | " \n", 346 | " \n", 347 | " \n", 348 | " \n", 349 | " \n", 350 | " \n", 351 | " \n", 352 | " \n", 353 | " \n", 354 | " \n", 355 | " \n", 356 | " \n", 357 | " \n", 358 | " \n", 359 | " \n", 360 | " \n", 361 | " \n", 362 | " \n", 363 | " \n", 364 | " \n", 365 | " \n", 366 | " \n", 367 | " \n", 368 | " \n", 369 | " \n", 370 | " \n", 371 | " \n", 372 | " \n", 373 | " \n", 374 | " \n", 375 | " \n", 376 | " \n", 377 | " \n", 378 | " \n", 379 | " \n", 380 | " \n", 381 | " \n", 382 | " \n", 383 | " \n", 384 | " \n", 385 | " \n", 386 | " \n", 387 | " \n", 388 | " \n", 389 | " \n", 390 | " \n", 391 | " \n", 392 | " \n", 393 | " \n", 394 | " \n", 395 | " \n", 396 | " \n", 397 | " \n", 398 | " \n", 399 | " \n", 400 | " \n", 401 | " \n", 402 | " \n", 403 | " \n", 404 | " \n", 405 | " \n", 406 | " \n", 407 | " \n", 408 | "
Data Columnmean radiusmean texturemean perimetermean areamean smoothnessmean compactnessmean concavitymean concave pointsmean symmetrymean fractal dimension...worst radiusworst textureworst perimeterworst areaworst smoothnessworst compactnessworst concavityworst concave pointsworst symmetryworst fractal dimension
Physical Typefloat64float64float64float64float64float64float64float64float64float64...float64float64float64float64float64float64float64float64float64float64
Logical TypeDoubleDoubleDoubleDoubleDoubleDoubleDoubleDoubleDoubleDouble...DoubleDoubleDoubleDoubleDoubleDoubleDoubleDoubleDoubleDouble
Semantic Tag(s)['numeric']['numeric']['numeric']['numeric']['numeric']['numeric']['numeric']['numeric']['numeric']['numeric']...['numeric']['numeric']['numeric']['numeric']['numeric']['numeric']['numeric']['numeric']['numeric']['numeric']
38111.0414.9370.67372.70.079870.070790.035460.0207400.20030.06246...12.09020.8379.73447.10.10950.19820.155300.067540.32020.07287
14410.7514.9768.26355.30.077930.051390.022510.0078750.13990.05688...11.95020.7277.79441.20.10760.12230.097550.034130.23000.06769
13611.7116.6774.72423.60.105100.060950.035920.0260000.13390.05945...13.33025.4886.16546.70.12710.10280.104600.069680.17120.07343
1168.9515.7658.74245.20.094620.124300.092630.0230800.13050.07163...9.41417.0763.34270.00.11790.18790.154400.038460.16520.07722
56720.6029.33140.101265.00.117800.277000.351400.1520000.23970.07016...25.74039.42184.601821.00.16500.86810.938700.265000.40870.12400
\n", 409 | "

5 rows × 30 columns

\n", 410 | "
" 411 | ], 412 | "text/plain": [ 413 | "Data Column mean radius mean texture mean perimeter mean area \\\n", 414 | "Physical Type float64 float64 float64 float64 \n", 415 | "Logical Type Double Double Double Double \n", 416 | "Semantic Tag(s) ['numeric'] ['numeric'] ['numeric'] ['numeric'] \n", 417 | "381 11.04 14.93 70.67 372.7 \n", 418 | "144 10.75 14.97 68.26 355.3 \n", 419 | "136 11.71 16.67 74.72 423.6 \n", 420 | "116 8.95 15.76 58.74 245.2 \n", 421 | "567 20.60 29.33 140.10 1265.0 \n", 422 | "\n", 423 | "Data Column mean smoothness mean compactness mean concavity \\\n", 424 | "Physical Type float64 float64 float64 \n", 425 | "Logical Type Double Double Double \n", 426 | "Semantic Tag(s) ['numeric'] ['numeric'] ['numeric'] \n", 427 | "381 0.07987 0.07079 0.03546 \n", 428 | "144 0.07793 0.05139 0.02251 \n", 429 | "136 0.10510 0.06095 0.03592 \n", 430 | "116 0.09462 0.12430 0.09263 \n", 431 | "567 0.11780 0.27700 0.35140 \n", 432 | "\n", 433 | "Data Column mean concave points mean symmetry mean fractal dimension ... \\\n", 434 | "Physical Type float64 float64 float64 ... \n", 435 | "Logical Type Double Double Double ... \n", 436 | "Semantic Tag(s) ['numeric'] ['numeric'] ['numeric'] ... \n", 437 | "381 0.020740 0.2003 0.06246 ... \n", 438 | "144 0.007875 0.1399 0.05688 ... \n", 439 | "136 0.026000 0.1339 0.05945 ... \n", 440 | "116 0.023080 0.1305 0.07163 ... \n", 441 | "567 0.152000 0.2397 0.07016 ... \n", 442 | "\n", 443 | "Data Column worst radius worst texture worst perimeter worst area \\\n", 444 | "Physical Type float64 float64 float64 float64 \n", 445 | "Logical Type Double Double Double Double \n", 446 | "Semantic Tag(s) ['numeric'] ['numeric'] ['numeric'] ['numeric'] \n", 447 | "381 12.090 20.83 79.73 447.1 \n", 448 | "144 11.950 20.72 77.79 441.2 \n", 449 | "136 13.330 25.48 86.16 546.7 \n", 450 | "116 9.414 17.07 63.34 270.0 \n", 451 | "567 25.740 39.42 184.60 1821.0 \n", 452 | "\n", 453 | "Data Column worst smoothness worst compactness worst concavity \\\n", 454 | "Physical Type float64 float64 float64 \n", 455 | "Logical Type Double Double Double \n", 456 | "Semantic Tag(s) ['numeric'] ['numeric'] ['numeric'] \n", 457 | "381 0.1095 0.1982 0.15530 \n", 458 | "144 0.1076 0.1223 0.09755 \n", 459 | "136 0.1271 0.1028 0.10460 \n", 460 | "116 0.1179 0.1879 0.15440 \n", 461 | "567 0.1650 0.8681 0.93870 \n", 462 | "\n", 463 | "Data Column worst concave points worst symmetry worst fractal dimension \n", 464 | "Physical Type float64 float64 float64 \n", 465 | "Logical Type Double Double Double \n", 466 | "Semantic Tag(s) ['numeric'] ['numeric'] ['numeric'] \n", 467 | "381 0.06754 0.3202 0.07287 \n", 468 | "144 0.03413 0.2300 0.06769 \n", 469 | "136 0.06968 0.1712 0.07343 \n", 470 | "116 0.03846 0.1652 0.07722 \n", 471 | "567 0.26500 0.4087 0.12400 \n", 472 | "\n", 473 | "[5 rows x 30 columns]" 474 | ] 475 | }, 476 | "execution_count": 55, 477 | "metadata": {}, 478 | "output_type": "execute_result" 479 | } 480 | ], 481 | "source": [ 482 | "X_train.head()" 483 | ] 484 | }, 485 | { 486 | "cell_type": "markdown", 487 | "metadata": {}, 488 | "source": [ 489 | "### Running the AutoML to select the best algorithm" 490 | ] 491 | }, 492 | { 493 | "cell_type": "code", 494 | "execution_count": 29, 495 | "metadata": {}, 496 | "outputs": [ 497 | { 498 | "data": { 499 | "text/plain": [ 500 | "[,\n", 501 | " ,\n", 502 | " ,\n", 503 | " ,\n", 504 | " ,\n", 505 | " ]" 506 | ] 507 | }, 508 | "execution_count": 29, 509 | "metadata": {}, 510 | "output_type": "execute_result" 511 | } 512 | ], 513 | "source": [ 514 | "import evalml\n", 515 | "evalml.problem_types.ProblemTypes.all_problem_types" 516 | ] 517 | }, 518 | { 519 | "cell_type": "code", 520 | "execution_count": 56, 521 | "metadata": {}, 522 | "outputs": [ 523 | { 524 | "name": "stdout", 525 | "output_type": "stream", 526 | "text": [ 527 | "Using default limit of max_batches=1.\n", 528 | "\n", 529 | "Generating pipelines to search over...\n", 530 | "*****************************\n", 531 | "* Beginning pipeline search *\n", 532 | "*****************************\n", 533 | "\n", 534 | "Optimizing for Log Loss Binary. \n", 535 | "Lower score is better.\n", 536 | "\n", 537 | "Using SequentialEngine to train and score pipelines.\n", 538 | "Searching up to 1 batches for a total of 9 pipelines. \n", 539 | "Allowed model families: xgboost, extra_trees, lightgbm, random_forest, catboost, decision_tree, linear_model\n", 540 | "\n" 541 | ] 542 | }, 543 | { 544 | "data": { 545 | "application/vnd.jupyter.widget-view+json": { 546 | "model_id": "8805e8da8bbe42aa80fe1e94d3413d3c", 547 | "version_major": 2, 548 | "version_minor": 0 549 | }, 550 | "text/plain": [ 551 | "FigureWidget({\n", 552 | " 'data': [{'mode': 'lines+markers',\n", 553 | " 'name': 'Best Score',\n", 554 | " 'type'…" 555 | ] 556 | }, 557 | "metadata": {}, 558 | "output_type": "display_data" 559 | }, 560 | { 561 | "name": "stdout", 562 | "output_type": "stream", 563 | "text": [ 564 | "Batch 1: (1/9) Mode Baseline Binary Classification P... Elapsed:00:00\n", 565 | "\tStarting cross validation\n", 566 | "\tFinished cross validation - mean Log Loss Binary: 12.904\n", 567 | "Batch 1: (2/9) Decision Tree Classifier w/ Imputer Elapsed:00:00\n", 568 | "\tStarting cross validation\n", 569 | "\tFinished cross validation - mean Log Loss Binary: 2.432\n", 570 | "High coefficient of variation (cv >= 0.2) within cross validation scores. Decision Tree Classifier w/ Imputer may not perform as estimated on unseen data.\n", 571 | "Batch 1: (3/9) LightGBM Classifier w/ Imputer Elapsed:00:00\n", 572 | "\tStarting cross validation\n", 573 | "\tFinished cross validation - mean Log Loss Binary: 0.133\n", 574 | "Batch 1: (4/9) Extra Trees Classifier w/ Imputer Elapsed:00:01\n", 575 | "\tStarting cross validation\n", 576 | "\tFinished cross validation - mean Log Loss Binary: 0.137\n", 577 | "Batch 1: (5/9) Elastic Net Classifier w/ Imputer + S... Elapsed:00:02\n", 578 | "\tStarting cross validation\n", 579 | "\tFinished cross validation - mean Log Loss Binary: 0.506\n", 580 | "Batch 1: (6/9) CatBoost Classifier w/ Imputer Elapsed:00:02\n", 581 | "\tStarting cross validation\n", 582 | "\tFinished cross validation - mean Log Loss Binary: 0.386\n", 583 | "Batch 1: (7/9) XGBoost Classifier w/ Imputer Elapsed:00:03\n", 584 | "\tStarting cross validation\n", 585 | "\tFinished cross validation - mean Log Loss Binary: 0.113\n", 586 | "High coefficient of variation (cv >= 0.2) within cross validation scores. XGBoost Classifier w/ Imputer may not perform as estimated on unseen data.\n", 587 | "Batch 1: (8/9) Random Forest Classifier w/ Imputer Elapsed:00:04\n", 588 | "\tStarting cross validation\n", 589 | "\tFinished cross validation - mean Log Loss Binary: 0.120\n", 590 | "Batch 1: (9/9) Logistic Regression Classifier w/ Imp... Elapsed:00:05\n", 591 | "\tStarting cross validation\n", 592 | "\tFinished cross validation - mean Log Loss Binary: 0.094\n", 593 | "High coefficient of variation (cv >= 0.2) within cross validation scores. Logistic Regression Classifier w/ Imputer + Standard Scaler may not perform as estimated on unseen data.\n", 594 | "\n", 595 | "Search finished after 00:08 \n", 596 | "Best pipeline: Logistic Regression Classifier w/ Imputer + Standard Scaler\n", 597 | "Best pipeline Log Loss Binary: 0.094015\n" 598 | ] 599 | } 600 | ], 601 | "source": [ 602 | "from evalml.automl import AutoMLSearch\n", 603 | "automl = AutoMLSearch(X_train=X_train, y_train=y_train, problem_type='binary')\n", 604 | "automl.search()" 605 | ] 606 | }, 607 | { 608 | "cell_type": "code", 609 | "execution_count": 57, 610 | "metadata": {}, 611 | "outputs": [ 612 | { 613 | "data": { 614 | "text/html": [ 615 | "
\n", 616 | "\n", 629 | "\n", 630 | " \n", 631 | " \n", 632 | " \n", 633 | " \n", 634 | " \n", 635 | " \n", 636 | " \n", 637 | " \n", 638 | " \n", 639 | " \n", 640 | " \n", 641 | " \n", 642 | " \n", 643 | " \n", 644 | " \n", 645 | " \n", 646 | " \n", 647 | " \n", 648 | " \n", 649 | " \n", 650 | " \n", 651 | " \n", 652 | " \n", 653 | " \n", 654 | " \n", 655 | " \n", 656 | " \n", 657 | " \n", 658 | " \n", 659 | " \n", 660 | " \n", 661 | " \n", 662 | " \n", 663 | " \n", 664 | " \n", 665 | " \n", 666 | " \n", 667 | " \n", 668 | " \n", 669 | " \n", 670 | " \n", 671 | " \n", 672 | " \n", 673 | " \n", 674 | " \n", 675 | " \n", 676 | " \n", 677 | " \n", 678 | " \n", 679 | " \n", 680 | " \n", 681 | " \n", 682 | " \n", 683 | " \n", 684 | " \n", 685 | " \n", 686 | " \n", 687 | " \n", 688 | " \n", 689 | " \n", 690 | " \n", 691 | " \n", 692 | " \n", 693 | " \n", 694 | " \n", 695 | " \n", 696 | " \n", 697 | " \n", 698 | " \n", 699 | " \n", 700 | " \n", 701 | " \n", 702 | " \n", 703 | " \n", 704 | " \n", 705 | " \n", 706 | " \n", 707 | " \n", 708 | " \n", 709 | " \n", 710 | " \n", 711 | " \n", 712 | " \n", 713 | " \n", 714 | " \n", 715 | " \n", 716 | " \n", 717 | " \n", 718 | " \n", 719 | " \n", 720 | " \n", 721 | " \n", 722 | " \n", 723 | " \n", 724 | " \n", 725 | " \n", 726 | " \n", 727 | " \n", 728 | " \n", 729 | " \n", 730 | " \n", 731 | " \n", 732 | " \n", 733 | " \n", 734 | "
idpipeline_namescorevalidation_scorepercent_better_than_baselinehigh_variance_cvparameters
08Logistic Regression Classifier w/ Imputer + St...0.0940150.06052999.271446True{'Imputer': {'categorical_impute_strategy': 'm...
16XGBoost Classifier w/ Imputer0.1130980.06904899.123568True{'Imputer': {'categorical_impute_strategy': 'm...
27Random Forest Classifier w/ Imputer0.1199720.09961499.070299False{'Imputer': {'categorical_impute_strategy': 'm...
32LightGBM Classifier w/ Imputer0.1327220.11067998.971496False{'Imputer': {'categorical_impute_strategy': 'm...
43Extra Trees Classifier w/ Imputer0.1369590.11116998.938661False{'Imputer': {'categorical_impute_strategy': 'm...
55CatBoost Classifier w/ Imputer0.3863870.37433897.005774False{'Imputer': {'categorical_impute_strategy': 'm...
64Elastic Net Classifier w/ Imputer + Standard S...0.5058620.49676796.079926False{'Imputer': {'categorical_impute_strategy': 'm...
71Decision Tree Classifier w/ Imputer2.4319162.72678281.154350True{'Imputer': {'categorical_impute_strategy': 'm...
80Mode Baseline Binary Classification Pipeline12.90438812.9520410.000000False{'Baseline Classifier': {'strategy': 'mode'}}
\n", 735 | "
" 736 | ], 737 | "text/plain": [ 738 | " id pipeline_name score \\\n", 739 | "0 8 Logistic Regression Classifier w/ Imputer + St... 0.094015 \n", 740 | "1 6 XGBoost Classifier w/ Imputer 0.113098 \n", 741 | "2 7 Random Forest Classifier w/ Imputer 0.119972 \n", 742 | "3 2 LightGBM Classifier w/ Imputer 0.132722 \n", 743 | "4 3 Extra Trees Classifier w/ Imputer 0.136959 \n", 744 | "5 5 CatBoost Classifier w/ Imputer 0.386387 \n", 745 | "6 4 Elastic Net Classifier w/ Imputer + Standard S... 0.505862 \n", 746 | "7 1 Decision Tree Classifier w/ Imputer 2.431916 \n", 747 | "8 0 Mode Baseline Binary Classification Pipeline 12.904388 \n", 748 | "\n", 749 | " validation_score percent_better_than_baseline high_variance_cv \\\n", 750 | "0 0.060529 99.271446 True \n", 751 | "1 0.069048 99.123568 True \n", 752 | "2 0.099614 99.070299 False \n", 753 | "3 0.110679 98.971496 False \n", 754 | "4 0.111169 98.938661 False \n", 755 | "5 0.374338 97.005774 False \n", 756 | "6 0.496767 96.079926 False \n", 757 | "7 2.726782 81.154350 True \n", 758 | "8 12.952041 0.000000 False \n", 759 | "\n", 760 | " parameters \n", 761 | "0 {'Imputer': {'categorical_impute_strategy': 'm... \n", 762 | "1 {'Imputer': {'categorical_impute_strategy': 'm... \n", 763 | "2 {'Imputer': {'categorical_impute_strategy': 'm... \n", 764 | "3 {'Imputer': {'categorical_impute_strategy': 'm... \n", 765 | "4 {'Imputer': {'categorical_impute_strategy': 'm... \n", 766 | "5 {'Imputer': {'categorical_impute_strategy': 'm... \n", 767 | "6 {'Imputer': {'categorical_impute_strategy': 'm... \n", 768 | "7 {'Imputer': {'categorical_impute_strategy': 'm... \n", 769 | "8 {'Baseline Classifier': {'strategy': 'mode'}} " 770 | ] 771 | }, 772 | "execution_count": 57, 773 | "metadata": {}, 774 | "output_type": "execute_result" 775 | } 776 | ], 777 | "source": [ 778 | "automl.rankings" 779 | ] 780 | }, 781 | { 782 | "cell_type": "markdown", 783 | "metadata": {}, 784 | "source": [ 785 | "### Getting The Best Pipeline" 786 | ] 787 | }, 788 | { 789 | "cell_type": "code", 790 | "execution_count": 58, 791 | "metadata": {}, 792 | "outputs": [ 793 | { 794 | "data": { 795 | "text/plain": [ 796 | "GeneratedPipeline(parameters={'Imputer':{'categorical_impute_strategy': 'most_frequent', 'numeric_impute_strategy': 'mean', 'categorical_fill_value': None, 'numeric_fill_value': None}, 'Logistic Regression Classifier':{'penalty': 'l2', 'C': 1.0, 'n_jobs': -1, 'multi_class': 'auto', 'solver': 'lbfgs'},})" 797 | ] 798 | }, 799 | "execution_count": 58, 800 | "metadata": {}, 801 | "output_type": "execute_result" 802 | } 803 | ], 804 | "source": [ 805 | "automl.best_pipeline" 806 | ] 807 | }, 808 | { 809 | "cell_type": "code", 810 | "execution_count": 59, 811 | "metadata": {}, 812 | "outputs": [], 813 | "source": [ 814 | "best_pipeline=automl.best_pipeline" 815 | ] 816 | }, 817 | { 818 | "cell_type": "markdown", 819 | "metadata": {}, 820 | "source": [ 821 | "### Let's Check the detailed desscription" 822 | ] 823 | }, 824 | { 825 | "cell_type": "code", 826 | "execution_count": 60, 827 | "metadata": {}, 828 | "outputs": [ 829 | { 830 | "name": "stdout", 831 | "output_type": "stream", 832 | "text": [ 833 | "***************************************************************\n", 834 | "* Logistic Regression Classifier w/ Imputer + Standard Scaler *\n", 835 | "***************************************************************\n", 836 | "\n", 837 | "Problem Type: binary\n", 838 | "Model Family: Linear\n", 839 | "\n", 840 | "Pipeline Steps\n", 841 | "==============\n", 842 | "1. Imputer\n", 843 | "\t * categorical_impute_strategy : most_frequent\n", 844 | "\t * numeric_impute_strategy : mean\n", 845 | "\t * categorical_fill_value : None\n", 846 | "\t * numeric_fill_value : None\n", 847 | "2. Standard Scaler\n", 848 | "3. Logistic Regression Classifier\n", 849 | "\t * penalty : l2\n", 850 | "\t * C : 1.0\n", 851 | "\t * n_jobs : -1\n", 852 | "\t * multi_class : auto\n", 853 | "\t * solver : lbfgs\n", 854 | "\n", 855 | "Training\n", 856 | "========\n", 857 | "Training for binary problems.\n", 858 | "Total training time (including CV): 3.4 seconds\n", 859 | "\n", 860 | "Cross Validation\n", 861 | "----------------\n", 862 | " Log Loss Binary MCC Binary AUC Precision F1 Balanced Accuracy Binary Accuracy Binary Sensitivity at Low Alert Rates # Training # Validation\n", 863 | "0 0.061 0.958 0.997 0.966 0.974 0.981 0.980 0.412 303.000 152.000\n", 864 | "1 0.128 0.930 0.984 0.981 0.955 0.960 0.967 0.333 303.000 152.000\n", 865 | "2 0.093 0.944 0.993 1.000 0.963 0.964 0.974 0.261 304.000 151.000\n", 866 | "mean 0.094 0.944 0.991 0.982 0.964 0.968 0.974 0.335 - -\n", 867 | "std 0.034 0.014 0.006 0.017 0.010 0.011 0.007 0.075 - -\n", 868 | "coef of var 0.359 0.015 0.007 0.018 0.010 0.011 0.007 0.225 - -\n" 869 | ] 870 | } 871 | ], 872 | "source": [ 873 | "automl.describe_pipeline(automl.rankings.iloc[0][\"id\"])" 874 | ] 875 | }, 876 | { 877 | "cell_type": "code", 878 | "execution_count": 61, 879 | "metadata": {}, 880 | "outputs": [ 881 | { 882 | "data": { 883 | "text/plain": [ 884 | "OrderedDict([('AUC', 0.9933862433862434),\n", 885 | " ('F1', 0.963855421686747),\n", 886 | " ('Precision', 0.975609756097561),\n", 887 | " ('Recall', 0.9523809523809523)])" 888 | ] 889 | }, 890 | "execution_count": 61, 891 | "metadata": {}, 892 | "output_type": "execute_result" 893 | } 894 | ], 895 | "source": [ 896 | "### Evaluate on hold out data\n", 897 | "best_pipeline.score(X_test, y_test, objectives=[\"auc\",\"f1\",\"Precision\",\"Recall\"])" 898 | ] 899 | }, 900 | { 901 | "cell_type": "markdown", 902 | "metadata": {}, 903 | "source": [ 904 | "### We can also optimize for a problem specific objective" 905 | ] 906 | }, 907 | { 908 | "cell_type": "code", 909 | "execution_count": 62, 910 | "metadata": {}, 911 | "outputs": [ 912 | { 913 | "name": "stdout", 914 | "output_type": "stream", 915 | "text": [ 916 | "Generating pipelines to search over...\n", 917 | "*****************************\n", 918 | "* Beginning pipeline search *\n", 919 | "*****************************\n", 920 | "\n", 921 | "Optimizing for AUC. \n", 922 | "Greater score is better.\n", 923 | "\n", 924 | "Using SequentialEngine to train and score pipelines.\n", 925 | "Searching up to 1 batches for a total of 9 pipelines. \n", 926 | "Allowed model families: xgboost, extra_trees, lightgbm, random_forest, catboost, decision_tree, linear_model\n", 927 | "\n" 928 | ] 929 | }, 930 | { 931 | "data": { 932 | "application/vnd.jupyter.widget-view+json": { 933 | "model_id": "eb34e094d01c4f1f8a6cbd317a515bdb", 934 | "version_major": 2, 935 | "version_minor": 0 936 | }, 937 | "text/plain": [ 938 | "FigureWidget({\n", 939 | " 'data': [{'mode': 'lines+markers',\n", 940 | " 'name': 'Best Score',\n", 941 | " 'type'…" 942 | ] 943 | }, 944 | "metadata": {}, 945 | "output_type": "display_data" 946 | }, 947 | { 948 | "name": "stdout", 949 | "output_type": "stream", 950 | "text": [ 951 | "Batch 1: (1/9) Mode Baseline Binary Classification P... Elapsed:00:00\n", 952 | "\tStarting cross validation\n", 953 | "\tFinished cross validation - mean AUC: 0.500\n", 954 | "Batch 1: (2/9) Decision Tree Classifier w/ Imputer Elapsed:00:00\n", 955 | "\tStarting cross validation\n", 956 | "\tFinished cross validation - mean AUC: 0.923\n", 957 | "Batch 1: (3/9) LightGBM Classifier w/ Imputer Elapsed:00:00\n", 958 | "\tStarting cross validation\n", 959 | "\tFinished cross validation - mean AUC: 0.991\n", 960 | "Batch 1: (4/9) Extra Trees Classifier w/ Imputer Elapsed:00:01\n", 961 | "\tStarting cross validation\n", 962 | "\tFinished cross validation - mean AUC: 0.993\n", 963 | "Batch 1: (5/9) Elastic Net Classifier w/ Imputer + S... Elapsed:00:02\n", 964 | "\tStarting cross validation\n", 965 | "\tFinished cross validation - mean AUC: 0.985\n", 966 | "Batch 1: (6/9) CatBoost Classifier w/ Imputer Elapsed:00:02\n", 967 | "\tStarting cross validation\n", 968 | "\tFinished cross validation - mean AUC: 0.991\n", 969 | "Batch 1: (7/9) XGBoost Classifier w/ Imputer Elapsed:00:03\n", 970 | "\tStarting cross validation\n", 971 | "\tFinished cross validation - mean AUC: 0.991\n", 972 | "Batch 1: (8/9) Random Forest Classifier w/ Imputer Elapsed:00:04\n", 973 | "\tStarting cross validation\n", 974 | "\tFinished cross validation - mean AUC: 0.992\n", 975 | "Batch 1: (9/9) Logistic Regression Classifier w/ Imp... Elapsed:00:05\n", 976 | "\tStarting cross validation\n", 977 | "\tFinished cross validation - mean AUC: 0.991\n", 978 | "\n", 979 | "Search finished after 00:08 \n", 980 | "Best pipeline: Extra Trees Classifier w/ Imputer\n", 981 | "Best pipeline AUC: 0.992791\n" 982 | ] 983 | } 984 | ], 985 | "source": [ 986 | "automl_auc = AutoMLSearch(X_train=X_train, y_train=y_train,\n", 987 | " problem_type='binary',\n", 988 | " objective='auc',\n", 989 | " additional_objectives=['f1', 'precision'],\n", 990 | " max_batches=1,\n", 991 | " optimize_thresholds=True)\n", 992 | "\n", 993 | "automl_auc.search()" 994 | ] 995 | }, 996 | { 997 | "cell_type": "code", 998 | "execution_count": 63, 999 | "metadata": {}, 1000 | "outputs": [ 1001 | { 1002 | "data": { 1003 | "text/html": [ 1004 | "
\n", 1005 | "\n", 1018 | "\n", 1019 | " \n", 1020 | " \n", 1021 | " \n", 1022 | " \n", 1023 | " \n", 1024 | " \n", 1025 | " \n", 1026 | " \n", 1027 | " \n", 1028 | " \n", 1029 | " \n", 1030 | " \n", 1031 | " \n", 1032 | " \n", 1033 | " \n", 1034 | " \n", 1035 | " \n", 1036 | " \n", 1037 | " \n", 1038 | " \n", 1039 | " \n", 1040 | " \n", 1041 | " \n", 1042 | " \n", 1043 | " \n", 1044 | " \n", 1045 | " \n", 1046 | " \n", 1047 | " \n", 1048 | " \n", 1049 | " \n", 1050 | " \n", 1051 | " \n", 1052 | " \n", 1053 | " \n", 1054 | " \n", 1055 | " \n", 1056 | " \n", 1057 | " \n", 1058 | " \n", 1059 | " \n", 1060 | " \n", 1061 | " \n", 1062 | " \n", 1063 | " \n", 1064 | " \n", 1065 | " \n", 1066 | " \n", 1067 | " \n", 1068 | " \n", 1069 | " \n", 1070 | " \n", 1071 | " \n", 1072 | " \n", 1073 | " \n", 1074 | " \n", 1075 | " \n", 1076 | " \n", 1077 | " \n", 1078 | " \n", 1079 | " \n", 1080 | " \n", 1081 | " \n", 1082 | " \n", 1083 | " \n", 1084 | " \n", 1085 | " \n", 1086 | " \n", 1087 | " \n", 1088 | " \n", 1089 | " \n", 1090 | " \n", 1091 | " \n", 1092 | " \n", 1093 | " \n", 1094 | " \n", 1095 | " \n", 1096 | " \n", 1097 | " \n", 1098 | " \n", 1099 | " \n", 1100 | " \n", 1101 | " \n", 1102 | " \n", 1103 | " \n", 1104 | " \n", 1105 | " \n", 1106 | " \n", 1107 | " \n", 1108 | " \n", 1109 | " \n", 1110 | " \n", 1111 | " \n", 1112 | " \n", 1113 | " \n", 1114 | " \n", 1115 | " \n", 1116 | " \n", 1117 | " \n", 1118 | " \n", 1119 | " \n", 1120 | " \n", 1121 | " \n", 1122 | " \n", 1123 | "
idpipeline_namescorevalidation_scorepercent_better_than_baselinehigh_variance_cvparameters
03Extra Trees Classifier w/ Imputer0.9927910.99575349.279119False{'Imputer': {'categorical_impute_strategy': 'm...
17Random Forest Classifier w/ Imputer0.9924820.99436749.248175False{'Imputer': {'categorical_impute_strategy': 'm...
28Logistic Regression Classifier w/ Imputer + St...0.9913420.99667649.134239False{'Imputer': {'categorical_impute_strategy': 'm...
35CatBoost Classifier w/ Imputer0.9913050.99390649.130502False{'Imputer': {'categorical_impute_strategy': 'm...
46XGBoost Classifier w/ Imputer0.9912650.99556849.126544False{'Imputer': {'categorical_impute_strategy': 'm...
52LightGBM Classifier w/ Imputer0.9907000.99150549.070044False{'Imputer': {'categorical_impute_strategy': 'm...
64Elastic Net Classifier w/ Imputer + Standard S...0.9849430.99686148.494262False{'Imputer': {'categorical_impute_strategy': 'm...
71Decision Tree Classifier w/ Imputer0.9233710.91929842.337093False{'Imputer': {'categorical_impute_strategy': 'm...
80Mode Baseline Binary Classification Pipeline0.5000000.5000000.000000False{'Baseline Classifier': {'strategy': 'mode'}}
\n", 1124 | "
" 1125 | ], 1126 | "text/plain": [ 1127 | " id pipeline_name score \\\n", 1128 | "0 3 Extra Trees Classifier w/ Imputer 0.992791 \n", 1129 | "1 7 Random Forest Classifier w/ Imputer 0.992482 \n", 1130 | "2 8 Logistic Regression Classifier w/ Imputer + St... 0.991342 \n", 1131 | "3 5 CatBoost Classifier w/ Imputer 0.991305 \n", 1132 | "4 6 XGBoost Classifier w/ Imputer 0.991265 \n", 1133 | "5 2 LightGBM Classifier w/ Imputer 0.990700 \n", 1134 | "6 4 Elastic Net Classifier w/ Imputer + Standard S... 0.984943 \n", 1135 | "7 1 Decision Tree Classifier w/ Imputer 0.923371 \n", 1136 | "8 0 Mode Baseline Binary Classification Pipeline 0.500000 \n", 1137 | "\n", 1138 | " validation_score percent_better_than_baseline high_variance_cv \\\n", 1139 | "0 0.995753 49.279119 False \n", 1140 | "1 0.994367 49.248175 False \n", 1141 | "2 0.996676 49.134239 False \n", 1142 | "3 0.993906 49.130502 False \n", 1143 | "4 0.995568 49.126544 False \n", 1144 | "5 0.991505 49.070044 False \n", 1145 | "6 0.996861 48.494262 False \n", 1146 | "7 0.919298 42.337093 False \n", 1147 | "8 0.500000 0.000000 False \n", 1148 | "\n", 1149 | " parameters \n", 1150 | "0 {'Imputer': {'categorical_impute_strategy': 'm... \n", 1151 | "1 {'Imputer': {'categorical_impute_strategy': 'm... \n", 1152 | "2 {'Imputer': {'categorical_impute_strategy': 'm... \n", 1153 | "3 {'Imputer': {'categorical_impute_strategy': 'm... \n", 1154 | "4 {'Imputer': {'categorical_impute_strategy': 'm... \n", 1155 | "5 {'Imputer': {'categorical_impute_strategy': 'm... \n", 1156 | "6 {'Imputer': {'categorical_impute_strategy': 'm... \n", 1157 | "7 {'Imputer': {'categorical_impute_strategy': 'm... \n", 1158 | "8 {'Baseline Classifier': {'strategy': 'mode'}} " 1159 | ] 1160 | }, 1161 | "execution_count": 63, 1162 | "metadata": {}, 1163 | "output_type": "execute_result" 1164 | } 1165 | ], 1166 | "source": [ 1167 | "automl_auc.rankings" 1168 | ] 1169 | }, 1170 | { 1171 | "cell_type": "code", 1172 | "execution_count": 66, 1173 | "metadata": {}, 1174 | "outputs": [ 1175 | { 1176 | "name": "stdout", 1177 | "output_type": "stream", 1178 | "text": [ 1179 | "*************************************\n", 1180 | "* Extra Trees Classifier w/ Imputer *\n", 1181 | "*************************************\n", 1182 | "\n", 1183 | "Problem Type: binary\n", 1184 | "Model Family: Extra Trees\n", 1185 | "\n", 1186 | "Pipeline Steps\n", 1187 | "==============\n", 1188 | "1. Imputer\n", 1189 | "\t * categorical_impute_strategy : most_frequent\n", 1190 | "\t * numeric_impute_strategy : mean\n", 1191 | "\t * categorical_fill_value : None\n", 1192 | "\t * numeric_fill_value : None\n", 1193 | "2. Extra Trees Classifier\n", 1194 | "\t * n_estimators : 100\n", 1195 | "\t * max_features : auto\n", 1196 | "\t * max_depth : 6\n", 1197 | "\t * min_samples_split : 2\n", 1198 | "\t * min_weight_fraction_leaf : 0.0\n", 1199 | "\t * n_jobs : -1\n", 1200 | "\n", 1201 | "Training\n", 1202 | "========\n", 1203 | "Training for binary problems.\n", 1204 | "Total training time (including CV): 0.9 seconds\n", 1205 | "\n", 1206 | "Cross Validation\n", 1207 | "----------------\n", 1208 | " AUC F1 Precision # Training # Validation\n", 1209 | "0 0.996 0.964 0.982 303.000 152.000\n", 1210 | "1 0.994 0.914 1.000 303.000 152.000\n", 1211 | "2 0.988 0.927 0.944 304.000 151.000\n", 1212 | "mean 0.993 0.935 0.975 - -\n", 1213 | "std 0.004 0.026 0.028 - -\n", 1214 | "coef of var 0.004 0.028 0.029 - -\n" 1215 | ] 1216 | } 1217 | ], 1218 | "source": [ 1219 | "automl_auc.describe_pipeline(automl_auc.rankings.iloc[0][\"id\"])" 1220 | ] 1221 | }, 1222 | { 1223 | "cell_type": "code", 1224 | "execution_count": 67, 1225 | "metadata": {}, 1226 | "outputs": [], 1227 | "source": [ 1228 | "best_pipeline_auc = automl_auc.best_pipeline" 1229 | ] 1230 | }, 1231 | { 1232 | "cell_type": "code", 1233 | "execution_count": 68, 1234 | "metadata": {}, 1235 | "outputs": [ 1236 | { 1237 | "data": { 1238 | "text/plain": [ 1239 | "OrderedDict([('AUC', 0.986111111111111)])" 1240 | ] 1241 | }, 1242 | "execution_count": 68, 1243 | "metadata": {}, 1244 | "output_type": "execute_result" 1245 | } 1246 | ], 1247 | "source": [ 1248 | "# get the score on holdout data\n", 1249 | "best_pipeline_auc.score(X_test, y_test, objectives=[\"auc\"])" 1250 | ] 1251 | }, 1252 | { 1253 | "cell_type": "code", 1254 | "execution_count": 69, 1255 | "metadata": {}, 1256 | "outputs": [], 1257 | "source": [ 1258 | "best_pipeline.save(\"model.pkl\")" 1259 | ] 1260 | }, 1261 | { 1262 | "cell_type": "markdown", 1263 | "metadata": {}, 1264 | "source": [ 1265 | "#### Loading the Model" 1266 | ] 1267 | }, 1268 | { 1269 | "cell_type": "code", 1270 | "execution_count": 70, 1271 | "metadata": {}, 1272 | "outputs": [], 1273 | "source": [ 1274 | "check_model=automl.load('model.pkl')" 1275 | ] 1276 | }, 1277 | { 1278 | "cell_type": "code", 1279 | "execution_count": 71, 1280 | "metadata": {}, 1281 | "outputs": [ 1282 | { 1283 | "data": { 1284 | "text/html": [ 1285 | "
\n", 1286 | "\n", 1299 | "\n", 1300 | " \n", 1301 | " \n", 1302 | " \n", 1303 | " \n", 1304 | " \n", 1305 | " \n", 1306 | " \n", 1307 | " \n", 1308 | " \n", 1309 | " \n", 1310 | " \n", 1311 | " \n", 1312 | " \n", 1313 | " \n", 1314 | " \n", 1315 | " \n", 1316 | " \n", 1317 | " \n", 1318 | " \n", 1319 | " \n", 1320 | " \n", 1321 | " \n", 1322 | " \n", 1323 | " \n", 1324 | " \n", 1325 | " \n", 1326 | " \n", 1327 | " \n", 1328 | " \n", 1329 | " \n", 1330 | " \n", 1331 | " \n", 1332 | " \n", 1333 | " \n", 1334 | " \n", 1335 | " \n", 1336 | " \n", 1337 | " \n", 1338 | " \n", 1339 | " \n", 1340 | " \n", 1341 | " \n", 1342 | " \n", 1343 | " \n", 1344 | " \n", 1345 | " \n", 1346 | " \n", 1347 | " \n", 1348 | " \n", 1349 | " \n", 1350 | " \n", 1351 | " \n", 1352 | " \n", 1353 | " \n", 1354 | " \n", 1355 | " \n", 1356 | " \n", 1357 | " \n", 1358 | " \n", 1359 | " \n", 1360 | " \n", 1361 | " \n", 1362 | " \n", 1363 | " \n", 1364 | "
benignmalignant
09.996252e-010.000375
19.845724e-010.015428
27.749595e-010.225040
39.907312e-010.009269
49.998272e-010.000173
.........
1099.990961e-010.000904
1107.981366e-010.201863
1119.999924e-010.000008
1121.082727e-081.000000
1139.999267e-010.000073
\n", 1365 | "

114 rows × 2 columns

\n", 1366 | "
" 1367 | ], 1368 | "text/plain": [ 1369 | " benign malignant\n", 1370 | "0 9.996252e-01 0.000375\n", 1371 | "1 9.845724e-01 0.015428\n", 1372 | "2 7.749595e-01 0.225040\n", 1373 | "3 9.907312e-01 0.009269\n", 1374 | "4 9.998272e-01 0.000173\n", 1375 | ".. ... ...\n", 1376 | "109 9.990961e-01 0.000904\n", 1377 | "110 7.981366e-01 0.201863\n", 1378 | "111 9.999924e-01 0.000008\n", 1379 | "112 1.082727e-08 1.000000\n", 1380 | "113 9.999267e-01 0.000073\n", 1381 | "\n", 1382 | "[114 rows x 2 columns]" 1383 | ] 1384 | }, 1385 | "execution_count": 71, 1386 | "metadata": {}, 1387 | "output_type": "execute_result" 1388 | } 1389 | ], 1390 | "source": [ 1391 | "check_model.predict_proba(X_test).to_dataframe()" 1392 | ] 1393 | }, 1394 | { 1395 | "cell_type": "code", 1396 | "execution_count": null, 1397 | "metadata": {}, 1398 | "outputs": [], 1399 | "source": [] 1400 | } 1401 | ], 1402 | "metadata": { 1403 | "kernelspec": { 1404 | "display_name": "Python 3", 1405 | "language": "python", 1406 | "name": "python3" 1407 | }, 1408 | "language_info": { 1409 | "codemirror_mode": { 1410 | "name": "ipython", 1411 | "version": 3 1412 | }, 1413 | "file_extension": ".py", 1414 | "mimetype": "text/x-python", 1415 | "name": "python", 1416 | "nbconvert_exporter": "python", 1417 | "pygments_lexer": "ipython3", 1418 | "version": "3.8.5" 1419 | } 1420 | }, 1421 | "nbformat": 4, 1422 | "nbformat_minor": 4 1423 | } 1424 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | GNU GENERAL PUBLIC LICENSE 2 | Version 3, 29 June 2007 3 | 4 | Copyright (C) 2007 Free Software Foundation, Inc. 5 | Everyone is permitted to copy and distribute verbatim copies 6 | of this license document, but changing it is not allowed. 7 | 8 | Preamble 9 | 10 | The GNU General Public License is a free, copyleft license for 11 | software and other kinds of works. 12 | 13 | The licenses for most software and other practical works are designed 14 | to take away your freedom to share and change the works. By contrast, 15 | the GNU General Public License is intended to guarantee your freedom to 16 | share and change all versions of a program--to make sure it remains free 17 | software for all its users. We, the Free Software Foundation, use the 18 | GNU General Public License for most of our software; it applies also to 19 | any other work released this way by its authors. You can apply it to 20 | your programs, too. 21 | 22 | When we speak of free software, we are referring to freedom, not 23 | price. Our General Public Licenses are designed to make sure that you 24 | have the freedom to distribute copies of free software (and charge for 25 | them if you wish), that you receive source code or can get it if you 26 | want it, that you can change the software or use pieces of it in new 27 | free programs, and that you know you can do these things. 28 | 29 | To protect your rights, we need to prevent others from denying you 30 | these rights or asking you to surrender the rights. Therefore, you have 31 | certain responsibilities if you distribute copies of the software, or if 32 | you modify it: responsibilities to respect the freedom of others. 33 | 34 | For example, if you distribute copies of such a program, whether 35 | gratis or for a fee, you must pass on to the recipients the same 36 | freedoms that you received. You must make sure that they, too, receive 37 | or can get the source code. And you must show them these terms so they 38 | know their rights. 39 | 40 | Developers that use the GNU GPL protect your rights with two steps: 41 | (1) assert copyright on the software, and (2) offer you this License 42 | giving you legal permission to copy, distribute and/or modify it. 43 | 44 | For the developers' and authors' protection, the GPL clearly explains 45 | that there is no warranty for this free software. For both users' and 46 | authors' sake, the GPL requires that modified versions be marked as 47 | changed, so that their problems will not be attributed erroneously to 48 | authors of previous versions. 49 | 50 | Some devices are designed to deny users access to install or run 51 | modified versions of the software inside them, although the manufacturer 52 | can do so. This is fundamentally incompatible with the aim of 53 | protecting users' freedom to change the software. The systematic 54 | pattern of such abuse occurs in the area of products for individuals to 55 | use, which is precisely where it is most unacceptable. Therefore, we 56 | have designed this version of the GPL to prohibit the practice for those 57 | products. If such problems arise substantially in other domains, we 58 | stand ready to extend this provision to those domains in future versions 59 | of the GPL, as needed to protect the freedom of users. 60 | 61 | Finally, every program is threatened constantly by software patents. 62 | States should not allow patents to restrict development and use of 63 | software on general-purpose computers, but in those that do, we wish to 64 | avoid the special danger that patents applied to a free program could 65 | make it effectively proprietary. To prevent this, the GPL assures that 66 | patents cannot be used to render the program non-free. 67 | 68 | The precise terms and conditions for copying, distribution and 69 | modification follow. 70 | 71 | TERMS AND CONDITIONS 72 | 73 | 0. Definitions. 74 | 75 | "This License" refers to version 3 of the GNU General Public License. 76 | 77 | "Copyright" also means copyright-like laws that apply to other kinds of 78 | works, such as semiconductor masks. 79 | 80 | "The Program" refers to any copyrightable work licensed under this 81 | License. Each licensee is addressed as "you". "Licensees" and 82 | "recipients" may be individuals or organizations. 83 | 84 | To "modify" a work means to copy from or adapt all or part of the work 85 | in a fashion requiring copyright permission, other than the making of an 86 | exact copy. The resulting work is called a "modified version" of the 87 | earlier work or a work "based on" the earlier work. 88 | 89 | A "covered work" means either the unmodified Program or a work based 90 | on the Program. 91 | 92 | To "propagate" a work means to do anything with it that, without 93 | permission, would make you directly or secondarily liable for 94 | infringement under applicable copyright law, except executing it on a 95 | computer or modifying a private copy. Propagation includes copying, 96 | distribution (with or without modification), making available to the 97 | public, and in some countries other activities as well. 98 | 99 | To "convey" a work means any kind of propagation that enables other 100 | parties to make or receive copies. Mere interaction with a user through 101 | a computer network, with no transfer of a copy, is not conveying. 102 | 103 | An interactive user interface displays "Appropriate Legal Notices" 104 | to the extent that it includes a convenient and prominently visible 105 | feature that (1) displays an appropriate copyright notice, and (2) 106 | tells the user that there is no warranty for the work (except to the 107 | extent that warranties are provided), that licensees may convey the 108 | work under this License, and how to view a copy of this License. If 109 | the interface presents a list of user commands or options, such as a 110 | menu, a prominent item in the list meets this criterion. 111 | 112 | 1. Source Code. 113 | 114 | The "source code" for a work means the preferred form of the work 115 | for making modifications to it. "Object code" means any non-source 116 | form of a work. 117 | 118 | A "Standard Interface" means an interface that either is an official 119 | standard defined by a recognized standards body, or, in the case of 120 | interfaces specified for a particular programming language, one that 121 | is widely used among developers working in that language. 122 | 123 | The "System Libraries" of an executable work include anything, other 124 | than the work as a whole, that (a) is included in the normal form of 125 | packaging a Major Component, but which is not part of that Major 126 | Component, and (b) serves only to enable use of the work with that 127 | Major Component, or to implement a Standard Interface for which an 128 | implementation is available to the public in source code form. A 129 | "Major Component", in this context, means a major essential component 130 | (kernel, window system, and so on) of the specific operating system 131 | (if any) on which the executable work runs, or a compiler used to 132 | produce the work, or an object code interpreter used to run it. 133 | 134 | The "Corresponding Source" for a work in object code form means all 135 | the source code needed to generate, install, and (for an executable 136 | work) run the object code and to modify the work, including scripts to 137 | control those activities. However, it does not include the work's 138 | System Libraries, or general-purpose tools or generally available free 139 | programs which are used unmodified in performing those activities but 140 | which are not part of the work. For example, Corresponding Source 141 | includes interface definition files associated with source files for 142 | the work, and the source code for shared libraries and dynamically 143 | linked subprograms that the work is specifically designed to require, 144 | such as by intimate data communication or control flow between those 145 | subprograms and other parts of the work. 146 | 147 | The Corresponding Source need not include anything that users 148 | can regenerate automatically from other parts of the Corresponding 149 | Source. 150 | 151 | The Corresponding Source for a work in source code form is that 152 | same work. 153 | 154 | 2. Basic Permissions. 155 | 156 | All rights granted under this License are granted for the term of 157 | copyright on the Program, and are irrevocable provided the stated 158 | conditions are met. This License explicitly affirms your unlimited 159 | permission to run the unmodified Program. The output from running a 160 | covered work is covered by this License only if the output, given its 161 | content, constitutes a covered work. This License acknowledges your 162 | rights of fair use or other equivalent, as provided by copyright law. 163 | 164 | You may make, run and propagate covered works that you do not 165 | convey, without conditions so long as your license otherwise remains 166 | in force. You may convey covered works to others for the sole purpose 167 | of having them make modifications exclusively for you, or provide you 168 | with facilities for running those works, provided that you comply with 169 | the terms of this License in conveying all material for which you do 170 | not control copyright. Those thus making or running the covered works 171 | for you must do so exclusively on your behalf, under your direction 172 | and control, on terms that prohibit them from making any copies of 173 | your copyrighted material outside their relationship with you. 174 | 175 | Conveying under any other circumstances is permitted solely under 176 | the conditions stated below. Sublicensing is not allowed; section 10 177 | makes it unnecessary. 178 | 179 | 3. Protecting Users' Legal Rights From Anti-Circumvention Law. 180 | 181 | No covered work shall be deemed part of an effective technological 182 | measure under any applicable law fulfilling obligations under article 183 | 11 of the WIPO copyright treaty adopted on 20 December 1996, or 184 | similar laws prohibiting or restricting circumvention of such 185 | measures. 186 | 187 | When you convey a covered work, you waive any legal power to forbid 188 | circumvention of technological measures to the extent such circumvention 189 | is effected by exercising rights under this License with respect to 190 | the covered work, and you disclaim any intention to limit operation or 191 | modification of the work as a means of enforcing, against the work's 192 | users, your or third parties' legal rights to forbid circumvention of 193 | technological measures. 194 | 195 | 4. Conveying Verbatim Copies. 196 | 197 | You may convey verbatim copies of the Program's source code as you 198 | receive it, in any medium, provided that you conspicuously and 199 | appropriately publish on each copy an appropriate copyright notice; 200 | keep intact all notices stating that this License and any 201 | non-permissive terms added in accord with section 7 apply to the code; 202 | keep intact all notices of the absence of any warranty; and give all 203 | recipients a copy of this License along with the Program. 204 | 205 | You may charge any price or no price for each copy that you convey, 206 | and you may offer support or warranty protection for a fee. 207 | 208 | 5. Conveying Modified Source Versions. 209 | 210 | You may convey a work based on the Program, or the modifications to 211 | produce it from the Program, in the form of source code under the 212 | terms of section 4, provided that you also meet all of these conditions: 213 | 214 | a) The work must carry prominent notices stating that you modified 215 | it, and giving a relevant date. 216 | 217 | b) The work must carry prominent notices stating that it is 218 | released under this License and any conditions added under section 219 | 7. This requirement modifies the requirement in section 4 to 220 | "keep intact all notices". 221 | 222 | c) You must license the entire work, as a whole, under this 223 | License to anyone who comes into possession of a copy. This 224 | License will therefore apply, along with any applicable section 7 225 | additional terms, to the whole of the work, and all its parts, 226 | regardless of how they are packaged. This License gives no 227 | permission to license the work in any other way, but it does not 228 | invalidate such permission if you have separately received it. 229 | 230 | d) If the work has interactive user interfaces, each must display 231 | Appropriate Legal Notices; however, if the Program has interactive 232 | interfaces that do not display Appropriate Legal Notices, your 233 | work need not make them do so. 234 | 235 | A compilation of a covered work with other separate and independent 236 | works, which are not by their nature extensions of the covered work, 237 | and which are not combined with it such as to form a larger program, 238 | in or on a volume of a storage or distribution medium, is called an 239 | "aggregate" if the compilation and its resulting copyright are not 240 | used to limit the access or legal rights of the compilation's users 241 | beyond what the individual works permit. Inclusion of a covered work 242 | in an aggregate does not cause this License to apply to the other 243 | parts of the aggregate. 244 | 245 | 6. Conveying Non-Source Forms. 246 | 247 | You may convey a covered work in object code form under the terms 248 | of sections 4 and 5, provided that you also convey the 249 | machine-readable Corresponding Source under the terms of this License, 250 | in one of these ways: 251 | 252 | a) Convey the object code in, or embodied in, a physical product 253 | (including a physical distribution medium), accompanied by the 254 | Corresponding Source fixed on a durable physical medium 255 | customarily used for software interchange. 256 | 257 | b) Convey the object code in, or embodied in, a physical product 258 | (including a physical distribution medium), accompanied by a 259 | written offer, valid for at least three years and valid for as 260 | long as you offer spare parts or customer support for that product 261 | model, to give anyone who possesses the object code either (1) a 262 | copy of the Corresponding Source for all the software in the 263 | product that is covered by this License, on a durable physical 264 | medium customarily used for software interchange, for a price no 265 | more than your reasonable cost of physically performing this 266 | conveying of source, or (2) access to copy the 267 | Corresponding Source from a network server at no charge. 268 | 269 | c) Convey individual copies of the object code with a copy of the 270 | written offer to provide the Corresponding Source. This 271 | alternative is allowed only occasionally and noncommercially, and 272 | only if you received the object code with such an offer, in accord 273 | with subsection 6b. 274 | 275 | d) Convey the object code by offering access from a designated 276 | place (gratis or for a charge), and offer equivalent access to the 277 | Corresponding Source in the same way through the same place at no 278 | further charge. You need not require recipients to copy the 279 | Corresponding Source along with the object code. If the place to 280 | copy the object code is a network server, the Corresponding Source 281 | may be on a different server (operated by you or a third party) 282 | that supports equivalent copying facilities, provided you maintain 283 | clear directions next to the object code saying where to find the 284 | Corresponding Source. Regardless of what server hosts the 285 | Corresponding Source, you remain obligated to ensure that it is 286 | available for as long as needed to satisfy these requirements. 287 | 288 | e) Convey the object code using peer-to-peer transmission, provided 289 | you inform other peers where the object code and Corresponding 290 | Source of the work are being offered to the general public at no 291 | charge under subsection 6d. 292 | 293 | A separable portion of the object code, whose source code is excluded 294 | from the Corresponding Source as a System Library, need not be 295 | included in conveying the object code work. 296 | 297 | A "User Product" is either (1) a "consumer product", which means any 298 | tangible personal property which is normally used for personal, family, 299 | or household purposes, or (2) anything designed or sold for incorporation 300 | into a dwelling. In determining whether a product is a consumer product, 301 | doubtful cases shall be resolved in favor of coverage. For a particular 302 | product received by a particular user, "normally used" refers to a 303 | typical or common use of that class of product, regardless of the status 304 | of the particular user or of the way in which the particular user 305 | actually uses, or expects or is expected to use, the product. A product 306 | is a consumer product regardless of whether the product has substantial 307 | commercial, industrial or non-consumer uses, unless such uses represent 308 | the only significant mode of use of the product. 309 | 310 | "Installation Information" for a User Product means any methods, 311 | procedures, authorization keys, or other information required to install 312 | and execute modified versions of a covered work in that User Product from 313 | a modified version of its Corresponding Source. The information must 314 | suffice to ensure that the continued functioning of the modified object 315 | code is in no case prevented or interfered with solely because 316 | modification has been made. 317 | 318 | If you convey an object code work under this section in, or with, or 319 | specifically for use in, a User Product, and the conveying occurs as 320 | part of a transaction in which the right of possession and use of the 321 | User Product is transferred to the recipient in perpetuity or for a 322 | fixed term (regardless of how the transaction is characterized), the 323 | Corresponding Source conveyed under this section must be accompanied 324 | by the Installation Information. But this requirement does not apply 325 | if neither you nor any third party retains the ability to install 326 | modified object code on the User Product (for example, the work has 327 | been installed in ROM). 328 | 329 | The requirement to provide Installation Information does not include a 330 | requirement to continue to provide support service, warranty, or updates 331 | for a work that has been modified or installed by the recipient, or for 332 | the User Product in which it has been modified or installed. Access to a 333 | network may be denied when the modification itself materially and 334 | adversely affects the operation of the network or violates the rules and 335 | protocols for communication across the network. 336 | 337 | Corresponding Source conveyed, and Installation Information provided, 338 | in accord with this section must be in a format that is publicly 339 | documented (and with an implementation available to the public in 340 | source code form), and must require no special password or key for 341 | unpacking, reading or copying. 342 | 343 | 7. Additional Terms. 344 | 345 | "Additional permissions" are terms that supplement the terms of this 346 | License by making exceptions from one or more of its conditions. 347 | Additional permissions that are applicable to the entire Program shall 348 | be treated as though they were included in this License, to the extent 349 | that they are valid under applicable law. If additional permissions 350 | apply only to part of the Program, that part may be used separately 351 | under those permissions, but the entire Program remains governed by 352 | this License without regard to the additional permissions. 353 | 354 | When you convey a copy of a covered work, you may at your option 355 | remove any additional permissions from that copy, or from any part of 356 | it. (Additional permissions may be written to require their own 357 | removal in certain cases when you modify the work.) You may place 358 | additional permissions on material, added by you to a covered work, 359 | for which you have or can give appropriate copyright permission. 360 | 361 | Notwithstanding any other provision of this License, for material you 362 | add to a covered work, you may (if authorized by the copyright holders of 363 | that material) supplement the terms of this License with terms: 364 | 365 | a) Disclaiming warranty or limiting liability differently from the 366 | terms of sections 15 and 16 of this License; or 367 | 368 | b) Requiring preservation of specified reasonable legal notices or 369 | author attributions in that material or in the Appropriate Legal 370 | Notices displayed by works containing it; or 371 | 372 | c) Prohibiting misrepresentation of the origin of that material, or 373 | requiring that modified versions of such material be marked in 374 | reasonable ways as different from the original version; or 375 | 376 | d) Limiting the use for publicity purposes of names of licensors or 377 | authors of the material; or 378 | 379 | e) Declining to grant rights under trademark law for use of some 380 | trade names, trademarks, or service marks; or 381 | 382 | f) Requiring indemnification of licensors and authors of that 383 | material by anyone who conveys the material (or modified versions of 384 | it) with contractual assumptions of liability to the recipient, for 385 | any liability that these contractual assumptions directly impose on 386 | those licensors and authors. 387 | 388 | All other non-permissive additional terms are considered "further 389 | restrictions" within the meaning of section 10. If the Program as you 390 | received it, or any part of it, contains a notice stating that it is 391 | governed by this License along with a term that is a further 392 | restriction, you may remove that term. If a license document contains 393 | a further restriction but permits relicensing or conveying under this 394 | License, you may add to a covered work material governed by the terms 395 | of that license document, provided that the further restriction does 396 | not survive such relicensing or conveying. 397 | 398 | If you add terms to a covered work in accord with this section, you 399 | must place, in the relevant source files, a statement of the 400 | additional terms that apply to those files, or a notice indicating 401 | where to find the applicable terms. 402 | 403 | Additional terms, permissive or non-permissive, may be stated in the 404 | form of a separately written license, or stated as exceptions; 405 | the above requirements apply either way. 406 | 407 | 8. Termination. 408 | 409 | You may not propagate or modify a covered work except as expressly 410 | provided under this License. Any attempt otherwise to propagate or 411 | modify it is void, and will automatically terminate your rights under 412 | this License (including any patent licenses granted under the third 413 | paragraph of section 11). 414 | 415 | However, if you cease all violation of this License, then your 416 | license from a particular copyright holder is reinstated (a) 417 | provisionally, unless and until the copyright holder explicitly and 418 | finally terminates your license, and (b) permanently, if the copyright 419 | holder fails to notify you of the violation by some reasonable means 420 | prior to 60 days after the cessation. 421 | 422 | Moreover, your license from a particular copyright holder is 423 | reinstated permanently if the copyright holder notifies you of the 424 | violation by some reasonable means, this is the first time you have 425 | received notice of violation of this License (for any work) from that 426 | copyright holder, and you cure the violation prior to 30 days after 427 | your receipt of the notice. 428 | 429 | Termination of your rights under this section does not terminate the 430 | licenses of parties who have received copies or rights from you under 431 | this License. If your rights have been terminated and not permanently 432 | reinstated, you do not qualify to receive new licenses for the same 433 | material under section 10. 434 | 435 | 9. Acceptance Not Required for Having Copies. 436 | 437 | You are not required to accept this License in order to receive or 438 | run a copy of the Program. Ancillary propagation of a covered work 439 | occurring solely as a consequence of using peer-to-peer transmission 440 | to receive a copy likewise does not require acceptance. However, 441 | nothing other than this License grants you permission to propagate or 442 | modify any covered work. These actions infringe copyright if you do 443 | not accept this License. Therefore, by modifying or propagating a 444 | covered work, you indicate your acceptance of this License to do so. 445 | 446 | 10. Automatic Licensing of Downstream Recipients. 447 | 448 | Each time you convey a covered work, the recipient automatically 449 | receives a license from the original licensors, to run, modify and 450 | propagate that work, subject to this License. You are not responsible 451 | for enforcing compliance by third parties with this License. 452 | 453 | An "entity transaction" is a transaction transferring control of an 454 | organization, or substantially all assets of one, or subdividing an 455 | organization, or merging organizations. If propagation of a covered 456 | work results from an entity transaction, each party to that 457 | transaction who receives a copy of the work also receives whatever 458 | licenses to the work the party's predecessor in interest had or could 459 | give under the previous paragraph, plus a right to possession of the 460 | Corresponding Source of the work from the predecessor in interest, if 461 | the predecessor has it or can get it with reasonable efforts. 462 | 463 | You may not impose any further restrictions on the exercise of the 464 | rights granted or affirmed under this License. For example, you may 465 | not impose a license fee, royalty, or other charge for exercise of 466 | rights granted under this License, and you may not initiate litigation 467 | (including a cross-claim or counterclaim in a lawsuit) alleging that 468 | any patent claim is infringed by making, using, selling, offering for 469 | sale, or importing the Program or any portion of it. 470 | 471 | 11. Patents. 472 | 473 | A "contributor" is a copyright holder who authorizes use under this 474 | License of the Program or a work on which the Program is based. The 475 | work thus licensed is called the contributor's "contributor version". 476 | 477 | A contributor's "essential patent claims" are all patent claims 478 | owned or controlled by the contributor, whether already acquired or 479 | hereafter acquired, that would be infringed by some manner, permitted 480 | by this License, of making, using, or selling its contributor version, 481 | but do not include claims that would be infringed only as a 482 | consequence of further modification of the contributor version. For 483 | purposes of this definition, "control" includes the right to grant 484 | patent sublicenses in a manner consistent with the requirements of 485 | this License. 486 | 487 | Each contributor grants you a non-exclusive, worldwide, royalty-free 488 | patent license under the contributor's essential patent claims, to 489 | make, use, sell, offer for sale, import and otherwise run, modify and 490 | propagate the contents of its contributor version. 491 | 492 | In the following three paragraphs, a "patent license" is any express 493 | agreement or commitment, however denominated, not to enforce a patent 494 | (such as an express permission to practice a patent or covenant not to 495 | sue for patent infringement). To "grant" such a patent license to a 496 | party means to make such an agreement or commitment not to enforce a 497 | patent against the party. 498 | 499 | If you convey a covered work, knowingly relying on a patent license, 500 | and the Corresponding Source of the work is not available for anyone 501 | to copy, free of charge and under the terms of this License, through a 502 | publicly available network server or other readily accessible means, 503 | then you must either (1) cause the Corresponding Source to be so 504 | available, or (2) arrange to deprive yourself of the benefit of the 505 | patent license for this particular work, or (3) arrange, in a manner 506 | consistent with the requirements of this License, to extend the patent 507 | license to downstream recipients. "Knowingly relying" means you have 508 | actual knowledge that, but for the patent license, your conveying the 509 | covered work in a country, or your recipient's use of the covered work 510 | in a country, would infringe one or more identifiable patents in that 511 | country that you have reason to believe are valid. 512 | 513 | If, pursuant to or in connection with a single transaction or 514 | arrangement, you convey, or propagate by procuring conveyance of, a 515 | covered work, and grant a patent license to some of the parties 516 | receiving the covered work authorizing them to use, propagate, modify 517 | or convey a specific copy of the covered work, then the patent license 518 | you grant is automatically extended to all recipients of the covered 519 | work and works based on it. 520 | 521 | A patent license is "discriminatory" if it does not include within 522 | the scope of its coverage, prohibits the exercise of, or is 523 | conditioned on the non-exercise of one or more of the rights that are 524 | specifically granted under this License. You may not convey a covered 525 | work if you are a party to an arrangement with a third party that is 526 | in the business of distributing software, under which you make payment 527 | to the third party based on the extent of your activity of conveying 528 | the work, and under which the third party grants, to any of the 529 | parties who would receive the covered work from you, a discriminatory 530 | patent license (a) in connection with copies of the covered work 531 | conveyed by you (or copies made from those copies), or (b) primarily 532 | for and in connection with specific products or compilations that 533 | contain the covered work, unless you entered into that arrangement, 534 | or that patent license was granted, prior to 28 March 2007. 535 | 536 | Nothing in this License shall be construed as excluding or limiting 537 | any implied license or other defenses to infringement that may 538 | otherwise be available to you under applicable patent law. 539 | 540 | 12. No Surrender of Others' Freedom. 541 | 542 | If conditions are imposed on you (whether by court order, agreement or 543 | otherwise) that contradict the conditions of this License, they do not 544 | excuse you from the conditions of this License. If you cannot convey a 545 | covered work so as to satisfy simultaneously your obligations under this 546 | License and any other pertinent obligations, then as a consequence you may 547 | not convey it at all. For example, if you agree to terms that obligate you 548 | to collect a royalty for further conveying from those to whom you convey 549 | the Program, the only way you could satisfy both those terms and this 550 | License would be to refrain entirely from conveying the Program. 551 | 552 | 13. Use with the GNU Affero General Public License. 553 | 554 | Notwithstanding any other provision of this License, you have 555 | permission to link or combine any covered work with a work licensed 556 | under version 3 of the GNU Affero General Public License into a single 557 | combined work, and to convey the resulting work. The terms of this 558 | License will continue to apply to the part which is the covered work, 559 | but the special requirements of the GNU Affero General Public License, 560 | section 13, concerning interaction through a network will apply to the 561 | combination as such. 562 | 563 | 14. Revised Versions of this License. 564 | 565 | The Free Software Foundation may publish revised and/or new versions of 566 | the GNU General Public License from time to time. Such new versions will 567 | be similar in spirit to the present version, but may differ in detail to 568 | address new problems or concerns. 569 | 570 | Each version is given a distinguishing version number. If the 571 | Program specifies that a certain numbered version of the GNU General 572 | Public License "or any later version" applies to it, you have the 573 | option of following the terms and conditions either of that numbered 574 | version or of any later version published by the Free Software 575 | Foundation. If the Program does not specify a version number of the 576 | GNU General Public License, you may choose any version ever published 577 | by the Free Software Foundation. 578 | 579 | If the Program specifies that a proxy can decide which future 580 | versions of the GNU General Public License can be used, that proxy's 581 | public statement of acceptance of a version permanently authorizes you 582 | to choose that version for the Program. 583 | 584 | Later license versions may give you additional or different 585 | permissions. However, no additional obligations are imposed on any 586 | author or copyright holder as a result of your choosing to follow a 587 | later version. 588 | 589 | 15. Disclaimer of Warranty. 590 | 591 | THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY 592 | APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT 593 | HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY 594 | OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, 595 | THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR 596 | PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM 597 | IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF 598 | ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 599 | 600 | 16. Limitation of Liability. 601 | 602 | IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING 603 | WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS 604 | THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY 605 | GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE 606 | USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF 607 | DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD 608 | PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), 609 | EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF 610 | SUCH DAMAGES. 611 | 612 | 17. Interpretation of Sections 15 and 16. 613 | 614 | If the disclaimer of warranty and limitation of liability provided 615 | above cannot be given local legal effect according to their terms, 616 | reviewing courts shall apply local law that most closely approximates 617 | an absolute waiver of all civil liability in connection with the 618 | Program, unless a warranty or assumption of liability accompanies a 619 | copy of the Program in return for a fee. 620 | 621 | END OF TERMS AND CONDITIONS 622 | 623 | How to Apply These Terms to Your New Programs 624 | 625 | If you develop a new program, and you want it to be of the greatest 626 | possible use to the public, the best way to achieve this is to make it 627 | free software which everyone can redistribute and change under these terms. 628 | 629 | To do so, attach the following notices to the program. It is safest 630 | to attach them to the start of each source file to most effectively 631 | state the exclusion of warranty; and each file should have at least 632 | the "copyright" line and a pointer to where the full notice is found. 633 | 634 | 635 | Copyright (C) 636 | 637 | This program is free software: you can redistribute it and/or modify 638 | it under the terms of the GNU General Public License as published by 639 | the Free Software Foundation, either version 3 of the License, or 640 | (at your option) any later version. 641 | 642 | This program is distributed in the hope that it will be useful, 643 | but WITHOUT ANY WARRANTY; without even the implied warranty of 644 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 645 | GNU General Public License for more details. 646 | 647 | You should have received a copy of the GNU General Public License 648 | along with this program. If not, see . 649 | 650 | Also add information on how to contact you by electronic and paper mail. 651 | 652 | If the program does terminal interaction, make it output a short 653 | notice like this when it starts in an interactive mode: 654 | 655 | Copyright (C) 656 | This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. 657 | This is free software, and you are welcome to redistribute it 658 | under certain conditions; type `show c' for details. 659 | 660 | The hypothetical commands `show w' and `show c' should show the appropriate 661 | parts of the General Public License. Of course, your program's commands 662 | might be different; for a GUI interface, you would use an "about box". 663 | 664 | You should also get your employer (if you work as a programmer) or school, 665 | if any, to sign a "copyright disclaimer" for the program, if necessary. 666 | For more information on this, and how to apply and follow the GNU GPL, see 667 | . 668 | 669 | The GNU General Public License does not permit incorporating your program 670 | into proprietary programs. If your program is a subroutine library, you 671 | may consider it more useful to permit linking proprietary applications with 672 | the library. If this is what you want to do, use the GNU Lesser General 673 | Public License instead of this License. But first, please read 674 | . 675 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # EVALML --------------------------------------------------------------------------------