├── README.md ├── cloud ├── bash.md ├── cloud.md └── version.md └── pytorch ├── resources ├── arch.jpeg ├── grad.jpeg ├── graph_backprop.png ├── rnn_components.png ├── rnn_schema.png ├── rules.png └── shakespeare.txt ├── udacity_cnn_tatiana_gaponova.pdf ├── udacity_nn_basics_tatiana_gaponova.pdf ├── udacity_style_transfer_tatiana_gaponova.pdf └── vanillaRNN.ipynb /README.md: -------------------------------------------------------------------------------- 1 | # udacity_notes 2 | 3 | These are my notes that were taken during various Udacity scholarship programs. All images (and sometimes text) belong to Udacity. 4 | 5 | ## DevOps Scholarship from Bertelsmann 6 | 7 | 1. [BASH basics](cloud/bash.md) 8 | 2. [Version control](cloud/version.md) 9 | 3. [Cloud](cloud/cloud.md) 10 | 11 | ## PyTorch Scholarship from Facebook 12 | 13 | 1. [Deep Learing basics](pytorch/udacity_nn_basics_tatiana_gaponova.pdf) 14 | 2. [CNN theory](pytorch/udacity_cnn_tatiana_gaponova.pdf) 15 | 3. [Style transfer](pytorch/udacity_style_transfer_tatiana_gaponova.pdf) 16 | 4. [RNN theory](pytorch/vanillaRNN.ipynb) 17 | -------------------------------------------------------------------------------- /cloud/bash.md: -------------------------------------------------------------------------------- 1 | # BASH basics 2 | 3 | 4 | - __echo X__ - repeat the string X back 5 | 6 | ``` bash 7 | echo hello 8 | hello 9 | ``` 10 |
11 |
12 | 13 | - __!!__ - re-run the last command 14 | 15 | ``` bash 16 | !! 17 | echo hello 18 | hello 19 | ``` 20 |
21 |
22 | 23 | - __$X__ - reference of the variable X 24 | 25 | ``` bash 26 | x=100 27 | echo $x 28 | 100 29 | ``` 30 |
31 |
32 | 33 | - __ls__ - list the contents of the current directory 34 | 35 | __ls -l__ - a more detailed list 36 | 37 | __cd X__ - change directory to X 38 | 39 | __pwd__ - print the current (working) directory 40 |
41 |
42 | 43 | - Shortcuts: 44 | 45 | . - working directory 46 | 47 | .. - parent directory 48 | 49 | ~ - home directory 50 |
51 |
52 |
53 | 54 | - __mkdir X__ - create a new directory X 55 | 56 | __mv X Y__ - more the file X to the directory Y 57 |
58 |
59 | 60 | - __curl 'http://www.google.com'__ - download the page 61 | 62 | __curl -L -o some_file.html 'http://www.google.com'__ - download the page following redirects (-L) and outputing the result to the _some_file.html_ (-o) 63 |
64 |
65 | 66 | - __cat X__ - display the contents of a file to the terminal 67 | 68 | __less X__ - display the contents of a file to the terminal with the possibility to search and scroll 69 |
70 |
71 | 72 | - Managing the __less__ command: 73 | 74 | scroll down - spacebar or down arrow 75 | 76 | scroll up - b key or up arrow 77 | 78 | search - forward slack / 79 | 80 | exit less - q key 81 |
82 |
83 | 84 | - __rm X__ - permanently delete the file X 85 | 86 | __rm -i X__ - permanently delete the file X but double check first 87 | 88 | __rm X Y Z__ permanently delete the files X, Y, Z 89 | 90 | __rmdir X__ - permanently delete the directory X (should be empty) 91 | 92 | __rmdir -r X__ - permanently delete the directory X with all it's content 93 | 94 | 95 | -------------------------------------------------------------------------------- /cloud/cloud.md: -------------------------------------------------------------------------------- 1 | # Cloud Computing 2 | 3 | **Cloud Computing** is the delivery of IT resources over the Internet, the cloud is like a virtual data center accessible via the Internet 4 | 5 | **Pay as you go** - you pay only for what you use and only when your code runs 6 | 7 | **Autoscaling** - the number of active servers can grow or shrink based on demand 8 | 9 | **Serverless** - allows you to write and deploy code without having to worry about the underlying infrastructure 10 | 11 | _Benefits_: 12 | 13 | - stop guessing about capacity 14 | - avoid huge capital investments up front 15 | - pay for only what you use 16 | - scale globally in minutes 17 | - deliver faster 18 | 19 | 20 | ## Types of Cloud Computing 21 | 22 | **Infrastructure-as-a-Service (IaaS)** - the provider supplies virtual server instances, storage, and mechanisms for you to manage servers 23 | 24 | **Platform-as-a-Service (PaaS)** - a platform of development tools hosted on a provider's infrastructure 25 | 26 | **Software-as-a-Service (SaaS)** - a software application that runs over the Internet and is managed by the service provider 27 | 28 | 29 | ## Cloud Deployment Models 30 | 31 | **Public Cloud** - a public cloud makes resources available over the Internet to the general public 32 | 33 | **Private Cloud** - a private cloud is a proprietary network that supplies services to a limited number of people 34 | 35 | **Hybrid Cloud** - a hybrid model contains a combination of both a public and a private cloud 36 | 37 | 38 | ## Logging In The Cloud 39 | 40 | **Logging** provides visibility into your cloud resources and applications. For applications that run in the cloud, you will need access to logging and auditing services to help you proactively monitor your resources and applications 41 | 42 | 43 | ## Cloud Trail 44 | **Cloud Trail** allows you to audit (or review) everything that occurs in your AWS account. Cloud Trail does this by recording all the AWS API calls occurring in your account and delivering a log file to you 45 | 46 | _Features_: 47 | - who has logged in 48 | - services that were accessed 49 | - actions performed 50 | - parameters for the actions 51 | - responses returned 52 | 53 | 54 | ## Cloud Watch 55 | **Cloud Watch** is a service that monitors resources and applications that run on AWS by collecting data in the form of logs, metrics, and events 56 | 57 | _Features_: 58 | - collect and track metrics 59 | - collect and monitor log files 60 | - set alarms and create triggers to run your AWS resources 61 | - react to changes in your AWS resources 62 | 63 | 64 | ## Infrastructure as Code 65 | **Infrastructure as Code** allows you to describe and provision all the infrastructure resources in your cloud environment. You can stand up servers, databases, runtime parameters, resources, etc. based on scripts that you write. Infrastructure as Code is a time-saving feature because it allows you to provision (or stand up) resources in a reproducible way 66 | 67 | 68 | ## Cloud Formation 69 | **AWS Cloud Formation** allows you to model your entire infrastructure in a text file template allowing you to provision AWS resources based on the scripts you write 70 | 71 | 72 | ## Global Infrastructure 73 | **Region** - a region is considered a geographic location or an area on a map 74 | **Availability Zone** - an availability zone is an isolated location within a geographic region and is a physical data center within a specific region 75 | **Edge Location** - an edge location is as a mini-data center used solely to cache large data files closer to a user's location 76 | 77 | _Notes_: 78 | - there are more Availability Zones (AZs) than there are Regions 79 | - there should be at least two AZs per Region 80 | - each region is located in a separate geographic area 81 | - AZs are distinct locations that are engineered to be isolated from failures 82 | 83 | 84 | ## Shared Responsibility Model 85 | 86 | AWS is responsible for security **OF the cloud**, we are responsible for security **IN the cloud** 87 | 88 | _AWS is responsible for_: 89 | - securing edge locations 90 | - monitoring physical device security 91 | - providing physical access control to hardware/software 92 | - database patching 93 | - discarding physical storage devices 94 | 95 | _You are responsible for_: 96 | - managing AWS Identity and Access Management (IAM) 97 | - encrypting data 98 | - preventing or detecting when an AWS account has been compromised 99 | - restricting access to AWS services to only those users who need it 100 | 101 | # Foundational and Compute Service 102 | 103 | ## Elastic Cloud Compute 104 | **Elastic Cloud Compute** or **EC2** is a foundational piece of AWS' cloud computing platform and is a service that provides servers for rent in the cloud 105 | 106 | _Pricing Options_: 107 | 1. On Demand - pay as you go, no contract 108 | 2. Dedicated Hosts - you have your own dedicated hardware and don't share it with others 109 | 3. spot - you place a bid on an instance price, if there is extra capacity that falls below your bid, an EC2 instance is provisioned, if the price goes above your bid while the instance is running, the instance is terminated 110 | 3. Reserved Instances - you earn huge discounts if you pay up front and sign a 1-year or 3-year contract 111 | 112 | 113 | ## Elastic Block Store 114 | **Elastic Block Store (EBS)** is a storage solution for EC2 instances and is a physical hard drive that is attached to the EC2 instance to increase storage 115 | 116 | ## Virtual Private Cloud (VPC) 117 | **Virtual Private Cloud (VPC)** allows you to create your own private network in the cloud. You can launch services, like EC2, inside of that private network. A VPC spans all the Availability Zones in the region 118 | 119 | VPC allows you to control your virtual networking environment, which includes: 120 | - IP address ranges 121 | - subnets 122 | - route tables 123 | - network gateways 124 | 125 | 126 | ## Lambda 127 | **AWS Lambda** provides you with computing power in the cloud by allowing you to execute code without standing up or managing servers. 128 | 129 | - the code you run on AWS Lambda is called a “Lambda function” 130 | - Lambda code can be triggered by other AWS services 131 | - AWS Lambda supports Java, Go, PowerShell, Node.js, C#/.NET, Python, and Ruby 132 | - there is a Runtime API that allows you to use other programming languages to author your functions 133 | - Lambda code can be authored via the console 134 | 135 | 136 | ## Elastic Beanstalk 137 | **Elastic Beanstalk** is an orchestration service that allows you to deploy a web application at the touch of a button by spinning up (or provisioning) all of the services that you need to run your application. Elastic Beanstalk can be used to deployed web applications developed with Java, .NET, PHP, Node.js, Python, Ruby, Go, and Docker. You can run your applications in a VPC. 138 | 139 | # Storage and content delivery 140 | 141 | ## S3 & S3 Glacier 142 | **Amazon Simple Storage Service (S3)** is an object storage system in the cloud. S3 offers several storage classes, which are different data access levels for your data at certain price points 143 | 144 | _Storage Classes_: 145 | - S3 Standard 146 | - S3 Glacier 147 | - S3 Glacier Deep Archive 148 | - S3 Intelligent-Tiering 149 | - S3 Standard Infrequent Access 150 | - S3 One Zone-Infrequent Access 151 | 152 | 153 | ## DynamoDB 154 | **DynamoDB** is a NoSQL document database service that is fully managed. Unlike traditional databases, NoSQL databases, are schema-less. Schema-less simply means that the database doesn't contain a fixed (or rigid) data structure 155 | 156 | - DynamoDB can handle more than 10 trillion requests per day 157 | - DynamoDB is serverless as there are no servers to provision, patch, or manage 158 | - DynamoDB supports key-value and document data models 159 | - DynamoDB synchronously replicates data across three AZs in an AWS Region 160 | - DynamoDB supports GET/PUT operations using a primary key 161 | 162 | 163 | ## Relational Database Service (RDS) 164 | **RDS (Relational Database Service)** is a service that aids in the administration and management of databases. RDS assists with database administrative tasks that include upgrades, patching, installs, backups, monitoring, performance checks, security, etc. 165 | 166 | 167 | ## Redshift 168 | **Redshift** is a cloud data warehousing service to help companies manage big data. Redshift allows you to run fast queries against your data using SQL, ETL, and BI tools. Redshift stores data in a column format to aid in fast querying 169 | 170 | - Redshift delivers great performance by using machine learning 171 | - Redshift Spectrum is a feature that enables you to run queries against data in Amazon S3 172 | - Redshift encrypts and keeps your data secure in transit and at rest 173 | - Redshift clusters can be isolated using Amazon Virtual Private Cloud (VPC) 174 | 175 | # Content Delivery In The Cloud 176 | A **Content Delivery Network (CDN)** speeds up delivery of your static and dynamic web content by caching content in an Edge Location close to your user base. 177 | 178 | 179 | ## Cloud Front 180 | **CloudFront** is used as a global content delivery network (CDN). Cloud Front speeds up the delivery of your content through Amazon's worldwide network of mini-data centers called Edge Locations 181 | 182 | - amazon countinously adds new Edge Locations 183 | - cloudFront ensures that end-user requests are served from the closest edge location 184 | - cloudFront works with non-AWS origin sources 185 | - you can use GeoIP blocking to serve content (or not serve content) to specific countries 186 | - cache control headers determine how frequently CloudFront needs to check the origin for an updated version your file 187 | - the maximum size of a single file that can be delivered through Amazon CloudFront is 20 GB 188 | 189 | # Security In The Cloud 190 | As adoption of cloud services has increased, so has the need for increased security in the cloud. The great thing about cloud security is that it not only protects data, it also protects applications that access the data. Cloud security even protects the infrastructure (like servers) that applications run on 191 | 192 | ## AWS Shield 193 | **AWS Shield** is a managed DDoS (or Distributed Denial of Service) protection service that safeguards web applications running on AWS. AWS Shield is a service that you get "out of the box", it is always running (automatically) and is a part of the free standard tier. If you want to use some of the more advanced features, you'll have to utilize the paid tier 194 | 195 | 196 | ## AWS WAF 197 | **AWS WAF (AWS Web Application Firewall)** provides a firewall that protects your web applications. WAF can stop common web attacks by reviewing the data being sent to your application and stopping well-known attacks 198 | 199 | - WAF can protect web sites not hosted in AWS through Cloud Front. 200 | - you can configure CloudFront to present a custom error page when requests are blocked 201 | 202 | 203 | ## Identity & Access Management 204 | **Identity & Access Management (IAM)** is an AWS service that allows us to configure who can access our AWS account, services, or even applications running in our account. IAM is a global service and is automatically available across ALL regions 205 | 206 | 207 | # Networking 208 | Networks reliably carry loads of data around the globe allowing for the delivery of content and applications with high availability. The network is the foundation of your infrastructure 209 | 210 | Cloud networking includes: 211 | 212 | - network architecture 213 | - network connectivity 214 | - application delivery 215 | - global performance 216 | - delivery 217 | 218 | ## Route 53 219 | **Route 53** is a cloud domain name system (DNS) service that has servers distributed around the globe used to translates human-readable names like www.google.com into the numeric IP addresses like 74.125.21.147 220 | 221 | _Features_: 222 | - scales automatically to manage spikes in DNS queries 223 | - allows you to register a domain name (or manage an existing) 224 | - routes internet traffic to the resources for your domain 225 | - checks the health of your resources 226 | - allows you to route users based on the user’s geographic location. 227 | 228 | 229 | # Elasticity in the Cloud 230 | One of the main benefits of the cloud is that it allows you to stop guessing about capacity when you need to run your applications. Sometimes you buy too much or you don't buy enough to support the running of your applications. With elasticity, your servers, databases, and application resources can automatically scale up or scale down based on load 231 | 232 | ## EC2 Auto Scaling 233 | **EC2 Auto Scaling** is a service that monitors your EC2 instances and automatically adjusts by adding or removing EC2 instances based on conditions you define in order to maintain application availability and provide peak performance to your users 234 | 235 | _Features_: 236 | - automatically scale in and out based on needs 237 | - included automatically with Amazon EC2 238 | - automate how your Amazon EC2 instances are managed 239 | - EC2 Auto Scaling adds instances only when needed, optimizing cost savings 240 | - EC2 predictive scaling removes the need for manual adjustment of auto scaling parameters over time 241 | 242 | 243 | ## Elastic Load Balancing 244 | **Elastic Load Balancing** automatically distributes incoming application traffic across multiple servers 245 | 246 | Elastic Load Balancer is a service that: 247 | - balances load between two or more servers 248 | - stands in front of a web server 249 | - provides redundancy and performance 250 | - elastic Load Balancing works with EC2 Instances, containers, IP addresses, and Lambda functions 251 | - you can configure Amazon EC2 instances to only accept traffic from a load balancer 252 | 253 | # Messaging in the Cloud 254 | There are often times that users of your applications need to be notified when certain events happen. Notifications, such as text messages or emails can be sent through services in the cloud. The use of the cloud offers benefits like lowered costs, increased storage, and flexibility 255 | 256 | 257 | ## Simple Notification Service 258 | **Amazon Simple Notification Service (SNS)** is a cloud service that allows you to send notifications to the users of your applications. SNS allows you to decouple the notification logic from being embedded in your applications and allows notifications to be published to a large number of subscribers 259 | 260 | _Features_: 261 | - SNS uses a publish/subscribe model 262 | - SNS can publish messages to Amazon SQS queues, AWS Lambda functions, and HTTP/S webhooks 263 | - SNS Topic names are limited to 256 characters 264 | - a notification can contain only one message 265 | 266 | 267 | ## Queues 268 | A *queue* is a data structure that holds requests called messages. Messages in a queue are commonly processed in order, first in, first out (or FIFO). Messaging queues improve performance, scalability and user experience 269 | 270 | ## Simple Queue Service 271 | **Amazon Simple Queue Service (SQS)** is a fully managed message queuing service that allows you to integrate queuing functionality in your application. SQS offers two types of message queues: standard and FIFO 272 | 273 | _Features_: 274 | - FIFO queues support up to 300 messages per second 275 | - FIFO queues guarantee the ordering of messages 276 | - standard queues offer best-effort ordering but no guarantees 277 | - standard queues deliver a message at least once, but occasionally more than one copy of a message is delivered 278 | 279 | # Containers in the Cloud 280 | Enterprises are adopting container technology at an explosive rate. A container consists of everything an application needs to run: the application itself and its dependencies (e.g. libraries, utilities, configuration files), all bundled into one package. Each container is an independent component that can run on its own and be moved from environment to environment 281 | 282 | 283 | ## Elastic Container Service (ECS) 284 | **ECS** is an orchestration service used for automating deployment, scaling, and managing of your containerized applications. ECS works well with Docker containers by: 285 | - launching and stopping Docker containers 286 | - scaling your applications 287 | - querying the state of your applications 288 | 289 | _Features_: 290 | - you can schedule long-running applications, services, and batch processeses using ECS 291 | - docker is the only container platform supported by Amazon ECS 292 | 293 | # AWS Management 294 | 295 | ## Infrastructure as Code 296 | **Infrastructure as Code** allows you to describe and provision all the infrastructure resources in your cloud environment. You can stand up servers, databases, runtime parameters, resources, etc. based on scripts that you write. Infrastructure as Code is a time-saving feature because it allows you to provision (or stand up) resources in a reproducible way 297 | 298 | ## Cloud Formation 299 | **AWS Cloud Formation** allows you to model your entire infrastructure in a text file template allowing you to provision AWS resources based on the scripts you write 300 | 301 | ## AWS Command Line Interface (CLI) 302 | The **AWS CLI (Command Line Interface)** allows you to access and control services running in your AWS account from the command line. To use the CLI, simply download, install, and configure it -------------------------------------------------------------------------------- /cloud/version.md: -------------------------------------------------------------------------------- 1 | # Version control 2 | 3 | ## Terminology 4 | 5 | **Version Control System (VCS)** / **Source Code Manager (SCM)** - a tool that manages different versions of source code 6 | 7 | **Commit** - a snapshot of the state of a project (like a save point in a game) 8 | 9 | **Repository** - a directory which contains your project work (made up of commits) 10 | 11 | **Checkout** - when content in the repository has been copied to the Working Directory 12 | 13 | **Staging Area** / **Staging Index** / **Index** - a file in the Git directory that stores information about what will go into your next commit 14 | 15 | **SHA (Secure Hash Algorithm)** - an ID number for each commit 16 | 17 | **Branch** - when a new line of development is created that diverges from the main line of development (can continue without altering the main line) 18 | 19 | **Forking** - an action that's done on a hosting service, like GitHub. Forking a repository creates an identical copy of the original repository and moves this copy to your account. You have total control over this forked repository. Modifying your forked repository does not alter the original repository in any way 20 | 21 | **Pull request** - a request for the source repository to pull in your commits and merge them with their project 22 | 23 | 24 | 25 | ## Initializing 26 | 27 | **git init** - sets up all of the necessary files and directories in a hidden directory called _.git_. The content of the _.git_ directory: 28 | 29 | - _config file_ - where all project specific configuration settings are stored 30 | 31 | - _description file_ - this file is only used by the GitWeb program, so we can ignore it 32 | 33 | - _hooks directory_ - this is where we could place client-side or server-side scripts that we can use to hook into Git's different lifecycle events 34 | 35 | - _info directory_ - contains the global excludes file 36 | 37 | - _objects directory_ - this directory will store all of the commits we make 38 | 39 | - _refs directory_ - this directory holds pointers to commits (basically the "branches" and "tags") 40 | 41 | 42 | **git clone path new_repo** - clone an existing repo (path) into new repo 43 | 44 | 45 | ## Logging 46 | 47 | **git status** - shows the state of our repository as Git sees it 48 | 49 | **git log** - displays the following info about each commit: the SHA, the author, the date, the message 50 | 51 | **git log --oneline** - a more succinct version of _git log_ 52 | 53 | **git log --stat** - displays the files that have been changed in the commit: the file(s) that have been modified, the number of lines that have been added/removed, a summary line with the total number of modified files and lines that have been added/removed 54 | 55 | **git log -p** / **git log --patch** - displays the actual changes made to a file, different flags can be used (for example, _-w_ or _--ignore-all-space_ ignores whitespace when comparing lines) 56 | 57 | **git log SHA** - by supplying a SHA, the _git log_ command will start at that commit 58 | 59 | **git show** - displays the most recent commit 60 | 61 | **git show SHA** - displays only one commit with the specified SHA, the output is exactly the same as the _git log -p SHA_: the commit, the author, the date, the commit message, the patch information. The command can be combined with the flags: _--stat_, _--patch_, _--ignore-all-space_ 62 | 63 | **git log --oneline --graph --all** - displays all branches at once in the git log output, the _--graph_ flag adds the bullets and lines to the leftmost part of the output, the _--all_ flag is what displays all of the branches in the repository 64 | 65 | **git shortlog** - groups commits by author and title 66 | 67 | **git log --author="Richard Kalehoff"** - filters commits with the _--author_ flag 68 | 69 | **git log --grep="border radius issue in Safari"** - filters commits using the _--grep_ flag 70 | 71 | 72 | ## Making commits 73 | 74 | **git add … ** - moves files from the Working Directory to the Staging Index 75 | 76 | **git commit -m message** - commits the changes to the repo (can be used without _-m_ flag with a code editor). The goal is that each commit has a single focus. Each commit should record a single-unit change 77 | 78 | **git diff** - displays the changes that have been made but haven't been committed: the files that have been modified, the location of the lines that have been added/removed, the actual changes that have been made 79 | 80 | **git tag -a tag_name** - adds a marker on a _most recent_ commit, the tag does not move around as new commits are added, the _-a_ flas creates an annotated flag (instead of a lightweight one), which includes the following info: the person who made the tag, the date the tag was made, a message for the tag 81 | 82 | **git tag -a tag_name SHA** - adds a marker on a _specific_ commit 83 | 84 | **git tag -d tag_name** - deletes the tag with the name _tag_name_ 85 | 86 | ## Resetting changes 87 | 88 | **git reset ** - erases commits, the exact behavior is different and is based on the flag used, possible options: 89 | _--mixed_ (default) moves the commit to the working directory, _--soft_ moves the commit to the staging index, _--hard_ erases the commit completely 90 | 91 | ## Branching 92 | 93 | **git branch** - can be used to list all branch names in the repository, create new branches, delete branches 94 | 95 | **git branch branch_name** - creates a branch _branch_name_ 96 | 97 | **git checkout branch_name** - switches to the branch _branch_name_ 98 | 99 | **git branch -d branch_name** - deletes the branch _branch_name_ (the deletion can be forced with _-D_) 100 | 101 | **git checkout -b branch_name** - creates a branch and switches to it all in one command 102 | 103 | 104 | ## Merging 105 | 106 | - _merging_ is combining branches together 107 | - making a merge makes a commit! 108 | - two main types of merges in Git: a regular merge and a fast-forward merge 109 | - a _fast-forward merge_ happens when one branch is ahead of another, so the merge will just move the currently checked out branch forward until it points to the same commit that the other branch, it is one of the easiest merges to do 110 | 111 | **git merge branch_name** - merges _branch_name_ into the current (checked-out) branch, the command finds a single commit that both branches have in their commit history, combines the lines of code that were changed on the separate branches, makes a commit to record the merge, a commit message needs to be supplied (!) 112 | 113 | **git reset --hard HEAD^** - the command to undo the merge 114 | 115 | 116 | ## Merge Conflicts 117 | 118 | ### Indicators Explanation 119 | 120 | ``` 121 | <<<<<<< HEAD everything below this line (until the next indicator) shows you what's on the current branch 122 | ||||||| merged common ancestors everything below this line (until the next indicator) shows you what the original lines were 123 | ======= is the end of the original lines, everything that follows (until the next indicator) is what's on the branch that's being merged in 124 | >>>>>>> heading-update is the ending indicator of what's on the branch that's being merged in (in this case, the heading-update branch) 125 | ``` 126 | 127 | ### Resolving A Merge Conflict 128 | 129 | - choose which line(s) to keep 130 | - remove all lines with indicators 131 | - save the file(s) 132 | - stage the file(s) 133 | - make a commit 134 | 135 | 136 | ## Making changes 137 | 138 | **git commit --amend** - alters the most-recent commit (change the message, add forgotten files), in order to add new files, those files should be stages as for a regular commit before running the command 139 | 140 | **git revert SHA** - reverts a specific commit (does the exact opposite of that commit), creates a new commit with the opposite action. Resetting is different from reverting, it erases commits from the repo history completely (!) 141 | 142 | 143 | ## Relative Commit References (Ancestry References) 144 | 145 | **^** – indicates the parent commit 146 | 147 | **~** – indicates the first parent commit 148 | 149 | The main difference between the ^ and the ~ is when a commit is created from a merge. A merge commit has two parents. With a merge commit, the ^ reference is used to indicate the first parent of the commit while ^2 indicates the second parent. The first parent is the branch you were on when you ran git merge while the second parent is the branch that was merged in. 150 | 151 | 152 | Examples: 153 | 154 | - The following indicates the parent commit of the current commit: _HEAD^_, _HEAD\~_, _HEAD\~1_ 155 | 156 | - The following indicates the grandparent commit of the current commit: _HEAD^^_, _HEAD\~2_ 157 | 158 | - The following indicates the great-grandparent commit of the current commit: _HEAD^^^_, _HEAD\~3_ 159 | 160 | 161 | ## Commit messages 162 | 163 | - do keep the message short (less than 60-ish characters) 164 | - do explain what the commit does (not how or why!) 165 | 166 | - do not explain why the changes are made (more on this below) 167 | - do not explain how the changes are made (that's what git log -p is for!) 168 | - do not use the word "and" 169 | 170 | ## Working with remotes 171 | 172 | **git remote -v** - shows the details about a connection to a remote 173 | 174 | **git remote add shortname url** - adds a connection to a new remote repository 175 | 176 | **git remote rename old_name new_name** - renames remotes 177 | 178 | _Note_: **origin** is a common shortname for remote repositories. 179 | 180 | **git push remote_repo branch_name** - sends commits from a local repository to a remote repository 181 | 182 | **git pull remote_repo branch_name** - pulls commits from a remote repository to a local repository, the command does two things: (1) fetching remote changes (which adds the commits to the local repository and moves the tracking branch to point to them), (2) merging the local branch (master) with the tracking branch (origin/master) 183 | 184 | **git fetch remote_repo branch_name** - retrieves the commits and moves the tracking branch 185 | 186 | 187 | ## Contributing to open source 188 | 189 | 190 | ### General guidelines: 191 | 192 | - before you start doing any work, make sure to look for the project's _CONTRIBUTING.md_ file 193 | - it's a good idea to look at the GitHub issues for the project, if necessary create a new issue 194 | - communicate the changes you'd like to make to the project maintainer in the issue 195 | - commit all of your work on a topic branch: do not work on the master branch and make sure to give the topic branch clear, descriptive name 196 | 197 | 198 | ### Creating a pull request: 199 | 200 | 1. you must fork the source repository 201 | 2. clone your fork down to your machine 202 | 3. make some commits (ideally on a topic branch!) 203 | 4. push the commits back to your fork 204 | 5. create a new pull request and choose the branch that has your new commits 205 | 206 | 207 | ### Staying in sync with a source repo: 208 | 209 | 1. get the cloneable URL of the source repository 210 | 2. create a new remote with the git remote add command 211 | 3. use the shortname upstream to point to the source repository 212 | 4. provide the URL of the source repository 213 | 5. fetch the new upstream remote 214 | 6. merge the upstream's branch into a local branch 215 | 7. push the newly updated local branch to your origin repo 216 | 217 | 218 | ## Rebasing 219 | 220 | **git rebase -i ** - interactive rebase 221 | 222 | The **git rebase** command is used to do a great many things. It can help you edit commit messages, reorder commits, combine commits, etc. Whenever you rebase commits, Git will create a new SHA for each commit. Inside the interactive list of commits, all commits start out as pick, but you can swap that out with one of the other commands (reword, edit, squash, fixup, exec, and drop). The best practice is to create a backup branch before rebasing, so that it's easy to return to your previous state. 223 | 224 | You should not rebase if you have already pushed the commits you want to rebase. If you're collaborating with other developers, then they might already be working with the commits you've pushed. 225 | 226 | 227 | ## Globbing and gitignore 228 | 229 | - the *.gitignore* file is used to tell Git about the files that Git should not track, this file should be placed in the same directory that the .git directory is in 230 | 231 | - **globbing** lets you use special characters to match patterns/characters, it is similar to regular expressions 232 | -------------------------------------------------------------------------------- /pytorch/resources/arch.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/baroquerock/udacity_notes/efd4c665916ba4953d6378e69331b6cb450a385c/pytorch/resources/arch.jpeg -------------------------------------------------------------------------------- /pytorch/resources/grad.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/baroquerock/udacity_notes/efd4c665916ba4953d6378e69331b6cb450a385c/pytorch/resources/grad.jpeg -------------------------------------------------------------------------------- /pytorch/resources/graph_backprop.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/baroquerock/udacity_notes/efd4c665916ba4953d6378e69331b6cb450a385c/pytorch/resources/graph_backprop.png -------------------------------------------------------------------------------- /pytorch/resources/rnn_components.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/baroquerock/udacity_notes/efd4c665916ba4953d6378e69331b6cb450a385c/pytorch/resources/rnn_components.png -------------------------------------------------------------------------------- /pytorch/resources/rnn_schema.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/baroquerock/udacity_notes/efd4c665916ba4953d6378e69331b6cb450a385c/pytorch/resources/rnn_schema.png -------------------------------------------------------------------------------- /pytorch/resources/rules.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/baroquerock/udacity_notes/efd4c665916ba4953d6378e69331b6cb450a385c/pytorch/resources/rules.png -------------------------------------------------------------------------------- /pytorch/resources/shakespeare.txt: -------------------------------------------------------------------------------- 1 | From fairest creatures we desire increase, 2 | That thereby beauty's rose might never die, 3 | But as the riper should by time decease, 4 | His tender heir might bear his memory: 5 | But thou contracted to thine own bright eyes, 6 | Feed'st thy light's flame with self-substantial fuel, 7 | Making a famine where abundance lies, 8 | Thy self thy foe, to thy sweet self too cruel: 9 | Thou that art now the world's fresh ornament, 10 | And only herald to the gaudy spring, 11 | Within thine own bud buriest thy content, 12 | And tender churl mak'st waste in niggarding: 13 | Pity the world, or else this glutton be, 14 | To eat the world's due, by the grave and thee. 15 | 16 | When forty winters shall besiege thy brow, 17 | And dig deep trenches in thy beauty's field, 18 | Thy youth's proud livery so gazed on now, 19 | Will be a tattered weed of small worth held: 20 | Then being asked, where all thy beauty lies, 21 | Where all the treasure of thy lusty days; 22 | To say within thine own deep sunken eyes, 23 | Were an all-eating shame, and thriftless praise. 24 | How much more praise deserved thy beauty's use, 25 | If thou couldst answer 'This fair child of mine 26 | Shall sum my count, and make my old excuse' 27 | Proving his beauty by succession thine. 28 | This were to be new made when thou art old, 29 | And see thy blood warm when thou feel'st it cold. 30 | 31 | Look in thy glass and tell the face thou viewest, 32 | Now is the time that face should form another, 33 | Whose fresh repair if now thou not renewest, 34 | Thou dost beguile the world, unbless some mother. 35 | For where is she so fair whose uneared womb 36 | Disdains the tillage of thy husbandry? 37 | Or who is he so fond will be the tomb, 38 | Of his self-love to stop posterity? 39 | Thou art thy mother's glass and she in thee 40 | Calls back the lovely April of her prime, 41 | So thou through windows of thine age shalt see, 42 | Despite of wrinkles this thy golden time. 43 | But if thou live remembered not to be, 44 | Die single and thine image dies with thee. 45 | 46 | Unthrifty loveliness why dost thou spend, 47 | Upon thy self thy beauty's legacy? 48 | Nature's bequest gives nothing but doth lend, 49 | And being frank she lends to those are free: 50 | Then beauteous niggard why dost thou abuse, 51 | The bounteous largess given thee to give? 52 | Profitless usurer why dost thou use 53 | So great a sum of sums yet canst not live? 54 | For having traffic with thy self alone, 55 | Thou of thy self thy sweet self dost deceive, 56 | Then how when nature calls thee to be gone, 57 | What acceptable audit canst thou leave? 58 | Thy unused beauty must be tombed with thee, 59 | Which used lives th' executor to be. 60 | 61 | Those hours that with gentle work did frame 62 | The lovely gaze where every eye doth dwell 63 | Will play the tyrants to the very same, 64 | And that unfair which fairly doth excel: 65 | For never-resting time leads summer on 66 | To hideous winter and confounds him there, 67 | Sap checked with frost and lusty leaves quite gone, 68 | Beauty o'er-snowed and bareness every where: 69 | Then were not summer's distillation left 70 | A liquid prisoner pent in walls of glass, 71 | Beauty's effect with beauty were bereft, 72 | Nor it nor no remembrance what it was. 73 | But flowers distilled though they with winter meet, 74 | Leese but their show, their substance still lives sweet. 75 | 76 | Then let not winter's ragged hand deface, 77 | In thee thy summer ere thou be distilled: 78 | Make sweet some vial; treasure thou some place, 79 | With beauty's treasure ere it be self-killed: 80 | That use is not forbidden usury, 81 | Which happies those that pay the willing loan; 82 | That's for thy self to breed another thee, 83 | Or ten times happier be it ten for one, 84 | Ten times thy self were happier than thou art, 85 | If ten of thine ten times refigured thee: 86 | Then what could death do if thou shouldst depart, 87 | Leaving thee living in posterity? 88 | Be not self-willed for thou art much too fair, 89 | To be death's conquest and make worms thine heir. 90 | 91 | Lo in the orient when the gracious light 92 | Lifts up his burning head, each under eye 93 | Doth homage to his new-appearing sight, 94 | Serving with looks his sacred majesty, 95 | And having climbed the steep-up heavenly hill, 96 | Resembling strong youth in his middle age, 97 | Yet mortal looks adore his beauty still, 98 | Attending on his golden pilgrimage: 99 | But when from highmost pitch with weary car, 100 | Like feeble age he reeleth from the day, 101 | The eyes (fore duteous) now converted are 102 | From his low tract and look another way: 103 | So thou, thy self out-going in thy noon: 104 | Unlooked on diest unless thou get a son. 105 | 106 | Music to hear, why hear'st thou music sadly? 107 | Sweets with sweets war not, joy delights in joy: 108 | Why lov'st thou that which thou receiv'st not gladly, 109 | Or else receiv'st with pleasure thine annoy? 110 | If the true concord of well-tuned sounds, 111 | By unions married do offend thine ear, 112 | They do but sweetly chide thee, who confounds 113 | In singleness the parts that thou shouldst bear: 114 | Mark how one string sweet husband to another, 115 | Strikes each in each by mutual ordering; 116 | Resembling sire, and child, and happy mother, 117 | Who all in one, one pleasing note do sing: 118 | Whose speechless song being many, seeming one, 119 | Sings this to thee, 'Thou single wilt prove none'. 120 | 121 | Is it for fear to wet a widow's eye, 122 | That thou consum'st thy self in single life? 123 | Ah, if thou issueless shalt hap to die, 124 | The world will wail thee like a makeless wife, 125 | The world will be thy widow and still weep, 126 | That thou no form of thee hast left behind, 127 | When every private widow well may keep, 128 | By children's eyes, her husband's shape in mind: 129 | Look what an unthrift in the world doth spend 130 | Shifts but his place, for still the world enjoys it; 131 | But beauty's waste hath in the world an end, 132 | And kept unused the user so destroys it: 133 | No love toward others in that bosom sits 134 | That on himself such murd'rous shame commits. 135 | 136 | For shame deny that thou bear'st love to any 137 | Who for thy self art so unprovident. 138 | Grant if thou wilt, thou art beloved of many, 139 | But that thou none lov'st is most evident: 140 | For thou art so possessed with murd'rous hate, 141 | That 'gainst thy self thou stick'st not to conspire, 142 | Seeking that beauteous roof to ruinate 143 | Which to repair should be thy chief desire: 144 | O change thy thought, that I may change my mind, 145 | Shall hate be fairer lodged than gentle love? 146 | Be as thy presence is gracious and kind, 147 | Or to thy self at least kind-hearted prove, 148 | Make thee another self for love of me, 149 | That beauty still may live in thine or thee. 150 | 151 | As fast as thou shalt wane so fast thou grow'st, 152 | In one of thine, from that which thou departest, 153 | And that fresh blood which youngly thou bestow'st, 154 | Thou mayst call thine, when thou from youth convertest, 155 | Herein lives wisdom, beauty, and increase, 156 | Without this folly, age, and cold decay, 157 | If all were minded so, the times should cease, 158 | And threescore year would make the world away: 159 | Let those whom nature hath not made for store, 160 | Harsh, featureless, and rude, barrenly perish: 161 | Look whom she best endowed, she gave thee more; 162 | Which bounteous gift thou shouldst in bounty cherish: 163 | She carved thee for her seal, and meant thereby, 164 | Thou shouldst print more, not let that copy die. 165 | 166 | When I do count the clock that tells the time, 167 | And see the brave day sunk in hideous night, 168 | When I behold the violet past prime, 169 | And sable curls all silvered o'er with white: 170 | When lofty trees I see barren of leaves, 171 | Which erst from heat did canopy the herd 172 | And summer's green all girded up in sheaves 173 | Borne on the bier with white and bristly beard: 174 | Then of thy beauty do I question make 175 | That thou among the wastes of time must go, 176 | Since sweets and beauties do themselves forsake, 177 | And die as fast as they see others grow, 178 | And nothing 'gainst Time's scythe can make defence 179 | Save breed to brave him, when he takes thee hence. 180 | 181 | O that you were your self, but love you are 182 | No longer yours, than you your self here live, 183 | Against this coming end you should prepare, 184 | And your sweet semblance to some other give. 185 | So should that beauty which you hold in lease 186 | Find no determination, then you were 187 | Your self again after your self's decease, 188 | When your sweet issue your sweet form should bear. 189 | Who lets so fair a house fall to decay, 190 | Which husbandry in honour might uphold, 191 | Against the stormy gusts of winter's day 192 | And barren rage of death's eternal cold? 193 | O none but unthrifts, dear my love you know, 194 | You had a father, let your son say so. 195 | 196 | Not from the stars do I my judgement pluck, 197 | And yet methinks I have astronomy, 198 | But not to tell of good, or evil luck, 199 | Of plagues, of dearths, or seasons' quality, 200 | Nor can I fortune to brief minutes tell; 201 | Pointing to each his thunder, rain and wind, 202 | Or say with princes if it shall go well 203 | By oft predict that I in heaven find. 204 | But from thine eyes my knowledge I derive, 205 | And constant stars in them I read such art 206 | As truth and beauty shall together thrive 207 | If from thy self, to store thou wouldst convert: 208 | Or else of thee this I prognosticate, 209 | Thy end is truth's and beauty's doom and date. 210 | 211 | When I consider every thing that grows 212 | Holds in perfection but a little moment. 213 | That this huge stage presenteth nought but shows 214 | Whereon the stars in secret influence comment. 215 | When I perceive that men as plants increase, 216 | Cheered and checked even by the self-same sky: 217 | Vaunt in their youthful sap, at height decrease, 218 | And wear their brave state out of memory. 219 | Then the conceit of this inconstant stay, 220 | Sets you most rich in youth before my sight, 221 | Where wasteful time debateth with decay 222 | To change your day of youth to sullied night, 223 | And all in war with Time for love of you, 224 | As he takes from you, I engraft you new. 225 | 226 | But wherefore do not you a mightier way 227 | Make war upon this bloody tyrant Time? 228 | And fortify your self in your decay 229 | With means more blessed than my barren rhyme? 230 | Now stand you on the top of happy hours, 231 | And many maiden gardens yet unset, 232 | With virtuous wish would bear you living flowers, 233 | Much liker than your painted counterfeit: 234 | So should the lines of life that life repair 235 | Which this (Time's pencil) or my pupil pen 236 | Neither in inward worth nor outward fair 237 | Can make you live your self in eyes of men. 238 | To give away your self, keeps your self still, 239 | And you must live drawn by your own sweet skill. -------------------------------------------------------------------------------- /pytorch/udacity_cnn_tatiana_gaponova.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/baroquerock/udacity_notes/efd4c665916ba4953d6378e69331b6cb450a385c/pytorch/udacity_cnn_tatiana_gaponova.pdf -------------------------------------------------------------------------------- /pytorch/udacity_nn_basics_tatiana_gaponova.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/baroquerock/udacity_notes/efd4c665916ba4953d6378e69331b6cb450a385c/pytorch/udacity_nn_basics_tatiana_gaponova.pdf -------------------------------------------------------------------------------- /pytorch/udacity_style_transfer_tatiana_gaponova.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/baroquerock/udacity_notes/efd4c665916ba4953d6378e69331b6cb450a385c/pytorch/udacity_style_transfer_tatiana_gaponova.pdf -------------------------------------------------------------------------------- /pytorch/vanillaRNN.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Forward pass for Vanilla RNN\n", 8 | "Taken from here: https://gist.github.com/karpathy/d4dee566867f8291f086" 9 | ] 10 | }, 11 | { 12 | "cell_type": "code", 13 | "execution_count": 1, 14 | "metadata": {}, 15 | "outputs": [], 16 | "source": [ 17 | "import numpy as np" 18 | ] 19 | }, 20 | { 21 | "cell_type": "markdown", 22 | "metadata": {}, 23 | "source": [ 24 | "### Reading the data" 25 | ] 26 | }, 27 | { 28 | "cell_type": "code", 29 | "execution_count": 5, 30 | "metadata": {}, 31 | "outputs": [], 32 | "source": [ 33 | "data = open('resources/shakespeare.txt', 'r').read() " 34 | ] 35 | }, 36 | { 37 | "cell_type": "code", 38 | "execution_count": 6, 39 | "metadata": {}, 40 | "outputs": [ 41 | { 42 | "name": "stdout", 43 | "output_type": "stream", 44 | "text": [ 45 | "From fairest creatures we desire increase,\n", 46 | "That thereby beauty's rose might never die,\n", 47 | "But as the riper should by time decease,\n", 48 | "His tender heir might bear his memory:\n", 49 | "But thou contracted to thine own bright eyes,\n", 50 | "Feed'st thy light's flame with self-substantial fuel,\n", 51 | "Making a famine where abundance lies,\n", 52 | "Thy self thy foe, to thy sweet self too cruel:\n", 53 | "Thou that art now the world's fresh ornament,\n", 54 | "And only herald to the gaudy spring,\n", 55 | "Within thine own bud buriest thy content,\n", 56 | "And tender churl mak'st waste in niggarding:\n", 57 | "Pity the world, or else this glutton be,\n", 58 | "To eat the world's due, by the grave and thee.\n", 59 | "\n" 60 | ] 61 | } 62 | ], 63 | "source": [ 64 | "print(data[:610])" 65 | ] 66 | }, 67 | { 68 | "cell_type": "markdown", 69 | "metadata": {}, 70 | "source": [ 71 | "### Encoding characters" 72 | ] 73 | }, 74 | { 75 | "cell_type": "code", 76 | "execution_count": 7, 77 | "metadata": {}, 78 | "outputs": [ 79 | { 80 | "name": "stdout", 81 | "output_type": "stream", 82 | "text": [ 83 | "data has 9837 characters, 57 unique\n" 84 | ] 85 | } 86 | ], 87 | "source": [ 88 | "chars = list(set(data))\n", 89 | "data_size, vocab_size = len(data), len(chars)\n", 90 | "# char-to-int encoding\n", 91 | "char_to_ix = {ch:i for i,ch in enumerate(chars)}\n", 92 | "ix_to_char = {i:ch for i,ch in enumerate(chars)}\n", 93 | "print('data has {} characters, {} unique'.format(data_size, vocab_size))" 94 | ] 95 | }, 96 | { 97 | "cell_type": "raw", 98 | "metadata": {}, 99 | "source": [] 100 | }, 101 | { 102 | "cell_type": "raw", 103 | "metadata": {}, 104 | "source": [] 105 | }, 106 | { 107 | "cell_type": "markdown", 108 | "metadata": {}, 109 | "source": [ 110 | "### RNN basics" 111 | ] 112 | }, 113 | { 114 | "cell_type": "markdown", 115 | "metadata": {}, 116 | "source": [ 117 | "#### RNN Structure\n", 118 | "\"Drawing\"" 119 | ] 120 | }, 121 | { 122 | "cell_type": "markdown", 123 | "metadata": {}, 124 | "source": [ 125 | "#### Components of RNN\n", 126 | "Taken from here: https://cs224d.stanford.edu/lecture_notes/LectureNotes4.pdf\n", 127 | "\n", 128 | "\"Drawing\"" 129 | ] 130 | }, 131 | { 132 | "cell_type": "raw", 133 | "metadata": {}, 134 | "source": [] 135 | }, 136 | { 137 | "cell_type": "markdown", 138 | "metadata": {}, 139 | "source": [ 140 | "Let's say that we defined the hyperparameters for the RNN layer as follows:
\n", 141 | "\n", 142 | "**hidden size** = 100
\n", 143 | "**sequence length** = 25\n", 144 | "\n", 145 | "\n", 146 | "So, what is the hidden size? \n", 147 | "\n", 148 | "In the example below we set the sequence length to 25. When we unroll the RNN layer, there will be 25 hidden states $h_{t}$.
Each hidden state will have the specified size of 100. So hidden size is just the size of every $h_{t}$. \n", 149 | "\n", 150 | "The output $h_{t}$ depends on two inputs: \n", 151 | "(1) previous hidden state $h_{t-1}$ \n", 152 | "(2) the current element of the sequence $x_t$\n", 153 | "\n", 154 | "Regular neural networks use only one element $x$ as an input,
\n", 155 | "so there was only one matrix of parameters $W$. \n", 156 | "\n", 157 | "But in the case of RNN the output depends on TWO inputs, so we should have TWO parameter matrices instead of just one:\n", 158 | "\n", 159 | "$$h_{t} = tanh(W^{hh}*h_{t-1} + W^{hx}*x_{t} + bias)$$\n", 160 | "\n", 161 | "\n", 162 | "$W^{hx}$ is associated with the inputs.\n", 163 | "\n", 164 | "$W^{hh}$ is associated with the previous hidden state.

\n", 165 | "\n", 166 | "\n", 167 | "\n", 168 | "$h_{t}$ should have dimensions of $(hidden\\_size, 1)$, it is $(100, 1)$ in our case.
\n", 169 | "In order for this to happen, both terms $W^{hh}*h_{t-1}$ and $W^{hx}*x_{t}$ should have the same dimension of $(100, 1)$.
\n", 170 | "\n", 171 | "We already know that $h_{t-1}$ is $(100, 1)$, then $W^{hh}$ should have dimensions of $(100, 100)$, because:\n", 172 | "
\n", 173 | "$$(100, 100) * (100, 1) = (100, 1)$$\n", 174 | "
\n", 175 | "The $W^{hx}$ should have dimensions of $(100, vocab\\_size)$, because:\n", 176 | "
\n", 177 | "$$(100, vocab\\_size) * (vocab\\_size, 1) = (100, 1)$$" 178 | ] 179 | }, 180 | { 181 | "cell_type": "raw", 182 | "metadata": {}, 183 | "source": [] 184 | }, 185 | { 186 | "cell_type": "markdown", 187 | "metadata": {}, 188 | "source": [ 189 | "### Defining the hyperparameters" 190 | ] 191 | }, 192 | { 193 | "cell_type": "code", 194 | "execution_count": 8, 195 | "metadata": {}, 196 | "outputs": [], 197 | "source": [ 198 | "# hyperparameters\n", 199 | "hidden_size = 100 # size of hidden layer of neurons\n", 200 | "seq_length = 25 # number of steps to unroll the RNN for\n", 201 | "learning_rate = 1e-1" 202 | ] 203 | }, 204 | { 205 | "cell_type": "markdown", 206 | "metadata": {}, 207 | "source": [ 208 | "### Initializing the weights" 209 | ] 210 | }, 211 | { 212 | "cell_type": "code", 213 | "execution_count": 9, 214 | "metadata": {}, 215 | "outputs": [], 216 | "source": [ 217 | "# model parameters\n", 218 | "Wxh = np.random.randn(hidden_size, vocab_size)*0.01 # input to hidden\n", 219 | "Whh = np.random.randn(hidden_size, hidden_size)*0.01 # hidden to hidden\n", 220 | "Why = np.random.randn(vocab_size, hidden_size)*0.01 # hidden to output\n", 221 | "bh = np.zeros((hidden_size, 1)) # hidden bias\n", 222 | "by = np.zeros((vocab_size, 1)) # output bias" 223 | ] 224 | }, 225 | { 226 | "cell_type": "code", 227 | "execution_count": 10, 228 | "metadata": {}, 229 | "outputs": [ 230 | { 231 | "name": "stdout", 232 | "output_type": "stream", 233 | "text": [ 234 | "Input to hidden weights: (100, 57)\n", 235 | "Hidden to hidden weights: (100, 100)\n", 236 | "Hidden to output weights: (57, 100)\n", 237 | "\n", 238 | "Hidden bias weights: (100, 1)\n", 239 | "Output bias weights: (57, 1)\n" 240 | ] 241 | } 242 | ], 243 | "source": [ 244 | "print('Input to hidden weights: {}'.format(Wxh.shape))\n", 245 | "print('Hidden to hidden weights: {}'.format(Whh.shape))\n", 246 | "print('Hidden to output weights: {}\\n'.format(Why.shape))\n", 247 | "print('Hidden bias weights: {}'.format(bh.shape))\n", 248 | "print('Output bias weights: {}'.format(by.shape))" 249 | ] 250 | }, 251 | { 252 | "cell_type": "markdown", 253 | "metadata": {}, 254 | "source": [ 255 | "### Extracting one sequence\n", 256 | "For the purpose of demonstration, I am using only one sequence of length 25 in the forward pass. \n", 257 | "*inputs* - one sequence of length 25\n", 258 | "*targets* - one sequence of length 25 shifted by 1 over the *inputs*" 259 | ] 260 | }, 261 | { 262 | "cell_type": "code", 263 | "execution_count": 11, 264 | "metadata": {}, 265 | "outputs": [ 266 | { 267 | "name": "stdout", 268 | "output_type": "stream", 269 | "text": [ 270 | "F(45) -> r(41)\n", 271 | "r(41) -> o(14)\n", 272 | "o(14) -> m(24)\n" 273 | ] 274 | } 275 | ], 276 | "source": [ 277 | "p = 0\n", 278 | "inputs = [char_to_ix[ch] for ch in data[p:p+seq_length]]\n", 279 | "targets = [char_to_ix[ch] for ch in data[p+1:p+seq_length+1]]\n", 280 | "\n", 281 | "# word 'from'\n", 282 | "for i in range(3):\n", 283 | " print('{}({}) -> {}({})'.format(ix_to_char[inputs[i]], inputs[i], ix_to_char[targets[i]], targets[i]))" 284 | ] 285 | }, 286 | { 287 | "cell_type": "markdown", 288 | "metadata": {}, 289 | "source": [ 290 | "### RNN Forward pass" 291 | ] 292 | }, 293 | { 294 | "cell_type": "code", 295 | "execution_count": 12, 296 | "metadata": {}, 297 | "outputs": [], 298 | "source": [ 299 | "# defining the previous hidden state\n", 300 | "# it equals to zeros since for the fisrt element there is no preious hidden state\n", 301 | "hprev = np.zeros((hidden_size,1))" 302 | ] 303 | }, 304 | { 305 | "cell_type": "code", 306 | "execution_count": 13, 307 | "metadata": {}, 308 | "outputs": [], 309 | "source": [ 310 | "# xs - contains one-hot encoded character in a sequence for timestep t\n", 311 | "# hs - contains hidden states for timestep t\n", 312 | "# ys - contains output for timestep t \n", 313 | "# ps - contains probabilities for timestep t \n", 314 | "xs, hs, ys, ps = {}, {}, {}, {}" 315 | ] 316 | }, 317 | { 318 | "cell_type": "code", 319 | "execution_count": 14, 320 | "metadata": {}, 321 | "outputs": [], 322 | "source": [ 323 | "hs[-1] = np.copy(hprev)\n", 324 | "loss = 0" 325 | ] 326 | }, 327 | { 328 | "cell_type": "code", 329 | "execution_count": 15, 330 | "metadata": {}, 331 | "outputs": [], 332 | "source": [ 333 | "for t in range(seq_length):\n", 334 | " #print('Timestep: {}'.format(t+1))\n", 335 | " xs[t] = np.zeros((vocab_size,1)) # one-hot\n", 336 | " xs[t][inputs[t]] = 1 # one-hot \n", 337 | " first_term = np.dot(Wxh, xs[t])\n", 338 | " second_term = np.dot(Whh, hs[t-1])\n", 339 | " hs[t] = np.tanh(first_term + second_term + bh)\n", 340 | " ys[t] = np.dot(Why, hs[t]) + by\n", 341 | " ps[t] = np.exp(ys[t]) / np.sum(np.exp(ys[t])) # probabilities for next chars\n", 342 | " loss += -np.log(ps[t][targets[t],0])" 343 | ] 344 | }, 345 | { 346 | "cell_type": "code", 347 | "execution_count": null, 348 | "metadata": {}, 349 | "outputs": [], 350 | "source": [] 351 | } 352 | ], 353 | "metadata": { 354 | "kernelspec": { 355 | "display_name": "Python 3", 356 | "language": "python", 357 | "name": "python3" 358 | }, 359 | "language_info": { 360 | "codemirror_mode": { 361 | "name": "ipython", 362 | "version": 3 363 | }, 364 | "file_extension": ".py", 365 | "mimetype": "text/x-python", 366 | "name": "python", 367 | "nbconvert_exporter": "python", 368 | "pygments_lexer": "ipython3", 369 | "version": "3.7.2" 370 | } 371 | }, 372 | "nbformat": 4, 373 | "nbformat_minor": 2 374 | } 375 | --------------------------------------------------------------------------------