├── README.md
├── cloud
├── bash.md
├── cloud.md
└── version.md
└── pytorch
├── resources
├── arch.jpeg
├── grad.jpeg
├── graph_backprop.png
├── rnn_components.png
├── rnn_schema.png
├── rules.png
└── shakespeare.txt
├── udacity_cnn_tatiana_gaponova.pdf
├── udacity_nn_basics_tatiana_gaponova.pdf
├── udacity_style_transfer_tatiana_gaponova.pdf
└── vanillaRNN.ipynb
/README.md:
--------------------------------------------------------------------------------
1 | # udacity_notes
2 |
3 | These are my notes that were taken during various Udacity scholarship programs. All images (and sometimes text) belong to Udacity.
4 |
5 | ## DevOps Scholarship from Bertelsmann
6 |
7 | 1. [BASH basics](cloud/bash.md)
8 | 2. [Version control](cloud/version.md)
9 | 3. [Cloud](cloud/cloud.md)
10 |
11 | ## PyTorch Scholarship from Facebook
12 |
13 | 1. [Deep Learing basics](pytorch/udacity_nn_basics_tatiana_gaponova.pdf)
14 | 2. [CNN theory](pytorch/udacity_cnn_tatiana_gaponova.pdf)
15 | 3. [Style transfer](pytorch/udacity_style_transfer_tatiana_gaponova.pdf)
16 | 4. [RNN theory](pytorch/vanillaRNN.ipynb)
17 |
--------------------------------------------------------------------------------
/cloud/bash.md:
--------------------------------------------------------------------------------
1 | # BASH basics
2 |
3 |
4 | - __echo X__ - repeat the string X back
5 |
6 | ``` bash
7 | echo hello
8 | hello
9 | ```
10 |
11 |
12 |
13 | - __!!__ - re-run the last command
14 |
15 | ``` bash
16 | !!
17 | echo hello
18 | hello
19 | ```
20 |
21 |
22 |
23 | - __$X__ - reference of the variable X
24 |
25 | ``` bash
26 | x=100
27 | echo $x
28 | 100
29 | ```
30 |
31 |
32 |
33 | - __ls__ - list the contents of the current directory
34 |
35 | __ls -l__ - a more detailed list
36 |
37 | __cd X__ - change directory to X
38 |
39 | __pwd__ - print the current (working) directory
40 |
41 |
42 |
43 | - Shortcuts:
44 |
45 | . - working directory
46 |
47 | .. - parent directory
48 |
49 | ~ - home directory
50 |
51 |
52 |
53 |
54 | - __mkdir X__ - create a new directory X
55 |
56 | __mv X Y__ - more the file X to the directory Y
57 |
58 |
59 |
60 | - __curl 'http://www.google.com'__ - download the page
61 |
62 | __curl -L -o some_file.html 'http://www.google.com'__ - download the page following redirects (-L) and outputing the result to the _some_file.html_ (-o)
63 |
64 |
65 |
66 | - __cat X__ - display the contents of a file to the terminal
67 |
68 | __less X__ - display the contents of a file to the terminal with the possibility to search and scroll
69 |
70 |
71 |
72 | - Managing the __less__ command:
73 |
74 | scroll down - spacebar or down arrow
75 |
76 | scroll up - b key or up arrow
77 |
78 | search - forward slack /
79 |
80 | exit less - q key
81 |
82 |
83 |
84 | - __rm X__ - permanently delete the file X
85 |
86 | __rm -i X__ - permanently delete the file X but double check first
87 |
88 | __rm X Y Z__ permanently delete the files X, Y, Z
89 |
90 | __rmdir X__ - permanently delete the directory X (should be empty)
91 |
92 | __rmdir -r X__ - permanently delete the directory X with all it's content
93 |
94 |
95 |
--------------------------------------------------------------------------------
/cloud/cloud.md:
--------------------------------------------------------------------------------
1 | # Cloud Computing
2 |
3 | **Cloud Computing** is the delivery of IT resources over the Internet, the cloud is like a virtual data center accessible via the Internet
4 |
5 | **Pay as you go** - you pay only for what you use and only when your code runs
6 |
7 | **Autoscaling** - the number of active servers can grow or shrink based on demand
8 |
9 | **Serverless** - allows you to write and deploy code without having to worry about the underlying infrastructure
10 |
11 | _Benefits_:
12 |
13 | - stop guessing about capacity
14 | - avoid huge capital investments up front
15 | - pay for only what you use
16 | - scale globally in minutes
17 | - deliver faster
18 |
19 |
20 | ## Types of Cloud Computing
21 |
22 | **Infrastructure-as-a-Service (IaaS)** - the provider supplies virtual server instances, storage, and mechanisms for you to manage servers
23 |
24 | **Platform-as-a-Service (PaaS)** - a platform of development tools hosted on a provider's infrastructure
25 |
26 | **Software-as-a-Service (SaaS)** - a software application that runs over the Internet and is managed by the service provider
27 |
28 |
29 | ## Cloud Deployment Models
30 |
31 | **Public Cloud** - a public cloud makes resources available over the Internet to the general public
32 |
33 | **Private Cloud** - a private cloud is a proprietary network that supplies services to a limited number of people
34 |
35 | **Hybrid Cloud** - a hybrid model contains a combination of both a public and a private cloud
36 |
37 |
38 | ## Logging In The Cloud
39 |
40 | **Logging** provides visibility into your cloud resources and applications. For applications that run in the cloud, you will need access to logging and auditing services to help you proactively monitor your resources and applications
41 |
42 |
43 | ## Cloud Trail
44 | **Cloud Trail** allows you to audit (or review) everything that occurs in your AWS account. Cloud Trail does this by recording all the AWS API calls occurring in your account and delivering a log file to you
45 |
46 | _Features_:
47 | - who has logged in
48 | - services that were accessed
49 | - actions performed
50 | - parameters for the actions
51 | - responses returned
52 |
53 |
54 | ## Cloud Watch
55 | **Cloud Watch** is a service that monitors resources and applications that run on AWS by collecting data in the form of logs, metrics, and events
56 |
57 | _Features_:
58 | - collect and track metrics
59 | - collect and monitor log files
60 | - set alarms and create triggers to run your AWS resources
61 | - react to changes in your AWS resources
62 |
63 |
64 | ## Infrastructure as Code
65 | **Infrastructure as Code** allows you to describe and provision all the infrastructure resources in your cloud environment. You can stand up servers, databases, runtime parameters, resources, etc. based on scripts that you write. Infrastructure as Code is a time-saving feature because it allows you to provision (or stand up) resources in a reproducible way
66 |
67 |
68 | ## Cloud Formation
69 | **AWS Cloud Formation** allows you to model your entire infrastructure in a text file template allowing you to provision AWS resources based on the scripts you write
70 |
71 |
72 | ## Global Infrastructure
73 | **Region** - a region is considered a geographic location or an area on a map
74 | **Availability Zone** - an availability zone is an isolated location within a geographic region and is a physical data center within a specific region
75 | **Edge Location** - an edge location is as a mini-data center used solely to cache large data files closer to a user's location
76 |
77 | _Notes_:
78 | - there are more Availability Zones (AZs) than there are Regions
79 | - there should be at least two AZs per Region
80 | - each region is located in a separate geographic area
81 | - AZs are distinct locations that are engineered to be isolated from failures
82 |
83 |
84 | ## Shared Responsibility Model
85 |
86 | AWS is responsible for security **OF the cloud**, we are responsible for security **IN the cloud**
87 |
88 | _AWS is responsible for_:
89 | - securing edge locations
90 | - monitoring physical device security
91 | - providing physical access control to hardware/software
92 | - database patching
93 | - discarding physical storage devices
94 |
95 | _You are responsible for_:
96 | - managing AWS Identity and Access Management (IAM)
97 | - encrypting data
98 | - preventing or detecting when an AWS account has been compromised
99 | - restricting access to AWS services to only those users who need it
100 |
101 | # Foundational and Compute Service
102 |
103 | ## Elastic Cloud Compute
104 | **Elastic Cloud Compute** or **EC2** is a foundational piece of AWS' cloud computing platform and is a service that provides servers for rent in the cloud
105 |
106 | _Pricing Options_:
107 | 1. On Demand - pay as you go, no contract
108 | 2. Dedicated Hosts - you have your own dedicated hardware and don't share it with others
109 | 3. spot - you place a bid on an instance price, if there is extra capacity that falls below your bid, an EC2 instance is provisioned, if the price goes above your bid while the instance is running, the instance is terminated
110 | 3. Reserved Instances - you earn huge discounts if you pay up front and sign a 1-year or 3-year contract
111 |
112 |
113 | ## Elastic Block Store
114 | **Elastic Block Store (EBS)** is a storage solution for EC2 instances and is a physical hard drive that is attached to the EC2 instance to increase storage
115 |
116 | ## Virtual Private Cloud (VPC)
117 | **Virtual Private Cloud (VPC)** allows you to create your own private network in the cloud. You can launch services, like EC2, inside of that private network. A VPC spans all the Availability Zones in the region
118 |
119 | VPC allows you to control your virtual networking environment, which includes:
120 | - IP address ranges
121 | - subnets
122 | - route tables
123 | - network gateways
124 |
125 |
126 | ## Lambda
127 | **AWS Lambda** provides you with computing power in the cloud by allowing you to execute code without standing up or managing servers.
128 |
129 | - the code you run on AWS Lambda is called a “Lambda function”
130 | - Lambda code can be triggered by other AWS services
131 | - AWS Lambda supports Java, Go, PowerShell, Node.js, C#/.NET, Python, and Ruby
132 | - there is a Runtime API that allows you to use other programming languages to author your functions
133 | - Lambda code can be authored via the console
134 |
135 |
136 | ## Elastic Beanstalk
137 | **Elastic Beanstalk** is an orchestration service that allows you to deploy a web application at the touch of a button by spinning up (or provisioning) all of the services that you need to run your application. Elastic Beanstalk can be used to deployed web applications developed with Java, .NET, PHP, Node.js, Python, Ruby, Go, and Docker. You can run your applications in a VPC.
138 |
139 | # Storage and content delivery
140 |
141 | ## S3 & S3 Glacier
142 | **Amazon Simple Storage Service (S3)** is an object storage system in the cloud. S3 offers several storage classes, which are different data access levels for your data at certain price points
143 |
144 | _Storage Classes_:
145 | - S3 Standard
146 | - S3 Glacier
147 | - S3 Glacier Deep Archive
148 | - S3 Intelligent-Tiering
149 | - S3 Standard Infrequent Access
150 | - S3 One Zone-Infrequent Access
151 |
152 |
153 | ## DynamoDB
154 | **DynamoDB** is a NoSQL document database service that is fully managed. Unlike traditional databases, NoSQL databases, are schema-less. Schema-less simply means that the database doesn't contain a fixed (or rigid) data structure
155 |
156 | - DynamoDB can handle more than 10 trillion requests per day
157 | - DynamoDB is serverless as there are no servers to provision, patch, or manage
158 | - DynamoDB supports key-value and document data models
159 | - DynamoDB synchronously replicates data across three AZs in an AWS Region
160 | - DynamoDB supports GET/PUT operations using a primary key
161 |
162 |
163 | ## Relational Database Service (RDS)
164 | **RDS (Relational Database Service)** is a service that aids in the administration and management of databases. RDS assists with database administrative tasks that include upgrades, patching, installs, backups, monitoring, performance checks, security, etc.
165 |
166 |
167 | ## Redshift
168 | **Redshift** is a cloud data warehousing service to help companies manage big data. Redshift allows you to run fast queries against your data using SQL, ETL, and BI tools. Redshift stores data in a column format to aid in fast querying
169 |
170 | - Redshift delivers great performance by using machine learning
171 | - Redshift Spectrum is a feature that enables you to run queries against data in Amazon S3
172 | - Redshift encrypts and keeps your data secure in transit and at rest
173 | - Redshift clusters can be isolated using Amazon Virtual Private Cloud (VPC)
174 |
175 | # Content Delivery In The Cloud
176 | A **Content Delivery Network (CDN)** speeds up delivery of your static and dynamic web content by caching content in an Edge Location close to your user base.
177 |
178 |
179 | ## Cloud Front
180 | **CloudFront** is used as a global content delivery network (CDN). Cloud Front speeds up the delivery of your content through Amazon's worldwide network of mini-data centers called Edge Locations
181 |
182 | - amazon countinously adds new Edge Locations
183 | - cloudFront ensures that end-user requests are served from the closest edge location
184 | - cloudFront works with non-AWS origin sources
185 | - you can use GeoIP blocking to serve content (or not serve content) to specific countries
186 | - cache control headers determine how frequently CloudFront needs to check the origin for an updated version your file
187 | - the maximum size of a single file that can be delivered through Amazon CloudFront is 20 GB
188 |
189 | # Security In The Cloud
190 | As adoption of cloud services has increased, so has the need for increased security in the cloud. The great thing about cloud security is that it not only protects data, it also protects applications that access the data. Cloud security even protects the infrastructure (like servers) that applications run on
191 |
192 | ## AWS Shield
193 | **AWS Shield** is a managed DDoS (or Distributed Denial of Service) protection service that safeguards web applications running on AWS. AWS Shield is a service that you get "out of the box", it is always running (automatically) and is a part of the free standard tier. If you want to use some of the more advanced features, you'll have to utilize the paid tier
194 |
195 |
196 | ## AWS WAF
197 | **AWS WAF (AWS Web Application Firewall)** provides a firewall that protects your web applications. WAF can stop common web attacks by reviewing the data being sent to your application and stopping well-known attacks
198 |
199 | - WAF can protect web sites not hosted in AWS through Cloud Front.
200 | - you can configure CloudFront to present a custom error page when requests are blocked
201 |
202 |
203 | ## Identity & Access Management
204 | **Identity & Access Management (IAM)** is an AWS service that allows us to configure who can access our AWS account, services, or even applications running in our account. IAM is a global service and is automatically available across ALL regions
205 |
206 |
207 | # Networking
208 | Networks reliably carry loads of data around the globe allowing for the delivery of content and applications with high availability. The network is the foundation of your infrastructure
209 |
210 | Cloud networking includes:
211 |
212 | - network architecture
213 | - network connectivity
214 | - application delivery
215 | - global performance
216 | - delivery
217 |
218 | ## Route 53
219 | **Route 53** is a cloud domain name system (DNS) service that has servers distributed around the globe used to translates human-readable names like www.google.com into the numeric IP addresses like 74.125.21.147
220 |
221 | _Features_:
222 | - scales automatically to manage spikes in DNS queries
223 | - allows you to register a domain name (or manage an existing)
224 | - routes internet traffic to the resources for your domain
225 | - checks the health of your resources
226 | - allows you to route users based on the user’s geographic location.
227 |
228 |
229 | # Elasticity in the Cloud
230 | One of the main benefits of the cloud is that it allows you to stop guessing about capacity when you need to run your applications. Sometimes you buy too much or you don't buy enough to support the running of your applications. With elasticity, your servers, databases, and application resources can automatically scale up or scale down based on load
231 |
232 | ## EC2 Auto Scaling
233 | **EC2 Auto Scaling** is a service that monitors your EC2 instances and automatically adjusts by adding or removing EC2 instances based on conditions you define in order to maintain application availability and provide peak performance to your users
234 |
235 | _Features_:
236 | - automatically scale in and out based on needs
237 | - included automatically with Amazon EC2
238 | - automate how your Amazon EC2 instances are managed
239 | - EC2 Auto Scaling adds instances only when needed, optimizing cost savings
240 | - EC2 predictive scaling removes the need for manual adjustment of auto scaling parameters over time
241 |
242 |
243 | ## Elastic Load Balancing
244 | **Elastic Load Balancing** automatically distributes incoming application traffic across multiple servers
245 |
246 | Elastic Load Balancer is a service that:
247 | - balances load between two or more servers
248 | - stands in front of a web server
249 | - provides redundancy and performance
250 | - elastic Load Balancing works with EC2 Instances, containers, IP addresses, and Lambda functions
251 | - you can configure Amazon EC2 instances to only accept traffic from a load balancer
252 |
253 | # Messaging in the Cloud
254 | There are often times that users of your applications need to be notified when certain events happen. Notifications, such as text messages or emails can be sent through services in the cloud. The use of the cloud offers benefits like lowered costs, increased storage, and flexibility
255 |
256 |
257 | ## Simple Notification Service
258 | **Amazon Simple Notification Service (SNS)** is a cloud service that allows you to send notifications to the users of your applications. SNS allows you to decouple the notification logic from being embedded in your applications and allows notifications to be published to a large number of subscribers
259 |
260 | _Features_:
261 | - SNS uses a publish/subscribe model
262 | - SNS can publish messages to Amazon SQS queues, AWS Lambda functions, and HTTP/S webhooks
263 | - SNS Topic names are limited to 256 characters
264 | - a notification can contain only one message
265 |
266 |
267 | ## Queues
268 | A *queue* is a data structure that holds requests called messages. Messages in a queue are commonly processed in order, first in, first out (or FIFO). Messaging queues improve performance, scalability and user experience
269 |
270 | ## Simple Queue Service
271 | **Amazon Simple Queue Service (SQS)** is a fully managed message queuing service that allows you to integrate queuing functionality in your application. SQS offers two types of message queues: standard and FIFO
272 |
273 | _Features_:
274 | - FIFO queues support up to 300 messages per second
275 | - FIFO queues guarantee the ordering of messages
276 | - standard queues offer best-effort ordering but no guarantees
277 | - standard queues deliver a message at least once, but occasionally more than one copy of a message is delivered
278 |
279 | # Containers in the Cloud
280 | Enterprises are adopting container technology at an explosive rate. A container consists of everything an application needs to run: the application itself and its dependencies (e.g. libraries, utilities, configuration files), all bundled into one package. Each container is an independent component that can run on its own and be moved from environment to environment
281 |
282 |
283 | ## Elastic Container Service (ECS)
284 | **ECS** is an orchestration service used for automating deployment, scaling, and managing of your containerized applications. ECS works well with Docker containers by:
285 | - launching and stopping Docker containers
286 | - scaling your applications
287 | - querying the state of your applications
288 |
289 | _Features_:
290 | - you can schedule long-running applications, services, and batch processeses using ECS
291 | - docker is the only container platform supported by Amazon ECS
292 |
293 | # AWS Management
294 |
295 | ## Infrastructure as Code
296 | **Infrastructure as Code** allows you to describe and provision all the infrastructure resources in your cloud environment. You can stand up servers, databases, runtime parameters, resources, etc. based on scripts that you write. Infrastructure as Code is a time-saving feature because it allows you to provision (or stand up) resources in a reproducible way
297 |
298 | ## Cloud Formation
299 | **AWS Cloud Formation** allows you to model your entire infrastructure in a text file template allowing you to provision AWS resources based on the scripts you write
300 |
301 | ## AWS Command Line Interface (CLI)
302 | The **AWS CLI (Command Line Interface)** allows you to access and control services running in your AWS account from the command line. To use the CLI, simply download, install, and configure it
--------------------------------------------------------------------------------
/cloud/version.md:
--------------------------------------------------------------------------------
1 | # Version control
2 |
3 | ## Terminology
4 |
5 | **Version Control System (VCS)** / **Source Code Manager (SCM)** - a tool that manages different versions of source code
6 |
7 | **Commit** - a snapshot of the state of a project (like a save point in a game)
8 |
9 | **Repository** - a directory which contains your project work (made up of commits)
10 |
11 | **Checkout** - when content in the repository has been copied to the Working Directory
12 |
13 | **Staging Area** / **Staging Index** / **Index** - a file in the Git directory that stores information about what will go into your next commit
14 |
15 | **SHA (Secure Hash Algorithm)** - an ID number for each commit
16 |
17 | **Branch** - when a new line of development is created that diverges from the main line of development (can continue without altering the main line)
18 |
19 | **Forking** - an action that's done on a hosting service, like GitHub. Forking a repository creates an identical copy of the original repository and moves this copy to your account. You have total control over this forked repository. Modifying your forked repository does not alter the original repository in any way
20 |
21 | **Pull request** - a request for the source repository to pull in your commits and merge them with their project
22 |
23 |
24 |
25 | ## Initializing
26 |
27 | **git init** - sets up all of the necessary files and directories in a hidden directory called _.git_. The content of the _.git_ directory:
28 |
29 | - _config file_ - where all project specific configuration settings are stored
30 |
31 | - _description file_ - this file is only used by the GitWeb program, so we can ignore it
32 |
33 | - _hooks directory_ - this is where we could place client-side or server-side scripts that we can use to hook into Git's different lifecycle events
34 |
35 | - _info directory_ - contains the global excludes file
36 |
37 | - _objects directory_ - this directory will store all of the commits we make
38 |
39 | - _refs directory_ - this directory holds pointers to commits (basically the "branches" and "tags")
40 |
41 |
42 | **git clone path new_repo** - clone an existing repo (path) into new repo
43 |
44 |
45 | ## Logging
46 |
47 | **git status** - shows the state of our repository as Git sees it
48 |
49 | **git log** - displays the following info about each commit: the SHA, the author, the date, the message
50 |
51 | **git log --oneline** - a more succinct version of _git log_
52 |
53 | **git log --stat** - displays the files that have been changed in the commit: the file(s) that have been modified, the number of lines that have been added/removed, a summary line with the total number of modified files and lines that have been added/removed
54 |
55 | **git log -p** / **git log --patch** - displays the actual changes made to a file, different flags can be used (for example, _-w_ or _--ignore-all-space_ ignores whitespace when comparing lines)
56 |
57 | **git log SHA** - by supplying a SHA, the _git log_ command will start at that commit
58 |
59 | **git show** - displays the most recent commit
60 |
61 | **git show SHA** - displays only one commit with the specified SHA, the output is exactly the same as the _git log -p SHA_: the commit, the author, the date, the commit message, the patch information. The command can be combined with the flags: _--stat_, _--patch_, _--ignore-all-space_
62 |
63 | **git log --oneline --graph --all** - displays all branches at once in the git log output, the _--graph_ flag adds the bullets and lines to the leftmost part of the output, the _--all_ flag is what displays all of the branches in the repository
64 |
65 | **git shortlog** - groups commits by author and title
66 |
67 | **git log --author="Richard Kalehoff"** - filters commits with the _--author_ flag
68 |
69 | **git log --grep="border radius issue in Safari"** - filters commits using the _--grep_ flag
70 |
71 |
72 | ## Making commits
73 |
74 | **git add … ** - moves files from the Working Directory to the Staging Index
75 |
76 | **git commit -m message** - commits the changes to the repo (can be used without _-m_ flag with a code editor). The goal is that each commit has a single focus. Each commit should record a single-unit change
77 |
78 | **git diff** - displays the changes that have been made but haven't been committed: the files that have been modified, the location of the lines that have been added/removed, the actual changes that have been made
79 |
80 | **git tag -a tag_name** - adds a marker on a _most recent_ commit, the tag does not move around as new commits are added, the _-a_ flas creates an annotated flag (instead of a lightweight one), which includes the following info: the person who made the tag, the date the tag was made, a message for the tag
81 |
82 | **git tag -a tag_name SHA** - adds a marker on a _specific_ commit
83 |
84 | **git tag -d tag_name** - deletes the tag with the name _tag_name_
85 |
86 | ## Resetting changes
87 |
88 | **git reset ** - erases commits, the exact behavior is different and is based on the flag used, possible options:
89 | _--mixed_ (default) moves the commit to the working directory, _--soft_ moves the commit to the staging index, _--hard_ erases the commit completely
90 |
91 | ## Branching
92 |
93 | **git branch** - can be used to list all branch names in the repository, create new branches, delete branches
94 |
95 | **git branch branch_name** - creates a branch _branch_name_
96 |
97 | **git checkout branch_name** - switches to the branch _branch_name_
98 |
99 | **git branch -d branch_name** - deletes the branch _branch_name_ (the deletion can be forced with _-D_)
100 |
101 | **git checkout -b branch_name** - creates a branch and switches to it all in one command
102 |
103 |
104 | ## Merging
105 |
106 | - _merging_ is combining branches together
107 | - making a merge makes a commit!
108 | - two main types of merges in Git: a regular merge and a fast-forward merge
109 | - a _fast-forward merge_ happens when one branch is ahead of another, so the merge will just move the currently checked out branch forward until it points to the same commit that the other branch, it is one of the easiest merges to do
110 |
111 | **git merge branch_name** - merges _branch_name_ into the current (checked-out) branch, the command finds a single commit that both branches have in their commit history, combines the lines of code that were changed on the separate branches, makes a commit to record the merge, a commit message needs to be supplied (!)
112 |
113 | **git reset --hard HEAD^** - the command to undo the merge
114 |
115 |
116 | ## Merge Conflicts
117 |
118 | ### Indicators Explanation
119 |
120 | ```
121 | <<<<<<< HEAD everything below this line (until the next indicator) shows you what's on the current branch
122 | ||||||| merged common ancestors everything below this line (until the next indicator) shows you what the original lines were
123 | ======= is the end of the original lines, everything that follows (until the next indicator) is what's on the branch that's being merged in
124 | >>>>>>> heading-update is the ending indicator of what's on the branch that's being merged in (in this case, the heading-update branch)
125 | ```
126 |
127 | ### Resolving A Merge Conflict
128 |
129 | - choose which line(s) to keep
130 | - remove all lines with indicators
131 | - save the file(s)
132 | - stage the file(s)
133 | - make a commit
134 |
135 |
136 | ## Making changes
137 |
138 | **git commit --amend** - alters the most-recent commit (change the message, add forgotten files), in order to add new files, those files should be stages as for a regular commit before running the command
139 |
140 | **git revert SHA** - reverts a specific commit (does the exact opposite of that commit), creates a new commit with the opposite action. Resetting is different from reverting, it erases commits from the repo history completely (!)
141 |
142 |
143 | ## Relative Commit References (Ancestry References)
144 |
145 | **^** – indicates the parent commit
146 |
147 | **~** – indicates the first parent commit
148 |
149 | The main difference between the ^ and the ~ is when a commit is created from a merge. A merge commit has two parents. With a merge commit, the ^ reference is used to indicate the first parent of the commit while ^2 indicates the second parent. The first parent is the branch you were on when you ran git merge while the second parent is the branch that was merged in.
150 |
151 |
152 | Examples:
153 |
154 | - The following indicates the parent commit of the current commit: _HEAD^_, _HEAD\~_, _HEAD\~1_
155 |
156 | - The following indicates the grandparent commit of the current commit: _HEAD^^_, _HEAD\~2_
157 |
158 | - The following indicates the great-grandparent commit of the current commit: _HEAD^^^_, _HEAD\~3_
159 |
160 |
161 | ## Commit messages
162 |
163 | - do keep the message short (less than 60-ish characters)
164 | - do explain what the commit does (not how or why!)
165 |
166 | - do not explain why the changes are made (more on this below)
167 | - do not explain how the changes are made (that's what git log -p is for!)
168 | - do not use the word "and"
169 |
170 | ## Working with remotes
171 |
172 | **git remote -v** - shows the details about a connection to a remote
173 |
174 | **git remote add shortname url** - adds a connection to a new remote repository
175 |
176 | **git remote rename old_name new_name** - renames remotes
177 |
178 | _Note_: **origin** is a common shortname for remote repositories.
179 |
180 | **git push remote_repo branch_name** - sends commits from a local repository to a remote repository
181 |
182 | **git pull remote_repo branch_name** - pulls commits from a remote repository to a local repository, the command does two things: (1) fetching remote changes (which adds the commits to the local repository and moves the tracking branch to point to them), (2) merging the local branch (master) with the tracking branch (origin/master)
183 |
184 | **git fetch remote_repo branch_name** - retrieves the commits and moves the tracking branch
185 |
186 |
187 | ## Contributing to open source
188 |
189 |
190 | ### General guidelines:
191 |
192 | - before you start doing any work, make sure to look for the project's _CONTRIBUTING.md_ file
193 | - it's a good idea to look at the GitHub issues for the project, if necessary create a new issue
194 | - communicate the changes you'd like to make to the project maintainer in the issue
195 | - commit all of your work on a topic branch: do not work on the master branch and make sure to give the topic branch clear, descriptive name
196 |
197 |
198 | ### Creating a pull request:
199 |
200 | 1. you must fork the source repository
201 | 2. clone your fork down to your machine
202 | 3. make some commits (ideally on a topic branch!)
203 | 4. push the commits back to your fork
204 | 5. create a new pull request and choose the branch that has your new commits
205 |
206 |
207 | ### Staying in sync with a source repo:
208 |
209 | 1. get the cloneable URL of the source repository
210 | 2. create a new remote with the git remote add command
211 | 3. use the shortname upstream to point to the source repository
212 | 4. provide the URL of the source repository
213 | 5. fetch the new upstream remote
214 | 6. merge the upstream's branch into a local branch
215 | 7. push the newly updated local branch to your origin repo
216 |
217 |
218 | ## Rebasing
219 |
220 | **git rebase -i ** - interactive rebase
221 |
222 | The **git rebase** command is used to do a great many things. It can help you edit commit messages, reorder commits, combine commits, etc. Whenever you rebase commits, Git will create a new SHA for each commit. Inside the interactive list of commits, all commits start out as pick, but you can swap that out with one of the other commands (reword, edit, squash, fixup, exec, and drop). The best practice is to create a backup branch before rebasing, so that it's easy to return to your previous state.
223 |
224 | You should not rebase if you have already pushed the commits you want to rebase. If you're collaborating with other developers, then they might already be working with the commits you've pushed.
225 |
226 |
227 | ## Globbing and gitignore
228 |
229 | - the *.gitignore* file is used to tell Git about the files that Git should not track, this file should be placed in the same directory that the .git directory is in
230 |
231 | - **globbing** lets you use special characters to match patterns/characters, it is similar to regular expressions
232 |
--------------------------------------------------------------------------------
/pytorch/resources/arch.jpeg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/baroquerock/udacity_notes/efd4c665916ba4953d6378e69331b6cb450a385c/pytorch/resources/arch.jpeg
--------------------------------------------------------------------------------
/pytorch/resources/grad.jpeg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/baroquerock/udacity_notes/efd4c665916ba4953d6378e69331b6cb450a385c/pytorch/resources/grad.jpeg
--------------------------------------------------------------------------------
/pytorch/resources/graph_backprop.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/baroquerock/udacity_notes/efd4c665916ba4953d6378e69331b6cb450a385c/pytorch/resources/graph_backprop.png
--------------------------------------------------------------------------------
/pytorch/resources/rnn_components.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/baroquerock/udacity_notes/efd4c665916ba4953d6378e69331b6cb450a385c/pytorch/resources/rnn_components.png
--------------------------------------------------------------------------------
/pytorch/resources/rnn_schema.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/baroquerock/udacity_notes/efd4c665916ba4953d6378e69331b6cb450a385c/pytorch/resources/rnn_schema.png
--------------------------------------------------------------------------------
/pytorch/resources/rules.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/baroquerock/udacity_notes/efd4c665916ba4953d6378e69331b6cb450a385c/pytorch/resources/rules.png
--------------------------------------------------------------------------------
/pytorch/resources/shakespeare.txt:
--------------------------------------------------------------------------------
1 | From fairest creatures we desire increase,
2 | That thereby beauty's rose might never die,
3 | But as the riper should by time decease,
4 | His tender heir might bear his memory:
5 | But thou contracted to thine own bright eyes,
6 | Feed'st thy light's flame with self-substantial fuel,
7 | Making a famine where abundance lies,
8 | Thy self thy foe, to thy sweet self too cruel:
9 | Thou that art now the world's fresh ornament,
10 | And only herald to the gaudy spring,
11 | Within thine own bud buriest thy content,
12 | And tender churl mak'st waste in niggarding:
13 | Pity the world, or else this glutton be,
14 | To eat the world's due, by the grave and thee.
15 |
16 | When forty winters shall besiege thy brow,
17 | And dig deep trenches in thy beauty's field,
18 | Thy youth's proud livery so gazed on now,
19 | Will be a tattered weed of small worth held:
20 | Then being asked, where all thy beauty lies,
21 | Where all the treasure of thy lusty days;
22 | To say within thine own deep sunken eyes,
23 | Were an all-eating shame, and thriftless praise.
24 | How much more praise deserved thy beauty's use,
25 | If thou couldst answer 'This fair child of mine
26 | Shall sum my count, and make my old excuse'
27 | Proving his beauty by succession thine.
28 | This were to be new made when thou art old,
29 | And see thy blood warm when thou feel'st it cold.
30 |
31 | Look in thy glass and tell the face thou viewest,
32 | Now is the time that face should form another,
33 | Whose fresh repair if now thou not renewest,
34 | Thou dost beguile the world, unbless some mother.
35 | For where is she so fair whose uneared womb
36 | Disdains the tillage of thy husbandry?
37 | Or who is he so fond will be the tomb,
38 | Of his self-love to stop posterity?
39 | Thou art thy mother's glass and she in thee
40 | Calls back the lovely April of her prime,
41 | So thou through windows of thine age shalt see,
42 | Despite of wrinkles this thy golden time.
43 | But if thou live remembered not to be,
44 | Die single and thine image dies with thee.
45 |
46 | Unthrifty loveliness why dost thou spend,
47 | Upon thy self thy beauty's legacy?
48 | Nature's bequest gives nothing but doth lend,
49 | And being frank she lends to those are free:
50 | Then beauteous niggard why dost thou abuse,
51 | The bounteous largess given thee to give?
52 | Profitless usurer why dost thou use
53 | So great a sum of sums yet canst not live?
54 | For having traffic with thy self alone,
55 | Thou of thy self thy sweet self dost deceive,
56 | Then how when nature calls thee to be gone,
57 | What acceptable audit canst thou leave?
58 | Thy unused beauty must be tombed with thee,
59 | Which used lives th' executor to be.
60 |
61 | Those hours that with gentle work did frame
62 | The lovely gaze where every eye doth dwell
63 | Will play the tyrants to the very same,
64 | And that unfair which fairly doth excel:
65 | For never-resting time leads summer on
66 | To hideous winter and confounds him there,
67 | Sap checked with frost and lusty leaves quite gone,
68 | Beauty o'er-snowed and bareness every where:
69 | Then were not summer's distillation left
70 | A liquid prisoner pent in walls of glass,
71 | Beauty's effect with beauty were bereft,
72 | Nor it nor no remembrance what it was.
73 | But flowers distilled though they with winter meet,
74 | Leese but their show, their substance still lives sweet.
75 |
76 | Then let not winter's ragged hand deface,
77 | In thee thy summer ere thou be distilled:
78 | Make sweet some vial; treasure thou some place,
79 | With beauty's treasure ere it be self-killed:
80 | That use is not forbidden usury,
81 | Which happies those that pay the willing loan;
82 | That's for thy self to breed another thee,
83 | Or ten times happier be it ten for one,
84 | Ten times thy self were happier than thou art,
85 | If ten of thine ten times refigured thee:
86 | Then what could death do if thou shouldst depart,
87 | Leaving thee living in posterity?
88 | Be not self-willed for thou art much too fair,
89 | To be death's conquest and make worms thine heir.
90 |
91 | Lo in the orient when the gracious light
92 | Lifts up his burning head, each under eye
93 | Doth homage to his new-appearing sight,
94 | Serving with looks his sacred majesty,
95 | And having climbed the steep-up heavenly hill,
96 | Resembling strong youth in his middle age,
97 | Yet mortal looks adore his beauty still,
98 | Attending on his golden pilgrimage:
99 | But when from highmost pitch with weary car,
100 | Like feeble age he reeleth from the day,
101 | The eyes (fore duteous) now converted are
102 | From his low tract and look another way:
103 | So thou, thy self out-going in thy noon:
104 | Unlooked on diest unless thou get a son.
105 |
106 | Music to hear, why hear'st thou music sadly?
107 | Sweets with sweets war not, joy delights in joy:
108 | Why lov'st thou that which thou receiv'st not gladly,
109 | Or else receiv'st with pleasure thine annoy?
110 | If the true concord of well-tuned sounds,
111 | By unions married do offend thine ear,
112 | They do but sweetly chide thee, who confounds
113 | In singleness the parts that thou shouldst bear:
114 | Mark how one string sweet husband to another,
115 | Strikes each in each by mutual ordering;
116 | Resembling sire, and child, and happy mother,
117 | Who all in one, one pleasing note do sing:
118 | Whose speechless song being many, seeming one,
119 | Sings this to thee, 'Thou single wilt prove none'.
120 |
121 | Is it for fear to wet a widow's eye,
122 | That thou consum'st thy self in single life?
123 | Ah, if thou issueless shalt hap to die,
124 | The world will wail thee like a makeless wife,
125 | The world will be thy widow and still weep,
126 | That thou no form of thee hast left behind,
127 | When every private widow well may keep,
128 | By children's eyes, her husband's shape in mind:
129 | Look what an unthrift in the world doth spend
130 | Shifts but his place, for still the world enjoys it;
131 | But beauty's waste hath in the world an end,
132 | And kept unused the user so destroys it:
133 | No love toward others in that bosom sits
134 | That on himself such murd'rous shame commits.
135 |
136 | For shame deny that thou bear'st love to any
137 | Who for thy self art so unprovident.
138 | Grant if thou wilt, thou art beloved of many,
139 | But that thou none lov'st is most evident:
140 | For thou art so possessed with murd'rous hate,
141 | That 'gainst thy self thou stick'st not to conspire,
142 | Seeking that beauteous roof to ruinate
143 | Which to repair should be thy chief desire:
144 | O change thy thought, that I may change my mind,
145 | Shall hate be fairer lodged than gentle love?
146 | Be as thy presence is gracious and kind,
147 | Or to thy self at least kind-hearted prove,
148 | Make thee another self for love of me,
149 | That beauty still may live in thine or thee.
150 |
151 | As fast as thou shalt wane so fast thou grow'st,
152 | In one of thine, from that which thou departest,
153 | And that fresh blood which youngly thou bestow'st,
154 | Thou mayst call thine, when thou from youth convertest,
155 | Herein lives wisdom, beauty, and increase,
156 | Without this folly, age, and cold decay,
157 | If all were minded so, the times should cease,
158 | And threescore year would make the world away:
159 | Let those whom nature hath not made for store,
160 | Harsh, featureless, and rude, barrenly perish:
161 | Look whom she best endowed, she gave thee more;
162 | Which bounteous gift thou shouldst in bounty cherish:
163 | She carved thee for her seal, and meant thereby,
164 | Thou shouldst print more, not let that copy die.
165 |
166 | When I do count the clock that tells the time,
167 | And see the brave day sunk in hideous night,
168 | When I behold the violet past prime,
169 | And sable curls all silvered o'er with white:
170 | When lofty trees I see barren of leaves,
171 | Which erst from heat did canopy the herd
172 | And summer's green all girded up in sheaves
173 | Borne on the bier with white and bristly beard:
174 | Then of thy beauty do I question make
175 | That thou among the wastes of time must go,
176 | Since sweets and beauties do themselves forsake,
177 | And die as fast as they see others grow,
178 | And nothing 'gainst Time's scythe can make defence
179 | Save breed to brave him, when he takes thee hence.
180 |
181 | O that you were your self, but love you are
182 | No longer yours, than you your self here live,
183 | Against this coming end you should prepare,
184 | And your sweet semblance to some other give.
185 | So should that beauty which you hold in lease
186 | Find no determination, then you were
187 | Your self again after your self's decease,
188 | When your sweet issue your sweet form should bear.
189 | Who lets so fair a house fall to decay,
190 | Which husbandry in honour might uphold,
191 | Against the stormy gusts of winter's day
192 | And barren rage of death's eternal cold?
193 | O none but unthrifts, dear my love you know,
194 | You had a father, let your son say so.
195 |
196 | Not from the stars do I my judgement pluck,
197 | And yet methinks I have astronomy,
198 | But not to tell of good, or evil luck,
199 | Of plagues, of dearths, or seasons' quality,
200 | Nor can I fortune to brief minutes tell;
201 | Pointing to each his thunder, rain and wind,
202 | Or say with princes if it shall go well
203 | By oft predict that I in heaven find.
204 | But from thine eyes my knowledge I derive,
205 | And constant stars in them I read such art
206 | As truth and beauty shall together thrive
207 | If from thy self, to store thou wouldst convert:
208 | Or else of thee this I prognosticate,
209 | Thy end is truth's and beauty's doom and date.
210 |
211 | When I consider every thing that grows
212 | Holds in perfection but a little moment.
213 | That this huge stage presenteth nought but shows
214 | Whereon the stars in secret influence comment.
215 | When I perceive that men as plants increase,
216 | Cheered and checked even by the self-same sky:
217 | Vaunt in their youthful sap, at height decrease,
218 | And wear their brave state out of memory.
219 | Then the conceit of this inconstant stay,
220 | Sets you most rich in youth before my sight,
221 | Where wasteful time debateth with decay
222 | To change your day of youth to sullied night,
223 | And all in war with Time for love of you,
224 | As he takes from you, I engraft you new.
225 |
226 | But wherefore do not you a mightier way
227 | Make war upon this bloody tyrant Time?
228 | And fortify your self in your decay
229 | With means more blessed than my barren rhyme?
230 | Now stand you on the top of happy hours,
231 | And many maiden gardens yet unset,
232 | With virtuous wish would bear you living flowers,
233 | Much liker than your painted counterfeit:
234 | So should the lines of life that life repair
235 | Which this (Time's pencil) or my pupil pen
236 | Neither in inward worth nor outward fair
237 | Can make you live your self in eyes of men.
238 | To give away your self, keeps your self still,
239 | And you must live drawn by your own sweet skill.
--------------------------------------------------------------------------------
/pytorch/udacity_cnn_tatiana_gaponova.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/baroquerock/udacity_notes/efd4c665916ba4953d6378e69331b6cb450a385c/pytorch/udacity_cnn_tatiana_gaponova.pdf
--------------------------------------------------------------------------------
/pytorch/udacity_nn_basics_tatiana_gaponova.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/baroquerock/udacity_notes/efd4c665916ba4953d6378e69331b6cb450a385c/pytorch/udacity_nn_basics_tatiana_gaponova.pdf
--------------------------------------------------------------------------------
/pytorch/udacity_style_transfer_tatiana_gaponova.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/baroquerock/udacity_notes/efd4c665916ba4953d6378e69331b6cb450a385c/pytorch/udacity_style_transfer_tatiana_gaponova.pdf
--------------------------------------------------------------------------------
/pytorch/vanillaRNN.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "# Forward pass for Vanilla RNN\n",
8 | "Taken from here: https://gist.github.com/karpathy/d4dee566867f8291f086"
9 | ]
10 | },
11 | {
12 | "cell_type": "code",
13 | "execution_count": 1,
14 | "metadata": {},
15 | "outputs": [],
16 | "source": [
17 | "import numpy as np"
18 | ]
19 | },
20 | {
21 | "cell_type": "markdown",
22 | "metadata": {},
23 | "source": [
24 | "### Reading the data"
25 | ]
26 | },
27 | {
28 | "cell_type": "code",
29 | "execution_count": 5,
30 | "metadata": {},
31 | "outputs": [],
32 | "source": [
33 | "data = open('resources/shakespeare.txt', 'r').read() "
34 | ]
35 | },
36 | {
37 | "cell_type": "code",
38 | "execution_count": 6,
39 | "metadata": {},
40 | "outputs": [
41 | {
42 | "name": "stdout",
43 | "output_type": "stream",
44 | "text": [
45 | "From fairest creatures we desire increase,\n",
46 | "That thereby beauty's rose might never die,\n",
47 | "But as the riper should by time decease,\n",
48 | "His tender heir might bear his memory:\n",
49 | "But thou contracted to thine own bright eyes,\n",
50 | "Feed'st thy light's flame with self-substantial fuel,\n",
51 | "Making a famine where abundance lies,\n",
52 | "Thy self thy foe, to thy sweet self too cruel:\n",
53 | "Thou that art now the world's fresh ornament,\n",
54 | "And only herald to the gaudy spring,\n",
55 | "Within thine own bud buriest thy content,\n",
56 | "And tender churl mak'st waste in niggarding:\n",
57 | "Pity the world, or else this glutton be,\n",
58 | "To eat the world's due, by the grave and thee.\n",
59 | "\n"
60 | ]
61 | }
62 | ],
63 | "source": [
64 | "print(data[:610])"
65 | ]
66 | },
67 | {
68 | "cell_type": "markdown",
69 | "metadata": {},
70 | "source": [
71 | "### Encoding characters"
72 | ]
73 | },
74 | {
75 | "cell_type": "code",
76 | "execution_count": 7,
77 | "metadata": {},
78 | "outputs": [
79 | {
80 | "name": "stdout",
81 | "output_type": "stream",
82 | "text": [
83 | "data has 9837 characters, 57 unique\n"
84 | ]
85 | }
86 | ],
87 | "source": [
88 | "chars = list(set(data))\n",
89 | "data_size, vocab_size = len(data), len(chars)\n",
90 | "# char-to-int encoding\n",
91 | "char_to_ix = {ch:i for i,ch in enumerate(chars)}\n",
92 | "ix_to_char = {i:ch for i,ch in enumerate(chars)}\n",
93 | "print('data has {} characters, {} unique'.format(data_size, vocab_size))"
94 | ]
95 | },
96 | {
97 | "cell_type": "raw",
98 | "metadata": {},
99 | "source": []
100 | },
101 | {
102 | "cell_type": "raw",
103 | "metadata": {},
104 | "source": []
105 | },
106 | {
107 | "cell_type": "markdown",
108 | "metadata": {},
109 | "source": [
110 | "### RNN basics"
111 | ]
112 | },
113 | {
114 | "cell_type": "markdown",
115 | "metadata": {},
116 | "source": [
117 | "#### RNN Structure\n",
118 | "
"
119 | ]
120 | },
121 | {
122 | "cell_type": "markdown",
123 | "metadata": {},
124 | "source": [
125 | "#### Components of RNN\n",
126 | "Taken from here: https://cs224d.stanford.edu/lecture_notes/LectureNotes4.pdf\n",
127 | "\n",
128 | "
"
129 | ]
130 | },
131 | {
132 | "cell_type": "raw",
133 | "metadata": {},
134 | "source": []
135 | },
136 | {
137 | "cell_type": "markdown",
138 | "metadata": {},
139 | "source": [
140 | "Let's say that we defined the hyperparameters for the RNN layer as follows:
\n",
141 | "\n",
142 | "**hidden size** = 100
\n",
143 | "**sequence length** = 25\n",
144 | "\n",
145 | "\n",
146 | "So, what is the hidden size? \n",
147 | "\n",
148 | "In the example below we set the sequence length to 25. When we unroll the RNN layer, there will be 25 hidden states $h_{t}$.
Each hidden state will have the specified size of 100. So hidden size is just the size of every $h_{t}$. \n",
149 | "\n",
150 | "The output $h_{t}$ depends on two inputs: \n",
151 | "(1) previous hidden state $h_{t-1}$ \n",
152 | "(2) the current element of the sequence $x_t$\n",
153 | "\n",
154 | "Regular neural networks use only one element $x$ as an input,
\n",
155 | "so there was only one matrix of parameters $W$. \n",
156 | "\n",
157 | "But in the case of RNN the output depends on TWO inputs, so we should have TWO parameter matrices instead of just one:\n",
158 | "\n",
159 | "$$h_{t} = tanh(W^{hh}*h_{t-1} + W^{hx}*x_{t} + bias)$$\n",
160 | "\n",
161 | "\n",
162 | "$W^{hx}$ is associated with the inputs.\n",
163 | "\n",
164 | "$W^{hh}$ is associated with the previous hidden state.
\n",
165 | "\n",
166 | "\n",
167 | "\n",
168 | "$h_{t}$ should have dimensions of $(hidden\\_size, 1)$, it is $(100, 1)$ in our case.
\n",
169 | "In order for this to happen, both terms $W^{hh}*h_{t-1}$ and $W^{hx}*x_{t}$ should have the same dimension of $(100, 1)$.
\n",
170 | "\n",
171 | "We already know that $h_{t-1}$ is $(100, 1)$, then $W^{hh}$ should have dimensions of $(100, 100)$, because:\n",
172 | "
\n",
173 | "$$(100, 100) * (100, 1) = (100, 1)$$\n",
174 | "
\n",
175 | "The $W^{hx}$ should have dimensions of $(100, vocab\\_size)$, because:\n",
176 | "
\n",
177 | "$$(100, vocab\\_size) * (vocab\\_size, 1) = (100, 1)$$"
178 | ]
179 | },
180 | {
181 | "cell_type": "raw",
182 | "metadata": {},
183 | "source": []
184 | },
185 | {
186 | "cell_type": "markdown",
187 | "metadata": {},
188 | "source": [
189 | "### Defining the hyperparameters"
190 | ]
191 | },
192 | {
193 | "cell_type": "code",
194 | "execution_count": 8,
195 | "metadata": {},
196 | "outputs": [],
197 | "source": [
198 | "# hyperparameters\n",
199 | "hidden_size = 100 # size of hidden layer of neurons\n",
200 | "seq_length = 25 # number of steps to unroll the RNN for\n",
201 | "learning_rate = 1e-1"
202 | ]
203 | },
204 | {
205 | "cell_type": "markdown",
206 | "metadata": {},
207 | "source": [
208 | "### Initializing the weights"
209 | ]
210 | },
211 | {
212 | "cell_type": "code",
213 | "execution_count": 9,
214 | "metadata": {},
215 | "outputs": [],
216 | "source": [
217 | "# model parameters\n",
218 | "Wxh = np.random.randn(hidden_size, vocab_size)*0.01 # input to hidden\n",
219 | "Whh = np.random.randn(hidden_size, hidden_size)*0.01 # hidden to hidden\n",
220 | "Why = np.random.randn(vocab_size, hidden_size)*0.01 # hidden to output\n",
221 | "bh = np.zeros((hidden_size, 1)) # hidden bias\n",
222 | "by = np.zeros((vocab_size, 1)) # output bias"
223 | ]
224 | },
225 | {
226 | "cell_type": "code",
227 | "execution_count": 10,
228 | "metadata": {},
229 | "outputs": [
230 | {
231 | "name": "stdout",
232 | "output_type": "stream",
233 | "text": [
234 | "Input to hidden weights: (100, 57)\n",
235 | "Hidden to hidden weights: (100, 100)\n",
236 | "Hidden to output weights: (57, 100)\n",
237 | "\n",
238 | "Hidden bias weights: (100, 1)\n",
239 | "Output bias weights: (57, 1)\n"
240 | ]
241 | }
242 | ],
243 | "source": [
244 | "print('Input to hidden weights: {}'.format(Wxh.shape))\n",
245 | "print('Hidden to hidden weights: {}'.format(Whh.shape))\n",
246 | "print('Hidden to output weights: {}\\n'.format(Why.shape))\n",
247 | "print('Hidden bias weights: {}'.format(bh.shape))\n",
248 | "print('Output bias weights: {}'.format(by.shape))"
249 | ]
250 | },
251 | {
252 | "cell_type": "markdown",
253 | "metadata": {},
254 | "source": [
255 | "### Extracting one sequence\n",
256 | "For the purpose of demonstration, I am using only one sequence of length 25 in the forward pass. \n",
257 | "*inputs* - one sequence of length 25\n",
258 | "*targets* - one sequence of length 25 shifted by 1 over the *inputs*"
259 | ]
260 | },
261 | {
262 | "cell_type": "code",
263 | "execution_count": 11,
264 | "metadata": {},
265 | "outputs": [
266 | {
267 | "name": "stdout",
268 | "output_type": "stream",
269 | "text": [
270 | "F(45) -> r(41)\n",
271 | "r(41) -> o(14)\n",
272 | "o(14) -> m(24)\n"
273 | ]
274 | }
275 | ],
276 | "source": [
277 | "p = 0\n",
278 | "inputs = [char_to_ix[ch] for ch in data[p:p+seq_length]]\n",
279 | "targets = [char_to_ix[ch] for ch in data[p+1:p+seq_length+1]]\n",
280 | "\n",
281 | "# word 'from'\n",
282 | "for i in range(3):\n",
283 | " print('{}({}) -> {}({})'.format(ix_to_char[inputs[i]], inputs[i], ix_to_char[targets[i]], targets[i]))"
284 | ]
285 | },
286 | {
287 | "cell_type": "markdown",
288 | "metadata": {},
289 | "source": [
290 | "### RNN Forward pass"
291 | ]
292 | },
293 | {
294 | "cell_type": "code",
295 | "execution_count": 12,
296 | "metadata": {},
297 | "outputs": [],
298 | "source": [
299 | "# defining the previous hidden state\n",
300 | "# it equals to zeros since for the fisrt element there is no preious hidden state\n",
301 | "hprev = np.zeros((hidden_size,1))"
302 | ]
303 | },
304 | {
305 | "cell_type": "code",
306 | "execution_count": 13,
307 | "metadata": {},
308 | "outputs": [],
309 | "source": [
310 | "# xs - contains one-hot encoded character in a sequence for timestep t\n",
311 | "# hs - contains hidden states for timestep t\n",
312 | "# ys - contains output for timestep t \n",
313 | "# ps - contains probabilities for timestep t \n",
314 | "xs, hs, ys, ps = {}, {}, {}, {}"
315 | ]
316 | },
317 | {
318 | "cell_type": "code",
319 | "execution_count": 14,
320 | "metadata": {},
321 | "outputs": [],
322 | "source": [
323 | "hs[-1] = np.copy(hprev)\n",
324 | "loss = 0"
325 | ]
326 | },
327 | {
328 | "cell_type": "code",
329 | "execution_count": 15,
330 | "metadata": {},
331 | "outputs": [],
332 | "source": [
333 | "for t in range(seq_length):\n",
334 | " #print('Timestep: {}'.format(t+1))\n",
335 | " xs[t] = np.zeros((vocab_size,1)) # one-hot\n",
336 | " xs[t][inputs[t]] = 1 # one-hot \n",
337 | " first_term = np.dot(Wxh, xs[t])\n",
338 | " second_term = np.dot(Whh, hs[t-1])\n",
339 | " hs[t] = np.tanh(first_term + second_term + bh)\n",
340 | " ys[t] = np.dot(Why, hs[t]) + by\n",
341 | " ps[t] = np.exp(ys[t]) / np.sum(np.exp(ys[t])) # probabilities for next chars\n",
342 | " loss += -np.log(ps[t][targets[t],0])"
343 | ]
344 | },
345 | {
346 | "cell_type": "code",
347 | "execution_count": null,
348 | "metadata": {},
349 | "outputs": [],
350 | "source": []
351 | }
352 | ],
353 | "metadata": {
354 | "kernelspec": {
355 | "display_name": "Python 3",
356 | "language": "python",
357 | "name": "python3"
358 | },
359 | "language_info": {
360 | "codemirror_mode": {
361 | "name": "ipython",
362 | "version": 3
363 | },
364 | "file_extension": ".py",
365 | "mimetype": "text/x-python",
366 | "name": "python",
367 | "nbconvert_exporter": "python",
368 | "pygments_lexer": "ipython3",
369 | "version": "3.7.2"
370 | }
371 | },
372 | "nbformat": 4,
373 | "nbformat_minor": 2
374 | }
375 |
--------------------------------------------------------------------------------