A collection of related data entries consisting of columns and rows in a structured format called a database.
16 |
Schema
17 |
The skeleton structure that represents the logical view of the entire database. It defines the constraints on the data and relations to other tables.
18 |
Primary key
19 |
A field in a table which uniquely identifies each row/record in a database table.
20 |
Foreign key
21 |
A field in a table used to link it to another table.
22 |
Database index
23 |
A copy of selected columns of data from a table that can be searched very efficiently and includes a direct link to the complete row of data from the original table.
24 |
Database normalization
25 |
A horizontal partition of data in a database. Each shards is held on a separate database server instance to efficiently spread load.
26 |
Database sharding
27 |
A vertical partition of data where columns of a table are moved to their own table and replaced with a foreign key.
28 |
Master/Slave model
29 |
A single Master server for all write operations, and one or many additional Slave servers that provide read-only operations for increasing overall performance. This approach has a write capacity bottleneck and replication is not always in real-time.
30 |
ACID
31 |
A set of properties of database transactions intended to guarantee validity even in the event of errors, power failures, etc. Atomicity guarantees that each transaction is treated as a single "unit", which either succeeds completely, or fails completely. Consistency ensures that a transaction can only bring the database from one valid state to another. Isolation ensures that concurrent execution of transactions leaves the database in the same state that would have been obtained if the transactions were executed sequentially. Durability guarantees that once a transaction has been committed, it will remain committed even in the case of a system failure.
SQL (Structured Query Language) lets you access and manipulate relational databases.
42 |
43 |
Commands
44 |
45 | ```
46 | SELECT - extracts data from a database
47 | UPDATE - updates data in a database
48 | DELETE - deletes data from a database
49 | INSERT INTO - inserts new data into a database
50 | CREATE DATABASE - creates a new database
51 | ALTER DATABASE - modifies a database
52 | CREATE TABLE - creates a new table
53 | ALTER TABLE - modifies a table
54 | DROP TABLE - deletes a table
55 | CREATE INDEX - creates an index (search key)
56 | DROP INDEX - deletes an index
57 | ```
58 |
59 |
60 |
Queries
61 |
62 | ```
63 | SELECT [columns]
64 | FROM [table]
65 | WHERE [comparison predicate on rows]
66 | GROUP BY [columns]
67 | HAVING [comparison predicate on groups]
68 | ORDER BY [columns]
69 | ```
70 |
71 |
Subqueries
72 |
Queries can be nested so that the results of one query can be used in another query via a relational operator or aggregation function. Joins and other table operations provide faster alternatives.
73 |
74 |
Other Keywords
75 |
The DISTINCT keyword eliminates duplicate data.
76 | The LIMIT clause specifies the number of records to return
77 | The COUNT() function returns the number of rows that matches a specified criteria.
78 | The AVG() function returns the average value of a numeric column.
79 | The SUM() function returns the total sum of a numeric column.
80 | The UNION operator is used to combine multiple similar schema result-sets.
81 |
82 |
Operators
83 |
84 |
85 |
86 |
87 |
88 |
Joins
89 |
A JOIN clause is used to combine rows from two or more tables, based on a related column between them.
Provides a mechanism for storage and retrieval of data that is modeled in means other than the tabular relations used in relational databases. NoSQL systems are also sometimes called "Not only SQL" to emphasize that they may support SQL-like query languages, or sit alongside SQL database in a polyglot persistence architecture.
102 |
Why NoSQL?
103 |
Simplicity of design, simpler "horizontal" scaling to clusters of machines (which is a problem for relational databases), and finer control over availability. The data structures used by NoSQL databases (e.g. key-value, wide column, graph, or document) are different from those used by default in relational databases, making some operations faster in NoSQL. Sometimes the data structures used by NoSQL databases are also viewed as "more flexible" than relational database tables due to not having to adhere to previously defined schemas.
A search engine that provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. Elasticsearch supports real-time GET requests, which makes it suitable as a NoSQL datastore, but it lacks distributed transactions.
114 |
115 |
Features
116 |
Elasticsearch is distributed, which means that indices can be divided into shards and each shard can have zero or more replicas. .[5] Related data is often stored in the same index, which consists of one or more primary shards, and zero or more replica shards. Once an index has been created, the number of primary shards cannot be changed. Elasticsearch is developed alongside a data-collection and log-parsing engine called Logstash, and an analytics and visualisation platform called Kibana. The three products are designed for use as an integrated solution, referred to as the "Elastic Stack".
117 |
Terminology
118 |
An index is like a ‘database’ in a relational database. It has a mapping which is the equivalent of a schema. A document is the equivalent of database record which is serialized as a JSON object.
119 |
Cluster
120 |
A collection of one or more nodes (servers) that together holds your entire data and provides federated indexing and search capabilities across all nodes. Each node hosts one or more shards, and acts as a coordinator to automatically delegate operations to the correct shard(s).
121 |
Shards
122 |
An index is split into shards. Documents are hashed to a particular shard. Each shard can be on a different node in a cluster. Follows the master/slave pattern with primary and replica shards. Primary shards are expensive so you do not want to overallocate too much. However, increasing the number of primary shards is a diffiult task that requires reindexing. Replica shards can be increased anytime.
123 |
Analyzer
124 |
Elasticsearch ships with a wide range of built-in analyzers, which can be used in any index mapping without further configuration. An example an analyzer would be autocomplete. Modifying these analyzers after they have already been created requires reindexing the data.
125 |
Aggregations
126 |
Collects all the data selected by the search query in order to build complex summaries of the data. Averages, moving averages, histograms, etc.
127 |
Full text queries
128 |
Includes fuzzy matching and proximity queries.
129 |
Elastic Stack
130 |
131 |
132 |
133 |
Technical notes
134 |
Use scripts to import CSV directly into Elasticsearch. Or use Logstash to import, parse, and transform data from various sources (Kafka, Apache Spark, Hadoop, MySQL, S3, etc). Logstash then exports into Elasticsearch. Logstash can scale across many nodes and guarantees at-least-once delivery.
135 |
Hardware requirements
136 |
RAM is the bottleneck, not CPU. Set the heap size to 32GB or half of the available system RAM for Elasticsearch. The remaining will be used for Lucene/OS to cache. Use an SSD with deadline or noop i/o scheduler for improved I/O latency. Redundancy is not important for file storage since the clusters are already redundant.
Node.js operates on a single thread event loop, using non-blocking I/O calls, allowing it to support tens of thousands of concurrent connections without incurring the cost of thread context switching. Node.js relies on the libuv library (written in C) which uses a fixed-sized (multiple) thread pool to handle the non-blocking asynchronous I/O operations.
16 |
A downside to the single-threaded approach is that Node.js doesn't allow vertical scaling by increasing the number of CPU cores of the machine it is running on without using a cluster module (such as pm2). Additionally, the number of threads in the libuv thread pool can be updated. The operating system will distribute these threads across multiple cores. Typically, the number of threads or forked processes should equal the number of CPU cores.
17 |
V8 Engine and Node.js bindings
18 |
V8 is a powerful open source Javascript engine provided by Google that compiles JavaScript source code to native machine code instead of interpreting it in real time. The core functionality of Node.js resides in a JavaScript library. The Node.js bindings, written in C++, connect these technologies to each other and to the operating system. These bindings allow the JavaScript to understand more than what the ECMAScript standard specifies the JavaScript should understand.
19 |
Event loop
20 |
Node.js registers with the operating system so the OS notifies it of connections and issues a callback. Within the Node.js runtime, each connection is a small heap allocation. Traditionally, processes or threads handled each connection. In contrast, Node.js uses an event loop that does not need to be called explicitly. Any function performing I/O must use a callback, and the server automatically enters the event loop at the end of the callback definition. Node.js exits the event loop when there are no further callbacks to be performed.
21 |
Process.env
22 |
The process.env global variable is injected by the Node at runtime for your application to use and it represents the state of the system environment your application is in when it starts. Secret variables are stored here rather than in accessible code.
npm is the pre-installed package manager for the Node.js server platform. It installs Node.js programs from the npm registry, organizing the installation and management of third-party Node.js programs. Packages in the npm registry can range from simple helper libraries such as Lodash to task runners such as Grunt.
36 |
37 |
38 |
NPM Scripts
39 |
Npm has a run command that can run scripts defined in the scripts property of a package.json file. It provides additional commands to the CLI, such as npm run test. This functionality reduces the need for third-party tooling like Grunt or Gulp.
40 |
You can add even more functionality to NPM scripts by installing open source dev dependencies. npm-run-all can run scripts in parallel (usually multiple tasks are run in sequence with &&). onchange will automatically re-run the task by watching for changes in your code.
54 | [Node.js Handbook](https://medium.freecodecamp.org/the-definitive-node-js-handbook-6912378afc6e) - The definitive Node.js handbook
55 | [Node best practices](https://github.com/i0natan/nodebestpractices) - The largest Node.JS best practices list
56 | [Nodeschool](http://nodeschool.io) - Learn Node.js with interactive lessons.
57 | [The Art of Node](https://github.com/maxogden/art-of-node/#the-art-of-node) - An introduction to Node.js.
58 | [stream-handbook](https://github.com/substack/stream-handbook) - How to write Node.js programs with streams.
59 | [module-best-practices](https://github.com/mattdesl/module-best-practices) - Some good practices when writing new npm modules.
60 | [You Don't Know Node.js](https://github.com/azat-co/you-dont-know-node) - Introduction to Node.js core features and asynchronous JavaScript.
61 | [The Node Way](http://thenodeway.io) - An entire philosophy of Node.js best practices and guiding principles exists for writing maintainable modules, scalable applications, and code that is actually pleasant to read.
65 |
66 |
--------------------------------------------------------------------------------
/content/backend/web_framework.md:
--------------------------------------------------------------------------------
1 | +++
2 | description = "REST, Express.js, Authentication, security, deployment"
3 | title = "Web Framework"
4 | draft = false
5 | weight = 300
6 | bref="A web framework is a software framework that is designed to support the development of web applications including web services, web resources, and web APIs"
7 | toc = true
8 | script = 'animation'
9 | +++
10 |
11 |
Representational State Transfer (REST) is a software architectural style that defines a set of constraints for creating web services over HTTP.
14 |
15 |
6 guiding constraints
16 |
17 |
Client–server architecture - a distributed application structure that partitions tasks or workloads between the providers of a resource or service, called servers, and service requesters, called clients.
18 |
Statelessness - Each request from any client contains all the information necessary to service the request, and session state is held in the client (not the server).
19 |
Cacheability - Responses must, implicitly or explicitly, define themselves as cacheable or not to prevent clients from getting stale or inappropriate data in response to further requests.
20 |
Layered system - A client cannot ordinarily tell whether it is connected directly to the end server, or to an intermediary along the way.
21 |
Code on demand (optional) - Servers can temporarily extend or customize the functionality of a client by transferring executable code such as JavaScript client side scripts.
22 |
Uniform interface - Constraints that decouple the client interface from the server implementation. These include how to identify resources, describe metadata, and represent data. See HTTP protocol for a better understanding of the constraints needed.
23 |
24 |
Other methods
25 |
GraphQL - an open-source data query and manipulation language for developing flexible APIs. It allows clients to define the structure of the data required, and exactly the same structure of the data is returned from the server, therefore preventing excessively large amounts of data from being returned, but this has implications for how effective web caching of query results can be. The flexibility and richness of the query language also adds complexity that may not be worthwhile for simple APIs.
26 |
Websockets - a computer communications protocol facilitating real-time data transfer between client and server. The protocol is located at the application layer of the TCP/IP model and depend on TCP at layer 4. The WebSocket handshake uses the HTTP Upgrade header to change from the HTTP protocol to the WebSocket protocol (also compatible with HTTP). This allows messages to be passed back and forth while keeping the connection open. The communications are done over TCP port number 80 (or 443 in the case of TLS-encrypted connections).
An open-source web application framework using Node.js designed for building web applications and APIs.
34 |
35 |
Routing
36 |
Routing refers to determining how an application responds to a client request to a particular endpoint, which is a URI (or path) and a specific HTTP request method (GET, POST, and so on). Express is configured to handle routes and HTTP verb with app.METHOD(PATH, HANDLER).
37 |
Static files
38 |
To serve static files such as images, CSS files, and JavaScript files, use the express.static built-in middleware function. Express also supports templating. A template engine enables you to use static template files in your application. At runtime, the template engine replaces variables in a template file with actual values, and transforms the template into an HTML file sent to the client. Pug, Handlebars, Jade are different template engines you can choose, each with their own syntax.
39 |
Middleware
40 |
Middleware functions are functions that have access to the request object (req), the response object (res), and the next middleware function in the application’s request-response cycle.
Hashing performs a one-way transformation on a password, turning the password into another string, called the hashed password. Salting is adding random data as an additional input to the hash function. Authentication then relies on comparing the salt + hash of the original password. This protects commonly used passwords or users who use the same password on several sites, by making all salted hash instances for the same password different from each other.
53 |
54 |
55 |
56 | Preventing reverse engineering via brute forcing the hash algorithm
57 |
58 |
59 |
Cookies vs tokens
60 |
Cookie-based authentication is stateful. This means that an authentication record or session must be kept both server and client-side. Token-based authentication is stateless. The server does not keep a record of which users are logged in or which tokens have been issued. Instead, every request to the server is accompanied by a signed token which the server uses to verify the authenticity of the request. Today, most authentication has moved from cookie-based to token-based.
61 |
62 |
63 |
64 |
Authentication libraries
65 |
Passport - authentication middleware for Node that provides "strategies" for handling all types of authentication mechanisms. JWT (JSON web tokens) and OAuth (login with Facebook/Google account) are the most common.
66 |
Third party - Ideally, you don't want to be handling sensitive data that can be hacked. Allowing a 3rd party to handle validation is best practice. AWS Cognito and Auth0 are two of the more popular authentication providers.
The easiest way to deploy Express apps is through Heroku, a cloud platform as a service supporting Node.js and many other languages. A platform-based service is a category of cloud computing services that provides a platform allowing customers to develop, run, and manage applications without the complexity of building and maintaining the infrastructure typically associated with developing and launching an app.
76 |
Once the Heroku CLI is installed, deployment is a simple heroku create command and Heroku will handle dependencies, installation, runtime environment, and all other deployment tasks. The Heroku Platform uses the container model to run and scale all Heroku apps.
77 |
78 |
79 |
AWS
80 |
AWS provides a similar service to Heroku called Elastic Beanstalk. They also provide virtual machines called EC2 instances. Deploying an app in this manner is a little more cumbersome.
81 |
82 |
Create EC2 instance
83 |
Access EC2 instance through SSH (using PEM/PPK file)
84 |
Download and Install node on the instance
85 |
Clone repository (from version control like GitHub)
86 |
Install app dependencies
87 |
Run the app in the node runtime
88 |
Expose ports on the instance for public access
89 |
90 |
Choose AWS when you need infrastructure flexibility and understand DevOps. It is also the cheaper option as the app grows in complexity and use. Choose Heroku for smaller projects or when you don't want to deal with DevOps.
Don’t use deprecated or vulnerable versions of your dependencies. Use npm audit fix to scan your project for vulnerabilities and automatically install any compatible updates to vulnerable dependencies. Alternatively, use Snyk.io.
100 |
SSL/TLS Certificate
101 |
In HTTPS, the communication protocol is encrypted using Transport Layer Security (TLS) to encrypt data before it is sent from the client to the server, thus preventing some common hacks (Man in the middle). HTTPS is based on public/private-key cryptography. This means that there is a key pair: The public key is used for encryption and the secret private key is required for decryption. A website certificate is a public key with a label identifying the owner. when your browser connects to an HTTPS server, the server will answer with its certificate. The browser checks if the certificate is valid and signed by a trusted certification authority. After the verification, the browser extracts the public key and uses it to encrypt information it sends back to the server. The server can decrypt it because the server has the matching private key.
102 |
103 |
104 |
105 |
106 |
Set security related HTTP headers
107 |
Cross-origin resource sharing (CORS) is a mechanism that allows restricted resources on a web page to be requested from another domain outside the domain from which the first resource was served. A web page may freely embed cross-origin images, stylesheets, scripts, iframes, and videos. Certain "cross-domain" requests, notably Ajax requests, are forbidden by default by the same-origin security policy. These have to be set explicitly on the header.
108 |
There are many other security related headers. Use a middleware function like Helmet to set these for you.
109 |
Learn common vulnerabilities
110 |
111 |
Cross-site scripting (XSS) - enables attackers to inject client-side scripts into web pages viewed by other users.
112 |
SQL injection - nefarious SQL statements are inserted into an entry field for execution (e.g. to dump the database contents to the attacker)
113 |
Cross-Site Request Forgery (CSRF) - unauthorized commands are transmitted from a user that the web application trusts.
114 |
Distributed Denial-of-service attack (DDoS) - flooding the server or resource with superfluous requests in an attempt to overload systems and prevent some or all legitimate requests from being fulfilled.
115 |
116 |
Secure development environments as well
117 |
118 |
Successful key management - involves dealing with the generation, exchange, storage, use, destruction and replacement of keys. It is the more challenging side of cryptography as it involves aspects of social engineering, system policy, user training, organizational and departmental interactions, and coordination between all of these elements.
119 |
Principle of least privilege - Every process, user, or program must be able to access only the information and resources that are necessary for its legitimate purpose and nothing more. This especially applies in teams of developers.
120 |
Use VPN - encrypt data if using unsecured or untrustworthy networks vulnerable to packet sniffing and man-in-the-middle attacks.
121 |
Monitor system - Check open ports and unrecognized running processes
Big O notation is the language we use for talking about how long an algorithm takes to run. It's how we compare the efficiency of different approaches to a problem.
16 |
To demonstrate Big O notation, let's use the example of a phone book. We will assume our phone book has businesses (the "Yellow Pages") which have unique names and people (the "White Pages") which may not have unique names. A phone number is assigned to at most one person or business. We will also assume that it takes constant time to flip to a specific page.
17 |
Here are the running times of some operations we might perform on the phone book, from best to worst:
18 |
19 |
O(1) (best case)
20 |
Given the page that a business's name is on and the business name, find the phone number.
21 |
22 |
23 |
O(1) (average case)
24 |
Given the page that a person's name is on and their name, find the phone number.
25 |
26 |
27 |
O(log n)
28 |
Given a person's name, find the phone number by picking a random point about halfway through the part of the book you haven't searched yet, then checking to see whether the person's name is at that point. Then repeat the process about halfway through the part of the book where the person's name lies. (This is a binary search for a person's name.)
29 |
30 |
31 |
O(n)
32 |
Find all people whose phone numbers contain the digit "5".
33 |
34 |
35 |
O(n)
36 |
Given a phone number, find the person or business with that number.
37 |
38 |
39 |
O(n log n)
40 |
There was a mix-up at the printer's office, and our phone book had all its pages inserted in a random order. Fix the ordering so that it's correct by looking at the first name on each page and then putting that page in the appropriate spot in a new, empty phone book.
41 |
42 |
For the below examples, we're now at the printer's office. Phone books are waiting to be mailed to each resident or business, and there's a sticker on each phone book identifying where it should be mailed to. Every person or business gets one phone book.
43 |
44 |
O(n log n)
45 |
We want to personalize the phone book, so we're going to find each person or business's name in their designated copy, then circle their name in the book and write a short thank-you note for their patronage.
46 |
47 |
48 |
O(n2)
49 |
A mistake occurred at the office, and every entry in each of the phone books has an extra "0" at the end of the phone number. Take some white-out and remove each zero.
50 |
51 |
52 |
O(n · n!)
53 |
We're ready to load the phonebooks onto the shipping dock. Unfortunately, the robot that was supposed to load the books has gone haywire: it's putting the books onto the truck in a random order! Even worse, it loads all the books onto the truck, then checks to see if they're in the right order, and if not, it unloads them and starts over. (This is the dreaded bogo sort.)
54 |
55 |
56 |
O(nn)
57 |
You fix the robot so that it's loading things correctly. The next day, one of your co-workers plays a prank on you and wires the loading dock robot to the automated printing systems. Every time the robot goes to load an original book, the factory printer makes a duplicate run of all the phonebooks! Fortunately, the robot's bug-detection systems are sophisticated enough that the robot doesn't try printing even more copies when it encounters a duplicate book for loading, but it still has to load every original and duplicate book that's been printed.
75 |
76 |
77 |
78 | Binary search trees keep their keys in sorted order, so that lookup and other operations can use the principle of binary search. This means that each comparison allows the operations to skip about half of the tree, so that each lookup is much better than the linear time required to find items by key in an (unsorted) array, but slower than the corresponding operations on hash tables.
79 |
80 |
81 |
82 |
83 |
84 |
85 |
86 | Hash functions are used in hash tables, to quickly locate a data record (e.g., a dictionary definition) given its search key. Specifically, the hash function is used to map the search key to a list; the index gives the place in the hash table where the corresponding record should be stored. Typically, the set of possible keys is larger than the number of different table indices, and so it will map several different keys to the same index which could result in collisions. So then, each slot of a hash table is associated with a set of records, often called a bucket.
87 |
88 |
89 |
str.test(reg)
105 | ```
106 | var str = "The best things in life are free";
107 | var reg = new RegExp("e");
108 | var res = reg.test(str);
109 | console.log(res) // expected output: true
110 | ```
111 |
112 | str.match(reg)
113 | ```
114 | var paragraph = 'The quick brown fox jumped over the lazy dog. It barked.';
115 | var reg = /[A-Z]/g;
116 | var found = paragraph.match(reg);
117 | console.log(found); // expected output: Array ["T", "I"]
118 | ```
119 |
120 | reg.exec(str)
121 | ```
122 | var str = "The best things in life are free";
123 | var reg = new RegExp("e");
124 | var res = reg.exec(str);
125 | console.log(res) // expected output: ["e", index: 2, input: "The best things in life are free", groups: undefined]
126 | ```
127 |
128 | str.replace(str1,str2)
129 | ```
130 | var p = 'The quick brown fox jumped over the lazy dog. If the dog reacted, was it really lazy?';
131 | var reg = /dog/gi;
132 | p.replace(reg,'cat')
133 | console.log(res) // expected output: "The quick brown fox jumped over the lazy cat. If the cat reacted, was it really lazy?"
134 | ```
135 |
136 | str.search(reg)
137 | ```
138 | let str = "A drop of ink may make a million think";
139 | alert( str.search( /a/i ) ); // 0 (the first position)
140 | ```
141 |
142 | str.includes("")
143 | ```
144 | var sentence = 'The quick brown fox jumped over the lazy dog.';
145 | var word = 'fox';
146 | console.log(sentence.includes(word)); // expected output: true
147 | ```
148 |
149 | str.split(' ') and words.join(' ')
150 | ```
151 | var str = 'The quick brown fox jumped over the lazy dog.';
152 | var words = str.split(' ');
153 | console.log(words[3]); // expected output: "fox"
154 | console.log(words.join(" ") === str) // expected output: true
155 | ```
156 |
--------------------------------------------------------------------------------
/content/basics/fundamentals.md:
--------------------------------------------------------------------------------
1 | +++
2 | description = "Hardware, software, binary, CLI"
3 | title = "Fundamentals"
4 | draft = false
5 | weight = 100
6 | bref="Computers consist of two major elements: hardware and software. Hardware is any physical device used in or with your machine. Software is a set of instructions for a computer to perform specific operations"
7 | toc = true
8 | script = 'animation'
9 | +++
10 |
11 |
The electronic circuitry within a computer that carries out the instructions of a computer program by performing the basic arithmetic, logic, controlling, and input/output (I/O) operations specified by the instructions. It consists of billions of transistors on the semiconductor material silicon. Transisters are assembled into logic gates (AND, OR, NOT) to make logical decisions based on the digital signals present on its inputs.
16 |
17 |
18 |
Memory (RAM)
19 |
Quick and temporary storage space for data. The CPU has exclusive access to RAM and processes data through memory rather than much slower storage devices. Once the process is over or the computer is shut down, the data from memory is wiped.
20 | CPUs also have a small cache which stores copies of frequently accessed RAM data for even faster retrieval.
21 |
22 |
23 |
Input/Output (IO)
24 |
Examples of input devices include the mouse and keybord. Output devices are monitors, printers, speakers, etc. A hard drive can be an input or output device depending on whether the CPU is reading or writing to the disk.
25 | There is also virtual memory which provides additional inactive memory from storage devices through a technique called paging.
26 |
27 |
28 |
Motherboard
29 |
The glue that connects all components together. It is connected to a power supply and contains expansion slots to allow for additional functionality. For example, a GPU can offload video processing from the CPU.
The operating system determines how to allocate resources (CPU and memory) for all the other applications running on your computer. Once an application is opened, the program is loaded into memory and it is called a "process".
41 |
Processes
42 |
Each process has a separate memory address space. It cannot efficiently communicate with other processes or access shared data in other processes. Switching from one process to another requires some time (relatively) for saving and loading registers, memory maps, and other resources.
43 |
Every process needs 3 items:
44 |
45 |
Register - a part of the CPU that holds an instruction, a storage address, or other kind of data needed by the process
46 |
Program counter - (aka instruction pointer) keeps track of where a computer is in its program sequence
47 |
Stack - a data structure that stores information in memory about the active subroutines of a computer program and is used as scratch space for the process. It includes the Heap which is the dynamically allocated memory portion of the stack.
48 |
49 |
Threads
50 |
A process can have one thread or have many threads. A thread is the unit of execution within a process. Each thread will have its own stack but all the threads in the process share the same heap. This means that communication between threads is efficient. However, an error in one thread can affect the other threads in the same process.
51 |
With a multithreaded process, the CPU can perform operations in parallel or concurrently.
52 |
53 |
Parallelism - Genuine simultaneous execution in a multi-core processor.
54 |
Concurrency - Interleaving of processes within one processor, giving the illusion of simultaneous execution.
Because a transmitter can only relay on and off, computers use the base-2 (binary) numeral system using only two symbols: "0" and "1".
66 |
67 |
68 |
69 |
Bit vs Byte
70 |
A bit is the basic unit of information representing either 0 or 1.
71 | A byte represents a string of 8 bits.
72 | A kilobyte is equal to 1,024 (210) bytes and a kilobit is equal to 1,024 (210) bits.
73 | A megabyte is equal to 1,024 (2100) bytes and a megabit is equal to 1,024 (2100) bits.
74 |
75 |
A common misunderstanding relating to bits and bytes occurs when referring to data size and data speed. Speed is measured in bits and size is measured in bytes. This means that 4G service at 100 megabits per second (Mb/s) allows downloading at 12.5 megabytes per second (MB/s). Note Mb vs MB.
76 |
Software to binary
77 |
Since software is written in human readable code, there are a few different ways for a CPU to execute that code.
78 |
79 |
Compilation - Source code gets converted into binary or machine code by a compiler and saved as an executable.
80 |
Interpretted - The program executes at runtime by an interpretter that parses code line by line. It has the benefit of not having to be compiled, but is slower than running native machine code.
81 |
Just-in-time compilation - A combination of the two above approaches where the interpreter profiles the program and compiles the most frequently executed parts into native code at runtime.
82 |
83 |
Quantum computing
84 |
Qubits are 1s and 0s that rely on superposition to have states determined at the time of observation. This means that a qubyte (8 qubits) can have up to 256 different outcomes when executed at runtime.
The Command line interface (CLI) is a means of interacting with a computer program where the user (or client) issues commands to the program in the form of successive lines of text (command lines). Most users rely upon graphical user interfaces and menu-driven interactions with a mouse. However, many software developers, system administrators and advanced users still rely heavily on command-line interfaces to perform tasks more efficiently, configure their machine, or access programs and program features that are not available through a graphical interface.
91 |
A Unix shell is a command-line interpreter or shell that provides a command line user interface for Unix-like operating systems. Bash is a Unix shell and command language that can be used for executing commands from the CLI. Windows has their own their shell and command language. Install Git Bash on Windows to run these commands.
A computer network that spans a relatively small area connected via ethernet and wifi.
16 |
17 |
18 |
WAN
19 |
A computer network that spans a large area by connecting multiple LANs. The largest WAN in existence is the Internet.
20 |
21 |
22 |
VPN
23 |
Extends a private network across a public network, and enables users to send and receive data across shared or public networks as if their computing devices were directly connected to the private network.
24 |
25 |
26 |
IP address
27 |
A numerical label assigned to each device connected to a computer network that uses the Internet Protocol for communication. When communicating with devices within the network, a private IP address is used. When communicating with devices outside the network, a public IP address is used. Since all devices within a private network share the same public IP address, port forwarding and network address translation is used by the router to determine how to route data among devices. CIDR is a method for dividing networks into subnetworks.
28 |
29 |
30 |
31 |
32 |
33 | An example of a cloud VPN on AWS
34 |
35 |
36 |
The Internet protocol suite (TCP/IP) provides end-to-end data communication specifying how data should be packetized, addressed, transmitted, routed, and received. This functionality is organized into abstraction layers.
43 |
44 |
Link/physical
45 |
The link layer is used to move packets between the Internet layer interfaces of two different hosts on the same link using MAC addresses. This is also the layer where packets may be selected to be sent over a VPN. In this scenario, the link layer data may be considered application data allowing it to traverse over the layers again before being transmitted.
46 |
47 |
48 |
Internet
49 |
The primary protocol in this scope is the Internet Protocol, which defines IP addresses. Its function in routing is to transport datagrams to the next IP router that has the connectivity to a network closer to the final data destination.
50 |
51 |
52 |
Transport
53 |
Handling host-to-host communication. TCP provides reliable, ordered, and error-checked delivery of a stream of octets (bytes) between applications running on hosts communicating via an IP network. Applications that do not require reliable data stream service may use the User Datagram Protocol (UDP), which provides a connectionless datagram service that emphasizes reduced latency over reliability.
54 |
55 |
56 |
Application
57 |
Providing process-to-process data exchange for applications. This is the layer in which all higher level protocols, such as SMTP, FTP, SSH, HTTP, operate. Processes are addressed via ports which essentially represent services.
The client and server communicate by sending plain-text (ASCII) messages. The client sends requests to the server and the server sends responses.
78 |
The request message consists of the following:
79 |
80 |
a request line with an HTTP method, URI, and HTTP version (e.g., GET /images/logo.png HTTP/1.1, which requests a resource called /images/logo.png from the server.)
The Domain Name System (DNS) is a worldwide, distributed database for translating easily memorized domain names into IP addresses needed for communication over the internet. Each domain has at least one authoritative DNS server that stores records such as IP addresses (A and AAAA), SMTP mail exchangers (MX), name servers (NS), and domain name aliases (CNAME). Name servers "point" your domain name to the company that controls its DNS settings.
120 |
121 |
DNS resolution occurs by first looking up the name server for the TLD (.com or .net), then the name server for the full domain (medium.com), and lastly the numeric IP of the subdomain with domain (www.medium.com). To reduce the load on the Domain Name System servers, results are cached locally or in intermediate resolver hosts. A time to live (TTL) is included with the cached results, an expiration time after which the results must be discarded or refreshed.
A version-control system for tracking changes in computer files and coordinating work on those files among multiple people. It is used for source-code management in software development. As a distributed revision-control system, it is aimed at speed, data integrity, and support for distributed, non-linear workflows. Every Git directory on every computer is a full-fledged repository with complete history and full version-tracking abilities, independent of network access or a central server.
An approach to software development under which requirements and solutions evolve through the collaborative effort of self-organizing and cross-functional teams and their customer(s)/end user(s). It advocates adaptive planning, evolutionary development, early delivery, and continual improvement, and it encourages rapid and flexible response to change. One of the differences between agile software development methods and waterfall is the approach to quality and testing. In the waterfall model, there is always a separate testing phase after a build phase; however, in agile software development testing is completed in the same iteration as programming.
29 |
30 |
31 |
Scrum
32 |
An agile framework for managing knowledge work, with an emphasis on software development. It is designed for teams of three to nine members, who break their work into actions that can be completed within timeboxed iterations, called "sprints", no longer than one month and most commonly two weeks, then track progress and re-plan in 15-minute stand-up meetings, called daily scrums.
CICD refers to the combined practices of continuous integration and continuous delivery. Continuous integration (CI) is the practice of merging all developer working copies to a shared mainline several times a day. Continuous delivery (CD) is a software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time. It aims at building, testing, and releasing software with greater speed and frequency. The approach helps reduce the cost, time, and risk of delivering changes by allowing for more incremental updates to applications in production.
63 |
64 |
The process in AWS
65 |
66 |
67 |
Setup AWS CodePipeline to trigger on changes to a source control like Git or AWS CodeCommit.
68 |
Use a Buildspec.yml containing build commands that AWS Codebuild can use to automatically create a build environment and run your build.
69 |
Any build output will be put into an Amazon S3 bucket. AWS CodePipeline will trigger AWS CodeDeploy to deploy these artifacts to an EC2 instance.
70 |
AWS CodePipeline will trigger automated integration & acceptance testing in this EC2 instance. This is the staging environment.
71 |
After tests have been run, AWS CodePipeline should wait for a manual checkpoint. After reviewing test reports, the decision maker can approve AWS CodeDeploy to deploy to production.
AWS whitepaper on the key concepts and design principles for architecting in the cloud. The 5 pillars are:
84 |
85 |
Operational Excellence
86 |
The operational excellence pillar focuses on running and monitoring systems to deliver business value, and continually improving processes and procedures. Key topics include managing and automating changes, responding to events, and defining standards to successfully manage daily operations.
87 |
88 |
89 |
Security
90 |
The security pillar focuses on protecting information & systems. Key topics include confidentiality and integrity of data, identifying and managing who can do what with privilege management, protecting systems, and establishing controls to detect security events.
91 |
92 |
93 |
Reliability
94 |
The reliability pillar focuses on the ability to prevent, and quickly recover from failures to meet business and customer demand. Key topics include foundational elements around setup, cross project requirements, recovery planning, and how we handle change.
95 |
96 |
97 |
Performance Efficiency
98 |
The performance efficiency pillar focuses on using IT and computing resources efficiently. Key topics include selecting the right resource types and sizes based on workload requirements, monitoring performance, and making informed decisions to maintain efficiency as business needs evolve.
99 |
100 |
101 |
Cost Optimization
102 |
Cost Optimization focuses on avoiding un-needed costs. Key topics include understanding and controlling where money is being spent, selecting the most appropriate and right number of resource types, analyzing spend over time, and scaling to meet business needs without overspending.
106 |
107 |
--------------------------------------------------------------------------------
/content/devops/scalability.md:
--------------------------------------------------------------------------------
1 | +++
2 | description = "Containers, Serverless, performance"
3 | title = "Scalability"
4 | draft = false
5 | weight = 500
6 | bref="Scalability is the capability of a system, network, or process to handle a growing amount of work, or its potential to be enlarged to accommodate that growth"
7 | toc = true
8 | script = 'animation'
9 | +++
10 |
11 |
An operating system feature in which the kernel allows the existence of multiple isolated user-space instances called containers. Containers isolate software from its environment so that it may run uniformly on all machines.
A container is a runtime instance of an image--what the image becomes in memory when executed (an image with state). An image is an executable package that includes everything needed to run an application--the code, a runtime, libraries, environment variables, and configuration files. Containers are a key enabling technology for microservices, providing a lightweight encapsulation of each component so that it is easy to maintain and replicate.
35 |
Microservices
36 |
An architectural style that structures an application as a collection of loosely coupled services. The benefit of decomposing an application into different smaller services is that it improves modularity. This makes the application easier to understand, develop, test, and become more resilient to architecture erosion. It parallelizes development by enabling small autonomous teams to develop, deploy and scale their respective services independently. It also allows the architecture of an individual service to emerge through continuous refactoring.
37 |
38 |
39 |
40 |
41 |
42 |
Docker
43 |
A platform for developers and sysadmins to develop, deploy, and run applications with containers.
44 |
The Docker Client runs commands like docker build or docker run. The Docker Daemon (aka Host or Engine) makes the system calls to create, operate and manage containers. Many of the configurations in Docker image will from Docker Registry where it will be downloaded.
45 |
46 |
47 |
48 |
Swarm is a native clustering tool for Docker. Swarm pools together several Docker hosts and exposes them as a single virtual Docker host. In Swarm mode, there are two types of nodes: Managers and workers. Manager nodes maintain cluster state and schedule services. Worker nodes execute containers. Clustering is an important feature for container technology, because it creates a cooperative group of systems that can provide redundancy through failover and scalability through automation.
49 |
50 |
51 |
52 |
53 |
AWS ECS and Kubernetes are two alternate services for maintaining and coordinating clusters of container. Their functionlity is similar.
A cloud-computing execution model in which the cloud provider acts as the server, dynamically managing the allocation of machine resources. Pricing is based on the actual amount of resources consumed by an application, rather than on pre-purchased units of capacity. This allows for infrastructure to be provisioned and terminated quickly and cheaply.
60 |
61 |
Benefits
62 |
63 |
Pay per use - paying for use rather than capacity can be significantly cheaper depending on use case
64 |
Scalability - horizontal scaling is built into the platform
65 |
Faster development - functions can be built standalone, independent from the rest of the application.
66 |
Maintenance - vendor handles server upgrades and maintenance.
67 |
68 |
69 |
Drawbacks
70 |
71 |
3rd party API - vendor lock-in
72 |
Lack of operational tools - debugging and testing are difficult
73 |
Architectural complexity - as functions get more granular, the wiring between those functions increases exponentially.
74 |
Timeouts & Latency - becuase serverless functions run in ephemeral containers, there is increased latency when the container does not yet exist. In addition, there is a timeout which prevents long running functions from executing to completion.
75 |
76 |
77 |
78 |
Serverless framework
79 |
80 |
81 |
vendor agnostic (AWS, Azure, Google Cloud, etc)
82 |
language independent (Node, Java, Python, etc)
83 |
packages functions AND provisions resources
84 |
large community that builds excellent plugins
85 |
86 |
87 |
Infrastructure as code (IaC) - The process of managing and provisioning computer data centers through definition files.
88 |
89 |
Cost - aims at helping not only the enterprise financially, but also in terms of people and effort, meaning that by removing the manual component, people are able to refocus their efforts towards other enterprise tasks.
90 |
Speed - Infrastructure automation enables speed through faster execution when configuring your infrastructure and aims at providing visibility to help other teams across the enterprise work quickly and more efficiently.
91 |
Risk - Automation removes the risk associated with human error, like manual misconfiguration; removing this can decrease downtime and increase reliability.
92 |
93 |
94 |
Serverless.yml
95 |
Services - A property of the serverless.yml that names the project.
96 |
Provider - A property describing the cloud provider and default parameters for the rest of the file. The IAM role and permissions will be set here as well.
97 |
Functions - A property of the serverless.yml which lists the names of the function in your service. Inside the function is a handler pointing to the code. Functions can be nested inside other functions. Environment variables can be set as well as tagging. If the function fails, a DLQ (SNS) can be published.
98 |
Events - things that trigger your functions to run (an S3 bucket upload, an SNS topic, and HTTP endpoints created via API Gateway)
99 |
Layers - Code that you've used else where that you can import into the serverless function.
100 |
Resources - A property of the serverless.yml which defines the infrastructure your functions depend on, like AWS DynamoDB or AWS S3. Resources deployed by Serverless have the naming scheme - {Function Name}{Cloud Formation Resource Type}{Resource Name}{SequentialID or Random String}. Does not apply to S3.
101 |
102 |
Variables - allow users to dynamically replace config values in serverless.yml config with ${} notation.
103 |
104 |
Self reference - ${self:someProperty}
105 |
Other files - ${file(../myFile.json):someProperty}
106 |
In JS files - reference JavaScript files to add dynamic data into your variables. ${file(../myFile.js):someModule}
Environment Variables - ${env:SOME_VAR}. Keep in mind that sensitive information which is provided through environment variables can be written into less protected or publicly accessible build logs, CloudFormation templates, et cetera.
109 |
Referencing CLI Options -${opt:some_option}. A common CLI option is the dev stage.
110 |
CloudFormation Outputs - allows your service to communicate with other services/stacks using ${cf:stackName.outputKey}.
111 | Output names are added to the Export field in the resources property:
112 |
113 | Import into other services:
114 |
115 |
A load balancer is a device that acts as a reverse proxy and distributes network or application traffic across a number of servers. Autoscaling is when computational resources in a server farm, typically measured in terms of the number of active servers, scales automatically based on the load on the farm. When using containers, orchestration services Kubernetes and Amazon ECS handle load balancing and autoscaling for you.
130 |
CDN
131 |
A content delivery network (CDN) refers to a geographically distributed group of servers which work together to provide fast delivery of Internet content. A CDN allows for the quick transfer of static assets needed for loading Internet content including HTML pages, javascript files, stylesheets, images, and videos. Using a CDN improves website load times, reduces bandwidth costs, increases content availability and redundancy, and improves website security (e.g. DDoS mitigation).
132 |
133 |
134 |
135 |
Caching
136 |
A cache A hardware or software component that stores data so that future requests for that data can be served faster; the data stored in a cache might be the result of an earlier computation or a copy of data stored elsewhere. A CDN is an example of a cache when a user retrieves assets from an edge location rather than a distant origin server. Web browsers also uses caches that store static data rather than re-retrieve them. The timeframe for how long these items stay cached can be set by a cache header or service worker. For databases, Redis can be used to intercept requests and send cached data rather than querying the database additional times for the same data.
137 |
Data transfer
138 |
QUIC - An experimental transport layer network protocol that is built on top of UDP. It handles error handling without the handshaking that the higher latency protocol TCP has.
139 |
Compression - compress all static assets. Also make sure your server uses gzip - a file format and a software application used for file compression and decompression.
140 |
Lazy loading - Transfer data to the client as it is needed like a stream, rather than having them wait for the entire bundle to complete downloading.
141 |
142 |
Language selection
143 |
Java - For some tasks a different language might be necessary for performance gains. Java is an excellent language for computationally intensive operations. IO operations, on the other hand, might be better served using Node.js.
144 |
WebAssembly - a web standard that defines a binary format and a corresponding assembly-like text format for executable code in Web pages. It is meant to enable executing code nearly as quickly as running native machine code. Many believe WebAssembly could replace JavaScript entirely in the distant future.
148 |
149 |
--------------------------------------------------------------------------------
/content/devops/testing.md:
--------------------------------------------------------------------------------
1 | +++
2 | description = "TDD, Chrome DevTools, error tracking"
3 | title = "Testing & Debugging"
4 | draft = false
5 | weight = 200
6 | bref="Software testing is an investigation conducted to provide stakeholders with information about the quality of the software product or service under test. Debugging is the process of finding and resolving defects or problems within a computer program that prevent correct operation of computer software or a system"
7 | toc = true
8 | script = 'animation'
9 | +++
10 |
11 |
13 |
14 | #### Whenever possible, use TDD
15 |
16 | TDD is a _design process_, not a testing process. TDD is a robust way of designing software components ("units") interactively so that their behaviour is specified through unit tests.
17 |
18 | How? Why?
19 |
20 | #### Test-first cycle
21 |
22 | 1. Write a simple failing test
23 | 2. Make the test pass by writing the minimum amount of code
24 | 3. Refactor the code by applying design principles/patterns
25 |
26 | During phase 2, don't bother with quality.
27 |
28 | #### Consequences of the test-first cycle
29 |
30 | + Writing a test first makes the code design more testable
31 | + Writing just the amount of code needed to implement the required functionality makes the resulting codebase minimal, thus more maintainable
32 | + The codebase can be enhanced using refactoring mechanisms, the tests give you confidence that the new code is not modifying the existing functionalities
33 | + Cleaning the code in each cycle makes the codebase more maintainable, it is much cheaper to change the code frequently and in small increments
34 | + Fast feedback for the developers, you know that you don't break anything and that you are evolving the system in a good direction
35 | + Generates confidence to add features, fix bugs, or explore new designs
36 |
37 | Note that code written without a test-first approach is often very hard to test!
38 |
Individual units/components of a software are tested. A unit is the smallest testable part of any software. It usually has one or a few inputs and usually a single output.
46 |
Integration testing
47 |
Individual software modules are combined and tested as a group.
48 |
System testing
49 |
A complete and integrated software is tested for compliance with the specified requirements.
50 |
Acceptance testing
51 |
A system is tested for acceptability and ready for delivery.
52 |
53 |
54 |
55 |
56 | The V-model of software development identifies testing tasks for each stage of development.
57 |
58 |
Syntax or interpretation errors - These include mismatched or missing quotes, parentheses and braces, incorrect case, spelling mistakes, illegal characters, and more. These erros can usually be caught using linters like ESlint or JSLint. Linters analyze your code before it is executed and point out syntax and other common errors.
72 |
Runtime exceptions - These are the errors that occur when your JavaScript code executes. Such errors can be triggered by referring to an undefined variable, dividing by zero, by a failed “assert” statement, or by using a “throw” statement in your (or a JavaScript library’s) code to manually specify that an exception has occurred. When a runtime exception occurs, any JavaScript code after that line of code is not executed so they can leave your web page or application in an unexpected state. You can catch and handle runtime exceptions by wrapping the code that fails with a try {code} catch(exception) {} block.
73 |
Incorrect logic - does not show any errors but causes your code to not do what you intend it to. Debugging such errors requires some practice using debugging tools that your browser provides.
74 |
75 |
Chrome DevTools
76 |
77 |
78 |
Console - Use console.log() to dump statements and other useful information like stack variables. Use console.assert() to ensure that certain conditions or assumptions are being met in your code. This also dumps the execution call stack and continues execution after the assert.
79 |
Debugger - To specify a breakpoint, click on its line number in your code editor or in the Sources panel in Chrome DevTools. This will cause the debugger to pause at that line and allow you to inspect variables, the call stack, and evaluate expressions. With the debugger you can continue execution by stepping over the code line-by-line. You can set watches for variables and see how they change as you step over the code line by line. DevTools also has conditional breakpoints where you can specify the condition when the debugger stops execution.
80 |
Working with APIs - The Networks panel in DevTools shows you, for each network request, the “method” (e.g. GET, POST), the HTTP status code (e.g. 200, 403), the MIMEtype (e.g. application/json), the content’s size and whether the network request was fulfilled from the browser’s cache.
81 |
Tips - To ensure that your web page is using the latest version of your code, disable the browser cache in the DevTools settings. Also, when inspecting JavaScript objects, remember to use console.log(JSON.parse(JSON.stringify(obj)));.
Logging - the practice of storing large volumes of computer generated audit logs for later analysis. The creation of logs is dependent on the goals of the application. In addition, there are middleware packages like Morgan that can automate some of these processes. For reporting, modern logging systems use Elasticsearch to visualize logs and metrics using string analysis.
87 | Logging can also be used for:
88 |
89 |
Compliance with security policies
90 |
Compliance with audit or regulation
91 |
System troubleshooting
92 |
Forensics (during investigations or in response to subpoena)
93 |
Security incident response
94 |
Understanding online user behavior
95 |
96 |
97 |
Monitoring
98 |
It is important to monitor systems and send automated notifications to respond to system-wide performance changes. On AWS, CloudWatch provides a unified view of AWS resources, applications and services that run on AWS. You can set alarms, visualize logs and metrics, take automated actions, configure health checks, troubleshoot issues, and discover insights to optimize your applications, and ensure they are running smoothly. Another service essential monitoring system health is Sentry.io which provides client-side error notifications.
A free and open-source front-end framework for developing websites and web applications. It contains HTML- and CSS-based design templates for typography, forms, buttons, navigation and other interface components, as well as optional JavaScript extensions.
Bootstrap has styling classes for tables, margins, padding, borders, alignment, color, and much more. Steer clear of type selectors (e.g., input[type="text"]) and extraneous parent classes (e.g., .parent .child) that make styles too specific to easily override.
17 |
These components make use of the popper JavaScript files added to the HTML file. Bootstrap comes built-in with the most commmon components such as navbar, forms, and buttons.
21 |
22 |
23 |
Grid system
24 |
A mobile-first flexbox grid to build layouts of all shapes and sizes thanks to a twelve column system. Each column of a row contains a col class specifying how many of the 12 columns it can possibly take up (no more than 12 total). These classes can also take a prefix (e.g. .col-sm-, .col-lg-) specifying which devices they apply to. Media queries are optionally used to specify the breakpoints (in pixels) for these class prefixes.
25 |
26 |
27 |
28 |
29 |
30 |
z-index
31 |
For overlapping components, Bootstrap uses a modifiable z-index to determine priority of overlapping.
A fast, small, and feature-rich JavaScript library. It makes things like HTML document traversal and manipulation, event handling, animation, and Ajax much simpler with an easy-to-use API that works across a multitude of browsers.
These methods are used to register behaviors to take effect when the user interacts with the browser, and to further manipulate those registered behaviors.
Refers to using plain JavaScript without any additional libraries. Usually this means the code you write will be A LOT more lightweight and faster to load than the entire jQuery library. For example, AJAX requests can be written using XMLHttpRequest. See "You might not need jQuery" for vanilla JS alternatives to common jQuery functions.
SASS/SCSS - a CSS preprocessor that runs on the server and compiles to CSS code that your browser understands. There are client-side alternatives to SASS that can be compiled in the browser using javascript such as LESS CSS, though it is advisable to compile to CSS for production use.
78 |
CSS Grid Layout - Alternative to Bootstrap grid. CSS Grid layout divides a page into major regions by defining the relationship in terms of size, position, and layer, between parts of a control built from HTML primitives.
--------------------------------------------------------------------------------
/content/frontend/react.md:
--------------------------------------------------------------------------------
1 | +++
2 | description = "React, React Router, Redux"
3 | title = "React"
4 | draft = false
5 | weight = 300
6 | bref="React is a JavaScript library for building user interfaces. It is maintained by Facebook and a community of individual developers and companies."
7 | toc = true
8 | script = 'animation'
9 | +++
10 |
11 |
React DOM - React is rendered in a single root DOM node. Unlike browser DOM elements, React elements are plain objects, and are cheap to create. React DOM efficiently compares the element and its children to the previous one, and only applies the DOM updates necessary to bring the DOM to the desired state.
16 |
JSX - Instead of artificially separating markup and logic into different files, JSX can be used to create React "elements" that contain both. You can put any valid JavaScript expression inside the curly braces in JSX and it will be compiled back to regular JavaScript using a Babel script.
17 |
18 |
19 |
Components
20 |
Components are created using JSX and rendered using React DOM. React has functional and class components. Functional components do not have state and class based components do have state. Components can be composed from other smaller components.
21 |
22 |
23 |
Props
24 |
When React sees a user-defined component, any attributes in that tag are passed to the component in an object called “props”. Props are read-only and cannot be modified. Everything between JSX tags will be passed to the child component as props.children. Note that there are a few differences in React Attributes. Use className instead of class and htmlFor instead of for.
25 |
26 |
27 |
State
28 |
Similar to props, but it is private to the component and modifiable. It is initialized in the constructor method of the component class. A component may choose to pass its state down as props to its child components.
29 |
30 |
31 |
LifeCycle
32 |
Component lifecycle methods are called in the following order when an instance of a component is being created and inserted into the DOM: constructor(), render(), componentDidMount(). constructor() is called before the component is mounted and is the only place where you should assign this.state directly. Otherwise, use this.setState(). render() is the only required method in a class component. It should return a React element or nothing at all. componentDidMount() is invoked immediately after a component is mounted (inserted into the tree). Initialization that requires DOM nodes and loading data from a remote endpoint should go here. There are also other less commonly used lifecycle methods.
33 |
34 |
35 |
Handling Events
36 |
With JSX you pass a function as the event handler, rather than a string. See all of React's SyntheticEvents here. Use ES6 functions for simplifying the binding of this.
37 |
38 |
39 |
Conditional Rendering
40 |
You can create distinct components that encapsulate behavior you need. Then, you can render only some of them, depending on the state of your application. You can use an inline if with the && operator or condition ? true: false ternary operator.
41 |
42 |
43 |
Lists and keys
44 |
Use map() to transform arrays into React functional component. Keys help React identify which items have changed, are added, or are removed. Keys should be given to these elements to give them a stable identity. Pass these keys as props to the functional component.
45 |
46 |
47 |
Additional features
48 |
Refs - provide a way to access DOM nodes or React elements created in the render method.
49 |
Fragments - A common pattern in React for returning multiple elements. Usually this is done by appending 1 div node containing JSX to the DOM.
50 |
Higher Order Component (HOC) - a function that takes a component and returns a new component. It is used for reusing logic by wrapping the original component with additional functionality.
51 |
Type checking - As your app grows, you can catch a lot of bugs with typechecking. You can use JavaScript extensions like Flow or TypeScript or the built-in propTypes.
52 |
53 |
54 |
Why I prefer React to Angular
55 |
JSX > TypeScript - JSX is easy to learn and allows for combining of markup and logic.
56 |
Virtual DOM > MVC - Unidirectional data flow from parent to child component is easy to understand and predictable as opposed to Angular's two-way binding.
57 |
Library > framework - React's library is flexible to use many open source packages. Angular's framework comes built-in with many components (forms, router, etc.)and a lot of packages are incompatible with TypeScript.
Offers a way to write your code so that it will show certain components of your app only if the route matches what you define. The 3 components you will interact the most when working with React Router are: BrowserRouter, Link, and Route. BrowserRouter wraps all your Route components. Link is used to generate links to your routes. Route is responsible for showing the components they contain.
67 |
React Router offers a way to match on URL params using : before the param. It also allows for nested routing. For this to work, the component needs to be wrapped in a higher order component from React Router called withRouter.
68 |
69 |
70 |
State management
71 |
Lifting state up - If several components need to reflect the same changing data, the shared state can be lifted to their closest common ancestor. The common ancestor will pass a function down to its child components. The child component will invoke the parent component's function with the common data that needs to be changed. The lifecycle method will fire and the child components will be re-rendered with updated props.
72 |
Redux - Lifting state can be tedious in a complex app so many developers prefer to use a library called Redux for state management. Provides <Provider/> which makes the Redux store available to the rest of the app. It also provides a connect function for components to connect to the store via mapStateToProps and mapDispatchToProps.
73 |
74 |
75 |
76 |
77 |
78 |
79 |
80 |
81 |
82 |
83 |
84 |
85 |
Context - A new feature of React that provides a way to share values like these between components without having to explicitly pass a prop through every level of the tree. In the parent component, create a context with React.createContext() and create a provider that returns the context.Provider. The child component can access the context through context.Consumer.
86 |
Redux vs Context - React Redux uses context internally but it doesn’t expose this in the public API. So you should feel much safer using React Redux because if it changes, the burden of updating the code will be on React Redux and not you. However, with a simple app, it might be easier to implement Context quickly.
87 |
88 |
89 |
Tools
90 |
Create React App - a toolchain used for rapid development. It includes a package manager (NPM), bundler (webpack), compiler (Babel), boilerplate code, and development environment for quickly writing code to production. Create React App produces a Single Page Application (SPA) that loads a single HTML page and dynamically updates that page as the user interacts with the app. Usually, the entire website bundle must be downloaded before anything is displayed (client side rendering), but there are ways of lazy loading (downloading when needed) for a faster client experience.
91 |
Gatsby - a static site generator that lets you use React components and outputs pre-rendered HTML/CSS. If your application doesn't contain dynamic content, using Gatsby will guarantee the fastest load time.
92 |
Next.js - a popular and lightweight framework for server‑rendered applications built with React. See diagram below for difference between server and client rendering
93 |
React Native - A framework for building native applications using JavaScript. React Native compiles to native app components, which makes it possible for you to build native mobile applications. In React JS, React is the base abstraction of React DOM for the web platform, while with React Native, React is still the base abstraction but of React Native. So the syntax and workflow remain similar, but the components are different.
--------------------------------------------------------------------------------
/content/languages/html_css.md:
--------------------------------------------------------------------------------
1 | +++
2 | description = "Markup language and styling"
3 | title = "HTML & CSS"
4 | draft = false
5 | weight = 100
6 | bref="HTML (the Hypertext Markup Language) and CSS (Cascading Style Sheets) are two of the core technologies for building Web pages. HTML provides the structure of the page, CSS the (visual and aural) layout, for a variety of devices"
7 | toc = true
8 | script = 'animation'
9 | +++
10 |
11 |
HTML should be used for structure and semantics to order the content and to specify its intended use (e.g. - distinguishing between the main body of an article and the links to other sections)
JavaScript is a powerful language that can do almost any type of programming (functional, object, imperative). It's growth has made it the most popular language used today. Thousands of libraries and frameworks are available to simplify the task of coding. JavaScript can be run on the client or the server, eliminating the need to switch between different languages and making it the perfect web development tool.
Containers for storing data values. Data values can be any of the 6 primitive data types or an object.
23 |
24 |
25 |
Primitive data types
26 |
Boolean - "true" or "false"
27 |
Null - the intentional absence of any value
28 |
Undefined - indicates that a variable has not been assigned a value, or not declared at all
29 |
Number - integer or float
30 |
String - a series of characters inside double or single quotes
31 |
Symbol - this data type is used as the key for an object property when the property is intended to be private, for the internal use of a class or an object type
32 |
33 |
34 |
Object
35 |
A non-primitive data type containing key-value pairs where the key is a string (also called “property name”), and value can be anything. Arrays, functions, dates, and errors are all of type object. Copying a variable referencing a primitive data type will pass the value to the new variable. However, because objects are a non-primitive data type, copying a variable yields a reference to the original object (NOT the value).
36 |
37 |
38 |
Array
39 |
An object containing an indexed list of values of any data type.
40 |
41 |
42 |
Function
43 |
An object containing a block of code designed to perform a particular task. It is executed when invoked with (). The function completes with a return statement. If the function has parameters, invoking it will pass an array named arguments to the function. Because functions are objects, they can themselves be passed as arguments to other functions. Passing functions as arguments enables asynchronous programming.
44 |
45 |
46 |
Class
47 |
An object containing a "blueprint" for creating many objects of the same "type". Classes contain a constructor function that is invoked when using the new keyword. Instances created through a class definition will inherit methods (functions) and properties of the parent class object through the prototype chain.
48 |
49 |
50 |
Conditionals
51 |
JavaScript has an "if/else if/else" statement, a ternary operator, and a switch statement.
52 |
53 |
54 |
Loops
55 |
A block of code useful for running the same code over and over again, each time with a different value. Often this is the case when working with arrays or objects. Use a for loop for looping every item in an array. "For/in" loops through the properties of an object. "While" loops through a block of code while a specified condition is true.
JavaScript's strict mode that eliminates some silent errors by changing them to throw errors. For example, an error will be thrown when using undeclared variables.
69 |
70 |
71 |
"this" keyword
72 |
As soon as the program runs, "this" is defaulted to the global object. As the program executes, "this" will change to the object it belongs to, which might be different each time the function is called. In a browser event, this refers to the element that received the event. There are call(), bind(), and apply() methods to set the value of a function's "this" regardless of how it's called. There are also arrow functions which don't provide their own this binding (it retains the this value of the enclosing lexical context).
73 |
74 |
75 |
Closures & Scopes
76 |
A closure is the combination of a function and the lexical scope within which that function was declared. Variables defined inside a function are not accessible from outside the function. Variables defined outside the function are accessible inside the function.
77 |
78 |
79 |
Hoisting
80 |
JavaScript's default behavior of moving declarations to the top. This occurs during the compilation step when the JavaScript engine creates the program's scopes before the interpreter executes the code.
81 |
82 |
83 |
Floating points
84 |
Unlike many other programming languages, JavaScript does not define different types of numbers, like integers, short, long, floating-point etc. JavaScript numbers are always stored as double precision floating point numbers which means that 1/10 in base 2 is a repeating decimal that will be rounded.
85 |
86 |
87 |
ECMAScript versions
88 |
A specification to standardize JavaScript with updates to the language every year. As a result, the latest features of JavaScript might not be able to run in older runtime environments. Fortunately, tools like Webpack and Babel can transpile JavaScript. The transpiler converts the latest versions of JavaScript into earlier versions like ES5 that are compatible with almost all runtimes.
89 |
90 |
91 |
Type coercion
92 |
The process of converting values from one type to another (such as string to number, object to boolean, and so on). See chart below for examples. Use === to check for equality AND type. Many developers prefer a statically typed language for issues like this. Using TypeScript adds static typing and other useful features before being transcompiled into JavaScript.
Python has many open-source libraries and tools that make doing certain tasks and working with data much easier than trying to do them in JavaScript. Luckily, once you learn the basics of one high level programming language, it is not too difficult to code in a new language. Beware of the whitespaces!
16 |
NumPy
17 |
NumPy is a library that provides a multidimensional array object, various derived objects (such as masked arrays and matrices), and an assortment of routines for fast operations on arrays, including mathematical, logical, shape manipulation, sorting, selecting, I/O, discrete Fourier transforms, basic linear algebra, basic statistical operations, random simulation and much more. SciPy extends NumPy functionality and provides many other numerical algorithms.
18 |
Pandas
19 |
Provides providing fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive. It is suited for tabular data, time series data, arbitrary matrix data, and any other form of observational/statistical data sets.
20 |
Matplotlib
21 |
Used to create 2D graphs and plots. It supports a very wide variety of graphs and plots namely - histogram, bar charts, power spectra, error charts etc. Seaborn is essentially a higher-level API based on the matplotlib library.
22 |
PyAutoGUI
23 |
A Python module for automation and programmatically controlling the mouse and keyboard.
24 |
Scrapy
25 |
Build and run your web spiders. Then deploy them to Scrapy Cloud.
26 |
Zipline
27 |
An algorithmic trading library for backtesting.
28 |
Keras
29 |
A high-level library for working with neural networks, running on top of the TensorFlow library.
30 |
NLTK
31 |
NLTK is a set of libraries, a whole platform for natural language processing.
48 |
--------------------------------------------------------------------------------
/content/latest/big_data.md:
--------------------------------------------------------------------------------
1 | +++
2 | description = "Hadoop, Spark, Streams"
3 | title = "Big data"
4 | draft = false
5 | weight = 300
6 | bref="Big data is a term used to refer to data sets that are too large or complex for traditional data-processing application software to adequately deal with"
7 | toc = true
8 | script = 'animation'
9 | +++
10 |
11 |
Techniques for analyzing data, such as A/B testing, machine learning and natural language processing
18 |
Big data technologies, like business intelligence, cloud computing and databases
19 |
Visualization, such as charts, graphs and other displays of the data
20 |
21 |
Architecture
22 |
Hadoop uses Hadoop Distributed File System (HDFS). HDFS is used to scale a single cluster to hundreds (and even thousands) of nodes. It is a distributed and highly fault-taolerant file system that handles large data sets running on commodity hardware. Spark initially reads from a file on HDFS, S3, or another filestore, into an established mechanism called the SparkContext. Out of that context, Apache Spark creates a structure called a DataFrame (an extension of the Resilient Distributed Dataset RDD with named columns), to represent an immutable collection of elements that can be operated on in parallel.
23 |
24 | Hadoop uses YARN to split up the functionalities of resource management and job scheduling/monitoring into separate daemons. The Resource Manager accepts job submissions and allocates resources for the first container. The Node Manager is the per-machine framework agent responsible for monitoring container usage and reporting it back to the Resource Manager. The per-application Application Master is tasked with negotiating resources from the ResourceManager and working with the NodeManager(s) to execute and monitor the tasks.
25 |
26 |
27 |
28 |
29 | Hadoop YARN
30 |
31 |
32 |
Spark vs Hadoop MapReduce
33 |
With MapReduce, queries are split and distributed across parallel nodes and processed in parallel (the Map step). The results are then gathered and delivered (the Reduce step). Apache Spark was developed in response to limitations in the MapReduce paradigm, as it adds the ability to set up many operations (not just map followed by reduce). In addition, Spark computations are carried out in memory and stored there, until the user actively persists them, whereas Hadoop MapReduce writes to a disk, making it up to 100 times slower. Spark also creates a Directed Acyclic Graph (DAG), to visualize the order of operations and the relationship between the operations in the DAG. DAGs enable optimizations between steps allowing for it to be up to 10 times faster on disk than Hadoop. However, the volume of data processed also differs: Hadoop MapReduce is able to work with far larger data sets than Spark and is the more economical solution when speed is not critical.
50 | Services like AWS Kinesis and Kafka are designed for large scale data ingestion and processing, with the ability to maximize write throughput for large volumes of data. Typical data streams include log files, e-commerce analytics, in-game player activity, information from social networks, financial trading floors, or geospatial services, and telemetry from connected devices or instrumentation in data centers.
51 |
52 |
Pubsub
53 |
A stream is a continuous inbound flow of messages. The messages themselves are encapsulated (individual) and received one at a time in quick succession. The workflow includes one or more message producers and any number of message consumers. This is commonly known as publish/subscribe or “pubsub”. When a message is published, the subscribers receive it nearly instantaneously.
54 |
55 |
56 |
57 |
Processing
58 |
Streams are processed differently from batch data – normal functions cannot operate on streams as a whole, as they have potentially unlimited data.
59 |
60 |
61 |
62 |
Idempotence
63 |
An idempotent operation is one that has no additional effect if it is called more than once with the same input parameters. Stream services usually provide "at least once" delivery to handle failed operations. This means that there should be some sort of "rollback" for mid-operation failure.
You and your neighbors ("nodes") have a spreadsheet of transactions (a "ledger"). Public accountants ("miners") have the same spreadsheet (it’s "distributed"). When any neighbor makes a new transaction, every accountant is notified by email. The accountants verify transactions 24/7. Every hour, each accountant submits totals in a new tab (a "block") and their calculations are provided ("Proof of Work"). If the majority of accountants agree, everyone updates their workbook ("blockchain") and the first to report it accurately gets their salary ( "mining reward"). The accountants use Excel to open spreadsheets, which Microsoft ("developers") updates every year.
Similar to Bitcoin, miners are tasked with solving a complex mathematical problem in order to successfully “mine” a block. Any computational problem that requires orders of magnitude more resources to solve algorithmically than it takes to verify the solution is a good candidate for proof of work (factorization of the product of two large prime numbers). In order to discourage centralisation due to the use of specialised hardware, as has occurred in the Bitcoin network, Ethereum chose a memory-hard computational problem which makes Ethereum’s Proof of Work ASIC-resistant.
is a prograddmmable blockchain. Rather than give users a set of pre-defined operations (e.g. bitcoin transactions), Ethereum allows users to create their own operations of any complexity they wish. In this way, it serves as a platform for many different types of decentralized blockchain applications, including but not limited to cryptocurrencies. Exchanges of any complexity could be carried out automatically and reliably using code running on Ethereum. Whereas the Bitcoin blockchain was purely a list of transactions, Ethereum’s basic unit is the account. The Ethereum blockchain tracks the state of every account, and all state transitions on the Ethereum blockchain are transfers of value and information between accounts. There are two types of accounts:
40 |
41 |
Externally Owned Accounts (EOAs), which are controlled by private keys
42 |
Contract Accounts, which are controlled by their contract code and can only be “activated” by an EOA
43 |
44 |
45 |
EVM
46 |
At the heart of it is the Ethereum Virtual Machine (“EVM”), which can execute code of arbitrary algorithmic complexity. Ethereum is “Turing complete” and includes a peer-to-peer network protocol. Each and every node of the network runs the EVM and executes the same instructions in order to maintain consensus across the blockchain. Decentralized consensus gives Ethereum extreme levels of fault tolerance, ensures zero downtime, and makes data stored on the blockchain forever unchangeable and censorship-resistant. Users must pay transaction fees to the nodes that validate the network. This protects the Ethereum blockchain from frivolous or malicious computational tasks, like DDoS attacks or infinite loops. The sender of a transaction must pay for each step of the “program” they activated, including computation and memory storage.
A language similar to JavaScript which is used for developing Ethereum smart contracts.
55 |
56 |
Smart Contract
57 |
A smart contract is a computer protocol intended to digitally facilitate, verify, or enforce the negotiation or performance of a contract. Smart contracts allow the performance of credible transactions without third parties. These transactions are trackable and irreversible. The aim of smart contracts is to provide security that is superior to traditional contract law and to reduce other transaction costs associated with contracting.
58 |
Flaws with Ethereum contracts
59 |
60 |
61 |
Ambiguities and easy-but-insecure constructs
62 |
Compiler bugs
63 |
Ethereum Virtual Machine bugs
64 |
Attacks on the blockchain network
65 |
The immutability of bugs
66 |
No central source documenting known vulnerabilities
67 |
68 |
69 |
Development Tools
70 |
71 |
Truffle is a tool for compiling Solidity contracts into EVM bytecode and deploying them to Ethereum networks. Also creates ABI interface for interacting with contract.
72 |
Web3 is a library used by JavaScript apps for programmatic access to contracts deployed on the Ethereum network.
73 |
MetaMask is a chrome extension users must have in their browser to transact with Ethereum applications on the web.
74 |
75 |
To get started with Solidity, you can use [Remix](https://remix.ethereum.org/), which is an
76 | browser-based IDE. Here are some example contracts:
77 |
78 | 1. [Voting](https://solidity.readthedocs.io/en/v0.4.24/solidity-by-example.html#voting)
79 | 2. [Blind Auction](https://solidity.readthedocs.io/en/v0.4.24/solidity-by-example.html#blind-auction)
80 | 3. [Safe remote purchase](https://solidity.readthedocs.io/en/v0.4.24/solidity-by-example.html#safe-remote-purchase)
81 | 4. [Micropayment Channel](https://solidity.readthedocs.io/en/v0.4.24/solidity-by-example.html#micropayment-channel)
82 |
83 |
Solidity vs JavaScript
84 |
85 |
86 |
Version Pragma - Source files can (and should) be annotated with a version pragma to reject being compiled with future compiler versions that might introduce incompatible changes.
87 |
Static typing - Argument and variable datatypes need to be declared
88 |
New types
89 |
90 |
Address - holds a 20 byte value (size of an Ethereum address) and serve as a base for all contracts.
91 |
Function modifiers - automatically check a condition prior to executing the function
92 |
Events - for interacting with EVM
93 |
Structs - custom defined types that can group several variables
94 |
Enums - sequential set of integer values
95 |
Mappings - hash tables which consist of key types and corresponding value type pairs
96 |
Time units - Suffixes like seconds, minutes, hours, days, weeks and years after literal numbers can be used to convert between units of time where seconds are the base unit
97 |
98 |
Data location - Every complex type, i.e. arrays and structs, has an additional annotation, the “data location”, about whether it is stored in memory (which is not persisting) or in storage (where the state variables are held).
99 |
Gas - each transaction is charged with a certain amount of Ether, whose purpose is to limit the amount of work that is needed to execute the transaction and to pay for this execution.
18 | A collection of software engineering notes published for the benefit of my students at University of Miami's coding bootcamp by Russell Rosario
20 |
35 | 
86 | 
92 | 
107 | 
132 | 
25 | 
43 | 
55 | 
63 | 
105 | 
61 | 
64 | 
72 | 
77 | 
85 | 
33 | 
58 | 
68 | 
93 | 
32 | 
64 | 
69 | 
124 | 
20 | 
56 | 
76 | 
18 | 
26 | 
40 | 
47 | 
52 | 
115 | 
134 | 
54 | 
15 |
16 |
17 | 
18 |
--------------------------------------------------------------------------------
/content/files/regex.md:
--------------------------------------------------------------------------------
1 | +++
2 | description = "String manipulation in JavaScript"
3 | title = "Regular Expressions"
4 |
5 | draft = false
6 | weight = 200
7 | bref="A regular expression, regex or regexp is, in theoretical computer science and formal language theory, a sequence of characters that define a search pattern"
8 | toc = false
9 | script = 'animation'
10 | +++
11 |
12 | 
13 | 
--------------------------------------------------------------------------------
/content/frontend/frameworks.md:
--------------------------------------------------------------------------------
1 | +++
2 | description = "Bootstrap & jQuery"
3 | title = "Frontend tools"
4 | draft = false
5 | weight = 400
6 | bref="JavaScript frameworks provide tooling to efficiently handle commonly faced problems in client-side development"
7 | toc = true
8 | script = 'animation'
9 | +++
10 |
11 | 
27 | 
60 | 
77 | 
82 | 
97 | 
16 | 
21 | 
26 | 
31 | 
41 | 
95 | 
39 | 
40 | 
46 | 
27 | 
36 | 
56 | 
61 | 
21 | 
30 |