├── books ├── computer-networking.md ├── database-management-system │ ├── chapter-16-recovery-system.md │ ├── chapter-17-system-architecture.md │ └── chapter-19-distributed-database.md ├── howto-win-friends-and-influence-people │ ├── how-to-win-friends-and-influence-people.md │ └── shortlist.md ├── modern-operating-system │ └── chapter-02-processes-and-threads.md ├── open-vpn.md ├── operating-system-concepts │ ├── chapter-03-processes.md │ ├── chapter-07-main-memory.md │ ├── chapter-08-virtual-memory.md │ ├── chapter-09-file-system-interface.md │ ├── chapter-10-file-system-implementation.md │ ├── chapter-11-mass-storage-structure.md │ └── images │ │ ├── base_limit.png │ │ ├── disk.png │ │ ├── dynamic_relocation.png │ │ ├── fcb.png │ │ ├── inode_index_block.png │ │ ├── job_queue.png │ │ ├── memory_mapping.png │ │ ├── paging_hardware.png │ │ ├── process_control_block.png │ │ ├── queuing_diagram.png │ │ └── tlb.png ├── teach-yourself-microsoft-sql-in-10-minutes.md └── transaction-processing.md ├── books_toc └── computer_networking_a_top_down_approach.md ├── computer-architecture.md ├── coursera └── cloud-computing-concepts-part-1 │ ├── week1.md │ ├── week2.md │ ├── week3.md │ └── week4.md ├── database ├── indexing.md ├── mongodb-design.md ├── mongoose.md ├── mysql │ └── mysql.md ├── sql.md └── write-ahead-logging.md ├── design-patterns └── index.md ├── devops ├── devop_checklist.md ├── docker.md ├── gcloud-commands.md └── gcloud.md ├── distributed-system ├── consistent-hashing.md ├── distributed-algorithms.md ├── distributed-system.md ├── images │ └── hash-ring.png ├── micro-service.md ├── raft.md ├── system-design.md └── time-calculation.md ├── favorite_blogposts.md ├── images ├── common-text-function-sql.png ├── computer-networking-go-back-n-protocol.png ├── computer-networking-packet-arrival-protocol.png ├── eee452.001.png ├── join.jpg ├── process-control-block-field.png ├── process_in_memory.png ├── process_states.png ├── proxy.png ├── pthread-function-calls.png ├── thread-sharing.png └── where-clause-operators.png ├── information_retrieval.md ├── language ├── cpp │ └── basics.md ├── css │ ├── bootstrap4.md │ └── element-ui.md ├── golang │ ├── assets │ │ └── markdown-img-paste-20180718120834171.png │ ├── context.md │ ├── embedding.md │ ├── golang.md │ ├── goroutine.md │ ├── interface.md │ ├── map.md │ ├── mutex.md │ ├── package.md │ ├── pprof.md │ ├── reflection.md │ └── slices.md ├── java │ └── java.md ├── javascript │ ├── async-await.md │ ├── error-handling.md │ ├── jquery.md │ ├── lodash.md │ ├── puppeteer.md │ ├── redux.md │ └── string.md ├── latex.md ├── python │ ├── csv.md │ ├── django.md │ ├── general.md │ ├── modules.md │ ├── pip.md │ ├── regex.md │ ├── script.md │ ├── virtualenv.md │ ├── wagtail.md │ └── web-scraping.md └── shell │ ├── bash.md │ └── shell.md ├── linux ├── linux.md ├── nginx.md └── ssh.md ├── machine-learning └── support-vector-machine.md ├── math ├── group-theory.md ├── lagrange-multiplier.md └── partial-derivative.md ├── networking ├── basics │ ├── assets │ │ ├── markdown-img-paste-20180716225230715.png │ │ ├── markdown-img-paste-20180716232316916.png │ │ ├── markdown-img-paste-20180716233624749.png │ │ ├── markdown-img-paste-20180716233642942.png │ │ ├── markdown-img-paste-20180716235137178.png │ │ ├── markdown-img-paste-20180717000859836.png │ │ ├── markdown-img-paste-20180717001634130.png │ │ ├── markdown-img-paste-20180717001913286.png │ │ ├── markdown-img-paste-20180717002216360.png │ │ ├── markdown-img-paste-20180718134607514.png │ │ ├── markdown-img-paste-2018071813491995.png │ │ └── markdown-img-paste-20180718135410900.png │ ├── hub.md │ ├── ports_nat.md │ ├── router.md │ └── switch.md ├── dos.md ├── http.md ├── ip-address.md ├── misc │ └── unix_domain_socket.md ├── network-address-translation.md ├── network-layer │ ├── assets │ │ ├── 2018-08-25-00-56-35.png │ │ ├── 2018-08-25-11-11-39.png │ │ ├── 2018-08-25-11-11-51.png │ │ ├── 2018-08-25-11-21-22.png │ │ ├── 2018-08-25-11-42-56.png │ │ ├── 2018-08-26-11-16-24.png │ │ ├── 2018-08-26-11-26-38.png │ │ ├── 2018-08-26-11-29-01.png │ │ ├── 2018-08-26-22-04-25.png │ │ └── 2018-08-26-22-14-46.png │ └── network-layer.md ├── networking.md ├── proxy.md ├── tcp-splitting.md └── transport_layer │ ├── assets │ ├── 2018-08-12-12-00-56.png │ ├── 2018-08-12-12-52-24.png │ ├── 2018-08-14-00-22-04.png │ ├── 2018-08-14-20-50-36.png │ ├── 2018-08-14-20-58-40.png │ ├── 2018-08-14-20-59-10.png │ ├── 2018-08-14-21-09-20.png │ ├── 2018-08-14-21-10-30.png │ ├── 2018-08-14-21-18-38.png │ ├── 2018-08-14-21-18-53.png │ ├── 2018-08-16-09-27-37.png │ ├── 2018-08-16-09-43-45.png │ ├── 2018-08-19-11-52-32.png │ ├── 2018-08-19-11-53-14.png │ ├── 2018-08-19-12-01-24.png │ ├── 2018-08-19-12-03-07.png │ ├── 2018-08-19-12-45-03.png │ ├── 2018-08-19-12-51-49.png │ ├── 2018-08-19-12-52-01.png │ ├── 2018-08-19-18-54-33.png │ ├── 2018-08-19-18-59-59.png │ ├── 2018-08-19-20-55-48.png │ ├── 2018-08-20-20-19-39.png │ ├── 2018-08-20-20-35-29.png │ ├── 2018-08-20-20-38-34.png │ ├── 2018-08-20-20-39-22.png │ ├── 2018-08-20-20-39-32.png │ └── 2018-08-24-20-47-03.png │ ├── reliable-data-transfer.md │ ├── tcp.md │ ├── transportlayer.md │ └── udp.md ├── nsu ├── cse331 │ ├── images │ │ └── mcu-vs-mpu.png │ └── lec1.1-microcontroller.md ├── eee111 │ ├── images │ │ ├── bias.png │ │ ├── diode.png │ │ └── np-junction.png │ └── lecture-1-diodes.md ├── eee452.md ├── eng115.md ├── his102.md ├── mat350.md └── phi104 │ ├── presentation.md │ ├── slide1.md │ ├── slide10.md │ ├── slide11.md │ ├── slide12.md │ ├── slide13.md │ ├── slide14.md │ ├── slide15.md │ ├── slide2.md │ ├── slide3.md │ ├── slide4.md │ ├── slide5.md │ ├── slide6.md │ ├── slide7.md │ ├── slide8.md │ └── slide9.md ├── operating-system ├── address-translation.md ├── advanced-page-tables.md ├── assets │ ├── base-and-bound.gif │ ├── markdown-img-paste-20180501093342227.png │ ├── markdown-img-paste-20180501122536680.png │ ├── markdown-img-paste-20180503220643690.png │ ├── markdown-img-paste-20180503223513809.png │ ├── markdown-img-paste-20180615003730499.png │ ├── markdown-img-paste-20180615011422807.png │ ├── markdown-img-paste-20180615012425329.png │ ├── markdown-img-paste-20180615145313928.png │ ├── markdown-img-paste-20180615150529748.png │ └── process-and-its-address-space.png ├── cache.md ├── consistency.md ├── operating-system.md ├── paging.md ├── segmentation.md └── tlb.md └── tools ├── assets ├── 2018-08-02-15-55-08.png ├── 2018-08-02-15-57-56.png └── markdown-img-paste-20180717103331600.png ├── atom.md ├── awk.md ├── bazel.md ├── docker-compose.md ├── elasticsearch.md ├── git.md ├── npm.md ├── protobuf.md ├── regex.md ├── tmux.md └── vim.md /books/database-management-system/chapter-17-system-architecture.md: -------------------------------------------------------------------------------- 1 | # Part 5: System Architecture 2 | 3 | # Chapter 17: Database System Architectures 4 | 5 | ## Centralized database system 6 | 7 | Single PC and single user. Processors have **Coarse Granularity Parallelism**. 8 | 9 | Even though PC has multiple cores, DB does not try to split query to multiple processor, instead it runs multiple queries concurrently. Hence, query throughput increases but individual query time does not decrease. 10 | 11 | ## Server Client System 12 | 13 | Client system simply consists of the front-end: user form, GUI, data mining tools and etc. 14 | 15 | The backend contains the server. 16 | 17 | **Transaction Server**: Transaction-server systems, also called query-server systems, provide an interface to which clients can send requests to perform an action, in response to which they execute the action and send back results to the client. 18 | 19 | Transaction Server consists of multiple processes accessing data in shared memory space: 20 | 21 | 1. **Server process**: These are processes that receive user queries (transactions), execute them, and send the results back. 22 | 1. **Lock manager process**: This process implements lock manager functionality, which includes lock grant, lock release, and deadlock detection. 23 | 1. **Database writer process**: There are one or more processes that output modified buffer blocks back to disk on a continuous basis. 24 | 1. **Log writer process**: This process outputs log records from the log record buffer to stable storage. 25 | 1. **Checkpoint process**: This process performs periodic checkpoints. 26 | 1. **Process monitor process**: This process monitors other processes, and if any of them fails, it takes recovery actions for the process, such as aborting any transaction being executed by the failed process, and then restarting the process. 27 | 28 | The shared memory contains all shared data, such as: 29 | 30 | - Buffer pool 31 | - Lock table 32 | - Log buffer 33 | - Cached query plans 34 | 35 | All database processes can access the data in shared memory. Since multiple processes may read or perform updates on data structures in shared memory, there must be a mechanism to ensure that a data structure is modified by at most one process at a time, and no process is reading a data structure while it is being written by others. Such mutual exclusion can be implemented by means of operating system functions called **semaphores**. 36 | 37 | ## Parallel Systems 38 | 39 | **Fine Grained vs Coarse Grained** 40 | 41 | | Fine Grained | Coarse Grained | 42 | | :------------- | :------------- | 43 | | program split into large number of small tasks | program split into large tasks. | 44 | | small computation |large amount of computation| 45 | | higher communication and synchronization cost | lower cost | 46 | | facilitates load balancing | load imbalance | 47 | | uses shared memory | uses message passing | 48 | 49 | **Speed up vs Scale up**: Running a given task in less time by increasing the degree of parallelism is called speedup. Handling larger tasks by increasing the degree of parallelism is called scaleup. 50 | 51 | 806 52 | -------------------------------------------------------------------------------- /books/database-management-system/chapter-19-distributed-database.md: -------------------------------------------------------------------------------- 1 | # Chapter 19 Distributed Databases 2 | 3 | # 19.1 Homogeneous and Heterogeneous Databases 4 | 5 | In a homogeneous distributed database system, all sites have identical database- management system software, are aware of one another, and agree to cooperate in processing users’ requests. 6 | 7 | We will deal with homogeneous distributed databases only. 8 | 9 | # 19.2 Distributed Data Storage 10 | 11 | There are two ways to distribute a database: 12 | 13 | 1. Replication 14 | 1. Fragmentation 15 | 16 | The two ways can be combined. 17 | 18 | ## 19.2.1 Data Replication 19 | 20 | 1. Availability 21 | 2. Increased Parallelism: If majority of the query is read, then multiple nodes can provide result. 22 | 3. Increased overhead on update 23 | 24 | ## 19.2.2 Data Fragmentation 25 | 26 | There are two different schemes for fragmenting a relation: horizontal fragmentation and vertical fragmentation. Horizontal fragmentation splits the relation by assigning each tuple of r to one or more fragments. Vertical fragmentation splits the relation by decomposing the scheme R of relation r. 27 | 28 | Horizontal fragmentation is usually used to keep tuples at the sites where they are used the most, to minimize data transfer. 29 | 30 | ## 19.2.3 Transparency 31 | 32 | The user of a distributed database system should not be required to know where the data are physically located nor how the data can be accessed at the specific local site. This property is called **data transparency**. 33 | 34 | 1. Fragmentation transparency 35 | 1. Replication transparency 36 | 1. Location transparency 37 | 38 | **Unique Names** 39 | 40 | Data items—such as relations, fragments, and replicas—must have unique names. This property is easy to ensure in a centralized database. In a distributed database, however, we must take care to ensure that two sites do not use the same name for distinct data items. 41 | 42 | One solution to this problem is to require all names to be registered in a central **name server**. The name server helps to ensure that the same name does not get used for different data items. We can also use the name server to locate a data item, given the name of the item. 43 | 44 | This approach, however, suffers from two major disadvantages. 45 | 1. the name server may become a performance bottle- neck when data items are located by their names, resulting in poor performance. 46 | 2. if the name server crashes, it may not be possible for any site in the distributed system to continue to run. 47 | 48 | A more widely used alternative approach requires that each site prefix its own site identifier to any name that it generates. This approach ensures that no two sites generate the same name (since each site has a unique identifier). Furthermore, no central control is required. This solution, however, fails to achieve location transparency, since site identifiers are attached to names. 49 | 50 | To overcome this problem, the database system can create a set of alternative names, or aliases, for data items. A user may thus refer to data items by simple names that are translated by the system to complete names. The mapping of aliases to the real names can be stored at each site. With aliases, the user can be unaware of the physical location of a data item. Furthermore, the user will be unaffected if the database administrator decides to move a data item from one site to another. 51 | -------------------------------------------------------------------------------- /books/howto-win-friends-and-influence-people/shortlist.md: -------------------------------------------------------------------------------- 1 | # Fundamental Techniques in Handling People 2 | 1. Don't criticize, condemn or complain 3 | 1. Give honest and sincere appreciation 4 | 1. Arouse in the other person an eager want 5 | 6 | # Six Ways to make people like you 7 | 8 | 4. Become genuinely interested in other people. 9 | 1. Smile 10 | 1. Remember that a person's name is to that person the sweetest and most important sound in any language. 11 | 1. Be a good listener. Encourage others to talk about themselves. 12 | 1. Talk in terms of the other person's interests. 13 | 1. Make the other person feel important -- and do it sincerely. 14 | 15 | # Win people to your way of thinking 16 | 17 | 10. The only way to get the best of an argument is to avoid it. 18 | 11. Show respect for the other person's opinions. Never say, "You're wrong". 19 | 12. If you are wrong, admit it quickly and emphatically. 20 | 13. Begin in a friendly way. 21 | 14. Get the other person saying "yes, yes" immediately. 22 | 15. Let the other person do a great deal of the talking. 23 | 16. Let the other person feel that the idea is his or hers. 24 | 17. Try honestly to see things from the other person's point of view. 25 | 18. Be sympathetic with the other person's ideas and desires. 26 | 19. Appeal to the nobler motives. 27 | 20. Dramatize your ideas. 28 | 21. Throw down a challenge. 29 | 30 | # Be a Leader 31 | 32 | 22. Begin with praise and honest appreciation. 33 | 1. Call attention to people's mistakes indirectly. 34 | 1. Talk about your own mistakes before criticizing the other person. 35 | 1. Ask questions instead of giving direct orders. 36 | 1. Let the other person save face. 37 | 1. Praise the slightest improvement and praise every improvement. Be "hearty in your approbation and lavish in your praise". 38 | 1. Give the other person a fine reputation to live up to. 39 | 1. Use encouragement. Make the fault seem easy to correct. 40 | 1. Make the other person happy about doing the thing you suggest. 41 | -------------------------------------------------------------------------------- /books/open-vpn.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/books/open-vpn.md -------------------------------------------------------------------------------- /books/operating-system-concepts/chapter-08-virtual-memory.md: -------------------------------------------------------------------------------- 1 | # Virtual Memory 2 | 3 | Virtual memory is a technique that allows the execution of processes that are not completely in memory. One major advantage of this scheme is that programs can be larger than physical memory. 4 | 5 | Further, virtual memory abstracts main memory into an extremely large, uniform array of storage, separating logical memory as viewed by the user from physical memory. This technique frees programmers from the concerns of memory-storage limitations. 6 | 7 | # 8.1 Background 8 | 9 | # 8.2 Demand Paging 10 | 11 | Process is loaded lazily. A pager swaps in pages as they are needed, swapping out pages that are no longer required. 12 | 13 | ## 8.2.1 14 | 15 | We use valid/invalid bit to mark pages that are present in memory. 16 | 17 | **pure demand paging**: never bring a page into memory until it is required. 18 | 19 | ## 8.2.2 Performance of Demand Paging 20 | 21 | **For most computer systems, the memory-access time, denoted ma, ranges from 10 to 200 nanoseconds.** 22 | 23 | The probability of a page fault needs to be really low so that demand paging does not slow system down. 24 | 25 | p < 0.0000025 would degrade the performance by less than 10%. 26 | 27 | 28 | -------------------------------------------------------------------------------- /books/operating-system-concepts/images/base_limit.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/books/operating-system-concepts/images/base_limit.png -------------------------------------------------------------------------------- /books/operating-system-concepts/images/disk.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/books/operating-system-concepts/images/disk.png -------------------------------------------------------------------------------- /books/operating-system-concepts/images/dynamic_relocation.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/books/operating-system-concepts/images/dynamic_relocation.png -------------------------------------------------------------------------------- /books/operating-system-concepts/images/fcb.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/books/operating-system-concepts/images/fcb.png -------------------------------------------------------------------------------- /books/operating-system-concepts/images/inode_index_block.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/books/operating-system-concepts/images/inode_index_block.png -------------------------------------------------------------------------------- /books/operating-system-concepts/images/job_queue.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/books/operating-system-concepts/images/job_queue.png -------------------------------------------------------------------------------- /books/operating-system-concepts/images/memory_mapping.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/books/operating-system-concepts/images/memory_mapping.png -------------------------------------------------------------------------------- /books/operating-system-concepts/images/paging_hardware.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/books/operating-system-concepts/images/paging_hardware.png -------------------------------------------------------------------------------- /books/operating-system-concepts/images/process_control_block.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/books/operating-system-concepts/images/process_control_block.png -------------------------------------------------------------------------------- /books/operating-system-concepts/images/queuing_diagram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/books/operating-system-concepts/images/queuing_diagram.png -------------------------------------------------------------------------------- /books/operating-system-concepts/images/tlb.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/books/operating-system-concepts/images/tlb.png -------------------------------------------------------------------------------- /books_toc/computer_networking_a_top_down_approach.md: -------------------------------------------------------------------------------- 1 | # Computer Networking: A Top Down Approach 2 | 3 | - Started Reading: 208-08-12 4 | - Finished Reading: Not yet 5 | 6 | ## Chapter 3 Transport Layer 7 | 8 | - [x] **2018-08-12** 3.1 Introduction and Transport-Layer Services 9 | - [x] **2018-08-12** 3.1.1 Relationship Between Transport and Network Layers 10 | - [x] **2018-08-12** 3.1.2 Overview of the Transport Layer in the Internet 11 | - [x] **2018-08-12** 3.2 Multiplexing and Demultiplexing 12 | - [x] TCP creates socket with 4 tuple. 13 | - [x] **2018-08-14** 3.3 Connectionless Transport: UDP 14 | - [x] **2018-08-14** 3.3.1 UDP Segment Structure 15 | - [x] **2018-08-14** 3.3.2 UDP Checksum 16 | - [x] **2018-08-19** 3.4 Principles of Reliable Data Transfer 17 | - [x] **2018-08-15** 3.4.1 Building a Reliable Data Transfer Protocol 18 | - [x] **2018-08-16** 3.4.2 Pipelined Reliable Data Transfer Protocols 19 | - [x] **2018-08-19** 3.4.3 Go-Back-N (GBN) 20 | - [x] **2018-08-19** 3.4.4 Selective Repeat (SR) 21 | - [x] **2018-08-20** 3.5 Connection-Oriented Transport: TCP 22 | - [x] **2018-08-19** 3.5.1 The TCP Connection 23 | - [x] **2018-08-19** 3.5.2 TCP Segment Structure 24 | - [x] **2018-08-19** 3.5.3 Round-Trip Time Estimation and Timeout 25 | - [x] **2018-08-19** 3.5.4 Reliable Data Transfer 26 | - [x] **2018-08-20** 3.5.5 Flow Control 27 | - [x] **2018-08-20** 3.5.6 TCP Connection Management 28 | - [x] **2018-08-24** 3.6 Principles of Congestion Control 29 | - [ ] 3.6.1 The Causes and the Costs of Congestion 30 | - [x] Costs of Congestion 31 | - [ ] 3.6.2 Approaches to Congestion Control 32 | - [ ] 3.6.3 Network-Assisted Congestion-Control Example: ATM ABR Congestion Control 33 | - [x] **2018-08-24** 3.7 TCP Congestion Control 34 | - [x] **2018-08-24** Congestion Control Algorithm 35 | - [ ] 3.7.1 Fairness 36 | - [ ] 3.8 Summary 37 | 38 | ## Chapter 4 The Network Layer 39 | 40 | - [x] **2018-08-24** 4.1 Introduction 41 | - [x] **2018-08-24** 4.1.1 Forwarding and Routing 42 | - [x] **2018-08-24** 4.1.2 Network Service Models (nothing much) 43 | - [ ] 4.2 Virtual Circuit and Datagram Networks 44 | - [ ] 4.2.1 Virtual-Circuit Networks 45 | - [x] **2018-08-25** 4.2.2 Datagram Networks 46 | - [ ] 4.2.3 Origins of VC and Datagram Networks 47 | - [x] **2018-08-26** 4.3 What's Inside a Router? 48 | - [x] **2018-08-26** 4.3.1 Input Processing 49 | - [x] **2018-08-26** 4.3.2 Switching 50 | - [x] **2018-08-26** 4.3.3 Output Processing 51 | - [x] **2018-08-26** 4.3.4 Where Does Queuing Occur? 52 | - [x] **2018-08-26** 4.3.5 The Routing Control Plane 53 | - [x] **2018-08-28** 4.4 The Internet Protocol (IP): Forwarding and Addressing in the Internet 54 | - [x] **2018-08-26** 4.4.1 Datagram Format 55 | - [x] **2018-08-26** 4.4.2 IPv4 Addressing 56 | - [x] **2018-08-28** 4.4.3 Internet Control Message Protocol (ICMP) 57 | - [x] **2018-08-28** 4.4.4 IPv6 (nothing much) 58 | - [x] **2018-08-28** 4.4.5 A Brief Foray into IP Security (nothing much) 59 | - [ ] 4.5 Routing Algorithms 60 | - [ ] 4.5.1 The Link-State (LS) Routing Algorithm 61 | - [ ] 4.5.2 The Distance-Vector (DV) Routing Algorithm 62 | - [ ] 4.5.3 Hierarchical Routing 63 | - [ ] 4.6 Routing in the Internet 64 | - [ ] 4.6.1 Intra-AS Routing in the Internet: RIP 65 | - [ ] 4.6.2 Intra-AS Routing in the Internet: OSPF 66 | - [ ] 4.6.3 Inter-AS Routing: BGP 67 | - [ ] 4.7 Broadcast and Multicast Routing 68 | - [ ] 4.7.1 Broadcast Routing Algorithms 69 | - [ ] 4.7.2 Multicast 70 | - [ ] 4.8 Summary 71 | 72 | ## Chapter 5 The Link Layer: Links, Access Networks, and LANs 73 | 74 | - [ ] 5.1 Introduction to the Link Layer 75 | - [ ] 5.1.1 The Services Provided by the Link Layer 76 | - [ ] Where Is the Link Layer Implemented? 77 | - [ ] 5.2 Error-Detection and -Correction Techniques 78 | - [ ] 5.2.1 Parity Checks 79 | - [ ] 5.2.2 Checksumming Methods 80 | - [ ] 5.2.3 Cyclic Redundancy Check (CRC) 81 | - [ ] 5.3 Multiple Access Links and Protocols 82 | - [ ] 5.3.1 Channel Partitioning Protocols 83 | - [ ] 5.3.2 Random Access Protocols 84 | - [ ] 5.3.3 Taking-Turns Protocols 85 | - [ ] 5.3.4 DOCSIS: The Link-Layer Protocol for Cable Internet Access 86 | - [ ] 5.4 Switched Local Area Networks 87 | - [ ] 5.4.1 Link-Layer Addressing and ARP 88 | - [ ] 5.4.2 Ethernet 89 | - [ ] 5.4.3 Link-Layer Switches 90 | - [ ] 5.4.4 Virtual Local Area Networks (VLANs) 91 | - [ ] 5.5 Link Virtualization: A Network as a Link Layer 92 | - [ ] 5.5.1 Multiprotocol Label Switching (MPLS) 93 | - [ ] 5.6 Data Center Networking 94 | - [ ] 5.7 Retrospective: A Day in the Life of a Web Page Request 95 | - [ ] 5.7.1 Getting Started: DHCP, UDP, IP, and Ethernet 96 | - [ ] 5.7.2 Still Getting Started: DNS and ARP 97 | - [ ] 5.7.3 Still Getting Started: Intra-Domain Routing to the DNS Server 98 | - [ ] 5.7.4 Web Client-Server Interactions: TCP and HTTP 99 | - [ ] 5.8 Summary -------------------------------------------------------------------------------- /computer-architecture.md: -------------------------------------------------------------------------------- 1 | # Little Endian and Big Endian 2 | 3 | Big Endian Byte Order: The most significant byte (the "big end") of the data is placed at the byte with the lowest address. 4 | 5 | Little Endian Byte Order: The least significant byte (the "little end") of the data is placed at the byte with the lowest address. 6 | 7 | Within a byte the order of the bits is the same for all computers (no matter how the bytes themselves are arranged). 8 | -------------------------------------------------------------------------------- /coursera/cloud-computing-concepts-part-1/week1.md: -------------------------------------------------------------------------------- 1 | # What is Cloud? 2 | 3 | Cloud is something that provides us lots of data storage with compute cycle nearby. It is data intensive. Moving around data is costly, so we bring computing power near the data. 4 | 5 | The servers in cloud are arranged in racks. Each rack has a common power supply and top of rack switch. Typically, all top of rack switch is then connected to core switch, creating a 2-tier network topology. 6 | 7 | ## Four Features New in Cloud 8 | 9 | 1. Massive scale 10 | 1. On demand access 11 | 1. Data-intensive nature 12 | 1. New cloud programming paradigm - Mapreduce, hadoop, nosql 13 | 14 | # Distributed System 15 | 16 | **Wrong definitions** 17 | 18 | A distributed system is a group of computer that work together and appear as one computer logically. 19 | 20 | The above definition is wrong because Internet is a distributed system and it doesn't appear as a single entity. 21 | 22 | Some say that distributed system follows server-client architecture. This is wrong because p2p bit-torrent has no server and yet it is a distributed system. 23 | 24 | **Definition using properties** 25 | 26 | > A distributed system is a collection of entities, each of which is **autonomous, programmable, asynchronous** and **failure prone**, and which communicates through an **unreliable communication medium**. 27 | 28 | A distributed system is different from parallel programming because it is asynchronous. In parallel programming, processors share a common clock. In distributed system, each entity has its own clock and they are not synchronized. 29 | 30 | # Map Reduce 31 | 32 | - Word count 33 | - Distributed grep 34 | - Reverse web link graph 35 | - Count url access frequency 36 | - Sort (need to use range based reducer) 37 | 38 | Output of map task is sorted by default. 39 | 40 | 41 | ## How map reduce works 42 | 43 | For learning purpose, we assume that there is a barrier between map phase and reduce phase, i.e, no reduce task starts before map tasks finish. 44 | 45 | Parallelizing map and reduce task is easy because there is no communication between any two task. They are all independent. 46 | 47 | Map input is stored in distributed file system. In order to process a map task, it first needs to fetch it from a server where the file block is situated. Hence, map task is fastest if it is assigned to the server where data is located. This is what we mean by **bringing computing near to data**. 48 | 49 | Output of map task is stored locally. Reduce task then gets its input from these servers where data is stored locally. Local file system is used in this intermediate process because using DFS has its own overhead. We want to get output of map task to reduce task as quick as possible. 50 | 51 | Output of reduce task is stored in DFS. 52 | 53 | ### Architecture 54 | 55 | Hadoop has three managers: 56 | 57 | Each machine (node) has some containers in it that run some task. 58 | 59 | 1. Resource Manager: Decides which nodes gets which task and tracks their progress. Its the center of the whole design. It knows everything. 60 | 1. Application Manager: Each task has an application manager. It is responsible for task management. Basically, it has the code for the task (map or reduce). When it gets alive for the first time, it reports to resource manager, asking for resource to run the task. 61 | 1. Node Manager: Each node has a node manager. Responsible for state of node. It has the ability to schedule task on its node. It lets resource manager know when ever a task is started or completed on its node. 62 | 63 | These managers communicate with each other through heartbeats. If a node fails, resource manager lets application managers know that their task failed and then restarts the node. If application manager fails, resource manager restarts it and synchronizes it with current state. If resource manager fails (trickiest), then a backup resource manager with checkpoint is started. 64 | 65 | **Stragglers (slow nodes)** 66 | Speculative Execution: Start replica task and consider the task done when any of the replica completes. 67 | 68 | **Locality** 69 | Manager tries to schedule task on same machine as the data, failing that on same rack, otherwise anywhere. 70 | -------------------------------------------------------------------------------- /coursera/cloud-computing-concepts-part-1/week2.md: -------------------------------------------------------------------------------- 1 | # Gossip 2 | 3 | ## Multicast Problem 4 | 5 | We have a node who wants to send information to all nodes in the group. 6 | 7 | Different from Broadcast as in broadcase all nodes in the network receive message but in Multicast only members of group in the network receive message. 8 | 9 | The multicast protocol needs to be fault-tolerant and scalable. 10 | 11 | ### Solution 12 | 13 | 1. Centralized: A single node sends to all nodes using a loop and TCP/UDP protocol. Does not scale well. If sender fails mid-way, then some nodes might miss the message. Latency overhead is O(n). Pressure on sender is also O(n). 14 | 15 | 1. Tree based: We could build a spanning tree. If we build it as a balance binary tree, then latency overhead drops down to O(log n). Sender overhead is O(1). But we have to setup and maintain the tree structure ourselves. Also, if a node near root fails, then it effects the whole sub-tree. 16 | 17 | 1. Gossip: Sender, at periodic interval of say 5 seconds, chooses a set of b nodes randomly and sends them messages using UDP. The nodes that receive the messages become infected nodes. These nodes does the same thing. This is called push-gossip. A variant is pull-gossip. 18 | 19 | ### Gossip Analysis 20 | 21 | [Epidemiology Analysis](https://www.coursera.org/learn/cloud-computing/lecture/jjieX/1-3-gossip-analysis) 22 | 23 | After $clog(n)$ rounds ( lets say $2log(n)$ ), number of infected nodes become $ (n+1) - \frac{1}{n^{cb-2}}$, or $(n+1) - 1/n^2$ if c = 2 and b = 2, where b is the gossip fanout. 24 | 25 | Hence gossip is **low latency** ($2log(n)$ rounds), **reliable** (only $1/n^2$ nodes miss out) and **lightweight** ($cblog(n) = 4log(n)$ messages are transmitted.) 26 | 27 | #### Fault Tolerance and Reliability 28 | If there is 50% packet loss, that means half the message we are sending is getting lost, that is same as analyzing with $b/2$ fanout. So to get same result as before, we simply need to run our gossip for $2clog(n)$ rounds. 29 | 30 | If 50% nodes fail, then it is same as $n/2$ and $b/2$. So we just need to run gossip for $2clog(2n)$ rounds to compensate (not sure about this). 31 | 32 | #### Can gossip die out 33 | Once gossip spreads, it is highly unlikely to die out as gossip fan out is selected at random. 34 | 35 | #### Pull vs Push 36 | Pull gossip has overall latency of O(logn), but on second half of the protocol, that is once half the nodes are infected, it grows faster than exponential with complexity of O(log(log(n))). 37 | 38 | A possible hybrid system could be push gossip for first half and pull gossip for later half. 39 | 40 | #### Topology Aware Gossip 41 | 42 | Suppose we have two subnets with n/2 nodes each. With normal gossip, at each round, $b*n/2$ messages will travel across the two net. With O(n) message passing, router will be overwhelmed. 43 | 44 | Instead, we can configure gossip so that it gossips with a node on different subnet with a probability of $1/n$. This way, the gossip quickly infects its own subnet and once everyone is infected, they all try to infect other subnet with $n \times 1/n = 1$ expected round. 45 | 46 | The load on router is now O(1) and latency overhead is $O(logn) + O(1) + O(logn) = O(logn)$. 47 | 48 | # Membership 49 | 50 | > **MTTF**: Mean time to failure 51 | 52 | We process the system as a group of members (individual process). 53 | 54 | We assume a **fail-stop** failure model where process stop completely once they fail. 55 | 56 | ## Failure Detection 57 | 58 | - Completeness: Each failure is detected 59 | - Accuracy: No mistake in detection 60 | - Speed 61 | - Scale 62 | 63 | Over a unreliable network, fulfilling both of completeness and accuracy property is impossible. 64 | 65 | So in real life, we design system so that **completeness is guaranteed** and **accuracy is partially guaranteed**. 66 | 67 | ### Possible Architecture 68 | 69 | 1. **Centralized Heartbeating**: All nodes send heartbeats to a single node. If the central node fails, failure detection becomes undefined. Also, the central node is under heavy load. 70 | 1. **Ring Heatbeating**: Each node is connected to two other nodes. Failures may go unnoticed if multiple nodes fail at same time. 71 | 1. **All-to-All Hearbeating**: Equally distributed load. But if a node is slow and receiving heartbeats at longer delay for some reason, it might wrongfully detect all other nodes to be dead. 72 | 1. **Gossip Style Heartbeating**: Each node keeps a membership table containing node number, heartbeat sequence number and a local time stamp. Periodically nodes share their membership list using gossip. 73 | 1. **SWIM** 74 | -------------------------------------------------------------------------------- /coursera/cloud-computing-concepts-part-1/week3.md: -------------------------------------------------------------------------------- 1 | # Distributed Hash Table 2 | 3 | [DHT Explanation in Chord video (need to watch again)](https://www.coursera.org/learn/cloud-computing/lecture/U0ILe/5-chord) 4 | 5 | Each nodes contains a **finger table** that contains pointer to next keys. Using finger table the routing can be brought down to log(n). 6 | 7 | Routing can be wrong if any node fails. 8 | -------------------------------------------------------------------------------- /coursera/cloud-computing-concepts-part-1/week4.md: -------------------------------------------------------------------------------- 1 | # Key Value / NOSQL 2 | 3 | NOSQL storage typically store a column together. 4 | 5 | # Cassandra 6 | 7 | Each data center is DHT Ring. Client requests to a node which acts as a **coordinator**. 8 | 9 | Write heavy. Lock free writing. 10 | 11 | Uses **Hinted Handoff** mechanism. Coordinator keeps the write to its buffer until some failed replica rejoins. Keeps it upto an hour. 12 | 13 | **Delete**: Tombstone 14 | 15 | For reading, value with the most updated timestamp among X replies is sent back. Then **read repair** is used to update stale values. 16 | 17 | **Gossip-style Membership** 18 | 19 | # Cap Theorem 20 | 21 | Availability: Fast and reliable read/write 22 | Consistency: All nodes see the same value 23 | Partition Tolerance: System can continue to work even when they are separated by network partition. 24 | 25 | ## Eventual Consistency 26 | 27 | If all writes to a key stop, then its value will eventually converge. 28 | 29 | If writes continue, the system will keep on trying to converge. 30 | 31 | May return stale values. 32 | 33 | **BASE**: Basically Available Soft-state Eventually Consistency 34 | 35 | ## Consistency Level 36 | 37 | 1. Any: Fastest. Coordinator caches the write. 38 | 1. All: Slowest. Strong consistency. 39 | 1. One 40 | 1. Quoram: Faster than all, still strong consistency. 41 | 42 | 43 | For strong consistency, $W + R > N$ and $W + W > N$. 44 | 45 | 46 | -------------------------------------------------------------------------------- /database/indexing.md: -------------------------------------------------------------------------------- 1 | # Indexing 2 | 3 | [How database indexing works](https://stackoverflow.com/questions/1108/how-does-database-indexing-work) 4 | -------------------------------------------------------------------------------- /database/mongodb-design.md: -------------------------------------------------------------------------------- 1 | # Mongodb Design 2 | 3 | # 1-N relationship 4 | 5 | - 1-to-few: like address, embed them 6 | - 1-to-many: like students in classroom, reference array 7 | - 1-to-tooMany: parent reference 8 | 9 | # Rule of Thumbs 10 | 11 | 1. favor embedding unless there is a compelling reason not to 12 | 1. needing to access an object on its own is a compelling reason not to embed it 13 | 1. Arrays should not grow without bound. If there are more than a couple of hundred documents on the “many” side, don’t embed them; if there are more than a few thousand documents on the “many” side, don’t use an array of ObjectID references. High-cardinality arrays are a compelling reason not to embed. 14 | 1. Don’t be afraid of application-level joins: if you index correctly and use the projection specifier (as shown in part 2) then application-level joins are barely more expensive than server-side joins in a relational database. 15 | 1. Consider the write/read ratio when denormalizing. A field that will mostly be read and only seldom updated is a good candidate for denormalization: if you denormalize a field that is updated frequently then the extra work of finding and updating all the instances is likely to overwhelm the savings that you get from denormalizing. 16 | 1. As always with MongoDB, how you model your data depends – entirely – on your particular application’s data access patterns. You want to structure your data to match the ways that your application queries and updates it. 17 | 18 | # Articles 19 | 20 | 1. [6 Rules of Thumb for MongoDB Schema Design: Part 1](https://www.mongodb.com/blog/post/6-rules-of-thumb-for-mongodb-schema-design-part-1) 21 | 1. [6 Rules of Thumb for MongoDB Schema Design: Part 3](https://www.mongodb.com/blog/post/6-rules-of-thumb-for-mongodb-schema-design-part-3) 22 | -------------------------------------------------------------------------------- /database/mongoose.md: -------------------------------------------------------------------------------- 1 | # Mongoose 2 | 3 | # Find one and update 4 | ``` 5 | const findObj = {} 6 | const updateFields = {a: "b", c: "d"} 7 | const options = {...} // new, upsert, fields, sort 8 | Model.findOneAndUpdate(findObj, {$set: updateFields}, options) 9 | // Returns original found document 10 | ``` 11 | 12 | # Update 13 | 14 | Don't forget to send "multi" true to update multiple object. 15 | ``` 16 | Coll.update(query,update,{ 17 | multi: true, 18 | upsert: true 19 | }); 20 | 21 | $set 22 | $push 23 | $pull 24 | $addToSet 25 | ``` 26 | 27 | # Mongoose Unique values in nested array of objects 28 | 29 | [Stack Overflow](https://stackoverflow.com/questions/15921700/mongoose-unique-values-in-nested-array-of-objects) 30 | 31 | ``` 32 | var user = { uid: userOid, ... }; 33 | Group.update( 34 | // This ensures users.uid is unique 35 | {name: 'admin', 'users.uid': {$ne: user.uid}}, 36 | {$push: {users: user}}, 37 | function(err, numAffected) { ... }); 38 | ``` 39 | 40 | # Aggregate 41 | 42 | ``` 43 | db.aggregate([ 44 | { 45 | // Select docs that match query 46 | $match: { 47 | // Normal query 48 | type: "folder" 49 | }, 50 | { 51 | // Used for getting random document 52 | $sample: { 53 | size: 1 54 | } 55 | } 56 | } 57 | ]) 58 | ``` 59 | 60 | # Query Array of Object 61 | 62 | Use `$elemMatch`. Returns the parent object. 63 | 64 | ``` 65 | const userList = await User.find({ 66 | ojStats: { 67 | $elemMatch: { 68 | ojname, 69 | solveList: problemId, 70 | }, 71 | }, 72 | }, {username: 1}).exec(); 73 | ``` 74 | 75 | [Query an Array of Embedded Documents](https://docs.mongodb.com/manual/tutorial/query-array-of-documents/) 76 | 77 | # Index 78 | 79 | ``` 80 | # Path level 81 | field1: { type: String, unique: true, index: true }, 82 | 83 | # Schema level (mandatory for compound index) 84 | mySchema.index({field1: 1, field2: 1}, {unique: true}); 85 | 86 | # More options 87 | sparse: true 88 | ``` 89 | 90 | # Populate Selected fields 91 | 92 | ``` 93 | .populate('field1 field2', 'select1 select2'); 94 | ``` 95 | 96 | # Backup 97 | 98 | ## mongodump vs mongoexport 99 | 100 | https://stackoverflow.com/questions/44562993/which-one-is-the-preferred-choice-mongodump-vs-mongoexport-for-upgrading-mongodb 101 | 102 | Mongodump and Mongorestore are better because: 103 | 104 | - They run faster 105 | - They preserve some data formats better than mongoexport and mongoimport, because the data is not translated from BSON into JSON and back. 106 | 107 | 108 | ``` 109 | sudo mongodump --db newdb --out /var/backups/mongobackups/`date +"%m-%d-%y"` 110 | sudo mongorestore --db newdb --drop /var/backups/mongobackups/01-20-16/newdb/ 111 | ``` 112 | -------------------------------------------------------------------------------- /database/mysql/mysql.md: -------------------------------------------------------------------------------- 1 | # Mysql 2 | 3 | ## Limit 4 | 5 | ```sql 6 | SELECT * FROM MyTable ORDER BY whatever LIMIT 0,1000 7 | SELECT * FROM MyTable ORDER BY whatever LIMIT 1000,1000 8 | SELECT * FROM MyTable ORDER BY whatever LIMIT 2000,1000 9 | ``` 10 | 11 | `OFFSET` is very bad. Instead of processing the data in $O(N)$ time, it will take $O(N^2)$. This is because of the effort to skip past the "offset" rows. -------------------------------------------------------------------------------- /database/sql.md: -------------------------------------------------------------------------------- 1 | # SQL Query 2 | 3 | - You can select distinct column items using `Distinct()` function like following: 4 | 5 | ``` 6 | select Distinct(city) 7 | from station 8 | where (id % 2) = 0 9 | ``` 10 | 11 | - Modular arithmetic works. Look above. 12 | - You can order by selected column as follow: 13 | ``` 14 | select city, char_length(city) 15 | from station 16 | order by 2 desc, 1 17 | limit 1; 18 | ``` 19 | 20 | [Exercise](https://www.hackerrank.com/challenges/weather-observation-station-5/problem) 21 | 22 | - You can match strings with regex like this: 23 | 24 | ``` 25 | select distinct(city) 26 | from station 27 | where city REGEXP "^[aeiou]+"; 28 | ``` 29 | 30 | # Function List 31 | 32 | - count 33 | - distinct 34 | - char_length(string s) 35 | - substring(sting s, int start) 36 | - RTrim() 37 | 38 | # Mysql Operators 39 | 40 | ## Like 41 | 42 | The LIKE operator is commonly used to select data based on patterns. Using the LIKE operator in the right way is essential to increase the query performance. 43 | 44 | The LIKE operator allows you to select data from a table based on a specified pattern. Therefore, the LIKE operator is often used in the WHERE clause of the SELECT statement. 45 | 46 | MySQL provides two wildcard characters for using with the LIKE operator, the percentage % and underscore _ . 47 | 48 | % works like * in regex, 0 or more characters. 49 | _ works like . in regex, exactly 1 character. 50 | 51 | Although it may seem that the % wildcard matches anything, there is one exception, NULL . Not even the clause WHERE prod_name LIKE ‘%’ will match a row with the value NULL as the product name. 52 | 53 | # Using Variables 54 | 55 | To create a user-defined variable, you use the format @variable_name, where the variable_name consists of alphanumeric characters. The maximum length of the user-defined variable is 64 characters as of MySQL 5.7.5 56 | 57 | The user-defined variables are not case-sensitive. It means that the @id and @ID are the same. 58 | 59 | You can assign the user-defined variable to a certain data types such as integer, floating point, decimal, string or NULL. 60 | 61 | There are two ways to assign values to variable: 62 | 1. `SET variable := value` 63 | 1. `SELECT variable := value` 64 | 65 | [HR Finding Median](https://www.hackerrank.com/challenges/weather-observation-station-20) 66 | 67 | # IF/ELSE 68 | 69 | It's like ternary operator: 70 | 71 | `IF( conditional, truthy, falsy ) AS col` 72 | 73 | [HR Problem](https://www.hackerrank.com/challenges/the-report?h_r=next-challenge&h_v=zen) 74 | 75 | # Join 76 | 77 | ![All possible joins](../images/join.jpg) 78 | -------------------------------------------------------------------------------- /database/write-ahead-logging.md: -------------------------------------------------------------------------------- 1 | # Write Ahead Logging (WAL) 2 | 3 | Data pages cannot be written to disk until the log records describing those changes are written to disk FIRST. 4 | -------------------------------------------------------------------------------- /design-patterns/index.md: -------------------------------------------------------------------------------- 1 | # General 2 | 3 | 1. Dependency Injection 4 | - [Youtube Video Duration 4:46 Minutes](https://www.youtube.com/watch?v=IKD2-MAkXyQ) 5 | -------------------------------------------------------------------------------- /devops/devop_checklist.md: -------------------------------------------------------------------------------- 1 | # DevOps Checklist 2 | 3 | Inspired from the article [The Must Know Checklist For DevOps & Site Reliability Engineers](https://hackernoon.com/the-must-know-checklist-for-devops-system-reliability-engineers-f74c1cbf259d). 4 | 5 | - [x] CPU/system info 6 | - [x] Most of the details can be found in `/proc` directory. 7 | - [x] version, uptime, cpuinfo 8 | - [ ] Cron jobs 9 | - [x] OS info 10 | - [x] `less /etc/lsb-release` 11 | - [x] lsb stands for linux standard base 12 | - [x] Environment Variables 13 | - [x] How to set/unset them? 14 | - [x] How to export permanent variables? 15 | - [x] How to view them? 16 | - [ ] Shell Configuration file 17 | - [ ] .bashrc, .bash_profile 18 | - [ ] How to source them 19 | - [ ] Logging 20 | - [ ] How it works 21 | - [ ] Logging levels 22 | - [ ] Log management tools 23 | - [ ] Swapping 24 | - [ ] How it works 25 | - [ ] What is swappiness 26 | - [ ] Network Configuration 27 | - [ ] View/Set Configuration 28 | - [ ] SSH 29 | - [ ] How it works 30 | - [ ] How to generate SSH key 31 | - [ ] DNS 32 | - [ ] Records 33 | - [ ] Setup 34 | - [ ] Nginx 35 | - [ ] vs Apache 36 | - [ ] Reverse proxy 37 | - [ ] Caching server 38 | - [ ] Load balancer 39 | - [ ] Init System 40 | - [ ] systemd/upstart 41 | - [ ] Firewall 42 | - [ ] iptable 43 | - [ ] Configuration Management 44 | - [ ] Puppet/Chef 45 | - [ ] Continuous Integration 46 | - [ ] Version Controlling 47 | - [ ] Git 48 | - [ ] IaC 49 | - [ ] Docker 50 | - [ ] Redis/Memcache 51 | 52 | There is more, but I got tired of listing. 53 | -------------------------------------------------------------------------------- /devops/docker.md: -------------------------------------------------------------------------------- 1 | # Building a docker from dockerfile 2 | 3 | Create a Dockerfile 4 | 5 | - It cannot be empty 6 | - Must start with "From " 7 | 8 | Then build the docker image 9 | 10 | - Basic: `docker build .` 11 | - With Title: `docker build -t .` 12 | 13 | Docker will then create an image 14 | 15 | ## More Dockerfile Macros 16 | - `WORKDIR `: creates and changes to path 17 | - `COPY `: copies file from host to container 18 | - `ADD `: adds file from host to container. Source can be url. zip or tar balls would be unzipped. 19 | - `RUN `: creates a new layer 20 | - `CMD []` 21 | 22 | # Docker Network 23 | All containers are connected to default network bridge named docker0. Using this bridge, any container can communicate using any other container using their ip address. DNS resolution using container name is available for user-created-networks only. 24 | 25 | - Create Network: `docker network create --driver bridge ` 26 | - Run container on network: `docker run --network -itd ` 27 | - Connect container to network: `docker network connect [OPTIONS] NETWORK CONTAINER` 28 | 29 | # Docker CLI commands 30 | 31 | ``` 32 | docker images # to view all images 33 | ``` 34 | 35 | # Remove 36 | 1. Remove image: `docker rmi ` 37 | 2. Remove stopped containers: `docker container prune` 38 | 3. Remove dangling images: `docker rmi $(docker images -f "dangling=true" -q)` 39 | 1. Remove Volume: `docker volume prune` to remove unused volume. 40 | 1. Remove dangling images: `docker image prune` 41 | 1. Remove **all** images not used by at least on container: `docker image prune -a` 42 | 2. Ultimate cleanup: `docker system prune -a` 43 | 44 | # Exec 45 | 46 | 1. `docker exec CONTAINER CMD` 47 | 1. `dcocker exec -it CONTAINER /bin/bash` to ssh into container. 48 | 1. You cannot reattach to a detached exec-process. 49 | 1. Docker exec creates a process. Use `ps aux | grep CMD` to find it out and kill it. 50 | 51 | # Docker Compose 52 | Writing `docker run` with various flags and then connecting them to networks and doing these all for multiple containers is tiresome. Docker compose makes things easy for us. 53 | 54 | `docker-compose` is a python script. Once installed, we need to create a `docker-compose.yml` file. `docker-compose` will work using that config file. 55 | 56 | ``` 57 | pip install --user docker-compose 58 | docker-compose up # creates and starts the containers and sets them up accordingly. 59 | docker-compose down # stops and destroys all containers and networks created during the process. Volumes are not destroyed. 60 | docker-compose down -v # to destroy the volumes. 61 | docker-compose start|stop # stops or starts without destroying the containers. 62 | ``` 63 | 64 | ## env File 65 | 66 | We can declare a `.env` file with following format: 67 | 68 | ``` 69 | key=value 70 | ``` 71 | 72 | Then we can use these values in docker-compose as following: 73 | 74 | ``` 75 | ... 76 | environment: 77 | - SOMETHING=${key} 78 | ``` 79 | 80 | # Docker Swarm 81 | 82 | Docker Swarm is a **Container Orchestration System**. 83 | 84 | # CMD or Entrypoint 85 | 86 | Define these parameters so that it is possible to run docker containers without passing any execution parameters, e.g, `docker run nginx`. 87 | 88 | Passing execution parameters will override cmd command. It is a bit of hassle to override entrypoint. Better to stick with cmd. 89 | 90 | # Misc 91 | 1. See running containers: `docker ps` 92 | 1. Exposed Port Binding: use -P flag during `run` 93 | -------------------------------------------------------------------------------- /devops/gcloud-commands.md: -------------------------------------------------------------------------------- 1 | # gcloud Configuration 2 | 3 | ``` 4 | gcloud init # Initiate project and region 5 | gcloud auth list # Check which users are authenticated on local system 6 | gcloud config list # Check current config values 7 | gcloud config get-value project # Get config value 8 | ``` 9 | 10 | # gcloud Docker 11 | ``` 12 | docker build -t gcr.io/${PROJECT_ID}/hello-app:v1 . # The title of container needs to have project ID 13 | gcloud docker -- push gcr.io/${PROJECT_ID}/hello-app:v1 # How to push to gcr.io 14 | ``` 15 | 16 | # gcloud Compute 17 | ``` 18 | gcloud compute instances list # Get list of VM running 19 | ``` 20 | 21 | # gcloud Containers 22 | ``` 23 | gcloud container clusters create hello-cluster --num-nodes=3 # Create a new cluster with 3 nodes 24 | ``` 25 | 26 | # gcloud Service 27 | ``` 28 | kubectl expose deployment nginx --target-port=80 --type=NodePort 29 | ``` 30 | 31 | # gcloud Storage 32 | ``` 33 | gcloud compute disks create [DISK_NAME] --size 100GB --type [pd-standard|pd-ssd] 34 | gcloud compute disks resize [DISK_NAME] --size [DISK_SIZE] 35 | ``` 36 | -------------------------------------------------------------------------------- /devops/gcloud.md: -------------------------------------------------------------------------------- 1 | # Geography and Regions 2 | 3 | Google Cloud Platform services are available in locations across North America, Europe, and Asia. 4 | 5 | These locations are then divided into **Regions** and **Zones**. 6 | 7 | Inter-Zone communication takes 5ms RTT on the 95th Percentile. 8 | 9 | Zones should be considered a single failure domain within a region. In order to deploy fault-tolerant applications with high availability, you should deploy your applications across multiple zones in a region to help protect against unexpected failures. 10 | 11 | To protect against the loss of an entire region due to natural disaster, you should have a disaster recovery plan and know how to bring up your application in the unlikely event that your primary region is lost. 12 | 13 | When using regional or zonal storage resources, it is highly recommended that you replicate data to another region or snapshot it to a multi-regional storage resources for disaster recovery purposes. 14 | 15 | # Cloud Computing 16 | 17 | > In **cloud computing**, the capital investment in **building and maintaining data centers** is replaced by **consuming IT resources** as an elastic, utility-like service from a cloud “provider” (including storage, computing, networking, data processing and analytics, application development, machine learning, and even fully managed services). 18 | 19 | # Security 20 | 21 | [Google Infrastructure Security Design](https://www.youtube.com/watch?time_continue=1&v=O-JXFQezWOc) 22 | 23 | 1. Hardware Layer 24 | Own data center built with security camera, biometric authentication and laser traps. 25 | Custom made manufactured hardware used to provide extra security, such as Titan chip 26 | 1. Boot 27 | 1. OS + IPC 28 | Everything is service. There is no trust between any service. Service need authentication so that server can limit access to rpc (**Interservice access management**). Services need **End User Permission Ticket** to access user-related sensitive data in another service. 29 | 30 | [Best Practices for DDoS Protection and Mitigation on Google Cloud Platform](https://cloud.google.com/files/GCPDDoSprotection-04122016.pdf): 31 | 32 | - When you enable HTTP(S) Load Balancing or SSL proxy Load Balancing, Google infrastructure mitigates and absorbs many Layer 4 and below attacks, such as SYN floods, IP fragment floods, port exhaustion, etc 33 | - projects are limited to an API rate limit of 20 requests/second. 34 | -------------------------------------------------------------------------------- /distributed-system/consistent-hashing.md: -------------------------------------------------------------------------------- 1 | # Consistent Hashing 2 | 3 | Keys and nodes map to same id space. 4 | 5 | ![](images/hash-ring.png) 6 | 7 | **How does key map to nodes?** 8 | 9 | Key id belongs to **successor** nodes. 10 | 11 | Suppose the the logical ring has 2^m positions. The nodes id can be found by using Sha1(ip_address) and simply taking the first m bits. 12 | 13 | # Information Stored 14 | 15 | Each node in the logical ring knows the ip-address of the immediate successor node. 16 | 17 | Each node also keeps a finger table. 18 | 19 | # Finger Table 20 | 21 | Each node keeps a table with m rows. Each row $i$ contains the successor of $x$, where $x = id + 2^i$ and $id$ is the id of node. 22 | 23 | # Key Search 24 | 25 | Ask a node to find out where key k is located. The node will look for the biggest value x <= k present in its finger table and send the request to that node containing x. If no such value is found, the request is simply sent to the next node. 26 | 27 | # Node Failure 28 | 29 | Instead of keeping just the successor, we can keep multiple successor. This helps when one of the intermediate node fails. Keeping 2*log(n) number of successors greatly reduces the chance of failure when n is large. 30 | 31 | 32 | 33 | # Property 34 | 35 | 1. **Load Balancing**: All nodes receives roughly the same amount of key. 36 | 1. **Flexibility**: If any node joins or leaves, only a fraction of the key need to be redistributed. 37 | 38 | # Resources 39 | 40 | 1. [Youtube - Consistent Hashing](https://www.youtube.com/watch?v=--4UgUPCuFM) 41 | -------------------------------------------------------------------------------- /distributed-system/distributed-algorithms.md: -------------------------------------------------------------------------------- 1 | - [Boyer–Moore majority vote algorithm](https://en.wikipedia.org/wiki/Boyer%E2%80%93Moore_majority_vote_algorithm) Finding majority element from a stream. 2 | 3 | [GCJ majority](https://code.google.com/codejam/contest/dashboard?c=4264486#s=p2&a=2) 4 | -------------------------------------------------------------------------------- /distributed-system/distributed-system.md: -------------------------------------------------------------------------------- 1 | # Distributed System 2 | 3 | ## Pattern 4 | 5 | 1. [Service Mesh](./http://philcalcado.com/2017/08/03/pattern_service_mesh.html) -------------------------------------------------------------------------------- /distributed-system/images/hash-ring.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/distributed-system/images/hash-ring.png -------------------------------------------------------------------------------- /distributed-system/micro-service.md: -------------------------------------------------------------------------------- 1 | # Micro Service 2 | 3 | > In short, the microservice architectural style is an approach to developing a single application as **a suite of small services**, each **running in its own process** and **communicating with lightweight mechanisms**, often an HTTP resource API. These services are built around business capabilities and independently deployable by fully automated deployment machinery. 4 | 5 | # Articles 6 | - [Martin Fowler - Microservice](https://martinfowler.com/articles/microservices.html) 7 | 8 | # Resources 9 | - [Udacity](https://classroom.udacity.com/courses/ud615/lessons/7826112332/concepts/80841806450923) 10 | - Kubernetes: Up and Running, Kelsey Hightower The definitive book on Kubernetes. This has been a great resource while making this course. 11 | - Building Microservices: Defining Fine-Grained Systems This is the book Kelsey reads before giving talks about microservices. It’s that good. 12 | 13 | **Videos** 14 | - Adrian Cockroft "The Evolution of Microservices" 15 | - Adrian Cockroft "The State of the Art in Microservices" (docker specific) 16 | - Martin Fowler "Microservices" at goto 17 | - Craig McLuckie "The Next Chapter in Native Cloud Computing" on cloud-native computing as being: container-packaged, dynamically-scheduled, and microservices-oriented 18 | -------------------------------------------------------------------------------- /distributed-system/raft.md: -------------------------------------------------------------------------------- 1 | # Replicated State Machine 2 | 3 | **Example**: Chubby, Zookeeper 4 | 5 | Replicated State Machine (RSM) is used to solve variety of fault-tolerance problem in distributed system. 6 | 7 | Replicated state machines are typically implemented using a replicated log. Each server stores a log containing a series of commands, which its state machine executes in order. Each log contains the same commands in the same order, so each state machine processes the same sequence of commands. Since the state machines are deterministic, each computes the same state and the same sequence of outputs. 8 | 9 | **Keeping the replicated log consistent is the job of the consensus algorithm.** 10 | 11 | 12 | 13 | # Consensus Algorithm (Raft) 14 | 15 | **Raft is an algorithm to maintain a set of replicated log in distributed system.** 16 | 17 | Consensus is a fundamental problem in fault-tolerant distributed systems. Consensus involves multiple servers agreeing on values. Once they reach a decision on a value, that decision is final. Typical consensus algorithms make progress when any majority of their servers is available; for example, a cluster of 5 servers can continue to operate even if 2 servers fail. If more servers fail, they stop making progress. 18 | 19 | Each machine can be thought as a node. Each node can be in three state: **Follower**, **Candidate** and **Leader**. 20 | 21 | ##Leader Election 22 | Each node has an election timeout. The **election timeout** is the amount of time a follower waits until becoming a candidate. In Raft, it is randomized between 150-300ms. 23 | 24 | After the election timeout, the follower becomes a candidate and starts a new election term. It votes for itself and sends out "Request Vote" messages to other nodes. If the receiving node has not voted yet in this term, then it votes for the candidate and resets its election timeout. 25 | 26 | Once a candidate has a majority of votes, it becomes a leader. The leader begins sending "Append Entries" messages to its followers. These messages are sent in intervals specified by the **heartbeat** timeout. The followers then respond to each "Append" message. This election term will continue until a follower stops receiving heartbeats and becomes a candidate. 27 | 28 | Requiring a majority of votes guarantees that only one leader can be elected per term. 29 | 30 | If two nodes become candidates at the same time then a split vote can occur. 31 | 32 | ## Log Replication 33 | 34 | Once we have a leader elected we need to replicate all changes to our system to all nodes. This is done by using the same Append Entries message that was used for heartbeats. 35 | 36 | First a client sends a change to the leader. The change is appended to the leader's log. Then the change is sent to the followers on the next heartbeat. An entry is committed once a majority of followers acknowledge it. Raft uses WAL protocol. 37 | 38 | Raft can even stay consistent in the face of network partitions. Cause even if there are multiple leaders, at most one leader can get majority acknowledgment for "Append Message". And during healing, old leaders will step down when they see that a new leader with higher term is present. 39 | 40 | ## Properties 41 | 42 | 1. **Safety**: never returns an incorrect result under all non-Byzantine conditions, including network delays, partitions, and packet loss, duplication, and reordering. 43 | 1. **Available**: They are fully functional as long as any majority of the servers are operational and can communicate with each other and with clients. Thus, a typical cluster of five servers can tolerate the failure of any two servers. Servers are assumed to fail by stopping; they may later recover from state on stable storage and rejoin the cluster. 44 | 1. **Clock Independent**: They do not depend on timing to ensure the consistency of the logs: faulty clocks and extreme message delays can, at worst, cause availability problems. 45 | 1. **Performance**: In the common case, a command can complete as soon as a majority of the cluster has responded to a single round of remote procedure calls; a minority of slow servers need not impact overall system performance. 46 | 1. **Election Safety**: at most one leader can be elected in a given term. 47 | 1. **Leader Append-Only**: a leader never overwrites or deletes entries in its log; it only appends new entries 48 | 1. **Log Matching**: if two logs contain an entry with the same index and term, then the logs are identical in all entries up through the given index. 49 | 1. **Leader Completeness**: if a log entry is committed in a given term, then that entry will be present in the logs of the leaders for all higher-numbered terms. 50 | 1. **State Machine Safety**: if a server has applied a log entry at a given index to its state machine, no other server will ever apply a different log entry for the same index. 51 | 52 | ## Resources 53 | 54 | 1. http://thesecretlivesofdata.com/raft 55 | 1. [How log commit works](https://stackoverflow.com/questions/37108309/raft-committed-entry-may-be-lost?rq=1) 56 | 1. [etcd/raft implementation](http://otm.github.io/2015/05/raft-a-first-implementation/) 57 | 1. [Hashicorp Raft Example](https://github.com/otoolep/hraftd) 58 | -------------------------------------------------------------------------------- /distributed-system/system-design.md: -------------------------------------------------------------------------------- 1 | # Common DS 2 | 3 | There are multiple nodes. 4 | 5 | A leader is elected using some kind of consensus algorithm. It's called Master node. 6 | 7 | Eventually consistent is preferred in most cases. During insert/update, we use quoram confirmation. 8 | 9 | How do we resolve conflicts? 10 | 11 | During get request, depending on the consistency need, it can be either 1 or quoram. For cases like Facebook we can simply use 1. For banking, qouram read should be used. 12 | 13 | # Read/Write Speed 14 | 15 | | Item | Read | Write | 16 | | :------------- | :------------- | :------| 17 | | SSD | 1 GB/s | ? 18 | | Disk seek | 10 ms | - 19 | | Memory Access | 100 ns | - | 20 | | Read 1 MB sequentially from memory| 250 us | 21 | | Read 1 MB sequentially from SSD | 1 ms | | 22 | | Read 1 MB sequentially from disk | 10 ms | | 23 | | Read 1 MB randomly from disk | 1 s | | 24 | 25 | # Read Repair 26 | 27 | Read-repair is lazy mechanism that ensures that the data you request from the database is accurate and consistent. 28 | 29 | Every time a read request occurs, it provides an opportunity for consistency improvement in the system. 30 | 31 | In read repair, system sends a digest request (a hash value for the data) to each replica not directly involved in the read. System compares all replicas and writes the most recent version to any replica node that does not have it. 32 | 33 | # Hinted Handoffs 34 | 35 | If the replica set for a given item X is nodes A, B, and C, but A is transiently unreachable, sloppy quorum allows a write of X to be sent to B, C, and D instead and considered successful once a quorum of that "sloppy" replica set is successful in acknowledging it. The write to D would be stored with a "hint" that it should ideally have been sent to A. Once A is detected to have recovered, D would attempt to hand off that data or operations. This "hinted handoff" is one of the mechanisms by which such a database can maintain expected levels of availability and durability across failure states. 36 | 37 | # ACID in Transaction 38 | 39 | - **Atomicity**: Either all operations of a transaction executes successfully or none. Partial executions are not allowed. 40 | - **Isolation**: Two atomic transactions must not overlap. 41 | 42 | Other definitions: 43 | - **Serializable**: The resulting database state is equivalent to a serial execution of component transactions. 44 | - **Recoverable**: A set of database transactions is considered recoverable if aborted or abandoned transactions will have no effect on the database state. 45 | 46 | ## Serializability 47 | Assume each transaction is a node. Two transactions are said to be in conflict if they have RW, WR or WW conflict over a resource. For each such conflict add an edge between the nodes. 48 | 49 | A set of transactions are serializable only if the graph is acyclic. 50 | 51 | 52 | # TODO 53 | 54 | 1. Vector Clocks 55 | 1. Byzantian conditions 56 | 1. [Distributed Atomicity CockroachDB](https://www.cockroachlabs.com/blog/how-cockroachdb-distributes-atomic-transactions/) 57 | 1. [Distributed Isolation CockroachDB](https://www.cockroachlabs.com/blog/serializable-lockless-distributed-isolation-cockroachdb/) 58 | -------------------------------------------------------------------------------- /distributed-system/time-calculation.md: -------------------------------------------------------------------------------- 1 | # Time Calculation Quiz 2 | 3 | **Question**: How much time does it require to flush Log-buffer in a database? 4 | 5 | **Answer**: 6 | 7 | A Log-buffer has capacity of [16 MB](http://www.dba-oracle.com/t_log_buffer_optimal_size.htm). 8 | 9 | So we have to flush 16 MB of data from RAM to some log file on disk in append only format. 10 | 11 | So first, we have to find the log file on disk. This would require Disk Access time = disk seek + rotational latency, $10 ms + 5 ms = 15 ms$. 12 | 13 | Next, a HDD can read/write at a speed of 100 MB/s. Hence, to flush 16 MB of data we will need $16 / 100 = 0.16s = 160 ms$. 14 | 15 | Therefore, it will take $160+15 = 175ms$ in total to flush the log buffer. 16 | 17 | 18 | # Time Table 19 | 20 | | type | time | 21 | | :------------- | :------------- | 22 | | HDD Seek Time | 10 ms | 23 | | SSD Seek Time | 0.10 ms (100X faster than HDD) | 24 | | Rotational Latency | 5 ms | 25 | | Disk-to-Buffer | 100 MB/s 26 | 27 | **Reference**: 28 | 29 | 1. [Hard Disk Drive Performance](https://en.wikipedia.org/wiki/Hard_disk_drive_performance_characteristics) 30 | -------------------------------------------------------------------------------- /favorite_blogposts.md: -------------------------------------------------------------------------------- 1 | # JOEL ON SOFTWARE 2 | 3 | 1. [Developers’ side projects](https://www.joelonsoftware.com/2016/12/09/developers-side-projects/) 4 | 5 | Who owns the code of your pet project? You or your company? 6 | 7 | 1. [Things You Should Never Do, Part I](https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/) 8 | 9 | Legacy codes are not mess. They are just hard to read. Starting from scratch is not the solution. 10 | 11 | 1. [The Joel Test: 12 Steps to Better Code](https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-steps-to-better-code/) 12 | 13 | Fix bugs before writing new code. 14 | 15 | # Julia Evans 16 | 17 | 1. [What happens when you start a process on Linux?](https://jvns.ca/blog/2016/10/04/exec-will-eat-your-brain/) 18 | - [fork() can fail: this is important](https://rachelbythebay.com/w/2014/08/19/fork/) 19 | 1. -------------------------------------------------------------------------------- /images/common-text-function-sql.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/images/common-text-function-sql.png -------------------------------------------------------------------------------- /images/computer-networking-go-back-n-protocol.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/images/computer-networking-go-back-n-protocol.png -------------------------------------------------------------------------------- /images/computer-networking-packet-arrival-protocol.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/images/computer-networking-packet-arrival-protocol.png -------------------------------------------------------------------------------- /images/eee452.001.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/images/eee452.001.png -------------------------------------------------------------------------------- /images/join.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/images/join.jpg -------------------------------------------------------------------------------- /images/process-control-block-field.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/images/process-control-block-field.png -------------------------------------------------------------------------------- /images/process_in_memory.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/images/process_in_memory.png -------------------------------------------------------------------------------- /images/process_states.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/images/process_states.png -------------------------------------------------------------------------------- /images/proxy.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/images/proxy.png -------------------------------------------------------------------------------- /images/pthread-function-calls.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/images/pthread-function-calls.png -------------------------------------------------------------------------------- /images/thread-sharing.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/images/thread-sharing.png -------------------------------------------------------------------------------- /images/where-clause-operators.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/images/where-clause-operators.png -------------------------------------------------------------------------------- /information_retrieval.md: -------------------------------------------------------------------------------- 1 | # Information Retrieval 2 | 3 | # Boolean Retrieval 4 | 5 | ## Inverted Index 6 | Usually, books have index at the end. If we consider each book as a document, then the index gives us where in the document can we find a particular word in the document. So the index belongs to the documents in this case. 7 | 8 | An inverted index is the opposite and hence, documents belong to index. When there are multiple documents, a word in inverted index points to which documents contain that particular word. 9 | 10 | A document can contain the same word multiple time. We merge all the same instances and keep only one postings. Since the word can be found in multiple document, we store a list for the word. The list is sorted according to docID. 11 | 12 | ## Boolean Query 13 | 14 | How do we process queries such as: Brutus AND Calpurnia? 15 | 16 | Grab the list of the two words and intersect them. 17 | 18 | Optimization: Sort the queries in increasing order of postings. That way, AND operator will quickly converge. 19 | 20 | # The term vocabulary and postings lists 21 | 22 | ## Document Representation and Character Sequence Decoding 23 | 24 | We need to first figure out how text has been encoded. Could be ASCII or UTF-8. Determining the correct encoding can be regarded as classification problem from machine learning. But often, the encoding is found from documents metadata. 25 | 26 | # R&D 27 | 28 | Jaccob's Coefficient 29 | Phrase query 30 | 31 | Bag of words model 32 | Term frequency 33 | Relevance is not proportional to term frequency 34 | Log-frequency weighting 35 | Sum over all intersection 36 | 37 | # Document frequency weighting 38 | Rare terms > common terms 39 | high weight to rare terms 40 | df -> document frequency 41 | term frequency vs document frequency (df) 42 | $df_t$: number of documents containing t. It is an inverse measure of the informativeness of t. The higher the value of df(t), the less important t is, cause it is not rare. 43 | df(t) <= N 44 | 45 | But usually we want to associate higher score with better term, so we inverse it, N/df(t). 46 | 47 | idf(t) = log10(N/df(t)), log taken to dampen the effect. 48 | 49 | A word that occurs in every doc, idf(t) = 0. 50 | 51 | No effect on 1 term queries, it simply acts as scale. Works for multi word query. 52 | 53 | # Collection 54 | collection frequency of t is the number of occurrences of t in the collection, counting multiple occurrences. df is better. 55 | 56 | # tf-idf weighting 57 | 58 | w(term,doc) = (1+log tf(t,d)) * log(N/df(t)) 59 | 60 | # Vector Space Model 61 | 62 | Vector Space Model is an algebraic model for representing text documents as vectors of identifiers. 63 | 64 | Each document is represented as V dimensional vector, where V is the number of identifiers. All the documents together form V dimensional vector space. 65 | 66 | Even queries are represented as vector of identifiers. 67 | 68 | Euclidean distance vs angle 69 | Decreasing angle or increasing cosine 70 | cosine similarity 71 | Easy to calculate using vector 72 | Length normalize queries and documents in advance 73 | 74 | # tf-idf cosine score in IR system 75 | 76 | # Evaluate search engine 77 | fast index 78 | fast search 79 | query language 80 | -------------------------------------------------------------------------------- /language/css/bootstrap4.md: -------------------------------------------------------------------------------- 1 | # Bootstrap 2 | 3 | # Buttons 4 | 5 | ## Full width button 6 | 7 | Create block level buttons—those that span the full width of a parent—by adding .btn-block. 8 | 9 | # Layout 10 | 11 | ## Vertical Alignment 12 | 13 | ``` 14 | 15 | ... all col gets aligned vertically 16 | 17 | ``` 18 | -------------------------------------------------------------------------------- /language/css/element-ui.md: -------------------------------------------------------------------------------- 1 | # Element UI 2 | 3 | ## Table 4 | 5 | ```pug 6 | el-table(:data="getUserFields") 7 | el-table-column(prop="any-object-key" label="Label it whatever you want") 8 | el-table-column 9 | template(slot-scope="scope") 10 | /* Access using scope.row */ 11 | fa-icon.vertical-middle(:name="scope.row.icon") 12 | 13 | 14 | // DATA 15 | getUserFields() { 16 | return [{ 17 | feature: 'Email', 18 | icon: 'envelope', 19 | value: this.user.email 20 | }] 21 | } 22 | ``` 23 | -------------------------------------------------------------------------------- /language/golang/assets/markdown-img-paste-20180718120834171.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/language/golang/assets/markdown-img-paste-20180718120834171.png -------------------------------------------------------------------------------- /language/golang/context.md: -------------------------------------------------------------------------------- 1 | # Context 2 | 3 | The context package provides propagated cancellation feature. 4 | 5 | Suppose you asked Tom to make a sandwitch and he starts making one. It requires many steps like buying bread and tomatoes. Suppose Tom asked Jerry to go and buy the ingredients. Now suddenly you don't feel like eating sandwitch anymore and let Tom know that. Tom now needs to stop what he was doing. This is cancellation. Tom also needs to let Jerry know that the task was cancelled. This is cancellation propagation. 6 | 7 | Using context package, we can stop other goroutines or functions for example. 8 | 9 | In Go servers, each incoming request is handled in its own goroutine. Request handlers often start additional goroutines to access backends such as databases and RPC services. The set of goroutines working on a request typically needs access to request-specific values such as the identity of the end user, authorization tokens, and the request's deadline. When a request is canceled or times out, all the goroutines working on that request should exit quickly so the system can reclaim any resources they are using. 10 | 11 | Context Provides the following methods: 12 | 13 | - Background 14 | - WithCancel 15 | - WithTimeout 16 | - WithDeadline 17 | 18 | > A Context carries a deadline, cancelation signal, and request-scoped values across API boundaries. Its methods are safe for simultaneous use by multiple goroutines. 19 | 20 | ## Derived Contexts 21 | 22 | - The context package provides functions to derive new Context values from existing ones. 23 | - These values form a tree: when a Context is canceled, all Contexts derived from it are also canceled. 24 | - Background 25 | - Background is the root of any Context tree; it is never canceled 26 | - Background returns an empty Context. It is never canceled, has no deadline, and has no values. Background is typically used in main, init, and tests, and as the top-level Context for incoming requests. 27 | 28 | ## Resource 29 | 30 | 1. [Youtube - justforfunc #9: The Context Package](https://www.youtube.com/watch?v=LSzR0VEraWw) 31 | 2. [The Go Blog - Go Concurrency Patterns: Context](https://blog.golang.org/context) 32 | 3. [Medium - How to correctly use context.Context in Go 1.7](https://medium.com/@cep21/how-to-correctly-use-context-context-in-go-1-7-8f2c0fafdf39) -------------------------------------------------------------------------------- /language/golang/embedding.md: -------------------------------------------------------------------------------- 1 | # Embedding in Go 2 | 3 | ## Resources 4 | 5 | 1. [Type embedding in Go](https://travix.io/type-embedding-in-go-ba40dd4264df) 6 | 2. [Eric Urban - Embedding in Go](http://www.hydrogen18.com/blog/golang-embedding.html) -------------------------------------------------------------------------------- /language/golang/golang.md: -------------------------------------------------------------------------------- 1 | # GoLang 2 | 3 | ## Build 4 | 5 | ```bash 6 | env GOOS=linux GOARCH=amd64 go build $FILEPATH 7 | ``` 8 | 9 | ## Basic 10 | 11 | 1. [Package](./package.md) 12 | - init & main 13 | - Execution Flow 14 | 1. [Slices](./slices.md) 15 | - Array vs Slice 16 | - Length & Capacity 17 | - Init, append and loop over 18 | 1. [Map](./map) 19 | 1. [Interface](./interface.md) 20 | 21 | ## Basic Questions 22 | 23 | 1. [String vs String Pointers](https://dhdersch.github.io/golang/2016/01/23/golang-when-to-use-string-pointers.html) 24 | 25 | ## Advanced 26 | 27 | 1. [Profiling with pprof](./pprof.md) 28 | - go-torch 29 | 1. [Go Routine](./goroutine.md) 30 | 1. [Mutex](./mutex.md) 31 | 1. [Reflection](./reflection.md) 32 | 1. [Embedding in Go](./embedding.md) 33 | 34 | ## Testing 35 | 36 | - Fail vs Errorf vs Fatalf 37 | - table of tests: slice of struct 38 | - Add name to tests 39 | - Sub Tests 40 | - http handler test 41 | - http.NewRecorder(); rec.Result() 42 | - defer res.Body.Close() 43 | - ioutil.ReadAll(res.body()) 44 | 45 | ## Blog/Tutorials 46 | 47 | 1. [Singleton Pattern in Go](http://marcio.io/2015/07/singleton-pattern-in-go/) 48 | - Check-Lock-Check Pattern 49 | - `sync.Once` 50 | - [Double-checked locking: Clever, but broken](https://www.javaworld.com/article/2074979/java-concurrency/double-checked-locking--clever--but-broken.html) 51 | 2. Just For Func #1: A Code Review 52 | 1. Do not capitalize error codes. 53 | 2. resp.Status vs resp.StatusCode. 54 | 3. Close channels to avoid goroutine leak. -------------------------------------------------------------------------------- /language/golang/goroutine.md: -------------------------------------------------------------------------------- 1 | # Go Routine 2 | 3 | ## Wait for all Go Routine to Finish 4 | 5 | ```go 6 | var wg sync.WaitGroup 7 | 8 | wg.Add(1) 9 | go func() { 10 | defer wg.Done() 11 | // do stuff 12 | }() 13 | wg.Wait() 14 | ``` 15 | 16 | Reference: [How to Wait for All Goroutines to Finish Executing Before Continuing](https://nathanleclaire.com/blog/2014/02/15/how-to-wait-for-all-goroutines-to-finish-executing-before-continuing/) 17 | 18 | ## Shutdown worker by closing channel 19 | 20 | ```go 21 | func main() { 22 | jobs := make(chan int, 5) 23 | done := make(chan bool) 24 | 25 | go func() { 26 | for { 27 | j, more := <-jobs 28 | if more { 29 | fmt.Println("received job", j) 30 | } else { 31 | fmt.Println("received all jobs") 32 | done <- true 33 | return 34 | } 35 | } 36 | }() 37 | 38 | for j := 1; j <= 3; j++ { 39 | jobs <- j 40 | fmt.Println("sent job", j) 41 | } 42 | close(jobs) 43 | fmt.Println("sent all jobs") 44 | 45 | <-done 46 | } 47 | ``` 48 | 49 | 50 | Reference [Go by Example: Closing Channels](https://gobyexample.com/closing-channels) -------------------------------------------------------------------------------- /language/golang/interface.md: -------------------------------------------------------------------------------- 1 | # Interface 2 | 3 | ## Resources 4 | 5 | 1. [Advance] [Go Data Structures: Interfaces](https://research.swtch.com/interfaces) 6 | -------------------------------------------------------------------------------- /language/golang/map.md: -------------------------------------------------------------------------------- 1 | # Map 2 | 3 | Key to Value map. Key can be any comparable element. Value can be literally anything. 4 | 5 | ## Using it 6 | 7 | - Map is like a slice. A pointer. So you must inititate it first 8 | - Unassigned map is like a nil pointer. 9 | - Nil Map: you can still "get" keys from them, but trying to assign to it will panic 10 | 11 | ```go 12 | // Initiate 13 | m = make(map[string]int) 14 | m = map[string]int{ 15 | "hello": 1, 16 | "world": 2, 17 | } 18 | 19 | // Access 20 | v := m[key] 21 | v, ok := m[key] 22 | 23 | // Length 24 | len(m) 25 | 26 | // Delete key 27 | delete(m, key) 28 | 29 | // Iterate 30 | for key, value := range m {} 31 | ``` -------------------------------------------------------------------------------- /language/golang/mutex.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/language/golang/mutex.md -------------------------------------------------------------------------------- /language/golang/package.md: -------------------------------------------------------------------------------- 1 | # Package 2 | 3 | - A package is a directory: **package = directory** 4 | - A package cannot have sub-directory. 5 | - All files in a package must be in same directory 6 | - All files must start with a package declaration 7 | 8 | ## Export 9 | 10 | To export a function, variable or anything, just capitalize the first letter. 11 | 12 | ## Name of packages 13 | 14 | - Keep them small and short. Usually one lowercase word. 15 | - No underscores 16 | 17 | ## main function 18 | 19 | From the spec: [Program execution](https://golang.org/ref/spec#Program_execution) 20 | > A complete program is created by linking a single, unimported package called the main package with all the packages it imports, transitively. The main package must have package name main and declare a function main that takes no arguments and returns no value. 21 | > 22 | > func main() { … } 23 | > 24 | > Program execution begins by initializing the main package and then invoking the function main. When that function invocation returns, the program exits. It does not wait for other (non-main) goroutines to complete. 25 | 26 | Therefore, a complete program must have a main package with a main function. Program will start from main function and once it returns, program will end. 27 | 28 | ## init function 29 | 30 | A package can also have an init function. It is guranteed that init function will run before main function. 31 | 32 | ## Execuation flow of packages 33 | 34 | ![](assets/markdown-img-paste-20180718120834171.png) 35 | 36 | First imports will be called. Next constants will be declared. After that variables will be initiated. Next comes init() and finally main() if any. 37 | 38 | Reference: [StackOverflow: When is the init() function run?](https://stackoverflow.com/questions/24790175/when-is-the-init-function-run) 39 | 40 | # Resources 41 | 42 | 1. [Go Talks - Organizing Go Code](https://talks.golang.org/2014/organizeio.slide#4) 43 | -------------------------------------------------------------------------------- /language/golang/pprof.md: -------------------------------------------------------------------------------- 1 | # Profilingd 2 | 3 | Read [How to profile Go applications inside a docker container](https://bruinsslot.jp/post/profiling-golang-docker/) 4 | 5 | ## Running pprof on Docker 6 | 7 | ### Step 1 8 | 9 | Add these to your main function 10 | 11 | ``` 12 | // main.go 13 | import ( 14 | _ "net/http/pprof" 15 | _ "expvar" 16 | ) 17 | 18 | func main() { 19 | go func() { 20 | http.ListenAndServe("0.0.0.0:6060", nil) 21 | }() 22 | ``` 23 | 24 | ### Step 2 25 | 26 | Then open 6060 port in your container through compose file. 27 | 28 | ``` 29 | // docker-compose.yaml 30 | version: "3" 31 | services: 32 | app: 33 | build: ./build-dir 34 | ports: 35 | - "6060:6060" 36 | ``` 37 | 38 | ### Step 3 39 | 40 | Start a cpu profile from shell by running the command: 41 | 42 | ``` 43 | $ go tool pprof http://localhost:6060/debug/pprof/profile 44 | ``` 45 | 46 | ### Step 4 47 | 48 | While running step 3, we need to hit our server with http requests so that it does something. Otherwise the profile would be empty. For web server we can hit our server with benchmarking tool like wrk. 49 | 50 | ### Step 5 51 | 52 | Once profiling is complete, you will be taken to a pprof interactive shell. 53 | 54 | ## pprof Shell 55 | 56 | ``` 57 | web # Opens the profile in chrome browser as a drawing 58 | top10 # To show top 10 functions by time spent only in that function 59 | top10 -cum # Top 10 functions by time spent by that function or function it called 60 | ``` 61 | 62 | ## Running go-torch on Docker 63 | 64 | Same thing as pprof. Once everything has been setup: 65 | 66 | ``` 67 | go-torch -u http://localhost:6060 -t 30 -f custom-name.svg 68 | ``` 69 | 70 | Make sure you open the svg in a chrome browser. The blocks are clickable. Colors do not mean anything. What matters is length. Longer blocks mean longer time. The whole stack is displayed. 71 | -------------------------------------------------------------------------------- /language/golang/reflection.md: -------------------------------------------------------------------------------- 1 | # Reflection 2 | 3 | - > Reflection in a computing is the ability of a program to exaine its own structure, particulary through types; it's a form of metaprogramming. - Rob Pike at The Laws of Reflection 4 | 5 | - > It's a power tool that should be used with care and avoided unless strictly necessary - Rob Pike at The Laws of Reflection 6 | 7 | - > Compared to a disk seek or network transfer, the cost of reflection will be negligible. - Dave Cheney 8 | 9 | - Without Reflection, you will have to write different code for every type when doing data transformation or resource injection. 10 | 11 | ## Kind vs Type 12 | 13 | Kind is builtin type. Type is user defined type. 14 | 15 | ## Relation to Interface 16 | 17 | Assigning an object to interface does not change the output of reflect. 18 | 19 | ## API 20 | 21 | 1. `reflect.ValueOf()` 22 | 2. `rv.Kind()` 23 | 3. `rv.Elem()` 24 | 4. `rv.NumField()` 25 | 26 | ```go 27 | import reflect; 28 | 29 | func display (v interface{}) { 30 | // Inspect the concrete type value that is passed in 31 | rv := reflect.ValueOf(v) 32 | 33 | if (rv.Kind() == reflect.Ptr) { 34 | rv = rv.Elem() 35 | } 36 | } 37 | ``` 38 | 39 | ## Laws of Reflection 40 | 41 | ### First Law: Reflection goes from interface value to reflection object 42 | 43 | I was really confused about what this line meant. Later I realized, they are simply introducing some functions that take `interface{}` as input and spits out `reflect` object as output. 44 | 45 | The functions introduced are: `reflect.TypeOf` and `reflect.ValueOf` which extracts `reflect.Type` and `reflect.Value` correspondingly. 46 | 47 | ### Second Law: Reflection goes from reflection object to interface value 48 | 49 | Basically, we have a function to convert `reflect` object to interface again. 50 | 51 | ```go 52 | func (v Value) Interface() interface{} 53 | ``` 54 | 55 | ### Third Law: To modify a reflection object, the value must be settable 56 | 57 | Not all reflect object have settable properties. 58 | 59 | ```go 60 | var x float64 = 3.4 61 | v := reflect.ValueOf(x) 62 | fmt.Println(v.CanSet()) // false 63 | v.SetFloat(7.1) // Error: will panic. 64 | ``` 65 | 66 | Basically, when we set a property of a reflect object, we want that to reflect in the original value too. But this is not possible above, as reflect object was provided a copy of x. 67 | 68 | In order to allow it to change the original value, we need to pass pointer. 69 | 70 | ```go 71 | var x float64 = 3.4 72 | p := reflect.ValueOf(&x) // Note: take the address of x. 73 | fmt.Println("type of p:", p.Type()) // Output: type of p: *float 74 | fmt.Println("settability of p:", p.CanSet()) // Output: settability of p: false 75 | 76 | v := p.Elem() // dereference the pointer 77 | fmt.Println("settability of v:", v.CanSet()) // Output: settability of v: true 78 | ``` 79 | 80 | ## Application of Reflect 81 | 82 | Read more on [Learning to Use Go Reflection — Part 2](https://medium.com/capital-one-developers/learning-to-use-go-reflection-part-2-c91657395066) 83 | 84 | - Marshaling and Unmarshaling data from a file or a network. 85 | 86 | ## Resources 87 | 88 | 1. [Go Blog: The Laws of Reflection](https://blog.golang.org/laws-of-reflection) 89 | 2. [GopherCon 2016: Emily Gu - The Power and Elegance of Reflection](https://www.youtube.com/watch?v=lI17OEJCPVw) 90 | 3. [Learning to Use Go Reflection — Part 2](https://medium.com/capital-one-developers/learning-to-use-go-reflection-part-2-c91657395066) -------------------------------------------------------------------------------- /language/golang/slices.md: -------------------------------------------------------------------------------- 1 | # Slices 2 | 3 | Slices are references to arrays. 4 | 5 | # Difference between slice and array 6 | 7 | Array has a fixed length. Slices are dynamic arrays. 8 | 9 | ```go 10 | var a [10]int 11 | // The length of the array is a part of it 12 | ``` 13 | 14 | Arrays can be hashed, slices cannot. 15 | 16 | # Length vs Capacity 17 | 18 | The length of a slice is the number of elements it contains. 19 | 20 | The capacity of a slice is the number of elements in the underlying array, counting from the first element in the slice. 21 | 22 | The length and capacity of a slice s can be obtained using the expressions `len(s)` and `cap(s)`. 23 | 24 | ## Nil Slices 25 | 26 | If slice has length 0, then it is a nil slice 27 | 28 | ```go 29 | var s []int 30 | if s == nil { 31 | fmt.Println("nil") 32 | } 33 | ``` 34 | 35 | # Initiate Slice 36 | 37 | Slices can be initated by assigning values directly. 38 | 39 | ```go 40 | x := []bool{true, false, true} 41 | ``` 42 | 43 | But what if we don't have initiate values and just want an empty slice? 44 | 45 | For that we have make. 46 | 47 | ```go 48 | a := make([]int, length, capacity) 49 | ``` 50 | 51 | # Append to Slice 52 | 53 | We can append items to slice using `append()` method. 54 | 55 | ```go 56 | func append(s []T, vs ...T) []T 57 | s := make(int[], 2) 58 | s = append(s, 1, 2, 3, 4, 5, 6) 59 | ``` 60 | 61 | # Iterating over Slice 62 | 63 | For iterating we have `range()` method 64 | 65 | ```go 66 | var pow = []int{1, 2, 4, 8, 16, 32, 64, 128} 67 | for i, v := range pow { 68 | fmt.Printf("2**%d = %d\n", i, v) 69 | } 70 | ``` 71 | `range()` returns index and copy of each value in slice. 72 | -------------------------------------------------------------------------------- /language/java/java.md: -------------------------------------------------------------------------------- 1 | # Memory Management 2 | 3 | | Stack | Heap | 4 | | :------------- | :------------- | 5 | | Where local variables and method invocation lives | Where object lives | 6 | 7 | - When JVM starts, it gets a chunk of memory from OS. 8 | - All objects are garbage-collectable on heap. 9 | - Xms Initial Java heap size; Xmx Maximum Java heap size; Xmn Size of the heap 10 | - For optimal performance, Xmn should be lower than Xmx 11 | - No memory for local variable or function calls, JVM will throw StackOverFlowError 12 | - No memory for object creation will throw OutOfMemoryError 13 | - Stack memory is visible to only its own thread, whereas heap memory is shared among threads 14 | - The heap is sometimes divided into two areas (or generations) called the nursery (or young space) and the old space. 15 | - The nursery is a part of the heap reserved for allocation of new objects. When the nursery becomes full, garbage is collected by running a special young collection, where all objects that have lived long enough in the nursery are promoted (moved) to the old space, thus freeing up the nursery for more object allocation. When the old space becomes full garbage is collected there, a process called an old collection. 16 | - The reasoning behind a nursery is that most objects are temporary and short lived. A young collection is designed to be swift at finding newly allocated objects that are still alive and moving them away from the nursery. Typically, a young collection frees a given amount of memory much faster than an old collection or a garbage collection of a single-generational heap (a heap without a nursery). 17 | -------------------------------------------------------------------------------- /language/javascript/async-await.md: -------------------------------------------------------------------------------- 1 | # Async-Await 2 | 3 | # Resource 4 | 5 | 1. [6 Reasons Why JavaScript’s Async/Await Blows Promises Away (Tutorial)](https://hackernoon.com/6-reasons-why-javascripts-async-await-blows-promises-away-tutorial-c7ec10518dd9) 6 | -------------------------------------------------------------------------------- /language/javascript/error-handling.md: -------------------------------------------------------------------------------- 1 | # Error Handling 2 | 3 | # Throw Error 4 | 5 | ``` 6 | throw new Error("some message"); 7 | ``` 8 | 9 | # Error.name Property 10 | 11 | Catch error using if-else and error.name 12 | -------------------------------------------------------------------------------- /language/javascript/jquery.md: -------------------------------------------------------------------------------- 1 | # Jquery 2 | 3 | # oninput vs onchange 4 | 5 | The oninput event occurs when an element gets user input. ... Tip: This event is similar to the onchange event. The difference is that the oninput event occurs immediately after the value of an element has changed, while onchange occurs when the element loses focus, after the content has been changed. 6 | -------------------------------------------------------------------------------- /language/javascript/lodash.md: -------------------------------------------------------------------------------- 1 | # Lodash 2 | 3 | ## Loop for N times 4 | ``` 5 | for(var i = 0; i < 5; i++) { 6 | // .... 7 | } 8 | 9 | _.times(5, function(i){ 10 | // ... 11 | }); 12 | ``` 13 | 14 | ## Deep-cloning Javascript object 15 | 16 | ``` 17 | var objA = { 18 | "name": "colin" 19 | } 20 | 21 | // Normal method? Too long. See Stackoverflow for solution: http://stackoverflow.com/questions/4459928/how-to-deep-clone-in-javascript 22 | 23 | // Lodash 24 | var objB = _.cloneDeep(objA); 25 | objB === objA // false 26 | ``` 27 | 28 | However, this will only work if there are no function within the object. 29 | 30 | # Resource 31 | 32 | 1. [Lodash: 10 Javascript Utility Functions That You Should Probably Stop Rewriting](https://colintoh.com/blog/lodash-10-javascript-utility-functions-stop-rewriting) 33 | -------------------------------------------------------------------------------- /language/javascript/puppeteer.md: -------------------------------------------------------------------------------- 1 | # Puppeteer 2 | 3 | ``` 4 | const browser = await puppeteer.launch({ 5 | args: ['--no-sandbox', '--disable-setuid-sandbox'], 6 | headless: false, 7 | }); 8 | const page = await browser.newPage(); 9 | 10 | await page.goto(`http://www.lightoj.com/login_main.php`, {waitUntil: 'networkidle2'}); 11 | 12 | await page.waitFor(100000); 13 | await browser.close(); 14 | ``` 15 | -------------------------------------------------------------------------------- /language/javascript/redux.md: -------------------------------------------------------------------------------- 1 | # State Management 2 | 3 | ## What to put in redux state? 4 | 5 | 1. Should this data be persisted between page refreshes? 6 | 1. Should this data be persisted across route changes? 7 | 1. Is this data used in multiple places in the UI? 8 | 9 | If answer to any of the question is yes, then it should go to redux state. 10 | -------------------------------------------------------------------------------- /language/javascript/string.md: -------------------------------------------------------------------------------- 1 | # String 2 | 3 | # strcmp() 4 | 5 | ``` 6 | str1.localeCompare(str2); 7 | // Returns -1, 0 or 1 8 | ``` 9 | -------------------------------------------------------------------------------- /language/latex.md: -------------------------------------------------------------------------------- 1 | # Latex 2 | 3 | # Labeling equations 4 | 5 | **Source** 6 | ``` 7 | $a + b = c \tag{1} \label{1}$ 8 | 9 | As we can see from $\eqref{1}$, ... 10 | ``` 11 | 12 | **Preview** 13 | 14 | $a + b = c \tag{1}$ 15 | 16 | As we can see from $\eqref{1}$, ... 17 | 18 | # Symbols 19 | 20 | ## Set 21 | - Union: $\cup$ 22 | - Set Subtract $\setminus$ 23 | - Intersection: $\cap$ 24 | -------------------------------------------------------------------------------- /language/python/csv.md: -------------------------------------------------------------------------------- 1 | # csv 2 | 3 | ```python 4 | import csv 5 | 6 | with open(filePath,"r") as f: 7 | fieldnames = ['first_name', 'last_name'] 8 | dict = csv.DictReader(f, delimiter=',', fieldnames=fieldnames) 9 | ``` 10 | -------------------------------------------------------------------------------- /language/python/django.md: -------------------------------------------------------------------------------- 1 | # Django 2 | 3 | ``` 4 | django-admin startproject [name] # Creates a new project 5 | 6 | python manage.py runserver # Starts development server at port 8000 7 | python manage.py startapp [name] # Start new app 8 | 9 | 10 | python manage.py migrate # Creates/Edits tables in db 11 | python manage.py makemigrations [name]# Where name is app config 12 | python manage.py sqlmigrate [app] [migrateID] # View corresponding sql commands 13 | 14 | python manage.py createsuperuser # Creates superuser 15 | ``` 16 | 17 | 18 | # Request and Response Object Structure 19 | -------------------------------------------------------------------------------- /language/python/general.md: -------------------------------------------------------------------------------- 1 | # File 2 | 3 | ``` 4 | f = open('workfile', 'w') 5 | f.write(string) 6 | f.close() 7 | ``` 8 | 9 | # List 10 | 11 | ``` 12 | myList.append(["a", "true"]) # Array appended 13 | myList.extend(["a", "true"]) # Elements appended separately 14 | 15 | myList.remove('a') # Remove a single 'a' 16 | myList = [x for x in myList if x != 'a'] # Multiple 'a' removed 17 | 18 | # Sorting 19 | 20 | ## Sorting list of objects by key 21 | newlist = sorted(list_to_be_sorted, key=lambda k: k['name']) 22 | ``` 23 | 24 | # Print 25 | 26 | ```python 27 | # Print without newline 28 | print('.', end='') 29 | ``` 30 | 31 | # String 32 | 33 | ``` 34 | # Long string without newline break 35 | s = ("this is a very" 36 | "long string too" 37 | "for sure ..." 38 | ) 39 | 40 | x = 'blue,red,green' 41 | x.split(",") # ['blue', 'red', 'green'] 42 | ``` 43 | -------------------------------------------------------------------------------- /language/python/modules.md: -------------------------------------------------------------------------------- 1 | # Pickle 2 | 3 | ``` 4 | import pickle 5 | 6 | obj = {'example': 'example'} 7 | 8 | "dump to file" 9 | with open('file', 'wb') as file: "notice the b flag in open" 10 | pickle.dump(obj, file) 11 | 12 | "read from file" 13 | with open('file', 'wb') as file: 14 | obj = pickle.load(file) 15 | ``` 16 | 17 | 18 | -------------------------------------------------------------------------------- /language/python/pip.md: -------------------------------------------------------------------------------- 1 | # pip 2 | 3 | ``` 4 | apt install python3-pip # install 5 | 6 | pip3 install --user # force local package installation 7 | pip3 install -r requirement.txt # install from file 8 | 9 | pip3 install --upgrade pip # upgrade 10 | pip3 uninstall # uninstall 11 | pip3 freeze # Output installed packages in requirements format. 12 | ``` 13 | -------------------------------------------------------------------------------- /language/python/regex.md: -------------------------------------------------------------------------------- 1 | # Regex 2 | 3 | 4 | If you need to match at the beginning of the string, or to match the entire string use `match`. It is faster. Otherwise use `search`. 5 | 6 | ``` 7 | import re 8 | 9 | m = re.match(pattern, string, flag) 10 | m.group(x) # Find x captured object. 1 based index. 11 | ``` 12 | -------------------------------------------------------------------------------- /language/python/script.md: -------------------------------------------------------------------------------- 1 | # Shebang 2 | 3 | `#!/usr/bin/env python3` 4 | 5 | # argparse 6 | 7 | ``` 8 | import argparse 9 | 10 | parser = argparse.ArgumentParser() 11 | 12 | # Positional argument 13 | parser.add_argument("echo", help="echo the string you use here") 14 | 15 | # argparse treats the options we give it as strings, unless we tell it otherwise. 16 | parser.add_argument("square", type=int, help="display a square of a given number") 17 | 18 | # Optional argument (if absent, value is set to None) 19 | parser.add_argument("--verbosity", help="increase output verbosity") 20 | 21 | # Optional boolean argument ( checks for presence of flag ) 22 | parser.add_argument("--verbose", help="increase output verbosity", action="store_true") 23 | 24 | # Short version 25 | parser.add_argument("-v", "--verbose", help="increase output verbosity", action="store_true") 26 | 27 | args = parser.parse_args() 28 | ``` 29 | **Learn more**: [Argparser turorial](https://docs.python.org/3/howto/argparse.html#id1) 30 | 31 | # pathlib 32 | 33 | ``` 34 | from pathlib import Path 35 | 36 | # current working directory 37 | str(Path.cwd()) 38 | 39 | # Join paths 40 | str(Path('/', 'Users', 'stephanfitzpatrick', 'git')) 41 | 42 | # Open files 43 | with Path('.', 'truth.txt').open() as foo: 44 | print(foo.read()) 45 | 46 | # Make path absolute 47 | filepath = Path.resolve(filepath); 48 | ``` 49 | 50 | **Learn more**: [Pathlib doc](https://docs.python.org/3/library/pathlib.html#pathlib.Path) 51 | 52 | # Resource 53 | 54 | 1. [Ultimate script tutorial](http://www.dreamsyssoft.com/python-scripting-tutorial/) 55 | -------------------------------------------------------------------------------- /language/python/virtualenv.md: -------------------------------------------------------------------------------- 1 | # Virtualenv 2 | 3 | ``` 4 | pip3 install --user virtualenv # Install 5 | virtualenv mycoolproject # Create new project 6 | source mycoolproject/bin/activate # Activate virtuaenv 7 | deactivate # Deactivate virtualenv 8 | ``` 9 | 10 | # Virtualenvwrapper 11 | 12 | It's a wrapper over basic virtualenv. Has extra features such as postactivate and postdeactivate scripts for maintaining environment variables. 13 | 14 | ``` 15 | pip3 install --user virtualenv 16 | pip3 install --user virtualenvwrapper 17 | mkdir ~/.virtualenvs 18 | 19 | # Add these to .bashrc 20 | export WORKON_HOME=$HOME/.virtualenvs 21 | export PROJECT_HOME=$HOME/me/projects/python # only needed for mkproject command 22 | export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3 23 | source $HOME/.local/bin/virtualenvwrapper.sh 24 | 25 | # Basic usage 26 | mkvirtualenv xyz # Creates new virtualenv at WORKON_HOME 27 | mkvirtualenv -p xyz # Different version of python 28 | workon # Displays list of all environment 29 | workon xyz # Switches to xyz environment 30 | rmvirtualenv xyz # Delete a venv. Deactivate before delete. 31 | lssitepackages # Show packages installed 32 | pip freeze > requirements.txt # Freeze list of your packages installed 33 | ``` 34 | 35 | # Reference 36 | 37 | 1. https://virtualenvwrapper.readthedocs.io/en/latest/command_ref.html 38 | 39 | # Troubleshoot 40 | If you perviously installed virtualenvwrapper using apt, and now its showing errors, try deleting the file in `/etc/bash_completion/virtualenvwrapper` file. 41 | -------------------------------------------------------------------------------- /language/python/wagtail.md: -------------------------------------------------------------------------------- 1 | # Wagtail 2 | 3 | ``` 4 | wagtail start [project] 5 | ``` 6 | -------------------------------------------------------------------------------- /language/python/web-scraping.md: -------------------------------------------------------------------------------- 1 | # Requests 2 | 3 | ``` 4 | import requests as rq 5 | 6 | response = rq.get(link) 7 | txt = response.text 8 | ``` 9 | 10 | # BeautifulSoup4 11 | 12 | ``` 13 | pip install beautifulsoup4 14 | 15 | from bs4 import BeautifulSoup as bs4 16 | 17 | soup = BeautifulSoup(response.text, 'html.parser') 18 | 19 | print(soup.prettify()) 20 | 21 | soup.select(css_selector) 22 | soup.get_text() 23 | ``` 24 | 25 | ## CSS Selector 26 | 27 | ``` 28 | element:nth-of-type(n) # Get nth tag 29 | ``` 30 | -------------------------------------------------------------------------------- /language/shell/bash.md: -------------------------------------------------------------------------------- 1 | # Double Expansion 2 | 3 | ```bash 4 | x="samiul" 5 | $x # "samiul" 6 | ${!x} # $samiul 7 | ``` 8 | 9 | It's like pointer. It points to a variable and we dereference it by `$((P)x)` 10 | 11 | # Loop 12 | 13 | ```bash 14 | for x in array; 15 | do 16 | echo $x; 17 | done 18 | ``` 19 | 20 | # Sub shell 21 | 22 | How to run command from shell? 23 | 24 | ```bash 25 | pwd = $(pwd) 26 | ``` 27 | 28 | `$()` creates a sub shell and runs the command. 29 | 30 | # Test 31 | 32 | `[[ ]]` is the extended test. It has less surprises. [Extended Test Manual](http://mywiki.wooledge.org/BashFAQ/031) 33 | 34 | `< > != =` all these are string comparison. Use `-ge -gt -ne` flags for integer comparison. 35 | 36 | ```bash 37 | # [ must be surrounded by space cause its a command 38 | if [ something ]; then 39 | echo "Something" 40 | elif [ something_else ]; then 41 | echo "Something else" 42 | else 43 | echo "None of the above" 44 | fi 45 | 46 | 47 | # More test commands 48 | if [ "$X" -lt "0" ] 49 | then 50 | echo "X is less than zero" 51 | fi 52 | if [ "$X" -gt "0" ]; then 53 | echo "X is more than zero" 54 | fi 55 | [ "$X" -le "0" ] && \ 56 | echo "X is less than or equal to zero" 57 | [ "$X" -ge "0" ] && \ 58 | echo "X is more than or equal to zero" 59 | [ "$X" = "0" ] && \ 60 | echo "X is the string or number \"0\"" 61 | [ "$X" = "hello" ] && \ 62 | echo "X matches the string \"hello\"" 63 | [ "$X" != "hello" ] && \ 64 | echo "X is not the string \"hello\"" 65 | [ -n "$X" ] && \ 66 | echo "X is of nonzero length" 67 | [ -f "$X" ] && \ 68 | echo "X is the path of a real file" || \ 69 | echo "No such file: $X" 70 | [ -x "$X" ] && \ 71 | echo "X is the path of an executable file" 72 | [ "$X" -nt "/etc/passwd" ] && \ 73 | echo "X is a file which is newer than /etc/passwd" 74 | 75 | # Escape meta characters for multiple condition inside if 76 | if [ $# != 1 -o \( $1 != "dev" -a $1 != "prod" \) ] ; then 77 | echo "Please enter a single argument to specify prod or dev" 78 | exit 0 79 | fi 80 | 81 | ``` 82 | 83 | # Flag and Option Parsing 84 | 85 | [How to get arguments with flags in bash script](https://stackoverflow.com/a/21128172/2042242) 86 | 87 | `getopts` does not support long flags. Single : means its has argument and :: means argument is optional. 88 | 89 | ```bash 90 | verbose='false' 91 | aflag='' 92 | bflag='' 93 | files='' 94 | 95 | while getopts 'abf:v' flag; do 96 | case "${flag}" in 97 | a) aflag='true' ;; 98 | b) bflag='true' ;; 99 | f) files="${OPTARG}" ;; 100 | v) verbose='true' ;; 101 | *) error "Unexpected option ${flag}" ;; 102 | esac 103 | done 104 | ``` -------------------------------------------------------------------------------- /language/shell/shell.md: -------------------------------------------------------------------------------- 1 | # Shell Scripting 2 | 3 | # How to check OS 4 | 5 | ``` 6 | var=$(uname -s) 7 | if [[ $var == "Darwin" ]]; then 8 | echo "Mac" 9 | fi 10 | ``` 11 | Possible names are: Darwin, Linux 12 | -------------------------------------------------------------------------------- /linux/linux.md: -------------------------------------------------------------------------------- 1 | ## Ack 2 | 3 | ``` 4 | sudo apt install ack 5 | ack [directory] # looks for the string in every file and lists them 6 | ``` 7 | 8 | ## Bash 9 | 10 | - [Double Quotes vs Single Quotes](https://stackoverflow.com/questions/6697753/difference-between-single-and-double-quotes-in-bash): Single quotes do not interpolate special characters. You can't use single quote inside single quote. 11 | 12 | ## Curl 13 | 14 | - **-o** flag for redirecting output 15 | - **-i** flag for showing hidden headers 16 | - **--head** flag for getting only headers 17 | - **--next** flag to separate multiple request 18 | - **--data|-d KEY=VALUE** to send post request 19 | - **curl -H "Host: yo" [link]** to add header file 20 | 21 | ## Environment Variables 22 | 23 | The environment is implemented as strings that represent key-value pairs. If multiple values are passed, they are typically separated by colon (:) characters 24 | 25 | `KEY=value1:value2:...` 26 | 27 | If the value contains significant white-space, quotations are used: 28 | 29 | `KEY="value with spaces"` 30 | 31 | The keys in these scenarios are variables. They can be one of two types, environmental variables or shell variables. 32 | 33 | **Environmental variables** are variables that are defined for the current shell and are inherited by any child shells or processes. Environmental variables are used to pass information into processes that are spawned from the shell. 34 | 35 | **Shell variables** are variables that are contained exclusively within the shell in which they were set or defined. They are often used to keep track of ephemeral data, like the current working directory. 36 | 37 | By convention, these types of variables are usually defined using all capital letters. This helps users distinguish environmental variables within other contexts. 38 | 39 | - **Printing Environment Variables**: use `printenv` to print all variables or `printenv VARIABLE` to print value of particular variable. 40 | - **Printing Shell Variables**: `set | grep VARIABLE` or `echo $VARIABLE` 41 | - **Creating Environment Variable**: `export VARIABLE` 42 | - **Demoting Environment Variable**: `export -n VARIABLE` 43 | - **Remove Variable**: `unset VARIABLE` 44 | 45 | **Process environment variables**: Each process has its corresponding env values located at `/proc/PID/environ`. 46 | 47 | ### Export 48 | 49 | `export VARIABLE` exposes variables to sub-process. This is useful in scripts. By exporting port numbers, I share value with docker-compose. 50 | 51 | ## File Compression 52 | 53 | Zip files: `zip -r FILENAME FILE-LIST-TO-ZIP` 54 | 55 | Extract: `extract [tar.gz,tgz,zip] ARCHIVE` 56 | 57 | `extract` is my own shell script. 58 | 59 | ## Procfile Management Tool 60 | 61 | Create a file named `Procfile` 62 | 63 | goreman (golang) 64 | 65 | ## Webmin 66 | 67 | A GUI through browser that allows us to configure our linux systems easily. For example, we can easily configure startup processes using webmin. 68 | 69 | [How to install](https://www.digitalocean.com/community/tutorials/how-to-install-webmin-on-ubuntu-16-04) 70 | 71 | You can stop/start/restart webmin from `/etc/webmin` 72 | 73 | You can view the dashboard from [https://ip6-localhost:10000/](https://ip6-localhost:10000/) 74 | 75 | ## Service 76 | 77 | View All Service: `service --status-all` 78 | 79 | ## Grep 80 | 81 | How to grep continuous stream of text? 82 | 83 | ```bash 84 | tail -f file | grep --line-buffered my_pattern 85 | ``` 86 | 87 | ## Continuous Read 88 | 89 | ```bash 90 | tail -f file 91 | tail -F file 92 | less +F file 93 | ``` 94 | -------------------------------------------------------------------------------- /linux/nginx.md: -------------------------------------------------------------------------------- 1 | # nginx 2 | 3 | A light weight reverse proxy. 4 | 5 | Suppose you have a server and you are running two web apps on it. One on port 8000 and one on 8001. Now suppose we have a domain name "example.com". We want "abc.example.com" to get routed to port 8000 of the machine and "xyz.example.com" to get routed to 8001 port of the machine. 6 | 7 | For that, we create 1 file for each web app we want to configure at `/etc/nginx/sites-enabled`. The file names could be anything. Let's just name them same as the sub-domain they represent. 8 | 9 | I mostly code in nodejs. So I usually have web app running on localhost. So the configuration looks like this: 10 | 11 | ``` 12 | server { 13 | listen 80; listen [::]:80; 14 | server_name abc.example.com; # <-- change this 15 | 16 | location / { 17 | proxy_pass http://localhost:8000; # <-- And this 18 | } 19 | } 20 | ``` 21 | 22 | # Starting and Stopping Nginx 23 | 24 | ``` 25 | nginx # start 26 | nginx -s stop 27 | nginx -s reload 28 | ``` 29 | 30 | # How to add a site 31 | 32 | ``` 33 | cd /etc/nginx/sites-enabled 34 | touch site.conf 35 | vim site.conf 36 | 37 | # Paste the following inside site.conf 38 | server { 39 | listen 80; listen [::]:80; 40 | server_name a.domain.com; # <-- change this 41 | 42 | location / { 43 | proxy_pass http://localhost:8080; 44 | proxy_set_header Host $http_host; 45 | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 46 | } 47 | } 48 | ``` 49 | 50 | # Adding SSL 51 | 52 | Adding ssl protection is very easy now. We just need to install certbot and use it. 53 | 54 | ``` 55 | # First install certbot 56 | $ sudo apt-get update 57 | $ sudo apt-get install software-properties-common 58 | $ sudo add-apt-repository ppa:certbot/certbot 59 | $ sudo apt-get update 60 | $ sudo apt-get install python-certbot-nginx 61 | 62 | # Next run it 63 | sudo certbot --nginx 64 | 65 | certbot will handle the rest. 66 | 67 | # Renew Certificate 68 | sudo apt upgrade certbot 69 | sudo certbot renew --dry-run 70 | sudo certbot renew 71 | 72 | # Check expiration dates 73 | certbot certificates 74 | ``` 75 | 76 | # How to enforce https? 77 | 78 | ``` 79 | if ($scheme != "https") { 80 | return 301 https://$host$request_uri; 81 | } # managed by Certbot 82 | ``` 83 | 84 | # Mixed content error 85 | 86 | We once got mixed content error after upgrading to ssl. In order to fix the issue, we simply had to add some headers to location block. 87 | 88 | ``` 89 | location / { 90 | proxy_pass http://localhost:8000; 91 | proxy_set_header Host $http_host; 92 | proxy_http_version 1.1; 93 | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 94 | proxy_set_header X-Forwarded-Proto https; 95 | } 96 | ``` 97 | -------------------------------------------------------------------------------- /linux/ssh.md: -------------------------------------------------------------------------------- 1 | # SSH 2 | 3 | 4 | ## Create SSH Key 5 | 6 | ``` 7 | ssh-keygen -t rsa -C EMAIL 8 | ssh-keygen -t rsa -f CUSTOM_FILE_NAME 9 | ``` 10 | 11 | ## Remove PassPhrase from existing key 12 | 13 | ``` 14 | ssh-keygen -p # Then it wil ask for existing key path, old pass and new pass 15 | ``` 16 | 17 | ## Add machine as trusted machine on server 18 | 19 | Create a file `~/.ssh/authorized_keys` and add public keys to it, one in each line. 20 | 21 | You can create `~/.ssh/config` to create shortcuts like `ssh SHORTCUT-NAME`. 22 | 23 | ## Config File 24 | 25 | Create a file: `~/.ssh/config` 26 | 27 | ```ssh 28 | Host example 29 | HostName 192.168.0.1 30 | port 22 31 | User forthright48 32 | IdentityFile ~/.ssh/id_rsa 33 | ``` -------------------------------------------------------------------------------- /machine-learning/support-vector-machine.md: -------------------------------------------------------------------------------- 1 | # Support Vector Machine 2 | 3 | A tool used to separate dataset such that the separation is as wide as possible. 4 | 5 | >In SVMs we are trying to find a decision boundary that maximizes the "margin" or the "width of the road" separating the positives from the negative training data points 6 | 7 | Imagine, we figured out the perfect separating line $S$. Now, from the origin, draw a unit vector $\vec{w}$ that connects to $S$ perpendicularly. Suppose this vector when extended cuts $S$ at $m$. Let the length from origin to $m$ be $c$. 8 | 9 | So now, when we get an unknown point $x$, how do we decide on which side of $S$ does it fall? We first consider it as a vector $\vec{u}$ and then project it upon $\vec{w}$. If the projection has length $ \ge c$, then it is on right side. 10 | 11 | $$\vec{w} \cdot \vec{u} \ge c $$ 12 | 13 | Without loss of generality: 14 | 15 | $$\vec{w} \cdot \vec{u} + b \ge 0 $$ 16 | 17 | where $b$ is a constant. 18 | -------------------------------------------------------------------------------- /math/group-theory.md: -------------------------------------------------------------------------------- 1 | # Group Theory 2 | 3 | Group -> Ring -> Field 4 | 5 | ## Group 6 | 7 | A group is a set $G$ with operation $*$ such that it has the following properties: 8 | 9 | - **Closure**: For all $x,y \in G$, $x*y = z\in G$ 10 | - **Associativity**: $a * (b * c) = (a * b) * c$ 11 | - **Identity**: There is an element $e \in G$ such that $e * g = g * e = g$ for all $g \in G$. 12 | - **Inverse**: For every $g \in G$ there exists $g^{-1} \in G$ such that $g * g^{-1} = g^{-1} * g = e$. 13 | 14 | A group does not need to be commutative, i.e, $x * y \not = y * x$. 15 | 16 | If a group is commutative then it is called **abelian**. 17 | 18 | If a set only has closure and associative property, then it is called **Semi-Group**. 19 | 20 | If a set has closure, associative and identity property, it is called **Monoid**. 21 | 22 | Example: $(Z,+)$, where $Z$ is the set of all integers. 23 | 24 | ## Ring 25 | 26 | A ring is a set $R$ with two operations $+$ and $*$, which has the following properties: 27 | 28 | - $R$ is a abelian group under $+$. 29 | - $R$ is monoid under $*$. 30 | - The operation $*$ distributes over $+$: 31 | $$a * (b + c) = ( a * b) + (a * c) \\\\ 32 | (a + b) * c = (a * c) + ( b * c)$$ 33 | 34 | ## Field 35 | 36 | A field is a set $F$ together with two operations $+$ and $*$ which has the following properties: 37 | 38 | - $F$ is commutative ring under $+$ and $*$. 39 | - Multiplicative Inverser: For every nonzero $f \in F$, there is $g \in F$ such that $f * g = g * f = 1$. 40 | 41 | For example, $Z_p$ is a field, where $Z_p$ is set of integers modulo a prime $p$. 42 | -------------------------------------------------------------------------------- /math/lagrange-multiplier.md: -------------------------------------------------------------------------------- 1 | # Lagrange Multiplier 2 | 3 | The Lagrange multiplier technique lets you find the maximum or minimum of a multivariable function $f(x,y,…)$ when there is some constraint on the input values you are allowed to use. 4 | 5 | This technique only applies to constraints that look something like this: $g(x,y,…)=c$. Here, $g$ is another multivariable function with the same input space as $f$, and $c$ is some constant. 6 | 7 | The core idea is to look for points where the contour lines of $f$ and $g$ are tangent to each other. 8 | -------------------------------------------------------------------------------- /math/partial-derivative.md: -------------------------------------------------------------------------------- 1 | # Partial Derivative 2 | 3 | Partial derivative is applicable for multivariable functions. Suppose we have the following function: 4 | 5 | $$ f(x,y) = x^2y$$ 6 | 7 | The $\partial$ sign is used to denote that all other variables are constant. Hence, 8 | 9 | $$ \frac{\partial f(x,y)}{\partial x} = 2x$$ 10 | 11 | , since $y$ is constant. 12 | 13 | # Symmetry of Second Derivative 14 | 15 | Technically, the symmetry of second derivatives is not always true. There is a theorem, referred to variously as Schwarz's theorem or Clairaut's theorem, which states that symmetry of second derivatives will always hold at a point if the second partial derivatives are continuous around that point. To really get into the meat of this, we'd need some real analysis. 16 | 17 | You should keep in the back of your mind that exceptions exist, but the symmetry of second derivatives work for just about every "normal" looking function that you will come across. 18 | 19 | # Gradient 20 | 21 | [TODO](https://www.khanacademy.org/math/multivariable-calculus/multivariable-derivatives/partial-derivative-and-gradient-articles/a/the-gradient) 22 | -------------------------------------------------------------------------------- /networking/basics/assets/markdown-img-paste-20180716225230715.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/basics/assets/markdown-img-paste-20180716225230715.png -------------------------------------------------------------------------------- /networking/basics/assets/markdown-img-paste-20180716232316916.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/basics/assets/markdown-img-paste-20180716232316916.png -------------------------------------------------------------------------------- /networking/basics/assets/markdown-img-paste-20180716233624749.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/basics/assets/markdown-img-paste-20180716233624749.png -------------------------------------------------------------------------------- /networking/basics/assets/markdown-img-paste-20180716233642942.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/basics/assets/markdown-img-paste-20180716233642942.png -------------------------------------------------------------------------------- /networking/basics/assets/markdown-img-paste-20180716235137178.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/basics/assets/markdown-img-paste-20180716235137178.png -------------------------------------------------------------------------------- /networking/basics/assets/markdown-img-paste-20180717000859836.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/basics/assets/markdown-img-paste-20180717000859836.png -------------------------------------------------------------------------------- /networking/basics/assets/markdown-img-paste-20180717001634130.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/basics/assets/markdown-img-paste-20180717001634130.png -------------------------------------------------------------------------------- /networking/basics/assets/markdown-img-paste-20180717001913286.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/basics/assets/markdown-img-paste-20180717001913286.png -------------------------------------------------------------------------------- /networking/basics/assets/markdown-img-paste-20180717002216360.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/basics/assets/markdown-img-paste-20180717002216360.png -------------------------------------------------------------------------------- /networking/basics/assets/markdown-img-paste-20180718134607514.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/basics/assets/markdown-img-paste-20180718134607514.png -------------------------------------------------------------------------------- /networking/basics/assets/markdown-img-paste-2018071813491995.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/basics/assets/markdown-img-paste-2018071813491995.png -------------------------------------------------------------------------------- /networking/basics/assets/markdown-img-paste-20180718135410900.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/basics/assets/markdown-img-paste-20180718135410900.png -------------------------------------------------------------------------------- /networking/basics/hub.md: -------------------------------------------------------------------------------- 1 | # Hub 2 | 3 | A hub is a network equipment with several interfaces called ports (not the one from OS). If we connect computers to these ports they will be able to talk with each other. 4 | 5 | ![](assets/markdown-img-paste-20180718134607514.png) 6 | 7 | People confuse hubs and switches. Know that hubs are antique legacy devices that no longer belong in any computer network. Switch on the other hand is a modern day device. 8 | 9 | ## Problems with Hub 10 | 11 | ### They are unintegillent 12 | 13 | All a hub ever does is to copy electrical signals that are entering one port to all other ports. 14 | 15 | This means that a hub is **unintegillent**. It doesn't care about network traffic or addresses at all. 16 | 17 | ### They do unnecessary work 18 | 19 | This results in lot of unnecessary work by the network. Apart from the actual receiver, all other computers must discard the packet. 20 | 21 | ![](assets/markdown-img-paste-2018071813491995.png) 22 | 23 | ### They are Half Duplex 24 | Another downside of hubs is that only one computer can talk at a time through a hub. When one computer is talking every other computer must stay completely silent. This is because the electrical signals of multiple computers that talk simultaneously will mix together in the hub, creating disturbances to the signals so that neither signal can be interpreted. 25 | 26 | If two computers talk at the same time a collision occurs. When this happens all computers will notice the disturbances and must stop talking for a while before trying again. This also makes a hub really slow. Only one device can communicate at a time, and when a collision accidentally occurs every device must be silent for a while. 27 | 28 | ![](assets/markdown-img-paste-20180718135410900.png) 29 | 30 | The more computers you connect to a hub the bigger the risk gets for collisions to happen since more involved devices will indirectly compete with each other for the available communication time slots. 31 | 32 | #### Half Duplex vs Full Duplex 33 | When only one device at a time can communicate on a network, it is called **Half Duplex**. Hub is always Half Duplex. 34 | 35 | If multiple device can talk at once then it is called **Full Duplex**. Switches are full duplex. 36 | -------------------------------------------------------------------------------- /networking/basics/ports_nat.md: -------------------------------------------------------------------------------- 1 | Go back to [Network Index](../README.md) 2 | 3 | # NAT and Port Forward 4 | 5 | # Public and Private IP Address 6 | 7 | In order to avoid ip address conflict, all clients should have unique ip address on the internet. But there are some execptions. 8 | 9 | ![](assets/markdown-img-paste-20180717000859836.png) 10 | 11 | The addresses above are reserved for using in LAN and they don't need to be unique on internet (but they must be unique within a single LAN). 12 | 13 | Tthe above special addresses are called **Private addresses**. They cannot be used on the Internet, they can only be used within local networks. If you try to use Private addresses on the Internet then your **Internet Service Provider will block your traffic** automatically, sensing that the traffic is coming from a Private IP address. This automatic block is being done to avoid any IP address conflicts on the Internet. These addresses are used in so many places that without the block we would have guaranteed and constant IP address conflicts all over the Internet. 14 | 15 | > If my pc is on a LAN and has a private IP address, then how am I able to surf the internet? 16 | 17 | # Address Translation (NAT) 18 | 19 | Since public IP addresses are limited and thus we are forced to use private IP addresses at home, how do we connect to internet? 20 | 21 | ![](assets/markdown-img-paste-20180717001634130.png) 22 | 23 | If we try to connect directly with the private IP, ISP will block us. 24 | 25 | The solution is to share a public IP. All the clients with private IP addresses on a LAN will share a single public address. This is done by using Address Translation or Network Address Translation. 26 | 27 | ## How NAT Works? 28 | 29 | When a computer wants to communicate it sends off a packet with data. The packet always has two IP addresses inscribed in the envelope or header of the packet. 30 | 31 | Source Address, which is the IP address of the sender. This has to be entered into the packet so the receiver knows where it should send its replies, like a “return address” 32 | Destination address, the IP address of the receiver that the packet is being sent to 33 | In a home network where a computer wants to talk to something on the Internet, the source address will be a Private IP address on the LAN. The destination address of the packet will be a Public IP address of a server on the Internet. If that packet is sent to the Internet then the ISP will block and throw away the packet since it has a private IP address as its source. 34 | 35 | To fix this problem the home router steps in and translates the source address from a private address to a public IP address. The router itself has a public IP address on its outside WAN interface. It got that public IP address from the ISP. The router will simply let every client on the inside LAN share that single public IP address. 36 | 37 | ![](assets/markdown-img-paste-20180717001913286.png) 38 | 39 | When the router sends the packet on to the Internet the packet will appear to come from the home router’s public IP address. From the perspective of the ISPs and the web server, the packet is coming from the public IP address of the home router. When the web server replies back to the computer it will send its reply back to the public IP address of the home router, and the ISP finds its way back there without any trouble. 40 | 41 | What about when there are multiple clients? Well, the router keeps track about which packet belongs to which client. You don't have to worry. 42 | 43 | ![](assets/markdown-img-paste-20180717002216360.png) 44 | 45 | # Port Forward 46 | 47 | > What if there is traffic from outside that wants to connect to a web server on our LAN. How would the router know, which client on the LAN is the web server? 48 | 49 | There is no way for the router to know. We have to configure the router such that when traffic from outside hits a particular port of the router, the router will send it to a host that we configured. This is called port forwarding. 50 | -------------------------------------------------------------------------------- /networking/basics/router.md: -------------------------------------------------------------------------------- 1 | Go back to [Network Index](../README.md) 2 | 3 | # Router 4 | 5 | The main role of a Router is to route data traffic. The router knows in which directions different destinations are, and when it receives data traffic it will forward the traffic in the direction of the destination. A router always picks the best route it has knowledge about for the data traffic that it is forwarding. 6 | 7 | In a home network, the router plays a vital role. It connects its two end points: outside and inside. 8 | 9 | ![Router Home](assets/markdown-img-paste-20180716225230715.png) 10 | 11 | On outside lies WAN (wide arean network) and on inside lies LAN (local area network). 12 | 13 | Router allows computers to communicate with each other, regardless of whether they are inside or outside. 14 | 15 | # Router is not just Router 16 | 17 | Our home router comes with more than "routing" features. 18 | 19 | ![](assets/markdown-img-paste-20180716232316916.png) 20 | 21 | A home router has the following parts: 22 | 23 | 1. Switch 24 | 2. Access point 25 | 3. Modem 26 | 4. Router 27 | 5. DHCP 28 | 6. Address translation 29 | 7. Port Forward 30 | 8. Firewall 31 | 9. DNS 32 | 10. Web interface 33 | 34 | From the picture we can see, internal computers can communicate through switch alone. No need to go through router. Same for wifi communication: access point to switch. 35 | 36 | # IP Address 37 | 38 | IP Address consists of two parts: **IP Network** and Host address. 39 | 40 | ## IP Network 41 | On a LAN, computers can talk with each other as long as they are under the same ip network. If the computers belong to different IP networks then they have to communicate with each other via a router. 42 | 43 | The main purpose of a router is to be able to forward traffic to different destinations. Within computer networking, those destinations are different IP networks. 44 | 45 | > So how do you know if two IP addresses are on the same IP network or on two different IP networks? 46 | 47 | ## Subnet Mask 48 | 49 | An IP address is always combined with a Subnet Mask, and it is the Subnet Mask that determines which part of the IP address that belongs to the IP network and which part that belongs to host addresses. 50 | 51 | ![](assets/markdown-img-paste-20180716233624749.png) 52 | 53 | ![](assets/markdown-img-paste-20180716233642942.png) 54 | 55 | Basically, you AND ip address with subnet mask. The result is the ip network. The remaining bits are for host address. 56 | 57 | ## Network Address 58 | 59 | Each ip network also has a name. The name of an ip network is the smallest possible ip address for that network. For example, in the figure above, 50.211.197.5 belongs to the ip network that has the following network address 50.211.197.0 60 | 61 | IP address is for a single device in a network. Network address is for the whole network. 62 | 63 | ## Default Gateway 64 | 65 | > So computers that are in same ip network can communicate directly with each other. What about computers that are in different ip network? How do they communicate? 66 | 67 | They must go through router. 68 | 69 | > But how do they find out where is the router? 70 | 71 | Using Default Gateway. 72 | 73 | In the local network, there must be a router for inter-network communication and the address of that router is configured to all clients of the LAN network. The address of the router is called Default Gateway. We could have just called it "Router Address". 74 | 75 | ![](assets/markdown-img-paste-20180716235137178.png) 76 | 77 | If you look at the picture above, you will notice that router has two IP. Using one IP it communicates with inside network and using another it communicates with outside. Because after all, two computers can communicate directly with each other if they are in same ip network. 78 | -------------------------------------------------------------------------------- /networking/basics/switch.md: -------------------------------------------------------------------------------- 1 | # Switch 2 | 3 | When two computers on same ip network communicate with each other directly, the communication is dealt by the switch. 4 | 5 | The task performed by switch is called **Switching**. Routers forward traffic based on IP address, switches forward traffic based on **MAC address**. 6 | 7 | # Hubs are not Switch 8 | 9 | Read more on [Hub](./hub.md). 10 | 11 | # MAC address 12 | 13 | All equipments that can be connected to computer networks (computers, routers, servers, printers and etc) have a MAC address. It is an address which is written into the network interface of the device during manufacturing. 14 | 15 | A MAC address consists of 12 hexadecimal characters and could look like this: 16 | 17 | - 01:23:45:67:89:ab 18 | - 00:fe:19:2a:73:dc 19 | - 02:0a:95:9d:68:16 20 | 21 | Each time a computer sends out network traffic the traffic has both a source and destination IP address, but it also has a source and destination MAC address. 22 | 23 | IP addresses are relevant on a global scale. They hold the final destination of the packet and can tell us which address the packet is originally coming from. In contrast, MAC addresses are used on a more local scale, and hold information about the **next hop destination** in the local LAN network. 24 | 25 | # Broadcasts 26 | 27 | > How does a new computer that just booted up knows where is the DHCP server is located? 28 | 29 | The computer sends out a broadcast which will reach every other device on the LAN to ask any available DHCP servers to reply back with an IP address. 30 | 31 | When a computer sends out a broadcast it will use a special destination MAC address, **FF:FF:FF:FF:FF:FF**. That address is called the **Broadcast Address** and is used specifically for this purpose. All other equipment on the LAN will then understand that the traffic is a broadcast that is directed at everybody else within the LAN. 32 | 33 | # Switch 34 | 35 | A switch is an intelligent device. 36 | 37 | What a switch does is that it constantly monitors the traffic which is entering the switch from connected devices. It then learns about where the different MAC addresses of those devices are connected. It does this by looking at the traffic that arrives from computers to read the source MAC address of the traffic. 38 | 39 | If it doesn't know where a particular device is, it acts like sends them to all unknown devices and waits for reply for them. 40 | 41 | Switch is like a HUB + Cache. 42 | 43 | # ARP (Adress Resolution Protocol) 44 | 45 | > How does the computer know MAC address of router? 46 | 47 | It sends a broadcast request known as ARP asking "Who has the default gateway IP? Send me your MAC address." Then once computer gets the address, it stores it in a cache for few minutes (refreshing after every successful packet transfer). The cache allows the computer to send packets without broadcasting for MAC address every time. 48 | 49 | -------------------------------------------------------------------------------- /networking/dos.md: -------------------------------------------------------------------------------- 1 | # Denial of Service 2 | 3 | ## DNS Amplification Attack 4 | 5 | [DOS via Open DNS Server](https://www.youtube.com/watch?v=Ajw79kWMYkY) 6 | 7 | A small request is sent to Open DNS server with ip spoofing of victim. The request asks for large data from DNS server over UDP protocol. The data gets sent to victim which results in DOS. This is an example of amplification where a small request from attacker gets amplified by DNS server. 8 | 9 | ## Syn Flood Attack 10 | 11 | In TCP’s three-way handshake, a server allocates and initializes connection variables and buffers in response to a received SYN. The server then sends a SYNACK in response, and awaits an ACK segment from the client. If the client does not send an ACK to complete the third step of this 3-way handshake, eventually (often after a minute or more) the server will terminate the half-open connection and reclaim the allocated resources. 12 | 13 | This TCP connection management protocol sets the stage for a classic Denial of Service (DoS) attack known as the **SYN flood attack**. In this attack, the attacker(s) send a large number of TCP SYN segments, without completing the third handshake step. With this deluge of SYN segments, the server’s connection resources become exhausted as they are allocated (but never used!) for half-open connections; legitimate clients are then denied service. Such SYN flooding attacks were among the first documented DoS attacks. Fortunately, an effective defense known as SYN cookies are now deployed in most major operating systems. 14 | 15 | SYN cookies work as follows: 16 | 17 | - When the server receives a SYN segment, it does not know if the segment is coming from a legitimate user or is part of a SYN flood attack. So, instead of creating a half-open TCP connection for this SYN, the server creates an initial TCP sequence number that is a complicated function (hash function) of source and destination IP addresses and port numbers of the SYN segment, as well as a secret number only known to the server. 18 | 19 | This carefully crafted initial sequence number is the so-called “cookie.” The server then sends the client a SYNACK packet with this special initial sequence number. Importantly, the server does not remember the cookie or any other state information corresponding to the SYN. 20 | 21 | - A legitimate client will return an ACK segment. When the server receives this ACK, it must verify that the ACK corresponds to some SYN sent earlier. But how is this done if the server maintains no memory about SYN segments? As you may have guessed, it is done with the cookie. 22 | 23 | Recall that for a legitimate ACK, the value in the acknowledgment field is equal to the initial sequence number in the SYNACK (the cookie value in this case) plus one. The server can then run the same hash function using the source and destination IP address and port numbers in the SYNACK (which are the same as in the original SYN) and the secret number. If the result of the function plus one is the same as the acknowledgment (cookie) value in the client’s SYNACK, the server concludes that the ACK corresponds to an earlier SYN segment and is hence valid. The server then creates a fully open connection along with a socket. 24 | 25 | - On the other hand, if the client does not return an ACK segment, then the original SYN has done no harm at the server, since the server hasn’t yet allocated any resources in response to the original bogus SYN. 26 | 27 | The SYN cookie is composed of timestamp (5 bits), MSS (3 bit) and the hash of secret key + IP information (24) bit. Here timestamp is not an actual timestamp, instead a slow incrementing value (e.g, time() >> 6). Timestamp is included to make sure session has not expired. 28 | 29 | ## IP Fragmented Attack (Teardrop) 30 | 31 | Fragments arrive to victim with incorrect offset, due to which victim is unable to reconstruct them. It eats up the buffer and prevents it from receiving anymore packets. 32 | -------------------------------------------------------------------------------- /networking/http.md: -------------------------------------------------------------------------------- 1 | # HTTP 2 | 3 | # CORS 4 | 5 | Cross Origin Resource Sharing. 6 | 7 | This is used to prevent a site (xyz.com) from pulling resources from another site (abc.com), unless abc.com explicitly allows it. 8 | 9 | Some requests are by default allowed, such as retrieving 'text/plain' documents using GET request. 10 | 11 | For requests which are deemed risky (trying to download and run a js script from another origin abc.com), the browser first sends a preflight request asking the server abc.com about what is allowed and what is not. This preflight request is a OPTION request. The server in return responds with what is allowed and what is not using http headers (Access-Control-Allow-Origin and Access-Control-Allow-Methods). 12 | 13 | Once the preflight response comes back, it is used to determine whether the browser should go forward with the risky request. 14 | 15 | **Resource**: [Cross-Origin Requests in Express.JS](http://justindavis.co/2015/08/31/CORS-in-Express/) 16 | -------------------------------------------------------------------------------- /networking/ip-address.md: -------------------------------------------------------------------------------- 1 | # Private IP Address 2 | 3 | Any address that starts with **192.168.X.X** or **10.X.X.X** is a private IP. 4 | -------------------------------------------------------------------------------- /networking/misc/unix_domain_socket.md: -------------------------------------------------------------------------------- 1 | # Unix Domain Socket 2 | 3 | A Unix domain socket or IPC socket (inter-process communication socket) is a data communications endpoint for exchanging data between processes executing on the **same host operating system**. 4 | 5 | Like named pipes, Unix domain sockets support: 6 | 7 | - Transmission of a reliable stream of bytes (SOCK_STREAM, compare to TCP). 8 | - Ordered and reliable transmission of datagrams (SOCK_SEQPACKET, compare to SCTP). 9 | - Unordered and unreliable transmission of datagrams (SOCK_DGRAM, compare to UDP). 10 | 11 | The Unix domain socket facility is a standard component of POSIX operating systems. 12 | 13 | ## Why use UDS instead of TCP 14 | 15 | IP sockets (especially TCP/IP sockets) are a mechanism allowing communication between processes over the network. In some cases, you can use TCP/IP sockets to talk with processes running on the same computer (by using the loopback interface). 16 | 17 | The API for Unix domain sockets is similar to that of an Internet socket, but rather than using an underlying network protocol, all communication occurs entirely within the operating system kernel. 18 | 19 | **UNIX domain sockets know that they’re executing on the same system**, so they can avoid some checks and operations (like routing); which makes them faster and lighter than IP sockets. So if you plan to communicate with processes on the same host, this is a better option than IP sockets. 20 | 21 | **UNIX domain sockets are subject to file system permissions**, while TCP sockets can be controlled only on the packet filter level. As a result, it is much easier to regulate which users have access to a UNIX domain socket than it is for a TCP socket 22 | -------------------------------------------------------------------------------- /networking/network-address-translation.md: -------------------------------------------------------------------------------- 1 | # Network Address Translation (NAT) 2 | 3 | [How NAT Works - Youtube](https://www.youtube.com/watch?v=QBqPzHEDzvo) 4 | 5 | Basically, NAT is a mapping carried out by our routers so that our computers on private network (with IP 192.168.0.100) is accessible from Internet. 6 | 7 | Whenever a packet is sent over network, the header must have a source IP. If the source IP is a private IP, how is the server going to send its reply back to us? 8 | 9 | What happens in reality is that our router replaces the source IP address (e.g, 192.168.0.100:12345) with its own public IP address and a random port (e.g, 12.13.14.15:54331) and stores the mapping in a table called **NAT Table**. Now the server can reply back to the router and the router forwards it according to the NAT Table. 10 | -------------------------------------------------------------------------------- /networking/network-layer/assets/2018-08-25-00-56-35.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/network-layer/assets/2018-08-25-00-56-35.png -------------------------------------------------------------------------------- /networking/network-layer/assets/2018-08-25-11-11-39.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/network-layer/assets/2018-08-25-11-11-39.png -------------------------------------------------------------------------------- /networking/network-layer/assets/2018-08-25-11-11-51.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/network-layer/assets/2018-08-25-11-11-51.png -------------------------------------------------------------------------------- /networking/network-layer/assets/2018-08-25-11-21-22.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/network-layer/assets/2018-08-25-11-21-22.png -------------------------------------------------------------------------------- /networking/network-layer/assets/2018-08-25-11-42-56.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/network-layer/assets/2018-08-25-11-42-56.png -------------------------------------------------------------------------------- /networking/network-layer/assets/2018-08-26-11-16-24.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/network-layer/assets/2018-08-26-11-16-24.png -------------------------------------------------------------------------------- /networking/network-layer/assets/2018-08-26-11-26-38.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/network-layer/assets/2018-08-26-11-26-38.png -------------------------------------------------------------------------------- /networking/network-layer/assets/2018-08-26-11-29-01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/network-layer/assets/2018-08-26-11-29-01.png -------------------------------------------------------------------------------- /networking/network-layer/assets/2018-08-26-22-04-25.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/network-layer/assets/2018-08-26-22-04-25.png -------------------------------------------------------------------------------- /networking/network-layer/assets/2018-08-26-22-14-46.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/network-layer/assets/2018-08-26-22-14-46.png -------------------------------------------------------------------------------- /networking/networking.md: -------------------------------------------------------------------------------- 1 | # TOC Networking 2 | 3 | ## Basics 4 | 5 | 1. [Router](./basics/router.md) 6 | - Router 7 | - IP Address 8 | - Subnet Mask 9 | - Default Gateway 10 | 2. [NAT & Port Forwarding](./basics/ports_nat.md) 11 | - Private & Public IP 12 | - NAT 13 | - Port Forward 14 | 3. [Switching](./basics/switch.md) 15 | - [Hub](./basics/hub.md) 16 | - Half Duplex vs Full Duplex 17 | 18 | ## Advanced 19 | 20 | 1. [Transport Layer](./transport_layer/transportlayer.md) 21 | - [UDP](./transport_layer/udp.md) 22 | - [Reliable Data Transfer](./transport_layer/reliable-data-transfer.md) 23 | - [TCP](./transport_layer/tcp.md) 24 | 25 | ## Misc 26 | 27 | 1. [Unix Domain Socket](./misc/unix_domain_socket.md) -------------------------------------------------------------------------------- /networking/proxy.md: -------------------------------------------------------------------------------- 1 | # Proxy (Forward Proxy) 2 | 3 | Proxy is a client side concept. Multiple client asks a proxy to send request on their behalf. For example, our router is a proxy, or when we can't access facebook from university, we often go to some proxy-service site and use their site to access fb. 4 | 5 | The flow goes as following: Client -> Proxy -> Internet -> Server 6 | 7 | Forward proxy is Internet facing proxy used to retrieve service from anywhere. 8 | 9 | # Reverse Proxy 10 | 11 | Reverse proxy is a server side concept. Here request from Internet is received by a reverse-proxy entity whose duty is to communicate with servers behind it to return proper response to client. For example, Nginx is a reverse proxy. 12 | 13 | The flow goes as following: Client -> Internet -> Proxy -> Server 14 | 15 | Reverse proxy is Internal facing proxy used as a front-end to control and protect access to servers on private network. 16 | 17 | ![proxy](../images/proxy.png) 18 | 19 | # Resources 20 | 21 | 1. [Wiki Proxy](https://en.wikipedia.org/wiki/Proxy_server) 22 | -------------------------------------------------------------------------------- /networking/tcp-splitting.md: -------------------------------------------------------------------------------- 1 | # TCP SPLITTING: OPTIMIZING THE PERFORMANCE OF CLOUD SERVICES 2 | For cloud services such as search, e-mail, and socialnetworks, it is desirable to provide a high-level ofresponsiveness, ideally giving users the illusion that theservices are running within their own end systems (includingtheir smartphones). This can be a major challenge, as usersare often located far away from the data centers that areresponsible for serving the dynamic content associated withthe cloud services. 3 | 4 | Indeed, if the end system is far from a data center, then theRTT will be large, potentially leading to poor response timeperformance due to TCP slow start. As a case study, considerthe delay in receiving a response for a search query.Typically, the server requires three TCP windows during slowstart to deliver the response. Thus the time from when an endsystem initiates a TCP connection until the time when itreceives the last packet of the response is roughly $4 \cdot RTT$ (one RTT to set up the TCP connection plus three RTTs forthe three windows of data) plus the processing time in thedata center. 5 | 6 | These RTT delays can lead to a noticeable delay in returningsearch results for a significant fraction of queries.Moreover, there can be significant packet loss in accessnetworks, leading to TCP retransmissions and even largerdelays. 7 | 8 | One way to mitigate this problem and improve user-perceivedperformance is to (1) deploy front-end servers closer to theusers, and (2) utilize TCP splitting by breaking the TCPconnection at the front-end server. 9 | 10 | With TCP splitting, the client establishes a TCP connection tothe nearby front-end, and the front-end maintains a persistentTCP connection to the data center with a very large TCPcongestion window. With this approach, the response timeroughly becomes $4 \cdot RTT_{FE} + RTT_{BE} + processing\ time$, where $RTT_{FE}$ is the round-trip time between clientand front-end server, and $RTT_{BE}$ is the round-trip timebetween the front-end server and the data center (back-endserver). 11 | 12 | If the front-end server is close to client, then this responsetime approximately becomes RTT plus processing time, since$RTT_{FE}$ is negligibly small and $RTT_{BE}$ is approximatelyRTT. 13 | 14 | In summary, TCP splitting can reduce the networking delay roughly from $4 \cdot RTT$ to $RTT$, significantly improving user-perceived performance, particularly for users who are far from the nearest data center. TCP splitting also helps reduce TCP retransmission delays caused by losses in access networks. 15 | 16 | Today, Google and Akamai make extensive use of their CDN servers in access networks to perform TCP splitting for the cloud services they support. 17 | -------------------------------------------------------------------------------- /networking/transport_layer/assets/2018-08-12-12-00-56.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/transport_layer/assets/2018-08-12-12-00-56.png -------------------------------------------------------------------------------- /networking/transport_layer/assets/2018-08-12-12-52-24.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/transport_layer/assets/2018-08-12-12-52-24.png -------------------------------------------------------------------------------- /networking/transport_layer/assets/2018-08-14-00-22-04.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/transport_layer/assets/2018-08-14-00-22-04.png -------------------------------------------------------------------------------- /networking/transport_layer/assets/2018-08-14-20-50-36.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/transport_layer/assets/2018-08-14-20-50-36.png -------------------------------------------------------------------------------- /networking/transport_layer/assets/2018-08-14-20-58-40.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/transport_layer/assets/2018-08-14-20-58-40.png -------------------------------------------------------------------------------- /networking/transport_layer/assets/2018-08-14-20-59-10.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/transport_layer/assets/2018-08-14-20-59-10.png -------------------------------------------------------------------------------- /networking/transport_layer/assets/2018-08-14-21-09-20.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/transport_layer/assets/2018-08-14-21-09-20.png -------------------------------------------------------------------------------- /networking/transport_layer/assets/2018-08-14-21-10-30.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/transport_layer/assets/2018-08-14-21-10-30.png -------------------------------------------------------------------------------- /networking/transport_layer/assets/2018-08-14-21-18-38.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/transport_layer/assets/2018-08-14-21-18-38.png -------------------------------------------------------------------------------- /networking/transport_layer/assets/2018-08-14-21-18-53.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/transport_layer/assets/2018-08-14-21-18-53.png -------------------------------------------------------------------------------- /networking/transport_layer/assets/2018-08-16-09-27-37.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/transport_layer/assets/2018-08-16-09-27-37.png -------------------------------------------------------------------------------- /networking/transport_layer/assets/2018-08-16-09-43-45.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/transport_layer/assets/2018-08-16-09-43-45.png -------------------------------------------------------------------------------- /networking/transport_layer/assets/2018-08-19-11-52-32.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/transport_layer/assets/2018-08-19-11-52-32.png -------------------------------------------------------------------------------- /networking/transport_layer/assets/2018-08-19-11-53-14.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/transport_layer/assets/2018-08-19-11-53-14.png -------------------------------------------------------------------------------- /networking/transport_layer/assets/2018-08-19-12-01-24.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/transport_layer/assets/2018-08-19-12-01-24.png -------------------------------------------------------------------------------- /networking/transport_layer/assets/2018-08-19-12-03-07.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/transport_layer/assets/2018-08-19-12-03-07.png -------------------------------------------------------------------------------- /networking/transport_layer/assets/2018-08-19-12-45-03.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/transport_layer/assets/2018-08-19-12-45-03.png -------------------------------------------------------------------------------- /networking/transport_layer/assets/2018-08-19-12-51-49.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/transport_layer/assets/2018-08-19-12-51-49.png -------------------------------------------------------------------------------- /networking/transport_layer/assets/2018-08-19-12-52-01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/transport_layer/assets/2018-08-19-12-52-01.png -------------------------------------------------------------------------------- /networking/transport_layer/assets/2018-08-19-18-54-33.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/transport_layer/assets/2018-08-19-18-54-33.png -------------------------------------------------------------------------------- /networking/transport_layer/assets/2018-08-19-18-59-59.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/transport_layer/assets/2018-08-19-18-59-59.png -------------------------------------------------------------------------------- /networking/transport_layer/assets/2018-08-19-20-55-48.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/transport_layer/assets/2018-08-19-20-55-48.png -------------------------------------------------------------------------------- /networking/transport_layer/assets/2018-08-20-20-19-39.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/transport_layer/assets/2018-08-20-20-19-39.png -------------------------------------------------------------------------------- /networking/transport_layer/assets/2018-08-20-20-35-29.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/transport_layer/assets/2018-08-20-20-35-29.png -------------------------------------------------------------------------------- /networking/transport_layer/assets/2018-08-20-20-38-34.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/transport_layer/assets/2018-08-20-20-38-34.png -------------------------------------------------------------------------------- /networking/transport_layer/assets/2018-08-20-20-39-22.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/transport_layer/assets/2018-08-20-20-39-22.png -------------------------------------------------------------------------------- /networking/transport_layer/assets/2018-08-20-20-39-32.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/transport_layer/assets/2018-08-20-20-39-32.png -------------------------------------------------------------------------------- /networking/transport_layer/assets/2018-08-24-20-47-03.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/networking/transport_layer/assets/2018-08-24-20-47-03.png -------------------------------------------------------------------------------- /networking/transport_layer/udp.md: -------------------------------------------------------------------------------- 1 | # UDP 2 | 3 | > Why would anyone choose udp? 4 | 5 | 1. **Finer application-level control over what data is sent, and when**: There is no congestion control like TCP. Each segment gets passed to network layer immediately. 6 | 2. **No connection establishment** 7 | 3. **No connection state** 8 | 4. **Small packet header overhead**: The TCP segment has 20 bytes of header over- head in every segment, whereas UDP has only 8 bytes of overhead. 9 | 10 | ## UDP Segment Structure 11 | 12 | ![](assets/2018-08-14-00-22-04.png) 13 | 14 | The UDP header has only four fields, each consisting of two bytes. 15 | 16 | - the port numbers allow the destination host to pass the application data to the correct process running on the destination end system (that is, to perform the demultiplexing function). 17 | - The length field specifies the number of bytes in the UDP segment (header plus data). An explicit length value is needed since the size of the data field may differ from one UDP segment to the next. 18 | - The checksum is used by the receiving host to check whether errors have been introduced into the segment. 19 | 20 | ## UDP Checksum 21 | 22 | The UDP checksum provides for error detection. That is, the checksum is used to determine whether bits within the UDP segment have been altered. 23 | 24 | UDP at the sender side performs the 1s complement of the sum of all the 16-bit words in the segment, with any overflow encountered during the sum being wrapped around. This result is put in the checksum field of the UDP segment. 25 | 26 | At the receiver, all 16-bit words are added, including the checksum. If no errors are introduced into the packet, then clearly the sum at the receiver will be 1111111111111111. If one of the bits is a 0, then we know that errors have been introduced into the packet. 27 | 28 | Although UDP provides error checking, it does not do anything to recover from an error. Some implementations of UDP simply discard the damaged segment; others pass the dam- aged segment to the application with a warning. -------------------------------------------------------------------------------- /nsu/cse331/images/mcu-vs-mpu.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/nsu/cse331/images/mcu-vs-mpu.png -------------------------------------------------------------------------------- /nsu/cse331/lec1.1-microcontroller.md: -------------------------------------------------------------------------------- 1 | # Microcontroller vs Microprocessor 2 | 3 | ## predefined tasks 4 | 5 | Microcontroller is used where tasks are predefined, like in oven, washing machine. 6 | 7 | Microprocessor is used where tasks vary and we have no way to it before hand. For example in pc we don't know what game we will play and each game has different requirements and calculations. 8 | 9 | ## integrated chip 10 | 11 | Since tasks of microcontroller is fixed, we know what amount of memory and other facilities it will need. All the facilities then can be fabricated into a single chip. So overall size of microcontroller becomes small. 12 | 13 | But in microprocessor, computational needs and other physical needs such as memory varies with the application. Hence in order to build a system, we have to attach those facilities externally. So overall size becomes larger. 14 | 15 | ## Processing power and memory 16 | 17 | Microprocessor have higher computational power with cpu cycle ranging in the gigahertz. Whereas microcontroller have cpu cycles in the range from 1 megahertz to 300 megahertz. 18 | 19 | Since microcontrollers' tasks are fixed, they have lower memory needs than microprocessor. 20 | 21 | Amount of data microprocessor can handle in a single cycle is higher than microcontrollers. 22 | 23 | # Summary 24 | 25 | ![](images/mcu-vs-mpu.png) 26 | -------------------------------------------------------------------------------- /nsu/eee111/images/bias.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/nsu/eee111/images/bias.png -------------------------------------------------------------------------------- /nsu/eee111/images/diode.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/nsu/eee111/images/diode.png -------------------------------------------------------------------------------- /nsu/eee111/images/np-junction.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/nsu/eee111/images/np-junction.png -------------------------------------------------------------------------------- /nsu/eee111/lecture-1-diodes.md: -------------------------------------------------------------------------------- 1 | # Semiconductors 2 | 3 | **Resources**: [How semiconductors work](https://www.youtube.com/watch?v=33vbFFFn04k&t=24s) 4 | 5 | 6 | Crystal of silicon with covalent bonds. Silicon has 4 valence electrons. 7 | 8 | **Why silicon?** Cause it has high **thermal stability**. Thermal stability is the ability of a compound to resist decomposing when heated. 9 | 10 | ## Generation and Recombination 11 | 12 | When an electron leaves an atom, a hole is created. This is called Generation. When an electron fills a hole, it is called recombination 13 | 14 | ## Doping 15 | 16 | Doping is the process where foreign substances are added to improve performance. 17 | 18 | ### N-type Semiconductor 19 | One of the common impurities added is **Phosphorus**. Phosphorus has 5 valence electrons. The phosphorus replaces silicon atom and takes it places sharing electron with 4 adjacent silicon atom and 1 free electron. 20 | 21 | Charge is carried by extra electron. 22 | 23 | It is called N type cause negative charge moves through it. 24 | 25 | ### P-type Semiconductor 26 | 27 | Another impurities is Boron that has 3 valence electron. So replacing Si with B introduces holes in the structure. 28 | 29 | Charge is carried by holes. 30 | 31 | It is called P type cause positive charge (holes) moves through it. 32 | 33 | # Diode 34 | 35 | Diode is a one way valve for electricity. 36 | 37 | ## PN Junction 38 | 39 | A PN semiconductor is formed by attaching a N-type semiconductor with P-type conductor. 40 | 41 | At the boundary of P and N type conductor, known as **PN-Junction**, a positive charge gets build up on P-type side and negative charge builds up on N-type side. This is cause some of the loose electron of Phosphorus gets attracted to holes in Boron. 42 | 43 | A region forms around the PN-Junction where there is no charge carrier. This region is called **Depletion Region**. This region works like an insulator, preventing current from flowing. 44 | 45 | It takes around 0.7 volts to cross the depletion region. 46 | 47 | ![](images/np-junction.png) 48 | 49 | Now, if we connect power source, depending on which end is connect to which, current will flow. That is, current flows in one direction but not in other. The other direction actually increases the insulation band width. 50 | 51 | ![](images/diode.png) 52 | 53 | When electron flows through diode, its called forward bias. 54 | 55 | ![](images/bias.png) 56 | 57 | # Transistor 58 | 59 | [How transistor work](https://www.youtube.com/watch?v=IcrBqCFLHIY) 60 | -------------------------------------------------------------------------------- /nsu/mat350.md: -------------------------------------------------------------------------------- 1 | # 3.1 Linear Models 2 | -------------------------------------------------------------------------------- /nsu/phi104/presentation.md: -------------------------------------------------------------------------------- 1 | # Risk and Liability in Engineering 2 | 3 | # 7.2 THE ENGINEER’S APPROACH TO RISK 4 | 5 | ## Risk as the Product of the Probability and Magnitude of Harm 6 | 7 | The usual engineering definition of risk is 8 | > "a compound measure of the probability and magnitude of adverse effect." 9 | 10 | A relatively slight harm that is highly likely might constitute a greater risk to more people than a relatively large harm that is far less likely. 11 | 12 | We can define a harm as an invasion or limitation of a person’s freedom or wellbeing. 13 | 14 | [Utilitarianism](https://www.youtube.com/watch?v=-a739VjqdSI) 15 | 16 | ## Utilitarianism and Acceptable Risk 17 | 18 | > An acceptable risk is one in which the product of the probability and magnitude of the harm is equaled or exceeded by the product of the probability and magnitude of the benefit, and there is no other option where the product of the probability and magnitude of the benefit is substantially greater. 19 | 20 | Basically it's acceptable if gain is more than risk and risk cannot be reduced anymore. 21 | 22 | ## Expanding the Engineering Account of Risk: The Capabilities Approach to Identifying Harm and Benefit 23 | 24 | Capability focuses on the effect of disasters on overall human well-being. Well-being is defined in terms of individual capabilities, or "the ability of people to lead the kind of life they have reason to value." 25 | 26 | **Problem with Utilitarianism** 27 | 28 | a person with limited resources might learn to take pleasure in small things, which are only minimally satisfying to a person with more ample means. The individual in a poverty-stricken situation might have all of his or her severely limited desires satisfied. From the utilitarian standpoint, the person would be described as happy and be said to enjoy a high standard of living. Yet this individual might still be objectively deprived. The problem here is that utilitarianism does not take into account the number and quality of options that are available to individuals, which is precisely what capabilities capture. 29 | 30 | From the capabilities standpoint, a risk is the probability that individuals’ capabilities might be reduced due to some hazard. 31 | 32 | # 7.3 THE PUBLIC’S APPROACH TO RISK 33 | 34 | ## Expert and Layperson: Differences in Factual Beliefs 35 | 36 | ## "Risky" Situations and Acceptable Risk 37 | 38 | ## Free and Informed Consent 39 | 40 | To give free and informed consent to the risks imposed by technology, three things are necessary. First, a person must not be coerced. Second, a person must have the relevant information. Third, a person must be rational and competent enough to evaluate the information. 41 | 42 | First, it is difficult to know when consent is free. Have workers given their free consent when they continue to work at a plant with known safety hazards? Perhaps they have no alternative form of employment. 43 | 44 | Second, people are often not adequately informed of dangers or do not evaluate them correctly. 45 | 46 | Third, it is often not possible to obtain meaningful informed consent from individuals who are subject to risks from technology. 47 | 48 | In light of the problems in getting free and informed consent, we could compensate individuals after the fact for actual harms done to them through technology. 49 | 50 | This approach has the advantage that consent does not have to be obtained, but it also has several disadvantages. First, it does not tell us how to determine adequate compensation. Second, it limits the freedom of individuals because some people would never have consented. Third, sometimes there is no adequate compensation for a harm, as in the case of serious injury or death. 51 | 52 | ## Equity or Justice 53 | 54 | > An acceptable risk is one in which (1) risk is assumed by free and informed consent, or properly compensated, and in which (2) risk is justly distributed, or properly compensated. 55 | 56 | # 7.4 THE GOVERNMENT REGULATOR’S APPROACH TO RISK 57 | 58 | > An acceptable risk is one in which protecting the public from harm has been weighted more heavily than benefiting the public. 59 | 60 | # 7.5 COMMUNICATING RISK AND PUBLIC POLICY 61 | 62 | ## Communicating Risk to the Public 63 | -------------------------------------------------------------------------------- /nsu/phi104/slide1.md: -------------------------------------------------------------------------------- 1 | # Ethics 2 | 3 | # Definition 4 | 5 | > “Ethics is a branch of philosophy; it is moral philosophy or philosophical thinking about morality, moral problems and moral judgments.” 6 | 7 | > **Philosophy** 8 | 1. Philosophy is a set of views or beliefs about life and the universe. 9 | 2. Philosophy is a process of reflecting on and criticizing our most deeply held conceptions and beliefs. 10 | 3. Philosophy is a rational attempt to look at the universe as a whole. 11 | 4. Philosophy is the logical analysis of language and the clarification of the meaning of words and concepts. 12 | 13 | # Ethicist 14 | 15 | > An ethicist has enough knowledge about ethical principles, ethical theories, ethical terms and their applications. 16 | 17 | # Ethical Expert 18 | 19 | > An ethical expert suggests how one can solve ethical dilemma or act ethically in a particular situation. He or she is like a consultant, lawyer or a software expert. 20 | -------------------------------------------------------------------------------- /nsu/phi104/slide10.md: -------------------------------------------------------------------------------- 1 | # Slide 10: Deontological Ethics 2 | 3 | The word deontology derives from the Greek words for duty (deon) and science (or study) of (logos). 4 | 5 | In contemporary moral philosophy, deontology is one of those kinds of normative theories regarding which choices are morally required, forbidden, or permitted. 6 | 7 | Deontology is against or in opposition to Consequentialism or utilitarianism. 8 | 9 | Deontology do not use effects of action to judge action. 10 | 11 | The most familiar forms of deontology holds that some choices cannot be justified by their effects—that no matter how morally good their consequences, some choices are morally forbidden. 12 | 13 | For such deontologists, what makes a choice right is its conformity with a moral norm. 14 | 15 | Such norms are to be simply obeyed by each moral agent; such norm-keepings are not to be maximized by each agent. 16 | 17 | In this sense, for such deontologists, the Right is said to have priority over the Good. 18 | 19 | # Types of Deontological Theory 20 | 21 | 1. Agent-Centered Deontological Theories 22 | 1. Patient-Centered Deontological Theories 23 | -------------------------------------------------------------------------------- /nsu/phi104/slide11.md: -------------------------------------------------------------------------------- 1 | # Slide 11: Kantian Ethics 2 | 3 | "the search for and establishment of the supreme principle of morality" 4 | 5 | That supreme principle, which Kant calls the categorical imperative, commands simply that our actions should have the form of moral conduct; that is, that they should be derivable from universal principles. 6 | 7 | # Moral Principles 8 | 9 | In other words, Kant is going to start from our ordinary ways of thinking about morality and analyze them to discover the principle behind them. 10 | 11 | # Good Will 12 | 13 | A person who does the right thing for the right reason evinces what Kant calls a good will. A good will is the only thing to which we attribute "unconditional worth.“ 14 | 15 | A number of the things which Kant says have only "conditional" value, such as health and happiness. 16 | 17 | # Motive, not Result 18 | 19 | When we attribute unconditional value to an action, it is because we have a certain conception of the motives from which the person acted. 20 | 21 | So what gives a morally good action its special value is the motivation behind it, the principle on the basis of which it is chosen or, in Kantian terms, willed. 22 | 23 | **Duty** is the good will. 24 | 25 | # Criticism 26 | 27 | One of the biggest criticisms of Kantian ethics is that it discounts outcome as a valid factor in evaluating the morality of an action. 28 | -------------------------------------------------------------------------------- /nsu/phi104/slide12.md: -------------------------------------------------------------------------------- 1 | # Kant’s Moral Law 2 | 3 | There are three moral laws: 4 | 5 | 1. **"The formula of universal law"**: act only in accordance with that maxim through which you can at the same time will that it become a universal law. 6 | 7 | 1. **"The formula of humanity"**: So act that you use humanity, whether in your own person or in the person of any other, always at the same time as an end, never merely as a means. 8 | 9 | 1. **"The Principle of Kingdom of Ends"**: the noble ideal of a universal kingdom of ends in themselves (rational beings) to which we can belong as members only when we carefully conduct ourselves in accordance with maxims of freedom as if they were laws of nature. 10 | 11 | Based on Kant’s formula of humanity, human life is sacred and inviolable, meaning one cannot enslave a few people even if it would enable more people to lead better lives. Killing one person to save the lives of millions is impermissible in Kantian ethics. 12 | -------------------------------------------------------------------------------- /nsu/phi104/slide13.md: -------------------------------------------------------------------------------- 1 | # Virtue Ethics 2 | 3 | Virtue ethics is a broad term for theories that emphasize the role of character and virtue in moral philosophy rather than either doing one’s duty or acting in order to bring about good consequences. 4 | 5 | A virtue ethicist is likely to give you this kind of moral advice: “Act as a virtuous person would act in your situation.” 6 | 7 | It may, initially, be identified as the one that emphasizes the virtues, or moral character, in contrast to the approach which emphasizes duties or rules (deontology) or that which emphasizes the consequences of actions (consequentialism). 8 | 9 | # Three Direction 10 | 11 | 1. Eudaimonism 12 | 1. agent-based theories, and the ethics of 13 | 1. care 14 | 15 | Eudaimonism bases virtues in human flourishing, where flourishing is equated with performing one’s distinctive function well. 16 | 17 | An agent-based theory emphasizes that virtues are determined by common-sense intuitions that we as observers judge to be admirable traits in other people. 18 | 19 | The third branch of virtue ethics, the ethics of care, was proposed predominately by feminist thinkers. It challenges the idea that ethics should focus solely on justice and autonomy; it argues that more feminine traits, such as caring and nurturing, should also be considered. 20 | -------------------------------------------------------------------------------- /nsu/phi104/slide14.md: -------------------------------------------------------------------------------- 1 | # Aristotle’s Virtue Ethics 2 | 3 | Seek highest good. 4 | 5 | # Characteristic 6 | 7 | 1. it is desirable for itself, 8 | 1. it is not desirable for the sake of some other good, and 9 | 1. all other goods are desirable for its sake. 10 | -------------------------------------------------------------------------------- /nsu/phi104/slide15.md: -------------------------------------------------------------------------------- 1 | # Application of Ethical Theories 2 | 3 | Applied ethics is a branch of ethics devoted to the treatment of moral problems, practices, and policies in personal life, professions, technology, and government. 4 | 5 | Applied ethics is a field of ethics that deals with ethical questions specific to a professional, disciplinary, or practical field. Subsets of applied ethics include medical ethics, bioethics, business ethics, legal ethics, and others. 6 | -------------------------------------------------------------------------------- /nsu/phi104/slide2.md: -------------------------------------------------------------------------------- 1 | # Slide 2 2 | 3 | # Moral Philosophy 4 | 5 | Moral philosophy is the philosophical thinking about moral terms, moral principles, moral judgments, moral language, what is morally good and bad, right and wrong. 6 | 7 | # Types of Ethics 8 | 9 | The word "ethics" is derived from the Greek "ethos" (meaning "custom" or "habit"). 10 | 11 | Ethics differs from morals and morality in that sense ethics denotes the theory of right action and the greater good, while morals indicate their practice. 12 | 13 | There are 4 types of Ethics: 14 | 15 | 1. Normative ethics 16 | 1. Descriptive ethics 17 | 1. Metaethics 18 | 1. Applied ethics 19 | 20 | # Normative Ethics 21 | 22 | Normative Ethics (or Prescriptive Ethics) is the branch of ethics concerned with establishing how things should be, how to value them, which things are good or bad, and which actions are right or wrong. It attempts to develop a set of rules governing human conduct, or a set of norms for action. 23 | 24 | Three main categories of normative ethics are: 25 | 26 | 1. Utilitarianism or Consequentialism: Consequentialism argues that the morality of an action is contingent on the action's outcome or result. Thus, a morally right action is one that produces a good outcome or consequence. 27 | 1. Deontology: Deontology is an approach to ethics that focuses on the rightness or wrongness of actions themselves, as opposed to the rightness or wrongness of the consequences of those actions. It argues that decisions should be made considering the factors of one's duties and other's rights (the Greek 'deon' means 'obligation' or 'duty'). 28 | 1. Virtue ethics: virtue ethics focuses on the inherent character of a person rather than on the nature or consequences of specific actions performed. 29 | 30 | # Other types 31 | 32 | 1. Descriptive Ethics is a value-free approach to ethics which examines ethics from the perspective of observations of actual choices made by moral agents in practice. It is mainly historical or anthropological. 33 | 1. Meta-Ethics is concerned primarily with the meaning of ethical judgments, and seeks to understand the nature of ethical properties, statements, attitudes, and judgments and how they may be supported or defended. 34 | 1. Applied Ethics is a discipline of philosophy that attempts to apply ethical theory to real-life situations. It is used in determining public policy. 35 | -------------------------------------------------------------------------------- /nsu/phi104/slide3.md: -------------------------------------------------------------------------------- 1 | # Slide 3: The Development of Morality 2 | 3 | Three stages in the development of morality: 4 | 5 | 1. The level of Instinct 6 | 7 | The conduct that appears right to the agent is the conduct determined by his fundamental needs and instinct-the innate tendencies. 8 | 9 | 1. The level of Custom 10 | The conduct that appears right to the agent is conduct in accordance with the customs of the group to which he belongs. 11 | 12 | 1. The level of Conscience 13 | The conduct that appears right to the agent is that approved by his own individual judgment of what is right and wrong. 14 | 15 | # How was the primitive human society? Is there any morality? 16 | 17 | Two views: 18 | 19 | Man was naturally both free and good, it was a life of ideal peace, harmony, goodwill and happiness. 20 | 21 | Every man feels by nature that he has a right to all things and as all are naturally acquisitive and ferocious, they are bound to be in a state of war with one another. 22 | 23 | # Moral, Immoral, Amoral or non-moral 24 | 25 | It is said to be neither moral nor immoral but amoral or non-moral, conduct to which moral predicates are not really applicable at all. 26 | 27 | # Discussion on level of custom 28 | 29 | Man considers to be right those forms of conduct which are approved by the standards or customary modes of behavior of the social group to which he belongs. 30 | 31 | At this level, the bad action is the action that is “not done”, and the good action is the action that has been “always done” by the majority people. 32 | 33 | # Discussion on level of conscience 34 | 35 | At the level of conscience, the moral authority is inside the individual, it is an inner voice that directs him, and the commands of conscience appears the obvious and proper thing to do. 36 | 37 | # Advancements 38 | 39 | 1. The standards of morality are now actively chosen by the individual after moral deliberation. 40 | 2. There is a new personal interest in morality. 41 | 3. Among other things including human welfare pure morality becomes the sphere of the individual. 42 | -------------------------------------------------------------------------------- /nsu/phi104/slide4.md: -------------------------------------------------------------------------------- 1 | # Slide 4: The Development of Ethical Theory 2 | 3 | Three periods in the development of ethical theory: 4 | 5 | In the Greek period the Greek city state formed the background of the moral life, and the man who performed his duties as a citizen was regarded as a good man. 6 | 7 | In the Medieval period morality was dominated by the Church and, generally speaking, the good life was identified with the holy life or the religious life. 8 | 9 | In the Modern period neither Church nor state are so important in the moral life , and morality is more concerned with the free individual and his rights and duties in relation to other free individuals. 10 | -------------------------------------------------------------------------------- /nsu/phi104/slide5.md: -------------------------------------------------------------------------------- 1 | # Slide 5: Autonomy 2 | 3 | Autonomy in ethics refers to individual freedom or one's right to make decisions without being coerced. It is the concept of social, political and ethical morals that give individuals the rational right to make their own informed choices. 4 | 5 | Autonomy means self-law, self-government, self-rule, or self-determination. 6 | 7 | Personal autonomy is, at minimum, self-rule that is free from both controlling interference by others and from limitations, such as inadequate understanding, that prevent meaningful choice. 8 | 9 | To be applied, the principle of autonomy must assume that you are free from the control of others and have the capacity to make your own life choices. 10 | 11 | # Several senses of autonomy: 12 | 13 | 1. Autonomy as informed consent 14 | 2. Autonomy as confidentiality 15 | 3. Autonomy as Truth-telling 16 | 4. Autonomy as Fidelity 17 | -------------------------------------------------------------------------------- /nsu/phi104/slide6.md: -------------------------------------------------------------------------------- 1 | # Slide 6: Rights 2 | 3 | A right is a justified claim on others. 4 | 5 | Beauchamp and Childress, have defined the term "right" to be a "justified claim that individuals and groups can make upon other individuals or upon society”. 6 | 7 | # Legal Rights 8 | 9 | Legal rights are, rights which exist under the rules of legal systems or by virtue of decisions of suitably authoritative bodies within them. 10 | 11 | # Moral Rights 12 | 13 | Moral rights are justified by moral standards that most people acknowledge, but which are not necessarily codified in law; these standards have also, however, been interpreted differently by different people. 14 | 15 | # Types of rights 16 | 17 | 1. **Natural and conventional** - natural rights pertain to us by virtue of our humanity; as such they apply to all persons. Natural rights are commonly called moral rights. Conventional rights are created by humans, generally within the context of social and political organizations. 18 | 19 | 1. **Negative and positive** -- negative rights impose duties of noninterference on others. Thus my right to life as a negative right is a right not be killed. positive rights impose duties of assistance on others. For example, welfare rights impose on the state the duty to assist those who cannot provide for themselves. 20 | 21 | # Universal Declaration of Human Rights 22 | 23 | 1. Article 1. All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. 24 | 1. Article 2. Everyone is entitled to all the rights and freedoms set forth in this Declaration, without distinction of any kind, such as race, colour, sex, language, religion, political or other opinion, national or social origin, property, birth or other status. Furthermore, no distinction shall be made on the basis of the political, jurisdictional or international status of the country or territory to which a person belongs, whether it be independent, trust, non-self-governing or under any other limitation of sovereignty. 25 | 1. Article 3. Everyone has the right to life, liberty and security of person. 26 | 1. Article 4. No one shall be held in slavery or servitude; slavery and the slave trade shall be prohibited in all their forms. 27 | 1. Article 5. No one shall be subjected to torture or to cruel, inhuman or degrading treatment or punishment. 28 | -------------------------------------------------------------------------------- /nsu/phi104/slide7.md: -------------------------------------------------------------------------------- 1 | # Slide 7: Utilitarianism 2 | 3 | # Moral Standard 4 | -------------------------------------------------------------------------------- /nsu/phi104/slide8.md: -------------------------------------------------------------------------------- 1 | # Slide 8: Utilitarianism Bentham 2 | 3 | "The greatest happiness for the greatest numbers" 4 | 5 | Bentham holds that pleasantness is the only quality because of which an experience is good or valuable. 6 | 7 | Each man ought to seek his own maximum pleasure. 8 | 9 | # Hedonic Calculation 10 | 11 | 1. Remoteness – how near it is 12 | 1. Purity – how free from pain it is 13 | 1. Richness – to what extent it will lead to other pleasures 14 | 1. Intensity – how powerful it is 15 | 1. Certainty – how likely it is to result 16 | 1. Extent – how many people it affects 17 | 1. Duration – how long it lasts 18 | -------------------------------------------------------------------------------- /nsu/phi104/slide9.md: -------------------------------------------------------------------------------- 1 | # Slide 9: Utilitarianism Mill 2 | 3 | 1. Pleasure is the only thing that is desirable. 4 | 2. The only proof that a thing is desirable is the fact that people do actually desire it. 5 | 3. Each person’s own pleasure or happiness is a good to that person, so the general happiness is a good to everybody. 6 | 4. Men do desire other objects, but they desire them as a means to pleasure. 7 | 5. If one of two pleasures is preferred by those who are completely acquainted with both we are justified in saying that this preferred pleasure is superior in quality to the other. 8 | 9 | # Limitations 10 | 11 | 1. The imperative to maximize happiness whenever possible is too strict. Should I buy milk for Africa rather than watching movie? 12 | 2. Sometimes maximizing happiness seems more than just right: jumping on grenade. 13 | 3. How are we supposed to calculate the utility for any given situation? We can’t predict the future, and even if we could, there are so many factors to most situations that it seems impossible for us to ever have a good idea of what the best possible consequence will be. 14 | 4. Utilitarianism seems to require us to do things that are not obviously morally permissible: Robin Hood. 15 | -------------------------------------------------------------------------------- /operating-system/address-translation.md: -------------------------------------------------------------------------------- 1 | # Address Translation 2 | 3 | > How do we maintain control over which memory locations an application can access, and thus ensure that application memory accesses are properly restricted? 4 | 5 | With address translation, the hardware transforms each **memory access** (e.g., an instruction fetch, load, or store), changing the virtual address provided by the instruction to a **physical address** where the desired information is actually located. 6 | 7 | ## Initial Assumptions 8 | 9 | Initially, we are going to assume the following things: 10 | 11 | 1. the user’s address space must be placed contiguously in physical memory 12 | 1. the size of the address space is not too big; specifically, that it is less than the size of physical memory 13 | 1. each address space is exactly the same size 14 | 15 | So a sample process will look like the following. 16 | 17 | ![Process and its address space](assets/process-and-its-address-space.png) 18 | 19 | ## Process perspective 20 | From the program’s perspective, its address space starts at address 0 and grows to a maximum of 16 KB; all memory references it generates should be within these bounds. However, to virtualize memory, the OS wants to place the process somewhere else in physical memory, not necessarily at address 0. 21 | 22 | > How can we relocate this process in memory in a way that is transparent to the process? How can we provide the illusion of a virtual address space starting at 0, when in reality the address space is located at some other physical address? 23 | 24 | ## Base and Bounds or Dynamic Relocation 25 | 26 | We’ll need two hardware registers within each CPU: one is called the base register, and the other the bounds (sometimes called a limit register). 27 | 28 | In this setup, each program is written and compiled as if it is loaded at address zero. However, when a program starts running, the OS decides where in physical memory it should be loaded and sets the base register to that value. 29 | 30 | Interesting things start to happen when the process is running. Now, when any memory reference is generated by the process, it is translated by the processor in the following manner: 31 | ``` 32 | physical address = virtual address + base 33 | ``` 34 | 35 | Each memory reference generated by the process is a **virtual address**; the hardware in turn adds the contents of the base register to this address and the result is a **physical address** that can be issued to the memory system. 36 | 37 | Transforming a virtual address into a physical address is exactly the technique we refer to as **address translation**; that is, the hardware takes a virtual address the process thinks it is referencing and transforms it into a physical address which is where the data actually resides. Because this relocation of the address happens at runtime, and because we can move address spaces even after the process has started running, the technique is often referred to as **dynamic relocation**. 38 | 39 | **What about bound register?** 40 | 41 | Bound register is used for comparing if the virtual address is within legal bound. 42 | 43 | If a process generates a virtual address that is greater than the bounds, or one that is negative, the CPU will raise an exception, and the process will likely be terminated. 44 | 45 | We should note that the base and bounds registers are hardware structures kept on the chip (one pair per CPU). Sometimes people call the part of the processor that helps with address translation the **memory management unit (MMU)** 46 | 47 | ![Base and Bound Registers](assets/base-and-bound.gif) 48 | -------------------------------------------------------------------------------- /operating-system/advanced-page-tables.md: -------------------------------------------------------------------------------- 1 | # Advanced Paging Table 2 | 3 | ## Linear Paging Table Memory Problem 4 | 5 | We now tackle the second problem that paging introduces: page tables are too big and thus consume too much memory. 6 | 7 | Let’s start out with a linear page table. As you might recall1 , linear page tables get pretty big. Assume again a 32-bit address space ($2^{32}$ bytes), with 4KB ($2^{12}$ byte) pages and a 4-byte page-table entry. An address space thus has roughly one million virtual pages in it ( $\frac{2^{32}}{2^{12}}$ ); multiply by the page-table entry size and you see that our page table is 4MB in size. 8 | 9 | Recall also: we usually have one page table for every process in the system! With a hundred active processes (not uncommon on a modern system), we will be allocating hundreds of megabytes of memory just for page tables! 10 | 11 | ## Simple Solution: Bigger Pages 12 | 13 | Increasing the page size by factor of four reduces memory requirement by factor of four. 14 | 15 | The major problem with this approach, however, is that big pages lead to waste within each page, a problem known as internal fragmentation (as the waste is internal to the unit of allocation). 16 | 17 | Thus, most systems use relatively small page sizes in the common case: **4KB** (as in x86) or 8KB (as in SPARCv9). 18 | 19 | ## Multi Level Page Tables 20 | 21 | How about we split the page table into pages itself? Recursive solution! 22 | 23 | The basic idea behind a multi-level page table is simple. First, chop up the page table into page-sized units; then, if an entire page of page-table entries (PTEs) is invalid, don’t allocate that page of the page table at all. To track whether a page of the page table is valid (and if valid, where it is in memory), use a new structure, called the **page directory**. The page directory thus either can be used to tell you where a page of the page table is, or that the entire page of the page table contains no valid pages. 24 | 25 | ![Page Directory](assets/markdown-img-paste-20180615145313928.png) 26 | 27 | The page directory, in a simple two-level table, contains one entry per page of the page table. It consists of a number of **page directory entries (PDE)**. A PDE (minimally) has a valid bit and a page frame number (PFN), similar to a PTE. 28 | 29 | | Advantages | Disadvantages | 30 | | :------------- | :------------- | 31 | | Memory Efficient: Supports sparse paging table | On a TLB miss, two loads from memory is required: one for the page directory, and one for the PTE itself | 32 | | Paging table split into pages. Easier to find space in RAM. | Complexity: both software and hardware support is now complex. | 33 | 34 | ![PDR Algo](assets/markdown-img-paste-20180615150529748.png) 35 | 36 | ## Inverted Page Tables 37 | 38 | An even more extreme space savings in the world of page tables is found with **inverted page tables**. Here, instead of having many page tables (one per process of the system), we keep a single page table that has an entry for each physical page of the system. The entry tells us which process is using this page, and which virtual page of that process maps to this physical page. 39 | 40 | Finding the correct entry is now a matter of searching through this data structure. A linear scan would be expensive, and thus a hash table is often built over the base structure to speed lookups. 41 | 42 | ## Resources 43 | 44 | 1. [Operating System Three Easy Pieces - Chapter 20](http://pages.cs.wisc.edu/~remzi/OSTEP/vm-smalltables.pdf) 45 | -------------------------------------------------------------------------------- /operating-system/assets/base-and-bound.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/operating-system/assets/base-and-bound.gif -------------------------------------------------------------------------------- /operating-system/assets/markdown-img-paste-20180501093342227.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/operating-system/assets/markdown-img-paste-20180501093342227.png -------------------------------------------------------------------------------- /operating-system/assets/markdown-img-paste-20180501122536680.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/operating-system/assets/markdown-img-paste-20180501122536680.png -------------------------------------------------------------------------------- /operating-system/assets/markdown-img-paste-20180503220643690.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/operating-system/assets/markdown-img-paste-20180503220643690.png -------------------------------------------------------------------------------- /operating-system/assets/markdown-img-paste-20180503223513809.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/operating-system/assets/markdown-img-paste-20180503223513809.png -------------------------------------------------------------------------------- /operating-system/assets/markdown-img-paste-20180615003730499.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/operating-system/assets/markdown-img-paste-20180615003730499.png -------------------------------------------------------------------------------- /operating-system/assets/markdown-img-paste-20180615011422807.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/operating-system/assets/markdown-img-paste-20180615011422807.png -------------------------------------------------------------------------------- /operating-system/assets/markdown-img-paste-20180615012425329.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/operating-system/assets/markdown-img-paste-20180615012425329.png -------------------------------------------------------------------------------- /operating-system/assets/markdown-img-paste-20180615145313928.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/operating-system/assets/markdown-img-paste-20180615145313928.png -------------------------------------------------------------------------------- /operating-system/assets/markdown-img-paste-20180615150529748.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/operating-system/assets/markdown-img-paste-20180615150529748.png -------------------------------------------------------------------------------- /operating-system/assets/process-and-its-address-space.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/operating-system/assets/process-and-its-address-space.png -------------------------------------------------------------------------------- /operating-system/cache.md: -------------------------------------------------------------------------------- 1 | # Cache 2 | 3 | Caching is one of the most fundamental performance techniques in computer systems, one that is used again and again to make the “commoncase fast”. 4 | 5 | The idea behind hardware caches is to take advantage of locality in instruction and data references. There are usually two types of locality: **temporal locality** and **spatial locality**. 6 | 7 | With temporal locality, the idea is that an instruction or data item that has been recently accessed will likely be re-accessed soon in the future. Think of loop variables or instructions in a loop; they are accessed repeatedly over time. 8 | 9 | With spatial locality, the idea is that if a program accesses memory at address x, it will likely soon access memory near x. Imagine here streaming through an array of some kind, accessing one element and then the next. Of course, these properties depend on the exact nature of the program, and thus are not hard-and-fast laws but more like rules of thumb. 10 | 11 | Hardware caches, whether for instructions, data, or address translations (as in our TLB) take advantage of locality by keeping copies of memory in small, fast on-chip memory. Instead of having to go to a (slow) memory to satisfy a request, the processor can first check if a nearby copy exists in a cache; if it does, the processor can access it quickly (i.e., in a few CPU cycles) and avoid spending the costly time it takes to access memory (many nanoseconds). 12 | 13 | You might be wondering: if caches (like the TLB) are so great, **why don’t we just make bigger caches and keep all of our data in them?** Unfortunately, this is where we run into more fundamental laws like those of physics. If you want a fast cache, it has to be small, as issues like the speed-of-light and other physical constraints become relevant. Any large cache by definition is slow, and thus defeats the purpose. Thus, we are stuck with small, fast caches; the question that remains is how to best use them to improve performance. 14 | -------------------------------------------------------------------------------- /operating-system/consistency.md: -------------------------------------------------------------------------------- 1 | # Consistency 2 | 3 | 1. No Consistency 4 | 2. Strong Consistency aka Linearizability 5 | 3. Sequential Consistency 6 | 7 | ## Strong Consistency aka Linearizability 8 | 9 | **Disclaimer**: This might be wrong. 10 | 11 | Imagine I have many processors in computer, processing various tasks. Here, let us consider tasks as method calls. Let's say we only have write and read calls. 12 | 13 | It is difficult for a programmer to reasons with so many things happening at once. So what the programmer does is, it tries to imagine that all these tasks are not happening in multiple processors, instead, they are being run on a single processor linearly. 14 | 15 | So the programmer takes all the operations from various processor and merges them together in a linear order. But, he needs to ensure that this linear model is equivalent to the previous multi-core model. 16 | 17 | What do I mean by equivalent? The read operation in the linear model must return what it returns in the multi-core model (**a read from a location to return the value of the last write to that location**). If he can ensure this, then this linear model is equivalent to multi-core model. 18 | 19 | Another thing he needs to ensure: **if a response precedes an invocation, then this ordering must be preseverd in linear model**. 20 | 21 | Not all multi-core model can be mapped to a linear model like this. Those that can be mapped are called **Linearizable**. 22 | 23 | ## Sequential Consistency 24 | 25 | For this, the linear model needs to simply make sure **if a response precedes an invocation, then this ordering must be preseverd in linear model**. That's it. Only the ordering of operations in individual processor is maintained. 26 | 27 | ## Resources 28 | 29 | 1. [A Foo Walks into a Bar: Consistency Models Explained Briefly](http://coldattic.info/post/88/) -------------------------------------------------------------------------------- /operating-system/operating-system.md: -------------------------------------------------------------------------------- 1 | # Process 2 | 3 | ## Memory 4 | 5 | 1. [Address Translation](./address-translation.md) 6 | - Base and Bound Registers 7 | - Dynamic Relocation 8 | 1. [Segmentation](./segmentation.md) 9 | - Remove waste between heap and stack 10 | - Code sharing 11 | - Flexible Size causes external fragmentation. 12 | 1. [Paging](paging.md) 13 | - Page vs Page Frames 14 | 1. [Translation Lookaside Buffer](tlb.md) 15 | - RAM isn't always RAM 16 | 1. [Advanced Page Tables](advanced-page-tables.md) 17 | - Page Directory 18 | - Inverse Page Index 19 | 1. [Cache](cache.md) 20 | - Spatial locality 21 | - Temporal locality 22 | 23 | ## Persistance 24 | -------------------------------------------------------------------------------- /operating-system/segmentation.md: -------------------------------------------------------------------------------- 1 | # Segmentation 2 | 3 | > How do we support a large address space with (potentially) a lot of free space between the stack and the heap? 4 | 5 | Instead of having just one base and bounds pair in our MMU, why not have a base and bounds pair per logical segment of the address space? 6 | 7 | We have three logically-different segments: code, stack, and heap. What segmentation allows the OS to do is to place each one of those segments in different parts of physical memory, and thus avoid filling physical memory with unused virtual address space. 8 | 9 | **Segmentation Fault**: The term segmentation fault or violation arises from a memory access on a segmented machine to an illegal address. Humorously, the term persists, even on machines with no support for segmentation at all. 10 | 11 | The hardware uses segment registers during translation. 12 | 13 | ## Segment Identification 14 | >How does it know the offset into a segment, and to which segment an address refers? 15 | 16 | One common approach, sometimes referred to as an explicit approach, is to chop up the address space into segments based on the top few bits of the virtual address 17 | 18 | ## Backwards Growth 19 | 20 | The stack grows backwards. So for each segment, we also need to store information about its direction of growth. 21 | 22 | ## Support for Sharing 23 | 24 | To save memory, sometimes it is useful to share certain memory segments between address spaces. In particular, **code sharing** is common and still in use in systems today. 25 | 26 | To support sharing, we need a little extra support from the hardware, in the form of **protection bits**. Basic support adds a few bits per segment, indicating whether or not a program can read or write a segment, or perhaps execute code that lies within the segment. By setting a code segment to read-only, the same code can be shared across multiple processes, without worry of harming isolation; while each process still thinks that it is accessing its own private memory, the OS is secretly sharing memory which cannot be modified by the process, and thus the illusion is preserved. 27 | 28 | ![Segments With Protection](assets/markdown-img-paste-20180501093342227.png) 29 | 30 | With protection bits, the hardware algorithm described earlier would also have to change. In addition to checking whether a virtual address is within bounds, the hardware also has to check whether a particular access is permissible. If a user process tries to write to a read-only segment, or execute from a non-executable segment, the hardware should raise an exception, and thus let the OS deal with the offending process. 31 | 32 | Segmentation is one method of implementing **memory protection**. 33 | 34 | # Problems 35 | 36 | - What if large but sparsely used heap falls in one logical segment? There is still waste. 37 | 38 | - What to do about external fragmentation? 39 | 40 | # Reference 41 | 42 | 1. [Operating Systems Three Little Pieces Chapter 16](http://pages.cs.wisc.edu/~remzi/OSTEP/vm-segmentation.pdf) 43 | 1. [Wiki Memory Segmentation](https://en.wikipedia.org/wiki/Memory_segmentation) 44 | -------------------------------------------------------------------------------- /operating-system/tlb.md: -------------------------------------------------------------------------------- 1 | # Translation Lookaside Buffer 2 | 3 | > How to avoid extra memory reference during translation? 4 | 5 | To speed address translation, we are going to add what is called a **translation-lookaside buffer**, or **TLB**. 6 | 7 | A TLB is part of the chip’s memory-management unit (MMU), and is simply a **hardware cache** of popular virtual-to-physical address translations; thus, a better name would be an address-translation cache. 8 | 9 | Upon each virtual memory reference, the hardware first checks the TLB to see if the desired translation is held therein; if so, the translation is performed (quickly) without having to consult the page table (which has all translations). 10 | 11 | ## TLB Basic Algorithm 12 | 13 | ![TLB Basci Algorithm](assets/markdown-img-paste-20180615003730499.png) 14 | 15 | We either have **TLB Hit** or **TLB Miss**. Notice that on **TLB Miss**, we retry the instruction at the end. 16 | 17 | For better understanding, read section 19.2 from [OS 3 Easy Pieces - TLB](http://pages.cs.wisc.edu/~remzi/OSTEP/vm-tlbs.pdf). 18 | 19 | Like any cache, TLBs rely upon both **spatial locality** and **temporal locality** for success, which are program properties. If the program of interest exhibits such locality (and many programs do), the TLB hit rate will likely be high. 20 | 21 | Read more on [Cache](./cache.md). 22 | 23 | ## Who should handle TLB Miss? 24 | 25 | Hardware vs Software 26 | 27 | If we let hardware handle it, like old systems, then they will need to know location of page table and it's exact format. 28 | 29 | The primary advantage of the software-managed approach is flexibility: the OS can use any data structure it wants to implement the page table, without necessitating hardware change. Another advantage is simplicity, as seen in picture below. 30 | 31 | ![OS Handling TLB Miss](assets/markdown-img-paste-20180615011422807.png) 32 | 33 | ## TLB Issue: Context Switches 34 | 35 | With TLBs, some new issues arise when switching between processes (and hence address spaces). Specifically, the TLB contains virtual-to-physical translations that are only valid for the currently running process; these translations are not meaningful for other processes. As a result, when switching from one process to another, the hardware or OS (or both) must be careful to ensure that the about-to-be-run process does not accidentally use translations from some previously run process. 36 | 37 | > How to manage TLB contents on a context switch? 38 | 39 | One approach is to simply flush the TLB on context switches, thus emptying it before running the next process. On a software-based system, this can be accomplished with an explicit (and privileged) hardware instruction; with a hardware-managed TLB, the flush could be enacted when the page-table base register is changed (note the OS must change the PTBR on a context switch anyhow). In either case, the flush operation simply sets all valid bits to 0, essentially clearing the contents of the TLB. However, there is a cost: each time a process runs, it must incur TLB misses as it touches its data and code pages. If the OS switches between processes frequently, this cost may be high. 40 | 41 | To reduce this overhead, some systems add hardware support to enable sharing of the TLB across context switches. In particular, some hardware systems provide an **address space identifier (ASID)** field in the TLB. You can think of the ASID as a process identifier (PID), but usually it has fewer bits (e.g., 8 bits for the ASID versus 32 bits for a PID). 42 | 43 | Here is a depiction of a TLB with the added ASID field: 44 | ![TLB With ASID](assets/markdown-img-paste-20180615012425329.png) 45 | 46 | ## TLB Coverage 47 | 48 | > RAM isn’t always RAM (Culler’s law) 49 | 50 | Due to TLB, access to pages from RAM may differ in time greatly. So all reading RAM randomly may not be ideal. 51 | 52 | In particular, if the number of pages a program accesses in a short period of time exceeds the number of pages that fit into the TLB, the program will generate a large number of TLB misses, and thus run quite a bit more slowly. We refer to this phenomenon as exceeding the **TLB coverage**, and it can be quite a problem for certain programs. 53 | 54 | ## Resources 55 | 56 | 1. [OS Three Little Pieces: Chapter 19 Paging - Faster Translations (TLBs)](http://pages.cs.wisc.edu/~remzi/OSTEP/vm-tlbs.pdf) 57 | -------------------------------------------------------------------------------- /tools/assets/2018-08-02-15-55-08.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/tools/assets/2018-08-02-15-55-08.png -------------------------------------------------------------------------------- /tools/assets/2018-08-02-15-57-56.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/tools/assets/2018-08-02-15-57-56.png -------------------------------------------------------------------------------- /tools/assets/markdown-img-paste-20180717103331600.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/forthright48/notes/4071d44bd54384d5fde48f1f8ad2cf4e4bfe3389/tools/assets/markdown-img-paste-20180717103331600.png -------------------------------------------------------------------------------- /tools/atom.md: -------------------------------------------------------------------------------- 1 | # Packages 2 | 3 | 1. [autocomplete-modules](https://atom.io/packages/autocomplete-modules): Auto-completes require "modules" from 'node_modules' folder. 4 | 1. [toggle-quotes](https://atom.io/packages/toggle-quotes) Toggle between "" and '' 5 | 1. [js-hyperclick](https://atom.io/packages/js-hyperclick) Goto definitions. 6 | 1. [vue-hyperclick](https://atom.io/packages/vue-hyperclick) 7 | 8 | # Shortcuts 9 | 10 | ## Fuzzy Finder 11 | ``` 12 | C+B Find from open files 13 | ``` 14 | -------------------------------------------------------------------------------- /tools/awk.md: -------------------------------------------------------------------------------- 1 | # AWK 2 | 3 | AWK is a programming language designed for text processing and typically used as a data extraction and reporting tool. It is a standard feature of most Unix-like operating systems. 4 | 5 | **For tasks that read files line by line and split on delimiters, AWK is probably the right tool for the job.** 6 | 7 | ## Number of field in each row 8 | 9 | ```bash 10 | awk "{print NF}" < pos_cut.txt 11 | ``` 12 | 13 | NF is an AWK built in variable and it stands for number of fields. 14 | 15 | ## Searching 16 | 17 | ```bash 18 | awk '/2410626/' pos_cut.txt 19 | ``` 20 | 21 | Will print the rows that contain the integer 2410626. 22 | 23 | ## Searching and Printing 24 | 25 | ```bash 26 | less file | awk '{ if (/samiul/) { print $2} }' 27 | ``` 28 | 29 | ## Custom separator 30 | 31 | ```bash 32 | awk -F ":" '/2410626/' pos_cut.txt 33 | awk 'BEGIN {OFS=","} /samiul/{ print $2} }' pos_cut.txt 34 | ``` 35 | 36 | ## Resources 37 | 38 | 1. [Tim-Dennis: Using AWK to Filter Rows](https://www.tim-dennis.com/data/tech/2016/08/09/using-awk-filter-rows.html) 39 | 2. [StackOverflow: What is the difference between sed and awk?](https://stackoverflow.com/questions/1632113/what-is-the-difference-between-sed-and-awk) 40 | 3. [Learn Awk in Y minutes](https://learnxinyminutes.com/docs/awk/) -------------------------------------------------------------------------------- /tools/bazel.md: -------------------------------------------------------------------------------- 1 | # Bazel 2 | 3 | ## How to run a module in BUILD 4 | 5 | In bazel, you must have a root somewhere. Just create a file name `WORKSPACE` somewhere and that becomes your root. 6 | 7 | Now, somewhere down the root, you will have a folder with BUILD file in it. The BUILD file will contain lots of rules. If you want to run a rule, you have to use relative path from the root. 8 | 9 | ``` 10 | build run //relative/path/to/build/file:rule 11 | ``` -------------------------------------------------------------------------------- /tools/docker-compose.md: -------------------------------------------------------------------------------- 1 | # Docker Compose 2 | 3 | Name your containers and network. Depending on default names might cause trouble sometimes. 4 | 5 | ```yml 6 | services: 7 | redis: 8 | container_name: entrytask_redis 9 | 10 | networks: 11 | ntw: 12 | name: entrytask_ntw 13 | ``` 14 | -------------------------------------------------------------------------------- /tools/elasticsearch.md: -------------------------------------------------------------------------------- 1 | # Elasticsearch 2 | 3 | ES runs on port 9200. 4 | 5 | # Kibana 6 | 7 | http://localhost:5601/app/kibana 8 | 9 | # Elastic Request 10 | 11 | `curl -X '://:/?' -d ''` 12 | 13 | In short form it looks like this: 14 | 15 | ``` 16 | PUT /megacorp/employee/2 17 | { 18 | "first_name" : "Jane", 19 | "last_name" : "Smith", 20 | "age" : 32, 21 | "about" : "I like to collect rock albums", 22 | "interests": [ "music" ] 23 | } 24 | ``` 25 | 26 | Here it follows the format `/[index]/[type]/[id]`. 27 | 28 | # Common Commands 29 | 30 | ``` 31 | # Search specific id 32 | GET /megacorp/employee/1 33 | 34 | # Search all 35 | GET /megacorp/employee/_search 36 | 37 | # Search with query 38 | GET /megacorp/employee/_search?q=last_name:Smith 39 | 40 | # Search using DSL 41 | GET /megacorp/employee/_search 42 | { 43 | "query" : { 44 | "match" : { 45 | "last_name" : "Smith" 46 | } 47 | } 48 | } 49 | 50 | # Auto generate id 51 | POST /website/blog/ 52 | { 53 | "title": "My second blog entry", 54 | "text": "Still trying this out...", 55 | "date": "2014/01/01" 56 | } 57 | 58 | # Get particular field of source 59 | GET /website/blog/123?_source=title,text 60 | 61 | # Get only source without any metadata 62 | GET /website/blog/123/_source 63 | 64 | # Create only if it doesn't exist 65 | PUT /website/blog/123/_create 66 | {...} 67 | 68 | 69 | ``` 70 | -------------------------------------------------------------------------------- /tools/git.md: -------------------------------------------------------------------------------- 1 | # Git 2 | 3 | ## Three Way Merge 4 | 5 | What happens when you merge a branch? For example, you want to merge Feature_A branch with master. 6 | 7 | What git does is, it finds 3 commits: the head of Feature_A, the head of master and lowest common ancestor of Feature_A and Master. Then uses these 3 commits to make decisions for merge. 8 | 9 | But why do we need three commits? Can't we just use head of each branch to make the decision? 10 | 11 | ![](assets/2018-08-02-15-55-08.png) 12 | 13 | In the picture above, is it possible to figure out which branch changed made the changes to line 1, 3 and 5? No. Hence, we have to manually decide for each line. 14 | 15 | But, if we use three commits to merge, then things get a bit easier. 16 | 17 | ![](assets/2018-08-02-15-57-56.png) 18 | 19 | Now, we can easily determine which branch made changes to each line. Only for the last line we have merge conflict. For the rest, we have auto merge. 20 | 21 | And this is the reason git uses three way merging. 22 | 23 | ## Git tag 24 | 25 | Create tags locally for easier rollback, e.g, before merging branch with master. 26 | 27 | ```bash 28 | git tag # show tags 29 | git tag -a v1.4 -m "my version 1.4" # Annotated tags 30 | git show v1.4 31 | git push origin v1.4 32 | git checkout v1.4 33 | git tag example # Lightweight 34 | git tag -d v1 # Delete 35 | ``` 36 | 37 | ## Squashing multiple commits 38 | 39 | `git rebase -i HEAD~3` or however many commits instead of 3. 40 | 41 | Turn this 42 | 43 | ```text 44 | pick YourCommitMessageWhatever 45 | pick YouGetThePoint 46 | pick IdkManItsACommitMessage 47 | ``` 48 | 49 | into this 50 | 51 | ```text 52 | pick YourCommitMessageWhatever 53 | s YouGetThePoint 54 | s IdkManItsACommitMessage 55 | ``` 56 | 57 | Save and come out. 58 | 59 | It's a lot easier than it looks. There are plenty of helpful comments to guide you once the rebase starts. 60 | 61 | ## Delete Submodule 62 | 63 | 1. Delete the relevant section from the .gitmodules file. 64 | 2. Stage the .gitmodules changes git add .gitmodules 65 | 3. Delete the relevant section from .git/config. 66 | 4. Run `git rm --cached path_to_submodule` (no trailing slash). 67 | 5. Run `rm -rf .git/modules/path_to_submodule` 68 | 6. Commit git commit -m "Removed submodule XXX" 69 | 7. Delete the now untracked submodule files `rm -rf path_to_submodule` 70 | 71 | ## Upstream 72 | 73 | Setting upstream allows you to view extra info during `git status` 74 | 75 | ![](assets/markdown-img-paste-20180717103331600.png) 76 | 77 | You can learn if your branch is ahead or behind a particular remote branch. 78 | 79 | ```bash 80 | git branch --set-upstream-to origin/my_branch 81 | git branch -u origin/my_branch 82 | ``` 83 | 84 | ## Default to ssh always 85 | 86 | This is useful when you have 2FA enabled in some website. You can't submit 2FA from terminal, can you? So force git to use ssh. 87 | 88 | ```bash 89 | git config --global url."git@github.com:".insteadOf "https://github.com/" 90 | ``` 91 | 92 | ## Caret vs Tilde 93 | 94 | > What is the diff between HEAD~ and HEAD^ ? 95 | 96 | A commit can have two parent if it's a merge commit. ~ means first parent and ^ means second parent. 97 | 98 | So, for day to day use, I think ~ is sufficient. 99 | 100 | ```bash 101 | git show HEAD # Shows details of last commit. 102 | git show HEAD~ # Shows details of second last commit. Same as HEAD~1. 103 | git show HEAD~2 # Shows details of third last commit. 104 | ``` 105 | 106 | ## git-diff 107 | 108 | ``` 109 | git diff # unstaged 110 | git diff --staged # staged 111 | git diff HEAD # Both staged and unstaged 112 | ``` -------------------------------------------------------------------------------- /tools/npm.md: -------------------------------------------------------------------------------- 1 | # NPM 2 | 3 | # How to publush 4 | 5 | ``` 6 | npm version [major|minor|patch] 7 | npm publish 8 | ``` 9 | -------------------------------------------------------------------------------- /tools/protobuf.md: -------------------------------------------------------------------------------- 1 | # Protobuf 2 | 3 | 1. 'required' tag has been lifted for sake of simplicity. [StackOverflow](https://stackoverflow.com/questions/31801257/why-required-and-optional-is-removed-in-protocol-buffers-3) 4 | 1. C++ style commenting supported. 5 | 6 | # Compile 7 | 8 | ```bash 9 | protoc --go_out=. *.proto 10 | ``` -------------------------------------------------------------------------------- /tools/regex.md: -------------------------------------------------------------------------------- 1 | # Javascript Regex 2 | 3 | Use the following syntax to build regex 4 | 5 | ``` 6 | regexp = /regex-expression/flag; 7 | match = regexp.exec(a_string_to_match_against); 8 | 9 | // If you want boolean only 10 | test = regexp.test(a_string_to_match_against); 11 | ``` 12 | 13 | Returns captured groups in a 1-indexed array (because match[0] contains the input). 14 | 15 | If no match occurs, it returns null. 16 | 17 | ## From String 18 | ``` 19 | var re = new RegExp("a|b", "i"); 20 | // same as 21 | var re = /a|b/i; 22 | ``` 23 | 24 | # Non Capturing Group 25 | `/(?:whatever)/` 26 | 27 | `(?:)` is a non-capturing group 28 | -------------------------------------------------------------------------------- /tools/tmux.md: -------------------------------------------------------------------------------- 1 | # Tmux 2 | 3 | A multiplexer for terminal 4 | 5 | # Install 6 | ``` 7 | apt install tmux 8 | ``` 9 | 10 | # Tmux commands 11 | ``` 12 | tmux -V # Check version 13 | tmux ls # Check running sessions 14 | ``` 15 | 16 | # Start new session 17 | ``` 18 | tmux new 19 | tmux new -s [name] 20 | ``` 21 | 22 | # Attach, Detach and Kill 23 | ``` 24 | ctrl+b d # Detach 25 | tmux a -t 3 # Attach session 26 | tmux a -t [name] 27 | tmux a # # Attach to last session 28 | ``` 29 | 30 | # Pane 31 | ``` 32 | ctrl+b " # Split Horizontal 33 | ctrl+b % # Split Vertical 34 | 35 | $resize-pane -D/U/L/R [line number] 36 | 37 | prefix + { # Swap Pane anti-clockwise 38 | prefix + } # Swap Pane clockwise 39 | ``` 40 | 41 | prefix + space # Changes to next layout 42 | 43 | # Exit 44 | ``` 45 | exit 46 | ``` 47 | 48 | # Tmuxinator 49 | 50 | A script manager for handling tmux sessions. 51 | 52 | # Navigation 53 | 54 | ``` 55 | prefix + [ # Enter Navigation mode 56 | q # Exit to normal mode 57 | ``` 58 | 59 | # Resources 60 | 61 | 1. **Book** - Tmux 2: Productive Mouse-Free Development 62 | 1. [Oh My Tmux!](https://github.com/gpakosz/.tmux) - configuration for tmux. 63 | -------------------------------------------------------------------------------- /tools/vim.md: -------------------------------------------------------------------------------- 1 | # Vim 2 | 3 | ## vimrc 4 | 5 | `.vimrc` should be created in $HOME directory. 6 | 7 | Get vimrc from [Ultimate vimrc](https://github.com/amix/vimrc) 8 | 9 | ## Movement 10 | 11 | - `G`: Move to last line 12 | 13 | ## Cut, Copy, Paste and Select 14 | 15 | - `V` for selecting a line 16 | - `v` for selecting characters 17 | - `d` for cutting and `y` for copying. `d` stands for delete in vim and `y` stands for yank 18 | - `yy` for copying a line. `y$` for copying line without newline. 19 | - Prefix `"+` to use clipboard 20 | - `P` for pasting before cursor and `p` for pasting after 21 | 22 | Install `vim-gnome` so that you can copy to clipboard. 23 | 24 | **Undo**: `u` 25 | **Redo**: `ctrl + r` 26 | 27 | ## Search and Replace 28 | 29 | - Number of times a pattern matched: `:%s/PATTERN//gn` for all matches or `:%s/PATTERN//n` for just number of lines. 30 | 31 | ## Mapping Commands 32 | 33 | `:map` and `:noremap` are recursive and non-recursive versions of the various mapping commands. What that means is that if you do: 34 | 35 | ```bash 36 | :map j gg 37 | :map Q j 38 | :noremap W j 39 | ``` 40 | 41 | - j will be mapped to gg. 42 | - Q will also be mapped to gg, because j will be expanded for the recursive mapping. 43 | - W will be mapped to j (and not to gg) because j will not be expanded for the non-recursive mapping. 44 | 45 | Now remember that Vim is a modal editor. It has a **normal** mode, **visual** mode and other modes. 46 | 47 | - For each of these sets of mappings, there is a mapping that works in normal, visual, select and operator modes (:map and :noremap) 48 | - one that works in normal mode (:nmap and :nnoremap) 49 | - one in visual mode (:vmap and :vnoremap) and so on. 50 | 51 | # Plugin 52 | 53 | ## NerdTree 54 | 55 | - Change window back to nerdTree: w 56 | - Toggle NerdTree: nn 57 | - [Create folder/file in nerdTree](https://sookocheff.com/post/vim/creating-a-new-file-or-directoryin-vim-using-nerdtree): go to the folder and press "m" to bring out the menu. 58 | - Toggle buffer using: + o 59 | 60 | --------------------------------------------------------------------------------