├── .gitignore ├── .replit ├── .vscode └── settings.json ├── LICENSE ├── README.md ├── computer-science └── algorithms_plus_data_structures_equals_programs.md ├── data-science ├── data_science_for_business.md ├── images │ └── nltk_pipeline.png └── natural_language_processing_with_python.md ├── databases ├── first_course_on_database_systems.md ├── sql_for_web_nerds.md └── stanford_db_course.md ├── developer-relations ├── business_value_of_developer_relations.md └── developer_marketing_and_relations__the_essential_guide.md ├── devops ├── docker.md └── kubernetes │ ├── create-cluster-using-kubeadm.md │ ├── gke.md │ ├── introduction.md │ ├── kubernetes-deep-dive.md │ ├── notes.md │ ├── todo.md │ └── yaml.md ├── django ├── django.md ├── django_testing.md └── test_drive_web_development_with_python.md ├── general ├── best_software_writing_i.md ├── how_to_read_a_book.md ├── learning_things_quickly.md └── software_debugging_udacity.md ├── javascript ├── javascript_lanugage_guide.md ├── javascript_the_good_parts.md ├── typescript-deep-dive │ └── promise-states-and-fates.png └── typescript_deep_dive.md ├── meetup ├── blockchain-in-healthcare-summit.md ├── chicago-startup-community.md ├── pycon2018.md └── react-chicago.md ├── product └── inspire.md ├── professional-development ├── 7_principles_of_public_speaking.md ├── passionate_programmer.md └── pragmatic_programmer.md ├── python ├── fluent_python.md └── logging.md ├── scala └── scala_for_the_impatient.md ├── snippets ├── docker.md ├── domain_modeling.md ├── git.md ├── kubernetes.md └── python.md ├── software-engineering ├── apprenticeship_patterns.md ├── clean_code.md ├── design_patterns.md ├── domain_driven_design.md ├── head_first_design_patterns.md ├── philosophy_of_software_design.md └── refactoring.md ├── testing ├── developer_testing_bootcamp.md └── mock.md ├── unix ├── vim_cheatsheet.md └── your_unix.md └── web-development ├── 12_factor_app.md ├── api.md ├── building_microservices.md ├── graphql.md ├── http.md ├── microservices.md └── wsgi.md /.gitignore: -------------------------------------------------------------------------------- 1 | *.ipynb 2 | .ipynb_checkpoints/ 3 | .DS_Store 4 | .envrc 5 | venv/ 6 | -------------------------------------------------------------------------------- /.replit: -------------------------------------------------------------------------------- 1 | language = "bash" -------------------------------------------------------------------------------- /.vscode/settings.json: -------------------------------------------------------------------------------- 1 | { 2 | "markdownlint.config": { 3 | "default": false, 4 | "MD007": false, 5 | "MD024": false, 6 | "MD028": false, 7 | "MD032": false, 8 | }, 9 | "editor.detectIndentation": false, 10 | "editor.rulers": [], 11 | "markdown-toc.depthFrom": 2, 12 | "markdown-toc.depthTo": 3, 13 | } 14 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | This is free and unencumbered software released into the public domain. 2 | 3 | Anyone is free to copy, modify, publish, use, compile, sell, or 4 | distribute this software, either in source code form or as a compiled 5 | binary, for any purpose, commercial or non-commercial, and by any 6 | means. 7 | 8 | In jurisdictions that recognize copyright laws, the author or authors 9 | of this software dedicate any and all copyright interest in the 10 | software to the public domain. We make this dedication for the benefit 11 | of the public at large and to the detriment of our heirs and 12 | successors. We intend this dedication to be an overt act of 13 | relinquishment in perpetuity of all present and future rights to this 14 | software under copyright law. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 17 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 18 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 19 | IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR 20 | OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 21 | ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR 22 | OTHER DEALINGS IN THE SOFTWARE. 23 | 24 | For more information, please refer to 25 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # SivNotes 2 | 3 | Notes from books I am reading. 4 | 5 | #### Table of Contents 6 | 7 | 8 | 9 | - [Computer Science](#computer-science) 10 | - [Data Science](#data-science) 11 | - [Databases](#databases) 12 | - [DevOps](#devops) 13 | - [Django](#django) 14 | - [General](#general) 15 | - [Javascript](#javascript) 16 | - [Meetup](#meetup) 17 | - [Product](#product) 18 | - [Professional Development](#professional-development) 19 | - [Python](#python) 20 | - [Scala](#scala) 21 | - [Snippets](#snippets) 22 | - [Software Engineering](#software-engineering) 23 | - [Testing](#testing) 24 | - [Unix](#unix) 25 | - [Web Development](#web-development) 26 | 27 | 28 | 29 | ## Computer Science 30 | 31 | - [Algorithms Plus Data Structures Equals Programs](computer-science/algorithms_plus_data_structures_equals_programs.md) 32 | 33 | 34 | ## Data Science 35 | 36 | - [Data Science For Business](data-science/data_science_for_business.md) 37 | - [Natural Language Processing With Python](data-science/natural_language_processing_with_python.md) 38 | 39 | 40 | ## Databases 41 | 42 | - [First Course On Database Systems](databases/first_course_on_database_systems.md) 43 | - [Sql For Web Nerds](databases/sql_for_web_nerds.md) 44 | - [Stanford Db Course](databases/stanford_db_course.md) 45 | 46 | 47 | # Developer Relations 48 | 49 | - [Business Value of Developer Relations](developer-relations/business_value_of_developer_relations.md) 50 | - [Developer Marketing + Relations: The Essential Guide](developer-relations/developer_marketing_and_relations__the_essential_guide.md) 51 | 52 | ## DevOps 53 | 54 | - [Docker](devops/docker.md) 55 | - [Kuberentes](devops/kubernetes) 56 | 57 | 58 | ## Django 59 | 60 | - [Django](django/django.md) 61 | - [Django Testing](django/django_testing.md) 62 | - [Test Drive Web Development With Python](django/test_drive_web_development_with_python.md) 63 | 64 | 65 | ## General 66 | 67 | - [How to Read a Book](general/how_to_read_a_book.md) 68 | - [Learning Things Quickly](general/learning_things_quickly.md) 69 | - [Software Debugging Udacity](general/software_debugging_udacity.md) 70 | 71 | 72 | ## Javascript 73 | 74 | - [Javascript Lanugage Guide](javascript/javascript_lanugage_guide.md) 75 | - [Javascript The Good Parts](javascript/javascript_the_good_parts.md) 76 | - [Typescript Deep Dive](javascript/typescript_deep_dive.md) 77 | 78 | 79 | ## Meetup 80 | 81 | - [Blockchain-In-Healthcare-Summit](meetup/blockchain-in-healthcare-summit.md) 82 | - [Chicago-Startup-Community](meetup/chicago-startup-community.md) 83 | - [Pycon2018](meetup/pycon2018.md) 84 | - [React-Chicago](meetup/react-chicago.md) 85 | 86 | ## Product 87 | 88 | - [Inspire](product/inspire.md) 89 | 90 | ## Professional Development 91 | 92 | - [7 Principles Of Public Speaking](professional-development/7_principles_of_public_speaking.md) 93 | - [Passionate Programmer](professional-development/passionate_programmer.md) 94 | - [Pragmatic Programmer](professional-development/pragmatic_programmer.md) 95 | 96 | 97 | ## Python 98 | 99 | - [Fluent Python](python/fluent_python.md) 100 | - [Logging](python/logging.md) 101 | 102 | 103 | ## Scala 104 | 105 | - [Scala For The Impatient](scala/scala_for_the_impatient.md) 106 | 107 | 108 | ## Snippets 109 | 110 | - [Docker](snippets/docker.md) 111 | - [Domain Modeling](snippets/domain_modeling.md) 112 | - [Git](snippets/git.md) 113 | - [Kubernetes](snippets/kubernetes.md) 114 | - [Python](snippets/python.md) 115 | 116 | 117 | ## Software Engineering 118 | 119 | - [Apprenticeship Patterns](software-engineering/apprenticeship_patterns.md) 120 | - [Clean Code](software-engineering/clean_code.md) 121 | - [Design Patterns](software-engineering/design_patterns.md) 122 | - [Domain Driven Design](software-engineering/domain_driven_design.md) 123 | - [Head First Design Patterns](software-engineering/head_first_design_patterns.md) 124 | - [Philosophy Of Software Design](software-engineering/philosophy_of_software_design.md) 125 | - [Refactoring](software-engineering/refactoring.md) 126 | 127 | 128 | ## Testing 129 | 130 | - [Developer Testing Bootcamp](testing/developer_testing_bootcamp.md) 131 | - [Mock](testing/mock.md) 132 | 133 | 134 | ## Unix 135 | 136 | - [Vim Cheatsheet](unix/vim_cheatsheet.md) 137 | - [Your Unix](unix/your_unix.md) 138 | 139 | 140 | ## Web Development 141 | 142 | - [12 Factor App](web-development/12_factor_app.md) 143 | - [Api](web-development/api.md) 144 | - [Building Microservices](web-development/building_microservices.md) 145 | - [Graphql](web-development/graphql.md) 146 | - [Http](web-development/http.md) 147 | - [Microservices](web-development/microservices.md) 148 | - [Wsgi](web-development/wsgi.md) 149 | -------------------------------------------------------------------------------- /computer-science/algorithms_plus_data_structures_equals_programs.md: -------------------------------------------------------------------------------- 1 | # Algorithms + Data Structures = Programs 2 | 3 | by Niklaus Wirth 4 | 5 | ## Chapter 1: Fundamental Data Stuctures 6 | 7 | > The importance of using a language that offers a convenient set of basic abstractions common to most problems of data processing lies mainly in the area of reliability of the resulting programs 8 | 9 | **cardinality** - number of distinct values 10 | -------------------------------------------------------------------------------- /data-science/data_science_for_business.md: -------------------------------------------------------------------------------- 1 | # Data Science for Business 2 | by Foster Provost and Tom Fawcett 3 | 4 | --- 5 | 6 | ## Chapter 1 7 | 8 | **Data Analytic Thinking** is a problem solving methodology that applies a well-defined approach to extract useful knowledge from data in order to solve business problems. 9 | 10 | ## Chapter 2 11 | 12 | Data mining is a process with well defined stages. We can break our task up into smaller subtasks. Apply a technique the matches best matches what we are trying to do, chapter has a brief discussion on algos. Put sub-results together in presentable way 13 | 14 | CRISP data mining process. List steps. 15 | 1. Business Understanding 16 | 2. Data Understanding 17 | 3. Data Preparation 18 | 4. Modeling 19 | 5. Evaluation 20 | 6. Deployment 21 | 22 | ## Chapter 3: Intro to Predictive Modeling 23 | 24 | One of the common tasks in analytics is finding which attributes correlate with the target of interest. When given a very large dataset, we can select a subset of of the data to analyze it; selecting the subset of attributes that are the most *informative* is a smart way to attack the problem 25 | 26 | **Supervised learning** is when we create a model to describe the relationship between attributes and a predefined target. 27 | 28 | > Common Problem: Supervised Segmentation (Theme) 29 | > 30 | > * Decision Tree... great for intuition 31 | > Pros / Cons later 32 | 33 | * Selecting Informative Attribute (use entropy. Information gain for classification problems. Variance for numeric problems). Ranking helps us: 34 | * Understand data better 35 | * Predict the target 36 | * Reduce size of data to analyze, select subset of attributes 37 | 38 | ## Chapter 4: Fitting a Model to Data 39 | 40 | Tree-based models try to infer structure. There is also another way, take a parameterized model and use our data to fit the parameters. 41 | 42 | Build different kinds of models, can get different kinds of insights from each 43 | 44 | Fitting a model is a generalized approach that can be used for: 45 | 46 | ### Support Vector Machines 47 | 48 | * Used to generate classification boundaries 49 | * Main idea: have wide decision boundaries 50 | * Basic idea is to penalize misclassifications using a hinge loss 51 | * If it's on the other side of the boundary, linear penalty 52 | 53 | ### Regression 54 | 55 | * Predicting a numeric value 56 | * Standard loss function uses sum of squared errors 57 | * Look at application, maybe absolute value makes sense 58 | 59 | ### Logistic Regression 60 | 61 | * Predicting a class 62 | * Uses a logit model to convert ```f(x)``` into a probability 63 | * Sigmoid function 64 | 65 | ## Chapter 5: Overfitting and Its Avoidance 66 | 67 | If we have the flexibility to find patterns in a dataset, we will find them.. We want models to apply not just to the exact training set, but to the general population. 68 | 69 | **Overfitting** is the tendency to tailor models to the training data vs generalizing to previously unseen data points. All algorithms overfit to some extent. Overfitting can be caused by training on the same data as we test against and overtly complex models (number of nodes for trees. number of attributes for regression). 70 | 71 | ### Avoiding Overfitting 72 | 73 | * Holdout (splitting data into training and test sets) 74 | * Cross-validation (splitting data into k partitions. Iterate thru k-1 training and k test sets) 75 | 76 | ### Avoiding Overfitting in Tree-Based Methods 77 | 78 | 1. Stop growing tree before it gets too complex 79 | * Limit tree size by specify min # of instances that must be present in leaf 80 | 2. Grow tree and then prune it back to reduce complexity 81 | * Estimate whether replacing a set of leaves or a branch with a leaf would reduce accuracy. If not, prune. 82 | 83 | ### General Method 84 | 85 | Called **nested holdout procedure** or **nested cross-validation** depending on which we use. 86 | 87 | #### Technique 88 | 89 | * Split training set into subtraining and testing set (aka **validation set**) 90 | * Build models on subtraining subset. Pick best model based on validation subset. **This steps tunes hyperparameters for model complexity** 91 | * Build model with tuned parameters on entire training set (subtraining + validation) 92 | 93 | ### Regularization 94 | 95 | For equations such as logistic regression, complexity can be controlled by choosing the "right" set of attributes. 96 | 97 | Idea: Instead of just optimizing the fit to the data, we optimize an equation that is a combination of fit and simplicity. This is called **regularization** 98 | 99 | We add a penalty to the objective function. 100 | 101 | * **L1 penalty** <=> ridge 102 | * **L2 penalty** <==> lasso 103 | * this ends up zeroing out coefficients 104 | 105 | ## Chapter 6: Similarity, Neighbors, and Clusters 106 | 107 | If two things are similar, they often share other characteristics as well. The closer two objects are in feature space, the more similar they are. Need a way to measure distance and similarity . 108 | 109 | ### Distance Functions 110 | 111 | * weighted contribution based on distance makes sense, not all neighbors should have same weight 112 | 113 | * Euclidean (||L2||) 114 | * Most widely used 115 | * Manhattan (||L1||) 116 | * Street distance 117 | * Jaccard 118 | * Best for comparing objects with sets of characteristics 119 | * Cosine 120 | * Useful in text processing (document analysis) 121 | * Edit Distance / Levenshtein 122 | * min # of edits to convert one string into another 123 | 124 | ### Nearest Neighbors (aka most similar instances) 125 | 126 | * Given a new example 127 | * Scan thru training examples and find the "nearest neighbors" 128 | * Use neighbor's data to predict example's target value 129 | * Task can be classification, probability estimation, or regression 130 | 131 | #### Issues 132 | 133 | * **Intelligibility** - lack of interpretable model 134 | * **Dimensionality** - lots of features, many of them irrelevant 135 | * **Computational Efficiency** - NN is fast because it stores only the instances, no need to build model 136 | 137 | ### Clustering 138 | 139 | * Finding natural groupings in the data. "find groups of objects where the objects within groups are similar, but the objects in different groups are not so similar" 140 | 141 | #### Hierarchical Clustering (dendogram) 142 | 143 | * Formed by starting with each node as its own cluster 144 | * Clusters are merged iteratively (using distance function) until only a single cluster remains 145 | 146 | #### k-means 147 | 148 | * Represent each cluster by its centroid 149 | * Start with k initial cluster centers 150 | * Clusters corresponding to centers are formed by figuring out the closest center to each point 151 | * Cluster centers have shifted 152 | * Recompute new clusters 153 | * Repeat until convergence 154 | 155 | **k-means will find local optimums** so do a bunch of random runs and average 156 | 157 | Runtime is efficient since it only computers distances between each point and the cluster center 158 | 159 | Can be used for EDA. What do the clusters mean? Make this into an interactive IPython widget. 160 | 161 | ### Using Supervised Learning to Generate Cluster Descriptions 162 | 163 | * Do unsupervised learning to find clusters 164 | * Use cluster assignments as class labels 165 | * Run a supervised learning algorithm to generate a classifier for each class/cluster 166 | * Inspect classifier description to get a description of the cluster 167 | 168 | Not guaranteed to work all the time. Think about **intelligibility** 169 | -------------------------------------------------------------------------------- /data-science/images/nltk_pipeline.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alysivji/notes/48e0c5920935fb7156e5b01fc6c3690b932d918e/data-science/images/nltk_pipeline.png -------------------------------------------------------------------------------- /data-science/natural_language_processing_with_python.md: -------------------------------------------------------------------------------- 1 | # Natural Language Processing with Python 2 | 3 | by Steven Bird, Ewan Klein, and Edward Loper 4 | 5 | ## Chapter 1: Language Processing and Python 6 | 7 | ### Computing with Language: Texts and Words 8 | 9 | * `concordance` permits us to see words in context 10 | * `similar` shows other words that appear in similar contexts 11 | * `common_contexts` to see shared context for two or more words 12 | * `dispersion_plot` gives visual of where word is used in document 13 | 14 | ```python 15 | def lexical_diversity(text): 16 | return len(set(text)) / len(text) 17 | ``` 18 | 19 | ### Computing with Language: Simple Statistics 20 | 21 | * `FreqDist()` creates a frequency distribution 22 | * `hapaxes` occur only once 23 | * *collocation* is a sequence of words that occur together unusually often, i.e. red wine 24 | * `bigrams()` or `.collocations` 25 | 26 | ### Automatic Natural Language Understanding 27 | 28 | * **word sense disambiguation** we want to work out which sense of a word was intended in a given context 29 | 30 | ## Chapter 2: Accessing Text Corpora and Lexical Resources 31 | 32 | * **corpus** - large body of text 33 | * **stylistics** - studying systematic differences between genres 34 | 35 | * **conditional frequency distribution** - collection of frequency distributions, each one for a different "condition" 36 | 37 | ```python 38 | def plural(word): 39 | if word.endswith('y'): 40 | return word[:-1] + 'ies' 41 | elif word[-1] in 'sx' or word[-2:] in ['sh', 'ch']: 42 | return word + 'es' 43 | elif word.endswith('an'): 44 | return word[:-2] + 'en' 45 | else: 46 | return word + 's' 47 | ``` 48 | 49 | * **stopwords** are high-frequency words that we want to filter out of a document before further processing 50 | 51 | ```python 52 | from nltk.corpus import stopwords 53 | stopwrods.words('english') 54 | 55 | def content_fraction(text): 56 | stopwords = nltk.corpus.stopwords.words('english') 57 | content = [w for w in text if w.lower() not in stopwords] 58 | return len(content) / len(text) 59 | ``` 60 | 61 | * `nltk.corpus.words.words()` is the `/usr/share/dict/words` file from Unix 62 | * `nltk.corpus.names` is the names corpus 63 | 64 | * WordNet is a semantically-oriented dictionary of English, similar to a traditional thesaurus but with a richer structure 65 | * `from nltk.corpus import wordnet as wn` 66 | 67 | * **hyponyms** - a word of more specific meaning than a general or superordinate term applicable to it 68 | * `nltk.app.wordnet()` 69 | 70 | * meronyms - items to their components 71 | * holonyms - things they are contained in 72 | * entailments - relationships between words 73 | * antonymy - relationships between lemmas 74 | 75 | * use synsets to find semantically related documents 76 | * article about car is also an article about a vechicle 77 | 78 | > Two synsets linked to the same root may have several hypernyms in common. If two synsets share a very specific hypernym — one that is low down in the hypernym hierarchy — they must be closely related. 79 | * `.path_similarity()` 80 | 81 | ## Chapter 3: Processing Raw Text 82 | 83 | `from nltk import word_tokenize` 84 | 85 | > texts found on the web may contain unwanted material, and there may not be an automatic way to remove it. But with a small amount of extra work we can extract the material we need 86 | 87 | ### Natural Language Processing Pipeline 88 | 89 | 90 | 91 | > The Processing Pipeline: We open a URL and read its HTML content, remove the markup and select a slice of characters; this is then tokenized and optionally converted into an nltk.Text object; we can also lowercase all the words and extract the vocabulary. 92 | 93 | * `nltk.Index()` converts this into a useful index 94 | 95 | ```python 96 | def stem(word): 97 | """Stem word using regular expression""" 98 | 99 | regexp = r'^(.*?)(ing|ly|ed|ious|ies|ive|es|s|ment)?$' 100 | stem, suffix = re.findall(regexp, word)[0] 101 | return stem 102 | ``` 103 | 104 | ### Normalizing Text 105 | 106 | > By using lower(), we have normalized the text to lowercase so that the distinction between The and the is ignored. Often we want to go further than this, and strip off any affixes, a task known as stemming. A further step is to make sure that the resulting form is a known word in a dictionary, a task known as lemmatization 107 | 108 | #### Tokenization 109 | 110 | > Tokenization is the task of cutting a string into identifiable linguistic units that constitute a piece of language data. 111 | 112 | * The function `nltk.regexp_tokenize()` is similar to `re.findall()`. However, `nltk.regexp_tokenize()` is more efficient for this task, and avoids the need for special treatment of parentheses 113 | 114 | > Tokenization turns out to be a far more difficult task than you might have expected. No single solution works well across-the-board, and we must decide what counts as a token depending on the application domain. 115 | > 116 | > When developing a tokenizer it helps to have access to raw text which has been manually tokenized, in order to compare the output of your tokenizer with high-quality (or "gold-standard") tokens. 117 | > 118 | > A final issue for tokenization is the presence of contractions, such as didn't. If we are analyzing the meaning of a sentence, it would probably be more useful to normalize this form to two separate forms: did and n't (or not). We can do this work with the help of a lookup table. 119 | 120 | #### Segmentation 121 | 122 | A more general form of tokenization, i.e. sentence segmentation. 123 | 124 | > Sentence segmentation is difficult because period is used to mark abbreviations, and some periods simultaneously mark an abbreviation and terminate a sentence, as often happens with acronyms like U.S.A. 125 | 126 | #### Stemming 127 | 128 | ```python 129 | porter = nltk.PorterStemmer() 130 | lancaster = nltk.LancasterStemmer() 131 | ``` 132 | 133 | > Stemming is not a well-defined process, and we typically pick the stemmer that best suits the application we have in mind. The Porter Stemmer is a good choice if you are indexing some texts and want to support search using alternative forms of words 134 | 135 | #### Lemmatization 136 | 137 | > The WordNet lemmatizer is a good choice if you want to compile the vocabulary of some texts and want a list of valid lemmas (or lexicon headwords). 138 | 139 | ### Summary 140 | 141 | * Texts found on the web may contain unwanted material (such as headers, footers, markup), that need to be removed before we do any linguistic processing. 142 | * Tokenization is the segmentation of a text into basic units — or tokens — such as words and punctuation. Tokenization based on whitespace is inadequate for many applications because it bundles punctuation together with words. NLTK provides an off-the-shelf tokenizer nltk.word_tokenize(). 143 | * Lemmatization is a process that maps the various forms of a word (such as appeared, appears) to the canonical or citation form of the word, also known as the lexeme or lemma (e.g. appear). 144 | 145 | ## Chapter 4: Writing Structured Programs 146 | 147 | > To copy the items from a list foo to a new list bar, you can write `bar = foo[:]`. This copies the object references inside the list. To copy a structure without copying any object references, use `copy.deepcopy()`. 148 | 149 | * `is` verfies object identity 150 | * `==` verfies object value 151 | 152 | > In the condition part of an `if` statement, a nonempty string or list is evaluated as true, while an empty string or list evaluates as false. 153 | 154 | > lists are **mutable**, while tuples are **immutable**. In other words, lists can be modified, while tuples cannot. 155 | 156 | * _procedural_ - dictating the machine operations step by step 157 | * _declarative_ - describe a result and get it via a black box 158 | (via [StackOverflow](https://stackoverflow.com/questions/1619834/difference-between-declarative-and-procedural-programming)) 159 | 160 | * __LGB rule__ of name resolution 161 | * local 162 | * global 163 | * built-in 164 | 165 | Note: A function can enable access to a global variable using the global declaration. 166 | -------------------------------------------------------------------------------- /databases/first_course_on_database_systems.md: -------------------------------------------------------------------------------- 1 | # A First Course on Database Systems 2 | 3 | By Jeffrey D. Ullman and Jennifer Widom 4 | 5 | ## Chapter 1: The World of Database Systems 6 | 7 | * A Database Management System (DBMS) is a powerful tool for creating and managing large amounts of data efficiently and allowing it to persist over long periods of time, safely 8 | 9 | DBMS is expected to: 10 | 1. Allow users to create new databases and specify their schemas using a data-definition language 11 | 2. Give users ability to query and modify data using a data manipulation (or query) language 12 | 3. Support storage of large amounts of data over a long period of time 13 | 4. Enable durability -- recovery of the database in the face of errors and failures 14 | 5. Control access to data from many users without allowing unexpected interactions amongst users and without actions on data to be performed partially 15 | 16 | * Query is parsed and optimized by a **query compiler**. The resulting query plan, or sequence of actions the DBMS will perform to answer the query, is passed to the execution engine. 17 | 18 | * **transactions** are units that must be executed atomically and in isolation from one another 19 | 20 | ### ACID Property of Transactions 21 | 22 | * **A**tomicity - all or nothing execution of transaction 23 | * **C**onsistency - transactions are expected to preserve consistency of database 24 | * **I**solation - transaction must appear to execute as if no other transaction is executing at the same time 25 | * **D**urability - effect of transaction on database must never be lost once the transaction is completed 26 | 27 | ## Chapter 2: The Relational Model of Data 28 | 29 | * Data model is a notation for describing data or information. Consists of three parts: 30 | * Structure of Data 31 | * Operations on Data 32 | * Constraints on Data 33 | 34 | * Relational data is stored in tables (rows and columns), although the underlying implementation maybe something complex 35 | * Semistructured data resembles trees or graphs, hierachically nested tags 36 | 37 | * semistructured models have more flexibility than relations, but since databases are large, efficiency of access to data and efficiency of modifications are of great importance. 38 | * limitations become features, we optimize for the one use case 39 | * if our data is relational, half the work is already done for us 40 | 41 | ### Relation (aka Table) 42 | 43 | * Attributes - named columns of a relation 44 | * Schemas - relation name + set of attributes 45 | * Tuples - rows of a relation 46 | * Domain - data type 47 | * Relation - sets of tuples 48 | 49 | * We expect tuples to change, but the schema is less likely to change 50 | * schema changes ('migrations') are expensive 51 | * if we add an attribute, we have to generate appropriate values for new components in the existing tuples 52 | 53 | * Key - set of attributes form a key if we do not allow two tuples in a relation instance to ahve the same values in all the attributes of the key 54 | * Databases use artifical keys to not make any assumptions about attribute values 55 | 56 | ### Defining a Relation Schema in SQL 57 | 58 | * `CREATE TABLE` 59 | * `ALTER TABLE` then `ADD` or `DROP [attribute_name]` 60 | 61 | * Two tuples in R cannot agree on all of the attributes in set S, unless one of them is NULL. Any attempt to insert or update a tuple that violates this rule causes the DBMS to reject the action that caused the violation. 62 | 63 | * if `PRIMARY_KEY` is used, then attributes in S are not allowed to have `NULL` as a value for their components 64 | 65 | ### Algebraic Query Language 66 | 67 | > By limiting what we can say or do in our query language, we get two huge rewards - ease of programming and the ability of the compiler to produce highly optimized code 68 | 69 | > If all we could do was to write single operations on one or two relations as queries, then relational algebra would not be nearly as useful as it is. However, relational algebra, like all algebras, allows us to form expressions of arbitrary complexity by applying operations to the result of other operations. 70 | 71 | > One can construct expressions of relational algebra by applying operators to subexpressions, using parentheses when necessary to indicate grouping of operands. It is also possible to represent expressions as expression trees; the latter often are easier for us to read, although they are less convenient as a machine-readable notation. 72 | 73 | ### Constraints on Relations 74 | 75 | > A common kind of constraint, called a referential integrity constraint, asserts that a value appearing in one context also appears in another, related context. 76 | -------------------------------------------------------------------------------- /databases/sql_for_web_nerds.md: -------------------------------------------------------------------------------- 1 | # [SQL for Web Nerds](http://philip.greenspun.com/sql/index.html) 2 | 3 | by Philip Greenspun 4 | 5 | ## [Chapter 1: Introduction](http://philip.greenspun.com/sql/introduction.html) 6 | 7 | ### ACID Test 8 | 9 | Transaction processing requires: 10 | 11 | * __A__tomicity - transaction is either committed or rolled back. All changes take efffect, or none do. 12 | 13 | * __C__onsistency - database is always transformed from valid state to valid state. Transaction is legal only if it meets all relation integrity contstraints; if not legal it is rolled back 14 | 15 | * __I__solation - each transaction is invisible to other transactions until it is complete 16 | 17 | * __D__urability - once a transaction is complete, the results are permanent 18 | 19 | > From an application programmer's point of view, the biggest innovation in the relational database is that one uses a declarative query language, SQL (an acronym for Structured Query Language and pronounced "ess-cue-el" or "sequel"). Most computer languages are procedural. The programmer tells the computer what to do, step by step, specifying a procedure. In SQL, the programmer says "I want data that meet the following criteria" and the RDBMS query planner figures out how to get it. There are two advantages to using a declarative language. The first is that the queries no longer depend on the data representation. The RDBMS is free to store data however it wants. The second is increased software reliability. It is much harder to have "a little bug" in an SQL query than in a procedural program. Generally it either describes the data that you want and works all the time or it completely fails in an obvious way. 20 | 21 | ### Declarative Query Language 22 | 23 | > In SQL, the programmer says "I want data that meet the following criteria" and the RDBMS query planner figures out how to get it. There are two advantages to using a declarative language. The first is that the queries no longer depend on the data representation. The RDBMS is free to store data however it wants. The second is increased software reliability. It is much harder to have "a little bug" in an SQL query than in a procedural program. Generally it either describes the data that you want and works all the time or it completely fails in an obvious way. 24 | 25 | ### Database Basics 26 | 27 | > A relational database is a big spreadsheet that several people can update simultaneously. 28 | > 29 | > Each table in the database is one spreadsheet. You tell the RDBMS how many columns each row has. For example, in our mailing list database, the table has two columns: name and email. Each entry in the database consists of one row in this table. An RDBMS is more restrictive than a spreadsheet in that all the data in one column must be of the same type, e.g., integer, decimal, character string, or date. Another difference between a spreadsheet and an RDBMS is that the rows in an RDBMS are not ordered. 30 | 31 | ### Data Manipulation Language 32 | 33 | * SQL 34 | 35 | ### Relational DB vs Object DB 36 | 37 | > Object databases bring back some of the bad features of 1960s pre-relational database management systems. The programmer has to know a lot about the details of data storage. If you know the identities of the objects you're interested in, then the query is fast and simple. But it turns out that most database users don't care about object identities; they care about object attributes. Relational databases tend to be faster and better at coughing up aggregations based on attributes. The critical difference between RDBMS and ODBMS is the extent to which the programmer is constrained in interacting with the data. With an RDBMS the application program--written in a procedural language such as C, COBOL, Fortran, Perl, or Tcl--can have all kinds of catastrophic bugs. However, these bugs generally won't affect the information in the database because all communication with the RDBMS is constrained through SQL statements. With an ODBMS, the application program is directly writing slots in objects stored in the database. A bug in the application program may translate directly into corruption of the database, one of an organization's most valuable assets. 38 | 39 | --- 40 | 41 | ## [Chapter 2: Data Modeling](http://philip.greenspun.com/sql/data-modeling.html) 42 | 43 | * __data modeling__ is the process of defining tables; we are defining: 44 | 45 | > * what elements of the data you will store 46 | > * how large each element can be 47 | > * what kind of information each element can contain 48 | > * what elements may be left blank 49 | > * which elements are constrained to a fixed range 50 | > * whether and how various tables are to be linked 51 | 52 | * When building the data model you must affirmatively decide whether a NULL value will be permitted for a column and, if so, what it means. 53 | -------------------------------------------------------------------------------- /databases/stanford_db_course.md: -------------------------------------------------------------------------------- 1 | # Databases 2 | 3 | ## Introduction to Databases (DB1) 4 | 5 | Table == relation 6 | 7 | ## Relational Algebra (DB4) 8 | 9 | * when a DBMS processes queries, the first thing that happens to a SQL query is that it gets translated into relational algebra or a very similar internal representation 10 | 11 | * An important job of the query “optimizer” dis­cussed is to replace one expression of relational algebra by an equivalent expression that is more efficiently evaluated. 12 | 13 | ## SQL (DB5) 14 | 15 | * 16 | -------------------------------------------------------------------------------- /developer-relations/business_value_of_developer_relations.md: -------------------------------------------------------------------------------- 1 | # Business Value of Developer Relations 2 | 3 | [TODO] 4 | -------------------------------------------------------------------------------- /devops/docker.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alysivji/notes/48e0c5920935fb7156e5b01fc6c3690b932d918e/devops/docker.md -------------------------------------------------------------------------------- /devops/kubernetes/create-cluster-using-kubeadm.md: -------------------------------------------------------------------------------- 1 | # How To Create a Kubernetes Cluster Using Kubeadm on Ubuntu 18.04 2 | 3 | [DO Article](https://www.digitalocean.com/community/tutorials/how-to-create-a-kubernetes-cluster-using-kubeadm-on-ubuntu-18-04) 4 | 5 | Kubeadm automates the installation and configuration of Kubernetes components such as the API server, Controller Manager, and Kube DNS. It does not, however, create users or handle the installation of operating-system-level dependencies and their configuration. Use Ansible for configuration management. 6 | -------------------------------------------------------------------------------- /devops/kubernetes/gke.md: -------------------------------------------------------------------------------- 1 | # Google Kubernetes Engine 2 | 3 | ## Create Docker Image 4 | 5 | ```console 6 | docker build -t myapp . 7 | docker tag myapp gcr.io/netes-sandbox/myapp:blue 8 | docker push gcr.io/netes-sandbox/myapp:blue 9 | ``` 10 | 11 | ## Set up GKE 12 | 13 | ```console 14 | gcloud config set project netes-sandbox 15 | gcloud container clusters get-credentials cluster-1 --zone=us-central1-c 16 | ```` 17 | 18 | ## Networking 19 | 20 | ```console 21 | k port-forward nginx 8080:80 22 | ``` 23 | -------------------------------------------------------------------------------- /devops/kubernetes/introduction.md: -------------------------------------------------------------------------------- 1 | # An Introduction to Kubernetes 2 | 3 | [DO Article](https://www.digitalocean.com/community/tutorials/an-introduction-to-kubernetes) 4 | 5 | Kubernetes, at its basic level, is a system for running and coordinating containerized applications across a cluster of machines. It is a platform designed to completely manage the life cycle of containerized applications and services using methods that provide predictability, scalability, and high availability. 6 | 7 | ## Kubernetes Architecture 8 | 9 | At its base, Kubernetes brings together individual physical or virtual machines into a cluster using a shared network to communicate between each server. This cluster is the physical platform where all Kubernetes components, capabilities, and workloads are configured. 10 | 11 | ### Overview 12 | 13 | - k8s runs applications and services in containers 14 | - underlying components make sure the desired state of the applications matches the actual state of the cluster 15 | - to start an app or service, submit a declarative plan in JSON or YAML 16 | 17 | ### Master Server 18 | 19 | One server is the master server. This server acts as a gateway and brain for the cluster by exposing an API for users and clients, health checking other servers, deciding how best to split up and assign work (known as “scheduling”), and orchestrating communication between other components. The master server acts as the primary point of contact with the cluster and is responsible for most of the centralized logic Kubernetes provides. 20 | 21 | > components on the master server work together to accept user requests, determine the best ways to schedule workload containers, authenticate clients and nodes, adjust cluster-wide networking, and manage scaling and health checking responsibilities. 22 | 23 | **etcd** -- distributed key-value store. K8s stores configuration data here 24 | **kube-apiserver** - main management point of the entire cluster as it allows a user to configure Kubernetes’ workloads and organizational units 25 | **kubectl** - client that can interact with the k8s cluster from a local computer 26 | **kube-controller-manager** - manages different controllers that regulate the state of the cluster, manage workload life cycles, and perform routine tasks. Watches `etcd` for changes 27 | **kube-scheduler** - process to assign workloads to specific nodes in the cluster 28 | **cloud-controller-manager** - interacts with various infrastructure providers to understand and manage the state of resources in the cluster 29 | 30 | ### Nodes 31 | 32 | Responsible for accepting and running workloads using local and external resources. Receives work instructions from master server and does whatever it needs to to route and forward traffic appropriately 33 | 34 | **container runtime** - responsible for starting and managing containers 35 | **kubelet** - esponsible for relaying information to and from the control plane services, as well as interacting with the `etcd` store to read configuration details or write new values. Recieves manifest and assumes responsibility for maintaining the state of the work on the node server 36 | **manifest** - defines workload and operating parameters 37 | **kube-proxy** - manage individual host subnetting and makes services available to other components. The process forwards requests to the correct containers, can do primitive load balancing, and is generally responsible for making sure the networking environment is predictable and accessible, but isolated where appropriate 38 | 39 | Worker nodes are the servers where your workloads (i.e. containerized applications and services) will run. A worker will continue to run your workload once they’re assigned to it, even if the master goes down once scheduling is complete 40 | 41 | ### Kubernetes Objects and Workloads 42 | 43 | Kubernetes uses additional layers of abstraction over the container interface to provide scaling, resiliency, and life cycle management features. Instead of managing containers directly, users define and interact with instances composed of various primitives provided by the Kubernetes object model. 44 | 45 | #### Pods 46 | 47 | Most basic unit that Kubernetes deals with. 48 | 49 | A pod generally represents one or more containers that should be controlled as a single application. Pods consist of containers that operate closely together, share a life cycle, and should always be scheduled on the same node. They are managed entirely as a unit and share their environment, volumes, and IP space. In spite of their containerized implementation, you should generally think of pods as a single, monolithic application to best conceptualize how the cluster will manage the pod’s resources and scheduling. 50 | 51 | Horizontal scaling is generally discouraged on the pod level because there are other higher level objects more suited for the task. Users are encouraged to work with higher level objects that use pods or pod templates as base components but implement additional functionality. 52 | 53 | #### Replication Controllers and Replication Sets 54 | 55 | Often, when working with Kubernetes, rather than working with single pods, you will instead be managing groups of identical, replicated pods. These are created from pod templates and can be horizontally scaled by controllers known as replication controllers and replication sets. 56 | 57 | **replication controller** - object that defines a pod template and control parameters to scale identical replicas of a pod horizontally by increasing or decreasing the number of running copies. Responsible for ensuring that the number of pods deployed in the cluster matches the number of pods in its configuration. Can also perform rolling updates to roll over a set of pods to a new version one by one, minimizing the impact on application availability. 58 | 59 | **replication sets** - An iteration on the replication controller design with greater flexibility in how the controller identifies the pods it is meant to manage. Replication sets are beginning to replace replication controllers because of their greater replica selection capabilities, but they are not able to do rolling updates to cycle backends to a new version like replication controllers can. Instead, replication sets are meant to be used inside of additional, higher level units that provide that functionality 60 | 61 | #### Deployments 62 | 63 | **deployments** - use replication sets as a building block, adding flexible life cycle management functionality to the mix. Deployments are a high level object designed to ease the life cycle management of replicated pods 64 | 65 | #### Stateful Sets 66 | 67 | **stateful sets** - specialized pod controllers that offer ordering and uniqueness guarantees. Primarily, these are used to have more fine-grained control when you have special requirements related to deployment ordering, persistent data, or stable networking. For instance, stateful sets are often associated with data-oriented applications, like databases, which need access to the same volumes even if rescheduled to a new node. 68 | 69 | Stateful sets provide a stable networking identifier by creating a unique, number-based name for each pod that will persist even if the pod needs to be moved to another node. Likewise, persistent storage volumes can be transferred with a pod when rescheduling is necessary. The volumes persist even after the pod has been deleted to prevent accidental data loss. 70 | 71 | When deploying or adjusting scale, stateful sets perform operations according to the numbered identifier in their name. This gives greater predictability and control over the order of execution, which can be useful in some cases. 72 | 73 | #### Daemon Sets 74 | 75 | **daemon sets** - specialized form of pod controller that run a copy of a pod on each node in the cluster (or a subset, if specified). This is most often useful when deploying pods that help perform maintenance and provide services for the nodes themselves 76 | 77 | For instance, collecting and forwarding logs, aggregating metrics, and running services that increase the capabilities of the node itself are popular candidates for daemon sets. Because daemon sets often provide fundamental services and are needed throughout the fleet, they can bypass pod scheduling restrictions that prevent other controllers from assigning pods to certain hosts. As an example, because of its unique responsibilities, the master server is frequently configured to be unavailable for normal pod scheduling, but daemon sets have the ability to override the restriction on a pod-by-pod basis to make sure essential services are running. 78 | 79 | #### Jobs and CRON jobs 80 | 81 | **jobs** provide a more task-based workflow where the running containers are expected to exit successfully after some time once they have completed their work. Jobs are useful if you need to perform one-off or batch processing instead of running a continuous service 82 | 83 | **cron jobs** provide an interface to run jobs with a scheduling component. Cron jobs can be used to schedule a job to execute in the future or on a regular, reoccurring basis. Kubernetes cron jobs are basically a reimplementation of the classic cron behavior, using the cluster as a platform instead of a single operating system 84 | 85 | ### Other Kubernetes Components 86 | 87 | #### Services 88 | 89 | **service** is a component that acts as a basic internal load balancer and ambassador for pods. A service groups together logical collections of pods that perform the same function to present them as a single entity 90 | 91 | Any time you need to provide access to one or more pods to another application or to external consumers, you should configure a service. For instance, if you have a set of pods running web servers that should be accessible from the internet, a service will provide the necessary abstraction. Likewise, if your web servers need to store and retrieve data, you would want to configure an internal service to give them access to your database pods. 92 | 93 | Although services, by default, are only available using an internally routable IP address, they can be made available outside of the cluster by choosing one of several strategies. 94 | 95 | **NodePort** works by opening a static port on each node’s external networking interface. Traffic to the external port will be routed automatically to the appropriate pods using an internal cluster IP service. 96 | 97 | **LoadBalancer** service type creates an external load balancer to route to the service using a cloud provider’s Kubernetes load balancer integration. The cloud controller manager will create the appropriate resource and configure it using the internal service service addresses 98 | 99 | #### Volumes and Persistent Volumes 100 | 101 | Kubernetes uses its own volumes abstraction that allows data to be shared by all containers within a pod and remain available until the pod is terminated. This means that tightly coupled pods can easily share files without complex external mechanisms. Container failures within the pod will not affect access to the shared files. Once the pod is terminated, the shared volume is destroyed, so it is not a good solution for truly persistent data. 102 | 103 | **Persistent volumes** are a mechanism for abstracting more robust storage that is not tied to the pod life cycle. Instead, they allow administrators to configure storage resources for the cluster that users can request and claim for the pods they are running. Once a pod is done with a persistent volume, the volume’s reclamation policy determines whether the volume is kept around until manually deleted or removed along with the data immediately. Persistent data can be used to guard against node-based failures and to allocate greater amounts of storage than is available locally. 104 | 105 | #### Labels and Annotations 106 | 107 | **label** - semantic tag that can be attached to Kubernetes objects to mark them as a part of a group. These can then be selected for when targeting different instances for management or routing. For instance, each of the controller-based objects use labels to identify the pods that they should operate on. Services use labels to understand the backend pods they should route requests to 108 | -------------------------------------------------------------------------------- /devops/kubernetes/kubernetes-deep-dive.md: -------------------------------------------------------------------------------- 1 | # Deep Dive 2 | 3 | ## Services 4 | 5 | ### Load Balancer 6 | 7 | - don't want to use the cloud native load balancer 8 | - how can i do it via DNS configuration? 9 | - NodePort? 10 | 11 | ### Service Network 12 | 13 | - kube-proxy `IPVS mode` 14 | 15 | ## RBAC and Admission Control 16 | 17 | - `ClusterRole` and `RoleBinding` 18 | - `kubectl config set-credentials --client-certificate --client-key` 19 | - PodSecurityPolicy 20 | 21 | ### Jobs 22 | 23 | - jobs 24 | - cronjobs 25 | 26 | ### Autoscaling 27 | 28 | - Pod resoruce requriests and limits 29 | - ResourceQuotas 30 | 31 | ### CustomResourceDefinition 32 | 33 | - create a new kind 34 | 35 | ## Servicee Messhes 36 | 37 | - Severless: OpenFAAS, KNative 38 | - Service Meshes: Istio 39 | - Prometheus 40 | - Security 41 | -------------------------------------------------------------------------------- /devops/kubernetes/notes.md: -------------------------------------------------------------------------------- 1 | # Notes 2 | 3 | ## kind 4 | 5 | oh yah 6 | 7 | ## Kubectl 8 | 9 | ```bash 10 | k get resource -o yaml 11 | 12 | kubectl diff -f configs/ 13 | kubectl apply -f configs/ 14 | kubectl delete -f configs/ 15 | 16 | # https://kubernetes.io/docs/concepts/overview/working-with-objects/field-selectors/ 17 | kubectl get pods --field-selector "" 18 | k delete pod -l app=mysql 19 | 20 | # https://kubernetes.io/docs/concepts/architecture/nodes/ 21 | kubectl describe node 22 | 23 | # https://kubernetes.io/docs/concepts/workloads/controllers/deployment/ 24 | kubectl get deployments 25 | kubectl rollout status deployment.v1.apps/nginx-deployment 26 | 27 | # to forward ports (find a better way) 28 | kubectl port-forward --namespace default svc/airflow-1584399113 8080:8080 29 | 30 | # secrets 31 | kubectl create secret generic mysql --from-literal=password=MYSECRETPASS 32 | ``` 33 | 34 | ## Pod 35 | 36 | - Atomic unit of kubernetes 37 | - Lifecycle: `Pending` -> `Running` -> `Succeeded`; `Pending` -> `Failed`; `Unknown` 38 | 39 | ## Services 40 | 41 | > When you create a Service, it creates a corresponding DNS entry. This entry is of the form ..svc.cluster.local, which means that if a container just uses , it will resolve to the service which is local to a namespace. This is useful for using the same configuration across multiple namespaces such as Development, Staging and Production. If you want to reach across namespaces, you need to use the fully qualified domain name (FQDN). 42 | 43 | - map service to endpoint to set up external services 44 | - use `ExternalName` Service 45 | - roundrobin IP addresses 46 | - `ClusterIP` service lets you use names like mysql:3306 47 | 48 | ## Logging 49 | 50 | [k8s docs](https://kubernetes.io/docs/concepts/cluster-administration/logging) 51 | 52 | - send to `stderr` or `stdout` 53 | 54 | ### Ingress 55 | 56 | https://www.youtube.com/watch?v=VicH6KojwCI 57 | 58 | - customizable way to expose traffic than a cloud load balancer 59 | - ingress object configures access to services from outside the cluster 60 | - designed for `HTTP` and `HTTPS` services 61 | - can provide SSL, name-based, and path-based routing 62 | 63 | - controller routes traffic to service based on ingress definition 64 | - usually fronted by a cloud load balancer 65 | - consolidate routing through a single resource 66 | 67 | ## Database Migrations 68 | 69 | https://cloudnativedevopsblog.com/posts/database-migrations/ 70 | 71 | - create a job and hook it up to helm to run before our deployment 72 | 73 | ## Secrets 74 | 75 | - consumed as environment variables or volumes 76 | - secrets are encoded, not encrypted by default 77 | 78 | ## ConfigMaps 79 | 80 | - decouple configuration from image content 81 | - created from files, directories, or literal values 82 | - values can be referenced as environment variables 83 | - can be mounted as a volume 84 | 85 | ## Deployment Patterns 86 | 87 | - rolling updates -- gradually replace pods with an updated spec 88 | - canary deployments -- deploy, send a small subset of traffic to canary 89 | - blue-green deployments -- maintain two versions of your application deployment: switch traffic from `blue` to `green` by updating selector in service 90 | 91 | ## Helm 92 | 93 | - use helm to update pods when config map changes 94 | - create a simple functional test 95 | 96 | ## Best Practices 97 | 98 | - [Recommended Labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/) 99 | - [ephemeral containers for debugging](https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/) 100 | - [daemonset](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) for logging 101 | - [CronJob](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/) to replace crontab 102 | - [kubeconfig](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/) to organize information about clusters, usres, namespaces, etc 103 | - [metrics](https://kubernetes.io/docs/concepts/cluster-administration/monitoring/) 104 | - [namespaces](https://kubernetes.io/docs/tasks/administer-cluster/namespaces-walkthrough/) to separate `production` and `staging` 105 | - horizontal pod autoscaler 106 | - Role-Based Access Control 107 | - namespaces == define a virtual cluster to isolate resources for multiple teams or projects (`default` and `kube-system`) 108 | - network policies (defining ingress and egress) using selectors 109 | - [Operators](https://www.youtube.com/watch?v=TKpQNKWRWHo) 110 | - [kopf](https://github.com/zalando-incubator/kopf) 111 | -------------------------------------------------------------------------------- /devops/kubernetes/todo.md: -------------------------------------------------------------------------------- 1 | # Learn 2 | 3 | - [ ] https://github.com/helm/monocular 4 | - what do we do when we deploy an app? do we update versions somewhere? 5 | -------------------------------------------------------------------------------- /devops/kubernetes/yaml.md: -------------------------------------------------------------------------------- 1 | # YAML Specification 2 | 3 | https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects/ 4 | 5 | > Every Kubernetes object includes two nested object fields that govern the object’s configuration: the object spec and the object status. The spec, which you must provide, describes your desired state for the object–the characteristics that you want the object to have. The status describes the actual state of the object, and is supplied and updated by the Kubernetes system. At any given time, the Kubernetes Control Plane actively manages an object’s actual state to match the desired state you supplied. 6 | 7 | - apiVersion - Which version of the Kubernetes API you’re using to create this object 8 | - kind - What kind of object you want to create 9 | - metadata - Data that helps uniquely identify the object, including a name string, UID, and optional namespace 10 | - spec - What state you desire for the object 11 | 12 | Use the API reference to find out the `spec` for each format https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/ 13 | 14 | ## Links 15 | 16 | - [writing a deployment spec](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#writing-a-deployment-spec) 17 | -------------------------------------------------------------------------------- /django/django.md: -------------------------------------------------------------------------------- 1 | # Django Cheatsheet 2 | 3 | --- 4 | 5 | ## Admin 6 | 7 | * Starting dev server 8 | ```console 9 | $ python manage.py runserver 10 | ``` 11 | 12 | 13 | * Create app 14 | ```console 15 | $ python manage.py startapp polls 16 | ``` 17 | 18 | * Can create custom admin site 19 | 20 | --- 21 | 22 | ## Models 23 | 24 | [models.Model reference](https://docs.djangoproject.com/en/1.11/ref/models/instances/) 25 | 26 | **Contains business logic**. 27 | 28 | > [A model is the single, definitive source of information about your data. It contains the essential fields and behaviors of the data you’re storing.](https://docs.djangoproject.com/en/1.11/topics/db/models/) 29 | 30 | * Subclass of [```django.db.models.Model```](https://docs.djangoproject.com/en/1.11/ref/models/instances/#django.db.models.Model) 31 | 32 | ### Relationships 33 | 34 | #### [Many-to-one](https://docs.djangoproject.com/en/1.11/topics/db/models/#many-to-one-relationships) 35 | 36 | [Example](https://docs.djangoproject.com/en/1.11/topics/db/examples/many_to_one/) 37 | 38 | * uses [```django.db.models.ForeignKey```](https://docs.djangoproject.com/en/1.11/ref/models/fields/#django.db.models.ForeignKey) 39 | * [ForeignKey docs](https://docs.djangoproject.com/en/1.11/ref/models/fields/#foreign-key-arguments) 40 | * [Following relationships backwards](https://docs.djangoproject.com/en/1.11/topics/db/queries/#backwards-related-objects) 41 | 42 | #### Many-to-many relationships 43 | 44 | * Uses [```ManyToManyField```](https://docs.djangoproject.com/en/1.11/ref/models/fields/#django.db.models.ManyToManyField) 45 | * Doesn't matter which model you put it in 46 | * If we need to have this relationship hold extra information (Players and Teams is many-to-many, but we want to know when players were on a team), we can create another Model to hold the details on the relationship ([docs](https://docs.djangoproject.com/en/1.11/topics/db/models/#extra-fields-on-many-to-many-relationships)) 47 | * Use ```through``` argument to associate 48 | 49 | #### One-to-one relationships 50 | 51 | * Uses [```OneToOneField```](https://docs.djangoproject.com/en/1.11/ref/models/fields/#django.db.models.OneToOneField) 52 | * Useful on the primary key of an object when one object "extends" another object 53 | * Have a Place model (Phone nubmer, address, etc) and a Resturant model. Restaurant is a Place (extends Place), so we can make a 1-1 relationship 54 | 55 | ### Meta options 56 | 57 | > Model metadata is “anything that’s not a field”, such as ordering options (ordering), database table name (db_table), or human-readable singular and plural names (verbose_name and verbose_name_plural). None are required, and adding class Meta to a model is completely optional. 58 | 59 | [List of possible Meta options](https://docs.djangoproject.com/en/1.11/ref/models/options/) 60 | 61 | ### Model Attributes 62 | 63 | **Manager** is interface thru which queries are performed. If no custom Manager, default name is objects 64 | 65 | ### Model Methods 66 | 67 | 68 | 69 | 70 | --- 71 | 72 | ## Migrations 73 | 74 | [docs](https://docs.djangoproject.com/en/1.11/topics/migrations/) 75 | 76 | Steps for making model changes: 77 | 78 | 1. Change models (```models.py```) 79 | 80 | 2. Create migrations 81 | ```console 82 | $ python manage.py makemigrations polls 83 | ``` 84 | 85 | 3. Run migrations 86 | ```console 87 | $ python manage.py migrate 88 | ``` 89 | 90 | * Can have [Data Migrations](https://simpleisbetterthancomplex.com/tutorial/2017/09/26/how-to-create-django-data-migrations.html) 91 | 92 | --- 93 | 94 | ## Querying 95 | 96 | * [Making queries](https://docs.djangoproject.com/en/1.11/topics/db/queries/) 97 | * [Field Lookups](https://docs.djangoproject.com/en/1.11/topics/db/queries/#field-lookups-intro) 98 | * [Related objects](https://docs.djangoproject.com/en/1.11/ref/models/relations/) aka field mappings 99 | 100 | * [Query expressions](https://docs.djangoproject.com/en/1.11/ref/models/expressions/#avoiding-race-conditions-using-f) 101 | 102 | ### Creating Objects 103 | 104 | Create a object. ```object.save()``` 105 | 106 | ### Saving Changes to Objects 107 | 108 | Change object attribute. ```object.save()``` 109 | 110 | ### Retrieving Objects 111 | 112 | * Construction a ```QuerySet```, collection of objects from your database 113 | * Get a query set by using your model's Manager (```.objects()```) 114 | 115 | #### Manager methods 116 | 117 | * ```.all()``` 118 | * ```.filter()``` 119 | * ```.exclude()``` 120 | * ```.get()``` *returns one object* 121 | 122 | * [Field lookups](https://docs.djangoproject.com/en/1.11/topics/db/queries/#field-lookups) aka ```WHERE``` clause 123 | 124 | #### ```QuerySet``` 125 | 126 | * [list of methods](https://docs.djangoproject.com/en/1.11/ref/models/querysets/#queryset-api) 127 | * Can change together filters 128 | * Lazy evaluation 129 | 130 | #### Queries across many relationships 131 | 132 | * ```Model.objects.filter(relatedmodel__field__contains='Value')``` to get all ```Model``` objects that have a ```field``` which contains ```Value``` in the related model 133 | 134 | #### Following relationships 'backwards' 135 | 136 | * If a model has a ```ForeignKey```, instances of the foreign-key model will have access to a ```Manager``` that returns all instances of the first model. By default, this ```Manager``` is named ```FOO_set```, where ```FOO``` is the source model name, lowercased. 137 | 138 | ##### Many-to-many relationship 139 | 140 | * The model that defines the ```ManyToManyField``` uses the attribute name of that field itself, whereas the “reverse” model uses the lowercased model name of the original model, plus ```_set``` (just like reverse one-to-many relationships). 141 | 142 | * 143 | 144 | 145 | --- 146 | 147 | ## Views 148 | 149 | * [```django.urls``` docs](https://docs.djangoproject.com/en/1.11/ref/urlresolvers/#module-django.urls) 150 | * [dynamic urls in templates](https://docs.djangoproject.com/en/1.11/intro/tutorial03/#removing-hardcoded-urls-in-templates) 151 | 152 | * [namespacing URL names](https://docs.djangoproject.com/en/1.11/intro/tutorial03/#namespacing-url-names) 153 | 154 | * [templates docs](https://docs.djangoproject.com/en/1.11/topics/templates/) 155 | 156 | ### Generic Views 157 | 158 | Lots of our views represent basic Web dev: getting data from the db according to the parameter that was passed, loading a template, and returning rendered template 159 | 160 | Shortcut: **generic views system** 161 | 162 | To convert: 163 | 1. Convert the URLconf 164 | 1. Delete old, unneeded views 165 | 1. Add views based on Django's generic views 166 | 167 | 168 | * [Generic display views](https://docs.djangoproject.com/en/1.11/ref/class-based-views/generic-display/#django.views.generic.list.ListView) 169 | * [Class-based views](https://docs.djangoproject.com/en/1.11/topics/class-based-views/) 170 | 171 | ### Function-Based and Class-Based Views 172 | 173 | * 174 | 175 | --- 176 | 177 | ## Testing 178 | 179 | * [Tutorial 5](https://docs.djangoproject.com/en/1.11/intro/tutorial05/) 180 | * [Django test Client docs](https://docs.djangoproject.com/en/1.11/topics/testing/tools/#django.test.Client) 181 | 182 | * Django test Client returns a ```response.context``` which we can use to make sure that the information we get back is correct 183 | 184 | ### Good Testing Rules of Thumb 185 | 186 | * separate TestClass for each model or view 187 | * separate test method for each set of conditions you want to test 188 | * test method names that describe their function 189 | 190 | ### Other Kinds of Test 191 | 192 | We described backend tests, but we might want to use Selenium to test the way HTML is rendered in the browser. 193 | 194 | Look into [```LiveServerTestCase```](https://docs.djangoproject.com/en/1.11/topics/testing/tools/#django.test.LiveServerTestCase) 195 | 196 | ### Additional Resources 197 | 198 | * [Testing in Django](https://docs.djangoproject.com/en/1.11/topics/testing/) 199 | * [Integration with coverage.py](https://docs.djangoproject.com/en/1.11/topics/testing/advanced/#topics-testing-code-coverage) 200 | 201 | 202 | --- 203 | 204 | ## Static Files 205 | 206 | * Throw static files into ```/app/static/app/``` directory 207 | 208 | * [Managing Static Files](https://docs.djangoproject.com/en/1.11/howto/static-files/) 209 | * [Deploying Static Files](https://docs.djangoproject.com/en/1.11/howto/static-files/deployment/) 210 | 211 | 212 | --- 213 | 214 | ## Reuseable App 215 | 216 | Can install [app as a package](https://docs.djangoproject.com/en/1.11/intro/reusable-apps/) 217 | 218 | > A Django application is just a Python package that is specifically intended for use in a Django project. An application may use common Django conventions, such as having models, tests, urls, and views submodules. 219 | 220 | 221 | 222 | 223 | 224 | 225 | 226 | 227 | 228 | 229 | 230 | 231 | 232 | 233 | 234 | 235 | 236 | 237 | 238 | -------------------------------------------------------------------------------- /django/django_testing.md: -------------------------------------------------------------------------------- 1 | # Django Testing 2 | 3 | ## Testing Basics 4 | 5 | ### Running tests 6 | 7 | Have files named test_*.py 8 | 9 | ```console 10 | $ python manage.py test 11 | ``` 12 | 13 | ### Test Database 14 | 15 | * Tests are not done on production databases, but on a blank database created for tests 16 | * This database is destroyed once all tests have been executed 17 | 18 | ### Test Order Execution 19 | 20 | 1. All TestCase subclasses are run first 21 | 1. All other Django-based tests are run 22 | 1. unitteset.TestCase tests are run 23 | 24 | ### Misc 25 | 26 | * Any initial data loaded in migrations will only be available in TestCase tests and not in TransactionTestCase tests 27 | * tests are run with ```DEBUG=False``` obviously ;) 28 | * [running tests in parallel](https://docs.djangoproject.com/en/1.11/ref/django-admin/#cmdoption-test-parallel) 29 | 30 | --- 31 | 32 | ## Django Test Client 33 | 34 | * check if correct template is being rendered 35 | * verify the correct context is being passed in 36 | * examine status codes and url 37 | 38 | * does not need Web server to be running 39 | * avoids overhead of HTTP and deals with Django framework directly 40 | 41 | ```python 42 | from django.test import Client 43 | 44 | c = Client() 45 | response = c.get('/endpoint/', 46 | data={'foo': 2, 'bar': 3}) 47 | ``` 48 | 49 | * test client is stateful, stores cookies are res 50 | 51 | ### Testing Responses 52 | 53 | * test response object has additional information that is useful for test code to verify 54 | 55 | * client 56 | * content - body of response 57 | * context - context instance that was used to render template 58 | * json - body of response, parsed as JSON 59 | * request 60 | * status_code 61 | * templates - templates used to generate response 62 | * resolver_match - use this to find which view served the response 63 | 64 | ### Authentication 65 | 66 | * test login using ```django.test.Client.login``` 67 | * After you call this method, the test client will have all the cookies and session data required to pass any login-based tests that may form part of a view. 68 | * After logging in, can test views that are only available to logged-in users 69 | * Done in test database so we will also need to create accounts 70 | 71 | ### Miscellaneous 72 | 73 | * exceptions that are not visible to test client are Http404, PermissionDenied, SystemExit, and Suspicious Operation 74 | * Django converts these to HTTP response codes 75 | 76 | * test client is stateful, reponse returns a cookie which will be stored in client and sent with all subsequent ```get()``` and ```post()``` reponses 77 | * ```Client.cookies``` and ```Client.session``` 78 | 79 | ### Django Test Case Classes 80 | 81 | * [SimpleTestCase](https://docs.djangoproject.com/en/1.11/topics/testing/tools/#simpletestcase) 82 | * Use this for all Django Tests 83 | * [TransactionTestCase](https://docs.djangoproject.com/en/1.11/topics/testing/tools/#transactiontestcase) **Test specific database transaction behavior** 84 | * [TestCase](https://docs.djangoproject.com/en/1.11/topics/testing/tools/#testcase) **Most common way to write test** 85 | * Use these for an y tests that make a database query 86 | * [LiveServerTestCase](https://docs.djangoproject.com/en/1.11/topics/testing/tools/#django.test.LiveServerTestCase) 87 | * Use this for Selenium client. Test rendered HTML and behavior of webpages 88 | * Used for functional testing inside browser and to simulate a real user's actions 89 | 90 | > SimpleTestCase and its subclasses (e.g. TestCase, …) rely on setUpClass() and tearDownClass() to perform some class-wide initialization (e.g. overriding settings). If you need to override those methods, don’t forget to call the super implementation 91 | 92 | --- 93 | 94 | ## Test Case Features 95 | 96 | https://docs.djangoproject.com/en/1.11/topics/testing/tools/#test-cases-features 97 | 98 | https://docs.djangoproject.com/en/1.11/topics/testing/advanced/ 99 | -------------------------------------------------------------------------------- /django/test_drive_web_development_with_python.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | https://www.obeythetestinggoat.com/book/part1.harry.html -------------------------------------------------------------------------------- /general/best_software_writing_i.md: -------------------------------------------------------------------------------- 1 | # The Best Software Writing I 2 | 3 | ## Adam Bosworth - ICSOC04 Talk 4 | 5 | > It is an ironic truth that those who seek to create systems that most assume the perfectibility of humans end up building the systems that are the most soul destroying and most rigid 6 | 7 | > Software which is flexible, simple, sloppy, tolerant, and altogether forgiving of human foibles and weaknesses turns out to be actually the most steel-cored, able to survive and grow, while software which is demanding, abstract, rich, but systematized turns out to collapse in on itself in a slow and grim implosion 8 | 9 | > Text search on Google: engineering is hard, but the user model is simple and sloppy 10 | 11 | > code is just a hammer they use to do the job 12 | -------------------------------------------------------------------------------- /general/how_to_read_a_book.md: -------------------------------------------------------------------------------- 1 | # How to Read a Book 2 | 3 | by Mortimer J. Adler and Charles Van Doren 4 | 5 | #### Table of Contents 6 | 7 | - [Part 1: Dimensions of Reading](#part-1-dimensions-of-reading) 8 | - [Chatper 1: The Activity and Art of Reading](#chatper-1-the-activity-and-art-of-reading) 9 | - [Chapter 2: The Levels of Reading](#chapter-2-the-levels-of-reading) 10 | - [Chapter 3: The First Level of Reading: Elementary Reading](#chapter-3-the-first-level-of-reading-elementary-reading) 11 | - [Chapter 4:](#chapter-4) 12 | 13 | ## Part 1: Dimensions of Reading 14 | 15 | ### Chatper 1: The Activity and Art of Reading 16 | 17 | - book is for those whose main purpose in reading books is to gain increased understanding 18 | - readers are people who gain a large share of their information about and their understanding of the world from the written word 19 | - media packages opinions into people's mind and then he pushes play on the opinion whenever it seems appropriate to do so; works without thinking 20 | - all reading must be to some degree active 21 | - one reader is better than another if he demands more of himself and of the text before him 22 | - reader / listener is like a catcher in baseball, i.e. works with the pitcher aka author / speaker 23 | - successful communication occurs in any case where what the writer wanted to have received finds its way into the reader's possession 24 | - the amount the reader "catches" will usually depend on the amount of activity he puts into the process, as well as upon the skill with which he executes the different mental acts involved 25 | - **art of reading** is the process whereby a mind, with nothing to operate on but the symbols of the readable matter, and with no help from outside, elevates itself by the power of its own operations 26 | - mind passes from understanding less to understanding more 27 | - distinction between 28 | - reading for information -- reading newspapers / anything that is at ocne thoroughly intelligible to us; increase our store of information, but they cannot improve our understanding 29 | - reading for understanding -- when a person tries to read something that at first he does not completely undrestanding; the thing to be read is better / higher than the reader 30 | - conditions under which reading for understanding takes place 31 | - initial inequality in understanding -- writer must be superior to the reader in understanding and the book must convey in readable form the insights he possesses and his potential readers lack 32 | - reader must be able to over this inequality to some degree 33 | - learning from our "betters" 34 | - to be enlightened is to know, in addition, what it is all about: why it is the case, what its connections are with other facts, in which respects it is the same, in what respects it is different, and so forth 35 | - differences between being able to remember something and being able to explain it 36 | - ways of learning 37 | - learning by discovery -- the process of learning something by research, by investigtion, or by reflection, without being taught 38 | - learning by instruction -- occurs when one person teaches another through speech or writing 39 | - art of reading includes all of the same skills that are involved in the art of unaided discovery 40 | - keenness of observation 41 | - readily available memory 42 | - range of imagination 43 | - an intllect trained in analysis and reflection 44 | - reading is learning by discovery, but with an instructor 45 | - listening is learning from a teacher that is present vs reading is learning from one who is absent 46 | 47 | > Students in school often read difficult books with the help and guidance of teachers. But for those of us who are not in school, and indeed also for those of us who are when we try to read books that are not required or assigned, our continuing education depends mainly on books alone, read without a teacher's help. Therefore if we are disposed to go on learning and discovering, we must know how to make books teach us well. That, indeed, is the primary goal of this book. 48 | 49 | ### Chapter 2: The Levels of Reading 50 | 51 | 1. Elementary Reading 52 | - going from nonliteracy to at least beginning literacy 53 | - answer: What does the sentence say? 54 | 2. Inspectional Reading 55 | - aim is to get the most out of a book within a short amount of time 56 | - skimming systematically 57 | - aim is to exaime the surface of the book, to learn everything the surface alone can teach you 58 | - answer: What is this book about? / What is the structure of the book? 59 | 3. Analytical Reading 60 | - thorough reading, complete reading, or good reading 61 | - best and most compelte reading that is possible given unlimited time 62 | - must ask many organized questions of what he is reading 63 | - reading for the sake of understanding 64 | 4. Syntopical Reading 65 | - comparative reading 66 | - read many books and place them in relation to one another and to a subject about which they all revolve 67 | - with the hep of the books read, the reader is able to construct an analysis of the subject that may not be in any of the books read 68 | 69 | ### Chapter 3: The First Level of Reading: Elementary Reading 70 | 71 | - 4 stages of learning how to read 72 | - Reading readiness 73 | - Read very simple materials 74 | - Rapid progress in vocabulary building and by increasing skill in "unlocking" the meaning of unfamiliar words through context clues 75 | - learn to read for different purposes 76 | - learn reading is also something one can do on one's own for fun, to satisfy curiosity, or even to expan one's horizons 77 | - Refinement and enhancement of the skills previously acquired 78 | - being to assimilate reading experiences 79 | - carry over concepts from one piece of writing to another 80 | - compare the views of different writers on the same subject 81 | 82 | ### Chapter 4: 83 | -------------------------------------------------------------------------------- /general/learning_things_quickly.md: -------------------------------------------------------------------------------- 1 | # Ten Step System to Learning Things Quickly 2 | 3 | by John Sonmez 4 | 5 | [Summary](https://www.linkedin.com/pulse/10-steps-systems-learn-john-sonmez-carlos-enrique-hernandez-ibarra/) 6 | 7 | ## 1. Get the Big Picture 8 | 9 | > Understand a little bit about the subject before we define into it 10 | 11 | ## 2. Determine Scope 12 | 13 | > Narrow down your focus to determine what exactly you want to learn 14 | 15 | ## 3. Define Success 16 | 17 | > Come up with a clear concise statement that will define success for your learning endeavor. __Be Specific__ 18 | 19 | ## 4. Find Resources 20 | 21 | > Gather as many soruces as you can find for the topic you want to learn about, quality does not matter 22 | 23 | ## 5. Create a Learning Plan 24 | 25 | > What do you want to learn 26 | 27 | ## 6. Filter Resources 28 | 29 | > Figure out which items helps you learn the content in your learing plan 30 | 31 | ## 7. Learning Enough to get Started 32 | 33 | > Don''t go too far 34 | 35 | ## 8. Play around 36 | 37 | > Create a small project during this step and test things out, write down questions you want answered 38 | 39 | ## 9. Learn enough to do something ueseful 40 | 41 | > Go thru resources we gathered to answer questions and learn about modules in depth. **Try to tie back to your ultimate goal** 42 | 43 | # 10. Teach 44 | 45 | > Best way to learn something is to teach somebody else 46 | -------------------------------------------------------------------------------- /general/software_debugging_udacity.md: -------------------------------------------------------------------------------- 1 | # CS259 Software Debugging 2 | 3 | [Course Link](https://classroom.udacity.com/courses/cs259/) 4 | [Course Wiki](https://www.udacity.com/wiki/cs259) 5 | 6 | The notes below are a translation of Chris McGinlay's [Summary Notes](https://docs.google.com/document/pub?id=1jesnYtfsVsnNPA3rSwmMC6vEh3k-F099b9tCD0UTl_I) into Markdown. 7 | 8 | Thanks to Chris doing all the hard work and adding a [Creative Commons license](https://en.wikipedia.org/wiki/Creative_Commons_license) to his notes. 9 | 10 | ## Lesson 1: How Debuggers Work 11 | 12 | * Idea of debugging by conducting a systematic experiment to establish cause(s) of failure. 13 | 14 | ```text 15 | Input -> 16 | Failed Program -> 17 | Compare expected and actual outputs for given inputs -> 18 | Find defect causing failure 19 | ``` 20 | 21 | ### Devil’s Guide to Debugging aka Debugging DONTs 22 | 23 | * Scatter output statements everywhere. (“print” or log entry). These waste time, have to enter statements, read and decipher horrible output, remove statements. Security issues if they are left in production code (eg. read passwords in clear in log file). 24 | * Just make it work. Don’t bother to understand it. (Causes unintended breakages later). 25 | * Never back up earlier working versions. 26 | * Don’t bother to understand what the code is meant to do 27 | Fix the symptom not the problem. 28 | * Apply the most obvious fix. 29 | 30 | ### Definitions 31 | 32 | **defect** - An error in code. Preferable to the term ‘bug’ which implies that something crawled in when you weren’t looking, whereas in fact the programmer put the defect there in the first place. 33 | 34 | **failure** - an externally visible error. 35 | 36 | **error** a deviation from correct behaviour. 37 | 38 | **infection** error in program state, propagating forward to successive states. 39 | 40 | **program** succession of states represented by a state vector (of variables, objects etc) 41 | 42 | A defective statement causes a change of state, such that a sane input state becomes an insane state. Marked by an infection with no precursor infection. 43 | 44 | Possible: defect -> infection -> failure 45 | Sometimes: defect -> infection -> no failure 46 | 47 | However a failure can always be traced back to a defect via cause-effect chain. This can be HARD; tens of thousands of variables, tens of thousands of intermediate states. 48 | 49 | ### Scientific Method 50 | 51 | Posit a hypothesis. (Something you expect to happen, or think may be happening to program state). Devise an experiment to test this hypothesis. E.g. use ‘assert False’ to mark code which should never be reached (such as an undefined branch of if..elif..else). 52 | 53 | Hypotheses -> Prediction -> Experiment -> Observation. 54 | 55 | Observation will result in either confirmation or rejection of hypothesis. So either you refine or reject your hypothesis. Finally, if you have established cause of defect ensure you: 56 | 1. prevent re-occurrence of defect 57 | 1. look for same essential error elsewhere in code 58 | 59 | ### Explicit Debugging... 60 | 61 | (...as opposed to keeping the whole situation in your head) 62 | 63 | Keep a written log of what you’ve tried. No need to memorise. Easy to interrupt work and resume later. The act of writing down structures forces thinking to take place and can reveal solution. Similarly, talk to someone about the problem. Act of explaining forces you to understand the situation and structure your thoughts prior to opening your mouth. 64 | 65 | ### Code from Lecture 66 | 67 | ```python 68 | #!/usr/bin/env python 69 | # Simple debugger 70 | import sys 71 | import readline 72 | 73 | def remove_html_markup(s): 74 | tag = False 75 | quote = False 76 | out = "" 77 | 78 | for c in s: 79 | if c == '<' and not quote: 80 | tag = True 81 | elif c == '>' and not quote: 82 | tag = False 83 | elif c == '"' or c == "'" and tag: 84 | quote = not quote 85 | elif not tag: 86 | out = out + c 87 | return out 88 | 89 | # main program that runs the buggy program 90 | def main(): 91 | print remove_html_markup('xyz') 92 | print remove_html_markup('"foo"') 93 | print remove_html_markup("'foo'") 94 | 95 | # globals 96 | breakpoints = {9: True, 14: True} 97 | stepping = False 98 | 99 | def debug(command, my_locals): 100 | global stepping 101 | global breakpoints 102 | 103 | if command.find(' ') > 0: 104 | arg = command.split(' ')[1] 105 | else: 106 | arg = None 107 | 108 | if command.startswith('s'): # step 109 | stepping = True 110 | return True 111 | elif command.startswith('c'): # continue 112 | stepping = False 113 | return True 114 | elif command.startswith('q'): # quit 115 | sys.exit(0) 116 | else: 117 | print "No such command", repr(command) 118 | 119 | return False 120 | 121 | commands = ["s", "s", "q"] 122 | 123 | def input_command(): 124 | #command = raw_input("(my-spyder) ") 125 | global commands 126 | command = commands.pop(0) 127 | return command 128 | 129 | def traceit(frame, event, trace_arg): 130 | global stepping 131 | 132 | if event == 'line': 133 | if stepping or breakpoints.has_key(frame.f_lineno): 134 | resume = False 135 | while not resume: 136 | print event, frame.f_lineno, frame.f_code.co_name, frame.f_locals 137 | command = input_command() 138 | resume = debug(command, frame.f_locals) 139 | return traceit 140 | 141 | sys.settrace(traceit) 142 | 143 | print remove_html_markup('xyz') 144 | ``` 145 | 146 | Add more features to the debuggers. 147 | -------------------------------------------------------------------------------- /javascript/javascript_lanugage_guide.md: -------------------------------------------------------------------------------- 1 | # JavaScript Language Guide 2 | 3 | ## Data Types 4 | 5 | [W3 Schools](https://www.w3schools.com/js/js_datatypes.asp) 6 | 7 | * String 8 | * Number 9 | * Boolean 10 | * Array (python list) 11 | * Object (python dict) 12 | * `undefined` 13 | * "" (i.e. empty string) 14 | * `null` (object type) 15 | 16 | * `undefined` and `null` are equal in value, but of different types 17 | 18 | * function 19 | 20 | ## Functions 21 | 22 | ```js 23 | function myFunction(p1, p2) { 24 | return p1 * p2; // The function returns the product of p1 and p2 25 | } 26 | ``` 27 | 28 | ## Objects 29 | 30 | [W3 Schools](https://www.w3schools.com/js/js_objects.asp) 31 | 32 | * created with `new` keyword 33 | -------------------------------------------------------------------------------- /javascript/javascript_the_good_parts.md: -------------------------------------------------------------------------------- 1 | # JavaScript: The Good Parts 2 | 3 | by Douglas Crockford 4 | 5 | ## Chapter 1 6 | 7 | https://www.safaribooksonline.com/library/view/javascript-the-good/9780596517748/ch01.html 8 | -------------------------------------------------------------------------------- /javascript/typescript-deep-dive/promise-states-and-fates.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alysivji/notes/48e0c5920935fb7156e5b01fc6c3690b932d918e/javascript/typescript-deep-dive/promise-states-and-fates.png -------------------------------------------------------------------------------- /javascript/typescript_deep_dive.md: -------------------------------------------------------------------------------- 1 | # Typescript Deep Dive 2 | 3 | by Basart Ali Syed 4 | 5 | ## Getting Started 6 | 7 | * Types can be implicit 8 | * [Definitely Typed](https://github.com/DefinitelyTyped/DefinitelyTyped) -- type definitions 9 | 10 | * `==` is for values 11 | * `===` is for references 12 | 13 | * Something hasn't been initialized : `undefined` 14 | * Something is currently unavailable: `null` 15 | * Recommend `== null` to check for both `undefined` or `null` 16 | * don't want to make a distinction between the two 17 | 18 | * `this` == Any access to this keyword within a function is actually controlled by how the function is actually called. It is commonly referred to as the “calling context.” 19 | 20 | * [TypeScript classes](https://basarat.gitbooks.io/typescript/content/docs/classes.html) 21 | 22 | * `var` is function scoped 23 | * `let` is block scoped 24 | 25 | * object destructuring is like equivalent to python's unpacking 26 | 27 | * `...rest` is like *args (spread operator) 28 | * `for ... of ...` loops over items not indexes 29 | 30 | * JS's version of f-strings 31 | 32 | ```js 33 | console.log(`${blah}`) 34 | ``` 35 | 36 | ### Async Functions 37 | 38 | A few points to keep in mind when working with async functions based on callbacks are: 39 | 40 | 1. Never call the callback twice. 41 | 1. Never throw an error. 42 | 43 | But we fall into callback hell, so use promises. 44 | 45 | ### Promises 46 | 47 | * A promise can be either `pending` or `fulfilled` or `rejected` 48 | 49 | 50 | 51 | Let's look at creating a promise. It's a simple matter of calling new on Promise (the promise constructor). The promise constructor is passed resolve and reject functions for settling the promise state: 52 | 53 | ```js 54 | const promise = new Promise((resolve, reject) => { 55 | // the resolve / reject functions control the fate of the promise 56 | }); 57 | ``` 58 | 59 | * The promise fate can be subscribed to using `.then` (if resolved) or `.catch` (if rejected). 60 | 61 | * Tips: 62 | * Quickly creating an already resolved promise: `Promise.resolve(result)` 63 | * Quickly creating an already rejected promise: `Promise.reject(error)` 64 | 65 | ## Compiler Options 66 | 67 | ```json 68 | { 69 | "compilerOptions": { 70 | 71 | /* Basic Options */ 72 | "target": "es5", /* Specify ECMAScript target version: 'ES3' (default), 'ES5', 'ES2015', 'ES2016', 'ES2017', or 'ESNEXT'. */ 73 | "module": "commonjs", /* Specify module code generation: 'commonjs', 'amd', 'system', 'umd' or 'es2015'. */ 74 | "lib": [], /* Specify library files to be included in the compilation: */ 75 | "allowJs": true, /* Allow javascript files to be compiled. */ 76 | "checkJs": true, /* Report errors in .js files. */ 77 | "jsx": "preserve", /* Specify JSX code generation: 'preserve', 'react-native', or 'react'. */ 78 | "declaration": true, /* Generates corresponding '.d.ts' file. */ 79 | "sourceMap": true, /* Generates corresponding '.map' file. */ 80 | "outFile": "./", /* Concatenate and emit output to single file. */ 81 | "outDir": "./", /* Redirect output structure to the directory. */ 82 | "rootDir": "./", /* Specify the root directory of input files. Use to control the output directory structure with --outDir. */ 83 | "removeComments": true, /* Do not emit comments to output. */ 84 | "noEmit": true, /* Do not emit outputs. */ 85 | "importHelpers": true, /* Import emit helpers from 'tslib'. */ 86 | "downlevelIteration": true, /* Provide full support for iterables in 'for-of', spread, and destructuring when targeting 'ES5' or 'ES3'. */ 87 | "isolatedModules": true, /* Transpile each file as a separate module (similar to 'ts.transpileModule'). */ 88 | 89 | /* Strict Type-Checking Options */ 90 | "strict": true, /* Enable all strict type-checking options. */ 91 | "noImplicitAny": true, /* Raise error on expressions and declarations with an implied 'any' type. */ 92 | "strictNullChecks": true, /* Enable strict null checks. */ 93 | "noImplicitThis": true, /* Raise error on 'this' expressions with an implied 'any' type. */ 94 | "alwaysStrict": true, /* Parse in strict mode and emit "use strict" for each source file. */ 95 | 96 | /* Additional Checks */ 97 | "noUnusedLocals": true, /* Report errors on unused locals. */ 98 | "noUnusedParameters": true, /* Report errors on unused parameters. */ 99 | "noImplicitReturns": true, /* Report error when not all code paths in function return a value. */ 100 | "noFallthroughCasesInSwitch": true, /* Report errors for fallthrough cases in switch statement. */ 101 | 102 | /* Module Resolution Options */ 103 | "moduleResolution": "node", /* Specify module resolution strategy: 'node' (Node.js) or 'classic' (TypeScript pre-1.6). */ 104 | "baseUrl": "./", /* Base directory to resolve non-absolute module names. */ 105 | "paths": {}, /* A series of entries which re-map imports to lookup locations relative to the 'baseUrl'. */ 106 | "rootDirs": [], /* List of root folders whose combined content represents the structure of the project at runtime. */ 107 | "typeRoots": [], /* List of folders to include type definitions from. */ 108 | "types": [], /* Type declaration files to be included in compilation. */ 109 | "allowSyntheticDefaultImports": true, /* Allow default imports from modules with no default export. This does not affect code emit, just typechecking. */ 110 | 111 | /* Source Map Options */ 112 | "sourceRoot": "./", /* Specify the location where debugger should locate TypeScript files instead of source locations. */ 113 | "mapRoot": "./", /* Specify the location where debugger should locate map files instead of generated locations. */ 114 | "inlineSourceMap": true, /* Emit a single file with source maps instead of having a separate file. */ 115 | "inlineSources": true, /* Emit the source alongside the sourcemaps within a single file; requires '--inlineSourceMap' or '--sourceMap' to be set. */ 116 | 117 | /* Experimental Options */ 118 | "experimentalDecorators": true, /* Enables experimental support for ES7 decorators. */ 119 | "emitDecoratorMetadata": true /* Enables experimental support for emitting type metadata for decorators. */ 120 | } 121 | } 122 | ``` 123 | -------------------------------------------------------------------------------- /meetup/blockchain-in-healthcare-summit.md: -------------------------------------------------------------------------------- 1 | # Blockchain in Healthcare Summit 2 | 3 | May 3rd, 2018 4 | 5 | ## Hyperledger Fabric 6 | 7 | https://www.ibm.com/blockchain/hyperledger.html 8 | 9 | https://hyperledger-fabric.readthedocs.io/en/release-1.1/ 10 | 11 | https://hyperledger-fabric.readthedocs.io/en/release-1.1/tutorials.html 12 | -------------------------------------------------------------------------------- /meetup/chicago-startup-community.md: -------------------------------------------------------------------------------- 1 | # Chicago Startup Community 2 | 3 | ## [Healthcare Innovation in Chicago (Design)](https://www.meetup.com/Chicago-Startup-Community/events/250817190/) 4 | 5 | May 22nd, 2018 6 | 7 | * can't design workflows without understanding how everybody is working 8 | * it's not about tech, it's about tech enabling healthcare professionals to improve outcomes 9 | * how can we disseminate healthcare information? we need to understand how healthcare is talked about in healthcare offices around the country 10 | * Talk to the pharmacists to know exactly what we need to show when we do analysis 11 | * keep track of who has access to data, it might be a good idea not to store things 12 | * healthcare data isn't like all other kind of data 13 | * find out who built the process, so we can work together and add technology to the process 14 | * what are pain points in the system? need to watch people use our platform 15 | * what's your day like? walk me thru how you do this? 16 | * what do you hate? 17 | * what are your workarounds? 18 | * iterate to success 19 | -------------------------------------------------------------------------------- /meetup/pycon2018.md: -------------------------------------------------------------------------------- 1 | # PyCon 2018 2 | 3 | ## Open Spaces (BoFs) 4 | 5 | ### Modern SQL BoF 6 | 7 | http://sedimental.org/pycon2018_bofs.html 8 | 9 | ### Testing Database using py.test BoF 10 | 11 | * session level fixture to return connection 12 | 13 | https://github.com/FactoryBoy/factory_boy 14 | 15 | http://pgtap.org/ 16 | 17 | ### Robust CLIs in Python BoF 18 | 19 | * term coloring 20 | * doc-opt 21 | -------------------------------------------------------------------------------- /meetup/react-chicago.md: -------------------------------------------------------------------------------- 1 | # React Chciago 2 | 3 | ## Books to Read 4 | 5 | * Don't make me think 6 | 7 | ## April 2018 8 | 9 | ### What's new in React 16.04 10 | 11 | * portals -- render react element outside react tree 12 | * fragments -- instead of rendering in `
.
`, we can render in `...` 13 | * [docs](https://reactjs.org/docs/fragments.html) 14 | 15 | * async rendering 16 | * lifecycle hooks are being depreccated and replaced with better methods for async 17 | * `componentWillMount`  — please use `componentDidMount` instead 18 | * `componentWillUpdate ` — please use `componentDidUpdate` instead 19 | * `componentWillReceiveProps ` — a new function, `static getDerivedStateFromProps` is introduced 20 | 21 | * context api 22 | * [more details](https://medium.freecodecamp.org/reacts-new-context-api-how-to-toggle-between-local-and-global-state-c6ace81443d0) 23 | 24 | ### React Native 25 | 26 | * react-native-sensitive-info 27 | * react-native-debugger 28 | * `detox` for testing 29 | * `app center` to do continuous builds for release in app store 30 | 31 | * build native components for iOS and Android and pull them into React Native as components 32 | 33 | ## March 2018 34 | 35 | ### Testing React Apps 36 | 37 | * Jest 101 38 | * jest comes with `create-react-app` 39 | * react-test-renderer vs enzyme 40 | * tests via snapshots... look into 41 | * deep render vs shallow render 42 | * snapshot-diff 43 | * jest-image-snapshot 44 | * jest-fetch-mock 45 | * vscode-jest 46 | * awesome-jest on Github 47 | * test interactions with enzyme 48 | -------------------------------------------------------------------------------- /professional-development/7_principles_of_public_speaking.md: -------------------------------------------------------------------------------- 1 | # The 7 Principles of Public Speaking 2 | 3 | By Richard Zeoli 4 | 5 | ## Principle 1: Perception 6 | 7 | > Stop trying to be a gerat public speaker 8 | 9 | * The best public speakers are those who seem to genuinely enjoy giving a speech 10 | * Great speakers speak to us, not at us 11 | * **Be prepared** 12 | * Time feels different at the podium than it does for the audience 13 | * Accomplished communicators handle questions by taking a few seconds to organize their thoughts before giving a carefullly constructed answer 14 | * Know your audience (this might involve some research) 15 | * Know the material inside and out. Pratice the material inside and out. Be able to give the speech without the benefits of having a written speech in front of you (but have notes that you can refer to as needed) 16 | * Approach humor with caution 17 | * Thank event host, organizers, staff, and any volunteers for their hard work 18 | * Approach publicspeaking as a conversation with the audience rather than as a "speech" 19 | 20 | ## Principle 2: Perfection 21 | 22 | > When you make a mistake, no one cares but you 23 | 24 | * People's attention spans constantly wander, chances are they won't even know you made a mistake 25 | * Need to write a speech to draw people's attention in 26 | * Eye Contact 27 | * Signals to people that they need to pay attention 28 | * Makes people feel important and makes them feel like you care about them 29 | * Tone & Pitch 30 | * Write remarks with phrases designed for vocal inflection (verbal cue to pay attention) 31 | * Write lines that cause your voice to naturally go up and down, even to the point of becoming quiet 32 | * People are with you during the intro and conclusion, we need to keep their minds from wandering during the middle 33 | * Try to insert a line that allows the tone of your voice to go up and/or down at elast once or twice in each paragraph 34 | * Dramatic Pause 35 | * Sends cue to listener to tune in because something good is coming 36 | * Always feels longer to speaker than to the audience so don't rush it 37 | 38 | ## Principle 3: Visualization 39 | 40 | > If you can see it, you can speak it 41 | 42 | * We might be born with a natural potential for a talent, but it takes a lot of training to bring this potential front and center 43 | * Practice in your mind, visualize yourself succeeding and you will 44 | * If you picture with intensity the scenarios you want, your mind has a funny way of making them happen 45 | 46 | ## Principle 4: Discipline 47 | 48 | > Practice makes good 49 | 50 | * Practicing consistently will make you good 51 | * Stand in front of a mirror and watch yourself talk. Look for mannerisms. The more you do it, the more comfortable you will feel 52 | * The five minutes before you are speaking are the most cruical as a speaker 53 | * Mentally picture yourself successfully taking the podium and addressing the crowd 54 | * Tell yourself positive affirmations that wll help you succeed 55 | * *I am prepared. I know this material. I know this topic. The audience is going to understand the depth of knowledge and appreciate my subject matter. But I recognize that not 100 of the audience will feel this way and that's okay. I am prepared. I am ready 56 | * When you tell yourself you are a champion, you will act like a champion 57 | 58 | ## Principle 5: Description 59 | 60 | > Make it personal, become a storyteller 61 | 62 | * People don't care about facts as much as they care about feelings 63 | * Don't inundate people with data, tell them something interesting, something that will help them grow while capturing their attention, something that they can take away with them 64 | * How to structure your story 65 | * Buy index cards 66 | * Write down facts about the memorable moments related to this topic. One card for each moment 67 | * No thought is too insignificant at the initial stage 68 | * Go thru each card and fill the feeling surround the facts 69 | * Write down everything: failures, successes, great moments you wish could happen again, moments you wish never happened, people you met along the way 70 | * Every story has a beginning, a middle, and an end 71 | * As you speak, your job is to take your listeners on a journey with you. Bring them with you as you take your new job, start your new business, and face the struggles that are an inevitable part of any new venture 72 | * Soon your story will take shape: you're telling the audience about a memorable event in yoru life and then you're telling them *why* it was memorable and *how* it affected you 73 | 74 | ## Principle 6: Inspiration 75 | 76 | > Speak to serve 77 | 78 | * The most important person in the room is the audience 79 | * Serve the audience by talking about something that interests them or by making an otherwise uninteresting topic interesting 80 | * Effective communicators are the ones who not only make us feel comfortable and relaxed but also have a purpose in what they're tryign to say. When we listen to them, we understand our personal stake in the subject. 81 | * While preparing and presenting, always think about how you can help the audience members achieve their goals. It will force you to come up with more details and examples to make your speech engaging 82 | * How can I serve my listeners? 83 | * How can I make this clear to them? 84 | * How can I show them what's at stake for them? 85 | * How can I show them how I can make their life, company, product or service better? 86 | * Never state what you do as a matter of fact, explain it in a way that shows how the audience will benefit from what you do 87 | 88 | ## Principle 7: Anticipation 89 | 90 | > Always leave your audience wantng more 91 | 92 | * less is usually more 93 | * How long should a speech be? Long enough to convey all the necessary information and not a second longer. As speakers, we must always maintain the goal of getting in and getting out. And of course convey the information somehwere in between. Do it in a way that's informative. Make it enjoyable, and then give your listeners the opportunity to get on with their lives 94 | * How do you know when enough is enough? 95 | * You have to make sure that you have effectively proven your thesis 96 | * Then summarize your point, plug your shit, thank the audience, and get out 97 | 98 | ## Bonus Principle: Staying on Message 99 | 100 | * Before you can prove your point, you must first prove your credibility before your audience. Only then will they believe you have the expertise to know what you are talking about 101 | * Gain credibility by talking about your experience, educational background, awards, published articles, specific knowledge, research, past accomplishments, detailed plan of action 102 | * Once you establish credibility, it's time to prove your point while staying on message using examples and stories 103 | * Consider your message like a laser beam. Everything you say should be used to support your message with precision 104 | * If it doesn't support your message, replace it with something that does 105 | * Simply stating experience isn't as powerful or effective as telling a story that shows experience 106 | * "Show, don't tell" You ust show the audience how you feel rather than telling them 107 | * Story along with picture does so much ot prove your point and make the presentation more valuable than a picture alone 108 | * When a speaker shows the visual, tells the story, makes it personal, and puts it into human terms to which we can all relate, that speaker is taking the visual to the next level 109 | -------------------------------------------------------------------------------- /professional-development/passionate_programmer.md: -------------------------------------------------------------------------------- 1 | # The Passionate Programmer 2 | 3 | By Chad Fowler 4 | 5 | ## Choosing Your Market 6 | 7 | ### Tip 1: Lead or Bleed 8 | 9 | > Both ends of the technology adoption curve might prove to be lucrative 10 | 11 | *Thoughts* 12 | 13 | Figuring out which technologies to invest time in learning is something we should be extremely deliberate about. 14 | 15 | * Do you want to jump on something new only to have it fade into obscurity? Learn how to read trends. Always stay on top of the latest happenings. Need to make this easier somehow. 16 | * Learning something entrenched means safety, but you are a dime a dozen coder. How to differentiate? Add a twist to the entrenched technology. 17 | * Learing something that is close to dying out means you can be an expert in a niche field. But always have to be on the lookout for signs where we need to move on to something else 18 | 19 | ### Tip 2: Supply and Demand 20 | 21 | > You cannot compete against offshore programmers on price, but you can compete on ability 22 | 23 | * Following the jobs will put you in price competition with offshore developers. 24 | * To compete in mainstream technology, we have to compete at a higher skill level 25 | * Or learn technologies that offshore companies will not learn 26 | 27 | *Thoughts* 28 | 29 | Again, it all comes down to finding your niche. What is your value add? How is your skillset unique? What do you bring to the table? 30 | 31 | ### Tip 3: Coding Doesn't Cut it Anymore 32 | 33 | > If you want to stay relevant, you're going to have to dive into the domain of the business you're in 34 | 35 | * Don't be "just a programmer", but be able to speak to your clients in their language 36 | * Put the same level of care into selecting which industry to serve as you put into selecting which technologies to master 37 | 38 | *Thoughts* 39 | 40 | Specialization does limit your options, but it also makes your more in demand for the specific niche you occupy. I didn't get my first fulltime developer job because I can code, I got it because I have domain expertise (in healthcare and clinical decision support) to compliment my programming skills. 41 | 42 | ### Tip 4: Be the Worst 43 | 44 | > Being the worst guy in the band means always playing iwth people who are better than you 45 | 46 | *Thoughts* 47 | 48 | * Challenge yourself and work with people better than you. Soon you will start emulating their good habits 49 | 50 | ### Tip 5: Invest in Your Intelligence 51 | 52 | > More technically competant people seek diverse experience because they learn to love new things or being forced into alien experiences and environments makes them more mature, well-rounded software developers 53 | 54 | * Stepping out of your comfort zone shows that you are motivated and engaged, but it also makes you deeper, better, smarter, and more creative 55 | 56 | *Thoughts* 57 | 58 | Invest in learning subjects that force you to think outside of the box. If you know OOP, learn functional programming! 59 | 60 | Being able to think in different ways will make us better problem solvers. We can never have enough tools in our repetoire. I learned Angular 1 a few years back in order to build a decision support tool. Even though the technology is obsolete, learning Angular taught me the principles of MVC which made me into a better programmer. 61 | 62 | Being interested in what you do also shows your passion. So start a blog, give talks, become a mentor. 63 | 64 | ### Tip 6: Don't Listen to Your Parents 65 | 66 | > Don't be fear-driven, take calculated risks with your career. Don't let fear consume you. And if you're not having fun, you're not going to be excellent 67 | 68 | *Thoughts* 69 | 70 | Programming for a living is a difficult job so you'll need to love what you do in order to be successful. Don't be content with mediocrity. 71 | 72 | ### Tip 7: Be a Generalist 73 | 74 | > Be the person everybody turns to when problems arise. Be useful. Your skills should transcend technology platforms 75 | 76 | *Thoughts* 77 | 78 | There is a lot of stuff to learn, but you don't need to master everything. As long as you can connect the dots fairl well, you'll have a leg up on everybody else. 79 | 80 | How many data scientists can actually code and fix their own problems without StackOverflow? Wouldn't they be more effective at their jobs if the actually learned the language they are using in a deeper way? 81 | 82 | ### Tip 8: Be a Specialist 83 | 84 | > Too many of us seem to believe that specializing in something simply means you don't know about other things 85 | 86 | *Thoughts* 87 | 88 | Make sure you understand the things you do. Don't just blindly code without thinking about what is going on behind the scenes. Abstraction is good, but you should be able to go one or two layers deep. 89 | 90 | ### Tip 9: Don't Put All Your Eggs in Someone Else's Basket 91 | 92 | > Vendor-centric views are typically myopic 93 | > 94 | > If you do have to invest in a technology, make sure it's an open source project 95 | 96 | *Thoughts* 97 | 98 | Be framework agnostic. Know how to solve problems. Know design patterns. Know algorithms. Concepts are universal. Frameworks are not, they are opinionated and you need to understand limitations and how to bypass. 99 | 100 | ### Tip 10: Love It or Leave It 101 | 102 | > You have to be passionate about your work if you want to be great at your work. If you don't care, it will show 103 | 104 | *Thoughts* 105 | 106 | Programming is hard. It will be you down. It will make you feel stupid. If you don't enjoy every single moment of the experience, you won't be remarkable. 107 | 108 | --- 109 | 110 | ## Investing in Your Product 111 | 112 | ### Tip 11: Learn to Fish 113 | 114 | > Who wants to be at the mercy of someone else? Don't let important things be voodoo to you. 115 | > 116 | > Don't let important things be like voodoo to you. 117 | > 118 | > The best software people become experts in their domains. They build better products and work faster than a develoepr who does not understand the business. 119 | 120 | *Thoughts* 121 | 122 | * In order to get projects off the ground, it helps to be able to do things across the entire stack. This is more true for solo projects, but having a deeper knowledgebase makes it so you don't need to wait or somebody to help you. You can solve your own problem. 123 | 124 | * If you don't understand the business problem you are trying to solve, you won't make a very valuable contributor to your team. 125 | 126 | ### Tip 12: Learn How Businesses Really Work 127 | 128 | > Figure out the business domain that best fits your skillset and current/future state of the market 129 | > 130 | > You can't creatively help a business be profitable if you don't understand how the business works 131 | 132 | *Thoughts* 133 | 134 | * Every single programmer has technical knowledge, but do they have domain knowledge? 135 | 136 | ### Tip 13: Find a Mentor 137 | 138 | > Mentor helps set the standard for what "good" means. A mentor can add structure to your learning process. Mentor is invested in your success. 139 | 140 | *Thoughts* 141 | 142 | * Having a mentor to bounce ideas off of made me better. I was putting together a very high level (read: meh) blog post / presentation on Python Context Managers. My mentor told me that it was a subject that a lot of people already knew about. Even though the material I was covering would be useful to newbie coders, I should go deeper and dig into the topic. Doing the research for my post (and then presentation) on Context Managers made me a better developer. It got me to think about Python language features, and language features in general, in a lot more depth. Without a guide, I would be lost. 143 | 144 | ### Tip 14: Be a Mentor 145 | 146 | > If you want to really learn something, try teaching it to someone else 147 | 148 | *Thoughts* 149 | 150 | * Giving talks, writing blog posts, pointing friends in the right direction. These are all the ways that I've been a mentor. Rewarding experience because you help somebody understand something, but also selfish because you take something away as well. 151 | 152 | ### Tip 15: Practice, Practice, Practice 153 | 154 | > Athletes push themselves to the limit during workouts so they can expand those limits for the real performances. Our industry tends to practice on the job. 155 | 156 | *Thoughts* 157 | 158 | * Figure out your weaknesses. Make them strengths by learning and practicing. Force yourself to learn something by throwing yourself into the deep end and forcing yourself to swim. 159 | 160 | ### Tip 16: The Way That You Do It 161 | 162 | > It's as important to focus on the process we're using as it is to focus on the product being developed. Much of this prior art has been codified into a group of methodologies. 163 | 164 | *Thoughts* 165 | 166 | * Don't really have a lot of experience with project methodologies, but reading Code Complete made me realize that structure is good. This is a topic I'm interested in learning more about. 167 | 168 | ### Tip 17: On the Shoulders of Giants 169 | 170 | > We can mine a huge body of existing code for patterns and tricks 171 | 172 | *Thoughts* 173 | 174 | * We learn how to get better by looking at how those better developers solve problems 175 | 176 | ### Tip 18: Automate Yourself into a Job 177 | 178 | > High skilled coders are worth more than an army of low paid / low skill workers 179 | 180 | *Thoughts* 181 | 182 | Be good at what you do. Understand the ins and outs of your profession. This is your career. Treat it like one 183 | 184 | --- 185 | 186 | ## Execution 187 | 188 | ### Tip 19: Right Now 189 | 190 | > Parkinson's Law: "Work expands so as to fill the time available for its completion" 191 | > 192 | > A sense of urgency, even if manufactured, will double or triple your productivity. You'll get it done, instead of talking about getting it done. 193 | 194 | *Thoughts* 195 | 196 | * Lots of mini-deadlines help you get a sense of accomplishment as you are working through a task list. 197 | 198 | ### Tip 20: Mind Reader 199 | 200 | > Listen to what people are saying and learn to anticipate what they want before they even know they want them 201 | 202 | *Thoughts* 203 | 204 | * Listen to what people want. Make it in the corner and then be the wizard who knows how to do everything. Don't unveil half complete projects, always make sure it is some sort of MVP. 205 | 206 | ### Tip 21: Daily Hit 207 | 208 | > Simply setting a goal and tracking your accomplishments can radically change your behavior 209 | 210 | *Thoughts* 211 | 212 | ### Tip 22: Remember Who You Work For 213 | 214 | > Align yourself with the needs of your business by solving your manager's problems 215 | 216 | *Thoughts* 217 | 218 | This is a lot more difficult to do in a larger organization where there are many layers between you and the organizational goal. Be useful to your manager and solve his / her problem and you'll be golden. 219 | 220 | ### Tip 23: Be Where You're At 221 | 222 | > Focusing on the present will get you further towards your goal than focusing on the goal itself 223 | 224 | *Thoughts* 225 | 226 | Take pride in what you do. If you phone it in, you'll never get ahead. 227 | 228 | ### Tip 24: How Good a Job Can I Do Today? 229 | 230 | > When we have more fun, we do better work 231 | 232 | *Thoughts* 233 | 234 | * Make your job into a game. Try to beat your time for a boring repetitive task. 235 | 236 | ### Tip 25: How Much Are You Worth? 237 | 238 | > How much value do I add to the bottom line? 239 | 240 | *Thoughts* 241 | 242 | * Always look for ways to improve things in order to save the company money, improve a process for a client, etc 243 | 244 | ### Tip 26: A Pebble in a Bucket of Water 245 | 246 | > Beware of being blinded by your own success 247 | 248 | *Thoughts* 249 | 250 | * As great as we think we are, nobody is irreplaceable. We are all just cogs in a wheel. Never get complacent. 251 | 252 | ### Tip 27: Learn to Love Maintenance 253 | 254 | > Embrace Maintencance programming and make it a mission to learn how to do stuff well 255 | 256 | *Thoughts* 257 | 258 | * Maintenance work isn't sexy, but it needs to be done. Take pride in what you do. Use the maintenance phase to learn the business inside and out. 259 | 260 | ### Tip 28: Eight-Hour Burn 261 | 262 | > Eight-Hour Burn is when you work so relentlessly that there is no way you could continue longer than eight hours. 263 | 264 | * Budget your work hours carefully. Work less, and you'll accomplish more. Work is always more wlecome when you've gven yourself time away from it 265 | 266 | *Thoughts* 267 | 268 | * Go in. Work hard for 4 hours. Take a break. Work hard for another 4 hours. 269 | * You have a live outside of work to waste time and surf the internet 270 | 271 | ### Tip 29: Learn How to Fail 272 | 273 | > Even masters play wrong notes, but they recover in such a way that the listener can't tell if the whole thing wasn't intentional 274 | 275 | How to deal with a technical, communication, or project management mistake: 276 | * Raise the issue as early as you know about it 277 | * Take the blame 278 | * Offer a solution 279 | * Ask for help 280 | 281 | *Thoughts* 282 | 283 | * It's not about messing, it's about learning from the screwup and figuring out a way to never let it happen again. We are judged by how we come back from setbacks, not the setback itself. 284 | 285 | ### Tip 30: Say No 286 | 287 | > The quickest path to missing your commitments is to make commitments that you know you can't meet. Don't be afraid to say, "I don't know" 288 | 289 | *Thoughts* 290 | 291 | * Don't act like you know everything or can do everything just to answer a question or make somebody happy. Be honest about your limits that way you will build trust with those who work around you. You are a person of your word 292 | 293 | ### Tip 31: Don't Panic 294 | 295 | > Heroes never panic. 296 | 297 | *Thoughts* 298 | 299 | * Panic leads to subpark performance. Stop, take a moment to access, continue after taking a deep breath. 300 | 301 | ### Tip 32: Say It, Do It, Show It 302 | 303 | > If you say what you're going to do and then you do it and show that it's done, you develop a reputation for being a doer 304 | 305 | * Planning can be liberating. Make a list of items to get done next. Keep going with the Pomodoro technique until you get your todo list done 306 | 307 | *Thoughts* 308 | 309 | * If you plan out your work, you are more likely to accomplish it. Use task management systems like Getting Things Done to stay on top of the workload. 310 | 311 | --- 312 | 313 | ## Marketing... Not Just for Suits 314 | 315 | ### Tip 33: Perceptions, Perschmeptions 316 | 317 | > There is no way to objectively measure the quality of a knowledge worker or their work, so you are measured on someone else's perception of you 318 | 319 | * Understanding what's important in each of your relationships is an important part of building credible perceptions with those you interact with 320 | 321 | *Thoughts* 322 | 323 | * As much as you shouldn't care what people think of you, you need to put in effort into every single relationship you ahve 324 | 325 | ### Tip 34: Adventure Tour Guide 326 | 327 | > Customers and managers are looking for someone who can make them comfortable about the project they are working on. Be cognizant that your customers may need topics toned down a bit when you've discussed software related matters 328 | 329 | *Thoughts* 330 | 331 | * Being able to talk about something intelligently means you are able to communicate concepts to an audience of varied skill levels 332 | 333 | ### Tip 35: Me Rite Reel Nice 334 | 335 | > Much of your job involves writing and people base their perceptions of you based on your writing ability so you need to get good at it 336 | 337 | *Thoughts* 338 | 339 | * Take some time to organize your thoughts before you respond, make sure you don't sound scatterbrained 340 | 341 | ### Tip 36: Being Present 342 | 343 | > Learn about your colleagues 344 | 345 | *Thoughts* 346 | 347 | Yes we are here to work, but make work enjoyable and get to know your coworkers. What do they like. It's not bad to start taking notes. 348 | 349 | ### Tip 37: Suit Speak 350 | 351 | > People that run businesses are interested in results so market your accomplishments in the language of your business 352 | 353 | *Thoughts* 354 | 355 | * How are you adding to the bottom line? You're not a cog in the wheel, you need to have a full scale understanding of everything that needs to be done 356 | 357 | ### Tip 38: Change the World 358 | 359 | > You need to come into work on a mission to effect change through your team, organization or company 360 | 361 | *Thoughts* 362 | 363 | * Have a purpose. You can't just go to work and only do what is expected of you. Look for ways to improve current processes. Take initiative 364 | 365 | ### Tip 39: Let Your Voice Be Heard 366 | 367 | > Think beyond your next promotion or the company you work for. You are a craftsperson and an artist within a large industry. 368 | 369 | *Thoughts* 370 | 371 | * Everybody had something to share. Become an expert by writing and giving talks. Fake it until you make it. 372 | 373 | ### Tip 40: Build Your Brand 374 | 375 | > Your name is your brand. Make your mark so that people recognize it and make sure that your mark is associated with positive traits. The most potentially destructive force for Brand You is yourself. 376 | 377 | *Thoughts* 378 | 379 | * You need to put yourself out there in a way that people will look at you as an expert. Don't fuck up and be a dick. Don't do lousy work. This is your life, get your shit together. 380 | 381 | ### Tip 41: Release Your Code 382 | 383 | > Contributing to open source is another way to makert yourself and it also shows you are passionate about your field 384 | 385 | * Refactorum - take a piece of open source code with unit tests, run tests thru test coverage analyzer, find least tested part of the system and write test to improve coverage of that code. Untested code is often untestable. Refactor to make code more testable. Submit changes as a patch. 386 | 387 | *Thoughts* 388 | 389 | * Public contributions to big projects raise your profile. 390 | 391 | ### Tip 42: Remarkability 392 | 393 | > To be remakrable, you ahve to be significantly different from those around you. You also need to be doing. It's not enough to be the best, you need to show you are the best 394 | 395 | *Thoughts* 396 | 397 | * Find a niche and dominate it. Combine a couple of different ideas into one 398 | 399 | ### Tip 43: Making the Hang 400 | 401 | > People like to be appreciated and they like to talk about the topics they are passionate about so don't be afraid to reach out to 'famous' people 402 | 403 | *Thoughts* 404 | 405 | * Putting yourself out there and connecting with people also increases your profile and your social connectiveness 406 | 407 | --- 408 | 409 | ## Maintaining Your Edge 410 | 411 | ### Tip 44: Already Obsolete 412 | 413 | > Success breeds hubris, which breeds complacency 414 | 415 | *Thoughts* 416 | 417 | * Our industry is always evolving and we have to be continuously learning. Yes it sucks to have to study all the time, but it will ensure that we don't go obsolete. Also, doesn't it sound kind of boring to do the same thing over and over again 418 | 419 | ### Tip 45: You've Already Lost Your Job 420 | 421 | > Don't buy too heavily into a long, imagined future. Can't afford to have tunnel vision with something too far off in the future. To hit a moving target, you have to aim to where the target is going to be 422 | 423 | *Thoughts* 424 | 425 | * Figure out where you want to go and take a little bit of a windy road to get where you want to be. By taking the scenic route, we have flexbility to change the set path we are on 426 | 427 | ### Tip 46: Path with No Destination 428 | 429 | > The real meat of your career is the time you spend working on yourself and your skills not on the advacements you make 430 | 431 | *Thoughts* 432 | 433 | * It's the journey, not the destination. If we aim for the moon and miss, we've still gone pretty far! 434 | 435 | ### Tip 47: Make Yourself a Map 436 | 437 | > It's a good idea to think about the story your skillset tells 438 | 439 | *Thoughts* 440 | 441 | * Are you just wandering around or are you structured about how you learn the things you want to learn. Are you working towards a goal? Track and find out. 442 | 443 | ### Tip 48: Watch the Market 444 | 445 | > Watch the alpha geeks 446 | 447 | *Thoughts* 448 | 449 | * Where is the technology going? Are you learning the right thing? Keep abreast of industry developments 450 | 451 | ### Tip 49: That Fat Man in the Mirror 452 | 453 | > If you're constantly exposed to something, it's hard to see it change unless the change happens rapidly 454 | 455 | *Thoughts* 456 | 457 | * Monitor your progress, but also ask for honest feedback about how you are doing 458 | 459 | ### Tip 50: The South Indian Monkey Trap 460 | 461 | > If you believe something so strongly that you no longer objectively question it, you have blind spots. 462 | 463 | *Thoughts* 464 | 465 | * If you only know how to hammer, everything looks like a nail. What do you believe strongly? What do you need to work on to think differently? 466 | 467 | ### Tip 51: Avoid Waterfall Career Planning 468 | 469 | > Iterating to a solution is always less stressful and more effective 470 | 471 | *Thoughts* 472 | 473 | * You need to be adaptable to change. Change isn't just possible, it's inevitable. 474 | 475 | ### Tip 52: Better Than Yesterday 476 | 477 | > The key to successfully solving big problems is to make small, incremental improvements each day. 478 | 479 | *Thoughts* 480 | 481 | * Life is reps. Put in the time and you will see results 482 | 483 | ### Tip 53: Go Independent 484 | 485 | > It's hard to be remarkable at a big company, go independent! 486 | 487 | *Thoughts* 488 | 489 | * Big organizations breed bureaucracy and kill innovation. Lesson learned more than a few times. 490 | -------------------------------------------------------------------------------- /professional-development/pragmatic_programmer.md: -------------------------------------------------------------------------------- 1 | 2 | Developer Tea podcast on Estimating https://spec.fm/podcasts/developer-tea/9981 3 | -------------------------------------------------------------------------------- /python/logging.md: -------------------------------------------------------------------------------- 1 | # Python Logging 2 | 3 | ## [Basic Logging Tutorial](https://docs.python.org/3/howto/logging.html) 4 | 5 | * We log contextual data and assign it a level 6 | * This helps us debug the program if certain events occur 7 | 8 | |Level|When it's used| 9 | |---|---| 10 | |`DEBUG`|Detailed information, typically of interest only when diagnosing problems.| 11 | |`INFO`|Confirmation that things are working as expected.| 12 | |`WARNING`|An indication that something unexpected happened, or indicative of some problem in the near future (e.g. ‘disk space low’). The software is still working as expected.| 13 | |`ERROR`|Due to a more serious problem, the software has not been able to perform some function.| 14 | |`CRITICAL`|A serious error, indicating that the program itself may be unable to continue running.| 15 | 16 | Default level is `WARNING` so all errors this level and above will be tracked 17 | 18 | ### Basic Config 19 | 20 | ```python 21 | import logging 22 | 23 | logging.basicConfig( 24 | filename='my_log.log', 25 | level=logging.DEBUG, 26 | format='', 27 | datefmt='', 28 | ) 29 | ``` 30 | 31 | * Can also set logging level via a command-line option `--log=LEVEL` 32 | * Access via `getattr(logging, loglevel.upper())` 33 | 34 | --- 35 | 36 | ## [Advanced Logging Tutorial](https://docs.python.org/3/howto/logging.html#logging-advanced-tutorial) 37 | 38 | ### Logging from Mutiple Modules 39 | 40 | > * __Loggers__ expose the interface that application code directly uses. 41 | > * __Handlers__ send the log records (created by loggers) to the appropriate destination. 42 | > * __Filters__ provide a finer grained facility for determining which log records to output. 43 | > * __Formatters__ specify the layout of log records in the final output. 44 | 45 | We pass information between loggers, handlers, and formatters in a `LogRecord` instance, which is created automatically by the `Logger` each time something is logged 46 | 47 | * Loggers perform the actual logging 48 | * module-level logger: `logger = logging.getLogger(__name__)` 49 | * Can save logs to different destinations using the handler 50 | * default is none 51 | * if no location is set, will go to `stderr` 52 | 53 | #### Logging Process Flow 54 | 55 | 56 | 57 | #### [Loggers](https://docs.python.org/3/howto/logging.html#loggers) 58 | 59 | * expose methods that application can use to log 60 | * determine which log messages to act on based upon severity 61 | * [logging level](https://docs.python.org/3/howto/logging.html#logging-levels) 62 | * default is `WARNING` 63 | * pass relevant log messages to log 64 | * configure handlers for top level logger and create child loggers as needed 65 | 66 | * `.setLevel()` specifies the lowest severity message logger will handle 67 | * `.addHandler()` / `.removeHandler()` to add/remove handler objects 68 | * `.addFilter()` / `.removeFilter()` to add/remove filter objects 69 | 70 | #### [Handlers](https://docs.python.org/3/howto/logging.html#handlers) 71 | 72 | * dispatch log message to handler's specified destination 73 | * can have multiple handlers 74 | * [useful handlers](https://docs.python.org/3/howto/logging.html#useful-handlers) 75 | 76 | #### [Formatters](https://docs.python.org/3/howto/logging.html#formatters) 77 | 78 | * configure how the log message looks 79 | 80 | `logging.Formatter.__init__(fmt=None, datefmt=None, style='%')` 81 | 82 | #### [Configuring Logging](https://docs.python.org/3/howto/logging.html#configuring-logging) 83 | 84 | > 1. Creating loggers, handlers, and formatters explicitly using Python code that calls the configuration methods listed above. 85 | > 1. Creating a logging config file and reading it using the `fileConfig()` function. 86 | > 1. Creating a dictionary of configuration information and passing it to the `dictConfig()` function. 87 | 88 | #### [Optimization](https://docs.python.org/3/howto/logging.html#optimization) 89 | 90 | * `logger.isEnabledFor(logging.LEVEL)` is a useful check to make sure we don't have to do expensive strip interpolations 91 | 92 | --- 93 | 94 | ## [Logging Cookbook](https://docs.python.org/3/howto/logging-cookbook.html) 95 | 96 | * Lots of great recipes 97 | * [Adding Contextual Information to Logging Output](https://docs.python.org/3/howto/logging-cookbook.html#adding-contextual-information-to-your-logging-output) 98 | 99 | --- 100 | 101 | ## [`logging`](https://docs.python.org/3/library/logging.html#module-logging) 102 | 103 | * [`LogRecord`](https://docs.python.org/3/library/logging.html#logrecord-objects) 104 | * [Configuration dictionary schema](https://docs.python.org/3/library/logging.config.html#configuration-dictionary-schema) 105 | * [Handlers](https://docs.python.org/3/library/logging.handlers.html) 106 | 107 | --- 108 | 109 | ## [PEP 282 -- A Logging System](https://www.python.org/dev/peps/pep-0282/) 110 | 111 | * A registery of [singleton](https://en.wikipedia.org/wiki/Singleton_pattern) logger object holds references to all initialized loggers 112 | 113 | ### Control Flow 114 | 115 | > Applications make logging calls on Logger objects. Loggers are organized in a hierarchical namespace and child Loggers inherit some logging properties from their parents in the namespace. 116 | > 117 | > These Logger objects create __LogRecord__ objects which are passed to __Handler__ objects for output. Both Loggers and Handlers may use logging __levels__ and (optionally) __Filters__ to decide if they are interested in a particular LogRecord. When it is necessary to output a LogRecord externally, a Handler can (optionally) use a __Formatter__ to localize and format the message before sending it to an I/O stream. 118 | > 119 | > The main benefit of a logging system like this is that one can control how much and what logging output one gets from an application without changing that application's source code. Therefore, although configuration can be performed through the logging API, it must also be possible to change the logging configuration without changing an application at all. For long-running programs like Zope, it should be possible to change the logging configuration while the program is running. 120 | 121 | Does this go against the principles of 12 factor? 122 | 123 | > The most simple configuration is that of a single handler, writing to stderr, attached to the root logger. This configuration is set up by calling the basicConfig() function once the logging module has been imported. 124 | > 125 | > To support use of the logging mechanism in short scripts and small applications, module-level functions debug(), info(), warn(), error(), critical() and exception() are provided. These work in the same way as the correspondingly named methods of Logger - in fact they delegate to the corresponding methods on the root logger. A further convenience provided by these functions is that if no configuration has been done, basicConfig() is automatically called. 126 | 127 | * At application exit we can flush all hanlers using `shutdown()` 128 | -------------------------------------------------------------------------------- /scala/scala_for_the_impatient.md: -------------------------------------------------------------------------------- 1 | # Scala for the Impatient (2nd Edition) 2 | 3 | by Cay S. Horstmann 4 | 5 | 6 | 7 | - [Chapter 1: The Basics](#chapter-1-the-basics) 8 | - [Chapter 2: Control Structures and Functions](#chapter-2-control-structures-and-functions) 9 | 10 | 11 | 12 | --- 13 | --- 14 | 15 | ## Chapter 1: The Basics 16 | 17 | * `val` is for constant, `var` is for variable 18 | * use `val` unless you need to change contents 19 | * do not need to specify type of value or variable, it is inferred from the expression type 20 | * semicolons are only required if you have multiple statements on the same line 21 | 22 | * Scala has 7 numeric types: 23 | * `Byte`, `Char`, `Short`, `Int`, `Long`, `Float`, `Double`, and `Boolean` 24 | 25 | * operators are actualy methods 26 | ```java 27 | a + b 28 | 29 | // same as 30 | a.+(b) 31 | ``` 32 | 33 | * does not have `++` or `--` operators, use `+=1` or `-=1` 34 | * *Rule of Thumb:* parameterless method that doesn't modify the object has no parentheses 35 | 36 | --- 37 | 38 | ## Chapter 2: Control Structures and Functions 39 | 40 | * 41 | -------------------------------------------------------------------------------- /snippets/docker.md: -------------------------------------------------------------------------------- 1 | # Docker Snippets 2 | 3 | ## Remove stopped containers 4 | 5 | `docker system prune -af` 6 | -------------------------------------------------------------------------------- /snippets/domain_modeling.md: -------------------------------------------------------------------------------- 1 | # Domain Modeling 2 | 3 | http://web.cerritos.edu/dwhitney/SitePages/CIS201/LectureNotesOnTalonNet/Chapter04Lecture.pdf 4 | 5 | ## Brainstorming Technique 6 | 7 | 1. Identify a user and a set of use cases. 8 | 2. Brainstorm with the user to identify things involved when carrying out the use case—that is, things about which information should be captured by the system. 9 | 3. Use the types of things (categories) to systematically ask questions about potential things, such as the following: Are there any tangible things you store information about? Are there any locations involved? Are there roles played by people that you need to remember? 10 | 4. Continue to work with all types of users and stakeholders to expand the brainstorming list. 11 | 5. Merge the results, eliminate any duplicates, and compile an initial list. 12 | 13 | ## Noun Technique 14 | 15 | 1. Using the use cases, actors, and other information about the system—including inputs and outputs—identify all nouns. 16 | 2. Using other information from existing systems, current procedures, and current reports or forms, add items or categories of information needed. 17 | 3. As this list of nouns builds, you will need to refine it. Ask these questions about each noun to help you decide whether you should include it: 18 | * Ask questions, such as is it important and inside the scope, to see if it should be included. 19 | * Ask questions to see if it really should be excluded, such as is it only a report or an input or is it an attribute. 20 | * Ask questions to see if it needs further research. In other words that you cannot answer whether it needs to be included or excluded. 21 | 4. Create a master list of all nouns identified and then note whether each one should be included, excluded, or researched further. 22 | 5. Review the list with users, stakeholders, and team members and then refine the list of things in the problem domain. 23 | -------------------------------------------------------------------------------- /snippets/git.md: -------------------------------------------------------------------------------- 1 | # Git Snippets 2 | 3 | ## Limiting changes to master 4 | 5 | ```console 6 | git checkout master 7 | git pull 8 | git checkout 9 | git rebase master 10 | ``` 11 | -------------------------------------------------------------------------------- /snippets/kubernetes.md: -------------------------------------------------------------------------------- 1 | # Kubernetes (k8s) 2 | 3 | - orchestrator with master and nodes 4 | - master looks after cluster 5 | - apiserver 6 | - api or `kubectl` 7 | - cluster store 8 | - persist store about cluster state and config 9 | - state of truth for cluster 10 | - controller manager 11 | - node, endpoints, namespace controllers 12 | - watches for changes 13 | - maintains desired state 14 | - kube-scheduler 15 | - watches apiserver for new pods and assigns work to nodes 16 | - nodes run workloads 17 | - kubelet 18 | - main kubernetes agent 19 | - registers node with cluster 20 | - watches apiserver 21 | - instantiates pods 22 | - reports back to master 23 | - exposes endpoints on `:10255` 24 | - container engine 25 | - does container management 26 | - kube-proxy 27 | - k8s networking 28 | - 1 IP per pod 29 | - load balances across all pods in a service 30 | - declarative model 31 | - give master a manifest file that describes the desired state 32 | - pods 33 | - atomic unit of scheduling and scaling 34 | - containers always run inside of pods 35 | - can have multiple containers (main container, sidecar container) 36 | - ring-fenced environment 37 | - network stack 38 | - kernel namespace 39 | - all containers in a pod share the pod environment 40 | - if a pod dies, a new pod is spun up (has a different IP) 41 | - services 42 | - we can't rely on pod IPs so we use services 43 | - provides a stable IP and DNS name 44 | - a higher level stable abstraction point for pods that provide load balancing 45 | - pod belongs to service via `label` 46 | - only send traffic to healthy pod 47 | - deployments 48 | - declarative description of how we want our architecture to look 49 | - simple rolling updates and rollbacks 50 | -------------------------------------------------------------------------------- /snippets/python.md: -------------------------------------------------------------------------------- 1 | # Python Snippets 2 | 3 | ## Testing 4 | 5 | * Running coverage with pytest 6 | 7 | ```console 8 | $ py.test --cov=backend tests/ 9 | ``` 10 | 11 | ## Debugging 12 | 13 | ### Q 14 | 15 | [Github](https://github.com/zestyping/q) 16 | 17 | ```python 18 | # print 19 | import q; q(foo) 20 | 21 | # trace function with decorator 22 | @q.t 23 | 24 | # interactive shell 25 | import q; q.d() 26 | ``` 27 | 28 | 29 | Output goes to `$TEMPDIR\q` or `\tmp\q` 30 | 31 | ```console 32 | tail -f $TMPDIR\q 33 | ``` 34 | 35 | 36 | Created an Bash Function 37 | ```console 38 | # connect to q log file for python debugging 39 | q () 40 | { 41 | if [ -z $TMPDIR ]; then tail -f \tmp\q; else tail -f $TMPDIR\q; fi 42 | } 43 | ``` 44 | 45 | 46 | ### IPython 47 | 48 | ```python 49 | from IPython import embed; embed() # interactive shell with IPython 50 | ``` 51 | 52 | ### Pdb 53 | 54 | ```python 55 | import ipdb 56 | ipdb.set_trace() 57 | ``` 58 | -------------------------------------------------------------------------------- /software-engineering/apprenticeship_patterns.md: -------------------------------------------------------------------------------- 1 | # Apprenticeship Patterns 2 | 3 | by David H Hoover & Adewale Oshineye 4 | -------------------------------------------------------------------------------- /software-engineering/design_patterns.md: -------------------------------------------------------------------------------- 1 | # Design Patterns 2 | Elements of Reusable Object-Oriented Software 3 | 4 | By Erich Gamma, Richard Helm, Ralph Johnson, John Vlissides 5 | 6 | #### Table of Contents 7 | 8 | 9 | 10 | - [Chapter 1: Introduction](#chapter-1-introduction) 11 | - [What is a Design Pattern?](#what-is-a-design-pattern) 12 | - [Organizing the Catalog](#organizing-the-catalog) 13 | - [How Design Patterns Solve Design Problems](#how-design-patterns-solve-design-problems) 14 | - [Conclusion](#conclusion) 15 | - [Facade](#facade) 16 | - [Uses](#uses) 17 | - [Collaborations](#collaborations) 18 | - [Consequences](#consequences) 19 | 20 | 21 | 22 | ## Chapter 1: Introduction 23 | 24 | ### What is a Design Pattern? 25 | 26 | A pattern has 4 essential elements: 27 | 28 | 1. **pattern name** is a handle we can use to describe a design problem, its solutions, and consequences in a word or two. It lets us design at a higher level of abstraction 29 | 2. **problem** describes when to apply the patter 30 | 3. **solution** describes the element that makes up the design, their relationships, responsibilities, and collaborations 31 | 4. **consequences** are the results and trade-offs of apply the pattern. Though consequences are often unvoiced when we describe design decisions, they are critical for evaluating design alternatives and for understanding the costs and benefits of apply the pattern 32 | 33 | - design patterns are descriptions of communicating objects and classes that are customized to solve a general design problem in a particular context 34 | 35 | ### Organizing the Catalog 36 | 37 | #### Purpose 38 | 39 | Reflects what a pattern does 40 | 41 | **creational** - cocnern the process of object creation 42 | **structural** - deal with the composition of classes or objects 43 | **behavioral** -- characterize the ways in which classes or objects interact and distribute responsibility 44 | 45 | #### Scope 46 | 47 | Specifies whether the pattern applies primarily to classes or to objects 48 | 49 | - **class patterns** are established through inheritance so they are fixed at compile time 50 | - **object patterns** deal with object relationships which are changed at run-time and are more dynamic 51 | 52 | ### How Design Patterns Solve Design Problems 53 | 54 | #### Finding Appropriate Objects 55 | 56 | - **object** packages both data and procedures that operate on that data 57 | - procedures are called **methods** or **operations** 58 | - an object performs an operation when it receives a **request** (or **message**) from a client 59 | - requests are the only way to get an object to execute an operation 60 | - operations are the only way to change an object's internal data 61 | - this means that an object's internal state is **encapsulated** - it cannot be accessed directly, and its representation is invisible from outside the object 62 | 63 | #### Specifying Object Interfaces 64 | 65 | - the set of all signatures defined by an object's operations is called the **interface** to the object 66 | - interfaces are fundamental in object-oriented systems; objects are only known thru their interface. There is no way to know anything about an object or to ask it to do anything without going through its interace 67 | - run-time association of a request to an object and one of its operations is known as **dynamic binding** 68 | - issuing a request doesn't commit you to a particular implementation until run-time 69 | - you can substitute objects with identical interfaces for each other at run-time - **polymorphism** 70 | - design patterns help you define interfaces by identifying their key elements and the kinds of data that get sent across an interface 71 | - design patterns also specify relationships between interfaces 72 | 73 | #### Class versus Interface Inheritance 74 | 75 | - class inheritance defines an object's implemtnation in terms of another object's implemtnation - it's a mechanism for code and representation sharing 76 | - interface inheritance (or subtyping) describes when an object can be used in place of another 77 | 78 | #### Programming to an Interface, not an Implementation 79 | 80 | - class inheritance is basically just a mechanism for extending an application's functionality by reusing functionality in parent classes 81 | - you can define a new kind of object rapidly in terms of an old man 82 | - get new implementations almost for free, inheriting most of what you need from existing classes 83 | - when inheritance is used carefully, all classes derived from an abstract class will share its interface 84 | - this implies a subclass merely adds or overrides operations and does not hide operations of the parent class 85 | - all subclasses can respond to requests in the interface of this abstract class, making t hem all subtypes of the abstract class 86 | - benefits of using the interface 87 | - clients remain unaware of the specific types of objects they use, as long as the objects adhere to the interface that clients expect 88 | - clients remain unaware of the classes that implement these objects = clients only know about the abstract classes defining the interface 89 | - variables should be defined by an abstract class, or interface 90 | 91 | #### Interitance versus Composition 92 | 93 | - object composition is an alternative to class inheritance - new functionality is obtained by assembling or composing objects to get more complex functionality 94 | - class inheritance is defined statically at compile-time; it makes it easy to modify the implementation being reused 95 | - "inheritance breaks encapsulation" - inheritance exposes a subclass to details of its parent's implemtnation 96 | - implementation dependencies can cause problems when you're trying to reuse a subclass - this limits flexibility and ultimately reusability 97 | - object composition is defined dynamically at run-time through objects acquiring references to other objects 98 | - any object can be replaced at run-time by another as long as it has the same type 99 | - favuoring object composition over class inheritance helps keep each class encapsulated and focused on one task 100 | 101 | #### Delegation 102 | 103 | - a way of making composition as powerful for reuse as inheritance 104 | - two objects are involved in handling a request: a receiving object delates operations to its delegate 105 | - the receiver passes itself to the delegate to let the delegated operation refer to the receiver 106 | - complex abstractions are harder to understand than static software 107 | 108 | #### Inheritance versus Parameterized Types 109 | 110 | - object composition lets you change the behavior being composed at run-time, but it also requires indirection and can be less efficient 111 | - inheritance lets you provide default implementations for operations and lets subclasses override them 112 | - parameterized types let you change the types that a class can use 113 | - special kind of generic 114 | 115 | #### Relating Run-Time and Compile-Time Structures 116 | 117 | - An object-oriented program's run-time structure often bears little resemblance to its code structure 118 | - code structure is frozen at compile-time, it consists of classes in fixed inheritance relationships 119 | - program's run-time strucutre consists of rapidly changing network of communicating ojbects 120 | - relationships between objects and their types must be designed with great care, because they determine how good or bad the run-time structure is 121 | 122 | #### Designing for Change 123 | 124 | - classes that are tightly coupled are hard to reuse in isolation, since they depend on each other 125 | - time coupling leads to monolithic systems where you cant' cahnge or remove a class without understanding and changing many other classes 126 | - lose coupling increases the probability that a class can be reused by itself 127 | - design patterns use techniques such as abstract coupling and layering to promote loosely coupled systems 128 | - looser coupling boosts the likelihood that one class of object can cooperate with several others 129 | - design patterns make an application more maintainable when they've used to limit platform dependencies and to layer a system 130 | 131 | ### Conclusion 132 | 133 | - design patterns should not be applied indiscriminately - often they achieve flexibility and variability by introducing additional levels of indirection, and that can complicate a design and/or cost you some performance 134 | - a design pattern should only be applied when the flexibility it affords is actually needed 135 | 136 | ## Facade 137 | 138 | - defines a higher-level interface that makes the subsystem easier to use 139 | - make life easier for most programmers without hiding the lower-level functionality from the few that need it 140 | 141 | ### Uses 142 | 143 | - provide a simple interface to a complex subsystem 144 | - decouple subsytem from clients 145 | - use a facade to define layers to your subsystem 146 | 147 | ### Collaborations 148 | 149 | - clients communicate with the subsystem by sending requests to Facade which forwards them to the appropriate subsystem objects; might have to do work to translate its interface to subsystem interfaces 150 | - clients that use facade don't have access to its subsystems objects directly 151 | 152 | ### Consequences 153 | 154 | - shields clients from subsystem components, reducing the number of objects that a client deals with / makes subsystem easier to use 155 | - promotes weak coupling between subsystem and its clients 156 | -------------------------------------------------------------------------------- /software-engineering/domain_driven_design.md: -------------------------------------------------------------------------------- 1 | # Domain-Driven Design 2 | 3 | by Eric Evans 4 | 5 | 6 | 7 | - [Preface](#preface) 8 | 9 | 10 | 11 | --- 12 | --- 13 | 14 | ## Preface 15 | 16 | > When complexity gets out of hand, developers can no longer understand the software well enough to change or extend it easily and safely. On the other hand, a good design can create opportunities to exploit those complex features. 17 | 18 | Book's main themes: 19 | 1. For most software projects, the primary focus should be on the domain and domain logic. 20 | 2. Complex domain designs should be based on a model. 21 | 22 | -------------------------------------------------------------------------------- /software-engineering/refactoring.md: -------------------------------------------------------------------------------- 1 | # Refactoring (2nd Edition) 2 | Improving the Design of Existing Code 3 | 4 | By Martin Fowler 5 | 6 | 7 | 8 | - [Preface](#preface) 9 | - [Chapter 1: Clean Code](#chapter-1-clean-code) 10 | 11 | 12 | 13 | --- 14 | --- 15 | 16 | ## Preface 17 | 18 | > Refactoring is the process of changing a software system in a way that does not alter the external behavior of the code yet improves its internal structure. It is a disciplined way to clean up code that minimizes the chances of introducing bugs . In essense, when you refactor, you are improving the design of the code after it has been written. 19 | 20 | ## Chapter 1: Clean Code 21 | 22 | asdfsdf 23 | 24 | 25 | -------------------------------------------------------------------------------- /testing/developer_testing_bootcamp.md: -------------------------------------------------------------------------------- 1 | # Developer Testing Bootcamp 2 | 3 | by Melvin Perez-Cedano 4 | Construx Software Training 5 | 6 | * We test to reduce the gap between the time when defects are intorduced into the code and the time when those defects are detected 7 | 8 | ## Adopt a Tester's Mindset 9 | 10 | * We need a good definition of testing 11 | * BDD - behavior driven development. Gerkin: [Given When Then](https://martinfowler.com/bliki/GivenWhenThen.html) 12 | 13 | ## Test Early; Test Often 14 | 15 | ## Automate your Tests 16 | 17 | * use a framework 18 | * don't build tests that test your business logic via the UI 19 | * UI is always changing so we should have less of these tests 20 | 21 | ## Grow Code with Tests 22 | 23 | * TDD 24 | 25 | ## Integrate Often 26 | 27 | * CI 28 | 29 | ## Ensure Functional Coverage 30 | 31 | * Design-by-Contract 32 | * Decision Tables 33 | 34 | ## Ensure Domain Coverage 35 | 36 | * Equivalence Partitioning 37 | * Boundary-Value Analysis 38 | * Pairwise Testing 39 | 40 | ## Ensure Code Coverage 41 | 42 | * Statement Coverage 43 | * Branch Coverage 44 | * Basis Path Coverage 45 | * Multiple Condition Coverage 46 | * MCDC Coverage 47 | 48 | ## Design for Testability 49 | 50 | * Simplify 51 | * Make Dependencies Replaceable 52 | * Test with Doubles 53 | 54 | * dependency lookup for higher level tests (integration tests) 55 | * check a config file 56 | 57 | ## Write Test Code with Care 58 | 59 | * Keep your Test Code Clean 60 | * Test your tests 61 | -------------------------------------------------------------------------------- /testing/mock.md: -------------------------------------------------------------------------------- 1 | # Mock Notes 2 | 3 | ## Michael Foord (creator of mock library) 4 | 5 | ### [Part 1](https://www.youtube.com/watch?v=d3_LdLzWSXQ) 6 | 7 | * Mock is a library to create mock (fake) objects that can be used to replace parts of your system under test. 8 | * Mock is a spy, fake, double object 9 | * Mock follows Arrange, Act, Assert pattern of testing 10 | * Arrange - set up system under test 11 | * Act - call the system under test 12 | * Assert - make sure the right things have happened 13 | 14 | * `from unittest.mock import:` 15 | * MagicMock (like Mock, but supports `__magic__` methods) 16 | * patch 17 | * call 18 | * ANY 19 | 20 | * Mock object are mocks all the way down 21 | * access attribute, returns mock object 22 | * call a function, returns a mock object 23 | 24 | * Mock objects track their usage (`.called`, `.call_count`, `.mock_calls`) 25 | * pass a mock object into our system under test 26 | * system uses it in a certain way 27 | * we make assertions about how it was used 28 | 29 | * PyCon 2016 mock talk by ana something 30 | * cause of truthy value, make sure your asserts are right 31 | * but we can use spec=True, to make sure we aren't magically passing tests 32 | 33 | ### [Part 2](https://www.youtube.com/watch?v=ADIcwu2GeEU) 34 | 35 | * Why use mock? 36 | * Isolation 37 | * Determinism (deterministic models produce the same output from a given startinc condition or initial state) 38 | * make test rliable, producing the same answer every time 39 | * Speed / Performance 40 | * Avoiding dependencies on external systems 41 | 42 | * Unit tests are isolated, need to be fast, and need useful tracebacks 43 | 44 | * Use mocks objects to fake out parts of an external API to make tests deterministic and compare to known preprocessed results 45 | 46 | ### [Part 3](https://www.youtube.com/watch?v=yFA-FFaEZPo) 47 | 48 | * Django test runner starts up a test database. All this time adds up 49 | * Use mocks to mock out things like the database 50 | * patch lets you replace objects in your system with mock objects 51 | * monkeypatching 52 | * easy to do, don't forget to replace 53 | * context manager is a good way because it has a closing step 54 | * and you control scope of patch 55 | * function decorator 56 | * class decorator, applies same patch to every single method in the class 57 | * but does it at creation time, decorator is not reapplied 58 | * Step to do mock 59 | * figure out how the object will be use 60 | * mock that behavior 61 | * use the mock object as it should be 62 | * assert it was used the way you intended 63 | 64 | * What to assert 65 | * response.status_code 66 | * assertIn('text', response.content) 67 | * use [pyquery](https://pythonhosted.org/pyquery/), jquery-esque lib from python, to test against actual stuff 68 | 69 | * patch is interesting 70 | * do the patch where the name is looked up 71 | * [timestamp](https://youtu.be/yFA-FFaEZPo?t=16m) 72 | 73 | #### Question 74 | 75 | * Should we test render? It's a django function after all 76 | * BUT YES we should because templates have logic 77 | * If you have branches for different templates, make sure you test those too 78 | 79 | ## [Stop Mocking, Start Testing](https://www.youtube.com/watch?v=Xu5EhKVZdV8) 80 | 81 | * dependency injection does not mean this 82 | 83 | ```python 84 | def my_func(database=None): 85 | if database is None: 86 | # construction 87 | ``` 88 | 89 | ## Fake It Til You Make It: Unit Testing Patterns With Mocks and Fakes 90 | 91 | [YouTube link](https://www.youtube.com/watch?v=hvPYuqzTPIk&t=20s) 92 | 93 | 94 | ## [The Tao of Testing](http://jasonpolites.github.io/tao-of-testing/index-1.1.html) 95 | 96 | * unit test - Test a single public method or function 97 | * black box test - test provides data and expects certain results without knowing how those results are derived 98 | * integration test - many parts of the system are tested at once, tests a complete end-to-end lifecycle of an application or a portion versus testing selected components independently 99 | * manual test - test completed by human 100 | * load/scale test - impose stress on application by simulating heavy load conditions / highly concurrent load conditions 101 | * infrastructure test - testing infrastructure underlying software 102 | * QA test - like manual tests 103 | * UAT (user acceptance testing) - human test assess implementation of software to see if it meets specifications 104 | 105 | ### [A Case for Testing](http://jasonpolites.github.io/tao-of-testing/ch1-1.1.html) 106 | 107 | * Bugs will happen, you can bank on it, only the severity of their impact will vary. 108 | * Bug incidents can increase dramatically as new features are added 109 | * Bug fix times can increase dramatically as new features are added 110 | * Software testing is NOT designed to make sure code works when it’s written but rather to ensure future changes do not break existing code. 111 | * A well run, tightly managed testing regime leads to more stable, reusable code. 112 | 113 | ### [Chapter 4 - How to Write Tests](http://jasonpolites.github.io/tao-of-testing/ch4-1.1.html#guidebook-for-writing-tests) 114 | 115 | #### Mocking dependencies to create isolated tests 116 | 117 | > A mock object is an object that implements the same signature as the object it is mocking, but does not possess any of the actual implementation. That is, as far as the application is concerned it “looks” like the real thing, but it’s actually a fake. 118 | > 119 | > The behavior of the mock object is orchestrated as part of the test so that it produces predictable outputs. 120 | 121 | 1. Mocks of real objects are easily created without needing to implement all required methods 122 | 1. Test conditions are easily orchestrated by setting expectations on mocks 123 | 1. Mocks allow assertions of methods called and method parameters as well as methods NOT called. 124 | 125 | 126 | 127 | 128 | ## To Read 129 | 130 | * [Fake It Til You Make It: Unit Testing Patterns With Mocks and Fakes](https://www.youtube.com/watch?v=hvPYuqzTPIk) 131 | * [Stop Mocking, Start Testing](https://www.youtube.com/watch?v=Xu5EhKVZdV8) 132 | 133 | -------------------------------------------------------------------------------- /unix/vim_cheatsheet.md: -------------------------------------------------------------------------------- 1 | # Vim commands 2 | 3 | ## Repeat Factor 4 | 5 | * Can hit # before command, do command. It will be repeated x times 6 | 7 | ## Insert Mode 8 | 9 | Mode where we enter text into the editor. This isn't what we are always supposed to do. 10 | 11 | * [Ctrl-w] to delete last word 12 | * [Ctrl-n] - activate completion 13 | * [Ctrl-n] / [Ctrl-p] | arrow keys - to move thru list 14 | 15 | ### Getting into Insert Mode 16 | 17 | * i - insert before cursor 18 | * I - insert at start of line 19 | 20 | * a - insert after cursor 21 | * A - insert at end of line 22 | 23 | * o - open new line after cursor 24 | * O - open new line before cursor 25 | 26 | * r - replace single character, stay in Commande Mode 27 | * R - replace more than a single character 28 | 29 | * s - replace current character and go into insert 30 | * S - delete line and go into insert 31 | 32 | * c - change operator 33 | 34 | ## Normal Mode 35 | 36 | This is the mode you are supposed to be in. It's **Normal** operation. 37 | 38 | ### Navigation 39 | 40 | For performing navigation and cut/paste operations. 41 | 42 | * h j k l - navigation keys 43 | * 0 - beginning of line 44 | * ^ - first chracater (nonblank character) 45 | * gm - move to middle of line 46 | * $ - end of line 47 | 48 | * Fx - previous x on line 49 | * fx - next x on line 50 | * ; - repeats character jump in same direction 51 | * , - repeats chracxter jump in other direction 52 | 53 | * b - moves back to beginning of word 54 | * e - moves to end of word 55 | * w - moves to beginning of word 56 | * B E W - skip punctuation 57 | 58 | * #G - goto line # (default is 1) 59 | 60 | * H / M / L - high / medium / low points on screen 61 | * zt / zz / zb - Move current line to top / middle / bottom of screen 62 | 63 | * [Ctrl-g] - get line number 64 | 65 | ### Text Navigation 66 | 67 | * { / } - move to start / end of paragrah 68 | * ( / ) - move to start / end of sentence 69 | 70 | ### Page Navigation 71 | 72 | * [Ctrl-f] - page down 73 | * [Ctrl-b] - page up 74 | 75 | * [Ctrl-d] - half-page down 76 | * [Ctrl-u] - half-page up 77 | 78 | * [Ctrl-e] - scroll one line down 79 | * [Ctrl-y] - scroll one line up 80 | 81 | ### Marks 82 | 83 | Disappear when file is close. 84 | 85 | * m[letter] - creates bookmark 86 | * '[ltter] - goto bookmark 87 | 88 | * lowercase - local 89 | * uppercase - global 90 | 91 | * Can combine marks with operators so delete till mark: ```d`[char]``` 92 | 93 | * '0 - goes to last line before vim closed 94 | * '1 - goes to last line before vim was closed two times ago 95 | 96 | * '' - jump back to previous spot 97 | * ', - jump to last edit 98 | * :marks - to see a list of marks 99 | 100 | ### Operators 101 | 102 | * d - delete operator (d$ deletes till end of line). Can also be used to cut text 103 | * D - deletes from cursor to end of line 104 | * y - yank (copy) 105 | 106 | * p - puts back taken by d after cursor 107 | * P - puts back taken by d before cursor 108 | 109 | * c - change operator (c$ changes all text until end of line) 110 | * C - change all text until end of line 111 | 112 | * ! - filter to act on text 113 | * x - deletes character under cursor 114 | * x - deletes previous character 115 | * J - joins current line with next line (removes new line char b/w two lines) 116 | 117 | * . - repeat last command 118 | 119 | * u - undo last edit 120 | * U - reverses changes made on a single line 121 | 122 | ### Search 123 | 124 | * /pat - searches forward for pattern pat 125 | * ?pat - searches backwards for pattern pat 126 | * n - repeats search in same direction 127 | * N - repeats search in opposite direction 128 | 129 | * If word is already selected: 130 | * ```*``` - search forward 131 | * # - search backward 132 | 133 | * f[char] - goto char on line going forward 134 | * F[char] - goto char on line going backward 135 | * t / T - do the same thing, but stop a character before 136 | 137 | * :iab ff Firefox - replace abbreviations with long words 138 | 139 | ## Folds 140 | 141 | TODO look into 142 | 143 | * :set foldmethod= 144 | * ```indent``` or ```syntax``` is probably best for now 145 | 146 | * z[x] - lots of fold commands 147 | * look into plugins 148 | 149 | ## Registers 150 | 151 | * Specify register you want to use before entering in command you would use anyways 152 | * "[char] - to access [char] register 153 | * a-zA-Z 154 | 155 | * "" - unnamed register (contained last deleted text) 156 | * "0p - always contains contents of last yank 157 | * "+ - system clipboard register 158 | 159 | ## Macros 160 | 161 | TODO lok into 162 | 163 | * qa 164 | * @a 165 | 166 | ## Visual Mode 167 | 168 | Mode where you can highlight text 169 | 170 | * V - Visual line mode, select lines 171 | * v - visual mode, select parts of text 172 | 173 | * ggVG - selects from first to last line 174 | 175 | ### Miscellaneous Commands 176 | 177 | * << / >> - indent / outdent if selected in ```VISUAL``` mode 178 | * = - format selection 179 | 180 | ## Last Line or ex Mode 181 | 182 | * :sh - Shell escape 183 | * :![cmd] - Run [cmd] 184 | * :# - Goes to line # 185 | 186 | * :n1,n2s/s1/s2 - replaces n1 with s1 and n2 with s2 187 | 188 | * [Ctrl-z] to go into shell 189 | * ```$ fg``` to bring background job to foreground 190 | 191 | * :e . - open file tree 192 | 193 | * :setpaste - Disables automatic indentation 194 | 195 | ## Window Management 196 | 197 | * :sp - split window 198 | * [Ctrl-w][Ctrl-w] - move between windows 199 | * [Ctrl-w][+/-] - make window bigger / smaller 200 | * :q - close active window 201 | * :on - close all other windows 202 | 203 | ## Buffers 204 | 205 | * multiple clipboards, figure out workflow 206 | -------------------------------------------------------------------------------- /unix/your_unix.md: -------------------------------------------------------------------------------- 1 | # Your Unix 2 | 3 | By Sumitabha Das 4 | 5 | ## Chapter 6: The File System 6 | 7 | * Everything in UNIX is a file 8 | 9 | Types of files: 10 | * ordinary file - contains data as a stream of characters 11 | * directory file - folder that contains the names of other files and directories 12 | * device file - represents a hardware device 13 | 14 | ### Directories 15 | 16 | * `/bin` and `/usr/bin` are where we keep commonly used user binaries 17 | * `/sbin` and `/usr/sbin` are for system admins 18 | * `/etc` holds system config files 19 | * `/dev` contains device files 20 | * `/home` is where all user files are 21 | * `/tmp` is best place to store temporary files 22 | * `/var` variable part of the files system 23 | * `/lib` is for library files 24 | 25 | * `$HOME` ENV VAR holds home directory 26 | 27 | `.` - current directory 28 | `..` - parent directory 29 | `./cmd` - Runs cmd in current directory (places it higher up in $PATH that if we just ran it without) 30 | -------------------------------------------------------------------------------- /web-development/12_factor_app.md: -------------------------------------------------------------------------------- 1 | # [The Twelve-Factor App](https://12factor.net/) 2 | 3 | ## I. [Codebase](https://12factor.net/codebase) 4 | 5 | > One codebase tracked in revision control, many deploys 6 | 7 | There is a one-to-one correlation between the codebase and the app. 8 | 9 | ## II. [Dependencies](https://12factor.net/dependencies) 10 | 11 | > Explicitly declare and isolate dependencies 12 | 13 | * Explicitly declare dependencies, don't rely on the implicit existence of site-packages; `pip` 14 | * Isolate dependencies from the rest of the system; `virtualenv` 15 | * Do not rely on the implicit existence of any system tools; might not exist on _all_ systems 16 | 17 | ## III. [Config](https://12factor.net/config) 18 | 19 | > Store config in the environment 20 | 21 | * __config__ is everything that can vary between different environments 22 | * Resource handles to the database 23 | * Credentials to external services 24 | * Hostname for deploy 25 | * strict separation of config from code 26 | * can we open source the code and not compromise credentials? 27 | * use config files that are not checked into version control 28 | * try to store config in environment variables 29 | 30 | ## IV. [Backing Services](https://12factor.net/backing-services) 31 | 32 | > Treat backing services as attached resources 33 | 34 | * __backing service__ is any service the app consumers over the networks 35 | * datastores 36 | * messaging / queueing systems 37 | * SMTP services for outbound email 38 | * caching systems 39 | * should not make any distinction between local and third party services 40 | * should be able to swap out local MySQL database with one production version without changing any code 41 | * should only have to change the config 42 | * resources can be attached to and detached from deploys at will; code / infrastructure should be able to handle 43 | 44 | ## V. [Build, release, run](https://12factor.net/build-release-run) 45 | 46 | > Strictly separate build and run stages 47 | 48 | * Three stages of deployment: 49 | * _build stage_ converts code repo into executable bundle (aka build) 50 | * _release stage_ takes build and combines with correct config; package is called release 51 | * _run stage_ run release in the execution environment by launching process 52 | * strict sepration between build, release, and run stages 53 | * every release should have a unique release ID so we can rollback if needed 54 | * build should be initiated everytime new code is deployed 55 | 56 | ## VI. [Processes](https://12factor.net/processes) 57 | 58 | > Execute the app as one or more stateless processes 59 | 60 | * App will be executed in the execution environment as one or more processes 61 | * Processes should be stateless and share nothing between instaces unless it uses a stateful backing service 62 | 63 | ## VII. [Port Binding](https://12factor.net/port-binding) 64 | 65 | > Export services via port binding 66 | 67 | * App is self-contained and does not require the injection of a webserver in order to create a service 68 | * Web app exports HTTP as a service by binding it to a port and listening to requests coming in on that port 69 | 70 | ## VIII. [Concurrency](https://12factor.net/concurrency) 71 | 72 | > Scale out via the process model 73 | 74 | * web apps usually have more than one process; server and worker (queue) 75 | * processes should be first class citizens 76 | * app must be able to span multiple processes running on multiple physical machines 77 | * makes it easier to scale out (aka scale horizontally) 78 | * need to reply on OS process manager to manage output streams and handling sysadmin work (restarts, shutdowns) 79 | 80 | ### Aside: [Unix Process Model](https://adam.herokuapp.com/past/2011/5/9/applying_the_unix_process_model_to_web_apps/) 81 | 82 | * Running program is a __process__ 83 | * can be started manually 84 | * run automatically; _managed process_ 85 | * __Process manager__ can put processes under management 86 | 87 | ## IX. [Disposability](https://12factor.net/disposability) 88 | 89 | > Maximize robustness with fast startup and graceful shutdown 90 | 91 | * Processes are _disposable_ and they can be started or stopped anytime 92 | * Should try to minimize startup time 93 | * Terminate gracefully when receiving a __`SIGTERM`__ from the process manage 94 | * Web apps should: 95 | * stop listening on service port 96 | * allow current requests to finish 97 | * exit 98 | * Worker processes should: 99 | * return current job to the work queue 100 | * jobs should be [reentrant](https://en.wikipedia.org/wiki/Reentrancy_(computing)) 101 | * function can stop during execution and restart without affect results 102 | * processes should be robust against suddent death 103 | 104 | ## X. [Development / Production Parity](https://12factor.net/dev-prod-parity) 105 | 106 | > Keep development, staging, and production as similar as possible 107 | 108 | * Gaps between development and production can be seen in three areas: 109 | 1. Time Gap -- a developer may work on code that takes days or longer to go into production 110 | 1. Personnel Gap - devs write code, ops deploy 111 | 1. Tools Gap - differences in development and production stacks 112 | * Continuous deployment requires a small gap between development and production enviornments 113 | * Try to use the same backing service across environments 114 | 115 | ## XI. [Logs](https://12factor.net/logs) 116 | 117 | > Treat logs as event streams 118 | 119 | * Logs provide visibility into the behavior of a running application; written to a file or streamed somewhere 120 | * App should not concern itself with routing or storage of output stream 121 | * Should event stream to `stdout` where it should be captured by the execution environment 122 | 123 | ## XII. [Admin Process](https://12factor.net/admin-processes) 124 | 125 | > Run admin/management tasks as one-off processes 126 | 127 | * Sometimes we need to do one-off administration or maintenance tasks 128 | * Ensure these processes are run in an identical environment as the app 129 | * Admin code must ship with application code to avoid sychronization issues 130 | -------------------------------------------------------------------------------- /web-development/api.md: -------------------------------------------------------------------------------- 1 | # Application Programming Interfaces 2 | 3 | ## [Best Practices for creating a well-designed API](TODO: link to service oriented architecture Alberta coursera) 4 | 5 | 1. Use only nounsd for your URI 6 | 1. Get methods should not alter the state of your resources 7 | 1. Use plural nouns for your URI 8 | 1. Use sub-resources for relationships between resources 9 | 1. Use HTTP headers to specify input/output format 10 | 1. Provide users with filtering and paging for collections 11 | 1. Version your API 12 | 1. Provide proper HTTP status codes 13 | -------------------------------------------------------------------------------- /web-development/graphql.md: -------------------------------------------------------------------------------- 1 | # GraphQL 2 | 3 | #### Table of Contents 4 | 5 | 6 | 7 | - [Queries and Mutations](#queries-and-mutations) 8 | - [Schemas and Types](#schemas-and-types) 9 | - [GraphQL Schema Language](#graphql-schema-language) 10 | - [Types](#types) 11 | - [Interfaces](#interfaces) 12 | - [Validation](#validation) 13 | - [Execution](#execution) 14 | - [Introspection](#introspection) 15 | - [Best Pratcies](#best-pratcies) 16 | - [Introduction](#introduction) 17 | - [Thinking in Graphs](#thinking-in-graphs) 18 | - [Pagination](#pagination) 19 | - [Global Object Identification](#global-object-identification) 20 | - [Caching](#caching) 21 | 22 | 23 | 24 | ## Queries and Mutations 25 | 26 | https://graphql.org/learn/queries/ 27 | 28 | > In a system like REST, you can only pass a single set of arguments - the query parameters and URL segments in your request. But in GraphQL, every field and nested object can get its own set of arguments, making GraphQL a complete replacement for making multiple API fetches. You can even pass arguments into scalar fields, to implement data transformations once on the server, instead of on every client separately. 29 | 30 | > While query fields are executed in parallel, mutation fields run in series, one after the other. 31 | 32 | ## Schemas and Types 33 | 34 | https://graphql.org/learn/schema/ 35 | 36 | > Every GraphQL service defines a set of types which completely describe the set of possible data you can query on that service. Then, when queries come in, they are validated and executed against that schema. 37 | 38 | ### GraphQL Schema Language 39 | 40 | > The most basic components of a GraphQL schema are object types, which just represent a kind of object you can fetch from your service, and what fields it has. 41 | 42 | ### Types 43 | 44 | GraphQL comes with a set of default scalar types out of the box: 45 | 46 | - `Int`: A signed 32‐bit integer. 47 | - `Float`: A signed double-precision floating-point value. 48 | - `String`: A UTF‐8 character sequence. 49 | - `Boolean`: true or false. 50 | - ID: The ID scalar type represents a unique identifier, often used to refetch an object or as the key for a cache. The ID type is serialized in the same way as a String; however, defining it as an ID signifies th`at` it is not intended to be human‐readable. 51 | - `Enums`: special kind of scalar that is restricted to a particular set of allowed values 52 | - `List`: `[]` 53 | - `Non-Null` (aka required): `!` 54 | 55 | ### Interfaces 56 | 57 | An Interface is an abstract type that includes a certain set of fields that a type must include to implement the interface. 58 | 59 | ## Validation 60 | 61 | https://graphql.org/learn/validation/ 62 | 63 | > By using the type system, it can be predetermined whether a GraphQL query is valid or not. This allows servers and clients to effectively inform developers when an invalid query has been created, without having to rely on runtime checks. 64 | 65 | > a fragment cannot refer to itself or create a cycle, as this could result in an unbounded result 66 | 67 | ## Execution 68 | 69 | https://graphql.org/learn/execution/ 70 | 71 | > You can think of each field in a GraphQL query as a function or method of the previous type which returns the next type. In fact, this is exactly how GraphQL works. Each field on each type is backed by a function called the resolver which is provided by the GraphQL server developer. When a field is executed, the corresponding resolver is called to produce the next value. 72 | > 73 | > If a field produces a scalar value like a string or number, then the execution completes. However if a field produces an object value then the query will contain another selection of fields which apply to that object. This continues until scalar values are reached. GraphQL queries always end at scalar values. 74 | 75 | > At the top level of every GraphQL server is a type that represents all of the possible entry points into the GraphQL API, it's often called the Root type or the Query type. 76 | 77 | ## Introspection 78 | 79 | https://graphql.org/learn/introspection/ 80 | 81 | > We designed the type system, so we know what types are available, but if we didn't, we can ask GraphQL, by querying the __schema field, always available on the root type of a Query. Let's do so now, and ask what types are available. 82 | 83 | ## Best Pratcies 84 | 85 | ### Introduction 86 | 87 | https://graphql.org/learn/best-practices/ 88 | 89 | - `Accept-Encoding: gzip` 90 | - While there's nothing that prevents a GraphQL service from being versioned just like any other REST API, GraphQL takes a strong opinion on avoiding versioning by providing the tools for the continuous evolution of a GraphQL schema. 91 | - in a GraphQL type system, every field is nullable by default 92 | 93 | ### Thinking in Graphs 94 | 95 | https://graphql.org/learn/thinking-in-graphs/ 96 | 97 | - model your business domain as a graph by defining a schema; within your schema, you define different types of nodes and how they connect/relate to one another 98 | - Your business logic layer should act as the single source of truth for enforcing business domain rules 99 | 100 | ### Pagination 101 | 102 | https://graphql.org/learn/pagination/ 103 | 104 | ### Global Object Identification 105 | 106 | https://graphql.org/learn/global-object-identification/ 107 | 108 | > The GraphQL schema is formatted to allow fetching any object via the node field on the root query object. This returns objects which conform to a "Node" interface. 109 | 110 | ### Caching 111 | 112 | https://graphql.org/learn/caching/ 113 | 114 | > In the same way that the URLs of a resource-based API provided a globally unique key, the id field in this system provides a globally unique key. 115 | 116 | > If the backend uses something like UUIDs for identifiers, then exposing this globally unique ID may be very straightforward! If the backend doesn't have a globally unique ID for every object already, the GraphQL layer might have to construct this. Oftentimes, that's as simple as appending the name of the type to the ID and using that as the identifier; the server might then make that ID opaque by base64-encoding it. 117 | -------------------------------------------------------------------------------- /web-development/http.md: -------------------------------------------------------------------------------- 1 | # [HTTP: Hypertext Transfer Protocol](https://developer.mozilla.org/en-US/docs/Web/HTTP) 2 | 3 | * [Application layer](https://en.wikipedia.org/wiki/Application_layer) protocol for transmitting hypermedia document 4 | * Application layer => abstraction layer containing communication protocols and interface methods 5 | 6 | * Follows [client-server model](https://en.wikipedia.org/wiki/Client%E2%80%93server_model) 7 | * client opens a connection to make a request 8 | * waits until it recieves a response 9 | 10 | * HTTP is [stateless](https://en.wikipedia.org/wiki/Stateless_protocol) 11 | * no information about the communication that occured is retained by either sender or reciver 12 | * Pros: No need to keep track of what happened, lost connections are not a problem 13 | * Cons: Sending extra information in each request that needs to be interpreted by the server 14 | 15 | ## [An Overview of HTTP](https://developer.mozilla.org/en-US/docs/Web/HTTP/Overview) 16 | 17 | * Basic idea: 18 | * a web document is made up of many different sub-documents 19 | * when a web browser goes to a page, it starts to fetch all the resources needed 20 | * once all the information is downloaded, the web browser can render the document as described 21 | 22 | **requests** - messages sent by the client 23 | **response** - messages sent by the server as an answer 24 | 25 | * Workflow: 26 | * *user-agent* sends a request 27 | * server processes request, sends back a response 28 | 29 | > To present a Web page, the browser sends an original request to fetch the HTML document from the page. It then parses this file, fetching additional requests corresponding to execution scripts, layout information (CSS) to display, and sub-resources contained within the page (usually images and videos). The Web browser then mixes these resources to present to the user a complete document, the Web page. Scripts executed by the browser can fetch more resources in later phases and the browser updates the Web page accordingly. 30 | 31 | * To the client, the server looks like a single machine. It "may actually be a collection of servers, sharing the load (load balancing) or a complex piece of software interrogating other computers (like cache, a DB server, e-commerce servers, …), totally or partially generating the document on demand." 32 | * Abstration is cool! 33 | 34 | * Proxies relay the HTTP messages between the client and server. Proxies can perform several functions such as: 35 | * caching (the cache can be public or private, like the browser cache) 36 | * filtering (like an antivirus scan, parental controls, …) 37 | * load balancing (to allow multiple servers to serve the different requests) 38 | * authentication (to control access to different resources) 39 | * logging (allowing the storage of historical information) 40 | 41 | * Note: Proxies must not alter requests methods, but they have the ability to change some headers. [More info](https://stackoverflow.com/questions/10369679/do-http-proxy-servers-modify-request-packets). 42 | 43 | > HTTP is stateless: there is no link between two requests being successively carried out on the same connection. This immediately has the prospect of being problematic for users attempting to interact with certain pages coherently, for example, using e-commerce shopping baskets. But while the core of HTTP itself is stateless, HTTP cookies allow the use of stateful sessions. Using header extensibility, HTTP Cookies are added to the workflow, allowing session creation on each HTTP request to share the same context, or the same state. 44 | 45 | ### HTTP Messages 46 | 47 | #### Request 48 | 49 | >Requests consists of the following elements: 50 | > * An HTTP method, usually a verb like GET, POST or a noun like OPTIONS or HEAD that defines the operation the client wants to perform. Typically, a client wants to fetch a resource (using GET) or post the value of an HTML form (using POST), though more operations may be needed in other cases. 51 | > * The path of the resource to fetch; the URL of the resource stripped from elements that are obvious from the context, for example without the protocol (http://), the domain (here developer.mozilla.org), or the TCP port (here 80). 52 | > * The version of the HTTP protocol. 53 | > * Optional headers that convey additional information for the servers. 54 | > * Or a body, for some methods like POST, similar to those in responses, which contain the resource sent. 55 | 56 | Example Request 57 | ```text 58 | GET / HTTP/1.1 59 | Host: developer.mozilla.org 60 | Accept-Language: fr 61 | ``` 62 | 63 | #### Response 64 | 65 | > Responses consist of the following elements: 66 | > * The version of the HTTP protocol they follow. 67 | > * A status code, indicating if the request has been successful, or not, and why. 68 | > * A status message, a non-authoritative short description of the status code. 69 | > * HTTP headers, like those for requests. 70 | > * Optionally, a body containing the fetched resource. 71 | 72 | Example Response 73 | ```text 74 | HTTP/1.1 200 OK 75 | Date: Sat, 09 Oct 2010 14:28:02 GMT 76 | Server: Apache 77 | Last-Modified: Tue, 01 Dec 2009 20:18:22 GMT 78 | ETag: "51142bc1-7449-479b075b2891b" 79 | Accept-Ranges: bytes 80 | Content-Length: 29769 81 | Content-Type: text/html 82 | 83 | Caching is a technique that stores a copy of a given resource and serves it back when requested. When a web cache has a requested resource in its store, it intercepts the request and returns its copy instead of re-downloading from the originating server. This achieves several goals: it eases the load of the server that doesn’t need to serve all clients itself, and it improves performance by being closer to the client, i.e., it takes less time to transmit the resource back. For a web site, it is a major component in achieving high performance. On the other side, it has to be configured properly as not all resources stay identical forever: it is important to cache a resource only until it changes, not longer. 89 | 90 | * [```Cache-control```](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control) is used in both requests and responses to specify directives (i.e. what can cache, for how long) for caching in both requests and responses 91 | 92 | * [304](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/304) (Not Modified) is what is sent when a cache wants to know if the requested resource is still "fresh" 93 | 94 | ## [HTTP Cookies](https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies) 95 | 96 | > An HTTP cookie (web cookie, browser cookie) is a small piece of data that a server sends to the user's web browser. The browser may store it and send it back with the next request to the same server. Typically, it's used to tell if two requests came from the same browser — keeping a user logged-in, for example. It remembers stateful information for the stateless HTTP protocol. 97 | > 98 | > Cookies are mainly used for three purposes: 99 | > * **Session management** - Logins, shopping carts, game scores, or anything else the server should remember 100 | > * **Personalization** - User preferences, themes, and other settings 101 | > * **Tracking** - Recording and analyzing user behavior 102 | 103 | * Cookies used to be used for storing client data, but now the recommended practice is to use [Web Storage API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Storage_API) or [IndexedDB API](https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API) 104 | * TODO 105 | 106 | * HttpOnly cookie attribute can help to mitigate cross-site scripting attacks by preventing access to cookie value through JavaScript 107 | 108 | ### Creating Cookies 109 | 110 | > When receiving an HTTP request, a server can send a Set-Cookie header with the response. The cookie is usually stored by the browser, and then the cookie is sent with requests made to the same server inside a Cookie HTTP header. An expiration date or duration can be specified, after which the cookie is no longer sent. Additionally, restrictions to a specific domain and path can be set, limiting where the cookie is sent. 111 | 112 | Example Response 113 | ```text 114 | HTTP/1.0 200 OK 115 | Content-type: text/html 116 | Set-Cookie: yummy_cookie=choco 117 | Set-Cookie: tasty_cookie=strawberry 118 | 119 | [page content] 120 | ``` 121 | 122 | Subsequent Request would be of the form: 123 | ```text 124 | GET /sample_page.html HTTP/1.1 125 | Host: www.example.org 126 | Cookie: yummy_cookie=choco; tasty_cookie=strawberry 127 | ``` 128 | 129 | ### Types of Cookies 130 | 131 | **Session cookie** - deleted when the client shuts down 132 | **Permanent cookie** - can set ```Expires``` or ```Max-Age``` to have it persist when client closes 133 | **Secure and HTTPOnly cookies** - only sent to the server with an encyrpted request over HTTPS 134 | 135 | **[Zombie cookie](https://en.wikipedia.org/wiki/Zombie_cookie)** - HTTP cookie that is recreated after deletion 136 | 137 | ### Scope of cookies 138 | 139 | > The ```Domain``` and ```Path``` directives define the scope of the cookie: what URLs the cookies should be sent to. 140 | 141 | ## [Cross-Origin Resource Sharing (CORS)](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) 142 | 143 | >Cross-Origin Resource Sharing (CORS) is a mechanism that uses additional HTTP headers to let a user agent gain permission to access selected resources from a server on a different origin (domain) than the site currently in use. A user agent makes a cross-origin HTTP request when it requests a resource from a different domain, protocol, or port than the one from which the current document originated. 144 | > 145 | > An example of a cross-origin request: A HTML page served from http://domain-a.com makes an src request for http://domain-b.com/image.jpg. Many pages on the web today load resources like CSS stylesheets, images, and scripts from separate domains, such as content delivery networks (CDNs). 146 | 147 | * TODO: Read more as needed 148 | 149 | ## [Evolution of HTTP](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Evolution_of_HTTP) 150 | 151 | Great read. 152 | 153 | ## [Security Guidelines](https://wiki.mozilla.org/Security/Guidelines/Web_Security) 154 | 155 | Lots of tips. 156 | 157 | TODO: work as you launch a website 158 | 159 | ## [HTTP Messages](https://developer.mozilla.org/en-US/docs/Web/HTTP/Messages) 160 | 161 | > HTTP messages are how data is exchanged between a server and a client. There are two types of messages: *requests* sent by the client to trigger an action on the server, and *responses*, the answer from the server. 162 | 163 | * Messages are text encoded in ASCII 164 | 165 | > HTTP requests, and responses, share similar structure and are composed of: 166 | > 1. A start-line describing the requests to be implemented, or its status of whether successful or a failure. This start-line is always a single line. 167 | > 1. An optional set of HTTP headers specifying the request, or describing the body included in the message. 168 | > 1. A blank line indicating all meta-information for the request have been sent. 169 | > 1. An optional body containing data associated with the request (like content of an HTML form), or the document associated with a response. The presence of the body and its size is specified by the start-line and HTTP headers. 170 | 171 | ### HTTP Request 172 | 173 | > HTTP requests are messages sent by the client to initiate an action on the server. 174 | 175 | #### Start line 176 | 177 | Contains three elements: 178 | 1. **HTTP method** - verb or noun that describes the action to be performed 179 | 2. **Request target** - URL or absolute path of resource 180 | 3. **HTTP version** - defines structure for rest of the reamining message and indicates the expected version to use for response 181 | 182 | #### Headers 183 | 184 | ```Header: value``` 185 | 186 | Different types of headers: 187 | * **General headers** - apply to message as whole 188 | * **Request headers** - give request context or conditionally restrict it 189 | * **Entity headers** - applies to the body of teh request (i.e. Content-Type) 190 | 191 | #### Body 192 | 193 | Not all requests have a body, but if we need to send data to the serve this is where it goes 194 | 195 | Two types: 196 | 1. **Single-resource bodies** - Single file defined by two headers (```Content-Type``` and ```Content-Length```) 197 | 1. **Multiple-resource bodies** - Multiparty body each containing different kind of information, associated with HTML Forms. 198 | 199 | ### HTTP Responses 200 | 201 | #### Status line 202 | 203 | Contains: 204 | 1. **Protocol version** 205 | 1. **Status code** - indiciation success or failure of request 206 | 1. **Status text** - Brief textual description of status code 207 | 208 | #### Headers 209 | 210 | ```Header: value``` 211 | 212 | Can be divided into several groups: 213 | * **General headers** - apply to whole message 214 | * **Response headers** - give additional information about server 215 | * **Entity headers** - apply to the body of the request 216 | 217 | #### Body 218 | 219 | Broadly divided into three categories: 220 | 1. **Single resource bodies of known length** - Defined by two headers (```Content-Type``` and ```Content-Length```) 221 | 2. **Single resource bodies of unknown length** - Encoded with chunks (```Transfer-Encoding:chunked```) 222 | 3. **Multiple resource bodies** - Consist of a multipart body each with different inforation 223 | 224 | ### HTTP/2 Frames 225 | 226 | * HTTP/2 introduces an extra step into the workflow, it divides the HTTP/1.x messages into frames which are then streamed. 227 | * Works when both browser and server have HTTP2 enabled. 228 | 229 | ## [A Typical HTTP Session](https://developer.mozilla.org/en-US/docs/Web/HTTP/Session) 230 | 231 | > Client-server sessions consist of three phases: 232 | > 1. The client establishes a TCP connection 233 | > 1. The client sends its request, and waits for the answer. 234 | > 1. The server processes the request, sending back its answer, providing a status code and appropriate data. 235 | 236 | > As of HTTP/1.1, the connection is no longer closed after completing the third phase, and the client is now granted a further request: this means the second and third phases can now be performed any number of times. 237 | 238 | ### Establishing a Connection 239 | 240 | * Client establishes a connection by initating a connection in the underlying transport layer 241 | 242 | ### Sending a client request 243 | 244 | The user-agent can send the request consisting of text directives, separated by CRLF (carriage return, followed by line feed), divided into three blocks: 245 | 1. First line contains request method and parameters (absolute path, HTTP protocol version) 246 | 1. Subsequent lines contain HTTP header 247 | 1. Optional data block 248 | 249 | ### Server response 250 | 251 | The web server processes the request and returns a reponse consisting of text directives, separated by CRLF (carriage return, followed by line feed), divided into three blocks: 252 | 1. Status line consisting of an acknowledgement of HTTP protocol, followed by a status request, and its meaning 253 | 1. Subsequent lines contain HTTP header 254 | 1. Optional data block 255 | 256 | ## [Connection Management in HTTP/1.x](https://developer.mozilla.org/en-US/docs/Web/HTTP/Connection_management_in_HTTP_1.x) 257 | 258 | 259 | 260 | ### Short-lived connections 261 | 262 | * Each HTTP request is completed in its own connection so a TCP handshake needs to occur before each request 263 | * Very slow since handshake process takes time 264 | 265 | ### Persistent connections (keep-alive connection) 266 | 267 | * TCP connection adapts to its load and becomes more efficient with sustained (aka warm) connections 268 | * Persistent connections remain open for a short period of time and can be reused for several requests 269 | * Idle connections are closed after some time 270 | * Drawbacks is that server resources are consumed and under heavy load, [DoS attacks](https://developer.mozilla.org/en-US/docs/Glossary/DoS_attack) can be conducted 271 | 272 | ### HTTP Pipelining 273 | 274 | Note: Not active by default in modern browsers 275 | 276 | * Process of sending successive reqeusts over the same persistent connection without waiting for an answer 277 | * Only idempotent methods (i.e. methods that don't change state of server when sent over and over again) such as ```GET```, ```HEAD```, ```PUT```, and ```DELETE``` 278 | 279 | ### Domain Sharding 280 | 281 | Deprecated, use HTTP/2 instead. 282 | 283 | ## [HTTP Headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers) 284 | 285 | > HTTP headers allow the client and the server to pass additional information with the request or the response. 286 | 287 | #### Types of Headers 288 | 289 | By context: 290 | 291 | * **General header** - Headers applying to both requests and responses but with no relation to the data eventually transmitted in the body. 292 | * **Request header** - Headers containing more information about the resource to be fetched or about the client itself. 293 | * **Response header** - Headers with additional information about the response, like its location or about the server itself (name and version etc.). 294 | * **Entity header** - Headers containing more information about the body of the entity, like its content length or its MIME-type. 295 | 296 | How proxies handle them: 297 | 298 | * **End-to-end headers** - These headers must be transmitted to the final recipient of the message; that is, the server for a request or the client for a response. Intermediate proxies must retransmit end-to-end headers unmodified and caches must store them. 299 | * **Hop-by-hop headers** - These headers are meaningful only for a single transport-level connection and must not be retransmitted by proxies or cached 300 | 301 | 302 | ```header: value``` 303 | 304 | ## [HTTP Requests Methods](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods) 305 | 306 | ### GET 307 | 308 | The GET method requests a representation of the specified resource. Requests using GET should only retrieve data. 309 | 310 | ### HEAD 311 | 312 | The HEAD method asks for a response identical to that of a GET request, but without the response body. 313 | 314 | ### POST 315 | 316 | The POST method is used to submit an entity to the specified resource, often causing a change in state or side effects on the server 317 | 318 | ### PUT 319 | 320 | The PUT method replaces all current representations of the target resource with the request payload. 321 | 322 | ### DELETE 323 | 324 | The DELETE method deletes the specified resource. 325 | 326 | ### CONNECT 327 | 328 | The CONNECT method establishes a tunnel to the server identified by the target resource. 329 | 330 | ### OPTIONS 331 | 332 | The OPTIONS method is used to describe the communication options for the target resource. 333 | 334 | ### TRACE 335 | 336 | The TRACE method performs a message loop-back test along the path to the target resource. 337 | 338 | ### PATCH 339 | 340 | The PATCH method is used to apply partial modifications to a resource. 341 | 342 | ## [HTTP Response Status Codes](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status) 343 | 344 | 100 - Information Responses 345 | 346 | 200 - Successful Responses 347 | 348 | 300 - Redirection Messages 349 | 350 | 400 - Client error reponses 351 | 352 | 500 - Server error responses 353 | 354 | ### Additional Resources 355 | 356 | * [HTTP Status Codes](https://httpstatuses.com/) 357 | 358 | 359 | 360 | 361 | 362 | 363 | 364 | 365 | 366 | 367 | 368 | 369 | 370 | 371 | 372 | 373 | 374 | 375 | 376 | 377 | 378 | 379 | 380 | 381 | 382 | 383 | 384 | 385 | -------------------------------------------------------------------------------- /web-development/microservices.md: -------------------------------------------------------------------------------- 1 | # Microservices 2 | 3 | ## Microservices in Python talk 4 | 5 | ### Lifecycle of a Microservice 6 | 7 | * On startup, the microservice registers with the service register, or is "discovered" by it 8 | * The load balancer watches the registery and updates itself to include the new microservice 9 | * The new service starts receiving traffic from the load balancer 10 | * If more than one instance of the service exist, the traffic is split among them 11 | * The service sends "keep-alive" signals, or responds to periodic health checks 12 | * When the service is stopped, or stops sending keep-alives, or fails a health check, it is removed from the registry, and in turn from the load balancer 13 | 14 | ## [Microservices](https://martinfowler.com/articles/microservices.html) by Martin Fowler 15 | 16 | > There is a bare minimum of centralized management of these services, which may be written in different programming languages and use different data storage technologies. 17 | 18 | > Using services like this does have downsides. Remote calls are more expensive than in-process calls, and thus remote APIs need to be coarser-grained, which is often more awkward to use. If you need to change the allocation of responsibilities between components, such movements of behavior are harder to do when you're crossing process boundaries. 19 | 20 | > Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure. -- Melvyn Conway, 1967 21 | 22 | > a team should own a product over its full lifetime 23 | 24 | ## [Testing Strategies in a Microservices Architecture](https://martinfowler.com/articles/microservice-testing/) 25 | 26 | * 27 | -------------------------------------------------------------------------------- /web-development/wsgi.md: -------------------------------------------------------------------------------- 1 | # Web Server Gateway Interfce (WSGI) 2 | 3 | ## [Wikipedia](https://en.wikipedia.org/wiki/Web_Server_Gateway_Interface) 4 | 5 | > WSGI was created as a low-level interface between web servers and web applications or frameworks to promote common ground for portable web application development 6 | 7 | ## [PEP-3333](https://www.python.org/dev/peps/pep-3333/) 8 | 9 | > 10 | 11 | 12 | first 5 minutes https://www.youtube.com/watch?v=H6Q3l11fjU0 13 | to understand wsgi 14 | 15 | other good bits of advice. friends don't let friends run docker as root. 16 | 17 | friends don't let friends use python without virtual environments 18 | --------------------------------------------------------------------------------