119 |
120 |
121 |
122 |
--------------------------------------------------------------------------------
/handouts/EnterpriseApplication.md:
--------------------------------------------------------------------------------
1 | ## Enterprise Application
2 |
3 | Enterprise applications usually have **a lot of persistent data**, usually managed by some kind of database management system. Usually this database is relational, but increasingly we're seeing NoSQL alternatives. This data will usually be longer lasting and more valuable than the applications that process it.
4 |
5 | This **data is accessed and manipulated concurrently**. The numbers vary a lot, in-house applications may have a few tens of users, but customer-facing web applications can easily have tens of thousands.
6 |
7 | With so much data, enterprise applications have **a lot of user interface screens** to handle it.
8 |
9 | You'll need to **integrate with other enterprise applications**. These systems are built by a wide range of teams, some from vendors who sell to many customers, others built internally just for your organization.
10 |
11 | Even when different applications access the same data there is considerable **conceptual dissonance** between them, a customer may mean something quite different to the sales organization than it does to technical support.
12 |
13 | If you want to change business rules you need sixty-seven meetings and three vice-presidents retiring, Do this a thousand times and you have the **complex business "illogic"** that lies in the heart of many enterprise applications.
--------------------------------------------------------------------------------
/handouts/Microservices.md:
--------------------------------------------------------------------------------
1 | ## Microservices
2 |
3 | The microservice architectural style is **an approach to developing a single application as a suite of small services**, each running in its own process and communicating with lightweight mechanisms, often an **HTTP resource API**. These services are built around business capabilities and independently deployable by fully automated deployment machinery.
4 |
5 | ### Monolithic application
6 |
7 | Monolithic application built as a single unit. Enterprise Applications are often built in **three main parts**:
8 |
9 | * **a client-side user interface** (consisting of HTML pages and javascript running in a browser on the user's machine)
10 |
11 | * **a database** (consisting of many tables inserted into a common, and usually relational, database management system),
12 |
13 | * **a server-side application** (handling HTTP requests, executing domain logic, retrieving and updating data from the database)
14 |
15 | This server-side application is a *monolith* - a **single logical executable**. Any changes to the system involve building and deploying a new version of the server-side application. You can horizontally scale the monolith by running many instances behind a load-balancer.
16 |
17 | * Change cycles are tied together
18 | * A change made to a small part of the application, requires the entire monolith to be rebuilt and deployed.
19 | * Over time it's often hard to keep a good modular structure,
20 | * Scaling requires scaling of the entire application rather than parts of it that require greater resource.
21 |
22 | 
23 |
24 | ### Characteristics of a Microservice Architecture
25 |
26 | There isn't a formal definition of the microservices architectural style, but we can attempt to describe what we see as common characteristics for architectures.
27 |
28 | #### Componentization via Services
29 |
30 |
--------------------------------------------------------------------------------
/handouts/TwelveFactorApp.md:
--------------------------------------------------------------------------------
1 | ## The Twelve-Factor App
2 |
3 | The twelve-factor app is a methodology for building software-as-a-service apps that:
4 | * Use **declarative** formats for setup automation, to minimize time and cost
5 | * Have a **clean contract** with the underlying operating system, offering **maximum portability**
6 | * Are suitable for **deployment** on modern **cloud platforms**
7 | * **Minimize divergence** between development and production
8 | * And can **scale up** without significant changes to tooling, architecture, or development practices.
9 |
10 | ### 1. Codebase
11 | > One codebase tracked in revision control, many deploys
12 |
13 | * If there are multiple codebases, it’s not an app – it’s a distributed system.
14 | * Multiple apps sharing the same code is a violation of twelve-factor
15 | * There is only one codebase per app, but there will be many deploys of the app
16 |
17 | 
18 |
19 | ### 2. Dependencies
20 | > Explicitly declare and isolate dependencies
21 |
22 | * Programming languages offer packaging system for distributing support libraries, such as **CPAN** for Perl or **Rubygems** for Ruby.
23 | * A twelve-factor app never relies on implicit existence of system-wide packages.
24 |
25 | | languages | declaration | isolation |
26 | |------------|--------------|-----------------|
27 | | ruby | Gemfile | bundle exec |
28 | | python | pip | virtualenv |
29 | | C | autoconf | static linking |
30 |
31 | ### 3. Config
32 | > Store config in the environment
33 |
34 |
35 | * Resource handles to the database, Memcached, and other backing services
36 | * Credentials to external services such as Amazon S3 or Twitter
37 | * Per-deploy values such as the canonical hostname for the deploy
38 |
39 | A litmus test for whether an app has all config correctly factored out of the code is whether the codebase could be made open source at any moment, without compromising any credentials.
40 |
41 | Another approach to config is the use of config files which are not checked into revision control, such as `config/database.yml` in Rails. Further, these formats tend to be language- or framework-specific.
42 |
43 | The twelve-factor app stores config in **environment variables** (often shortened to *env vars* or *env*).
44 |
45 | They are a language- and OS-agnostic standard.
46 |
47 | ### 4. Backing services
48 | > Treat backing services as attached resources
49 |
50 | A backing service is any service the app consumes over the network as part of its normal operation. Examples include:
51 |
52 | * datastores (such as MySQL or CouchDB)
53 | * messaging/queueing systems (such as RabbitMQ or Beanstalkd)
54 | * SMTP services for outbound email (such as Postfix)
55 | * and caching systems (such as Memcached).
56 |
57 | In addition to these locally-managed services, the app may also have services provided and managed by third parties. Examples include:
58 | * SMTP services (such as Postmark)
59 | * metrics-gathering services (such as New Relic or Loggly)
60 | * binary asset services (such as Amazon S3)
61 | * and even API-accessible consumer services (such as Twitter, Google Maps, or Last.fm).
62 |
63 | **The code for a twelve-factor app makes no distinction between local and third party services.** To the app, both are attached resources, accessed via a URL or other locator/credentials stored in the config.
64 |
65 | The twelve-factor app treats these databases as attached resources, which indicates their loose coupling to the deploy they are attached to.
66 |
67 | 
68 |
69 | ### 5. Build, release, run
70 | > Strictly separate build and run stages
71 |
72 | A codebase is transformed into a (non-development) deploy through three stages:
73 |
74 | * The **build stage** is a transform which converts a code repo into an executable bundle known as a build. Using a version of the code at a commit specified by the deployment process, the build stage fetches vendors dependencies and compiles binaries and assets.
75 | * The **release stage** takes the build produced by the build stage and combines it with the deploy’s current config. The resulting release contains both the build and the config and is ready for immediate execution in the execution environment.
76 | * The **run stage** (also known as “runtime”) runs the app in the execution environment, by launching some set of the app’s processes against a selected release.
77 |
78 | 
79 |
80 | **The twelve-factor app uses strict separation between the build, release, and run stages.** For example, it is impossible to make changes to the code at runtime, since there is no way to propagate those changes back to the build stage.
81 |
82 | Every release should always have a unique release ID, Releases are an append-only ledger and a release cannot be mutated once it is created. Any change must create a new release.
83 |
84 | ### 6. Processes
85 | > Execute the app as one or more stateless processes
86 |
87 | In the simplest case, the code is a stand-alone script, the execution environment is a developer’s local laptop with an installed language runtime, and the process is launched via the command line (for example, `python my_script.py`).
88 |
89 | **Twelve-factor processes are stateless and share-nothing.** Any data that needs to persist must be stored in a stateful backing service, typically a database.
90 |
91 | The twelve-factor app never assumes that anything cached in memory or on disk will be available on a future request or job.
92 |
93 | Some web systems rely on “sticky sessions” – that is, caching user session data in memory of the app’s process and expecting future requests from the same visitor to be routed to the same process. **Sticky sessions are a violation of twelve-factor and should never be used or relied upon.** Session state data is a good candidate for a datastore that offers time-expiration, such as *Memcached* or *Redis*.
94 |
95 | ### 7. Port binding
96 | > Export services via port binding
97 |
98 | **The twelve-factor app is completely self-contained** and does not rely on runtime injection of a webserver into the execution environment to create a web-facing service. The web app **exports HTTP as a service by binding to a port**, and listening to requests coming in on that port.
99 |
100 | In deployment, a routing layer handles routing requests from a public-facing hostname to the port-bound web processes.
101 |
102 | This is typically implemented by using **dependency declaration** to add a webserver library to the app, such as *Tornado* for Python, *Thin* for Ruby, or *Jetty* for Java and other JVM-based languages. This happens entirely in user space, that is, within the app’s code. The contract with the execution environment is binding to a port to serve requests.
103 |
104 | Note also that the port-binding approach means that one app can become the backing service for another app, by providing the URL to the backing app as a resource handle in the config for the consuming app.
105 |
106 | ### 8. Concurrency
107 | > Scale out via the process model
108 |
109 | **In the twelve-factor app, processes are a first class citizen.** Processes in the twelve-factor app take strong cues from the unix process model for running service daemons. Using this model, the developer can architect their app to handle diverse workloads by assigning each type of work to a process type. For example, HTTP requests may be handled by a web process, and long-running background tasks handled by a worker process.
110 |
111 | 
112 |
113 | Twelve-factor app processes should never daemonize or write PID files. Instead, rely on the operating system’s process manager (such as systemd) to manage output streams, respond to crashed processes, and handle user-initiated restarts and shutdowns.
114 |
115 | ### 9. Disposability
116 | > Maximize robustness with fast startup and graceful shutdown
117 |
118 | **The twelve-factor app’s processes are disposable, meaning they can be started or stopped at a moment’s notice.** This facilitates fast elastic scaling, rapid deployment of code or config changes, and robustness of production deploys.
119 |
120 | Short startup time provides more agility for the release process and scaling up; and it aids robustness, because the process manager can more easily move processes to new physical machines when warranted.
121 |
122 | Processes **shut down gracefully when they receive a SIGTERM signal** from the process manager. For a web process, graceful shutdown is achieved by ceasing to listen on the service port (thereby refusing any new requests), allowing any current requests to finish, and then exiting.
123 |
124 | For a worker process, graceful shutdown is achieved by returning the current job to the work queue. For example, on RabbitMQ the worker can send a `NACK`.
125 |
126 | Processes should also be **robust against sudden death**, in the case of a failure in the underlying hardware. While this is a much less common occurrence than a graceful shutdown with `SIGTERM`, it can still happen. A recommended approach is use of a robust queueing backend, such as Beanstalkd, that returns jobs to the queue when clients disconnect or time out.
127 |
128 | ### 10. Dev/prod parity
129 | > Keep development, staging, and production as similar as possible
130 |
131 | Historically, there have been substantial gaps between development (a developer making live edits to a local deploy of the app) and production (a running deploy of the app accessed by end users). These gaps manifest in three areas:
132 |
133 | * **The time gap:** A developer may work on code that takes days, weeks, or even months to go into production.
134 | * **The personnel gap:** Developers write code, ops engineers deploy it.
135 | * **The tools gap:** Developers may be using a stack like Nginx, SQLite, and OS X, while the production deploy uses Apache, MySQL, and Linux.
136 |
137 | **The twelve-factor app is designed for continuous deployment by keeping the gap between development and production small.**
138 |
139 | | | Traditional app | Twelve-factor app |
140 | |------------|--------------|-----------------|
141 | | **Time between deploys** | Weeks | Hours |
142 | | **Code authors vs code deployers** | Different people | Same people |
143 | | **Dev vs production environments** | Divergent | As similar as possible |
144 |
145 | Developers sometimes find great appeal in using a lightweight backing service in their local environments, while a more serious and robust backing service will be used in production. For example, using SQLite locally and PostgreSQL in production; or local process memory for caching in development and Memcached in production. **The twelve-factor developer resists the urge to use different backing services between development and production.**
146 |
147 | ### 11. Logs
148 | > Treat logs as event streams
149 |
150 | Logs provide visibility into the behavior of a running app. In server-based environments they are commonly written to a file on disk (a “logfile”); but this is only an output format.
151 |
152 | Logs are the stream of aggregated, time-ordered events collected from the output streams of all running processes and backing services.
153 |
154 | **A twelve-factor app never concerns itself with routing or storage of its output stream.**
155 |
156 | In staging or production deploys, each process’ stream will be captured by the execution environment, collated together with all other streams from the app, and routed to one or more final destinations for viewing and long-term archival. These archival destinations are not visible to or configurable by the app, and instead are completely managed by the execution environment. Open-source log routers (such as Logplex and Fluentd) are available for this purpose.
157 |
158 | These systems allow for great power and flexibility for introspecting an app’s behavior over time, including:
159 |
160 | * Finding specific events in the past.
161 | * Large-scale graphing of trends (such as requests per minute).
162 | * Active alerting according to user-defined heuristics (such as an alert when the quantity of errors per minute exceeds a certain threshold).
163 |
164 | ### 12. Admin processes
165 | > Run admin/management tasks as one-off processes
166 |
167 | Developers will often wish to do one-off administrative or maintenance tasks for the app, such as:
168 |
169 | * Running database migrations (e.g. `manage.py migrate` in Django, `rake db:migrate` in Rails).
170 | * Running a console (also known as a REPL shell) to run arbitrary code or inspect the app’s models against the live database. Most languages provide a REPL by running the interpreter without any arguments (e.g. `python` or `perl`) or in some cases have a separate command (e.g. `irb` for Ruby, `rails console` for Rails).
171 | * Running one-time scripts committed into the app’s repo (e.g. `php scripts/fix_bad_records.php`).
172 |
173 | One-off admin processes should be run in an identical environment as the regular long-running processes of the app. They run against a release, using the same codebase and config as any process run against that release. Admin code must ship with application code to avoid synchronization issues.
174 |
175 | Twelve-factor strongly favors languages which provide a REPL shell out of the box, and which make it easy to run one-off scripts. In a local deploy, developers invoke one-off admin processes by a direct shell command inside the app’s checkout directory. In a production deploy, developers can use ssh or other remote command execution mechanism provided by that deploy’s execution environment to run such a process.
--------------------------------------------------------------------------------
/img/MonolithsandMicroservices.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/pyguy/lpi-devops-study/ca47f351055a89253bc0943fc7fad1b2047edf2c/img/MonolithsandMicroservices.png
--------------------------------------------------------------------------------
/img/attached-resources.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/pyguy/lpi-devops-study/ca47f351055a89253bc0943fc7fad1b2047edf2c/img/attached-resources.png
--------------------------------------------------------------------------------
/img/codebase-deploys.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/pyguy/lpi-devops-study/ca47f351055a89253bc0943fc7fad1b2047edf2c/img/codebase-deploys.png
--------------------------------------------------------------------------------
/img/handout-20.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/pyguy/lpi-devops-study/ca47f351055a89253bc0943fc7fad1b2047edf2c/img/handout-20.png
--------------------------------------------------------------------------------
/img/process-types.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/pyguy/lpi-devops-study/ca47f351055a89253bc0943fc7fad1b2047edf2c/img/process-types.png
--------------------------------------------------------------------------------
/img/release.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/pyguy/lpi-devops-study/ca47f351055a89253bc0943fc7fad1b2047edf2c/img/release.png
--------------------------------------------------------------------------------