├── .github ├── CODEOWNERS └── PULL_REQUEST_TEMPLATE.md ├── .gitignore ├── API ├── Authentication.md ├── CORS.md ├── Documentation.md ├── Intro.md ├── JSON:API.md ├── Pagination.md └── Versioning.md ├── Databases ├── Migrations.md ├── PG extensions.md └── SQL │ ├── General.md │ ├── Indexes.md │ ├── Query Examples │ ├── 01 - The Most Recent Record.md │ ├── 02 - Top N Records.md │ ├── 03 - Appointments.md │ ├── 04 - Search Optimization.md │ ├── 05 - CSV.md │ ├── 06 - Chat.md │ ├── 07 - Circular Sort.md │ ├── 08 - UPDATE with subquery.md │ ├── 09 - LATERAL JOIN.md │ └── 10 - Keywords Count.md │ └── Tips & Tricks.md ├── Development practices ├── API.md ├── Automated tests.md ├── Background and scheduled jobs.md ├── Code style.md ├── Design patterns.md ├── Emails.md ├── File uploads.md ├── Folder Structure.md └── __Other important guidelines.md ├── Fiscalizer.md ├── Hosting API Documentation.md ├── Misc └── Fiscalization.md ├── Starting a new project.md ├── Workflows ├── Architecture review.md ├── Dependencies.md ├── Deployment.md ├── GIT │ ├── Branches.md │ ├── Clean changes.md │ └── Pull requests.md ├── Production release.md ├── Rails upgrades.md └── Starting a new project.md └── intro.slim /.github/CODEOWNERS: -------------------------------------------------------------------------------- 1 | # For more info about the file read https://help.github.com/en/articles/about-code-owners 2 | 3 | * @cukoreva @cilim @nikajukic 4 | -------------------------------------------------------------------------------- /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | Task: [#__TASK_NUMBER__](__ADD_URL_TO_PRODUCTIVE_TASK__) 2 | 3 | #### Aim 4 | 5 | 6 | #### Solution 7 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.DS_Store 2 | -------------------------------------------------------------------------------- /API/Authentication.md: -------------------------------------------------------------------------------- 1 | Authentication is the process of verifying someone's identity. There are many ways to verify identities over HTTP APIs, the most popular of which will be described below. 2 | 3 | When we talk about API authentication, it's important to distinguish its two components: 4 | 5 | 1. the mechanism — the medium utilized by the authentication protocol 6 | 2. the method — the protocol for authenticating the user 7 | 8 | These two work in tandem and sometimes they're indistinguishable, but we'll explain each one separately and show how they work together. 9 | 10 | ## Mechanisms 11 | 12 | As mentioned above, a mechanism is a medium utilized by the authentication protocol. The architecture of HTTP allows for several authentication mechanisms. 13 | 14 | ### Cookies 15 | 16 | Cookies are pieces of data exchanged between the client and the server through headers. The server sends one or more cookies via a `Set-Cookie` response header and then the browser saves them and includes them in subsequent requests to the server in a `Cookie` header. The browser can also send cookies to the server without receiving any first. The goal here is to somehow hold the state between server and client, meaning that the server will have a way to maintain a session with the stateless client. 17 | 18 | When setting cookies, the server can apply one or more restrictions, like expiration date and domains where they're supposed to be used. You can learn more about how cookies work on [MDN](https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies). 19 | 20 | There are two important cookie attributes you should know about: 21 | 22 | - `Secure` - When set, the cookie will only be sent to the server and saved in the browser if the protocol through which they're sent/received is `https`. Insecure `http` protocol can't access cookies with that attribute. If you're communicating over `https`, make sure to set this attribute. 23 | - `HttpOnly` - These cookies cannot be accessed by JavaScript code. They are exchanged only between the browser and the server. If the cookie is used to store a server-side session (a method described below), this attribute should be enabled because the cookie is supposed to be saved and used only by the browser, not read or changed by the user through JS. Note that this attribute doesn't prevent the tampering of cookies as users can still edit them manually (for example, in a developer console). 24 | 25 | _Note_: If you have a cookie based authentication using the `HttpOnly` flag is among the safest option for storing sensitive information in a browser, so if you have a JS-based frontend application that will consume your APIs, and you need to keep the secrets really secret, consider using this technique. 26 | 27 | ### Authorization header 28 | 29 | HTTP supports sending a standard `Authorization` header in requests. The header consists of a type (indicating what kind of a credential it is) and a value separated by whitespace. An example header can look like this: `Basic aHR0cHM6Ly95b3V0dS5iZS9kUXc0dzlXZ1hjUQ==`. 30 | 31 | [MDN](https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication#authentication_schemes) has a nice list of types. The one most commonly seen in APIs is the so-called `Bearer` token. As explained in the [Bearer Token Usage RFC](https://datatracker.ietf.org/doc/html/rfc6750#section-1.2), a bearer token is: 32 | 33 | > A security token with the property that any party in possession of the token (a "bearer") can use the token in any way that any other party in possession of it can. 34 | When the server generates a token, it should make it opaque (meaning they can't be interpreted by anyone other than the server). 35 | 36 | The `Authorization` header is the preferred mechanism if your API is to be used by platforms other than the browser (browsers using it are susceptible to XSS attacks). It's easier to control than cookies outside of browsers, and it doesn't come with security concerns around using query strings (described below). If your API will be consumed by both browsers and mobile platforms, you should provide for both cookies and the `Authorization` header as cookies can complicate the implementation on mobile platforms, and JS clients can't store/exchange authorization information securely with the server, other than through a cookie. 37 | 38 | #### Basic auth 39 | 40 | Basic auth is the simplest authentication method. It's a type of `Authorization` header authentication, where the "type" value is `Basic`. 41 | 42 | ### Query string 43 | 44 | The query string is the part of the URL used to provide additional parameters. The query string starts after the question mark character (`?`) and delimits multiple parameters with ampersands (`&`). Each parameter consists of a name and a string value separated by an equal sign (`=`). 45 | 46 | For example, the URL `example.com?api_key=123&token=abc` has two query params: `api_key` with the value `123`, and `token` with the value `abc`. 47 | 48 | The query string can be used to transmit keys and tokens in requests to the server, similar to cookies and the `Authorization` header. At the first glance, this might seem insecure, but if it's sent over the `https` protocol, the query string is encrypted with the rest of the request and it's visible only to the client and the server. 49 | 50 | However, there are other concerns when using query strings to transmit sensitive data: 51 | 52 | - URLs (including query strings) are visible in server logs (although, you can [filter parameters](https://guides.rubyonrails.org/action_controller_overview.html#parameters-filtering) like these in Rails) 53 | - URLs are saved in browser history (when used in navigation) 54 | - you have to include the key/token in every request, but normally you want to reserve the query string for other parameters (like filters, pagination, sorting, etc.) 55 | - XSS attacks 56 | - length constraints (if you want to ensure your apps work on most modern browsers, exceeding 2047 characters isn't suggested) 57 | 58 | Consider using query strings for authentication only when other methods are unavailable. 59 | 60 | ## Methods 61 | 62 | ### API keys 63 | 64 | API keys are unique strings used to identify clients. Think of them like normal passwords, and you should also treat them as such. 65 | 66 | API keys in most cases aren't used to authenticate the users themselves, but rather external access to user's resources. In those scenarios, users authenticate through other means (like a server-side session) and then they request one or more keys for API access. 67 | 68 | Regarding the format of API keys, they usually consist of alphanumeric characters and have a minimum length to prevent brute-force attacks. 69 | 70 | If your system allows access via API keys, it should also provide a way to revoke them, in case they're stolen. 71 | 72 | Remind users of security measures they can take to decrease the likelihood of a malicious party accessing their keys and resources: 73 | - never store an API key in a public place (private repos on Github count as public too) and never embed it directly in code (unless the key is meant to be public) 74 | - regenerate API keys periodically 75 | - delete a key if you're not using it anymore 76 | 77 | One option for generating API keys is [`SecureRandom`](https://ruby-doc.org/stdlib-3.0.0/libdoc/securerandom/rdoc/SecureRandom.html), a Ruby standard library. 78 | 79 | ### Access and refresh tokens 80 | 81 | Access tokens and refresh tokens are artifacts used in the OAuth 2.0 authorization protocol. This protocol enables a user to grant permissions for accessing resources from a 3rd party application. It is most widely used for features like Single Sign On (SSO) via a variety of social media platforms (eg: Facebook, Instagram, LinkedIn). 82 | 83 | Once the user logs into the authorization server and grants access to their personal data, an `access token` is returned to the client server. This token can be used for subsequent communication with the server on behalf of the user. For that reason, it's critical to have security strategies that minimize the risk of compromising access tokens, for example creating access tokens with a short lifespan. The downside of short-lived access tokens is the fact that the client application must prompt the user to log in again, which makes for bad UX. A better solution would be to use `refresh tokens`. 84 | 85 | Refresh tokens are artifacts that let the client application refresh an access token without asking the user to log in. They usually livie longer than access tokens, but that also means that if the refresh token is stolen, the malicious user has the power to create new access tokens, so additional security techniques need to be in place, like refresh token rotation or automatic reuse detection. 86 | 87 | ### JWT 88 | 89 | The JSON Web Token standard is a method for creating the mentioned access tokens or any other artifact used for transmitting information between 2 servers. You can read more about it on [jwt.io](https://jwt.io/introduction). 90 | 91 | ### Server-side sessions 92 | 93 | Believe it or not, cookies aren't the answer sometimes. Here are some usual arguments against them: 94 | 95 | - file size (only about 4kb of data can be saved into a cookie) 96 | - they are sent with every request, making it slower 97 | - storing wrong data inside a cookie can be insecure ([replay attacks](https://guides.rubyonrails.org/security.html#replay-attacks-for-cookiestore-sessions)) 98 | 99 | When you can't store your session data inside a cookie, Rails has other options that are easily configurable: 100 | 101 | _config/initializers/session_store.rb_ 102 | 103 | ```ruby 104 | Rails.application.config.session_store :cookie_store 105 | ``` 106 | 107 | Two of the most widely used alternatives for storing session data are: 108 | 109 | - `redis_store` - toggle storage to be backed by Redis 110 | - `active_record_store` - toggle storage to be backed by ActiveRecord 111 | 112 | Both solutions come out of the box with their corresponding dependencies: [redis-rails](https://github.com/redis-store/redis-rails) & [activerecord-session_store](https://github.com/rails/activerecord-session_store). 113 | 114 | 115 | ## On reinventing authentication 116 | 117 | For whatever reasons, you may feel like building your own authentication solution. Pursue this motivation, but not with the intention to use it in a production environment. 118 | 119 | Authentication is a complex beast encompassing many aspects, most of which you won't be aware of until you start building your own solution, and some of which you'll only learn about when someone bypasses its security. 120 | 121 | There are megapopular public libraries available for all common authentication methods, which come with the assurance: 122 | 123 | - that they have been tested in real, production environments by thousands if not millions of users, 124 | - that they had their implementation examined by security experts, 125 | - that they had suffered and fixed serious security breaches. 126 | 127 | If you think your custom solution will live up to those expectations in the time you have to implement it, then feel free to build it. However, there are better and more productive ways to use your time: solving problems which haven't been solved already. 128 | -------------------------------------------------------------------------------- /API/CORS.md: -------------------------------------------------------------------------------- 1 | CORS (Cross-Origin Resource Sharing) is a mechanism for allowing web requests between domains (origins). Cross-origin requests are by default prevented by the browser [same-origin policy](https://developer.mozilla.org/en-US/docs/Web/Security/Same-origin_policy) — CORS exists to lift that restriction when required. 2 | 3 | An excellent guide to CORS is available [here](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS). This mechanism concerns frontend and backend developers alike and should be understood by both. The omnipresence of CORS cannot be understated — if your work concerns the Web, CORS will crop up eventually. Therefore, don't hesitate to understand how it works, take your time and read the guide. 4 | 5 | ## Misconceptions 6 | 7 | Assuming you have read the guide, you still might feel a lack of confidence as to how it works. This is not uncommon with CORS, it is one of the most misunderstood aspects of the Web. Therefore, the following list of misconceptions has been compiled to help you understand it better: 8 | 9 | - "CORS is enforced by the server" 10 | - Servers only supply CORS headers, the enforcer is actually **the browser**. It sends a preflight request to discover the server's CORS policy, and based on that info decides whether to execute the request or raise an error.
11 | This is also the reason why CORS errors don't show up in tools like cURL or Postman — they don't send a preflight request or enforce the server headers. 12 | - "I can just allow all origins with `*` and be done with it" 13 | - It depends. Setting `*` only works when the request doesn't include credentials (cookies or the Authorization header). If you allow all origins *and* include credentials, the browser will [raise an error](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSNotSupportingCredentials). When you need to include credentials, the `Access-Control-Allow-Origin` header must have a concrete origin value. 14 | - "I can copy the `Origin` header to `Access-Control-Allow-Origin` to satisfy CORS" 15 | - In certain situations (like hosting public APIs) this is okay, but in others you can expose your users to malicious websites that can execute requests on the user's behalf to your server with potentially disastrous results for the user. Read [this](https://portswigger.net/research/exploiting-cors-misconfigurations-for-bitcoins-and-bounties) article for more info. 16 | - "Browser clients can use `no-cors` mode to skip CORS" 17 | - In a way, yes, but that option comes with severe limitations. The only methods allowed are `HEAD`, `GET` and `POST`, with the constraint that all headers are [simple](https://fetch.spec.whatwg.org/#simple-header) and that they **cannot** read the response in JS. [[Reference](https://developer.mozilla.org/en-US/docs/Web/API/Request/mode#value)] 18 | 19 | To learn more about CORS misconceptions and misconfigurations, read [this](https://web-in-security.blogspot.com/2017/07/cors-misconfigurations-on-large-scale.html) article which shows how even the most popular sites fail to properly serve CORS headers. 20 | 21 | ## How to set up headers 22 | 23 | For Rack-based applications (like Ruby on Rails), [`rack-cors`](https://github.com/cyu/rack-cors) is the most popular choice for CORS headers management. Follow the gem README and make sure to read the [Common Gotchas](https://github.com/cyu/rack-cors#common-gotchas) chapter. 24 | 25 | If you're setting up CORS for environments which are accessed locally by frontend developers (like staging), add `localhost` to the list of allowed domains so they can send requests from their local environment. 26 | -------------------------------------------------------------------------------- /API/Documentation.md: -------------------------------------------------------------------------------- 1 | An API without accompanying documentation practically doesn't even exist. Furthermore, without good, thorough documentation, no API can be good either. As the developer behind an API, you are naturally familiar with what you've created and how to use it. However, your clients are not. Documentation exists to transfer that knowledge to clients. 2 | 3 | Documentation writing requires thinking about what the API client needs to set up their environment, what they're able to do with the API, what to send in requests and which response scenarios to expect. All of this naturally requires providing lots of information and structuring it in a comprehensible way. 4 | 5 | Writing isn't easy and developers tasked with writing documentation sometimes do a poor job. Poorly-written documentation necessitates further communication and explanations from the API developer — imagine then what would happen if you had to explain your API to a thousand confused clients. Therefore, to save time in the long run, you have to invest enough time to write proper documentation from the start. 6 | 7 | ## Essentials 8 | 9 | The following checklist contains documentation essentials. This is not an exhaustive list — APIs always have specifics which cannot be included in such lists and you should document them as well. A good rule of thumb is, if someone would ask about it, it should be documented. 10 | 11 | - **Overview** - Placed at the top of the documentation, the overview should explain: what the API does, architecture (e.g. REST), format (e.g. JSON:API or GraphQL) and it should provide general information (base URL, response types, etc.). 12 | - **Format requirements** - If you use JSON:API, GraphQL or some other format, they typically come with requirements, like setting `Content-Type`/`Accept` headers, which the clients must abide by. 13 | - **CORS** - If applicable, document your CORS policy (allowed origins, methods, exposed headers, credentials, etc.). 14 | - **Authentication** - Applicable if endpoints are authenticated. Explain the authentication method(s) used (API key, cookies, JWT, etc.) and how to obtain credentials. 15 | - **Authorization** - If the outcome of requests depends on the user's permissions, then that should also be documented. If authorization logic varies between endpoints, the specific logic should be explained for each endpoint. 16 | - **Errors** - All APIs are subject to errors. Errors are roadblocks, preventing clients from doing what they need to do. When a roadblock appears with either no explanation or a vague one (`PC LOAD LETTER`), frustration ensues.
17 | Document all common errors and supply descriptions. Descriptions should explain what happened and what the client can do to remove the error. Document even the 5xx messages, no matter how obvious they seem. 18 | - **Changelog** - Major changes to the API should be documented for historical reference. It's an optional step if the API will be used by a limited number of users to whom you can communicate changes directly, though it's useful to have it even then. 19 | - **Endpoints** - The main body of the documentation, each endpoint should: describe what it is, document available HTTP verbs, and show example requests and responses for happy and sad paths. If a response error is specific to an endpoint, document it. 20 | 21 | *Bonus points:* 22 | 23 | - **Quickstart guide** - Onboard clients quickly by providing an example request (e.g. in cURL) which they can copy/paste to test out and modify. This approach is the fastest way to get clients to learn how to use the API. 24 | - **Diagrams** - Complex flows are hard to explain with just words. If the API, for example, controls a state machine, words will struggle to explain its states and transitions. Diagrams, even simple ones created in tools like [draw.io](https://draw.io), will help clients tremendously in understanding the underlying logic.
25 | *Tip*: keep raw, editable files in the repository along with exported formats (like PNG). That way other developers can later update existing diagrams instead of having to create new ones. 26 | 27 | ## Hall of fame 28 | 29 | Everything written about documentation practices in the preceding paragraphs sets a basis for writing API documentation. But it's just that — a basis. Real-world APIs are big and complex and to write comprehensible documentation for them means to consider many aspects not pointed out here. Instead of trying to put everything into words, here are some examples of API documentation that prove that it's possible to manage complexity and present it in an understandable and user-friendly way to the clients. 30 | 31 | - [Stripe API](https://stripe.com/docs/api) 32 | - [Productive](https://developer.productive.io/) 33 | - [Example App](https://cekila.byinfinum.co/api/v1/docs/) 34 | 35 | ## Automating documentation 36 | 37 | Documentation writing is laborious. Most of it has to be done by hand — nothing can replace that — but some things can be automated. One of the things we can automate are example requests and responses. 38 | 39 | ### Dox 40 | 41 | Infinum's very own gem that does the majority of work in regards to API documentation for the developer. It's super easy to set up and with just a couple of changes to your existing request specs, you get an updated API documentation after the specs finish running. 42 | Check the dox documentation by following [this link](https://github.com/infinum/dox). 43 | -------------------------------------------------------------------------------- /API/Intro.md: -------------------------------------------------------------------------------- 1 | APIs are sets of defined protocols enabling applications to communicate with other systems, that can also be applications. In the context of the Web, APIs are a way for Web clients (browsers, mobile apps, terminals, servers, and other platforms) to communicate with Web servers. 2 | 3 | Our APIs use the HTTP protocol and are usually in JSON format, but if necessary, we might implement file uploads, XML or something else the project requires. 4 | 5 | The following chapters provide information on our practices regarding JSON APIs. Although some gems are mentioned throughout, primary focus is on aspects surrounding API design and delivery. 6 | 7 | ## API design principles 8 | 9 | Before talking about specific topics like authentication and pagination, we should outline a set of principles all APIs should follow. Since it's challenging to understand generic definitions, all of the principles will be introduced with an example showing what happens when they're *not* followed to try to illustrate the importance of following them. 10 | 11 | *Note*: all the examples below are inspired by situations developers have been in when dealing with terrible APIs. 12 | 13 | ### Predictable 14 | 15 | What we mean by "predictable API" is that it never aims to surprise the client. As an example, let's take an imaginary API endpoint for retrieving song details: 16 | 17 | ```json 18 | // GET api/songs/123 19 | { 20 | "id": 123, 21 | "name": "Modern Love", 22 | "artist": "David Bowie", 23 | "album": "Let's Dance" 24 | } 25 | ``` 26 | 27 | There is nothing wrong here, the endpoint returns data which you use in your application. Everything works fine, until one day your application breaks. You debug the code, find that it has to do with the endpoint, so you fetch it, and get the following result: 28 | 29 | ```json 30 | // GET api/songs/123 31 | { 32 | "id": 123, 33 | "name": "Modern Love", 34 | "artist_id": 491, 35 | "album_id": 528 36 | } 37 | ``` 38 | 39 | The API developer decided that instead of serializing artist and album names, they want to return their IDs. Technically, this is not a bad decision, it's quite a common change actually. However, what they failed to do is communicate this change to their clients in advance, so that clients can update their code to support the new response. 40 | 41 | Clients have expectations about APIs. If you tell clients that an endpoint returns data in a particular format, they will start to build their code around it. An API changed without prior notice breaks those assumptions and client applications which depend on them. 42 | 43 | Fortunately, the remedy is simple: communicate with your clients. Anticipate that a change might break client applications and tell them about it in advance. 44 | 45 | ### Consistent design 46 | 47 | Let's imagine an API endpoint which returns country data: 48 | 49 | ```json 50 | // GET api/countries/hrv 51 | { 52 | "Ime": "Croatia", 53 | "nativeName": "Hrvatska", 54 | "top_level_domain": ".hr" 55 | } 56 | ``` 57 | 58 | What's the issue here? As you can see, the API is using different naming conventions for attributes: 59 | 60 | * `Ime` - Croatian PascalCase 61 | * `nativeName` - English camelCase 62 | * `top_level_domain` - English snake_case 63 | 64 | The problem is not the choice of cases (every option is good), nor the language, but the inconsistency. When building an API, pick one style and stick with it. 65 | 66 | This doesn't apply just to request and response bodies, URLs are also subject to consistency rules, e.g. endpoints `/api/country_data` and `/api/currency-data` are bad design because they're mixing snake_case and kebab-case. 67 | 68 | ### Following conventions and standards 69 | 70 | Here is an example of an endpoint which returns album data: 71 | 72 | ```json 73 | // GET api/albums/123 74 | { 75 | "name": "Elephant", 76 | "explicit": "false" 77 | } 78 | ``` 79 | 80 | The API serializes the attribute `explicit` which tells the client if the album contains strong language. This has been implemented as a boolean field, however the API developer decided to stringify the boolean, so instead of returning `false`, they return `"false"`. 81 | 82 | ### Performant 83 | 84 | As the project grows, the user base will also grow, thus increasing the amount of requests sent to your server. Without a fast and optimized API, the response times will sky-rocket and the user experience will plummet, which might cause users to delete their accounts and abandon your product. 85 | 86 | There are many ways to improve the performance of an API, from caching to database sharding, but please be aware that the amount of information fetched from the database and wrapped in an API response should be as small as it can be. Increasing the speed of SQL queries and avoiding any [performance issues](https://infinum.com/handbook/books/devproc/general-coding-practices/api-design#performance-issues) are crucial measures that should always be undertaken. 87 | 88 | ### Robust 89 | 90 | A strong, healthy, and flexible API is a delight to work with. This goes for the producers as well as the consumers of the API. 91 | To have a robust API, you'll need to ensure: 92 | 93 | - a nice interface: existing behaviour is easily applicable throughout the system (e.g.: new filter or sort option) 94 | - bad requests don't cause problems for subsequent usages 95 | - integration tests that go through every endpoint and supported behaviour before deploying new builds 96 | 97 | ### Debuggable 98 | 99 | Delivering a feature usually entails many steps that have to be resolved before shipping to production. When the project is split into *backend* and *frontend*, it often results in time spent communicating how an API is working. This time increases, especially if the API's consumer doesn't understand something and wastes time debugging a certain endpoint or a collection of endpoints. 100 | 101 | The debugging time can be decreased by using appropriate HTTP status codes and providing descriptive error messages in API responses. 102 | 103 | Common HTTP statuses for client-related errors are: 104 | 105 | - _Bad Request (400)_ - user must use valid and supported format in API communication 106 | - _Unauthorized (401)_ - user needs to be logged in in order to use an endpoint 107 | - _Forbidden (403)_ - user needs to be permitted to perform a certain action 108 | - _Not Found (404)_ - user can only fetch a resource that exists 109 | - _Unprocessable entity (422)_ - user tries to create/update a resource, but has provided incomplete or invalid data 110 | 111 | The HTTP status codes help the consumer distinguish if the error is *data-related* (for which access to the database is needed, providing valid credentials, updating permissions, finding a resource to work with), or if it's *request-related* (for which the HTTP request itself would need further investigation). 112 | 113 | Request-related errors can be improved so that the frontend developer has a painless experience figuring out the problem. Imagine an endpoint where a user can update a song. The *Song* model validates that the _name_ attribute is always present, meaning that a song can't exist in the database without its name. Attempting to update a song with an invalid name would look like this: 114 | 115 | ```JSON 116 | PATCH api/v1/songs/123 117 | { 118 | "name": "", 119 | "artist": "David Bowie", 120 | "album": "Let's Dance" 121 | } 122 | ``` 123 | 124 | A good error structure should consist of: 125 | 126 | - title - short categorization of the error, usually the name of the HTTP status code 127 | - detail - coherent and explicit description of the error (e.g.: _"The name of the song must be present."_) 128 | - code - usually HTTP status code, but if you have a requirement for errors to be categorized in one way or another, this can be a good attribute to place the values from the error legend 129 | - source - object that tells the API consumer what the *name* of the parameter with the error is and its *path* in the prior request 130 | 131 | ```JSON 132 | { 133 | "errors": [ 134 | { 135 | "title": "Unprocessable entity", 136 | "detail": "Song must contain a name", 137 | "code": 422, 138 | "source": { 139 | "parameter": "name", 140 | "pointer": "/name" 141 | } 142 | } 143 | ] 144 | } 145 | ``` 146 | 147 | ### Documented 148 | 149 | Transparency is key to not losing any data or information that might be beneficial to anyone on the project. Human beings are faulty, and we're not capable of keeping everything at the forefront of our minds. Over time, some insights might get lost or forgotten and spending time debugging and reminding ourselves is costly. 150 | 151 | An API should always live alongside its documentation. Read more in the [API Documentation](Documentation) chapter. 152 | -------------------------------------------------------------------------------- /API/JSON:API.md: -------------------------------------------------------------------------------- 1 | JSON:API is a specification for tailoring APIs that respond with JSON. Its purpose is to take the API `design` out of the equation so that developers can focus on API `delivery`. Due to its structure, tooling can be built around the specification for handling the bulk of the work regarding parsing requests and formatting responses. 2 | 3 | If your work consists of building or working with JSON:API APIs, read the [specification](https://jsonapi.org/) to understand the concepts behind it. If you're interested in what the community has to say about JSON:API, check out these blog posts: 4 | 5 | - [Standardizing RESTful JSON APIs with OpenAPI Spec](https://oozou.com/blog/standardizing-restful-json-apis-with-openapi-spec-53) 6 | - [The Benefits of Using JSON API](https://nordicapis.com/the-benefits-of-using-json-api/) 7 | 8 | ### Standardized APIs 9 | 10 | Developers at Infinum have implemented dozens of APIs and the [JSON:API query builder](https://github.com/infinum/jsonapi-query_builder) is a collection of the most required features a modern API has to offer. 11 | 12 | The gem allows a developer to use a nice DSL for the so-called *query classes*, which are classes that contain JSON:API-related logic for resource manipulation (filtering, sorting, pagination, etc...). 13 | 14 | If you're just starting a new project, install the gem and make your life easier. 15 | -------------------------------------------------------------------------------- /API/Pagination.md: -------------------------------------------------------------------------------- 1 | Some API endpoints are designed to return a list of resources. We call those *list* or *index* endpoints. 2 | 3 | Servers can, in practice, have thousands of resources, and in theory, return all of them when asked to, but this can be costly, slow, and impractical. It's costly and slow for the server to fetch everything from the database, serialize it, and for the client to download and parse the response. It is also impractical because in most cases users don't care about all resources but instead only want to see top results (e.g. if the endpoint is used for search). 4 | 5 | Rather than showing everything at once, we can limit the number of resources returned in a request and provide a way to fetch the rest. In other words, we can page through resources. 6 | 7 | There are different ways to enable pagination and paginate resources. The following paragraphs are limited to our most commonly used methods. That is not to say these are the only methods available. You are encouraged to use something else if it benefits your project. 8 | 9 | ## Enabling pagination 10 | 11 | In most cases, pagination is enabled through the query string. The query string enables clients to pass additional parameters, which are unsuitable for sending in headers or the body. A URL with pagination applied might look like this: `/api/v1/countries?page=5`. In the example, we're fetching countries and requesting the 5th page of the results. The query string is the preferred way to enable pagination, and you should also keep it enabled by default, so a request without the pagination query string would be handled as `page=1`. 12 | 13 | ## Types 14 | 15 | There are many pagination types, and working with APIs you'll find different implementations. We'll stick to two types which are the most prevalent. 16 | 17 | ### Page-based 18 | 19 | Page-based pagination divides a set of records into pages of equal size (except for the last page which can have fewer records than other pages). A number is assigned to every page, starting with 1. 20 | 21 | ![page-based-pagination](/img/page-based-pagination.png) 22 | 23 | The API should support client-set page size and page number params. An endpoint with pagination params applied could look like this: `/api/v1/countries?page[size]=50&page[number]=3`. In the example, the client requests the 3rd page of countries with maximum 50 records. 24 | This means that the API will respond with countries from 101 to 150, if the API database contains that many countries. Here are some cases when it doesn't: 25 | 26 | - 120 countries: the client would receive only 20 countries (from 101 to 120) 27 | - 160 countries: the client would receive 50 of them (from 101 to 150) 28 | 29 | If the client omits the page size param, you should set a default value. You should also set a maximum value (so clients can't request all records by providing an absurdly large number). If a client requests more records than the maximum allows, you can either clamp the value to the maximum or respond with an error, whatever you think is the right choice. 30 | 31 | When the page number param is omitted, you should respond with the first page. Respond with an error if the param is less than 1, but if it's greater than the total number of pages, then return the last page. Returning the last page instead of responding with an error prevents an edge case when pages disappear due to records being deleted but clients are still requesting the missing page (which they think exists). 32 | 33 | The response should optionally include pagination metadata: 34 | 35 | - current page (in case it differs from the requested page) 36 | - total number of pages 37 | - total number of items 38 | 39 | This information is optional because it won't always be used by clients, and, since it typically requires executing an additional SQL COUNT query, it can be a costly operation. 40 | 41 | Since this type of pagination allows clients to request any page, it is usually represented in the UI with a paginator where you can select a page (e.g. `<- 2 3 4 5 6 ->`). If this is the kind of UI you're implementing the pagination for, then the page-based type is the right choice. 42 | 43 | This type is also popular because it's easy to implement using SQL's `OFFSET` clause, so you may often hear it being referred to as offset-based pagination. `page[size]=50&page[number]=3` simply translates to `LIMIT 50 OFFSET 50 * (3 - 1) => LIMIT 50 OFFSET 100` in SQL. However, offset-based pagination has a negative side. The clause requires scanning _all_ rows included by the offset value, which is inefficient for large offsets and causes performance to suffer the more pages you have (e.g. page 100 is slower to fetch than page 1). If you don't have to paginate many records, then it's not that much of an issue. 44 | 45 | ### Cursor-based 46 | 47 | Cursor-based pagination, also called keyset pagination, uses a cursor which points to a specific record in the dataset and returns a number of records before or after it. 48 | 49 | ![cursor-based-pagination](/img/cursor-based-pagination.png) 50 | 51 | An endpoint with pagination params applied could look like this: `/api/v1/countries?page[after]=dQw4w9`. `page[after]` can be replaced with `page[before]` if we want to retrieve records preceding a cursor. 52 | 53 | Notice that the cursor in the example is a random unique string — this is intentional. Some implementations opt for opaque strings (which have no meaning to clients), while others use record IDs for cursors (take for example [Stripe API](https://stripe.com/docs/api/pagination)). 54 | 55 | This pagination type should also support the client-set page size param (same rules apply as for the page-based type). 56 | 57 | Due to the nature of this pagination type, you cannot show a paginator element in the UI (like you can for the page-based type). Based on the current page (which is defined with a cursor), clients can jump only page-by-page back or forth, but only one page at a time. This type is appropriate for the so-called infinite scroll design, where we load the next page when the client reaches the end of the current page. It can also be used in situations where users can paginate only with `Previous page` and `Next page` buttons. 58 | 59 | ## Common pitfalls 60 | 61 | Always ensure that the records are sorted by a unique attribute (eg: ID). If the collection contains duplicate values of the attribute by which they are sorted, there may be occurrences when records appear to be missing. 62 | Imagine a scenario where we have an API that responds with countries sorted by the number of counties and we only show 2 countries per page: 63 | 64 | - Canada, 50 65 | - Chile, 45 66 | - Colombia, 45 67 | - Denmark, 40 68 | - Ecuador, 35 69 | 70 | The 1st page will show Canada and Chile, whereas the 2nd page shows Chile and Denmark, instead of Colombia and Denmark. This issue should be fixed by sorting the collection by the count **and** some unique attribute of the record (especially if it also has an index). 71 | -------------------------------------------------------------------------------- /API/Versioning.md: -------------------------------------------------------------------------------- 1 | Project requirements are prone to changes. Even with a good, stable API design, the new requirement might force the API to change as well, which breaks client code unless it's adapted. Sometimes, it's not possible for all clients to adopt the change (like mobile applications which depend on users to update them), but we still have to push the change to other clients. At that point, we have to backward support one version of an endpoint and simultaneously release a new one. This is called versioning. 2 | 3 | In technical terms, API versioning is a strategy for serving the same resource in multiple, incompatible formats. We will define what "incompatible" means later on. 4 | 5 | ## To version, or not to version 6 | 7 | ### No breaking changes 8 | 9 | To give an example of a non-breaking change, imagine you are serving an endpoint which returns country data: 10 | 11 | ```json 12 | // GET api/v1/countries/hr 13 | { 14 | "name": "Croatia", 15 | "code": "HR" // alpha2 code 16 | } 17 | ``` 18 | 19 | This endpoint fulfills all needs — for the time being. After a while, a new requirement is to serialize alpha-3 country code along with the existing alpha-2 code. You might be tempted to just add a new key `alpha_3_code` to the response, but why should `code` by default imply alpha-2 code? Now that you have to serialize multiple country codes, wouldn't it make more sense to have keys `alpha_2_code` and `alpha_3_code` so it's immediately obvious what each of them represents? 20 | 21 | If the answer to that question is affirmative, and you know that some clients depend on the `code` key and that they won't be able to rename it to `alpha_2_code` in a timely fashion, but you still want to add the new alpha-3 code, then you could add 2 new attributes alongside the existing ones and communicate the deprecation to the API consumers. 22 | 23 | ```json 24 | // GET api/v1/countries/hr 25 | { 26 | "name": "Croatia", 27 | "code": "HR", // deprecated 28 | "alpha_2_code": "HR", 29 | "alpha_3_code": "HRV" 30 | } 31 | ``` 32 | 33 | *Side note*: notice that if the key for alpha-2 code was named `alpha_2_code` from the start, a change like the mentioned one wouldn't be necessary. That is, the only change would be the addition of a new key `alpha_3_code`, which is not breaking anything. This is one of the reasons why it's important to carefully think about API design and naming in particular. 34 | 35 | ### Breaking changes 36 | 37 | Let's continue using the same example with countries, but this time we'll introduce a breaking change. If you take a closer look at the response, you might notice we lack a way to paginate the resources. 38 | 39 | ```json 40 | // GET api/v1/countries 41 | [ 42 | { 43 | "name": "Croatia", 44 | "alpha_2_code": "HR", 45 | "alpha_3_code": "HRV" 46 | }, 47 | { 48 | "name": "Germany", 49 | "alpha_2_code": "DE", 50 | "alpha_3_code": "DEU" 51 | } 52 | ] 53 | ``` 54 | 55 | The usual way is via a new object, called `meta`, where we can keep some calculated properties the frontend might use, but since we only have an array of country objects, there's no way to distinguish a new object that contains some other stuff, so the response will have to be changed to: 56 | 57 | ```json 58 | { 59 | "results": [ 60 | // here come our country objects 61 | ], 62 | "meta": { 63 | "current_page": 1, 64 | "next_page": 2, 65 | "total_pages": 20, 66 | "total_count": 150 67 | } 68 | } 69 | ``` 70 | 71 | The problem is that we can't introduce this modification to the existing endpoint `api/v1/countries`, so we'll have to leave it as is, and suffer the consequences of having an unpaginated endpoint with potentially many objects. Luckily, we can open a new endpoint `api/v2/countries` and notify our API consumers that the `v1` endpoint is deprecated and will be dropped in the future. 72 | 73 | In short, here are the usual operations upon an existing endpoint and whether they are breaking or not: 74 | 75 | - adding a new attribute - no break 76 | - renaming an attribute - break 77 | - removing an attribute - break 78 | - changing the value type for an attribute - break 79 | - changing the format or structure (eg: JSON to GraphQL) - break 80 | 81 | ## How to version 82 | 83 | There are a number of strategies for API versioning. This is still a widely discussed and contentious topic, but this chapter won't go into details (if you're interested, see [this blog post](https://www.troyhunt.com/your-api-versioning-is-wrong-which-is/)). Rather, we'll propose a versioning strategy which works for us. 84 | 85 | We version APIs by adding a version number in the URL (prefixed with `v`). For example, an API for retrieving countries might have endpoints `/api/v1/countries` and `/api/v2/countries`. 86 | 87 | We include the version in APIs *from the beginning*, starting with `v1`. This choice is a pragmatic one even if you'll never have to release a second version. The cost of adding a version to the URL initially is imperceptible and once it's set up, you don't have to think about it anymore. Adding new versions is also easy because you won't have to change/refactor code from version `v1` in order to support the new version. 88 | 89 | In Rails, this kind of versioning has the following structure in routes: 90 | 91 | ```ruby 92 | # config/routes.rb 93 | namespace :api do 94 | namespace :v1 do 95 | resource :countries 96 | end 97 | 98 | namespace :v2 do 99 | resource :countries 100 | end 101 | end 102 | ``` 103 | 104 | Controllers are then also namespaced under the API version: `Api::V1::CountriesController` for `/api/v1/countries`, and `Api::V2::CountriesController` for `/api/v2/countries`. Maintain this consistency with other abstractions as well: serializers, policies, specs, documentation or any custom-made constructs. 105 | -------------------------------------------------------------------------------- /Databases/Migrations.md: -------------------------------------------------------------------------------- 1 | ## Schema migrations 2 | 3 | The [Rails Migrations Guide](https://edgeguides.rubyonrails.org/active_record_migrations.html) does a very good job at explaining how schema migrations work in Rails. 4 | 5 | Not many people know this, but you can [pass modifiers](https://edgeguides.rubyonrails.org/active_record_migrations.html#passing-modifiers) when you are generating migrations from the command line. 6 | 7 | Whenever possible, **write reversible migrations**. If a migration can't be reversed, then make sure you raise `ActiveRecord::IrreversibleMigration` exception. 8 | 9 | 10 | **Must read** - take a look at [strong_migrations](https://github.com/ankane/strong_migrations) for tips on how to write safer migrations. Please don't hesitate to add the gem to your project, it will help you catch unsafe migrations in development. 11 | 12 | 13 | ## Data migrations 14 | 15 | Here is where things get a bit complicated. As your project grows and evolves, so does your data. At some point, you might realize you forgot to add a default to a field. Or that you want to change an enumeration. Or manipulate the existing data in your database in any way. 16 | 17 | There are two ways of dealing with these problems, and both have its use cases: 18 | 19 | 1. Write a rake task 20 | 2. Write a data migration 21 | 22 | Your first instinct would be to write a simple rake task. You have access to all your models, and it is easily testable and runnable. You can even delete the file afterwards. The problem with the rake task is that you have to remember to run it. It does not run automatically. Also, if you are writing a big feature and need that change in the middle of your schema migrations, then you have a problem. 23 | But if the data migration can be ran after the deployment at any time, and the migration is really really complicated and involves multiple models, then a rake task is the way to go. 24 | 25 | 26 | Writing a data migration is a bit trickier. You start of by writing a schema migration, but instead of doing `add_column` or `rename_column` in your `change` method, you do a `Model.update_all()`. Problems start to arise when, after a month of developing a new feature, you introduce some validations that break that migration. Or you remove a method you are using in that migration. Or you remove the model entirely. And you notice the problem only when you try to deploy your application to the server. 27 | 28 | ## Make your data migrations foolproof 29 | 30 | __Write raw sql queries inside of your data migrations.__ 31 | 32 | Example: 33 | 34 | ``` ruby 35 | def change 36 | execute(<<-SQL 37 | UPDATE users 38 | SET role = 39 | CASE role 40 | WHEN '2' THEN 'developer' 41 | WHEN '3' THEN 'client' 42 | END 43 | SQL 44 | ) 45 | end 46 | ``` 47 | 48 | When writing raw SQL queries, it's best to use a database client, such as [TablePlus](https://tableplus.com/) or [Postico](https://eggerapps.at/postico/) where you can check the syntax and test the query faster. 49 | You can also write the query with ActiveRecord and call the `to_sql` method on the AR query object. 50 | 51 | **Reversible data migration** 52 | 53 | If you try to roll back the migration, the example above will, by default, throw `ActiveRecord::IrreversibleMigration`. If you need to be able to do a rollback, you will need to write your own `down` method: 54 | 55 | ``` ruby 56 | def up 57 | execute(<<-SQL 58 | UPDATE users 59 | SET role = 60 | CASE role 61 | WHEN '2' THEN 'developer' 62 | WHEN '3' THEN 'client' 63 | END 64 | SQL 65 | ) 66 | end 67 | 68 | def down 69 | execute(<<-SQL 70 | UPDATE users 71 | SET role = 72 | CASE role 73 | WHEN 'developer' THEN '2' 74 | WHEN 'client' THEN '3' 75 | END 76 | SQL 77 | ) 78 | end 79 | ``` 80 | -------------------------------------------------------------------------------- /Databases/PG extensions.md: -------------------------------------------------------------------------------- 1 | If you want to use postgres extensions (for example postgis or uuid) on Amazons RDS Postgres database add this migration to your application: 2 | 3 | ```ruby 4 | def change 5 | if Rails.env.in? ['development', 'test'] 6 | enable_extension 'postgis' 7 | else 8 | STDERR.puts 'TELL DEVOPS TO ENABLE Postgis ON DATABASE' 9 | end 10 | end 11 | ``` 12 | 13 | This is because of restrictions on Amazon RDS databases. To be able to enable extension on RDS you need to be a superuser (to have access to scripts on disk). The users we are using on RDS to read and write in our databases do not have superuser rights. That is why you will need to humbly ask our devops to enable it for you. 14 | -------------------------------------------------------------------------------- /Databases/SQL/General.md: -------------------------------------------------------------------------------- 1 | ## Intro 2 | 3 | Working on backend systems almost always necessitates the use of a database where application data can be stored. 4 | The majority of our applications use a relational database as a data storage solution. 5 | 6 | SQL (structured query language) is the tool for communicating with the database. 7 | We usually communicate with the database through the ORM (`ActiveRecord`). But sometimes, the ORM is not powerful enough for a specific problem and you must write raw SQL. 8 | 9 | To be able to **solve complex problems** and to get a **better understanding** of how the ORM works, here are some query examples and advice to help you with that! 10 | 11 | 12 | ## Convention 13 | 14 | All query examples will be compatible with the **PostgreSQL standard**. 15 | 16 | SQL keywords are formatted in all capitals to make them stand out from the text, as in **SELECT**. 17 | 18 | SQL tables and columns are spelled in lowercase, and words are separated by underscores, as in `users`, `first_name`. 19 | 20 | It would make your life easier if you align and format a query you are currently working on. It makes it easier to debug and identify parts of the query. 21 | 22 | SQL is correctly pronounced "ess-cue-ell", not "see-quell". 23 | 24 | In the context of database-related usage, the word index refers to an ordered collection of information. The preferred plural of this word is *indexes*. 25 | 26 | 27 | ## SQL Playground Database 28 | 29 | We have set up a PostgreSQL database instance that you can use to test your queries. 30 | The credentials are stored in our **1Password**. 31 | 32 | ![db-er-diagram](https://user-images.githubusercontent.com/10612221/161582654-63ce291d-c6e7-4b67-b37d-8d858a591cd5.svg) 33 | 34 | ## DB GUI Clients 35 | 36 | There are plenty of GUI tools for interacting with PostgreSQL. Here is the list of the most popular GUI clients - feel free to choose the one you like: 37 | 38 | * [Postico](https://eggerapps.at/postico/) (only for Mac - we have a full version license that you can acquire from your team lead) 39 | * [TablePlus](https://tableplus.com/) 40 | * [Azure DataStudio](https://docs.microsoft.com/en-us/sql/azure-data-studio) 41 | * [ArcType](https://arctype.com/) 42 | * [pgAdmin](https://www.pgadmin.org/) 43 | * [JetBrains DataGrip](https://www.jetbrains.com/datagrip/) 44 | * [SQLPro for Postgres](http://macpostgresclient.com/) 45 | 46 | 47 | Additional Resources: 48 | 49 | * [Best PostgreSQL GUIs in 2021](https://retool.com/blog/best-postgresql-guis-in-2020/) 50 | * [Awesome Postgres / GUIs](https://dhamaniasad.github.io/awesome-postgres/#gui) 51 | 52 | 53 | ## Tools & Resources 54 | 55 | * [How to Interpret PostgreSQL Explain Analyze Output](https://www.cybertec-postgresql.com/en/how-to-interpret-postgresql-explain-analyze-output/) 56 | * [PostgreSQL execution plan visualizer #1](https://explain.dalibo.com/) 57 | * [PostgreSQL execution plan visualizer #2](http://tatiyants.com/pev/#/plans/new) 58 | * [PostgreSQL execution plan visualizer #3](https://explain.depesz.com/) 59 | * [PostgreSQL Cheat Sheet](https://postgrescheatsheet.com) 60 | * [Most Useful PostgreSQL Commands with Examples](https://technobytz.com/most-useful-postgresql-commands.html) 61 | * [DB Fiddle](https://www.db-fiddle.com/) 62 | * [PostgreSQL Ecosystem](https://github.com/EfficiencyGeek/postgresql-ecosystem) 63 | * [PGTune](https://pgtune.leopard.in.ua/) 64 | * [PG Wiki - Don't Do This](https://wiki.postgresql.org/wiki/Don't_Do_This) 65 | * [How the PostgreSQL Query Optimizer Works](https://www.cybertec-postgresql.com/en/how-the-postgresql-query-optimizer-works/) 66 | 67 | 68 | ## Ruby & Rails Database Related Gems 69 | 70 | * [active_record_extended](https://github.com/georgekaraszi/ActiveRecordExtended) 71 | * [rollup](https://github.com/ankane/rollup) 72 | * [groupdate](https://github.com/ankane/groupdate) 73 | * [hightop](https://github.com/ankane/hightop) 74 | * [active_median](https://github.com/ankane/active_median) 75 | * [jsonb_accessor](https://github.com/madeintandem/jsonb_accessor) 76 | * [fx](https://github.com/teoljungberg/fx) 77 | * [activerecord-postgres_enum](https://github.com/bibendi/activerecord-postgres_enum) 78 | * [active_record_doctor](https://github.com/gregnavis/active_record_doctor) 79 | * [hairtrigger](https://github.com/jenseng/hair_trigger) 80 | * [strong_migrations](https://github.com/ankane/strong_migrations) 81 | * [database_consistency](https://github.com/djezzzl/database_consistency) 82 | -------------------------------------------------------------------------------- /Databases/SQL/Indexes.md: -------------------------------------------------------------------------------- 1 | > An index makes the query fast. 2 | 3 | ## Data Structure 4 | 5 | The data structure that is used for indexes is a **B-tree** (balanced tree). That's because the tree depth is equal for all nodes (the distance between root and leaf nodes). 6 | 7 | ## Why is a query not using an index? 8 | 9 | The execution plan of a query can show that the database does not use the index, and uses **sequential scan**. 10 | Possible situations: 11 | 12 | * majority of rows are getting fetched as part of the SQL query 13 | * although the database reads more data, it might need to execute fewer read operations 14 | * there is no index available for specified columns 15 | 16 | > NOTE: Using an index does not mean a query is executed in the best way! 17 | 18 | 19 | ## Composite Index 20 | 21 | It is possible to define an index on two or more columns - that kind of an index is called **composite** (or **concatenated**, **combined**) index. 22 | 23 | **Order matters** - the most important thing is how to choose the column order so the index can be used as often as possible! We as developers should have a feeling for the data (business domain) and properly choose columns for an index. 24 | 25 | Example: 26 | 27 | ```sql 28 | CREATE INDEX idx_employee_department_1 ON employees(employee_id, department_id); 29 | 30 | -- different approach 31 | 32 | CREATE INDEX idx_employee_department_2 ON employees(department_id, employee_id); 33 | ``` 34 | 35 | What index would you create, depends on data usage from the domain perspective. If queries will mostly use the `department_id` column in the `WHERE` part of a query, then it's recommended to use the `idx_employee_department_2` index. Otherwise, if `employee_id` will be used repeatedly, it would probably be more performant to use the `idx_employee_department_1` index. 36 | 37 | When both of the columns would be used on a similar frequency, use the column with more scattered data, on the most left position of an index (in our example it would be `emplyee_id`). 38 | 39 | 40 | ## Function-Based Index 41 | 42 | When you have an index whose definition contains a function or an expression, you have a **function-based index**. 43 | Instead of copying the column data directly into the index, the database applies a function on the column and stores the result into the index. 44 | 45 | Example: 46 | 47 | ```sql 48 | SELECT * 49 | FROM users 50 | WHERE LOWER(email) = 'user@mail.com' 51 | 52 | -- we store lower-cased email as an index value 53 | CREATE INDEX idx_users_email ON users(LOWER(email)) 54 | ``` 55 | 56 | 57 | ## Partial Index 58 | 59 | In some situations, where we query only a part of the table, it makes sense to index only a specific partition of the table. 60 | For example, we don't want to index posts that are deleted (`deleted_at IS NOT NULL`). So, we will create a partial index: 61 | 62 | ```sql 63 | -- index not deleted posts by published_at 64 | CREATE INDEX idx_posts_undeleted ON posts(published_at) WHERE deleted_at IS NULL 65 | ``` 66 | 67 | The main benefit of partial indexes is the smaller size of the index. 68 | 69 | The index size is reduced vertically and horizontally: 70 | 71 | * reduced number of rows - only posts that are not deleted (`deleted_at IS NULL`) 72 | * reduced number of columns - you can skip `deleted_at` column from the index definition 73 | 74 | Resources: 75 | 76 | * [PostgreSQL Documentation](https://www.postgresql.org/docs/current/indexes-partial.html) 77 | -------------------------------------------------------------------------------- /Databases/SQL/Query Examples/01 - The Most Recent Record.md: -------------------------------------------------------------------------------- 1 | ## Problem 2 | 3 | Write a query that will return the most recent post for each post category. 4 | 5 | 6 | ## Input Data 7 | 8 | 9 | | id | title | content | category | created_at | updated_at | 10 | | -- | ------- | ------- | -------- | ------------------- | -------------------------- | 11 | | 1 | Title A | --- | backend | 2021-05-12 00:00:00 | 2021-05-12 15:32:27.423371 | 12 | | 2 | Title B | --- | devops | 2021-04-12 00:00:00 | 2021-05-12 15:32:27.423371 | 13 | | 3 | Title C | --- | devops | 2021-04-26 00:00:00 | 2021-05-12 15:32:27.423371 | 14 | | 4 | Title D | --- | dba | 2021-05-13 00:00:00 | 2021-05-12 15:32:27.423371 | 15 | | 5 | Title E | --- | backend | 2021-05-13 00:00:00 | 2021-05-12 15:32:27.423371 | 16 | 17 | 18 | ## Expected Result 19 | 20 | | id | title | content | category | created_at | updated_at | 21 | | -- | ------- | ------- | -------- | ------------------- | -------------------------- | 22 | | 5 | Title E | --- | backend | 2021-05-13 00:00:00 | 2021-05-12 15:32:27.423371 | 23 | | 4 | Title D | --- | dba | 2021-05-13 00:00:00 | 2021-05-12 15:32:27.423371 | 24 | | 3 | Title C | --- | devops | 2021-04-26 00:00:00 | 2021-05-12 15:32:27.423371 | 25 | 26 | 27 | ## Setup Script 28 | 29 | ```sql 30 | DROP TABLE IF EXISTS posts; 31 | DROP SEQUENCE IF EXISTS posts_id_seq; 32 | 33 | CREATE SEQUENCE posts_id_seq; 34 | 35 | CREATE TABLE posts ( 36 | id BIGINT NOT NULL DEFAULT NEXTVAL('posts_id_seq'::regclass), 37 | title VARCHAR, 38 | content VARCHAR, 39 | category VARCHAR, 40 | created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, 41 | updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, 42 | PRIMARY KEY (id) 43 | ); 44 | 45 | INSERT INTO posts (title, content, category, created_at) VALUES 46 | ('Title A', '---', 'backend', '2021-05-12'), 47 | ('Title B', '---', 'devops', '2021-04-12'), 48 | ('Title C', '---', 'devops', '2021-04-26'), 49 | ('Title D', '---', 'dba', '2021-05-13'), 50 | ('Title E', '---', 'backend', '2021-05-13'); 51 | ``` 52 | 53 | 54 | ## Solution 55 | 56 | ```sql 57 | SELECT DISTINCT ON (category) * 58 | FROM posts 59 | ORDER BY category, created_at DESC; 60 | 61 | 62 | -- alternative solution, using WINDOW function 63 | 64 | SELECT * 65 | FROM ( 66 | SELECT posts.*, 67 | ROW_NUMBER() OVER ( 68 | PARTITION BY category ORDER BY created_at DESC 69 | ) AS rn 70 | FROM posts 71 | ) posts 72 | WHERE rn = 1; 73 | ``` 74 | 75 | 76 | ## Keywords 77 | 78 | * [`DISTINCT ON` (PostgreSQL specific)](https://www.postgresql.org/docs/current/sql-select.html#SQL-DISTINCT) 79 | * basic usage of [`WINDOW` functions](https://www.postgresql.org/docs/current/functions-window.html) 80 | * [`ROW_NUMBER()`](https://www.postgresql.org/docs/current/functions-window.html#FUNCTIONS-WINDOW-TABLE) 81 | -------------------------------------------------------------------------------- /Databases/SQL/Query Examples/02 - Top N Records.md: -------------------------------------------------------------------------------- 1 | ## Problem 2 | 3 | Write a query that will return 3 most recent images for each post. 4 | 5 | 6 | ## Input Data 7 | 8 | | id | post_id | image_url | created_at | updated_at | 9 | |----|---------|--------------|---------------------|----------------------------| 10 | | 1 | 1 | images.com/1 | 2021-03-12 00:00:00 | 2022-02-11 10:26:06.634393 | 11 | | 2 | 2 | images.com/2 | 2021-03-15 00:00:00 | 2022-02-11 10:26:06.634393 | 12 | | 3 | 2 | images.com/3 | 2021-03-20 00:00:00 | 2022-02-11 10:26:06.634393 | 13 | | 4 | 3 | images.com/4 | 2021-04-13 00:00:00 | 2022-02-11 10:26:06.634393 | 14 | | 5 | 1 | images.com/5 | 2021-04-15 00:00:00 | 2022-02-11 10:26:06.634393 | 15 | | 6 | 3 | images.com/6 | 2021-04-17 00:00:00 | 2022-02-11 10:26:06.634393 | 16 | | 7 | 3 | images.com/7 | 2021-04-18 00:00:00 | 2022-02-11 10:26:06.634393 | 17 | | 8 | 1 | images.com/8 | 2021-05-01 00:00:00 | 2022-02-11 10:26:06.634393 | 18 | | 9 | 1 | images.com/9 | 2021-05-02 00:00:00 | 2022-02-11 10:26:06.634393 | 19 | | 10 | 3 | images.com/0 | 2021-05-13 00:00:00 | 2022-02-11 10:26:06.634393 | 20 | 21 | 22 | ## Expected Result 23 | 24 | | id | post_id | image_url | created_at | updated_at | 25 | |----|---------|--------------|---------------------|----------------------------| 26 | | 9 | 1 | images.com/9 | 2021-05-02 00:00:00 | 2022-02-11 10:26:06.634393 | 27 | | 8 | 1 | images.com/8 | 2021-05-01 00:00:00 | 2022-02-11 10:26:06.634393 | 28 | | 5 | 1 | images.com/5 | 2021-04-15 00:00:00 | 2022-02-11 10:26:06.634393 | 29 | | 3 | 2 | images.com/3 | 2021-03-20 00:00:00 | 2022-02-11 10:26:06.634393 | 30 | | 2 | 2 | images.com/2 | 2021-03-15 00:00:00 | 2022-02-11 10:26:06.634393 | 31 | | 10 | 3 | images.com/0 | 2021-05-13 00:00:00 | 2022-02-11 10:26:06.634393 | 32 | | 7 | 3 | images.com/7 | 2021-04-18 00:00:00 | 2022-02-11 10:26:06.634393 | 33 | | 6 | 3 | images.com/6 | 2021-04-17 00:00:00 | 2022-02-11 10:26:06.634393 | 34 | 35 | 36 | ## Setup Script 37 | 38 | ```sql 39 | DROP TABLE IF EXISTS post_images; 40 | DROP SEQUENCE IF EXISTS post_images_id_seq; 41 | 42 | CREATE SEQUENCE post_images_id_seq; 43 | 44 | CREATE TABLE post_images ( 45 | id BIGINT NOT NULL DEFAULT NEXTVAL('post_images_id_seq'::regclass), 46 | post_id BIGINT, 47 | image_url VARCHAR, 48 | created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, 49 | updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, 50 | PRIMARY KEY (id) 51 | ); 52 | 53 | INSERT INTO post_images (post_id, image_url, created_at) VALUES 54 | (1, 'images.com/1', '2021-03-12'), 55 | (2, 'images.com/2', '2021-03-15'), 56 | (2, 'images.com/3', '2021-03-20'), 57 | (3, 'images.com/4', '2021-04-13'), 58 | (1, 'images.com/5', '2021-04-15'), 59 | (3, 'images.com/6', '2021-04-17'), 60 | (3, 'images.com/7', '2021-04-18'), 61 | (1, 'images.com/8', '2021-05-01'), 62 | (1, 'images.com/9', '2021-05-02'), 63 | (3, 'images.com/0', '2021-05-13'); 64 | ``` 65 | 66 | 67 | ## Solution 68 | 69 | ```sql 70 | SELECT 71 | id, post_id, image_url, created_at, updated_at 72 | 73 | FROM ( 74 | SELECT 75 | post_images.*, 76 | ROW_NUMBER() OVER ( 77 | PARTITION BY post_id ORDER BY created_at DESC 78 | ) AS rn 79 | 80 | FROM 81 | post_images 82 | ) post_images 83 | 84 | WHERE rn <= 3; 85 | ``` 86 | 87 | 88 | ## Keywords 89 | 90 | * basic usage of [`WINDOW` functions](https://www.postgresql.org/docs/current/functions-window.html) 91 | * [partitions and frames](https://www.postgresql.org/docs/current/sql-expressions.html#SYNTAX-WINDOW-FUNCTIONS) 92 | * [`ROW_NUMBER()`](https://www.postgresql.org/docs/current/functions-window.html#FUNCTIONS-WINDOW-TABLE) 93 | * [subqueries](https://www.postgresql.org/docs/current/queries-table-expressions.html#QUERIES-SUBQUERIES) 94 | -------------------------------------------------------------------------------- /Databases/SQL/Query Examples/03 - Appointments.md: -------------------------------------------------------------------------------- 1 | ## Problem 2 | 3 | Write a query that will generate appointment slots of 30 minutes (consultations with the content editor), between 08:00 and 16:00 for the current date and for the **content editor with ID = 1**. For each slot, you need to put status *BUSY* if there is a scheduled appointment for the slot, otherwise *FREE*. 4 | 5 | 6 | ## Input Data 7 | 8 | | id | content_editor_id | start_time | end_time | 9 | |----|-------------------|---------------------|---------------------| 10 | | 1 | 1 | 2022-02-11 08:30:00 | 2022-02-11 09:00:00 | 11 | | 2 | 1 | 2022-02-11 09:30:00 | 2022-02-11 10:00:00 | 12 | | 3 | 1 | 2022-02-11 09:30:00 | 2022-02-11 10:00:00 | 13 | | 4 | 1 | 2022-02-11 13:30:00 | 2022-02-11 14:00:00 | 14 | 15 | 16 | ## Expected Result 17 | 18 | | start_time | end_time | status | 19 | |---------------------|---------------------|--------| 20 | | 2022-02-11 08:00:00 | 2022-02-11 08:30:00 | FREE | 21 | | 2022-02-11 08:30:00 | 2022-02-11 09:00:00 | BUSY | 22 | | 2022-02-11 09:00:00 | 2022-02-11 09:30:00 | FREE | 23 | | 2022-02-11 09:30:00 | 2022-02-11 10:00:00 | BUSY | 24 | | 2022-02-11 10:00:00 | 2022-02-11 10:30:00 | FREE | 25 | | 2022-02-11 10:30:00 | 2022-02-11 11:00:00 | FREE | 26 | | 2022-02-11 11:00:00 | 2022-02-11 11:30:00 | FREE | 27 | | 2022-02-11 11:30:00 | 2022-02-11 12:00:00 | FREE | 28 | | 2022-02-11 12:00:00 | 2022-02-11 12:30:00 | FREE | 29 | | 2022-02-11 12:30:00 | 2022-02-11 13:00:00 | FREE | 30 | | 2022-02-11 13:00:00 | 2022-02-11 13:30:00 | FREE | 31 | | 2022-02-11 13:30:00 | 2022-02-11 14:00:00 | BUSY | 32 | | 2022-02-11 14:00:00 | 2022-02-11 14:30:00 | FREE | 33 | | 2022-02-11 14:30:00 | 2022-02-11 15:00:00 | FREE | 34 | | 2022-02-11 15:00:00 | 2022-02-11 15:30:00 | FREE | 35 | | 2022-02-11 15:30:00 | 2022-02-11 16:00:00 | FREE | 36 | 37 | 38 | ## Setup Script 39 | 40 | ```sql 41 | DROP TABLE IF EXISTS appointments; 42 | DROP SEQUENCE IF EXISTS appointments_id_seq; 43 | 44 | CREATE SEQUENCE appointments_id_seq; 45 | 46 | CREATE TABLE appointments ( 47 | id BIGINT NOT NULL DEFAULT NEXTVAL('appointments_id_seq'::regclass), 48 | content_editor_id BIGINT NOT NULL, 49 | start_time TIMESTAMP, 50 | end_time TIMESTAMP, 51 | PRIMARY KEY (id) 52 | ); 53 | 54 | INSERT INTO appointments (content_editor_id, start_time, end_time) VALUES 55 | (1, CURRENT_DATE + INTERVAL '08:30', CURRENT_DATE + INTERVAL '09:00'), 56 | (1, CURRENT_DATE + INTERVAL '09:30', CURRENT_DATE + INTERVAL '10:00'), 57 | (2, CURRENT_DATE + INTERVAL '09:30', CURRENT_DATE + INTERVAL '10:00'), 58 | (1, CURRENT_DATE + INTERVAL '13:30', CURRENT_DATE + INTERVAL '14:00'); 59 | ``` 60 | 61 | 62 | ## Solution 63 | 64 | ```sql 65 | WITH generated_appointments AS ( 66 | SELECT 67 | start_time, 68 | (start_time + INTERVAL '30' MINUTE) AS end_time 69 | 70 | FROM GENERATE_SERIES( 71 | CURRENT_DATE + INTERVAL '08:00', 72 | CURRENT_DATE + INTERVAL '15:30', 73 | INTERVAL '30' MINUTE 74 | ) AS t(start_time) 75 | ) 76 | 77 | SELECT 78 | g.start_time, 79 | g.end_time, 80 | CASE WHEN a.id IS NULL 81 | THEN 'FREE' 82 | ELSE 'BUSY' 83 | END AS status 84 | 85 | FROM generated_appointments AS g 86 | LEFT JOIN appointments AS a ON 87 | (g.start_time, g.end_time) = (a.start_time, a.end_time) AND 88 | a.content_editor_id = 1; 89 | ``` 90 | 91 | 92 | ## Keywords 93 | 94 | * [`LEFT JOIN`](https://www.postgresql.org/docs/current/queries-table-expressions.html#QUERIES-JOIN) 95 | * [`NULL` checking](https://www.postgresql.org/docs/current/functions-comparison.html#FUNCTIONS-COMPARISON-PRED-TABLE) 96 | * [`SWITCH CASE`](https://www.postgresql.org/docs/current/functions-conditional.html#FUNCTIONS-CASE) 97 | * [`DATETIME` manipulation](https://www.postgresql.org/docs/current/functions-datetime.html) 98 | * [`INTERVAL`](https://www.postgresql.org/docs/current/datatype-datetime.html#DATATYPE-DATETIME) 99 | * [`CTE` - Common Table Expressions (or `WITH` clause)](https://www.postgresql.org/docs/current/queries-with.html) 100 | -------------------------------------------------------------------------------- /Databases/SQL/Query Examples/04 - Search Optimization.md: -------------------------------------------------------------------------------- 1 | ## Problem 2 | 3 | You have a mission to optimize querying users by their name and surname by using this query: 4 | 5 | ```sql 6 | SELECT * 7 | FROM users 8 | WHERE first_name || ' ' || last_name ILIKE '%joe%' 9 | ``` 10 | 11 | How would you speed up this query? 12 | 13 | 14 | ## Input Data 15 | 16 | | id | first_name | last_name | role | 17 | |----|------------|-----------|--------| 18 | | 1 | John | Doe | author | 19 | | 2 | Joe | Smith | author | 20 | | 3 | Mark | Cohen | admin | 21 | 22 | 23 | ## Setup 24 | 25 | ```sql 26 | DROP TABLE IF EXISTS users; 27 | DROP SEQUENCE IF EXISTS users_id_seq; 28 | 29 | CREATE SEQUENCE users_id_seq; 30 | 31 | CREATE TABLE users ( 32 | id BIGINT NOT NULL DEFAULT NEXTVAL('users_id_seq'::regclass), 33 | first_name VARCHAR, 34 | last_name VARCHAR, 35 | role VARCHAR, 36 | PRIMARY KEY (id) 37 | ); 38 | 39 | INSERT INTO users (first_name, last_name, role) VALUES 40 | ('John', 'Doe', 'author'), 41 | ('Joe', 'Smith', 'author'), 42 | ('Mark', 'Cohen', 'admin'); 43 | ``` 44 | 45 | 46 | ## Solution 47 | 48 | ```sql 49 | CREATE EXTENSION pg_trgm; 50 | 51 | CREATE INDEX idx_users 52 | ON users USING GIN ((first_name || ' ' || last_name) gin_trgm_ops); 53 | ``` 54 | 55 | > Go ahead and check the performance benefits (you can use `EXPLAIN` to check the execution plan) 56 | 57 | 58 | ## Keywords 59 | 60 | * [defining index on expression (concatenation of first name and last name)](https://www.postgresql.org/docs/current/sql-createindex.html) 61 | * [extension `pg_trgm`](https://www.postgresql.org/docs/current/pgtrgm.html) 62 | * [`GIN` index](https://www.postgresql.org/docs/current/textsearch-indexes.html) 63 | -------------------------------------------------------------------------------- /Databases/SQL/Query Examples/05 - CSV.md: -------------------------------------------------------------------------------- 1 | ## Problem (1) 2 | 3 | You have to export users with the surname “Smith” to CSV. 4 | 5 | 6 | ## Problem (2) 7 | 8 | You have to import new users from a large CSV file. 9 | 10 | 11 | ## Input Data 12 | 13 | | id | first_name | last_name | role | 14 | |----|------------|-----------|--------| 15 | | 1 | John | Doe | author | 16 | | 2 | Joe | Smith | author | 17 | | 3 | Mark | Cohen | admin | 18 | 19 | 20 | ## Setup 21 | 22 | ```sql 23 | DROP TABLE IF EXISTS users; 24 | DROP SEQUENCE IF EXISTS users_id_seq; 25 | 26 | CREATE SEQUENCE users_id_seq; 27 | 28 | CREATE TABLE users ( 29 | id BIGINT NOT NULL DEFAULT NEXTVAL('users_id_seq'::regclass), 30 | first_name VARCHAR, 31 | last_name VARCHAR, 32 | role VARCHAR, 33 | PRIMARY KEY (id) 34 | ); 35 | 36 | INSERT INTO users (first_name, last_name, role) VALUES 37 | ('John', 'Doe', 'author'), 38 | ('Joe', 'Smith', 'author'), 39 | ('Mark', 'Cohen', 'admin'); 40 | ``` 41 | 42 | 43 | ## Solution (1) 44 | 45 | ```sql 46 | COPY (SELECT * FROM users WHERE last_name = 'Smith') 47 | TO '/tmp/users_smith.csv' 48 | WITH DELIMITER ';' CSV HEADER; 49 | ``` 50 | 51 | 52 | ## Solution (2) 53 | 54 | ```sql 55 | COPY users 56 | FROM '/tmp/new_users.csv' 57 | CSV HEADER; 58 | ``` 59 | 60 | > You can read more about PostgreSQL's `COPY` [command on our blog](https://infinum.com/the-capsized-eight/superfast-csv-imports-using-postgresqls-copy). 61 | 62 | ## Keywords 63 | 64 | * [`COPY`](https://www.postgresql.org/docs/current/sql-copy.html) 65 | -------------------------------------------------------------------------------- /Databases/SQL/Query Examples/06 - Chat.md: -------------------------------------------------------------------------------- 1 | ## Problem 2 | 3 | You have to find a chat between two users with IDs 1 and 4. Users can be provided in no particular order. 4 | 5 | 6 | ## Input Data 7 | 8 | | id | user1_id | user2_id | created_at | 9 | |----|----------|----------|---------------------| 10 | | 1 | 1 | 2 | 2022-02-11 08:30:00 | 11 | | 2 | 3 | 2 | 2022-02-12 09:30:00 | 12 | | 3 | 4 | 1 | 2022-02-13 10:30:00 | 13 | 14 | 15 | ## Expected Result 16 | 17 | | id | user1_id | user2_id | created_at | 18 | |----|----------|----------|---------------------| 19 | | 3 | 4 | 1 | 2022-02-13 10:30:00 | 20 | 21 | 22 | ## Setup 23 | 24 | ```sql 25 | DROP TABLE IF EXISTS chats; 26 | DROP SEQUENCE IF EXISTS chats_id_seq; 27 | 28 | CREATE SEQUENCE chats_id_seq; 29 | 30 | CREATE TABLE chats ( 31 | id BIGINT NOT NULL DEFAULT NEXTVAL('chats_id_seq'::regclass), 32 | user1_id VARCHAR, 33 | user2_id VARCHAR, 34 | created_at TIMESTAMP, 35 | PRIMARY KEY (id) 36 | ); 37 | 38 | INSERT INTO chats (user1_id, user2_id, created_at) VALUES 39 | (1, 2, '2022-02-11 08:30:00'), 40 | (3, 2, '2022-02-12 09:30:00'), 41 | (4, 1, '2022-02-13 10:30:00'); 42 | ``` 43 | 44 | 45 | ## Solution (1) 46 | 47 | ```sql 48 | SELECT 49 | * 50 | 51 | FROM 52 | chats 53 | 54 | WHERE 55 | (1, 4) IN ((user1_id, user2_id), (user2_id, user1_id)) 56 | ``` 57 | 58 | 59 | ## Solution (2) 60 | 61 | ```sql 62 | SELECT 63 | * 64 | 65 | FROM 66 | chats 67 | 68 | WHERE 69 | (user1_id, user2_id) IN ((1, 4), (4, 1)) 70 | ``` 71 | 72 | 73 | ## Keywords 74 | 75 | * tuples 76 | -------------------------------------------------------------------------------- /Databases/SQL/Query Examples/07 - Circular Sort.md: -------------------------------------------------------------------------------- 1 | ## Problem 2 | 3 | Write a query that will return five posts published/created after a specific post (let's use a post with ID = 5 as a reference). 4 | If there are less than five posts published after that post, populate the list with the posts from the same category (`backend`) first (ordered by `created_at` ascending). Then use all other posts ordered by the `created_at` ascending. 5 | 6 | 7 | ## Input Data 8 | 9 | | id | title | content | category | created_at | updated_at | 10 | | -- | ------- | ------- | -------- | ------------------- | ------------------- | 11 | | 1 | Title A | --- | backend | 2021-03-17 00:00:00 | 2021-03-17 00:00:00 | 12 | | 2 | Title B | --- | devops | 2021-04-12 00:00:00 | 2021-04-12 00:00:00 | 13 | | 3 | Title C | --- | devops | 2021-04-26 00:00:00 | 2021-04-26 00:00:00 | 14 | | 4 | Title D | --- | dba | 2021-05-13 00:00:00 | 2021-05-13 00:00:00 | 15 | | 5 | Title E | --- | backend | 2021-05-14 00:00:00 | 2021-05-14 00:00:00 | 16 | | 6 | Title F | --- | dba | 2021-06-13 00:00:00 | 2021-06-13 00:00:00 | 17 | | 7 | Title G | --- | backend | 2021-07-22 00:00:00 | 2021-07-22 00:00:00 | 18 | 19 | 20 | ## Expected Result 21 | 22 | | id | title | content | category | created_at | updated_at | 23 | |----|---------|---------|----------|---------------------|---------------------| 24 | | 6 | Title F | --- | backend | 2021-06-13 00:00:00 | 2021-06-13 00:00:00 | 25 | | 7 | Title G | --- | backend | 2021-07-22 00:00:00 | 2021-07-22 00:00:00 | 26 | | 1 | Title A | --- | backend | 2021-03-17 00:00:00 | 2021-03-17 00:00:00 | 27 | | 2 | Title B | --- | devops | 2021-04-12 00:00:00 | 2021-04-12 00:00:00 | 28 | | 3 | Title C | --- | devops | 2021-04-26 00:00:00 | 2021-04-26 00:00:00 | 29 | 30 | 31 | ## Setup Script 32 | 33 | ```sql 34 | DROP TABLE IF EXISTS posts; 35 | DROP SEQUENCE IF EXISTS posts_id_seq; 36 | 37 | CREATE SEQUENCE posts_id_seq; 38 | 39 | CREATE TABLE posts ( 40 | id BIGINT NOT NULL DEFAULT NEXTVAL('posts_id_seq'::regclass), 41 | title VARCHAR, 42 | content VARCHAR, 43 | category VARCHAR, 44 | created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, 45 | updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, 46 | PRIMARY KEY (id) 47 | ); 48 | 49 | INSERT INTO posts (title, content, category, created_at, updated_at) VALUES 50 | ('Title A', '---', 'backend', '2021-03-17', '2021-03-17'), 51 | ('Title B', '---', 'devops', '2021-04-12', '2021-04-12'), 52 | ('Title C', '---', 'devops', '2021-04-26', '2021-04-26'), 53 | ('Title D', '---', 'dba', '2021-05-13', '2021-05-13'), 54 | ('Title E', '---', 'backend', '2021-05-14', '2021-05-14'), 55 | ('Title F', '---', 'backend', '2021-06-13', '2021-06-13'), 56 | ('Title G', '---', 'backend', '2021-07-22', '2021-07-22'); 57 | ``` 58 | 59 | 60 | ## Solution 61 | 62 | ```sql 63 | SELECT 64 | * 65 | 66 | FROM 67 | posts 68 | 69 | WHERE 70 | id != 5 71 | 72 | ORDER BY 73 | CASE WHEN id > 5 THEN 0 ELSE 1 END, 74 | CASE WHEN category = 'backend' THEN 0 ELSE 1 END, 75 | created_at 76 | 77 | LIMIT 78 | 5 79 | ``` 80 | 81 | 82 | ## Keywords 83 | 84 | * circular [order](https://www.postgresql.org/docs/current/queries-order.html) 85 | * custom order with [`CASE`](https://www.postgresql.org/docs/current/functions-conditional.html#FUNCTIONS-CASE) 86 | -------------------------------------------------------------------------------- /Databases/SQL/Query Examples/08 - UPDATE with subquery.md: -------------------------------------------------------------------------------- 1 | ## Problem 2 | 3 | For the posts table, we want to separate category to its own table and then reference category by the foreign key. 4 | Write a query that will update the current posts table data to the suggested structure. 5 | 6 | 7 | ## Input Data 8 | 9 | `posts` 10 | 11 | | id | title | content | category | created_at | updated_at | 12 | | -- | ------- | ------- | -------- | ------------------- | ------------------- | 13 | | 1 | Title A | --- | backend | 2021-03-17 00:00:00 | 2021-03-17 00:00:00 | 14 | | 2 | Title B | --- | devops | 2021-04-12 00:00:00 | 2021-04-12 00:00:00 | 15 | | 3 | Title C | --- | devops | 2021-04-26 00:00:00 | 2021-04-26 00:00:00 | 16 | | 4 | Title D | --- | dba | 2021-05-13 00:00:00 | 2021-05-13 00:00:00 | 17 | | 5 | Title E | --- | backend | 2021-05-14 00:00:00 | 2021-05-14 00:00:00 | 18 | | 6 | Title F | --- | dba | 2021-06-13 00:00:00 | 2021-06-13 00:00:00 | 19 | | 7 | Title G | --- | backend | 2021-07-22 00:00:00 | 2021-07-22 00:00:00 | 20 | 21 | 22 | `categories` 23 | 24 | | id | name 25 | | -- | ------- 26 | | 1 | backend 27 | | 2 | devops 28 | | 3 | dba 29 | 30 | 31 | ## Expected Result 32 | 33 | | id | title | content | category | created_at | updated_at | 34 | | -- | ------- | ------- | -------- | ------------------- | ------------------- | 35 | | 1 | Title A | --- | 1 | 2021-03-17 00:00:00 | 2021-03-17 00:00:00 | 36 | | 2 | Title B | --- | 2 | 2021-04-12 00:00:00 | 2021-04-12 00:00:00 | 37 | | 3 | Title C | --- | 2 | 2021-04-26 00:00:00 | 2021-04-26 00:00:00 | 38 | | 4 | Title D | --- | 3 | 2021-05-13 00:00:00 | 2021-05-13 00:00:00 | 39 | | 5 | Title E | --- | 1 | 2021-05-14 00:00:00 | 2021-05-14 00:00:00 | 40 | | 6 | Title F | --- | 3 | 2021-06-13 00:00:00 | 2021-06-13 00:00:00 | 41 | | 7 | Title G | --- | 1 | 2021-07-22 00:00:00 | 2021-07-22 00:00:00 | 42 | 43 | 44 | ## Setup Script 45 | 46 | ```sql 47 | DROP TABLE IF EXISTS posts; 48 | DROP SEQUENCE IF EXISTS posts_id_seq; 49 | 50 | CREATE SEQUENCE posts_id_seq; 51 | 52 | CREATE TABLE posts ( 53 | id BIGINT NOT NULL DEFAULT NEXTVAL('posts_id_seq'::regclass), 54 | title VARCHAR, 55 | content VARCHAR, 56 | category VARCHAR, 57 | created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, 58 | updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, 59 | PRIMARY KEY (id) 60 | ); 61 | 62 | INSERT INTO posts (title, content, category, created_at, updated_at) VALUES 63 | ('Title A', '---', 'backend', '2021-03-17', '2021-03-17'), 64 | ('Title B', '---', 'devops', '2021-04-12', '2021-04-12'), 65 | ('Title C', '---', 'devops', '2021-04-26', '2021-04-26'), 66 | ('Title D', '---', 'dba', '2021-05-13', '2021-05-13'), 67 | ('Title E', '---', 'backend', '2021-05-14', '2021-05-14'), 68 | ('Title F', '---', 'backend', '2021-06-13', '2021-06-13'), 69 | ('Title G', '---', 'backend', '2021-07-22', '2021-07-22'); 70 | 71 | DROP TABLE IF EXISTS categories; 72 | DROP SEQUENCE IF EXISTS categories_id_seq; 73 | 74 | CREATE SEQUENCE categories_id_seq; 75 | 76 | CREATE TABLE categories ( 77 | id BIGINT NOT NULL DEFAULT NEXTVAL('categories_id_seq'::regclass), 78 | name VARCHAR, 79 | PRIMARY KEY (id) 80 | ); 81 | 82 | INSERT INTO categories (name) VALUES 83 | ('backend'), 84 | ('devops'), 85 | ('dba'); 86 | ``` 87 | 88 | 89 | ## Solution 90 | 91 | ```sql 92 | UPDATE 93 | posts 94 | 95 | SET 96 | category = categories.id 97 | 98 | FROM 99 | categories 100 | 101 | WHERE 102 | posts.category = categories.name 103 | ``` 104 | 105 | 106 | ## Keywords 107 | 108 | * [`UPDATE` with `FROM`](https://www.postgresql.org/docs/current/sql-update.html) 109 | * using `FROM` as `JOIN` within `UPDATE` statement 110 | -------------------------------------------------------------------------------- /Databases/SQL/Query Examples/09 - LATERAL JOIN.md: -------------------------------------------------------------------------------- 1 | ## Problem 2 | 3 | Prepare a database view which will hold the data for similar posts. 4 | For calculating the similarity score, use the following formula: 5 | 6 | * if post is in the same category - `2 points` 7 | * for common keywords - `1 point` for each common keyword 8 | 9 | 10 | ## Input Data 11 | 12 | `posts` 13 | 14 | | id | title | content | category | created_at | updated_at | 15 | | -- | ------- | ------- | -------- | ------------------- | ------------------- | 16 | | 1 | Title A | --- | backend | 2021-03-17 00:00:00 | 2021-03-17 00:00:00 | 17 | | 2 | Title B | --- | devops | 2021-04-12 00:00:00 | 2021-04-12 00:00:00 | 18 | | 3 | Title C | --- | devops | 2021-04-26 00:00:00 | 2021-04-26 00:00:00 | 19 | | 4 | Title D | --- | dba | 2021-05-13 00:00:00 | 2021-05-13 00:00:00 | 20 | | 5 | Title E | --- | backend | 2021-05-14 00:00:00 | 2021-05-14 00:00:00 | 21 | | 6 | Title F | --- | dba | 2021-06-13 00:00:00 | 2021-06-13 00:00:00 | 22 | | 7 | Title G | --- | backend | 2021-07-22 00:00:00 | 2021-07-22 00:00:00 | 23 | 24 | 25 | `keywords` 26 | 27 | | id | content | created_at | updated_at | 28 | | --- | -------- | -------------------------- | -------------------------- | 29 | | 1 | database | 2022-03-14 14:25:47.778108 | 2022-03-14 14:25:47.778108 | 30 | | 2 | sql | 2022-03-14 14:25:47.778108 | 2022-03-14 14:25:47.778108 | 31 | | 3 | rails | 2022-03-14 14:25:47.778108 | 2022-03-14 14:25:47.778108 | 32 | | 4 | ruby | 2022-03-14 14:25:47.778108 | 2022-03-14 14:25:47.778108 | 33 | | 5 | aws | 2022-03-14 14:25:47.778108 | 2022-03-14 14:25:47.778108 | 34 | | 6 | linux | 2022-03-14 14:25:47.778108 | 2022-03-14 14:25:47.778108 | 35 | | 7 | .net | 2022-03-14 14:25:47.778108 | 2022-03-14 14:25:47.778108 | 36 | | 8 | python | 2022-03-14 14:25:47.778108 | 2022-03-14 14:25:47.778108 | 37 | | 9 | csv | 2022-03-14 14:25:47.778108 | 2022-03-14 14:25:47.778108 | 38 | 39 | 40 | `post_keywords` 41 | 42 | | id | post_id | keyword_id | created_at | updated_at | 43 | | --- | ------- | ---------- | -------------------------- | -------------------------- | 44 | | 1 | 1 | 1 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 45 | | 2 | 1 | 2 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 46 | | 3 | 1 | 3 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 47 | | 4 | 2 | 5 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 48 | | 5 | 2 | 8 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 49 | | 6 | 3 | 5 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 50 | | 7 | 3 | 6 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 51 | | 8 | 4 | 1 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 52 | | 9 | 4 | 2 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 53 | | 10 | 4 | 5 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 54 | | 11 | 4 | 6 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 55 | | 12 | 5 | 1 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 56 | | 13 | 5 | 2 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 57 | | 14 | 5 | 3 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 58 | | 15 | 5 | 4 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 59 | | 16 | 5 | 9 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 60 | | 17 | 6 | 1 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 61 | | 18 | 7 | 1 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 62 | | 19 | 7 | 5 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 63 | | 20 | 7 | 6 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 64 | | 21 | 7 | 7 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 65 | | 22 | 8 | 1 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 66 | 67 | 68 | ## Expected Result 69 | 70 | | reference_post_id | similar_post_id | similarity_score | 71 | | ----------------- | --------------- | ---------------- | 72 | | 1 | 5 | 5 | 73 | | 1 | 7 | 3 | 74 | | 1 | 4 | 2 | 75 | | 1 | 8 | 1 | 76 | | 1 | 6 | 1 | 77 | | 2 | 3 | 3 | 78 | | 2 | 4 | 1 | 79 | | 2 | 7 | 1 | 80 | | 3 | 2 | 3 | 81 | | 3 | 4 | 2 | 82 | | 3 | 7 | 2 | 83 | | 4 | 7 | 3 | 84 | | 4 | 6 | 3 | 85 | | 4 | 5 | 2 | 86 | | 4 | 1 | 2 | 87 | | 4 | 3 | 2 | 88 | | 4 | 8 | 1 | 89 | | 4 | 2 | 1 | 90 | | 5 | 1 | 5 | 91 | | 5 | 7 | 3 | 92 | | 5 | 4 | 2 | 93 | | 5 | 8 | 1 | 94 | | 5 | 6 | 1 | 95 | | 6 | 4 | 3 | 96 | | 6 | 5 | 1 | 97 | | 6 | 8 | 1 | 98 | | 6 | 1 | 1 | 99 | | 6 | 7 | 1 | 100 | | 7 | 5 | 3 | 101 | | 7 | 4 | 3 | 102 | | 7 | 1 | 3 | 103 | | 7 | 3 | 2 | 104 | | 7 | 8 | 1 | 105 | | 7 | 2 | 1 | 106 | | 7 | 6 | 1 | 107 | | 8 | 5 | 1 | 108 | | 8 | 6 | 1 | 109 | | 8 | 4 | 1 | 110 | | 8 | 1 | 1 | 111 | | 8 | 7 | 1 | 112 | 113 | 114 | ## Setup Script 115 | 116 | ```sql 117 | DROP TABLE IF EXISTS posts CASCADE; 118 | DROP SEQUENCE IF EXISTS posts_id_seq; 119 | 120 | CREATE SEQUENCE posts_id_seq; 121 | 122 | CREATE TABLE posts ( 123 | id BIGINT NOT NULL DEFAULT NEXTVAL('posts_id_seq'::regclass), 124 | title VARCHAR, 125 | content VARCHAR, 126 | category VARCHAR, 127 | created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, 128 | updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, 129 | PRIMARY KEY (id) 130 | ); 131 | 132 | INSERT INTO posts (title, content, category, created_at, updated_at) VALUES 133 | ('Title A', '---', 'backend', '2021-03-17', '2021-03-17'), 134 | ('Title B', '---', 'devops', '2021-04-12', '2021-04-12'), 135 | ('Title C', '---', 'devops', '2021-04-26', '2021-04-26'), 136 | ('Title D', '---', 'dba', '2021-05-13', '2021-05-13'), 137 | ('Title E', '---', 'backend', '2021-05-14', '2021-05-14'), 138 | ('Title F', '---', 'dba', '2021-06-13', '2021-06-13'), 139 | ('Title G', '---', 'backend', '2021-07-22', '2021-07-22'), 140 | ('Title H', '---', 'design', '2021-07-22', '2021-07-22'); 141 | 142 | DROP TABLE IF EXISTS keywords CASCADE; 143 | DROP SEQUENCE IF EXISTS keywords_id_seq; 144 | 145 | CREATE SEQUENCE keywords_id_seq; 146 | 147 | CREATE TABLE keywords ( 148 | id BIGINT NOT NULL DEFAULT NEXTVAL('keywords_id_seq'::regclass), 149 | content VARCHAR, 150 | created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, 151 | updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, 152 | PRIMARY KEY (id) 153 | ); 154 | 155 | INSERT INTO keywords (content) VALUES 156 | ('database'), 157 | ('sql'), 158 | ('rails'), 159 | ('ruby'), 160 | ('aws'), 161 | ('linux'), 162 | ('.net'), 163 | ('python'), 164 | ('csv'); 165 | 166 | DROP TABLE IF EXISTS post_keywords CASCADE; 167 | DROP SEQUENCE IF EXISTS post_keywords_id_seq; 168 | 169 | CREATE SEQUENCE post_keywords_id_seq; 170 | 171 | CREATE TABLE post_keywords ( 172 | id BIGINT NOT NULL DEFAULT NEXTVAL('post_keywords_id_seq'::regclass), 173 | post_id BIGINT NOT NULL, 174 | keyword_id BIGINT NOT NULL, 175 | created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, 176 | updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, 177 | PRIMARY KEY (id), 178 | FOREIGN KEY (post_id) REFERENCES posts (id), 179 | FOREIGN KEY (keyword_id) REFERENCES keywords (id) 180 | ); 181 | 182 | INSERT INTO post_keywords (post_id, keyword_id) VALUES 183 | (1, 1), 184 | (1, 2), 185 | (1, 3), 186 | (2, 5), 187 | (2, 8), 188 | (3, 5), 189 | (3, 6), 190 | (4, 1), 191 | (4, 2), 192 | (4, 5), 193 | (4, 6), 194 | (5, 1), 195 | (5, 2), 196 | (5, 3), 197 | (5, 4), 198 | (5, 9), 199 | (6, 1), 200 | (7, 1), 201 | (7, 5), 202 | (7, 6), 203 | (7, 7), 204 | (8, 1); 205 | ``` 206 | 207 | 208 | ## Solution 209 | 210 | ```sql 211 | CREATE VIEW similar_posts AS 212 | 213 | SELECT 214 | posts.id AS reference_post_id, 215 | 216 | COALESCE( 217 | similar_posts_by_category.post_id, 218 | similar_posts_by_common_keywords.post_id 219 | ) AS similar_post_id, 220 | 221 | COALESCE(similar_posts_by_category.similarity_score, 0) + 222 | COALESCE(similar_posts_by_common_keywords.similarity_score, 0) AS similarity_score 223 | 224 | FROM 225 | posts, 226 | 227 | LATERAL ( 228 | SELECT 229 | posts.id reference_post_id, 230 | similar_category_posts.id AS post_id, 231 | 2 AS similarity_score 232 | 233 | FROM 234 | posts AS similar_category_posts 235 | 236 | WHERE 237 | posts.id != similar_category_posts.id 238 | AND posts.category = similar_category_posts.category 239 | ) similar_posts_by_category 240 | 241 | FULL JOIN LATERAL ( 242 | SELECT 243 | posts.id reference_post_id, 244 | post_keywords.post_id, 245 | COUNT(*) AS similarity_score 246 | 247 | FROM 248 | post_keywords 249 | 250 | WHERE 251 | posts.id != post_keywords.post_id 252 | AND post_keywords.keyword_id IN ( 253 | SELECT 254 | keyword_id 255 | 256 | FROM 257 | post_keywords 258 | 259 | WHERE 260 | post_id = posts.id 261 | ) 262 | 263 | GROUP BY 264 | post_keywords.post_id 265 | ) similar_posts_by_common_keywords 266 | ON similar_posts_by_category.post_id = similar_posts_by_common_keywords.post_id 267 | 268 | ORDER BY 269 | reference_post_id, 270 | similarity_score DESC 271 | ``` 272 | 273 | 274 | ## Keywords 275 | 276 | * [`LATERAL` join](https://www.postgresql.org/docs/current/queries-table-expressions.html#QUERIES-LATERAL) 277 | * [`CROSS` join](https://www.postgresql.org/docs/current/queries-table-expressions.html#id-1.5.6.6.5.6.4.2.1.1) 278 | * [subqueries](https://www.postgresql.org/docs/current/queries-table-expressions.html#QUERIES-SUBQUERIES) 279 | * [`COALESCE`](https://www.postgresql.org/docs/current/functions-conditional.html#FUNCTIONS-COALESCE-NVL-IFNULL) 280 | -------------------------------------------------------------------------------- /Databases/SQL/Query Examples/10 - Keywords Count.md: -------------------------------------------------------------------------------- 1 | ## Problem 2 | 3 | Write a query that will return the list of posts, where each post can have: 4 | 5 | * only `database` keyword 6 | * only `sql` keyword 7 | * exactly `database` and `sql` keywords 8 | 9 | 10 | ## Input Data 11 | 12 | `posts` 13 | 14 | | id | title | content | category | created_at | updated_at | 15 | | -- | ------- | ------- | -------- | ------------------- | ------------------- | 16 | | 1 | Title A | --- | backend | 2021-03-17 00:00:00 | 2021-03-17 00:00:00 | 17 | | 2 | Title B | --- | devops | 2021-04-12 00:00:00 | 2021-04-12 00:00:00 | 18 | | 3 | Title C | --- | devops | 2021-04-26 00:00:00 | 2021-04-26 00:00:00 | 19 | | 4 | Title D | --- | dba | 2021-05-13 00:00:00 | 2021-05-13 00:00:00 | 20 | | 5 | Title E | --- | backend | 2021-05-14 00:00:00 | 2021-05-14 00:00:00 | 21 | | 6 | Title F | --- | dba | 2021-06-13 00:00:00 | 2021-06-13 00:00:00 | 22 | | 7 | Title G | --- | backend | 2021-07-22 00:00:00 | 2021-07-22 00:00:00 | 23 | 24 | 25 | `keywords` 26 | 27 | | id | content | created_at | updated_at | 28 | | --- | -------- | -------------------------- | -------------------------- | 29 | | 1 | database | 2022-03-14 14:25:47.778108 | 2022-03-14 14:25:47.778108 | 30 | | 2 | sql | 2022-03-14 14:25:47.778108 | 2022-03-14 14:25:47.778108 | 31 | | 3 | rails | 2022-03-14 14:25:47.778108 | 2022-03-14 14:25:47.778108 | 32 | | 4 | ruby | 2022-03-14 14:25:47.778108 | 2022-03-14 14:25:47.778108 | 33 | | 5 | aws | 2022-03-14 14:25:47.778108 | 2022-03-14 14:25:47.778108 | 34 | | 6 | linux | 2022-03-14 14:25:47.778108 | 2022-03-14 14:25:47.778108 | 35 | | 7 | .net | 2022-03-14 14:25:47.778108 | 2022-03-14 14:25:47.778108 | 36 | | 8 | python | 2022-03-14 14:25:47.778108 | 2022-03-14 14:25:47.778108 | 37 | | 9 | csv | 2022-03-14 14:25:47.778108 | 2022-03-14 14:25:47.778108 | 38 | 39 | 40 | `post_keywords` 41 | 42 | | id | post_id | keyword_id | created_at | updated_at | 43 | | --- | ------- | ---------- | -------------------------- | -------------------------- | 44 | | 1 | 1 | 1 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 45 | | 2 | 1 | 2 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 46 | | 3 | 1 | 3 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 47 | | 4 | 2 | 5 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 48 | | 5 | 2 | 8 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 49 | | 6 | 3 | 5 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 50 | | 7 | 3 | 6 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 51 | | 8 | 4 | 1 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 52 | | 9 | 4 | 2 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 53 | | 10 | 4 | 5 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 54 | | 11 | 4 | 6 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 55 | | 12 | 5 | 1 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 56 | | 13 | 5 | 2 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 57 | | 14 | 5 | 3 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 58 | | 15 | 5 | 4 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 59 | | 16 | 5 | 9 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 60 | | 17 | 6 | 1 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 61 | | 18 | 6 | 2 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 62 | | 19 | 7 | 1 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 63 | | 20 | 7 | 5 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 64 | | 21 | 7 | 6 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 65 | | 22 | 7 | 7 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 66 | | 23 | 8 | 1 | 2022-03-14 14:25:47.803197 | 2022-03-14 14:25:47.803197 | 67 | 68 | 69 | ## Expected Result 70 | 71 | | id | title | content | category | created_at | updated_at | 72 | | -- | ------- | ------- | -------- | ------------------- | ------------------- | 73 | | 6 | Title F | --- | dba | 2021-06-13 00:00:00 | 2021-06-13 00:00:00 | 74 | 75 | 76 | ## Setup Script 77 | 78 | ```sql 79 | DROP TABLE IF EXISTS posts CASCADE; 80 | DROP SEQUENCE IF EXISTS posts_id_seq; 81 | 82 | CREATE SEQUENCE posts_id_seq; 83 | 84 | CREATE TABLE posts ( 85 | id BIGINT NOT NULL DEFAULT NEXTVAL('posts_id_seq'::regclass), 86 | title VARCHAR, 87 | content VARCHAR, 88 | category VARCHAR, 89 | created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, 90 | updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, 91 | PRIMARY KEY (id) 92 | ); 93 | 94 | INSERT INTO posts (title, content, category, created_at, updated_at) VALUES 95 | ('Title A', '---', 'backend', '2021-03-17', '2021-03-17'), 96 | ('Title B', '---', 'devops', '2021-04-12', '2021-04-12'), 97 | ('Title C', '---', 'devops', '2021-04-26', '2021-04-26'), 98 | ('Title D', '---', 'dba', '2021-05-13', '2021-05-13'), 99 | ('Title E', '---', 'backend', '2021-05-14', '2021-05-14'), 100 | ('Title F', '---', 'dba', '2021-06-13', '2021-06-13'), 101 | ('Title G', '---', 'backend', '2021-07-22', '2021-07-22'), 102 | ('Title H', '---', 'design', '2021-07-22', '2021-07-22'); 103 | 104 | DROP TABLE IF EXISTS keywords CASCADE; 105 | DROP SEQUENCE IF EXISTS keywords_id_seq; 106 | 107 | CREATE SEQUENCE keywords_id_seq; 108 | 109 | CREATE TABLE keywords ( 110 | id BIGINT NOT NULL DEFAULT NEXTVAL('keywords_id_seq'::regclass), 111 | content VARCHAR, 112 | created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, 113 | updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, 114 | PRIMARY KEY (id) 115 | ); 116 | 117 | INSERT INTO keywords (content) VALUES 118 | ('database'), 119 | ('sql'), 120 | ('rails'), 121 | ('ruby'), 122 | ('aws'), 123 | ('linux'), 124 | ('.net'), 125 | ('python'), 126 | ('csv'); 127 | 128 | DROP TABLE IF EXISTS post_keywords CASCADE; 129 | DROP SEQUENCE IF EXISTS post_keywords_id_seq; 130 | 131 | CREATE SEQUENCE post_keywords_id_seq; 132 | 133 | CREATE TABLE post_keywords ( 134 | id BIGINT NOT NULL DEFAULT NEXTVAL('post_keywords_id_seq'::regclass), 135 | post_id BIGINT NOT NULL, 136 | keyword_id BIGINT NOT NULL, 137 | created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, 138 | updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, 139 | PRIMARY KEY (id), 140 | FOREIGN KEY (post_id) REFERENCES posts (id), 141 | FOREIGN KEY (keyword_id) REFERENCES keywords (id) 142 | ); 143 | 144 | INSERT INTO post_keywords (post_id, keyword_id) VALUES 145 | (1, 1), 146 | (1, 2), 147 | (1, 3), 148 | (2, 5), 149 | (2, 8), 150 | (3, 5), 151 | (3, 6), 152 | (4, 1), 153 | (4, 2), 154 | (4, 5), 155 | (4, 6), 156 | (5, 1), 157 | (5, 2), 158 | (5, 3), 159 | (5, 4), 160 | (5, 9), 161 | (6, 1), 162 | (6, 2), 163 | (7, 1), 164 | (7, 5), 165 | (7, 6), 166 | (7, 7), 167 | (8, 1); 168 | ``` 169 | 170 | 171 | ## Solution 172 | 173 | ```sql 174 | SELECT 175 | posts.* 176 | 177 | FROM 178 | posts 179 | 180 | WHERE 181 | posts.id IN ( 182 | SELECT 183 | post_id 184 | 185 | FROM 186 | post_keywords 187 | JOIN keywords ON keywords.id = post_keywords.keyword_id 188 | 189 | GROUP BY 190 | post_id 191 | 192 | HAVING 193 | COUNT(keywords.id) = SUM(CASE WHEN keywords.content IN ('sql', 'database') THEN 1 ELSE 0 END) 194 | ) 195 | ``` 196 | 197 | 198 | ## Keywords 199 | 200 | * [subqueries](https://www.postgresql.org/docs/current/queries-table-expressions.html#QUERIES-SUBQUERIES) 201 | * [`SWITCH CASE`](https://www.postgresql.org/docs/current/functions-conditional.html#FUNCTIONS-CASE) 202 | * [aggregate functions](https://www.postgresql.org/docs/current/functions-aggregate.html) 203 | -------------------------------------------------------------------------------- /Databases/SQL/Tips & Tricks.md: -------------------------------------------------------------------------------- 1 | ## N-queries 2 | 3 | You should avoid making (generating) multiple DB queries from the application if you can fetch the needed data in a single query. Always check what queries are executed in your (Rails) log. 4 | 5 | Also, when you work on a complex SQL calculation, avoid pulling intermediate results to your application, which will be used for the next step in your query. It can be a very resource-intensive operation. 6 | 7 | 8 | ## UNION (ALL) 9 | 10 | When you use the `UNION` operator for combining result sets of two or more query statements, double-check if you should use `UNION` or `UNION ALL`. 11 | 12 | The `UNION` operator is *less performant* because it **removes all duplicate rows** from the combined data set. 13 | To retain the duplicate rows, you use the `UNION ALL` instead. 14 | 15 | 16 | ## VIEWs 17 | 18 | A database `VIEW` is a pretty useful SQL mechanism. You can use it for: 19 | 20 | * hiding the complexity of queries 21 | * reducing complexity by naming a part of the complex query 22 | * exposing a data subset to the outer world 23 | 24 | A plain `VIEW` contains only a definition - that means the query will be executed each time you fetch data from the view. 25 | 26 | The results can be "cached" by using a `MATERIALIZED VIEW`. Basically, a `MATERIALIZED VIEW` acts as a read-only plain table. To update data in materialized views, use the `REFRESH` statement. 27 | 28 | `REFRESH` completely replaces the content of a materialized view. During the refresh process, the view is locked. To refresh a materialized view without locking, use `REFRESH MATERIALIZED VIEW CONCURRENTLY`. 29 | 30 | 31 | ### Plain or materialized views? 32 | 33 | So, the rule of thumb when you have to choose between plain or materialized VIEW is the frequency of data updates. 34 | `VIEWs` are generally used when data is to be accessed infrequently and data in the table get updated frequently. 35 | On the other hand, `MATERIALIZED VIEWs` are used when data is to be accessed frequently and data in the table does not get updated frequently. 36 | 37 | > You can add indexes to materialized views, but not plain database views. 38 | 39 | 40 | ### Handling views from Rails 41 | 42 | When using Rails and `ActiveRecord`, views behave like a plain models: 43 | 44 | ```ruby 45 | class Report < ApplicationRecord 46 | belongs_to :property 47 | 48 | def readonly? 49 | true 50 | end 51 | end 52 | 53 | Report.where(property: property) 54 | .group(:date) 55 | .sum(:revenue) 56 | ``` 57 | 58 | Also, for managing (materialized) views in your Rails application, it's recommended to use [Scenic gem](https://github.com/scenic-views/scenic). 59 | The gem adds methods for easier management (create, delete, update, and refresh) of database views. 60 | 61 | 62 | ## PostGIS 63 | 64 | When you want to work with geographical data, most likely you'll end up using [PostGIS](https://postgis.net/). PostGIS is an extension for the PostgreSQL database. 65 | 66 | In the official [documentation](https://postgis.net/docs/), you can find more information about data types, functions, and operators. 67 | Here's the list of some useful functions and geometry constructors, which can be your starting point when you start playing with PostGIS: 68 | 69 | * [`ST_Point`](https://postgis.net/docs/ST_Point.html) 70 | * [`ST_ClosestPoint`](https://postgis.net/docs/ST_ClosestPoint.html) 71 | * [`ST_Contains`](https://postgis.net/docs/ST_Contains.html) 72 | * [`ST_Covers`](https://postgis.net/docs/ST_Covers.html) 73 | * [`ST_Collect`](https://postgis.net/docs/ST_Collect.html) 74 | * [`ST_Centroid`](https://postgis.net/docs/ST_Centroid.html) 75 | * [`ST_DWithin`](https://postgis.net/docs/ST_DWithin.html) 76 | * [`ST_ClusterDBSCAN`](https://postgis.net/docs/ST_ClusterDBSCAN.html) 77 | 78 | PostGIS might be overkill in some scenarios - read more here [why you (probably) don't need PostGIS](https://blog.rebased.pl/2020/04/07/why-you-probably-dont-need-postgis.html). 79 | -------------------------------------------------------------------------------- /Development practices/API.md: -------------------------------------------------------------------------------- 1 | # Intro 2 | 3 | Building an API is different than building a regular HTML application. There are no HTML views to worry about, but instead you should take extra care of response statuses, response format and documentation, since your application will be used by other developers. 4 | 5 | ## Creating an application 6 | 7 | To generate a Rails application with only the elements necessary for an API application, use the `--api` flag, e.g., `rails new my_application --api`. 8 | 9 | Use the `--api` flag only if you're sure that your application will be exclusively an API. If you'll have an admin backend, or use the view layer in some capacity, the Rails API mode will not suffice, and it's best to use the standard `rails new` command. 10 | 11 | We write our APIs using the JSON format, more specifically, we use the [JSON API standard](http://jsonapi.org/). 12 | This allows those who consume our APIs to know what to expect regarding the document structure, and they can use libraries which enable them to build their clients faster and easier. 13 | 14 | ## Example app 15 | 16 | Check out this [example API application](https://github.com/infinum/rails-infinum-jsonapi_example_app) that uses a combination of gems used for simplifying our APIs. 17 | 18 | 19 | ## Gems 20 | 21 | We use a few gems that make our lives easier when writing APIs. 22 | 23 | 24 | ### Controller responses 25 | 26 | [json\_api\_responders](https://github.com/infinum/json_api_responders) simplifies the code in controllers. 27 | 28 | Responding with a status `200` (_OK_) if the object is valid and with `422` (_Unprocessable Entity_) would usually look like this: 29 | 30 | ```Ruby 31 | if resource.valid? 32 | render json: resource, status: 200 33 | else 34 | render error: errors, status: 422 35 | end 36 | ``` 37 | 38 | `json_api_responders`'s method `respond_with` enables you to simplify the code: 39 | 40 | ```Ruby 41 | respond_with resource 42 | ``` 43 | 44 | ### Data serialization/deserialization 45 | 46 | So far we've used the [jsonapi-rails](https://github.com/jsonapi-rb/jsonapi-rails), which takes care of serialization/deserialization of Ruby objects into/out of JSON objects. It's done according to the JSON API standard, so you don't have to think about those details. 47 | 48 | Another alternative is [fast_jsonapi](https://github.com/Netflix/fast_jsonapi). Netflix's gem proved it can do serialization faster. You can check out the [benchmark specs](https://github.com/Netflix/fast_jsonapi/blob/master/spec/lib/object_serializer_performance_spec.rb) if you're interested. 49 | 50 | ### Testing & Documentation 51 | 52 | [rspec](https://github.com/rspec/rspec-rails) 53 | * Testing is always important when building software, but doubly so when writing an API since we can use tests to generate documentation. 54 | 55 | [dox](https://github.com/infinum/dox) 56 | * Generates documentation from the RSpec tests in the API Blueprint format. 57 | 58 | ## Documentation 59 | 60 | You can generate documentation from your controller or request specs. It's very 61 | important that you cover all the possible cases and return codes, e.g., a valid response, 62 | response when there are errors, and any other outcome of the action. Also, document pagination, enumerations, filtering parameters, sorting parameters, and anything else that may be used to refine the results of the API call. 63 | 64 | Take special care to document crucial parts of your API, such as authentication and session management. 65 | 66 | Generating documentation should be a part of the Continuous Integration process so that you can be sure your documentation is always up-to-date. You can use a service like [Apiary](https://apiary.io/) to host the documentation. 67 | 68 | ## Versioning 69 | 70 | APIs must be versioned from the start. That way, in case of large changes, clients can still use the older version of the API while adjusting their application to the new API specification. Otherwise, if the API is not versioned, changes to the API would break the client's applications frequently, and the client would have to take a lot of care to track the changes. 71 | 72 | Namespace different API versions in the URL, e.g., a new version of ```api/v1/resource``` would look like ```api/v2/resource```. Also, make sure that your application code, that is, your controllers, serializers, tests, and documentation, are also namespaced. 73 | 74 | ## Pagination 75 | 76 | It's often possible that the API call provides a huge number of results—it could be hundreds of thousands of records. That's impractical if the client needs only a small part of the results. That's why we can limit the number of results returned with pagination, which allows the client to choose which part of the results they want to see in the request, for example, results 500-600 out of 10000. 77 | 78 | Pagination is required for every index action, and should be described in the documentation with information on which query parameters are used, what the default and max page size is, etc. 79 | 80 | ## HTTP headers and status codes 81 | 82 | If the JSON API standard is used, clients have to set the `content-type` to `application/vnd.api+json` in their request headers. In other cases, when using the JSON format, `application/json` will suffice. 83 | 84 | Some of the more commonly used HTTP status codes are: 85 | #### Success codes 86 | * `200 OK`—request has succeeded 87 | * `201 Created`—new resource has been created 88 | * `204 No Content`—no content needs to be returned (e.g., when deleting a resource) 89 | 90 | #### Client error codes 91 | * `400 Bad Request`—request is malformed in some way (e.g., wrong formatting of JSON) 92 | * `401 Unauthorized`—authentication failed 93 | * `403 Forbidden`—user does not have permissions for the resource 94 | * `404 Not Found`—resource could not be found 95 | * `422 Unprocessable Entity`—resource hasn't be saved (e.g., validations on the resource failed) 96 | 97 | #### Server error codes 98 | * `500 Internal Server Error`—something exploded in your application 99 | 100 | 101 | There are many more HTTP status codes. You can go [here](https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html) to read the whole specification. 102 | -------------------------------------------------------------------------------- /Development practices/Automated tests.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/infinum/rails-handbook/1d7e3845ae57fb6f78f8016a84926226ef520685/Development practices/Automated tests.md -------------------------------------------------------------------------------- /Development practices/Background and scheduled jobs.md: -------------------------------------------------------------------------------- 1 | Our goal is to have applications that process incoming web requests in a timely fashion. 2 | Some requests don't just manipulate data from your database, but also perform time-consuming operations, eg: 3 | 4 | - send emails 5 | - send push notifications 6 | - execute transactions 7 | - generate PDFs 8 | - interact with 3rd party APIs 9 | - perform other long-running tasks, eg. updating database indices (Algolia) 10 | 11 | Performing time-consuming tasks during the request processing negatively impact response time. 12 | 13 | ## Background jobs 14 | The solution is to extract time-consuming tasks and execute them asynchronously. 15 | 16 | A common idiom is to extract time-consuming tasks into units of work called jobs. The main process (the request) can defer (schedule) a job for later execution (in the background). 17 | 18 | There are many Ruby projects dealing with background jobs 19 | * Sidekiq 20 | * DelayedJob 21 | * Resque 22 | * ... 23 | 24 | After trying them all, we decided on Sidekiq. Sidekiq offers the optimal performance, versatility and support for our purposes. 25 | 26 | To incorporate Sidekiq into your project, you'll need the following infrastructure: 27 | 28 | - database to store jobs - [Redis](https://redis.io/) 29 | - processor that executes jobs - [Sidekiq](https://github.com/mperham/sidekiq) 30 | 31 | Instructions on how to configure Sidekiq to work with Redis can be found [here](https://github.com/mperham/sidekiq/wiki/Using-Redis). 32 | After you [setup](https://github.com/mperham/sidekiq/wiki/Getting-Started) a worker (background job), you can hook it up wherever it is required and choose when it's going to be executed. 33 | 34 | ### Error handling 35 | Another great benefit of using background jobs is improving fault tolerance, especially if you have an API that communicates with another API. 36 | 37 | We can never be 100% sure that the dependent API works OK, which could cause errors in the database or 500 HTTP errors. 38 | 39 | Sidekiq background jobs provide [options](https://github.com/mperham/sidekiq/wiki/Advanced-Options#workers) that are defined as class methods which can be used to improve error handling. 40 | 41 | ### Best practices 42 | Some of the best practices have been covered in the official [Sidekiq Wiki pages](https://github.com/mperham/sidekiq/wiki/Best-Practices). 43 | 44 | ### Active Job 45 | Active Job is a framework for declaring jobs and making them run on a variety of [queuing backends](https://guides.rubyonrails.org/active_job_basics.html#starting-the-backend). 46 | 47 | To setup Sidekiq as a queuing library for Active Job read [this Wiki](https://github.com/mperham/sidekiq/wiki/Active-Job). 48 | 49 | ## Scheduled jobs 50 | Sometimes a service needs to be run recurrently and not in an HTTP request lifecycle: 51 | 52 | - every N (seconds/minutes/hours/days) 53 | - on a certain day of every month 54 | - on a certain hour of every day 55 | - etc... 56 | 57 | This is where [sidekiq-scheduler](https://github.com/moove-it/sidekiq-scheduler) comes in handy. 58 | It also provides the `cron` option for specifying when the service will be called if you prefer or require it. 59 | 60 | ## Sidekiq Monitoring 61 | APIs usually require a dashboard interface for admin users. It makes sense to also include a monitoring tool that has a lot of useful info in regards to background jobs. 62 | 63 | You can use [built-in methods](https://github.com/mperham/sidekiq/wiki/Monitoring#devise) in your `config/routes.rb` file to enable this feature. 64 | #### _NOTE:_ 65 | If you have nginx configured to only look up assets in the `current/public` folder, the Monitoring dashboard will load without the `sidekiq/web` assets. To solve this, you have to create a symlink to the Sidekiq assets folder using the `link_sidekiq_assets` mina task: 66 | 67 | ```ruby 68 | task :deploy do 69 | invoke :'git:ensure_pushed' 70 | deploy do 71 | invoke :'git:clone' 72 | invoke :'deploy:link_shared_paths' 73 | invoke :'bundle:install' 74 | invoke :'secrets:pull' 75 | invoke :'rails:db_migrate' 76 | invoke :'rails:assets_precompile' 77 | invoke :'deploy:cleanup' 78 | 79 | on :launch do 80 | invoke :link_sidekiq_assets 81 | invoke :restart_application 82 | invoke :restart_sidekiq 83 | end 84 | end 85 | end 86 | ``` 87 | 88 | In case your Sidekiq monitoring tool is namespaced behind an authenticated route: 89 | 90 | ```ruby 91 | authenticate :administrator do 92 | mount Sidekiq::Web => '/admin/sidekiq' 93 | end 94 | ``` 95 | 96 | then you also need to: 97 | 98 | `set :sidekiq_web_namespace, :admin` 99 | 100 | in your `config/deploy.rb` file. 101 | 102 | 103 | ### Interesting articles: 104 | * [Which Ruby background job framework is right for you?](https://scoutapm.com/blog/which-ruby-background-job-framework-is-right-for-you) 105 | * [Improving Rails Performance with Better Background Jobs](https://rollout.io/blog/improving-rails-performance-better-background-jobs/) 106 | * [How to use sidekiq with devise to send emails without devise-async](https://stackoverflow.com/a/56265427/1339894) 107 | -------------------------------------------------------------------------------- /Development practices/Code style.md: -------------------------------------------------------------------------------- 1 | The Infinum Backend Team writes code following the community guidelines: 2 | 3 | * [Ruby style guide](https://github.com/rubocop-hq/ruby-style-guide) 4 | * [Rails style guide](https://github.com/rubocop-hq/rails-style-guide) 5 | * [Betterspecs](http://www.betterspecs.org/) 6 | 7 | Please read them. 8 | 9 | We use a Ruby style-checking analyzer called [Rubocop](https://github.com/bbatsov/rubocop) to preserve syntax consistency in our applications. It's packaged as a Ruby gem and comes with a CLI to check style consistency. 10 | We run the checks through an [Overcommit](https://github.com/sds/overcommit) hook on almost all our projects (if it's missing on a legacy project, check with your colleagues whether it makes sense to install it). 11 | 12 | We also use it as a plugin in our editors to get real time in-editor warnings just as we're writing code. Whether you're using Vim, Atom or Sublime, please check that it's properly installed after you run our installation scripts. 13 | 14 | Rubocop is highly configurable, and we deviate only slightly from the suggested guidelines. 15 | Our configuration can be found [here](https://github.com/infinum/default_rails_template/blob/master/.rubocop.yml). 16 | Each project must have its own copy of rubocop.yml where project specific configuration can be added. 17 | 18 | 19 | Some exceptions from our config file: 20 | 21 | * We avoid documenting every class or module. It doesn't make sense to write about a User, Post, or a Comment class and all of its methods. However, applications tend to have a Domain-Specific Business Logic that requires a documentation since, without a few comments, it's often not clear what's going on. If that's the case, please explain the interactions in the project README. 22 | If you're adding a *hack* or some method is implemented in a non-standard way, inline comment is **mandatory**. 23 | 24 | * The community guidelines suggest to write arrays with string elements with the [percent notation](https://en.wikibooks.org/wiki/Ruby_Programming/Syntax/Literals#The_.25_Notation): 25 | 26 | ``` 27 | developers = %w(Lucas John Tommy) 28 | ``` 29 | 30 | instead of 31 | 32 | ``` 33 | developers = ['Lucas', 'John', 'Tommy'] 34 | ``` 35 | 36 | We find it OK to write string arrays the second way for two reasons: 37 | 38 | * We never remember to write them with the percent notation in the first place, so we always have to change them afterwards. 39 | * The percent notation looks odd when you have multiple words in one string: 40 | 41 | ``` 42 | developers = %w(Lucas John Tommy Herman\ Zvonimir) 43 | ``` 44 | -------------------------------------------------------------------------------- /Development practices/Design patterns.md: -------------------------------------------------------------------------------- 1 | 2 | ## Classic design patterns 3 | 4 | Design patterns are typical solutions to common problems in software design. 5 | Each pattern is like a customizable blueprint used to solve a particular design problem in your code. 6 | They define a language that efficiently improves team communication. 7 | 8 | They can also be used to improve code readability and cleanness, as well as to simplify the design. 9 | 10 | [Refactoring.guru](https://refactoring.guru) is a great online place to learn more about them. 11 | 12 | ## Rails patterns 13 | 14 | ### Factory 15 | Class that has one creation method with a large conditional. 16 | 17 | #### Problem 18 | You have a method which instantiates a different type of object depending on a particular parameter. 19 | 20 | #### Solution 21 | Read more [here](https://www.sihui.io/design-pattern-factory/). 22 | 23 | ### Form object 24 | Move form-specific logic away from your ActiveRecord models into a separate class. 25 | 26 | #### Problem 27 | * Your controllers are huge and you want to make them more readable by extracting some of their logic, but you don't want to clutter your models 28 | * Your opinion is that models shouldn't have validations 29 | * You have a virtual attribute that works only in a couple of places, but it doesn't make sense to add `attr_accessor` to the model 30 | 31 | #### Solution 32 | Read more [here](https://thoughtbot.com/blog/activemodel-form-objects). 33 | _Note_: Form objects work great with [active_type](https://github.com/makandra/active_type). 34 | 35 | ### Value object 36 | A small simple object, like a date range or location, whose equality isn’t based on identity. 37 | 38 | #### Problem 39 | The problem can be spotted by finding: 40 | 41 | * attributes that don't make sense on their own 42 | * logic that is tightly coupled with primitives (attributes with behaviour - methods) 43 | 44 | #### Solution 45 | Read more [here](https://revs.runtime-revolution.com/value-objects-in-ruby-on-rails-9df64bc8db34). 46 | [dry-rb](https://dry-rb.org/gems/dry-initializer/3.0/) is a Ruby gem that can be used for value objects. 47 | 48 | ### Null object 49 | Instead of returning null, or some odd value, return a Special Case that has the same interface as what the caller expects. 50 | 51 | #### Problem 52 | You have a lot of conditionals scattered around many classes or view files that check the same thing. It's usually a low-level error handling part that covers calls to `nil`s. 53 | 54 | #### Solution 55 | Read more [here](https://thoughtbot.com/blog/handling-associations-on-null-objects). 56 | 57 | ### Query object 58 | Represent a database query or a set of database queries related to the same table. 59 | 60 | #### Problem 61 | * You want to simplify a class that fetches some data from your database using a complex query 62 | * You want to simplify your model scope to use another class, instead of keeping the query in the caller 63 | * You have a complex SQL query that is used in multiple callers 64 | 65 | #### Solution 66 | Read more [here](https://medium.flatstack.com/query-object-in-ruby-on-rails-56ea434365f0). 67 | 68 | ### Service object 69 | Concentrate the core logic of a request operation into a separate object. 70 | 71 | #### Problem 72 | You have a complex set of operations that need to be executed in a sequence synchronously or asynchronously. 73 | 74 | #### Solution 75 | Read more [here](http://brewhouse.io/blog/2014/04/30/gourmet-service-objects.html). 76 | _Note_: Service objects can easily become a code smell if not handled properly. If you find yourself with a service object that has too many methods and/or becomes generally unreadable and hard to understand, you might have to find a [bounded context](https://blog.carbonfive.com/bring-clarity-to-your-monolith-with-bounded-contexts/) or use one of the design patterns mentioned in this chapter. 77 | -------------------------------------------------------------------------------- /Development practices/Emails.md: -------------------------------------------------------------------------------- 1 | ## Guidelines 2 | 3 | ### Templates 4 | In most cases we don't need a custom design for the emails. We use [simple HTML templates](https://github.com/mailgun/transactional-email-templates) and modify them slightly with project's theme colors, and add a logo if needed. 5 | 6 | For implementing a fully custom template we use [MJML](https://mjml.io/) with the [mjml-rails gem](https://github.com/sighmon/mjml-rails). Before setting it up on a new project, check with someone from the JS team whether this still is the best tool for the job. 7 | 8 | If somebody from the other team is expected to work on emails, please make sure you first set up the Letter opener, prepare the previews for all mailers and instruct your colleague on how to use the [mailer previews](#mailer-previews). 9 | 10 | ### Send emails asynchronously 11 | Emails should be sent asynchronously from a background job: 12 | 13 | * so they don't block the current HTTP request longer than needed 14 | * if the email server is temporarily unavailable, the background job will fail and re-try again (check your retry options!). 15 | 16 | See the [Background jobs chapter](/books/rails/development-practices/background-and-scheduled-jobs) on how to configure those. 17 | 18 | If you're using Devise, don't forget to configure it for ActiveJob as described [here](https://github.com/heartcombo/devise#activejob-integration). 19 | 20 | ## Local development 21 | 22 | We usually don't need nor want to send real emails to users in development, so we use [Letter opener](https://github.com/ryanb/letter_opener). Instead of sending an email, it will render the email in a new tab. 23 | 24 | ### Mailer previews 25 | Rails has a built-in feature for [previewing emails](https://guides.rubyonrails.org/action_mailer_basics.html#previewing-emails). Using it allows you to see what the email looks like without sending a real email. Please read the setup instructions on the previous link. 26 | 27 | We typically add mailer preview classes to the `spec/mailers/previews` folder. 28 | 29 | Here's an example of a preview class: 30 | 31 | ```ruby 32 | # Preview all emails at http://localhost:3000/rails/mailers 33 | class UserMailerPreview < ActionMailer::Preview 34 | def storage_limit_warning_email 35 | UserMailer.storage_limit_warning_email(user) 36 | end 37 | 38 | private 39 | 40 | def user 41 | User.new(email: 'rails@is.best', username: 'mockyMock', space_usage: 450) 42 | end 43 | end 44 | ``` 45 | 46 | The important thing is not to load existing objects from your development database here, because you will have problems when you delete or change the data. Other developers working on the project probably won't have the same data in their database. 47 | A better solution would be to just build the necessary objects with all attributes and relationships used in the template. 48 | 49 | 50 | ## Production and staging 51 | 52 | In production/staging/uat we mainly use [Mailgun](https://mailgun.com) to send emails from our apps. 53 | 54 | Someone from the DevOps team should open a Mailgun account and set up everything for production/staging and store credentials in [Vault](/books/rails/development-practices/secrets). 55 | 56 | To set up Mailgun in the app, add the official [mailgun-ruby](https://github.com/mailgun/mailgun-ruby#rails) gem and set it up per environment as described in the gem's Readme. 57 | 58 | Mailgun has [webhooks](https://documentation.mailgun.com/en/latest/api-webhooks.html), which are useful if you need to track failed deliveries caused by non-existing/non-verified emails in the database. We've created the [mailgun_catcher](https://github.com/infinum/rails-infinum-mailgun_catcher) gem to simplify webhook integration in our apps. 59 | 60 | ### Sending emails on behalf of users 61 | Sometimes we need to send an email on behalf of the app's users, i.e. Contact form submissions. **Never set _from_ field to user's email address - use Reply-To.**. Mailgun won't deliver emails with _from_ address with a different domain than the one configured in Mailgun settings. 62 | 63 | ### Intercepting emails 64 | Emails in non-production environments should have prefixed subjects, i.e. `[Staging] Confirmation Instructions`. Creating a custom interceptor is simple, for more info read [here](https://guides.rubyonrails.org/action_mailer_basics.html#intercepting-emails). 65 | 66 | In some apps you should intercept emails in staging to prevent sending them to real users, or to avoid Mailgun failures if you have fake/non-existing emails in the database. 67 | There's the [mail_interceptor](https://github.com/bigbinary/mail_interceptor) gem which intercepts and forwards emails to a given address, and also sends the emails only to whitelisted addresses. 68 | -------------------------------------------------------------------------------- /Development practices/File uploads.md: -------------------------------------------------------------------------------- 1 | Our standard file upload setup includes: 2 | 3 | * using Shrine gem 4 | * direct S3 uploads 5 | * [S3](https://aws.amazon.com/s3/) as file storage for production and staging servers, and disk storage for development. 6 | 7 | ## Shrine 8 | There are several file upload gems out there. We use [Shrine](https://github.com/janko-m/shrine) because of its flexibility and maintainability. 9 | It is a general-purpose file uploader gem. Since it has no direct dependency, 10 | __it can be used in both Rails and non-Rails projects.__ 11 | 12 | Official docs can be found [here](https://shrinerb.com/) 13 | There is also [a demo application](https://github.com/erikdahlstrand/shrine-rails-example). 14 | 15 | ### Plugins 16 | Shrine features a rich plugin system which makes it incredibly easy to change its behavior to suit your needs. 17 | 18 | Here are some of the plugins we use often: 19 | 20 | * [activerecord](https://shrinerb.com/docs/plugins/activerecord) 21 | * [bakgrounding](https://shrinerb.com/docs/plugins/backgrounding) 22 | * [cached\_attachment\_data](https://shrinerb.com/docs/plugins/cached_attachment_data) - for forms 23 | * [derivatives](https://shrinerb.com/docs/plugins/derivatives) 24 | * [determine\_mime\_type](https://shrinerb.com/docs/plugins/determine_mime_type) 25 | * [presign_endpoint](https://shrinerb.com/docs/plugins/presign_endpoint) - for direct S3 uploads 26 | * [upload_options](https://shrinerb.com/docs/plugins/upload_options) 27 | * [url_options](https://shrinerb.com/docs/plugins/url_options) 28 | * [validation](https://shrinerb.com/docs/validation) 29 | * [validation_helpers](https://shrinerb.com/docs/plugins/validation_helpers) 30 | 31 | ### Security 32 | Files stored on S3 are private by default. This means file URLs will be signed and they will expire after some specified time. 33 | 34 | It's best to explicitly set the expiration time for **each uploader class** using [url_options](https://shrinerb.com/docs/plugins/url_options) plugin. 35 | 36 | Some files need to be public, i.e. albums' covers. In that case, set the `acl` to `public-read` via [upload_options](https://shrinerb.com/docs/plugins/upload_options) plugin for that uploader class. 37 | 38 | ### Other guidelines 39 | 40 | - Use **jsonb** data type for file columns when possible. 41 | 42 | - If you're uploading images, you should process them (file compression) and create **multiple versions** (derivatives in Shrine) of different sizes. Consult with frontend devs on required dimensions. 43 | **Backgrounding** plugin should be combined with the processing for a better user experience. 44 | 45 | - Always validate **mime type** for uploaded files, and extension if needed. 46 | 47 | - Shrine doesn't automatically delete files from cache storage when moving them to store storage. Tell a DevOps to **set a lifecycle policy** with an appropriate amount of time **for cache storage** prefix. 48 | 49 | ## Direct S3 upload 50 | Mobile or web front ends often upload files through the app server, which means that the file does a double hop: from the frontend to the backend, then from the backend to the cloud storage service. 51 | 52 | Direct upload solves this double-hop performance problem by giving one-time credentials to the frontend app to upload files directly to the cloud, and it sends out references to those files to the backend. 53 | 54 | This is extremely useful if you want to speed up the uploading process and improve user experience. 55 | 56 | To implement direct S3 upload with Shrine, read the instructions [here](https://shrinerb.com/docs/direct-s3). 57 | 58 | ## Accessing files on S3 59 | To access files on s3, ask someone from the DevOps team for the S3 console access. 60 | -------------------------------------------------------------------------------- /Development practices/Folder Structure.md: -------------------------------------------------------------------------------- 1 | > The first concern of the architect is to make sure that the house is usable; it is not to ensure that the house is made of brick. 2 | > — Uncle Bob 3 | 4 | # What it looks like now 5 | Ruby on Rails is an opinionated framework and forces developers into a specific folder structure. 6 | Most gems also follow the Rails guidelines of how the folder structure should look like. 7 | On smaller projects this can be fine, but on bigger monoliths the files can get out of hand and it becomes too difficult to find the right class. 8 | 9 | Here is a typical Rails folder structure: 10 | 11 | ``` 12 | app 13 | ├── assets # sprockets 14 | ├── channels # active_cable 15 | ├── components # view_component 16 | ├── controllers # Rails 17 | ├── decorators # draper 18 | ├── enumerations # enumerations 19 | ├── forms # reform/active_type 20 | ├── helpers # Rails 21 | ├── javascript # webpacker 22 | ├── jobs # active_job/sidekiq 23 | ├── mailers # Rails 24 | ├── models # Rails 25 | ├── policies # pundit/active_policy 26 | ├── serializers # AMS/jsonapi-serializer 27 | ├── services # PORO 28 | ├── uploaders # shrine 29 | └── views # Rails 30 | ``` 31 | 32 | This type of folder structure usually encourages a flat hierarchy of classes, which, again, is fine for a smaller projects but 33 | falls short when the application grows. 34 | It is said that you can discern the application's purpose just by looking at files in your `services` folder. 35 | 36 | Here is an example of a project services folder. Can you tell me what this application does? 37 | 38 |
39 | Services 40 | 41 | 
 42 | 
 43 | app/services
 44 | ├── add_points_to_user.rb
 45 | ├── aes_crypt.rb
 46 | ├── assign_geo_keywords.rb
 47 | ├── assign_location_attributes.rb
 48 | ├── braintree
 49 | │   ├── customer_and_token_creator.rb
 50 | │   └── webhook_handler.rb
 51 | ├── calculate_bearing.rb
 52 | ├── developers
 53 | │   └── api
 54 | │       └── v1
 55 | │           └── process_occupancy_params.rb
 56 | ├── devise
 57 | │   └── strategies
 58 | │       └── jwt.rb
 59 | ├── export_service
 60 | │   ├── config.rb
 61 | │   ├── generator.rb
 62 | │   ├── model.rb
 63 | │   └── options_formatter.rb
 64 | ├── exporter
 65 | │   ├── creator.rb
 66 | │   └── generator.rb
 67 | ├── exporters
 68 | │   └── csv
 69 | │       ├── base.rb
 70 | │       └── generator.rb
 71 | ├── feature.rb
 72 | ├── filters
 73 | │   ├── creator.rb
 74 | │   └── updater.rb
 75 | ├── generate_friends_report.rb
 76 | ├── indigo
 77 | │   ├── cancel_reservation.rb
 78 | │   └── create_reservation.rb
 79 | ├── jwt_serializer.rb
 80 | ├── locations
 81 | │   └── actions.rb
 82 | ├── loyalty_points_awarder.rb
 83 | ├── mailgun_notify_email_forwarder.rb
 84 | ├── merge_parking_places.rb
 85 | ├── migrations
 86 | │   └── file_mover.rb
 87 | ├── notifications
 88 | │   ├── amazon_notifications.rb
 89 | │   ├── firebase_notifications.rb
 90 | │   ├── sender.rb
 91 | │   └── shared_actions.rb
 92 | ├── parking_place_awarder.rb
 93 | ├── parking_place_exporter.rb
 94 | ├── parking_places
 95 | │   ├── category_updater.rb
 96 | │   ├── country_code_mapper.rb
 97 | │   ├── creator.rb
 98 | │   ├── exporter.rb
 99 | │   ├── generators
100 | │   │   ├── icons
101 | │   │   │   ├── combiner.rb
102 | │   │   │   └── type.rb
103 | │   │   └── icons.rb
104 | │   ├── icon_types.rb
105 | │   ├── icons
106 | │   │   └── updater.rb
107 | │   ├── icons.rb
108 | │   ├── occupancies
109 | │   │   └── setter.rb
110 | │   ├── security_rating.rb
111 | │   └── updater.rb
112 | ├── payment_exporter.rb
113 | ├── payments
114 | │   └── exporter.rb
115 | ├── promiles
116 | │   ├── add_driving_distance.rb
117 | │   ├── build_polyline.rb
118 | │   ├── corridor_search.rb
119 | │   ├── create_or_update_parking_places.rb
120 | │   ├── create_or_update_promiles_locations.rb
121 | │   └── get_route.rb
122 | ├── provider_occupancies_service.rb
123 | ├── providers
124 | │   ├── client
125 | │   │   └── mdm.rb
126 | │   └── parser
127 | │       ├── a63_france.rb
128 | │       └── mdm.rb
129 | ├── request_logs
130 | │   ├── archive_partition.rb
131 | │   ├── create_archive.rb
132 | │   └── create_partition.rb
133 | ├── reservations
134 | │   ├── booking_id_generator.rb
135 | │   ├── charger.rb
136 | │   ├── creator.rb
137 | │   ├── emails.rb
138 | │   ├── integrations_caller.rb
139 | │   ├── mails_sender.rb
140 | │   ├── notifications_sender.rb
141 | │   ├── settler.rb
142 | │   ├── status_setter.rb
143 | │   ├── updater.rb
144 | │   ├── users_setter.rb
145 | │   └── voider.rb
146 | ├── shared
147 | │   └── filter_suggested.rb
148 | ├── stats_maker.rb
149 | ├── subscriptions
150 | │   └── actions.rb
151 | ├── tln
152 | │   └── number_checker.rb
153 | ├── translators
154 | │   └── google.rb
155 | ├── units.rb
156 | ├── update_user_to_developer.rb
157 | ├── user_loyalty_points.rb
158 | ├── users
159 | │   ├── actions.rb
160 | │   ├── creator.rb
161 | │   ├── destroyer.rb
162 | │   ├── exporter.rb
163 | │   ├── o_auth_creator.rb
164 | │   ├── retrieve_all_data
165 | │   │   ├── base.rb
166 | │   │   ├── favorites.rb
167 | │   │   ├── friends_list.rb
168 | │   │   ├── notifications.rb
169 | │   │   ├── personal_information.rb
170 | │   │   └── reviews.rb
171 | │   ├── retrieve_all_data.rb
172 | │   └── settings.rb
173 | ├── vat_checker.rb
174 | ├── version_converter.rb
175 | └── x_server
176 |     ├── add_detour_distance.rb
177 |     ├── add_driving_distance.rb
178 |     ├── base.rb
179 |     ├── build_polyline.rb
180 |     ├── corridor_search.rb
181 |     ├── get_detour_distance.rb
182 |     ├── get_driving_distance.rb
183 |     ├── get_geo_keywords.rb
184 |     ├── get_location_attributes.rb
185 |     ├── get_route.rb
186 |     └── leave_reachable.rb
187 | 
188 |
189 |
190 | 191 | At some point this folder became a dumpall folder without rhyme or reason. 192 | There is also an issue of boundaries: there are none. Each class is used from anywhere in the application and that can 193 | cause headaches. 194 | 195 | # Desired structure 196 | 197 | Here is a folder structure we should strive for 198 | 199 | ``` 200 | app 201 | ├── controllers # Rails 202 | ├── frontend # webpacker 203 | ├── domains # custom 204 | ├── models # Rails 205 | ├── views # Rails 206 | └── web # custom 207 | ``` 208 | 209 | Let's first talk about Rails-specific folders (`controllers`, `views` and `models`). 210 | 211 | The `views` folder needs to exist because it is hardcoded in the Rails itself (https://github.com/rails/rails/blob/main/railties/lib/rails/engine.rb#L610) 212 | 213 | The `controllers` folder is there because of the way Rails finds controller names (it appends `"Controller"` to the resource name) (https://github.com/rails/rails/blob/main/actionpack/lib/action_dispatch/http/request.rb#L88) 214 | 215 | `models` is a folder where we put all our ActiveRecord models. It is technically not necessary but we will talk about it in the [Further Improvements](#further-improvements) chapter. 216 | 217 | 218 | # `domains` vs `web` folder 219 | `web` is for classes used in controller actions: forms, policies, serializers, queries and similar. 220 | `domains` is for business logic. 221 | 222 | ## `web` folder 223 | The `web` folder structure should follow your routes. Having these routes: 224 | 225 | ```ruby 226 | namespace: :dasboard do 227 | resources :projects 228 | end 229 | namespace :api do 230 | namespace :v1 do 231 | resources :projects 232 | end 233 | end 234 | ``` 235 | 236 | it would have this folder structure: 237 | 238 | ``` 239 | web 240 | ├── dashboard 241 | │ ├── projects 242 | │ │ ├── presenter.rb 243 | │ │ ├── decorator.rb 244 | │ │ ├── form.rb 245 | │ │ └── policy.rb 246 | ├── api 247 | │ ├── v1 248 | │ │ ├── v1 249 | │ │ │ ├── projects 250 | │ │ │ │ ├── policy.rb 251 | │ │ │ │ ├── query.rb 252 | │ │ │ │ └── serializer.rb 253 | ├── mailers 254 | │ ├── projects 255 | │ │ ├── mailer.rb 256 | │ │ └── decorator.rb 257 | ``` 258 | 259 | ## `domains` folder 260 | The `domains` folder should be organized into domains. 261 | 262 | The main idea is to create cohesive groups of domain objects that are involved in the same business use case(s). The same way we try to build classes that are loosely coupled but highly cohesive we should build our domains as well. 263 | Here is an example of domains folder: 264 | 265 | ``` 266 | domains 267 | ├── base 268 | │ ├── decorator.rb 269 | │ ├── mailer.rb 270 | │ ├── policy.rb 271 | │ ├── presenter.rb 272 | │ ├── query.rb 273 | │ └── serializer.rb 274 | ├── exports 275 | │ ├── formatters 276 | │ ├── csvs 277 | │ └── spreadsheets 278 | ├── external 279 | │ └── zapier.rb 280 | ├── projects 281 | │ ├── counter.rb 282 | │ └── presister.rb 283 | └── utils 284 | ├── cache.rb 285 | └── slug.rb 286 | ``` 287 | 288 | ## Further improvements 289 | * Move mailers into domains folder, where they are actually used. There might be an issue of template files discovery. 290 | * Move controllers to web folder as well 291 | * Move models to domains and web folders (build DTOs) 292 | * Move views to web folder 293 | * Move frontend to web folder 294 | -------------------------------------------------------------------------------- /Development practices/__Other important guidelines.md: -------------------------------------------------------------------------------- 1 | ## Do! 2 | * [Check the return value of **#save**, otherwise use **#save!**](http://rails-bestpractices.com/posts/2012/11/02/check-the-return-value-of-save-otherwise-use-save) The same goes for #create, #update and #destroy. 3 | * Mirror model validations to database constraints and defaults as much as possible. 4 | * Validate the associated `belongs_to` object (`user`), not the database column (`user_id`). 5 | * All non-actions controller methods should be **private**. 6 | * Use the user's name in the `From` header and email in the `Reply-To` when [**delivering an email on behalf of the app's users**]( http://robots.thoughtbot.com/post/3215611590/recipe-delivering-email-on-behalf-of-users) 7 | 8 | 9 | ## Don't! 10 | * Never use `has_and_belongs_to_many`—use `has_many :through` instead. The first one has unexpected hidden behaviors and, if you find out that you need an extra column in the intermediate table, converting it to the second macro requires a fair amount of work. 11 | * [**Never rescue the Exception class**](http://rails-bestpractices.com/posts/2012/11/01/don-t-rescue-exception-rescue-standarderror/) 12 | * Don't change a migration after it's been merged into master if the desired change can be solved with another migration. 13 | * Don't reference a model class directly from a view. 14 | * Don't use instance variables in partials. Pass local variables to partials from view templates. 15 | * Don't use before_actions for setting instance variables. Use them only for changing the application flow, such as redirecting if a user is not authenticated. 16 | * Avoid the `:except` option in routes. Use the `:only` option to explicitly state exposed routes. 17 | 18 | 19 | ## Naming 20 | * Name date columns with `_on` suffixes. 21 | * Name datetime columns with `_at` suffixes. 22 | * Name time columns (referring to a time of day with no date) with `_time` suffixes. 23 | * Name initializers for their gem name. 24 | 25 | 26 | ## Organization 27 | * Order ActiveRecord associations above ActiveRecord validations. 28 | * Order i18n translations alphabetically by key name. 29 | -------------------------------------------------------------------------------- /Fiscalizer.md: -------------------------------------------------------------------------------- 1 | ## Fiscalizer 2 | 3 | We use the [fiscalizer gem](https://github.com/infinum/fiscalizer) to transfer various document types from a business entity to the tax authorities. 4 | 5 | ### Setup & Usage 6 | 7 | Everything about setup and usage of this gem is explained at [fiscalizer gem](https://github.com/infinum/fiscalizer). 8 | 9 | ### Certificates 10 | 11 | #### Environments: 12 | 13 | + ##### Production environment 14 | 15 | Requires only one certificate - application certificate. It is most often called `FISCAL_1.p12`. It contains the public and private key, and the CA (`Certificate authority`) certificate. 16 | 17 | Secrets file example: 18 | 19 | ```Ruby 20 | fiscalization_app_cert_path: 'cert/production/FISKAL_1.p12' 21 | fiscalization_ca_cert_path: '' 22 | ``` 23 | 24 | 25 | + ##### Development & staging environment 26 | 27 | Requires 2 certificates - both application and server certificate. The application certificate (`FISCAL_1.p12`) contains the public and private key. The server certificate (most often `fina_ca.pem`) contains the required CA certificates. 28 | 29 | Secrets file example: 30 | 31 | ```Ruby 32 | fiscalization_app_cert_path: 'cert/development/FISKAL_1.p12' 33 | fiscalization_ca_cert_path: 'cert/development/fina_ca.pem' 34 | ``` 35 | 36 | #### Application certificate unpacking 37 | 38 | To view the content of the application certificate locally, you need to unpack it with this command: 39 | 40 | ``` 41 | openssl pkcs12 -in FISKAL_1.p12 -out FISKAL_1.pem -nodes 42 | ``` 43 | 44 | After running this command, you are prompted to input the `fiscalization password`. 45 | 46 | #### Create your own certificate 47 | 48 | If you need to test some features that require certificates but you don't have them and your request cannot be sent, you can create a certificate following this [tutorial](https://datacenteroverlords.com/2012/03/01/creating-your-own-ssl-certificate-authority/). 49 | Once done, create a `.p12` file following these two steps: 50 | 51 | 1. Copy the private key and SSL certificate to a plain text file. The private key should be at the top with the SSL certificate below. In the example, we use "filename.txt" as the name of the file containing the private key and SSL certificate. 52 | 53 | 2. `openssl pkcs12 -export -in filename.txt -out filename.p12` 54 | -------------------------------------------------------------------------------- /Hosting API Documentation.md: -------------------------------------------------------------------------------- 1 | # Summary 2 | 3 | This is the flow we use to generate and host our documentation: 4 | 5 | - Write tests using rspec and annotate them using [dox](https://github.com/infinum/dox) 6 | - Once tests on [semaphore](https://semaphoreci.com) pass we move to deploy 7 | - In the deploy phase we do the following: 8 | * deploy the application using [mina](https://github.com/mina-deploy/mina) 9 | * run only dox tagged specs (request specs) to generate the documentation 10 | * copy the generated documentation to the application server inside `public` folder 11 | 12 | # Why 13 | 14 | Hosting the API documentation directly from the application server is convenient and does not cost extra. 15 | Solutions like apiary.io require having one project for each environment, can't easily set 16 | a desired domain and can't easily hide behind some kind of authentication (like Basic Authentication). 17 | 18 | To ensure that the documentation is always up to date it is copied only after a successful deploy 19 | 20 | By using the `public` folder nginx takes care of serving the static file and the requests don't go through the whole Rails stack. 21 | 22 | # How 23 | 24 | ## Dox generating task 25 | 26 | Here is an advanced example of a rake task that generates the documentation: 27 | It has the feature to write ERB code inside your accompanying .md files. 28 | 29 | ```ruby 30 | namespace :dox do 31 | task :md, [:version, :docs_path, :host] do |_, args| 32 | require 'rspec/core/rake_task' 33 | 34 | version = args[:version] || :v1 35 | docs_path = args[:docs_path] || "api/#{version}/docs" 36 | 37 | RSpec::Core::RakeTask.new(:api_spec) do |t| 38 | t.pattern = "spec/requests/#{version}/" 39 | t.rspec_opts = 40 | "-f Dox::Formatter --order defined -t dox -o public/#{docs_path}/apispec.md.erb" 41 | end 42 | 43 | Rake::Task['api_spec'].invoke 44 | end 45 | 46 | task :html, [:version, :docs_path, :host] => :erb do |_, args| 47 | version = args[:version] || :v1 48 | docs_path = args[:docs_path] || "api/#{version}/docs" 49 | `node_modules/.bin/aglio -i public/#{docs_path}/apispec.md -o public/#{docs_path}/index.html --theme-full-width --theme-variables flatly` 50 | end 51 | 52 | task :open, [:version, :docs_path, :host] => :html do |_, args| 53 | version = args[:version] || :v1 54 | docs_path = args[:docs_path] || "api/#{version}/docs" 55 | 56 | `open public/#{docs_path}/index.html` 57 | end 58 | 59 | task :erb, [:version, :docs_path, :host] => [:environment, :md] do |_, args| 60 | require 'erb' 61 | 62 | host = args[:host] || 'http://localhost:3000' 63 | version = args[:version] || :v1 64 | docs_path = args[:docs_path] || "api/#{version}/docs" 65 | 66 | File.open("public/#{docs_path}/apispec.md", 'w') do |f| 67 | f.write(ERB.new(File.read("public/#{docs_path}/apispec.md.erb")).result(binding)) 68 | end 69 | end 70 | end 71 | ``` 72 | 73 | ```markdown 74 | FORMAT: 1A 75 | HOST: <%= host %> 76 | 77 | 78 | # My app API 79 | An API documentation for my app 80 | ``` 81 | 82 | ```sh 83 | rake 'api:doc:html[v1, api/v1/docs, https://example.com]' 84 | ``` 85 | 86 | The above example can be simplified if some of the features are not needed. 87 | 88 | ## Deploy script 89 | 90 | The mina-dox plugin includes the `dox:publish` task, which runs `rake dox:html` and copies the generated documentation to the server. 91 | 92 | ```sh 93 | echo "=========== bundle install ===========" 94 | bundle install --deployment --path vendor/bundle 95 | echo "=========== deploy ===========" 96 | bundle exec mina production ssh_keyscan_domain deploy 97 | 98 | echo "=========== setup for tests ===========" 99 | bundle exec secrets pull -e development -y 100 | bundle exec rake db:setup 101 | bundle exec rake db:test:prepare 102 | 103 | echo "=========== install aglio and themes ===========" 104 | yarn install 105 | 106 | echo "=========== publish api doc ===========" 107 | bundle exec mina production dox:publish 108 | ``` 109 | -------------------------------------------------------------------------------- /Misc/Fiscalization.md: -------------------------------------------------------------------------------- 1 | ## Fiscalizer 2 | 3 | We use the [fiscalizer gem](https://github.com/infinum/fiscalizer) to transfer various document types from a business entity to the tax authorities. 4 | 5 | ### Setup & Usage 6 | 7 | Everything about setup and usage of this gem is explained at [fiscalizer gem](https://github.com/infinum/fiscalizer). 8 | 9 | ### Certificates 10 | 11 | #### Environments: 12 | 13 | + ##### Production environment 14 | 15 | Requires only one certificate - application certificate. It is most often called `FISCAL_1.p12`. It contains the public and private key, and the CA (`Certificate authority`) certificate. 16 | 17 | Secrets file example: 18 | 19 | ```Ruby 20 | fiscalization_app_cert_path: 'cert/production/FISKAL_1.p12' 21 | fiscalization_ca_cert_path: '' 22 | ``` 23 | 24 | 25 | + ##### Development & staging environment 26 | 27 | Requires 2 certificates - both application and server certificate. The application certificate (`FISCAL_1.p12`) contains the public and private key. The server certificate (most often `fina_ca.pem`) contains the required CA certificates. 28 | 29 | Secrets file example: 30 | 31 | ```Ruby 32 | fiscalization_app_cert_path: 'cert/development/FISKAL_1.p12' 33 | fiscalization_ca_cert_path: 'cert/development/fina_ca.pem' 34 | ``` 35 | 36 | #### Application certificate unpacking 37 | 38 | To view the content of the application certificate locally, you need to unpack it with this command: 39 | 40 | ``` 41 | openssl pkcs12 -in FISKAL_1.p12 -out FISKAL_1.pem -nodes 42 | ``` 43 | 44 | After running this command, you are prompted to input the `fiscalization password`. 45 | 46 | #### Create your own certificate 47 | 48 | If you need to test some features that require certificates but you don't have them and your request cannot be sent, you can create a certificate following this [tutorial](https://datacenteroverlords.com/2012/03/01/creating-your-own-ssl-certificate-authority/). 49 | Once done, create a `.p12` file following these two steps: 50 | 51 | 1. Copy the private key and SSL certificate to a plain text file. The private key should be at the top with the SSL certificate below. In the example, we use "filename.txt" as the name of the file containing the private key and SSL certificate. 52 | 53 | 2. `openssl pkcs12 -export -in filename.txt -out filename.p12` 54 | -------------------------------------------------------------------------------- /Starting a new project.md: -------------------------------------------------------------------------------- 1 | ## Starting a new project 2 | 3 | When you're starting a new project, there are a few things you need to do: 4 | 5 | #### 1. Open a repository 6 | 7 | Ask your team lead or DevOps to open a repository. 8 | 9 | #### 2. Fill out the project setup sheet 10 | 11 | We've come up with a project setup template in cooperation with the DevOps team. Its purpose is to provide the DevOps with all the details needed for deploying your application to staging and production servers. 12 | 13 | It contains general info about your project that your project manager can fill out, as well as more technical information like frameworks and technologies used, the build and deploy setup, third party services that need to be set up, etc. 14 | 15 | Ask your PM for a link to the sheet template. 16 | 17 | **Fill in the blanks on backend-related stuff.** 18 | 19 | If you need any other services that aren't mentioned, **provide as much information as possible** 20 | 21 | **The project setup sheet needs to be up to date** with new changes to the project setup and services used. If you're adding some new dependency or service or changing something, please update it in the project setup sheet and notify DevOps to make these changes. 22 | 23 | 24 | #### 3. Generate a new application 25 | 26 | Use the [**default rails template**](https://github.com/infinum/default_rails_template/) to start your new application. 27 | 28 | Most of our work is based on the Rails framework. We have some standard configurations and gems that are basically in every Rails application. To make our lives easier, we've made a default template to generate new Rails applications that meet those standards. 29 | 30 | Please always update the default rails template if something gets broken or out of date when you're starting a new project. 31 | 32 | 33 | #### 4. Check for new/better gems 34 | 35 | We usually use specific gems for some purposes, e.g. Devise for authentication, Pundit for authorization, etc. That doesn't mean we need to use those gems necessarily. 36 | 37 | When adding a gem to your application, check if there's some new and better gem for that purpose. 38 | 39 | When checking out a gem, always look for these things: 40 | 41 | * is the gem maintained? when was the last commit? 42 | * how many stars does it have? 43 | * what is the ratio between open and closed issues? 44 | 45 | 46 | #### 5. Fill out the readme file 47 | 48 | After setting up a project, fill out the readme file with all the necessary details about the application: 49 | 50 | * dependencies & prerequisites 51 | * setup instructions 52 | * development instructions 53 | * deployment instructions 54 | * test instructions 55 | * any other project specific information that someone who takes over a project needs to know 56 | 57 | 58 | #### 6. Create an ER diagram 59 | 60 | Before you start coding, you always need to create an ER diagram for your database. Go through the project specification and list all the entities and their attributes and construct relationships between listed entities. 61 | 62 | For creating an ER diagram use: 63 | 64 | * [DB diagram](https://dbdiagram.io/) 65 | * [MySQL Workbench](https://dev.mysql.com/downloads/workbench/) 66 | 67 | After creating an ER diagram, you need to go through an architecture review. You can read more about it in the [Architecture review chapter](Architecture review) 68 | 69 | If you need to create an application or a server architecture diagram use: 70 | 71 | * [Gliffy](https://www.gliffy.com/) 72 | 73 | If you need to create a sequence diagram use: 74 | 75 | * [Web Sequence Diagram](https://www.websequencediagrams.com/) 76 | -------------------------------------------------------------------------------- /Workflows/Architecture review.md: -------------------------------------------------------------------------------- 1 | ## The reason behind it 2 | 3 | Before you start programming, your application should go through an architecture review. 4 | When we say architecture review, we mostly have the database design and API in mind. 5 | 6 | We do architecture planning and review in order to: 7 | 8 | * have a clear overview of the project before we start coding 9 | * have a clear way of communicating the project architecture to the client 10 | * prevent scope creep 11 | * have better architecture 12 | 13 | **Database design is by far the most important part of a web application.** 14 | 15 | ## Architecture review timeline 16 | 17 | Please follow this timeline: 18 | 19 | 1. A project manager asks you to go into an initial project meeting. 20 | 2. You should never go alone as the only Rails developer in the meeting—always take another team member with you. 21 | 3. There are no stupid questions in these meetings. It's better to understand everything at this point, than make assumptions two weeks later. 22 | 4. Ask the project manager to give you the project specification. 23 | 5. Once you're done with the initial project meeting and have the specification, sit down with the member of your team that was at the meeting with you and design the database. You can do this on a whiteboard or a piece of paper, or use a tool, such as [Gliffy](https://www.gliffy.com/) or [MySQL Workbench](https://dev.mysql.com/downloads/workbench/). If you need to draw a sequence diagram, use [Web Sequence Diagram](https://www.websequencediagrams.com/). 24 | 6. Consult the project manager at will and add new discoveries to the specification if you find that something's missing. 25 | 7. Don't hesitate to involve other team members since this is the most important part of the app. 26 | 8. Once you've finished the database design, it has to be approved by someone from the management. 27 | 9. Once your architecture has been approved, you can generate a new Rails project. Read more about that on the next page. 28 | 29 | In order to start coding, the whole project must go through an architecture review. 30 | If this is not possible for any reason, the project should be split into phases. In that case, each phase needs to go through an architecture review before coding can begin. 31 | 32 | Sometimes, you will notice some obvious mistakes you couldn't have seen when you started the project. If these make a large architecture overturn, go back to step 8. 33 | 34 | **Think of this as a never bending rule.** 35 | -------------------------------------------------------------------------------- /Workflows/Dependencies.md: -------------------------------------------------------------------------------- 1 | > Every dependency in your application has the potential to bloat your app, to destabilize your app, to inject odd behaviour via monkeypatching or buggy native code. 2 | > 3 | > When you are considering adding a dependency to your Rails app, it’s a good idea to do a quick sanity check, in order of preference: 4 | > 5 | > Do I really need this at all? Kill it. 6 | > 7 | > Can I implement the required minimal functionality myself? Own it. 8 | > 9 | > If you need a gem: 10 | > 11 | > Does the gem have a native extension? Look for pure ruby alternatives. 12 | > 13 | > Does the gem transitively pull in a lot of other gems? Look for simpler alternatives. 14 | > 15 | > Gems with native extensions can destabilize your system; they can be the source of mysterious bugs and crashes. Avoid gems which pull in more dependencies than their value warrants. 16 | > 17 | > -- [Kill your dependencies](https://www.mikeperham.com/2016/02/09/kill-your-dependencies/) 18 | 19 | ### Why should you bother with updates? 20 | Convincing clients that we need to spend time regularly updating dependencies is sometimes challenging. The application code already works and it doesn't need changing to keep functioning. Properly updating a dependency takes not only development time (research and execution) but also warrants testing of affected features on all environments. From the client's naive point of view there's no added benefit. It is our job to communicate the following points clearly and take care of overall project health. 21 | 22 | - All dependencies have a certain **support life cycle**. If you're using an older version and a new crucial security update is announced you may be forced to do a rapid upgrade of multiple versions to secure your application. Doing that is risky, especially in a time crunch. 23 | - **Regular smaller dependency upgrades** are more predictable, safer and easier to manage in the context of a sprint. Failing to update dependencies regularly can turn the upgrade process from several bite size chunks which can be done independently to a massive undertaking that takes days and blocks all development. Estimating these big upgrades is always hard and we usually underestimate it by a factor of 3 or 4. 24 | - Dependency upgrades usually introduce **new features, bug fixes and performance enchancements**. With widely used dependencies you're usually not the first person to run into a problem or see room for improvement. In almost all cases somebody has already reported the problem you're having, notified the maintainer and if you're lucky even fixed the issue. The maintainers and community are doing the hard work for free, all you need to do is perform the upgrade and reap the benefits. 25 | 26 | ## Identifying outdated dependencies 27 | The Ruby ecosystem is quite large and active, so being up to date on all your dependency changes can be a challenge. Manual options like checking the [Ruby security mailing lists](https://groups.google.com/g/ruby-security-ann) and Github repositiories have their place, but they require developers to put in extra time and effort. Luckily the community has developed some great automated tools to help you ease the load. 28 | ### Bundle outdated 29 | Bundler [already includes](https://bundler.io/man/bundle-outdated.1.html) a handy tool which can be used to get a list of outdated gems in your project. 30 | 31 | ``` 32 | Outdated gems included in the bundle: 33 | * aws-partitions (newest 1.367.0, installed 1.338.0) 34 | * blueprinter (newest 0.25.1, installed 0.25.0) in groups "default" 35 | * brakeman (newest 4.9.1, installed 4.9.0) in groups "development" 36 | ``` 37 | 38 | You should have a periodic reminder in your calendar to run `bundle outdated` and triage the result. You can't always be on the edge, but you should at least be aware how far behind you are. This will help you estimate the time needed to update your dependencies and fit them in your schedule. 39 | ### [Bundler-audit](https://github.com/rubysec/bundler-audit) 40 | Running `bundle-audit` give you a list of all the dependencies in your project that contain publicly known security vulnerabilities. We mandate this check to be executed before each commit (using [overcommit](https://github.com/sds/overcommit) or [lefthook](https://github.com/Arkweid/lefthook)). This gives developers a clear message - 41 | > Triage and deal with the problem immediately, then continue your work. 42 | 43 | Once a vulnerability is made public and added to the [Ruby Advisory Database](https://github.com/rubysec/ruby-advisory-db) you don't have much time to spare. By the time that you get the warning the vulnerability is probably already being exploited in the wild. Arrange with your project manager to get this sorted **immediately**. 44 | 45 | ### [Dependabot](https://docs.github.com/en/github/administering-a-repository/about-github-dependabot-version-updates) 46 | Can you imagine having a co-worker who would check your dependencies for updates, summarise the changelogs and open a PR for each new one each week? You're in luck, Dependabot does exactly that. 47 | 48 | To trigger Dependabot you will have to add a file named `dependabot.yml` to the `.github` directory, located in the root of the project. The configuration should loosely follow the following format 49 | 50 | ``` 51 | version: 2 52 | updates: 53 | - package-ecosystem: "bundler" 54 | directory: "/" 55 | schedule: 56 | interval: "weekly" 57 | open-pull-requests-limit: 2 58 | 59 | - package-ecosystem: "npm" 60 | directory: "/" 61 | schedule: 62 | interval: "weekly" 63 | open-pull-requests-limit: 2 64 | ``` 65 | 66 | We recommend that you adjust the limit of weekly pull requests based on the circumstance of your project. Dependencies get updated rapidly, and you won't always have the time sift through all the updates in the same week. 67 | 68 | ## Dependency upgrades 69 | 70 | Dependency upgrades should be treated no differently than a feature request. They should be planned, estimated, implemented, reviewed, deployed to staging, checked by the QA team, and then finally deployed to production and monitored. 71 | 72 | You will seldom understand all the inner workings of your downstream dependencies, so you should in fact take even more care than usual when upgrading dependencies. 73 | 74 | The key to reducing dependency upgrade risk is making small steps. You need to understand what's being changed and how it's going to affect your project. 75 | > It may take a bit more planning to pull off, but by finding a way to avoid disrupting the rest of the business, housekeeping activities like keeping your dependencies up-to-date can be done over the course of everyday development—hopefully helping the team avoid ever again falling several years behind on a major dependency like Rails. 76 | > 77 | > -- [3 keys to upgrading Rails](https://blog.testdouble.com/posts/2019-09-03-3-keys-to-upgrading-rails/) 78 | 79 | The following steps can serve as a guide to help you on your path: 80 | 81 | 1. Check the dependency policy regarding versioning. Is the dependency using [Semantic versioning](https://semver.org)? What version number changes include breaking API changes? 82 | 2. Survey the changelog for breaking changes. It is usually located in the `CHANGELOG.md` file, placed at the root level of the source code. **Always** go through all the intermediate steps between the current and future dependency version tag. You probably won't be familiar with all the phrases and terminology, so take your time. Be especially careful when the major version is bumped. 83 | 3. When in doubt, look at the source code diffs. Commit messages and associated PRs usually tell most of the story, but you may still have to dig into the commit content sometimes. 84 | 4. Check for any open issues related to the new version. 85 | 5. Make an overview of all the places in the codebase that are affected by the dependency. Note the testing steps in the associated task. 86 | 6. Apply the required code changes and bump the dependency version (make sure to use the **--conservative** bundler flag to only update the necessary dependencies) 87 | 7. Run tests and verify the behaviour locally. 88 | 8. Identify & remedy [deprecation warnings](https://www.fastruby.io/blog/rails/upgrades/deprecation-warnings-rails-guide.html). 89 | 9. Deploy to a live environment and have one of your colleagues check the behaviour. In most cases it would be too expensive to run a full project smoke test, so focus on the areas that are most likely to be affected by the dependency. 90 | 10. If all goes well then deploy to production and **monitor** for regressions. 91 | 92 | -------------------------------------------------------------------------------- /Workflows/Deployment.md: -------------------------------------------------------------------------------- 1 | We use the [continuous delivery](https://en.wikipedia.org/wiki/Continuous_delivery) 2 | approach, sometimes also called [continuous integration](https://en.wikipedia.org/wiki/Continuous_integration), to deploy new versions of our websites. 3 | 4 | This approach includes an automated build service that runs your automated test 5 | suite, which can usually also deploy the latest changes to the server if the 6 | build passes. 7 | 8 | ## Why? 9 | 10 | * By having an extensive test suite that runs automatically after pushing to the 11 | remote repository, you can be certain that your new changes didn't break any 12 | old functionality. 13 | 14 | * The build server can deploy the changes automatically if it's set up to do so 15 | and the build passes successfully, which removes the need to deploy changes 16 | manually after a push to the remote repository. 17 | 18 | * The build server can also send build status notifications to Slack, emails, and 19 | other communication channels. This is useful because all members of your 20 | team get notified if a build or deployment fails. 21 | 22 | ## Mina 23 | 24 | [Mina](https://github.com/mina-deploy/mina) is the deploy tool we're currently 25 | using. It's being maintained by our very own [@d4be4st](https://github.com/d4be4st/), so if you run into any issues with it, you can always ping him. 26 | 27 | Since Mina shouldn't be used directly for deployment, we won't go into extreme lengths about it in this article. You can read more about the deployment process with Mina in the README.md of the Mina repository. 28 | 29 | ## Semaphore 30 | 31 | [Semaphore](http://www.semaphoreci.com) is the build service we're currently 32 | using. Semaphore supports notifications through instant messaging software, 33 | emails, and other methods. It also supports automated deployments, which have to 34 | be configured separately. 35 | 36 | Semaphore uses Mina for deployment in the background, and you can check other 37 | projects for details on how to set up deployment. 38 | 39 | New projects on Semaphore can be created by admin users only so, if you need to 40 | create a new project, be sure to contact your team lead. 41 | 42 | ## Git and continuous delivery 43 | 44 | Using continuous delivery for deploying our applications requires some care when 45 | handling Git branches. 46 | 47 | * The master branch is the production branch, while the develop branch is the 48 | staging branch. 49 | * Changes should never be committed or pushed to the master branch directly—they should always be merged into master from the develop branch instead. 50 | * All changes should be made in a separate branch, created from the develop 51 | branch. 52 | * When changes are completed and reviewed, they should be merged into the 53 | develop branch and deployed to the staging server. 54 | * Once the changes have passed any additional checks on the staging server, the 55 | develop branch can be merged into the master branch and deployed to production. 56 | 57 | ## Sources 58 | * [A Ruby on Rails Continuous Integration Process Using Github, Semaphore, 59 | CodeClimate and 60 | HipChat](https://infinum.co/the-capsized-eight/articles/a-ruby-on-rails-continous-integration-process-using-semaphore-github-codeclimate-and-hipchat), by Jan Varljen 61 | 62 | 63 | 64 | 65 | # Mina Workshop 66 | 67 | ## 1. Mina is just a wrapper around rake 68 | 69 | ```ruby 70 | task :hello do 71 | puts 'Hello World' 72 | end 73 | 74 | task :question do 75 | puts 'How are you?' 76 | end 77 | 78 | task answer: :question do 79 | puts 'I am fine, thanks.' 80 | end 81 | 82 | namespace :db do 83 | task :migrate do 84 | puts 'I am migrating database' 85 | end 86 | end 87 | ``` 88 | 89 | ```bash 90 | rake hello # invocation 91 | rake hello question # chainging invocation 92 | rake answer # dependencies 93 | rake db:migrate # namespaces 94 | ``` 95 | ## 2. Installing mina 96 | 97 | ```bash 98 | gem install mina 99 | ``` 100 | 101 | or add it to your Gemfile with require: false 102 | 103 | ## 3. Using mina 104 | 105 | ### 3.1 initializing 106 | 107 | ```bash 108 | mina init # creates config/deploy.rb -- Also works with ['Minafile', '.deploy.rb', 'deploy.rb', 'config/deploy.rb', minafile] 109 | ``` 110 | 111 | Least code for mina to work: 112 | 113 | ```ruby 114 | # config/deploy.rb 115 | 116 | require 'mina/default' 117 | 118 | task :deploy do 119 | puts 'deploying' 120 | end 121 | ``` 122 | 123 | ### 3.2 variables 124 | 125 | ```ruby 126 | require 'mina/default' 127 | 128 | set :hello, 'world' # setting a variable 129 | set :lambda, -> { "#{fetch(:hello)} world" } # setting as lambda 130 | 131 | task :write do 132 | puts fetch(:hello) # using variables 133 | puts fetch(:world) # default is nil 134 | puts fetch(:world, 'Heja') # can assign a defualt if nonexisting 135 | puts fetch(:lambda) 136 | end 137 | ``` 138 | 139 | Can override varibles with ENV 140 | 141 | ```bash 142 | hello=buja mina write 143 | ``` 144 | 145 | ### 3.3 commands 146 | 147 | ```ruby 148 | require 'mina/default' 149 | 150 | set :execution_mode, :pretty # modes: [:pretty, :system, :exec, :printer](https://github.com/mina-deploy/mina/blob/master/docs/how_mina_works.md#execution-modes-runners) 151 | 152 | task :deploy do 153 | run :local do # backend: [:local, :remote] 154 | invoke :predeploy # invoking other tasks 155 | command 'echo $PATH' # running shell commands 156 | end 157 | end 158 | 159 | task :predeploy do 160 | command 'pwd' 161 | end 162 | 163 | ``` 164 | ### 3.4 options 165 | 166 | ```bash 167 | mina -AT # list tasks 168 | mina -d # display configuration 169 | mina -v # verbose mode 170 | mina -s # simulate 171 | ``` 172 | 173 | ### 3.5 running on remote 174 | 175 | ```ruby 176 | require 'mina/default' 177 | 178 | set :execution_mode, :pretty 179 | set :domain, 'localhost' 180 | 181 | task :deploy do 182 | run :remote do 183 | invoke :predeploy 184 | command 'echo $PATH' 185 | end 186 | end 187 | 188 | task :predeploy do 189 | command 'pwd' 190 | end 191 | ``` 192 | 193 | ## 4. Deploying with mina 194 | 195 | ```ruby 196 | # config/deploy.rb 197 | require 'mina/deploy' 198 | require 'mina/git' 199 | 200 | set :application_name, 'mina_test' 201 | set :domain, 'localhost' 202 | set :deploy_to, '/Users/stef/dev/misc/mina_workshop/examples/4_www' 203 | set :repository, '/Users/stef/dev/misc/mina_workshop/examples/4_deploy' 204 | set :branch, 'master' 205 | 206 | task :deploy do 207 | deploy do 208 | invoke :'git:clone' 209 | 210 | on :launch do 211 | command 'pwd' 212 | end 213 | end 214 | end 215 | ``` 216 | 217 | ```bash 218 | mina setup 219 | mina deploy 220 | tree ../4_www 221 | ../4_www 222 | ├── current -> /Users/stef/dev/misc/mina_workshop/examples/4_www/releases/2 223 | ├── releases 224 | │   ├── 1 225 | │   └── 2 226 | │   └── config 227 | │   └── deploy.rb 228 | ├── scm 229 | │   ├── HEAD 230 | │   ├── config 231 | │   ├── description 232 | │   ├── hooks 233 | │   │   ├── applypatch-msg.sample 234 | │   │   ├── commit-msg.sample 235 | │   │   ├── post-update.sample 236 | │   │   ├── pre-applypatch.sample 237 | │   │   ├── pre-commit.sample 238 | │   │   ├── pre-push.sample 239 | │   │   ├── pre-rebase.sample 240 | │   │   ├── pre-receive.sample 241 | │   │   ├── prepare-commit-msg.sample 242 | │   │   └── update.sample 243 | │   ├── info 244 | │   │   └── exclude 245 | │   ├── objects 246 | │   │   ├── 03 247 | │   │   │   └── 2ebb0ede1ede2ef44ca2a0ed2182e554e22548 248 | │   │   ├── 55 249 | │   │   │   └── 3506a9a535dbdd64f3e8016a50a4f2bee39bba 250 | │   │   ├── ab 251 | │   │   │   └── 87456cb4f6725a6b39e64e22bfd84650d353c5 252 | │   │   ├── c8 253 | │   │   │   └── d683fbe9cc6ede0e82a360ab01b85f78f00f2c 254 | │   │   ├── info 255 | │   │   └── pack 256 | │   ├── packed-refs 257 | │   └── refs 258 | │   ├── heads 259 | │   └── tags 260 | ├── shared 261 | └── tmp 262 | ``` 263 | 264 | more info at https://github.com/mina-deploy/mina/blob/master/docs/deploying.md 265 | and https://github.com/mina-deploy/mina/blob/master/data/deploy.sh.erb 266 | 267 | ## 5. Avaliable plugins 268 | 269 | * deploy 270 | * git 271 | * bundler 272 | * rails 273 | * managers (rbenv, ry, rvm, chruby) 274 | -------------------------------------------------------------------------------- /Workflows/GIT/Branches.md: -------------------------------------------------------------------------------- 1 | Our Git process is an upgraded version of [Github Flow](https://guides.github.com/introduction/flow/). 2 | 3 | ![](https://ftp.infinum.co/stjepan_hadjic/git-flow-2.jpg) 4 | 5 | Instead of a single `main` branch, we use two branches to record the history of the project. The `main` branch stores code that's currently in production. The `staging` branch serves as an integration branch for features and fixes. It holds data that reflects what's currently on staging. 6 | 7 | **The main branch must always be deployable.** 8 | 9 | As we use [Productive](https://productive.io) to manage our tasks, we use **feature branches** for fixes and new features. Our branch naming convention is {type}/{task-number}{descriptive-task-name}. For example, a branch for adding authentication would be named `feature/123-authentication`. 10 | 11 | *Types* 12 | 13 | * _feature_ - new feature or an improvement to an existing feature. Goes through the code review process. 14 | * _fix_ - non-critical bugfix, improvement, paying the technical debt. Goes through the code review process. 15 | * _hotfix_ - time sensitive critical bugfix, that should be deployed to production as soon as possible. It is not necessary for it to go through a code review, but it should be revisited at a later stage, and properly fixed or improved. 16 | 17 | Feature branches should always be branched out of the `main` branch. They are also integrated first into `staging`, and then into the `main` branch if they are ready for production. 18 | 19 | **Never branch out of the `staging` branch and never integrate `staging` into the `main` branch** 20 | 21 | ## Why you need the `staging` branch and why you should never integrate it into the `main` branch 22 | We use the `staging` server so that our QA team could test our applications in an environment that is as close to production as possible. We can also work on multiple features in parallel. Those fixes and features need to be verified by our QA team and sometimes by the client as well. Sometimes, a feature may be ready for production while others aren't and are still being worked on. In that case, the `staging` branch contains multiple features, but only one needs to end up on the `main` branch. That is why we do not branch out of `staging` and do not integrate `staging` into the `main` branch. 23 | 24 | ## Note on the workflow during early development: 25 | While the application has not been deployed to a production server yet, you can omit the `staging` branch. Once the production server has been set up, and the first deploy is up, create a `staging` branch. 26 | 27 | ## Other important notes on using Git: 28 | **Commit messages are important**, especially since Git tracks your changes and then displays them as commits once they've been pushed to the server. By writing clear commit messages, you can make it easier for other people to follow and provide feedback. 29 | Commits should have a descriptive subject as well as a quick explanation of the reason for the change in the commit body. This makes it easier to check changes in the code editor as you do not have to find the pull request and open it on GitHub. 30 | Read more about writing proper commit messages [here](https://chris.beams.io/posts/git-commit/). 31 | 32 | **Follow the Single Resposibility Principle in git commits.** This makes commits easier to review when making pull requests, and it's easier to notice what's going on when something's wrong. 33 | 34 | **Don't use `git add .`** Review what you're adding to your repo — this is the #1 cause of making unwanted changes. 35 | -------------------------------------------------------------------------------- /Workflows/GIT/Clean changes.md: -------------------------------------------------------------------------------- 1 | After a while you'll get notifications about comments on your pull requests from your reviewers. Oftentimes you'll need to change something before the branch is ready. 2 | 3 | ## Solving change requests 4 | Change requests should be applied in separate _fixup_ or _squash_ commits. Rebasing the branch during an ongoing review is not appreciated unless there is a good reason for it, like pulling in some new and necessary changes from the `main` branch, because it makes it harder for the reviewers to know what the new changes are and what they have already reviewed. 5 | 6 | These commits should be integrated into `staging` as well when they are done. 7 | 8 | ## Integration Methodology 9 | 10 | ### Staging 11 | Once the PR is opened, the branch can be integrated into `staging`, unless it contains considerable 12 | logic in the migrations, in which case the reviewers should prioritize reviewing the migrations, 13 | and giving a thumbs up before integration. 14 | 15 | We use cherry picking methodology on staging: 16 | 17 | ```bash 18 | git switch staging 19 | git fetch origin staging && git pull --rebase 20 | git cherry-pick -n {BASE-OF-BRANCH}..{branch-name} 21 | ``` 22 | 23 | _Note_: BASE-OF-BRANCH is one commit prior to the first commit of the branch. 24 | 25 | The commit message should be in the following format: 26 | 27 | ``` 28 | Merge-squash {branch-name} 29 | 30 | (pull request {pull-request-link}) 31 | (cherry picked from {commit-sha}) 32 | ``` 33 | 34 | Including the pull request link in the message results with a reference in the pull request feed, which is helpful because we'll know when the branch was deployed to `staging`. 35 | 36 | You can use this [script](https://app.productive.io/1-infinum/docs/doc/113800?filter=MjQ3Nzgw&page=148751) to generate commit message. 37 | 38 | Make sure that everything is OK by running the specs, then push. 39 | 40 | #### Keep it clean 41 | After a while the `staging` branch history can become quite different than that of the `main` branch, because of the different branch integrations. You'll sometimes even find yourself pushing a couple of _fixup_ commits to `staging` if you're fixing something and then applying the changes to the `fix/branch`. 42 | 43 | The `staging` branch is reset to the `main` branch after each sprint, or more frequently as we see fit: 44 | 45 | ```bash 46 | git switch staging 47 | git reset --hard origin/master 48 | git push origin staging --force 49 | ``` 50 | 51 | ### Main branch 52 | Once the PR has at least one approval, the branch has been successfully deployed to `staging` and tested, and 53 | there are no failing specs, it can be integrated into the `main` branch. 54 | 55 | We use one of these two methods to integrate branches to the main branch: 56 | 57 | 1. Squash and merge 58 | 2. Merge without squashing 59 | 60 | Before merging we also rebase the feature branch onto the latest `main` branch, so that the git 61 | history is nice and clean, and that the latest code can be run on CI before actually merging into 62 | the `main` branch. 63 | 64 | While on feature branch: 65 | 66 | ```bash 67 | git fetch origin master 68 | git rebase -i origin/master 69 | ``` 70 | 71 | This will start interactive rebase. You can edit it with your editor of choice. 72 | To set the default editor you'll need to edit the git configuration: 73 | 74 | ```bash 75 | git config core.editor "{editor-name} --wait" 76 | ``` 77 | 78 | Examples for visual studio code, atom and vim: 79 | 80 | ```bash 81 | git config core.editor "code --wait" 82 | git config core.editor "atom --wait" 83 | git config core.editor "vim --wait" 84 | ``` 85 | 86 | Refer to [this](https://git-scm.com/book/en/v2/Appendix-C%3A-Git-Commands-Setup-and-Config) guide 87 | for more information. 88 | 89 | Once the interactive rebase has started, you will see feature branch's commits listed. Each commit will consist of command, SHA (commit hash) and commit's text. Each commit will, initially, have `pick` command. You'll see something similar to this (may vary depending on the editor): 90 | 91 | ```bash 92 | pick 07c5abd First commit 93 | pick de9b1eb Add controller 94 | pick 3e7ee36 Remove controller 95 | pick fa20af3 Finish up 96 | 97 | # Rebase 8db7e8b..fa20af3 onto 8db7e8b 98 | # 99 | # Commands: 100 | # p, pick = use commit 101 | # r, reword = use commit, but edit the commit message 102 | # e, edit = use commit, but stop for amending 103 | # s, squash = use commit, but meld into previous commit 104 | # f, fixup = like "squash", but discard this commit's log message 105 | # x, exec = run command (the rest of the line) using shell 106 | # 107 | # These lines can be re-ordered; they are executed from top to bottom. 108 | # 109 | # If you remove a line here THAT COMMIT WILL BE LOST. 110 | # 111 | # However, if you remove everything, the rebase will be aborted. 112 | # 113 | # Note that empty commits are commented out 114 | ``` 115 | 116 | Save and close the editor to finish the rebase and then force push to update remote's history: 117 | 118 | ```bash 119 | git push --force-with-lease 120 | ``` 121 | 122 | #### Squash and merge 123 | 124 | After there are no conflicts, select the Squash and merge option on the pull request. 125 | This will combine all commits into a single one and merge it into the master branch. 126 | 127 | #### Merge without squashing 128 | 129 | If you want to keep all commits from your pull request on the main branch, use the merge without squashing method: 130 | 131 | ```bash 132 | git fetch origin master 133 | git switch master && git pull 134 | git merge --no-ff --no-edit {branch-name} 135 | ``` 136 | 137 | We perform non-fast-forward merges to group commits of a single feature together in a meaningful way. 138 | 139 | Make sure the history graph is nice and clean by entering the following command or similar. No lines should "cross over". 140 | 141 | ```bash 142 | git log --oneline --graph 143 | ``` 144 | 145 | Bad: 146 | 147 | ```bash 148 | * 1b82b9f (HEAD -> master) Merge branch 'feature/add-git-process-to-readme' 149 | |\ 150 | | * a25b3dc (origin/feature/add-git-process-to-readme, feature/add-git-process-to-readme) Add git process to readme 151 | * | bfe1152 (origin/master) Merge branch 'feature/xx-some-other-feature' 152 | |\ \ 153 | | |/ 154 | |/| 155 | | * 3345dbb Some other feature subject 156 | |/ 157 | * 7eade95 Merge branch 'feature/xx-another-other-feature' 158 | |\ 159 | | * 0a80385 Another feature subject 160 | |/ 161 | * 162 | ``` 163 | 164 | Good: 165 | 166 | ```bash 167 | * 1a164b4 (HEAD -> master) Merge branch 'feature/add-git-process-to-readme' 168 | |\ 169 | | * 497dcd7 (origin/feature/add-git-process-to-readme, feature/add-git-process-to-readme) Add git process to readme 170 | |/ 171 | * bfe1152 (origin/master) Merge branch 'feature/xx-some-other-feature' 172 | |\ 173 | | * 3345dbb Some other feature subject 174 | |/ 175 | * 7eade95 Merge branch 'feature/xx-another-other-feature' 176 | |\ 177 | | * 0a80385 Another feature subject 178 | |/ 179 | * 180 | ``` 181 | 182 | Make sure that everything is OK by running the specs, then push. 183 | 184 | If you're having trouble understanding the differences between squashing, rebasing or any other topics covered in these chapters, you can learn more from the [Git book](https://git-scm.com/book/en/v2). 185 | -------------------------------------------------------------------------------- /Workflows/GIT/Pull requests.md: -------------------------------------------------------------------------------- 1 | Pull requests initiate discussion about your commits. Since they're tightly integrated with the underlying Git repository, everyone can see exactly what changes will be merged if they accept your request. 2 | 3 | ## New pull request 4 | Once a feature or fix is done, a PR for it should be opened. 5 | 6 | It's essential to write a good pull request description. Reviewers will usually read it before 7 | looking at the diff, so make sure it gives them enough context to know what they are looking at. 8 | 9 | Another purpose of a good description is to provide the documentation for a future reference. 10 | A new developer might join the project in the future. The description will help him understand 11 | the motivation behind the implementation of a specific feature. Please note that any important information 12 | from the description should also be added to the main documentation in the project as a central reference point 13 | where the latest state of the project is documented. 14 | 15 | A new Rails project created with the [Infinum rails template](https://github.com/infinum/default_rails_template) 16 | includes a [pull request description template](https://help.github.com/en/articles/creating-a-pull-request-template-for-your-repository). 17 | The template is defined in the `rails_project/.github/PULL_REQUEST_TEMPLATE.md` file. 18 | 19 | ``` 20 | Task: [#__TASK_NUMBER__](__ADD_URL_TO_PRODUCTIVE_TASK__) 21 | 22 | #### Aim 23 | 24 | 25 | #### Solution 26 | 27 | ``` 28 | 29 | ### Link to the task 30 | Replace `__TASK_NUMBER__` with a number of the task from Productive (eg. `149`) and 31 | `__ADD_URL_TO_PRODUCTIVE_TASK__` with the url to the task (eg. `https://app.productive.io/1-infinum/m/task/487456`). 32 | 33 | ### Aim 34 | In the Aim section, provide enough information for reviewers to have context for your changes. 35 | 36 | - Why are we introducing this change? 37 | - Why now? 38 | - What problems is the code solving now? 39 | - How does it work on a business level? 40 | - Why have we picked this solution historically, etc.? 41 | 42 | ### Solution 43 | Explain why things are done the way they are in this PR. Highlight the most important and/or controversial design decisions you have taken. 44 | 45 | - Why is it implemented the way it is? 46 | - What alternative implementations have you considered and why have you chosen this one, etc.? 47 | 48 | The description is a good place to include questions that came up during development. 49 | 50 | - Is this class/method name the best one? 51 | - Could approach B be more applicable in this case, etc.? 52 | 53 | This is also a good place to talk about the performance and security considerations if there are any. 54 | 55 | ## Default pull request reviewers 56 | It's a bit tedious to add the same reviewers to pull requests over and over again. GitHub allows us to set a 57 | list of default PR reviewers. 58 | 59 | A new rails project created with the [Infinum rails template](https://github.com/infinum/default_rails_template) 60 | includes a list of default pull request reviewers known as [Code Owners](https://help.github.com/en/articles/about-code-owners)). 61 | The list is defined in the `rails_project/.github/CODEOWNERS` file. 62 | 63 | When `rails new` command is run, a developer is prompted to enter a list of GitHub username handles 64 | that are automatically added to the file as code owners. 65 | -------------------------------------------------------------------------------- /Workflows/Production release.md: -------------------------------------------------------------------------------- 1 | ## Web application checklist 2 | 3 | Before you release a Rails application to production, you **must** go through this checklist. 4 | 5 | Not all checklist items are necessary in every app. 6 | 7 | If you're not sure whether you should add some feature (e.g., Google Analytics) to the web app you're releasing, consult with the project manager and/or your team leader. 8 | 9 | In your project management app (e.g., Productive, Trello...), create a _Production Checklist_ task and link to this handbook chapter. Finish it prior to production release. 10 | 11 | ## Checklist items: 12 | 13 | * Are you uploading all files to S3 and not to the local (public/system) filesystem? 14 | * Check that the app doesn't have any security issues on Code Climate. If it isn't on Code Climate, why isn't it? If there is a specific reason for not using Code Climate, check the security of your application with [Brakeman](https://github.com/presidentbeef/brakeman). 15 | * Check that Bugsnag is included. 16 | * Gzip the content with [Rack Deflater](http://robots.thoughtbot.com/content-compression-with-rack-deflater/). 17 | * Check that indices are all set up. If you have the Postgres `pg_stat_statements` extension enabled, check what are some of the slow queries and consult with the team on ways to decrease their execution time. Also, check out [lol_dba](https://github.com/plentz/lol_dba)—a static checker for indices based on foreign keys. 18 | * Check if models have actions dependent on their associations. 19 | 20 | **HTML apps only:** 21 | 22 | * Test the app on various browsers. The usual set of browsers is (this can vary from project to project): Chrome, FireFox, Opera, Safari, and IE8+. Do this before the app goes to QA so they don't have to return the app because of trivial mistakes. 23 | * Go through the final design review with the designer. 24 | * Pass on the application to the QA team so that they can try to find potential bugs before the client or customers do. 25 | * Add the cookies_eu gem. 26 | * Add Google Analytics. 27 | * Add metatags. 28 | * Add a favicon. 29 | * Add a custom 404/500 page if necessary. 30 | * Did you add credits (i.e., a link to www.infinum.co and Developed by Infinum)? 31 | 32 | **DevOps-specific:** 33 | 34 | * Is your application listed in Infinum's Catalog? Is all data up-to-date? 35 | * Are we monitoring the app? 36 | * Where's the DNS zone? If there are no dependencies (such as Cpanel, lots of subdomains, etc.), you're probably better off hosting it on our DNS servers (PointHQ). It's faster to make changes that way than by contacting a third-party DNS host. 37 | * Lower TTL for all A/CNAME records to 300. 38 | * Are you using SSL? 39 | * Check if the log rotation is set up on the server. 40 | -------------------------------------------------------------------------------- /Workflows/Rails upgrades.md: -------------------------------------------------------------------------------- 1 | Rails is probably one of your biggest project dependencies. While it may look like a big blob of magic, Rails is in essence a composition of multiple opinionated ([omakase](https://dhh.dk/2012/rails-is-omakase.html)) frameworks. Each of the individual functionalities (object-relation mapping, view templating, email sending, ...) is handled by its own gem: 2 | 3 | `actioncable, actionmailbox, actionmailer, actionpack, actiontext, actionview, activejob, activemodel, activerecord, activestorage, activesupport` 4 | 5 | Upgrading Rails is thus much more [challenging](https://engineering.shopify.com/blogs/engineering/upgrading-shopify-to-rails-5-0) and [tends to hurt](https://blog.testdouble.com/posts/2019-09-03-3-keys-to-upgrading-rails) because of the big surface area. You're essentially upgrading 11 gems at one time, which are themselves dependencies to a lot of your project's dependencies. 6 | 7 | 8 | ## Upgrade strategy 9 | Planning a Rails upgrade strategy is trivial on well maintained projects where we're only one minor/major Rails version behind, but there are projects where we're severly behind and need to set a precise plan of action. The steps below can guide you in both of those cases and should be formally noted as part of the upgrade task. 10 | 11 | ### Assessing the current state of the project 12 | The prerequisite for creating a roadmap is knowing your current location. You should make note of these properties before commencing a Rails upgrade cycle: 13 | 14 | - Ruby version 15 | - how many minor/major versions away from the latest version is it 16 | - is it [EOL](https://endoflife.date/ruby) 17 | - Rails version 18 | - how many minor/major versions away from the latest version is it 19 | - is it [EOL](https://endoflife.date/rails) 20 | - test suite confidence 21 | - number of tests 22 | - code coverage 23 | - types of tests 24 | - do tests pass locally 25 | - deployment pipeline 26 | - is it functional 27 | - do we have access to it 28 | - dependency age 29 | - how many dependencies does the project have 30 | - when were they last updated 31 | - how far away from the latest version are they 32 | - context 33 | - do we have anyone from the client side available to answer basic questions 34 | - do we have anyone with in-depth knowledge of the project available internally 35 | - does the QA team have a set of test cases or someone with in-depth knowledge of the project to perform testing 36 | - do we have established deployment slots (e.g. scheduled, manually approved) 37 | 38 | ### Determining upgrade steps 39 | You should **always** update one minor version of Rails or Ruby at a time. Each new minor version increment can introduce **important deprecation warnings** to warn you of changes in the next minor/major release. Skipping minor versions can lead to unexpected, hard to diagnose errors. It may seem like you're going slower one minor version at a time, but you'll be faster (and safer) in the long run. Minimizing the complexity and risk of code changes is always the better option. 40 | 41 | Our goal is to follow the [Ruby](https://www.ruby-lang.org/en/downloads/releases/) and [Rails](https://rubyonrails.org/category/releases) release cycles closely and be as aligned with new releases as we can be. 42 | 43 | Writing out the individual upgrade steps will enable you to better visualize the scope of work and determine the order of execution. The upgrades generally follow a predetermined cycle for each Rails upgrade 44 | 45 | - upgrade Ruby [if the next version of Rails requires it](https://www.fastruby.io/blog/ruby/rails/versions/compatibility-table.html) 46 | - [upgrade gems so they support the next version of Rails](#1-upgrading-gems-for-rails-x-support) 47 | - [upgrade Rails to the next minor version](#2-rails-x-upgrade) 48 | - [enable new Rails defaults](#3-enabling-rails-x-defaults) 49 | 50 | Here is an example of a roadmap for a very outdated project where we want to upgrade Ruby and Rails from 5.1.7 to 7.0.4 and Ruby from 2.5.5 to 3.1.3. 51 | 52 | ``` 53 | - Rails 5.1.7 -> Rails 5.2.8.1 54 | - lots of point releases with CVEs 55 | - Upgrading gems for Rails 6 support 56 | - Rails 5.2.8.1 -> Rails 6.0.6 57 | - add zeitwerk autoloader support 58 | - Rails 6 enable defaults 59 | - Rails 6.0.6 -> Rails 6.1.7 60 | - Rails 6.1 enable defaults 61 | - Ruby 2.5.5 -> Ruby 2.6.9 62 | - Ruby 2.6.9 -> Ruby 2.7.7 63 | - 2.7.6 introduces 3.0 deprecation warnings which are disabled by default 64 | - 2.7.x required by Rails 7 65 | - Upgrading gems for Rails 7 support 66 | - Rails 6.1.7 -> Rails 7.0.4 67 | - Rails 7.0 enable defaults 68 | - Fixing Ruby 3 deprecation warnings 69 | - updating gems 70 | - updating application code 71 | - RUBYOPT='-W:deprecated' bundle exec rspec 72 | - Ruby 2.7.7 -> 3.0.5 73 | - possible Passenger issues due to bundled gems 74 | - Ruby 3.0.5 -> 3.1.3 75 | - possible Passenger issues due to bundled gems 76 | ``` 77 | 78 | Each top level item should have **its own task** (with specific acceptance criteria, estimate and test steps for QA) and **should be deployed individually**. 79 | 80 | You can find 4 more comprehensive examples in our [project upgrade plan folder](https://drive.google.com/drive/folders/1JIVRJRS_RD2wGSmUEiIqGUEyWTItaxH6?usp=share_link). 81 | 82 | #### Priorities 83 | If a project is in bad shape, you may be required to combine upgrade efforts with patching security vulnerabilities. The budget and technical constraints might vary by project, but in general you should prioritize the high-priority low-hanging security patches first. 84 | 85 | 1. High priority CVEs for dependencies not related to Rails 86 | 2. High priority Brakeman warnings not related to Rails 87 | 3. Rails upgrades 88 | 4. Ruby upgrades 89 | 5. CVE leftovers 90 | 5. Brakeman leftovers 91 | 92 | ### Estimations 93 | Estimating upgrades is tough work due to the Rails gravitational pull. Auxiliary gems are tightly coupled to it and so is our application. We can rely on our gut in some cases, but that can be easily thrown off by a seemingly simple one line changelog notice or an outdated auxiliary dependency that will require major application level changes before we can upgrade it. 94 | 95 | Splitting the work into phases might help us visualize the scope of work a bit better, but we should also perform a check of potential challenges **in the estimation phase**. 96 | 97 | - assess the [current state of the project](#assessing-the-current-state-of-the-project) 98 | - prepare a [roadmap](#upgrade-strategy) and estimate each phase individually 99 | - find exactly which gems [explicitly block the Rails upgrade](#explicit-dependency-rails-support) and estimate their upgrades individually 100 | - find exactly which gems [implicitly block the Rails upgrade](#implicit-dependency-rails-support) and estimate their upgrades individually 101 | - check the [Rails changelogs](#2-rails-x-upgrade) ahead of time and triage which breaking changes are likely to affect the estimate (e.g. the introduction of Zeitwerk autoloading might require you to rename modules and classes) 102 | - check in with the rest of the Rails team and ask very specific questions 103 | - has anyone performed **this exact** upgrade? 104 | - what was the initial estimation and actual time spent on the upgrade? 105 | - was the work deployed in phases? 106 | - who performed the testing and to what extent? 107 | - was the project similar in size or requirements (big/small, internal/client)? 108 | - did we discover any gotchas? 109 | - do a trial upgrade of the Gemfile if possible and note the count of deprecation warnings or application errors when running specs 110 | - check the dependency project health, projects which haven't been updated for a long time will likely require more work 111 | - sync with the project team to determine the amount of manual testing you want to do in each specific phase (e.g. testing yourself, small/complete smoke tests by the QA team). Communicate separate development and QA testing estimates 112 | - sync with the devops team to provide a Ruby installation estimate if applicable 113 | 114 | Projects in bad health will require extra care in the estimation phase. The following factors may severly (e.g. 3x) impact the estimate 115 | 116 | - low test coverage 117 | - low internal application functionality knowledge 118 | - being more than 1 major Rails or Ruby version behind 119 | - incomplete or proprietary deployment pipeline 120 | - poor codebase health 121 | - lack of functional specification 122 | - forked dependencies 123 | - unmaintained dependencies (e.g. paperclip) that may need to be replaced 124 | - dependencies which haven't received support for the next version of Rails that may need to be replaced 125 | - the app is not bootable locally 126 | - there is no staging environment 127 | 128 | **Please perform a sanity check with a TL/LE for all major Rails upgrades.** 129 | 130 | ## Upgrade steps 131 | The upgrade process can be split into multiple smaller phases. Splitting the work, reviewing and deploying it individually enables us to: 132 | 133 | - introduce smaller code changes, 134 | - observe and review a single change at a time, 135 | - perform smaller, more targeted smoke tests of the application, 136 | - more easily plan multiple smaller amounts of work (usually a a day or so per phase) as opposed to a big (weeks) chunk of work, 137 | - estimate the work with a higher degree of confidence. 138 | 139 | The phases noted below should each contain their own task, acceptance criteria and verification steps (e.g. partial or full smoke test). They should be deployed to the production environment **individually** and there should be some grace period between them when required. 140 | 141 | ### Exceptions 142 | Please check in with a fellow TL/LE if you won't be following these guidelines due to project specific constraints. Bundling the upgrade steps together does introduce extra risk so we should be explicit and communicate the tradeoffs to stakeholders. You should still follow the upgrade procedure order locally and keep a representative commit history, even in cases where certain upgrade steps will be deployed together. 143 | 144 | We strongly discourage bundled Ruby and Rails upgrades, but gem upgrades, minor Rails version upgrades and enabling defaults can be combined into one step in cases when [application](#application-level-backwards-compatibility-with-previous-versions-of-rails) or [framework](#framework-level-backwards-compatibility-with-previous-versions-of-rails) level compatibility layers are not needed. 145 | 146 | ### 1. Upgrading gems for Rails X support 147 | Rails has a strong gravitational pull towards the general gem ecosystem, so it is very likely that most Rails upgrades will require some auxiliary dependencies to be upgraded as well. Compiling a list of gems that need to be upgraded to support the next Rails version will allow you to split the upgrade into multiple parts and deploy smaller, safer changes. 148 | 149 | Gems declare their runtime (`add_runtime_dependency` or `add_dependency`) dependency version constraints in the `gemspec` file, but that is not a solid guarantee that a dependency supports the next version of Ruby or Rails. 150 | 151 | ``` 152 | s.required_ruby_version = '>= 2.3' 153 | s.add_runtime_dependency 'activesupport', '>= 3.0.0' 154 | s.add_runtime_dependency 'uniform_notifier', '~> 1.11' 155 | ``` 156 | 157 | Gem development dependencies (`add_development_dependency`) are not relevant for your application upgrade so you can ignore them. 158 | 159 | #### Explicit dependency Rails support 160 | All project dependencies and their dependency requirements are listed in the `Gemfile.lock` file. Some gems constrict their dependency requirements quite conservatively while others keep an open mind. Relaxing these dependencies (usually by upgrading them) is the first step towards enabling you to upgrade to the next Rails version. 161 | 162 | In the following case we present a situation where a project is trying to upgrade to Rails 6.1, but is blocked by one of the dependencies (`delayed_job`) to Rails versions betwen 3.0 and 6.0.x. 163 | 164 | ``` 165 | delayed_job (4.1.8) 166 | activesupport (>= 3.0, < 6.1) 167 | ``` 168 | 169 | An upgrade of the Rails gem (`bundle update rails --conservative`) will fail in this case. 170 | 171 | Finding all of the gems that explicitly require upgrades will likely require a repetitive process and some of your time. 172 | 173 | - running `bundle update rails --conservative` 174 | - checking the output `Bundler attempted to update rails but its version stayed the same` 175 | - checking `Gemfile.lock` and bundler output for dependencies which might be blocking the upgrade 176 | 177 | ``` 178 | delayed_job (4.1.8) 179 | activesupport (>= 3.0, < 6.1) 180 | delayed_job_active_record (4.1.4) 181 | activerecord (>= 3.0, < 6.1) 182 | delayed_job (>= 3.0, < 5) 183 | audited (4.9.0) 184 | activerecord (>= 4.2, < 6.1) 185 | ``` 186 | 187 | - adding the dependency to the upgrade list and retrying the upgrade `bundle update rails --conservative delayed_job delayed_job_active_record audited` 188 | 189 | Once the `bundle update` command passes successfully you'll have a list of the exact **explicit** dependencies which are blocking the Rails upgrades. 190 | 191 | #### Implicit dependency Rails support 192 | Some dependencies might have very open constraints in terms of supported Rails versions. 193 | 194 | ``` 195 | bullet (6.1.5) 196 | activesupport (>= 3.0.0) 197 | uniform_notifier (~> 1.11) 198 | ``` 199 | 200 | These requirements would allow you to upgrade the Rails gem from version 6 to 7, but that does not guarantee that the selected gem and its functionalities are compatible with the new Rails version. 201 | 202 | In these types of cases the [gem changelog](https://github.com/flyerhzm/bullet/blob/master/CHANGELOG.md) is your best source of information as it usually indicates exactly when support for a new Rails version was added. 203 | 204 | > #### 7.0.0 (12/18/2021) 205 | > * Support **rails 7** 206 | > * Fix Mongoid 7 view iteration 207 | >* Move CI from Travis to Github Actions 208 | 209 | > #### 6.1.5 (08/16/2021) 210 | >* Rename whitelist to safelist 211 | >* Fix onload called twice 212 | >* Support Rack::Files::Iterator responses 213 | >* Ensure HABTM associations are not incorrectly labeled n+1 214 | 215 | In case the changelog is empty, you still have some options: 216 | 217 | - check the gem readme for mentions of compatibility 218 | - chech the gem (open and closed) issues for mentions of compatiblity 219 | - check with the Rails team if any projects with that dependency have already upgraded Rails 220 | - check the commit messages and diffs. 221 | 222 | Going through changelogs of all dependencies is painstaking work so focus on the core set of your dependencies (e.g. authentication, email sending, background jobs) first. The test suite and QA process is likely to suss out any remaining incompatibilities, but we should at least do our due diligence on the most important dependencies. 223 | 224 | #### Gem upgrades 225 | Once you've identified the suite of gems that are incompatible with the Rails upgrade, you should update them and deploy them to production **before** upgrading Rails. 226 | 227 | In some cases dependency upgrades can be grouped together to reduce the amount of QA effort and overhead. We advocate for this with tightly coupled gems (e.g. `devise` and `devise_invitable`) since they support the same functionality and can be tested together. Development and test dependencies (e.g. `puma`, `rspec`, `factory_bot`) are also a good candidate since they don't require a formal QA step. 228 | 229 | ### 2. Rails X upgrade 230 | Your primary source of information for minor and major upgrades should be the [Rails upgrade docs](https://guides.rubyonrails.org/upgrading_ruby_on_rails.html). **Read about the relevant changes once, then read again.** An ounce of prevention is worth a pound of cure. Don't start playing error whack-a-mole without fully understanding the framework changes introduced by a new version. 231 | 232 | Minor version upgrades have their changelogs linked on the [Rails release page](https://rubyonrails.org/2022/9/9/Rails-7-0-4-6-1-7-6-0-6-have-been-released). 233 | 234 | There are two levels of changelogs to go through: 235 | 236 | - changelogs for each individual Rails gem (e.g. [ActiveRecord](https://github.com/rails/rails/blob/v6.0.6/activerecord/CHANGELOG.md)) 237 | - [condensed changelog summary](https://github.com/rails/rails/releases/tag/v6.0.6) for that specific Rails version. 238 | 239 | If you need to upgrade through multiple patch versions (e.g. Rails 6.0.1 to Rails 6.0.6) you should [compare the tags](https://github.com/rails/rails/compare/v6.0.1...v6.0.6) to get a look at all the changelogs. Sometimes even patch versions might introduce breaking changes to [patch security vulnerabilities](https://github.com/advisories/GHSA-3hhc-qp5v-9p2j). 240 | 241 | #### Gemfile changes 242 | The Gemfile contains your Rails version requirement (e.g. `gem rails, '~> 6.0.3'`), but the exact version of you're using is noted in `Gemfile.lock`. If the requirement is too restrictive, then you will need to relax it (e.g. `gem rails, '~> 6.0'`) before running the bundle update command 243 | 244 | ``` 245 | bundler update rails --conservative 246 | ``` 247 | 248 | #### Configuration file changes 249 | Updating Rails configuration files is sometimes tricky due to the amount of configuration file changes. The interactive update tool `rails app:update` is the best resource to guide you through the process with well established options like overwrite, show diff and run merge tools. While this process will run smoothly for vanilla apps, you will run into merge conflicts with apps that have a lot of custom edits in the configuration files. Approach each file change separately and make sure you do this step right. You can use [Rails diff](http://railsdiff.org) to get a better (cleaner) view of the before and after state of the configuration files. 250 | 251 | If you wish to make your next upgrade easier, then make sure to add custom configuration options at the end of the environment file (e.g. `config/environments/production.rb`) to reduce the chance of merge conflicts. 252 | 253 | The update command may also generate migrations (e.g. `activestorage`) and a new framework defaults file (`config/initializers/new_framework_defaults_x_y.rb`) which you will tackle [later](#3-enabling-rails-x-defaults). 254 | 255 | #### Deprecation warnings 256 | Running the test suite at this point is likely to bring some new deprecation warnings to the surface. Addressing these deprecation warnings is crucial since they will start raising runtime errors in the next release cycle. 257 | 258 | ### 3. Enabling Rails X defaults 259 | Most Rails upgrades introduce some amount of breaking changes. The purpose of the `new_framework_defaults_x_y.rb` file is to allow you to **opt-in** to these changes **at your own pace**, separating the upgrade of Rails and introduction of breaking behaviour. 260 | 261 | The defaults file should be carefully triaged. You should be able to answer the following set of question before enabling each of the options: 262 | 263 | - how does the behavior change look in practice? 264 | - which parts of the application code will this affect? 265 | - how would we verify if this change will cause us an issue? 266 | - what is the estimated effort of changing the application level code to comply with this option? 267 | 268 | If the risk or effort of an individual option is deemed too high, then you should tackle it separately from the rest of the options. Most of these low-hanging-fruit defaults can be packaged and deployed to production together, with risky ones being tackled in separate tasks since they usually require some application level changes and a more focused QA effort. 269 | 270 | Once **all** of the defaults have been uncommented in the `new_framework_defaults_x_y.rb` file, you can change the `config.load_defaults` directive in `application.rb` to match your Rails version. At this point the framework defaults file is redundant and can be deleted. 271 | 272 | #### Application level backwards compatibility with previous versions of Rails 273 | In the following example Rails 7 changed the default digest class and introduced a breaking change. 274 | 275 | ``` 276 | # Change the digest class for the key generators to `OpenSSL::Digest::SHA256`. 277 | # Changing this default means invalidate all encrypted messages generated by 278 | # your application and, all the encrypted cookies. Only change this after you 279 | # rotated all the messages using the key rotator. 280 | # 281 | # See upgrading guide for more information on how to build a rotator. 282 | # https://guides.rubyonrails.org/v7.0/upgrading_ruby_on_rails.html 283 | # Rails.application.config.active_support.key_generator_hash_digest_class = OpenSSL::Digest::SHA256 284 | ``` 285 | 286 | Encrypted cookies created by earlier (e.g. 6.1) Rails versions would immediately become invalid once the upgrade was deployed if the option was enabled by default. In this case the grace period between deploying Rails 7 and enabling this setting allows us to implement the cookie rotator in application code safely. We can turn this option on after some time (e.g. a few days) when we're sure that all the users' cookies have been safely rotated on the production environment. 287 | 288 | #### Framework level backwards compatibility with previous versions of Rails 289 | 290 | In the following example Rails 6.1 introduced a change in how CSRF tokens are encoded. 291 | 292 | ``` 293 | # Generate CSRF tokens that are encoded in URL-safe Base64. 294 | # 295 | # This change is not backwards compatible with earlier Rails versions. 296 | # It's best enabled when your entire app is migrated and stable on 6.1. 297 | # Rails.application.config.action_controller.urlsafe_csrf_tokens = true 298 | ``` 299 | The defaults file explicitly states that previous versions of Rails (e.g. 6.0) are not compatible with this change. If you enabled this option, deployed the change to the production environment, let user traffic to it (generating new CRSF tokens) and for any reason had to revert to the previous version of Rails, you would end up errors. The new CSRF tokens generated by Rails 6.1 and still stored on the client side would not be valid when checked by Rails 6.0 application code so users would be unable to save any existing forms they might have had open, potentially even losing data. 300 | 301 | You will not hit any such issues if you follow the explicit recomendation of only enabling this option once the new version of Rails has been stable in production for a while. 302 | -------------------------------------------------------------------------------- /Workflows/Starting a new project.md: -------------------------------------------------------------------------------- 1 | ## Starting a new project 2 | 3 | When you're starting a new project, there are a few things you need to do: 4 | 5 | #### 1. Open a repository 6 | 7 | Ask your team lead or DevOps to open a repository. 8 | 9 | #### 2. Fill out the project setup sheet 10 | 11 | We've come up with a project setup template in cooperation with the DevOps team. Its purpose is to provide the DevOps with all the details needed for deploying your application to staging and production servers. 12 | 13 | It contains general info about your project that your project manager can fill out, as well as more technical information like frameworks and technologies used, the build and deploy setup, third party services that need to be set up, etc. 14 | 15 | Ask your PM for a link to the sheet template. 16 | 17 | **Fill in the blanks on backend-related stuff.** 18 | 19 | If you need any other services that aren't mentioned, **provide as much information as possible** 20 | 21 | **The project setup sheet needs to be up to date** with new changes to the project setup and services used. If you're adding some new dependency or service or changing something, please update it in the project setup sheet and notify DevOps to make these changes. 22 | 23 | 24 | #### 3. Generate a new application 25 | 26 | Use the [**default rails template**](https://github.com/infinum/default_rails_template/) to start your new application. 27 | 28 | Most of our work is based on the Rails framework. We have some standard configurations and gems that are basically in every Rails application. To make our lives easier, we've made a default template to generate new Rails applications that meet those standards. 29 | 30 | Please always update the default rails template if something gets broken or out of date when you're starting a new project. 31 | 32 | 33 | #### 4. Check for new/better gems 34 | 35 | We usually use specific gems for some purposes, e.g. Devise for authentication, Pundit for authorization, etc. That doesn't mean we need to use those gems necessarily. 36 | 37 | When adding a gem to your application, check if there's some new and better gem for that purpose. 38 | 39 | When checking out a gem, always look for these things: 40 | 41 | * is the gem maintained? when was the last commit? 42 | * how many stars does it have? 43 | * what is the ratio between open and closed issues? 44 | 45 | 46 | #### 5. Fill out the readme file 47 | 48 | After setting up a project, fill out the readme file with all the necessary details about the application: 49 | 50 | * dependencies & prerequisites 51 | * setup instructions 52 | * development instructions 53 | * deployment instructions 54 | * test instructions 55 | * any other project specific information that someone who takes over a project needs to know 56 | 57 | 58 | #### 6. Create an ER diagram 59 | 60 | Before you start coding, you always need to create an ER diagram for your database. Go through the project specification and list all the entities and their attributes and construct relationships between listed entities. 61 | 62 | For creating an ER diagram use: 63 | 64 | * [DB diagram](https://dbdiagram.io/) 65 | * [sqlDBM](https://sqldbm.com/) 66 | * [MySQL Workbench](https://dev.mysql.com/downloads/workbench/) 67 | 68 | After creating an ER diagram, you need to go through an architecture review. You can read more about it in the [Architecture review chapter](Architecture review) 69 | 70 | If you need to create an application or a server architecture diagram use: 71 | 72 | * [Gliffy](https://www.gliffy.com/) 73 | 74 | If you need to create a sequence diagram use: 75 | 76 | * [Web Sequence Diagram](https://www.websequencediagrams.com/) 77 | -------------------------------------------------------------------------------- /intro.slim: -------------------------------------------------------------------------------- 1 | .intro 2 | section.lead.intro__lead 3 | .lead__image.lead__image--rails-header.lead__image--shorter.lead__image--eat-mobile-gutter 4 | .lead__hashtag.lead__hashtag--dark #Don’tGoOffTheRails 5 | h1.lead__title Infinum Rails Handbook 6 | .lead__info.lead__info--confined Our handbook is based on 15+ years of experience in Rails development. Here are some numbers behind it: 7 | 8 | .generic-container 9 | section 10 | .intro__testimonials 11 | .testimonial.intro__testimonial 12 | .testimonial__number.testimonial__number--small 200,000+ 13 | .label.label--hours 14 | .label__desc.label__desc--small hours of coding 15 | 16 | .testimonial.intro__testimonial 17 | .testimonial__number.testimonial__number--small 600,000+ 18 | .label.label--code 19 | .label__desc.label__desc--small lines of code 20 | 21 | .testimonial.intro__testimonial 22 | .testimonial__number.testimonial__number--small 100+ 23 | .label.label--projects 24 | .label__desc.label__desc--small launched apps 25 | 26 | .testimonial.intro__testimonial 27 | .testimonial__number.testimonial__number--small 20+ 28 | .label.label--people 29 | .label__desc.label__desc--small developers 30 | 31 | .paragraph.intro__paragraph 32 | .paragraph__title Introduction 33 | .paragraph__content Rails is a web framework written in Ruby that allows developers to rapidly build web applications. In this handbook, we expand on the cornerstones that Rails is built upon, and deal with tools, processes and patterns that allow us to build and maintain large Rails applications, while keeping the code readable and extensible. 34 | --------------------------------------------------------------------------------