├── README.md ├── LICENSE ├── kibana_dev_principles.md └── development_constitution.md /README.md: -------------------------------------------------------------------------------- 1 | Elastic Engineering 2 | =================== 3 | 4 | Sharing our engineering philosophies with the public. 5 | 6 | * [Elasticsearch Team Development Constitution](./development_constitution.md) 7 | * [Kibana Development Principles](./kibana_dev_principles.md) 8 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "{}" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright {yyyy} {name of copyright owner} 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /kibana_dev_principles.md: -------------------------------------------------------------------------------- 1 | # Kibana development principles 2 | 3 | Over time, the Kibana project has been shaped by certain principles. Like Kibana itself, some of these principles were formed by intention while others were the result of evolution and circumstance, but today all are important for the continued success and maintainability of Kibana. 4 | 5 | This document is inspired by and at times reflects points already covered in the [Elasticsearch Development Constitution](./development_constitution.md). When this happens it is either to reframe a principle from the development constitution in terms that are more clearly relevant to Kibana or to emphasize principles that are underrepresented in the Kibana project today. 6 | 7 | While the Elasticsearch Development Constitution is written with Elasticsearch in mind, its principles are broadly relevant throughout Elastic's engineering culture and especially for the Kibana project. For that reason, Kibana developers should read and internalize that document as well. 8 | 9 | * [Don't compromise on quality](#dont-compromise-on-quality) 10 | * [Do the right thing even when it's the hard thing](#do-the-right-thing-even-when-its-the-hard-thing) 11 | * [Automate tests through CI](#automate-tests-through-ci) 12 | * [Organize by domain](#organize-by-domain) 13 | * [Prefer one way to do things](#prefer-one-way-to-do-things) 14 | * [Prioritize consistency within Kibana](#prioritize-consistent-code-within-kibana) 15 | * [TypeScript over JavaScript](#typescript-over-javascript) 16 | * [Be wary of dependencies](#be-wary-of-dependencies) 17 | * [Don't share code prematurely](#dont-share-code-prematurely) 18 | * [Don't abstract prematurely](#dont-abstract-prematurely) 19 | * [Error cases are features](#error-cases-are-features) 20 | * [Document for others](#document-for-others) 21 | 22 | ## Don't compromise on quality 23 | 24 | Kibana is a mission-critical piece of software, but it's not a service we deploy and manage. When we ship a bug that causes instability, regression, or vulnerability, the issue will significantly impact production deployments for our users, and we will not be able to rollback the deployment to fix it. Even if an issue is severe enough to warrant an emergency patch, it will continue to hurt users in production until we release a fix (at least a week) and users upgrade to it (days to weeks). 25 | 26 | It's natural for a developer to want to get an important change merged into the product so it can be released, and it's good for our users to have access to frequent improvements in our releases, but we must not sacrifice the quality of our changes in order to meet these pressures. 27 | 28 | **Build small features of high quality, and then enhance those features with additional high quality improvements over multiple iterations. Never merge code that is not release-ready.** 29 | 30 | ## Do the right thing even when it's the hard thing 31 | 32 | To ensure Kibana remains reliable and relevant in the long term, we will occasionally need to make big changes to existing infrastructure and features where incremental progress is impractical. It takes a lot of hard work and time to get these changes right, and they may break behaviors that developers and end-users have come to rely on, but we must not be afraid to do them if they put the project in a better place. 33 | 34 | For changes like this, it's important to clearly define goals, establish scope, gain consensus, and communicate the changes effectively and transparently. 35 | 36 | The amount of work associated with a change should give us pause to reflect, but **we should not shy away from making big changes to the product when they're the correct solution.** 37 | 38 | ## Automate tests through CI 39 | 40 | Kibana is far too large a project for us to reliably find breakages through manual QA efforts. It would take years of developer time to manually test all of the features of Kibana in a single version, and even with a team of QA engineers and hundreds of thousands of lines of automated tests, it still takes months of cumulative developer time to manually QA the releases we have. 41 | 42 | It's imperative for the future success and stability of the project as well as for team velocity that all changes have extensive automated testing that give us confidence that ongoing churn on the codebase does not break existing functionality. We must achieve this level of testing without increasing the manual QA burden. 43 | 44 | Additionally, since all merged code should be release-ready, it's imperative that automated testing be run through our CI, preferably as blocking status checks on pull requests. 45 | 46 | **A feature that requires manual QA to ensure it continues to work is worse than not having that feature at all.** 47 | 48 | ## Organize by domain 49 | 50 | The Kibana codebase should be organized roughly into software domains, which is to say that modules are defined based on their role within the product rather than by the technical type of code that comprise them. The goal is to help developers find code quickly by placing services in intuitive locations. In practice the two highest technical levels we use to express domains are that of core and plugins, but even within these modules we seek to organize the code by more specific domains. 51 | 52 | Within core itself we express domains as services such as HTTP, config, elasticsearch, etc. These things are essential elements to constructing most useful features of Kibana, which is why they belong to the "core" domain. 53 | 54 | Each plugin is one top-level domain of the Kibana project, and will generally be constructed by many sub-domains within. In this way, plugins are modeled in the same spirit as core itself. 55 | 56 | Be cautious when adding new domains as doing so inherently expands the scope of the parent domain, which in the case of core or plugin boundaries means the scope of Kibana itself. 57 | 58 | **Code should be organized by domain up until you've reached a level of granularity in a module where it is no longer practical.** 59 | 60 | ## Prefer one way to do things 61 | 62 | When building APIs, whether they be programmatic or through HTTP, there are usually multitudes of use cases, and since it's natural as API authors to want to provide a convenient experience for consumers, this often results in a variety of different APIs that achieve similar results. Unfortunately, each API we create causes a burden in maintenance, testing, tooling, and documentation, and this burden is almost always greater than the work needed to use an alternative API. 63 | 64 | In order to achieve our goals of creating a consistent product experience as well as an efficient-to-maintain project, we should aim to leverage existing features, APIs, and standards rather than reinventing the wheel. **Unless it's part of a broader effort to change the way Kibana does something, it's best to leverage existing APIs and prefer existing conventions where possible.** 65 | 66 | ## Prioritize consistent code within Kibana 67 | 68 | Kibana was around before most of its dependencies existed, and we expect Kibana to be around after most of its current dependencies are no longer useful, relevant, or appropriate for the project. We must ensure long term sustainability by building Kibana independently of the popular patterns, tools, and idioms of the day. 69 | 70 | As with any codebase that evolves organically over time, Kibana has a fair share of inconsistency in style, patterns, and conventions, but we aim to reverse this trend in favor of a more consistent codebase. **To accomplish this, we must write new code in the spirit of conventions established by existing code, or we must seek consensus for introducing new styles, patterns, and conventions to Kibana.** 71 | 72 | We can leverage the cumulative knowledge of the broader developer community to iteratively improve the quality of our software, but the fundamental building blocks of Kibana are designed to support Kibana's needs as a project rather than the conveniences of popular tools and developer community sentiment. 73 | 74 | Whenever possible, automated tools should be used to aid in consistency (e.g. linting, prettier). 75 | 76 | ## TypeScript over JavaScript 77 | 78 | JavaScript is no longer the language of choice for Kibana, and any **JavaScript that gets transpiled in the project is considered technical debt. Our goal is near-complete replacement of JavaScript with TypeScript throughout the repo.** 79 | 80 | Static typing gives us more reliable mechanisms to do changes across the project, whether it be upgrading systemic dependencies, making changes to the file structure, or refactoring important modules that have many downstream dependencies. Static analysis allows for more robust tooling to help identify inter-module dependencies, security vulnerabilities, and orphaned code. 81 | 82 | Having one definitively preferred language helps developers navigate the broader project and work more freely across teams. 83 | 84 | ## Be wary of dependencies 85 | 86 | The Kibana project is not just the code we commit to the repo but rather the combined total of all of the source code from our own repo and all of the external dependencies we rely on. When a user encounters a deficiency in Kibana, it matters not whether the root cause is in code we've written or external code we depend on. Additionally, relying on a dependency is a considerable expense in terms of cognitive burden, maintenance overhead, and risk. 87 | 88 | Except for highly specialized functionality, dependencies often do more harm in the long term than their short term benefits justify. **Always be critical of new external dependencies being added to the project, and frequently re-evaluate the use of existing dependencies.** 89 | 90 | When the use of an external dependency is necessary, ensure there is sufficient integration testing in Kibana to ensure it continues to operate the way we'd expect when we change the consuming code in Kibana or upgrade the dependency code. 91 | 92 | Except in specific cases where widespread consensus was gained and clear ownership is established, third party dependencies should not be exposed directly as features of Kibana, whether it be through the UI, HTTP API, or programmatic interfaces. 93 | 94 | ## Don't share code prematurely 95 | 96 | There are many modules throughout Kibana's codebase where we have generic utilities that seem appropriate for use elsewhere in the codebase, but doing so is creating another public interface which has a cost in terms of maintenance, testing, documentation, and complexity that increases with each usage. Over the long term, shared utilities tend to accumulate additional complexity in order to be flexible enough for the various use cases of its consumers, particularly when they are shared across domain/plugin boundaries. 97 | 98 | **Just because some code can be reused does not mean that it should be reused.** Duplicating code has drawbacks, but the complexity scales linearly, which is easier for us to manage across so many teams in a project this size. 99 | 100 | When we have many duplicate implementations of the same code across different modules, then we can more effectively evaluate how similar the functions are in practice to determine whether creating a new public interface for the sake of code sharing is appropriate. 101 | 102 | ## Don't abstract prematurely 103 | 104 | Overuse of abstractions makes it harder to build a mental model about Kibana, and we've already encountered scenarios in Kibana's history where so many abstractions were in place it was difficult to maintain them in practice, and no one person understood how all the pieces worked together. 105 | 106 | At the same time, abstractions are necessary in a project like Kibana. We must be cautious about introducing new abstractions and ensure there is sufficient testing and documentation in place when we do create them. 107 | 108 | Every situation is different, but in general it's better to have abstractions emerge when multiple different implementations of the same service exist rather than in anticipation of that possibility in the future. It's much easier to introduce a new abstraction when we need it than it is to remove an unnecessary abstraction when we don't. For the same reason, it's important that we only write code that we intend to use instead of code that we think might be useful in the future. 109 | 110 | **Just because something can be abstracted does not mean that it should be abstracted.** 111 | 112 | ## Error cases are features 113 | 114 | When two or more modules interact with one another, and especially when two or more independent systems (i.e. integrations) interface with one another, a possible outcome is an error case. **We must build Kibana in such a way that errors behave consistently in documented ways that are verified through automated testing, which means these errors are features of Kibana.** 115 | 116 | In some scenarios the error cases are already codified by the module or service being consumed, and we simply need to handle those cases explicitly from the consuming side (e.g. HTTP response codes). In other cases, an error might occur as the result of an exception being thrown, in which case the consuming code should have a defined way to handle uncaught exceptions with the least impact to unrelated functionality as possible (e.g. render a generic error message in place of this component, return with a 500 error, etc). 117 | 118 | Other than in exceptionally rare cases, errors should never be swallowed, and error messages should usually be descriptive and actionable. 119 | 120 | ## Document for others 121 | 122 | The primary consumers of the code we write, the APIs that we create, and the features we ship are people other than ourselves, which means the people using our tools do not share the cumulative domain knowledge and expertise that crafted them. As authors of these tools, we must share our knowledge in the form of documentation. 123 | 124 | Features that we anticipate end users, admins, and plugin developers consuming should be documented through our official docs, but module-level READMEs and code comments are also appropriate. 125 | 126 | **Documentation is critical part of developing features and code, so an undocumented feature is an incomplete feature.** 127 | -------------------------------------------------------------------------------- /development_constitution.md: -------------------------------------------------------------------------------- 1 | Elasticsearch Team Development Constitution 2 | 3 | # Preamble 4 | 5 | We, the team of Elasticsearch core developers, want to move as fast as we can toward a system that is reliable, robust, secure, scalable, and straightforward to use. We want to strive for innovation, replace legacy constructs and features, remove fragile code, and work toward a better user experience while keeping our users onboard with our rapid changes. 6 | 7 | It’s crucial for us to have a shared vision of where the team is heading and maybe even more importantly why the team is going down a certain path. When Elasticsearch was the *new kid on the block* it shone with endless flexibility, ease of use, and rich APIs. We formed a company around that young kid and suddenly its user base shot through the roof. The support organization could barely keep up with the growing number of customers, which is a good problem to have. Yet as the number of users grew, so did the chance of things going sideways, unfortunately much more quickly than we could ever hire support engineers. We learned that much of the flexibility came from leniency, from features that worked in most cases but not all. For instance, having scripts that users can send with the request is basically a remote code execution engine and if it goes wrong it’s fatal. Even the most basic features, like settings, were very flexible but enormously fragile. Specifying a number without a unit was perfectly fine except that many users didn’t know what the default unit was. We just tried to do the right thing, which turned out to not be the right thing all the time. 8 | 9 | Today we are in a different position. Our user base is much larger than it was in 2013 but our support organization hasn’t grown at the same rate. Yes, we handle an order of magnitude more support cases than in 2013, but this would not have been possible with the system we had back then. Now we’ve moved from a fragile but flexible system toward software that is narrower in scope. We have defined many more boundaries: stricter input validation, a security model that allows us fine grained control over permissions, and even a plugin model that provides great flexibility to add riskier features. 10 | 11 | But hold on, we are not there yet! There are still endless problems that can have fatal consequences. Aggregations can blow up servers with a single request. Users feel the need to run Elasticsearch with 30+ GB heaps. We are still offering 27 different ways of specifying a boolean value. And this list continues… 12 | 13 | We have a massive responsibility to our users, the support organization, the cloud hosting teams, and third party providers to offer a system that is reliable, robust, secure, and straightforward to use. For this reason we all should strive for innovation, replace legacy constructs and features, remove fragile code, and improve the user experience. Our advantage over other companies is our innovation, and innovation requires velocity. We must move and embrace change to innovate without leaving the user behind. 14 | 15 | The following sections are a collection of principles and guidelines for designing, refactoring, or removing code from the Elasticsearch codebase. These points are unordered and mostly uncategorized and should be seen as a constitution of software development within the Elasticsearch team. 16 | 17 | # Designing features 18 | 19 | * *Progress over perfection.* We have followed this approach for many years now which allows us to make large changes over time without big bang commits originating from massive pull requests. For example, the completion suggester was added in the early days of Elasticsearch without support for realtime updates and specifically deletes. This means that deleting a document in Elasticsearch wasn’t immediately reflected in the suggestions. It was a hard problem at the time and about three years later we added support for bitset filters to the Lucene suggester as well as to Elasticsearch. Meanwhile it’s been an acceptable solution for many users out there, with many bugs fixed and the evolution toward a document based suggester. It was all about progress over perfection. 20 | 21 | * *Design for today! Use abstractions with care.* Computer Science professors teach students to make extensive use of abstraction layers in the name of flexibility and information hiding. Certainly Elasticsearch makes extensive use of abstractions; no project involving several million lines of code could do otherwise and survive. But experience has shown that excessive or premature abstraction can be just as harmful as premature optimization. Abstraction should be used to the level required and no further. 22 | 23 | As a simple exercise, consider a function which has an argument which is always passed as zero by all callers. One could retain that argument just in case somebody eventually needs to use the extra flexibility that it provides. By that time, though, chances are good that the code never noticed — because it has never been used. Or when the need for extra flexibility arises, it does not do so in a way which matches the programmer's early expectation. We should routinely submit patches to remove unused arguments; in general they should not be added in the first place. (adapted from [the Linux kernel's dev process](https://www.kernel.org/doc/html/latest/process/4.Coding.html)) 24 | 25 | * *Start simple; don’t be smart.* Everybody wants to write code that is top-notch, cutting edge, fast as hell, robust to the end of days, elegant, and efficient. Unfortunately, this doesn’t happen overnight. As with everything we’ve got to learn to walk before we can run. A new feature should start in the simplest way possible. Even if it’s desirable to prevent moving all shards to shrink an index from 8 to 4 shards, it’s better to have a solid shared infrastructure that requires all shards to be on the same node. Special allocation logic can come later, in subsequent releases. 26 | 27 | * *Beware: removing code is hard.* Removing even the smallest feature can be extremely hard. Be aware of this when you add code to the code base and choose wisely what gets added. We might need to stick with it for years or break many users out there when we attempt to remove it. 28 | 29 | * *Strict, unambiguous, reliable, and simple.* Elasticsearch has a history of adding lenient, ambiguous, unreliable, and complex options. The direction along those lines has changed a while ago. At first glance it seems to help with being friendly to users but it comes with a high price. It comes with a combinatorial explosion of options and code paths which are not tested and which hide bugs. A perfect example is the gazillion options to specify a boolean value. Someone would think it’s as simple as comparing the value to the string "true" or “false” and if it doesn’t match we throw an exception. No, it accepts the values “false”, “0”, “no”, and “off” and if it doesn’t match any of those it’s interpreted as “true”. What could possibly go wrong? If you add code, do it in the simplest way possible that is also strict, unambiguous, and reliable. 30 | 31 | * *Stick to core responsibilities.* It is crucial to our system that we build *solid* and *reliable* features. To do this, we need to remember our core responsibilities as a distributed scalable search engine. For example, we used to provide a limited web server called site-plugins, but it didn’t represent our core responsibilities, so we removed it. When features align with making Elasticsearch a better distributed scalable search engine, it becomes more solid and reliable. (The same principle applies to all our products.) 32 | 33 | * *You are the expert; act like it.* Elasticsearch has become popular. The user base is huge and growing exponentially. The subset of power users is shrinking such that one of our core responsibilities now is to streamline API usage and reduce the risk of "shooting yourself in the foot". Our core APIs offer a lot of flexibility which makes them easy to misuse. The result is often slow performance, cluster outages, and wrong results. Going forward, we should use our experience and deeper knowledge of the system to prevent these pitfalls. Build APIs and features that do their one thing very well. Don’t design it to be a workaround for other problems. 34 | 35 | * *Build features in isolation.* Always prefer adding a feature as a plugin rather than adding the feature to core. The best way to make clear APIs and extension points is by using them. To arrive at a maintainable core we have to keep it lean. Our plugin model allows classloader isolation as well as dedicated permissions to third party components. An isolated implementation is always preferable. If it needs to ship with the distribution it can be a module. 36 | 37 | * *Remove first, fix later.* Often the removal of dangerous or trappy features stalls because there is no replacement for its functionality. We will remove those features and prioritize reimplementing them as though they were new features. If the features were important we’ll make the reimplementation a blocker for the next release. If they weren’t it might be several releases until they are reimplemented. Or they might never be reimplemented, becoming a relic of time long gone by. Removing dangerous features is crucial to the success of the organization. For instance, *delete-by-query* was repeatedly causing massive outages that took days to debug and fix. Its removal has potentially saved us a large amount of money and time that we didn’t spend with our customers. Now given the fact that our user base is growing fast, it’s our responsibility to make the right decision for our users even if that decision is not popular. The *remove first* approach is mandatory when it comes to security, cluster stability, and data corruption. 38 | 39 | * *Be fast by default; slow is optional.* Performance is key in our business. Slow is considered esoteric. This is a very difficult topic since for instance O(n) can be acceptable with 10k documents but not for 10m. A perfect example is `*_source*` access in scripts. There are some scripts (search for instance) that should simply not allow the access of `*_source*` since it loads the JSON source of every scored document. That is a round-trip to disk per doc the script is executed for. Features like this must be disabled by default or should not be available on performance critical parts of the system. There is always the argument of prototyping, small documents sets, little webshops etc. Yet our message here should focus on reindexing and future improvements to our defaults to make these APIs obsolete. 40 | 41 | * *Focus on upgrade experience.* With time-based release the upgrade experience is crucial for us since we want users to move to new releases ASAP. We had several problems in the past with breaking too much and users suffered from long cluster restarts. Our feature development and ideas for improvement should focus on smoothing the path forward. 42 | 43 | * *Break on majors, not on minors.* Breaking changes must be done on major releases. We try to scope the changes to not break too much at the same time especially if we are getting closer to the GA date. It’s OK to push out a breaking change to the next major if we are close. 44 | 45 | * *Test bottom up.* If you write code, write unit tests first. Write many of them. Write code so you can write many of them. Integration testing is the last step. Focus on adding more tests that execute fast and are easy to debug, like unit tests. This is crucial for developer velocity. 46 | 47 | * *Consider Java APIs expert APIs*. With the exception of the HTTP-Client and its dedicated APIs, all APIs in Elasticsearch, all extension points and plugins are expert APIs. Accordingly, expert users can handle API changes and removals. The most reliable way to make sure folks don’t use deprecated APIs is to remove them. Don’t hesitate, especially when it’s an internal API. Non-expert users should always go through a REST interface. 48 | 49 | * *Be critical, doubt all the code, and embrace mistakes.* Everybody writes code that must be fixed, refactored, or removed at some point. But most of the time the effective half-life of code is pretty short. Add comments describing why things are done in a certain way. We can never know the full extent of a problem or all the use cases when we develop a feature. 50 | 51 | When someone criticizes the code, they are not criticizing you, so don’t take it personally. Help them understand why you wrote it that way. When someone rewrites something you wrote, it’s not a rejection of your ideas. At one point Mike pushed a change to Lucene that Adrien obsoleted just two days later! It’s great when other people take interest in code you wrote: it means the code is alive. See improvements to your code as that code becoming a growing, thriving being. 52 | 53 | Don’t fear making a mistake and, more importantly, don’t let fear paralyze you from adding something that might not be totally right. See mistakes and failures as *feedback*, *discovery* and *knowledge* that can make our product better. 54 | 55 | * *Don’t be afraid of big changes.* Often the solution to a problem is hard. The hardest part is to solve it correctly. It comes together with tons of work, risk, and changes in the system that will affect others — mostly users. Prefer incremental changes (see *Progress over perfection*), but be willing to make big changes in big chunks where incremental change is impossible. 56 | 57 | * *Don’t be afraid to say no.* Elasticsearch is at a point where it can’t accept every change. If we tried to make everybody happy we would stall and be paralyzed. There are certain things that just don’t work with a system like Elasticsearch. Think of *[joins](https://github.com/elastic/elasticsearch/pull/3278)* or *real type isolation*. To these we have to say *thanks, but no thanks!* 58 | 59 | * *Only accept features that scale.* We often get feature requests that would work fine in the single-node case (e.g., joins or precise cardinality aggregations) but would be a disaster given the distributed nature of Elasticsearch. These feature requests should always be rejected because they violate our core Elasticsearch responsibilities of scalability and distributed nature. In principle we do not add features that only work in the single-node case. 60 | 61 | * *Always start with a dream.* It’s wise to start with an idea of what a perfect solution would look like, even if it involves backward breaks or removing core features and replacing them later with better solutions. Sometimes it’s possible to implement the ideal even if it takes time. By the same token, it’s wise to think about the simplest possible solution, and in many cases the *biggest bang for the buck* is pretty close to the simplest solution. 62 | 63 | * *Focus on error reporting.* In software development lots of things are binary. If it didn’t work as expected it should fail fast and hard. Focus on good error reporting; avoid swallowing exceptions, rather declare checked exceptions and forcing the caller to handle the situation, Exceptions are part of a method’s contract! Guard preconditions for methods with actual checks. If we don’t know or document what exceptions can happen, and when, then we have no idea really how the method behaves. Add checks/javadocs and try to make this better. Look at the JDK code for examples of this, even Lucene code which we use as a reference often is not a good example here. Understand how this can make your code faster, e.g. checking for array index up front in the method is not only more clear, it fails fast and hard, and can also eliminate bounds checks ("dominating test"). When reporting errors, ask yourself a) what message you would want to see if you are debugging a problem, and b) what information would give the operational production-support team enough insight to diagnose problems. 64 | 65 | * *Document the code.* You may think your code is obvious, but it may not be. Give a high level overview of it to someone who is unfamiliar with those thousands of lines of code, such that you both would be able to divide and conquer. Document a summary of what things do at package, class, and method level. If you think your own code is tricky or hairy, do even more to try to compensate. Long living code is written once but read and reread many times! 66 | 67 | * *Private by default.* Java’s access levels are a good way to encapsulate code: separate the interface/contract from the implementation detail. Private is best, package-private is good, public is last resort. Be careful about what you expose, so that your class or API is simple easy to use. 68 | 69 | * *Every change deserves review.* Our system is complex and every change can have potential side effects. We expect everyone to work hard and think things through but there will be times when implications are missed. Every change should be proposed and receive at least one LGTM. For complex changes, two reviewers are better. On some teams two is the minimum number of LGTMs, three for complex changes. Coders and reviewers share responsibility for failures associated with a change; this encourages careful review. Sometimes a feature fails unexpectedly because something it depends on has changed or broken. We should all take ownership for failures and unexpected problems for customers, rather than blaming a few people. 70 | 71 | * *Rules are meant to be broken.* Sometimes the code has to break the rules. Maybe it’s five times easier to understand if your comment contains a table 150 characters wide. Nasty abstractions to try to enforce "DRY" can end out far worse than a simple duplication of code! 72 | 73 | # Interacting with people 74 | 75 | * *Voice your opinion with precision and respect.* Always share what you have to say but leave room for another opinion. Always explain your reasons. Ultimatums kill conversation. Phrases like "This will never work" and “This is stupid” are lazy and imprecise. Say "I think this will be problematic in the case that... because..." Don’t say "This is wrong" rather "I think this is wrong because..." Don’t say "Is this really needed?"; ask "*Why* is this needed?" Don’t say "I am not open to anything else" or "There is nothing to discuss." See the point about vetoes instead. 76 | 77 | * *Be kind.* The written form is hard. What you intend with your words may not be obvious to the reader. Make the effort to explain your reasoning clearly. Be quick to apologise if you haven’t made a good job of explaining. Assume misunderstanding rather than malice. When in any doubt about communicating your idea, get on video or voice chat. 78 | 79 | From [Being Kind](http://boz.com/articles/be-kind.html): 80 | 81 | *Being kind is fundamentally about taking responsibility for your impact on the people around you. It requires you be mindful of their feelings and considerate of the way your presence affects them. It isn’t the same as being nice. It isn’t about superficial praise. It doesn’t mean dulling your opinions. And it shouldn’t diminish the passion with which you present them.* 82 | 83 | * *Thank people.* Say it when someone has done a good job. Take the time to include some detail on why you think it was a good job to make it sincere and specific. 84 | 85 | (Beware of the bad audio of the recording: [How to Delegate, Like a Boss](https://www.youtube.com/watch?v=h3MPewsk5PU&t=5m55s)) 86 | 87 | * *With power comes responsibility.* You have the power to veto. A veto or in other words a `-1` is a strong call. Only use it as a last resort when you are 100% convinced that a certain change should not be made. Don’t use it if you only disagree or do not like a change. **Beware that a veto will kill progress of the issue/change and won’t be overruled unless retracted**, so be conscious of the seriousness of a veto. 88 | 89 | Use wise words to explain your objection. Your veto must come with a technical reason. Be prepared to discuss and explain. The vetoed change is guaranteed to be seen as good by the persons who proposed it and they deserve that discussion. Of course, they also deserve the chance to convince you of their reason. In that past, such healthy discussion actually ended with the veto being pulled back. 90 | 91 | * *Empathy for passion.* Some of your coworkers have unlimited passion. Unfortunately it doesn’t always come with unlimited patience. It’s always good to change communication channels if discussions go sideways. Face to face communication is crucial in such a development environment. For example, some of our code cleanup efforts last several months. Think of settings refactoring or removing google guice. If you argue on an issue, keep in mind that the other person might have spent months on this and you may be missing something in the big picture. It’s enormously hard to ensure both people are on the same page! If in doubt do zoom on slack or go on aOn. 92 | 93 | * *Empathy for the pressured.* You will face situations where an argument goes sideways. You will see situations where people do not use a polite voice. Don’t accept it but try to look behind the scenes, speak about it, and forgive. 94 | 95 | * *Report abusive comments to our Code of Conduct team.* If you see abusive comments, even if you are not part of the conversation, please report it (push back). Don’t fuel a discussion, and don’t provide a forum for more abusive comments. End the discussion and notify others to back you up. If you don’t feel like doing this, it’s fine to ping others directly to do this for you. Use github emoji (and reaction emoji) to amplify pushback from others. Then move on with technical arguments. Be the bigger personality and help others to reduce the impact of abusive or aggressive comment. 96 | 97 | * *Ask questions if in doubt.* There are many complex areas in Elasticsearch. If you are in doubt or you are not sure how to fix a certain problem, if you don’t even know how to approach the problem or you are stuck, go and get help. Having zoom sessions to explain the problem to others might even help in realizing that the solution to the problem is a totally different one. 98 | 99 | * *Solve conflicts by speaking to each other. Accept decisions, even if not yours and move on.* We are all passionate and opinionated people. This is what makes us good at our job and moves the code forward. It also means we will not always agree. Talk things through and try to see the other side. Almost always there is also a third way that will make both parties happy. In the worst case, there will be times that consensus is not reached and leadership has to make a call. [Disagree and commit](https://www.amazon.jobs/en/principles). Remember: nothing is final and things can be changed if they have been proven wrong. 100 | 101 | Inspired by: 102 | 103 | [Zen of Python](https://en.wikipedia.org/wiki/Zen_of_Python) 104 | 105 | [Contributor Covenant](http://contributor-covenant.org/) 106 | 107 | [Amazon’s Leadership Principles](https://www.amazon.jobs/principles) 108 | 109 | [Rust’s Code of Conduct](https://www.rust-lang.org/en-US/conduct.html), [Rust video on Conduct](https://youtu.be/dIageYT0Vgg?t=7m2s) 110 | 111 | 112 | 113 | --------------------------------------------------------------------------------