├── .gitignore ├── CONTRIBUTING.md ├── LICENSE ├── Makefile ├── README.md ├── docs ├── README.md ├── developer-guide │ └── README.md ├── mantav2.md ├── operator-guide │ ├── README.md │ ├── architecture.md │ ├── deployment.md │ ├── maintenance.md │ └── mantav2-migration.md ├── sample-training.md └── user-guide │ ├── .htaccess │ ├── README.md │ ├── commands-reference.md │ ├── rbac.md │ ├── sdks.md │ └── storage-reference.md ├── package-lock.json ├── package.json └── tools └── jr-manifest.json /.gitignore: -------------------------------------------------------------------------------- 1 | /tmp 2 | /build 3 | /docs/operator-guide.html 4 | /docs/operator-guide.json 5 | /docs/user-guide/examples/*/job.id 6 | /docs/user-guide/examples/*/job.err 7 | /docs/user-guide/examples/*/job.out 8 | /docs/user-guide/examples/*/index.md 9 | /node_modules 10 | .DS_Store 11 | .idea/ 12 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | 6 | 7 | 11 | 12 | # Manta Contribution Guidelines 13 | 14 | Thanks for using Manta and for considering contributing to it! 15 | 16 | # Code 17 | 18 | All changes to Manta project repositories go through code review via a GitHub 19 | pull request. 20 | 21 | If you're making a substantial change, you probably want to contact developers 22 | [on the mailing list or IRC](README.md#community) first. If you have any trouble 23 | with the contribution process, please feel free to contact developers [on the 24 | mailing list or IRC](README.md#community). 25 | 26 | See the [developer guide](docs/developer-guide) for useful information about 27 | building and testing the software. 28 | 29 | Manta repositories use the same [Joyent Engineering 30 | Guidelines](https://github.com/TritonDataCenter/eng/blob/master/docs/index.md) as 31 | the Triton project. Notably: 32 | 33 | * The #master branch should be first-customer-ship (FCS) quality at all times. 34 | Don't push anything until it's tested. 35 | * All repositories should be "make check" clean at all times. 36 | * All repositories should have tests that run cleanly at all times. 37 | 38 | Typically each repository has `make check` to lint and check code style. 39 | Specific code style can vary by repository. 40 | 41 | ## Issues 42 | 43 | There are two separate issue trackers that are relevant for Manta code: 44 | 45 | * An internal JIRA instance. 46 | 47 | A JIRA ticket has an ID like `MANTA-380`, where "MANTA" is the JIRA project 48 | name. A read-only view of many JIRA tickets is made available at 49 | (e.g. 50 | ). 51 | 52 | * GitHub issues for the relevant repository. 53 | 54 | Before Manta was open sourced, Joyent engineering used a private JIRA instance. 55 | While Joyent continues to use JIRA internally, we also use GitHub issues for 56 | tracking -- primarily to allow interaction with those without access to JIRA. 57 | 58 | 59 | ## Code of Conduct 60 | 61 | All persons and/or organizations contributing to, or interacting with our 62 | repositories or communities are required to abide by the 63 | [illumos Code of Conduct][coc]. 64 | 65 | [coc]: https://github.com/TritonDataCenter/illumos-joyent/blob/master/CODE_OF_CONDUCT.md 66 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Mozilla Public License Version 2.0 2 | ================================== 3 | 4 | 1. Definitions 5 | -------------- 6 | 7 | 1.1. "Contributor" 8 | means each individual or legal entity that creates, contributes to 9 | the creation of, or owns Covered Software. 10 | 11 | 1.2. "Contributor Version" 12 | means the combination of the Contributions of others (if any) used 13 | by a Contributor and that particular Contributor's Contribution. 14 | 15 | 1.3. "Contribution" 16 | means Covered Software of a particular Contributor. 17 | 18 | 1.4. "Covered Software" 19 | means Source Code Form to which the initial Contributor has attached 20 | the notice in Exhibit A, the Executable Form of such Source Code 21 | Form, and Modifications of such Source Code Form, in each case 22 | including portions thereof. 23 | 24 | 1.5. "Incompatible With Secondary Licenses" 25 | means 26 | 27 | (a) that the initial Contributor has attached the notice described 28 | in Exhibit B to the Covered Software; or 29 | 30 | (b) that the Covered Software was made available under the terms of 31 | version 1.1 or earlier of the License, but not also under the 32 | terms of a Secondary License. 33 | 34 | 1.6. "Executable Form" 35 | means any form of the work other than Source Code Form. 36 | 37 | 1.7. "Larger Work" 38 | means a work that combines Covered Software with other material, in 39 | a separate file or files, that is not Covered Software. 40 | 41 | 1.8. "License" 42 | means this document. 43 | 44 | 1.9. "Licensable" 45 | means having the right to grant, to the maximum extent possible, 46 | whether at the time of the initial grant or subsequently, any and 47 | all of the rights conveyed by this License. 48 | 49 | 1.10. "Modifications" 50 | means any of the following: 51 | 52 | (a) any file in Source Code Form that results from an addition to, 53 | deletion from, or modification of the contents of Covered 54 | Software; or 55 | 56 | (b) any new file in Source Code Form that contains any Covered 57 | Software. 58 | 59 | 1.11. "Patent Claims" of a Contributor 60 | means any patent claim(s), including without limitation, method, 61 | process, and apparatus claims, in any patent Licensable by such 62 | Contributor that would be infringed, but for the grant of the 63 | License, by the making, using, selling, offering for sale, having 64 | made, import, or transfer of either its Contributions or its 65 | Contributor Version. 66 | 67 | 1.12. "Secondary License" 68 | means either the GNU General Public License, Version 2.0, the GNU 69 | Lesser General Public License, Version 2.1, the GNU Affero General 70 | Public License, Version 3.0, or any later versions of those 71 | licenses. 72 | 73 | 1.13. "Source Code Form" 74 | means the form of the work preferred for making modifications. 75 | 76 | 1.14. "You" (or "Your") 77 | means an individual or a legal entity exercising rights under this 78 | License. For legal entities, "You" includes any entity that 79 | controls, is controlled by, or is under common control with You. For 80 | purposes of this definition, "control" means (a) the power, direct 81 | or indirect, to cause the direction or management of such entity, 82 | whether by contract or otherwise, or (b) ownership of more than 83 | fifty percent (50%) of the outstanding shares or beneficial 84 | ownership of such entity. 85 | 86 | 2. License Grants and Conditions 87 | -------------------------------- 88 | 89 | 2.1. Grants 90 | 91 | Each Contributor hereby grants You a world-wide, royalty-free, 92 | non-exclusive license: 93 | 94 | (a) under intellectual property rights (other than patent or trademark) 95 | Licensable by such Contributor to use, reproduce, make available, 96 | modify, display, perform, distribute, and otherwise exploit its 97 | Contributions, either on an unmodified basis, with Modifications, or 98 | as part of a Larger Work; and 99 | 100 | (b) under Patent Claims of such Contributor to make, use, sell, offer 101 | for sale, have made, import, and otherwise transfer either its 102 | Contributions or its Contributor Version. 103 | 104 | 2.2. Effective Date 105 | 106 | The licenses granted in Section 2.1 with respect to any Contribution 107 | become effective for each Contribution on the date the Contributor first 108 | distributes such Contribution. 109 | 110 | 2.3. Limitations on Grant Scope 111 | 112 | The licenses granted in this Section 2 are the only rights granted under 113 | this License. No additional rights or licenses will be implied from the 114 | distribution or licensing of Covered Software under this License. 115 | Notwithstanding Section 2.1(b) above, no patent license is granted by a 116 | Contributor: 117 | 118 | (a) for any code that a Contributor has removed from Covered Software; 119 | or 120 | 121 | (b) for infringements caused by: (i) Your and any other third party's 122 | modifications of Covered Software, or (ii) the combination of its 123 | Contributions with other software (except as part of its Contributor 124 | Version); or 125 | 126 | (c) under Patent Claims infringed by Covered Software in the absence of 127 | its Contributions. 128 | 129 | This License does not grant any rights in the trademarks, service marks, 130 | or logos of any Contributor (except as may be necessary to comply with 131 | the notice requirements in Section 3.4). 132 | 133 | 2.4. Subsequent Licenses 134 | 135 | No Contributor makes additional grants as a result of Your choice to 136 | distribute the Covered Software under a subsequent version of this 137 | License (see Section 10.2) or under the terms of a Secondary License (if 138 | permitted under the terms of Section 3.3). 139 | 140 | 2.5. Representation 141 | 142 | Each Contributor represents that the Contributor believes its 143 | Contributions are its original creation(s) or it has sufficient rights 144 | to grant the rights to its Contributions conveyed by this License. 145 | 146 | 2.6. Fair Use 147 | 148 | This License is not intended to limit any rights You have under 149 | applicable copyright doctrines of fair use, fair dealing, or other 150 | equivalents. 151 | 152 | 2.7. Conditions 153 | 154 | Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted 155 | in Section 2.1. 156 | 157 | 3. Responsibilities 158 | ------------------- 159 | 160 | 3.1. Distribution of Source Form 161 | 162 | All distribution of Covered Software in Source Code Form, including any 163 | Modifications that You create or to which You contribute, must be under 164 | the terms of this License. You must inform recipients that the Source 165 | Code Form of the Covered Software is governed by the terms of this 166 | License, and how they can obtain a copy of this License. You may not 167 | attempt to alter or restrict the recipients' rights in the Source Code 168 | Form. 169 | 170 | 3.2. Distribution of Executable Form 171 | 172 | If You distribute Covered Software in Executable Form then: 173 | 174 | (a) such Covered Software must also be made available in Source Code 175 | Form, as described in Section 3.1, and You must inform recipients of 176 | the Executable Form how they can obtain a copy of such Source Code 177 | Form by reasonable means in a timely manner, at a charge no more 178 | than the cost of distribution to the recipient; and 179 | 180 | (b) You may distribute such Executable Form under the terms of this 181 | License, or sublicense it under different terms, provided that the 182 | license for the Executable Form does not attempt to limit or alter 183 | the recipients' rights in the Source Code Form under this License. 184 | 185 | 3.3. Distribution of a Larger Work 186 | 187 | You may create and distribute a Larger Work under terms of Your choice, 188 | provided that You also comply with the requirements of this License for 189 | the Covered Software. If the Larger Work is a combination of Covered 190 | Software with a work governed by one or more Secondary Licenses, and the 191 | Covered Software is not Incompatible With Secondary Licenses, this 192 | License permits You to additionally distribute such Covered Software 193 | under the terms of such Secondary License(s), so that the recipient of 194 | the Larger Work may, at their option, further distribute the Covered 195 | Software under the terms of either this License or such Secondary 196 | License(s). 197 | 198 | 3.4. Notices 199 | 200 | You may not remove or alter the substance of any license notices 201 | (including copyright notices, patent notices, disclaimers of warranty, 202 | or limitations of liability) contained within the Source Code Form of 203 | the Covered Software, except that You may alter any license notices to 204 | the extent required to remedy known factual inaccuracies. 205 | 206 | 3.5. Application of Additional Terms 207 | 208 | You may choose to offer, and to charge a fee for, warranty, support, 209 | indemnity or liability obligations to one or more recipients of Covered 210 | Software. However, You may do so only on Your own behalf, and not on 211 | behalf of any Contributor. You must make it absolutely clear that any 212 | such warranty, support, indemnity, or liability obligation is offered by 213 | You alone, and You hereby agree to indemnify every Contributor for any 214 | liability incurred by such Contributor as a result of warranty, support, 215 | indemnity or liability terms You offer. You may include additional 216 | disclaimers of warranty and limitations of liability specific to any 217 | jurisdiction. 218 | 219 | 4. Inability to Comply Due to Statute or Regulation 220 | --------------------------------------------------- 221 | 222 | If it is impossible for You to comply with any of the terms of this 223 | License with respect to some or all of the Covered Software due to 224 | statute, judicial order, or regulation then You must: (a) comply with 225 | the terms of this License to the maximum extent possible; and (b) 226 | describe the limitations and the code they affect. Such description must 227 | be placed in a text file included with all distributions of the Covered 228 | Software under this License. Except to the extent prohibited by statute 229 | or regulation, such description must be sufficiently detailed for a 230 | recipient of ordinary skill to be able to understand it. 231 | 232 | 5. Termination 233 | -------------- 234 | 235 | 5.1. The rights granted under this License will terminate automatically 236 | if You fail to comply with any of its terms. However, if You become 237 | compliant, then the rights granted under this License from a particular 238 | Contributor are reinstated (a) provisionally, unless and until such 239 | Contributor explicitly and finally terminates Your grants, and (b) on an 240 | ongoing basis, if such Contributor fails to notify You of the 241 | non-compliance by some reasonable means prior to 60 days after You have 242 | come back into compliance. Moreover, Your grants from a particular 243 | Contributor are reinstated on an ongoing basis if such Contributor 244 | notifies You of the non-compliance by some reasonable means, this is the 245 | first time You have received notice of non-compliance with this License 246 | from such Contributor, and You become compliant prior to 30 days after 247 | Your receipt of the notice. 248 | 249 | 5.2. If You initiate litigation against any entity by asserting a patent 250 | infringement claim (excluding declaratory judgment actions, 251 | counter-claims, and cross-claims) alleging that a Contributor Version 252 | directly or indirectly infringes any patent, then the rights granted to 253 | You by any and all Contributors for the Covered Software under Section 254 | 2.1 of this License shall terminate. 255 | 256 | 5.3. In the event of termination under Sections 5.1 or 5.2 above, all 257 | end user license agreements (excluding distributors and resellers) which 258 | have been validly granted by You or Your distributors under this License 259 | prior to termination shall survive termination. 260 | 261 | ************************************************************************ 262 | * * 263 | * 6. Disclaimer of Warranty * 264 | * ------------------------- * 265 | * * 266 | * Covered Software is provided under this License on an "as is" * 267 | * basis, without warranty of any kind, either expressed, implied, or * 268 | * statutory, including, without limitation, warranties that the * 269 | * Covered Software is free of defects, merchantable, fit for a * 270 | * particular purpose or non-infringing. The entire risk as to the * 271 | * quality and performance of the Covered Software is with You. * 272 | * Should any Covered Software prove defective in any respect, You * 273 | * (not any Contributor) assume the cost of any necessary servicing, * 274 | * repair, or correction. This disclaimer of warranty constitutes an * 275 | * essential part of this License. No use of any Covered Software is * 276 | * authorized under this License except under this disclaimer. * 277 | * * 278 | ************************************************************************ 279 | 280 | ************************************************************************ 281 | * * 282 | * 7. Limitation of Liability * 283 | * -------------------------- * 284 | * * 285 | * Under no circumstances and under no legal theory, whether tort * 286 | * (including negligence), contract, or otherwise, shall any * 287 | * Contributor, or anyone who distributes Covered Software as * 288 | * permitted above, be liable to You for any direct, indirect, * 289 | * special, incidental, or consequential damages of any character * 290 | * including, without limitation, damages for lost profits, loss of * 291 | * goodwill, work stoppage, computer failure or malfunction, or any * 292 | * and all other commercial damages or losses, even if such party * 293 | * shall have been informed of the possibility of such damages. This * 294 | * limitation of liability shall not apply to liability for death or * 295 | * personal injury resulting from such party's negligence to the * 296 | * extent applicable law prohibits such limitation. Some * 297 | * jurisdictions do not allow the exclusion or limitation of * 298 | * incidental or consequential damages, so this exclusion and * 299 | * limitation may not apply to You. * 300 | * * 301 | ************************************************************************ 302 | 303 | 8. Litigation 304 | ------------- 305 | 306 | Any litigation relating to this License may be brought only in the 307 | courts of a jurisdiction where the defendant maintains its principal 308 | place of business and such litigation shall be governed by laws of that 309 | jurisdiction, without reference to its conflict-of-law provisions. 310 | Nothing in this Section shall prevent a party's ability to bring 311 | cross-claims or counter-claims. 312 | 313 | 9. Miscellaneous 314 | ---------------- 315 | 316 | This License represents the complete agreement concerning the subject 317 | matter hereof. If any provision of this License is held to be 318 | unenforceable, such provision shall be reformed only to the extent 319 | necessary to make it enforceable. Any law or regulation which provides 320 | that the language of a contract shall be construed against the drafter 321 | shall not be used to construe this License against a Contributor. 322 | 323 | 10. Versions of the License 324 | --------------------------- 325 | 326 | 10.1. New Versions 327 | 328 | Mozilla Foundation is the license steward. Except as provided in Section 329 | 10.3, no one other than the license steward has the right to modify or 330 | publish new versions of this License. Each version will be given a 331 | distinguishing version number. 332 | 333 | 10.2. Effect of New Versions 334 | 335 | You may distribute the Covered Software under the terms of the version 336 | of the License under which You originally received the Covered Software, 337 | or under the terms of any subsequent version published by the license 338 | steward. 339 | 340 | 10.3. Modified Versions 341 | 342 | If you create software not governed by this License, and you want to 343 | create a new license for such software, you may create and use a 344 | modified version of this License if you rename the license and remove 345 | any references to the name of the license steward (except to note that 346 | such modified license differs from this License). 347 | 348 | 10.4. Distributing Source Code Form that is Incompatible With Secondary 349 | Licenses 350 | 351 | If You choose to distribute Source Code Form that is Incompatible With 352 | Secondary Licenses under the terms of this version of the License, the 353 | notice described in Exhibit B of this License must be attached. 354 | 355 | Exhibit A - Source Code Form License Notice 356 | ------------------------------------------- 357 | 358 | This Source Code Form is subject to the terms of the Mozilla Public 359 | License, v. 2.0. If a copy of the MPL was not distributed with this 360 | file, You can obtain one at http://mozilla.org/MPL/2.0/. 361 | 362 | If it is not possible or desirable to put the notice in a particular 363 | file, then You may include the notice in a location (such as a LICENSE 364 | file in a relevant directory) where a recipient would be likely to look 365 | for such a notice. 366 | 367 | You may add additional accurate notices of copyright ownership. 368 | 369 | Exhibit B - "Incompatible With Secondary Licenses" Notice 370 | --------------------------------------------------------- 371 | 372 | This Source Code Form is "Incompatible With Secondary Licenses", as 373 | defined by the Mozilla Public License, v. 2.0. 374 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | # 2 | # This Source Code Form is subject to the terms of the Mozilla Public 3 | # License, v. 2.0. If a copy of the MPL was not distributed with this 4 | # file, You can obtain one at http://mozilla.org/MPL/2.0/. 5 | # 6 | # Copyright 2020 Joyent, Inc. 7 | # 8 | 9 | # 10 | # Repo-specific targets 11 | # 12 | .PHONY: all 13 | all: docs 14 | 15 | ./node_modules/.bin/doctoc: 16 | npm install 17 | 18 | # Make a table of contents in Markdown docs that are setup to use it. This 19 | # changes those files in-place, so one should do this before commit. 20 | .PHONY: docs 21 | docs: | ./node_modules/.bin/doctoc 22 | ./node_modules/.bin/doctoc --notitle --maxlevel 3 docs/developer-guide/README.md 23 | ./node_modules/.bin/doctoc --notitle --maxlevel 3 docs/operator-guide/architecture.md 24 | ./node_modules/.bin/doctoc --notitle --maxlevel 3 docs/operator-guide/deployment.md 25 | ./node_modules/.bin/doctoc --notitle --maxlevel 3 docs/operator-guide/maintenance.md 26 | ./node_modules/.bin/doctoc --notitle --maxlevel 3 docs/operator-guide/mantav2-migration.md 27 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 6 | 7 | 11 | 12 | # Manta: a scalable, distributed object store 13 | 14 | Manta is an open-source, scalable, HTTP-based object store. All the pieces 15 | required to deploy and operate your own Manta are open source. This repo 16 | provides documentation for the overall Manta project and pointers to the other 17 | repositories that make up a complete Manta deployment. 18 | 19 | ## Getting started 20 | 21 | The fastest way to get started with Manta depends on what exactly one 22 | wishes to do. 23 | 24 | * To use Manta see the [Getting Started](./docs/user-guide/#getting-started) 25 | section of the User Guide. 26 | 27 | * To learn about installing and operating your own Manta deployment, see the 28 | [Manta Operator Guide](./docs/operator-guide/). 29 | 30 | * To understand Manta's architecture, see [Bringing Arbitrary Compute to 31 | Authoritative Data](http://queue.acm.org/detail.cfm?id=2645649), the [ACM 32 | Queue](http://queue.acm.org/) article on its design and implementation. 33 | 34 | * To understand the [CAP tradeoffs](http://en.wikipedia.org/wiki/CAP_theorem) in Manta, 35 | see [Fault Tolerence in Manta](http://dtrace.org/blogs/dap/2013/07/03/fault-tolerance-in-manta/) -- 36 | which received [some notable praise](https://twitter.com/eric_brewer/status/352804538769604609). 37 | 38 | * For help with working on Manta and building and testing your changes, 39 | see the [developer guide](docs/developer-guide) 40 | 41 | ## Community 42 | 43 | Community discussion about Manta happens in two main places: 44 | 45 | * The *manta-discuss* 46 | [mailing list](https://mantastorage.topicbox.com/groups/manta-discuss). 47 | If you wish to send mail to the list you'll need to join, but you can view 48 | and search the archives online without being a member. 49 | 50 | * In the *#manta* IRC channel on the 51 | [Libera.chat IRC network](https://libera.chat/). 52 | 53 | ## Dependencies 54 | 55 | Manta is composed of a number of services that deploy on top of Joyent's 56 | [Triton DataCenter](https://github.com/TritonDataCenter/triton) platform (just "Triton" 57 | for short), which is also open-source. Triton provides services for operating 58 | physical servers (compute nodes), deploying services in containers, monitoring 59 | services, transmitting and visualizing real-time performance data, and a bunch 60 | more. Manta primarily uses Triton for initial deployment, service upgrade, and 61 | service monitoring. 62 | 63 | Triton itself depends on [SmartOS](http://smartos.org). Manta also directly 64 | depends on several SmartOS features, notably ZFS. 65 | 66 | ## Building and Deploying Manta 67 | 68 | Manta service images are built and packaged using the same mechanisms as 69 | building the services that are part of Triton. Once you have Triton set up, 70 | follow the instructions in the [Manta Operator Guide](./docs/operator-guide/) 71 | to deploy Manta. The easiest way to play around with your own Manta 72 | installation is to first set up a Triton cloud-on-a-laptop (COAL) installation 73 | in VMware and then follow those instructions to deploy Manta on it. 74 | 75 | If you want to deploy your own builds of Manta components, see "Deploying your 76 | own Manta Builds" below. 77 | 78 | ## Repositories 79 | 80 | This repository is just a wrapper containing documentation about Manta. Manta 81 | is made up of several components from many repositoies. This section highlights 82 | some of the more important ones. 83 | 84 | A full list of repositories relevant to Manta is maintained in a [repo manifest 85 | file](./tools/jr-manifest.json) in this repo. To more conveniently list those 86 | repos, you can use the [`jr` tool](https://github.com/TritonDataCenter/joyent-repos#jr). 87 | 88 | The front door services respond to requests from the internet at large: 89 | 90 | * [muppet](https://github.com/TritonDataCenter/muppet): the haproxy-based "loadbalancer" 91 | service 92 | * [muskie](https://github.com/TritonDataCenter/manta-muskie): the node.js-based "webapi" 93 | service, this is Manta's "Directory API" 94 | * [buckets-api](https://github.com/TritonDataCenter/manta-buckets-api): Node.js-based 95 | "buckets-api" service, this is Manta's "Buckets API" 96 | 97 | The metadata tiers for the Directory and Buckets APIs store the entire object 98 | namespace (not object data) as well as backend storage system capacity: 99 | 100 | * [manatee](https://github.com/TritonDataCenter/manatee): the "postgres" service, a 101 | high-availability postgres cluster using synchronous replication and automatic 102 | fail-over 103 | * [moray](https://github.com/TritonDataCenter/moray): Node-based key-value store built on 104 | top of manatee. Also responsible for monitoring manatee replication topology 105 | (i.e., which postgres instance is the master). 106 | * [electric-moray](https://github.com/TritonDataCenter/electric-moray): Node-based service 107 | that provides the same interface as Moray, but which directs requests to one 108 | or more Moray+Manatee *shards* based on hashing the Moray key. 109 | * [buckets-mdapi](https://github.com/TritonDataCenter/manta-buckets-mdapi): a Rust-based 110 | API for managing all metadata for the Buckets API 111 | * [buckets-mdplacement](https://github.com/TritonDataCenter/manta-buckets-mdplacement): a 112 | Rust-based API for handling routing of Buckets API objects to appropriate 113 | nodes in the storage tier. 114 | 115 | The storage tier is responsible for actually storing bits on disk: 116 | 117 | * [mako](https://github.com/TritonDataCenter/manta-mako): the "storage" service, a 118 | nginx-based server that receives PUT/GET requests from the front door services 119 | to store object data on disk 120 | * [minnow](https://github.com/TritonDataCenter/manta-minnow): a Node-based agent that 121 | runs inside storage instances to periodically report storage capacity to the 122 | metadata tier 123 | 124 | There are a number of services not part of the data path that are critical for 125 | Manta's operation. For example: 126 | 127 | * [binder](https://github.com/TritonDataCenter/binder): hosts both ZooKeeper (used for 128 | manatee leader election and for group membership) and a Node-based DNS server 129 | that keeps track of which instances of each service are online at any given 130 | time 131 | * [mahi](https://github.com/TritonDataCenter/mahi): The "authcache" service for handling authn/authz. 132 | 133 | Most of the above components are *services*, of which there may be multiple 134 | *instances* in a single Manta deployment. Except for the last category of 135 | non-data-path services, these can all be deployed redundantly for availability 136 | and additional instances can be deployed to increase capacity. 137 | 138 | For more details on the architecture, including how these pieces actually fit 139 | together, see the [Architecture](./docs/operator-guide/architecture.md) section 140 | of the Operator Guide. 141 | 142 | ## Deploying your own Manta Builds 143 | 144 | As described above, as part of the normal Manta deployment process, you start 145 | with the "manta-deployment" zone that's built into Triton. Inside that zone, you 146 | run "manta-init" to fetch the latest Joyent build of each Manta component. Then 147 | you run Manta deployment tools to actually deploy zones based on these builds. 148 | 149 | The easiest way to use your own custom build is to first deploy Manta using the 150 | default Joyent build and *then* replace whatever components you want with your 151 | own builds. This will also ensure that you're starting from a known-working set 152 | of builds so that if something goes wrong, you know where to start looking. To 153 | do this: 154 | 155 | 1. Complete the Manta deployment procedure from the operator guide. 156 | 2. Build a zone image for whatever zone you want to replace. See the 157 | instructions for building [Triton](https://github.com/TritonDataCenter/triton) 158 | zone images. Manta zones work the same way. The output of this process 159 | will be a zone **image**, identified by uuid. The image is comprised of 160 | two files: an image manifest (a JSON file) and the image file itself 161 | (a binary blob). 162 | 3. Import the image into the Triton DataCenter that you're using to deploy Manta. 163 | (If you've got a multi-datacenter Manta deployment, you'll need to import the 164 | image into each datacenter separately using this same procedure.) 165 | 1. Copy the image and manifest files to the Triton headnode where the Manta 166 | deployment zone is deployed. For simplicity, assume that the 167 | manifest file is "/var/tmp/my_manifest.json" and the image file is 168 | "/var/tmp/my_image". You may want to use the image uuid in the filenames 169 | instead. 170 | 2. Import the image using: 171 | 172 | sdc-imgadm import -m /var/tmp/my_manifest.json -f /var/tmp/my_image 173 | 174 | 4. Now you can use the normal Manta zone update procedure (from the operator 175 | guide). This involves saving the current configuration to a JSON 176 | file using "manta-adm show -sj > config.json", updating the configuration 177 | file, and then applying the changes with "manta-adm update < config.json". 178 | When you modify the configuration file, you can use your image's uuid in 179 | place of whatever service you're trying to replace. 180 | 181 | If for some reason you want to avoid deploying the Joyent builds at all, you'll 182 | have to follow a more manual procedure. One approach is to update the SAPI 183 | configuration for whatever service you want (using sdc-sapi -- see 184 | [SAPI](https://github.com/TritonDataCenter/sdc-sapi)) *immediately after* running 185 | manta-init but before deploying anything. Note that each subsequent 186 | "manta-init" will clobber this change, though the SAPI configuration is normally 187 | only used for the initial deployment anyway. The other option is to apply the 188 | fully-manual install procedure from the Operator Guide (i.e., instead of 189 | using manta-deploy-coal or manta-deploy-lab) and use a custom "manta-adm" 190 | configuration file in the first place. If this is an important use case, file 191 | an issue and we can improve this procedure. 192 | 193 | The above procedure works to update Manta *zones*, which are most of the 194 | components above. The other two kinds of components are the *platform* and 195 | *agents*. Both of these procedures are documented in the Operator Guide, 196 | and they work to deploy custom builds as well as the official Joyent builds. 197 | 198 | ## Contributing to Manta 199 | 200 | To report bugs or request features, you can submit issues to the Manta project 201 | on Github. If you're asking for help with Joyent's production Manta service, 202 | you should contact Joyent support instead. 203 | 204 | See the [Contribution Guidelines](./CONTRIBUTING.md) for information about 205 | contributing changes to the project. 206 | 207 | ## Design principles 208 | 209 | Manta assumes several constraints on the data storage problem: 210 | 211 | 1. There should be one *canonical* copy of data. You shouldn't need to copy 212 | data in order to analyze it, transform it, or serve it publicly over the 213 | internet. 214 | 2. The system must scale horizontally in every dimension. It should be possible 215 | to add new servers and deploy software instances to increase the system's 216 | capacity in terms of number of objects, total data stored, or compute 217 | capacity. 218 | 3. The system should be general-purpose. 219 | 4. The system should be strongly consistent and highly available. In terms of 220 | [CAP](http://en.wikipedia.org/wiki/CAP_theorem), Manta sacrifices 221 | availability in the face of network partitions. (The reasoning here is that 222 | an AP cache can be built atop a CP system like Manta, but if Manta were AP, 223 | then it would be impossible for anyone to get CP semantics.) 224 | 5. The system should be transparent about errors and performance. The public 225 | API only supports atomic operations, which makes error reporting and 226 | performance easy to reason about. (It's hard to say anything about the 227 | performance of compound operations, and it's hard to report failures in 228 | compound operations.) Relatedly, a single Manta deployment may span multiple 229 | datacenters within a region for higher availability, but Manta does not 230 | attempt to provide a global namespace across regions, since that would imply 231 | uniformity in performance or fault characteristics. 232 | 233 | From these constraints, we define some design principles: 234 | 235 | 1. Manta presents an HTTP interface (with REST-based PUT/GET/DELETE operations) 236 | as the primary way of reading and writing data. Because there's only one 237 | copy of data, and some data needs to be available publicly (e.g., on the 238 | internet over standard protocols), HTTP is a good choice. 239 | 2. Manta is an *object store*, meaning that it only provides PUT/GET/DELETE for 240 | *entire objects*. You cannot write to the middle of an object or append to 241 | the end of one. This constraint makes it possible to guarantee strong 242 | consistency and high availability, since only the metadata tier (i.e., the 243 | namespace) needs to be strongly consistent, and objects themselves can be 244 | easily replicated for availability. 245 | 246 | It's easy to underestimate the problem of just reliably storing bits on disk. 247 | It's commonly assumed that the only components that fail are disks, that they 248 | fail independently, and that they fail cleanly (e.g., by reporting errors). In 249 | reality, there are a lot worse failure modes than disks failing cleanly, 250 | including: 251 | 252 | * disks or HBAs dropping writes 253 | * disks or HBAs redirecting both read and write requests to the wrong physical 254 | blocks 255 | * disks or HBAs retrying writes internally, resulting in orders-of-magnitude 256 | latency bubbles 257 | * disks, HBAs, or buses corrupting data at any point in the data path 258 | 259 | Manta delegates to ZFS to solve the single-system data storage problem. To 260 | handle these cases, 261 | 262 | * ZFS stores block checksums *separately* from the blocks themselves. 263 | * Filesystem metadata is stored redundantly (on separate disks). Data is 264 | typically stored redundantly as well, but that's up to user configuration. 265 | * ZFS is aware of how the filesystem data is stored across several disks. As a 266 | result, when reads from one disk return data that doesn't match the expected 267 | checksum, it's able to read another copy and fix the original one. 268 | 269 | ## Further reading 270 | 271 | For background on the overall design approach, see ["There's Just No Getting 272 | Around It: You're Building a Distributed 273 | System"](http://queue.acm.org/detail.cfm?id=2482856). 274 | 275 | For information about how Manta is designed to survive component failures and 276 | maintain strong consistency, see [Fault tolerance in 277 | Manta](http://dtrace.org/blogs/dap/2013/07/03/fault-tolerance-in-manta/). 278 | 279 | For information on the latest recommended production hardware, see [Joyent 280 | Manufacturing Matrix](http://eng.joyent.com/manufacturing/matrix.html) and 281 | [Joyent Manufacturing Bill of 282 | Materials](http://eng.joyent.com/manufacturing/bom.html). 283 | -------------------------------------------------------------------------------- /docs/README.md: -------------------------------------------------------------------------------- 1 | # Manta documentation 2 | 3 | Please see one of the following for Manta documentation: 4 | 5 | - The [README](../README.md) has a general introduction. 6 | - To use Manta see the [User Guide](./user-guide). 7 | - To install and operate Manta see the [Operator Guide](./operator-guide). 8 | - To develop/contribute to Manta, there is a [Developer Guide](./developer-guide) 9 | to help get you started. 10 | - See [Mantav2](./mantav2.md) for information on the mantav1 and mantav2 11 | separate active versions of Manta. 12 | -------------------------------------------------------------------------------- /docs/developer-guide/README.md: -------------------------------------------------------------------------------- 1 | # Manta Developer Guide 2 | 3 | This document has miscellaneous documentation for Manta developers, including 4 | how components are put together during the build and example workflows for 5 | testing changes. 6 | 7 | Anything related to the *operation* of Manta, including how Manta works at 8 | runtime, should go in the [Manta Operator Guide](../operator-guide) instead. 9 | 10 | The high-level basics are documented in the [main README](../../README.md) in 11 | this repo. That includes information on dependencies, how to build and deploy 12 | Manta, the repositories that make up Manta, and more. This document is for more 13 | nitty-gritty content than makes sense for the README. 14 | 15 | 16 | ## Contents 17 | 18 | 19 | 20 | 21 | 22 | - [Development Tips](#development-tips) 23 | - [Manta zone build and setup](#manta-zone-build-and-setup) 24 | - [Testing changes inside an actual Manta deployment](#testing-changes-inside-an-actual-manta-deployment) 25 | - [Zone-based components (including Manta deployment tools)](#zone-based-components-including-manta-deployment-tools) 26 | - [Building and deploying your own zone images](#building-and-deploying-your-own-zone-images) 27 | - [Building with your changes](#building-with-your-changes) 28 | - [What if you're changing dependencies?](#what-if-youre-changing-dependencies) 29 | - [Advanced deployment notes](#advanced-deployment-notes) 30 | - [Service Size Overrides](#service-size-overrides) 31 | - [Configuration](#configuration) 32 | - [Configuration Updates](#configuration-updates) 33 | - [Directory API Shard Management](#directory-api-shard-management) 34 | - [Buckets API Shard Management](#buckets-api-shard-management) 35 | 36 | 37 | 38 | 39 | # Development Tips 40 | 41 | To update the manta-deployment zone (a.k.a. the "manta0" zone) on a Triton 42 | headnode with changes you've made to your local sdc-manta git clone: 43 | 44 | sdc-manta.git$ ./tools/rsync-to 45 | 46 | To see which manta zones are deployed, use manta-adm show: 47 | 48 | headnode$ manta-adm show 49 | 50 | To tear down an existing manta deployment, use manta-factoryreset: 51 | 52 | manta$ manta-factoryreset 53 | 54 | 55 | # Manta zone build and setup 56 | 57 | You should look at the instructions in the README for actually building and 58 | deploying Manta. This section is a reference for developers to understand how 59 | those procedures work under the hood. 60 | 61 | Most Manta components are deployed as *zones*, based on *images* built from a 62 | single *repo*. Examples are above, and include *muppet* and *muskie*. 63 | 64 | For a typical zone (take "muppet"), the process from source code to deployment 65 | works like this: 66 | 67 | 1. Build the repository itself. 68 | 2. Build an image (a zone filesystem template and some metadata) from the 69 | contents of the built repository. 70 | 3. Optionally, publish the image to updates.tritondatacenter.com. 71 | 4. Import the image into a Triton instance. 72 | 5. Provision a new zone from the imported image. 73 | 6. During the first boot, the zone executes a one-time setup script. 74 | 7. During the first and all subsequent boots, the zone executes another 75 | configuration script. 76 | 77 | There are tools to automate most of this: 78 | 79 | * The build tools contained in the `eng.git` submodule, usually found in 80 | `deps/eng` in manta repositories include a tool called `buildimage` 81 | which assembles an image containing the built Manta component. The image 82 | represents a template filesystem with which instances of this 83 | component will be stamped out. After the image is built, it can be uploaded 84 | to updates.tritondatacenter.com. Alternatively, the image can be manually imported to 85 | a Triton instance by copying the image manifest and image file 86 | (a compressed zfs send stream) to the headnode and running 87 | "sdc-imgadm import". 88 | * The "manta-init" command takes care of step 4. You run this as part of any 89 | deployment. See the [Manta Operator's Guide](https://joyent.github.io/manta) 90 | for details. After the first run, subsequent runs find new images in 91 | updates.tritondatacenter.com, import them into the current Triton instance, and mark 92 | them for use by "manta-deploy". Alternatively, if you have images that were 93 | manually imported using "sdc-imgadm import", then "manta-init" can be run 94 | with the "-n" flag to use those local images instead. 95 | * The "manta-adm" and "manta-deploy" commands (whichever you choose to use) take 96 | care of step 5. See the Manta Operator's Guide for details. 97 | * Steps 6 and 7 happen automatically when the zone boots as a result of the 98 | previous steps. 99 | 100 | For more information on the zone setup and boot process, see the 101 | [manta-scripts](https://github.com/TritonDataCenter/manta-scripts) repo. 102 | 103 | 104 | # Testing changes inside an actual Manta deployment 105 | 106 | There are automated tests in many repos, but it's usually important to test 107 | changed components in the context of a full Manta deployment as well. You have 108 | a few options, but for all of them you'll need to have a local Manta deployment 109 | that you can deploy to. 110 | 111 | Some repos (including marlin, mola, and mackerel) may have additional 112 | suggestions for testing them. 113 | 114 | 115 | # Zone-based components (including Manta deployment tools) 116 | 117 | You have a few options: 118 | 119 | * Build your own zone image and deploy it. This is the normal upgrade process, 120 | it's the most complete test, and you should definitely do this if you're 121 | changing configuration or zone setup. It's probably the most annoying, but 122 | please help us streamline it by testing it and sending feedback. For details, 123 | see below. 124 | * Assuming you're doing your dev work in a zone on the Manta network, run 125 | whatever component you're testing inside that zone. You'll have to write a 126 | configuration file for it, but you may be able to copy most of the 127 | configuration from another instance. 128 | * Copy your code changes into a zone already deployed as part of your Manta. 129 | This way you don't have to worry about configuring your own instance, but 130 | it's annoying because there aren't great ways of synchronizing your changes. 131 | 132 | 133 | # Building and deploying your own zone images 134 | 135 | As described above, Manta's build and deployment model is exactly like Triton's, 136 | which is that most components are delivered as zone images and deployed by 137 | provisioning new zones from these images. While the deployment tools are 138 | slightly different than Triton's, the build process is nearly identical. The 139 | common instructions for building zone images are part of the [Triton 140 | documentation](https://github.com/TritonDataCenter/triton/blob/master/docs/developer-guide/building.md). 141 | 142 | ## Building with your changes 143 | 144 | Building a repository checked out to a given git branch will include those 145 | changes in the resulting image. 146 | 147 | One exception, is any `agents` (for example 148 | [`amon`](https://github.com/TritonDataCenter/sdc-amon), 149 | [`config-agent`](https://github.com/TritonDataCenter/sdc-config-agent/), 150 | [`registrar`](https://github.com/TritonDataCenter/registrar), (there are others)) that 151 | are bundled within the image. 152 | 153 | At build-time, the build will attempt to build agents from the same branch 154 | name as the checked-out branch of the component being built. If that branch 155 | name doesn't exist in the respective agent repository, the build will use 156 | the `master` branch of the agent repository. 157 | 158 | To include agents built from alternate branches at build time, set 159 | `$AGENT_BRANCH` in the shell environment. The build will then try to build 160 | all required agents from that branch. If no matching branch is found for a given 161 | agent, the build then will try to checkout the agent repository at the same 162 | branch name as the checked-out branch of the component you're building, before 163 | finally falling back to the `master` branch of that agent repository. 164 | 165 | The mechanism used is described in the 166 | [`Makefile.agent_prebuilt.defs`](https://github.com/TritonDataCenter/eng/blob/master/tools/mk/Makefile.agent_prebuilt.defs), 167 | [`Makefile.agent_prebuilt.targ`](https://github.com/TritonDataCenter/eng/blob/master/tools/mk/Makefile.agent_prebuilt.targ), 168 | and 169 | [`agent-prebuilt.sh`](https://github.com/TritonDataCenter/eng/blob/master/tools/agent_prebuilt.sh) 170 | files, likely appearing as a git submodule beneath `deps/eng` in the 171 | component repository. 172 | 173 | ## What if you're changing dependencies? 174 | 175 | In some cases, you may be testing a change to a single zone that involves more 176 | than one repository. For example, you may need to change not just madtom, but 177 | the node-checker module on which it depends. One way to test this is to push 178 | your dependency changes to a personal github clone (e.g., 179 | "davepacheco/node-checker" rather than "joyent/node-checker") and then commit a 180 | change to your local copy of the zone's repo ("manta-madtom", in this case) that 181 | points the repo at your local dependency: 182 | 183 | diff --git a/package.json b/package.json 184 | index a054b43..8ef5a35 100644 185 | --- a/package.json 186 | +++ b/package.json 187 | @@ -8,7 +8,7 @@ 188 | "dependencies": { 189 | "assert-plus": "0.1.1", 190 | "bunyan": "0.16.6", 191 | - "checker": "git://github.com/TritonDataCenter/node-checker#master", 192 | + "checker": "git://github.com/davepacheco/node-checker#master", 193 | "moray": "git://github.com/TritonDataCenter/node-moray.git#master", 194 | "posix-getopt": "1.0.0", 195 | "pg": "0.11.3", 196 | 197 | This approach ensures that the build picks up your private copy of both 198 | madtom and the node-checker dependency. But when you're ready for the final 199 | push, be sure to push your changes to the dependency first, and remember to 200 | remove (don't just revert) the above change to the zone's package.json! 201 | 202 | 203 | 204 | # Advanced deployment notes 205 | 206 | ## Service Size Overrides 207 | 208 | Application and service configs can be found under the `config` directory in 209 | sdc-manta.git. For example: 210 | 211 | config/application.json 212 | config/services/webapi/service.json 213 | 214 | Sometimes it is necessary to have size-specific overrides for these services 215 | within these configs that apply during setup. The size-specific override 216 | is in the same directory as the "normal" file and has `.[size]` as a suffix. 217 | For example, this is the service config and the production override for the 218 | webapi: 219 | 220 | config/services/webapi/service.json 221 | config/services/webapi/service.json.production 222 | 223 | The contents of the override are only the *differences*. Taking the above 224 | example: 225 | 226 | $ cat config/services/webapi/service.json 227 | { 228 | "params": { 229 | "networks": [ "manta", "admin" ], 230 | "ram": 768 231 | } 232 | } 233 | $ cat config/services/webapi/service.json.production 234 | { 235 | "params": { 236 | "ram": 32768, 237 | "quota": 100 238 | } 239 | } 240 | 241 | You can see what the merged config with look like with the 242 | `./bin/manta-merge-config` command. For example: 243 | 244 | $ ./bin/manta-merge-config -s coal webapi 245 | { 246 | "params": { 247 | "networks": [ 248 | "manta", 249 | "admin" 250 | ], 251 | "ram": 768 252 | }, 253 | "metadata": { 254 | "MUSKIE_DEFAULT_MAX_STREAMING_SIZE_MB": 5120 255 | } 256 | } 257 | $ ./bin/manta-merge-config -s production webapi 258 | { 259 | "params": { 260 | "networks": [ 261 | "manta", 262 | "admin" 263 | ], 264 | "ram": 32768, 265 | "quota": 100 266 | } 267 | } 268 | 269 | Note that after setup, the configs are stored in SAPI. Any changes to these 270 | files will *not* result in accidental changes in production (or any other 271 | stage). Changes must be made via the SAPI api (see the SAPI docs for details). 272 | 273 | 274 | ## Configuration 275 | 276 | Manta is deployed as a single SAPI application. Each manta service (moray, 277 | postgres, storage, etc.) has a corresponding SAPI service. Every zone which 278 | implements a manta service had a corresponding SAPI instance. 279 | 280 | Within the config/ and manifests/ directories, there are several subdirectories 281 | which provide the SAPI configuration used for manta. 282 | 283 | config/application.json Application definition 284 | config/services Service definitions 285 | manifests Configuration manifests 286 | manifests/applications Configuration manifests for manta application 287 | 288 | There's no static information for certain instances. Instead, manta-deploy will 289 | set a handful of instance-specific metadata (e.g. shard membership). 290 | 291 | ### Configuration Updates 292 | 293 | Once Manta has been deployed there will be cases where the service manifests 294 | must be changed. Only changing the manifest in this repository isn't 295 | sufficient. The manifests used to configure running instances (new and old) are 296 | the ones stored in SAPI. The service templates in the zone are **not used after 297 | initial setup**. To update service templates in a running environment (coal or 298 | production, for example): 299 | 300 | 1) Verify that your changes to configuration are backward compatible or that the 301 | updates will have no effect on running services. 302 | 303 | 2) Get the current configuration for your service: 304 | 305 | headnode$ sdc-sapi /services?name=[service name] 306 | 307 | If you can't find your service name, look for what you want with the following 308 | command: 309 | 310 | headnode$ sdc-sapi /services?application_uuid=$(sdc-sapi /applications?name=manta | \ 311 | json -gHa uuid) | json -gHa uuid name 312 | 313 | Take note of the service uuid and make sure you can fetch it with: 314 | 315 | headnode$ sdc-sapi /services/[service uuid] 316 | 317 | 3) Identify the differences between the template in this repository and what is 318 | in SAPI. 319 | 320 | 4) Update the service template in SAPI. If it is a simple, one-parameter 321 | change, and the value of the key is a string type, it can be done like this: 322 | 323 | headnode$ sapiadm update [service uuid] json.path=value 324 | #Examples: 325 | headnode$ sapiadm update 8386d8f5-d4ff-4b51-985a-061832b41179 \ 326 | params.tags.manta_storage_id=2.stor.us-east.joyent.us 327 | headnode$ sapiadm update update 0b48c067-01bd-41ca-9f70-91bda65351b2 \ 328 | metadata.PG_DIR=/manatee/pg/data 329 | 330 | If you require a complex type (an object or array) or a value that is not a 331 | string, you will need to hand-craft the differences and `|` to `sapiadm`. For 332 | example: 333 | 334 | headnode$ echo '{ "metadata": { "PORT": 5040 } }' | \ 335 | sapiadm update fde6c6ed-eab6-4230-bb39-69c3cba80f15 336 | 337 | Or if you want to "edit" what comes back from sapi: 338 | 339 | headnode$ sapiadm get [service uuid] | json params >/tmp/params.json 340 | #Edit params.txt to wrap the json structure in { "params": ... } 341 | headnode$ cat /tmp/params.json | json -o json-0 | sapiadm update [service uuid] 342 | 343 | 5) Once the service in SAPI has been modified, make sure to get it to verify 344 | what SAPI has is what it should be. 345 | 346 | 347 | ## Directory API Shard Management 348 | 349 | A shard is a set of moray buckets, backed by >1 moray instances and >=3 350 | Postgres instances. No data is shared between any two shards. Many other manta 351 | services may said to be "in a shard", but more accurately, they're using a 352 | particular shard. 353 | 354 | There are two pieces of metadata which define how shards are used: 355 | 356 | INDEX_MORAY_SHARDS Shards used for the indexing tier 357 | STORAGE_MORAY_SHARD Shard used for minnow (manta_storage) records 358 | 359 | Currently, the hash ring topology for electric-moray is created once during 360 | Manta setup and stored as an image in a Triton imgapi. The image uuid and 361 | imgapi endpoint are stored in the following sapi parameters: 362 | 363 | HASH_RING_IMAGE The hash ring image uuid 364 | HASH_RING_IMGAPI_SERVICE The imageapi that stores the image. 365 | 366 | In a cross-datacenter deployment, the HASH_RING_IMGAPI_SERVICE may be in 367 | another datacenter. This limits your ability to deploy new electric-moray 368 | instances in the event of DC failure. 369 | 370 | This topology is **independent** of what's set in manta-shardadm. **WARNING 371 | UNDER NO CIRCUMSTANCES SHOULD THIS TOPOLOGY BE CHANGED ONCE MANTA HAS BEEN 372 | DEPLOYED, DOING SO WILL RESULT IN DATA CORRUPTION** 373 | 374 | See manta-deploy-lab for hash-ring generation examples. 375 | 376 | The manta-shardadm tool lists shards and allows the addition of new ones: 377 | 378 | manta$ manta-shardadm 379 | Manage manta shards 380 | 381 | Usage: 382 | manta-shardadm [OPTIONS] COMMAND [ARGS...] 383 | manta-shardadm help COMMAND 384 | 385 | Options: 386 | -h, --help Print help and exit. 387 | --version Print version and exit. 388 | 389 | Commands: 390 | help (?) Help on a specific sub-command. 391 | list List manta shards. 392 | set Set manta shards. 393 | 394 | In addition, the -z flag to manta-deploy specifies a particular shard for that 395 | instance. In the case of moray and postgres, that value defines which shard 396 | that instance participates in. For all other services, that value defines which 397 | shard an instance will consume. 398 | 399 | Note that deploying a postgres or moray instance into a previously undefined 400 | shard will not automatically update the set of shards for the indexing tier. 401 | Because of the presence of the electric-moray proxy, adding an additional shard 402 | requires coordination with all existing shards, lest objects and requests be 403 | routed to an incorrect shard (and thereby inducing data corruption). If you 404 | find yourself adding additional capacity, deploy the new shard first, coordinate 405 | with all existing shards, then use manta-shardadm to add the shard to list of 406 | shards for the indexing tier. 407 | 408 | 409 | ## Buckets API Shard Management 410 | 411 | Buckets API is an experimental feature that serves similar functions as the 412 | Directory API but has a different paradigm for object organization. As opposed 413 | to the hierarchical object support provided by Directory API that comes with 414 | a limit on the maximum number of objects per directory, Buckets API offers a 415 | simpler structure for storing an unlimited number of objects in groups. The 416 | two-level object structure incurs a much smaller overhead in request processing 417 | and is more cost-efficient from a metadata perspective. It is a better option 418 | when the objects in the same bucket are loosely related and do not need a 419 | finer categorization. 420 | 421 | Buckets shards are defined and stored in the same way as directory shards 422 | but are separate entities altogether. They are also managed with 423 | `manta-shardadm` which generates the `BUCKETS_MORAY_SHARDS` and 424 | `BUCKETS_HASH_RING_IMAGE` configurations in SAPI metadata. 425 | 426 | Note: The "buckets" that Buckets API manages are not to be confused with moray 427 | "buckets" which represent the object namespaces for the data stored in Postgres. 428 | -------------------------------------------------------------------------------- /docs/mantav2.md: -------------------------------------------------------------------------------- 1 | # Mantav2 2 | 3 | Starting November 2019, there will be two separate active versions of Manta: 4 | 5 | - "**mantav1**" - a long term support branch of Manta that maintains current 6 | Manta features. 7 | - "**mantav2**" - a new major version of Manta that adds (Buckets API, storage 8 | "rebalancer" service) and removes (jobs, snaplinks, etc.) some major 9 | features, and becomes the focus of future Manta development. 10 | 11 | At this time, mantav1 is the recommended version for production usage, but 12 | that is expected to change to mantav2 during 2020. 13 | 14 | 15 | ## What is mantav2? 16 | 17 | "Mantav2" is a new major version of Manta. Its purpose is to focus on: 18 | 19 | - improved API latency; 20 | - exploring alternative storage backends that improve efficiency; 21 | - improved operation and stability at larger scales. 22 | 23 | It is a backward incompatible change, because it drops some API features. 24 | Significant changes are: 25 | 26 | - The following features of the current API (now called the "Directory API") 27 | are being removed. 28 | - jobs (a.k.a. compute jobs) 29 | - snaplinks 30 | - metering data under `//reports/...` 31 | Otherwise the Directory API remains a part of Manta. 32 | - A new "Buckets API" (S3-like) is added. This is the API for which latency 33 | improvements are being made. 34 | - A "rebalancer" system is added for storage tier maintenance. 35 | - The garbage collection (GC) system is improved for larger scale. 36 | - Improved per-account usage data for operators. 37 | 38 | The "master" branch of Manta-related git repos is for mantav2. Mantav1 39 | development has moved to "mantav1" branches. 40 | 41 | 42 | ## How do I know if I have mantav1 or mantav2? 43 | 44 | A user can tell from the "Server" header in Manta API responses. 45 | A mantav1 API responds with `Server: Manta`: 46 | 47 | $ curl -is $MANTA_URL/ | grep -i server 48 | server: Manta 49 | 50 | and a mantav2 API responds with `Server: Manta/2`: 51 | 52 | $ curl -is $MANTA_URL/ | grep -i server 53 | server: Manta/2 54 | 55 | 56 | An operator can tell from the `MANTAV` metadatum on the "manta" SAPI 57 | application. If `MANTAV` is `1` or empty, this is a mantav1: 58 | 59 | [root@headnode (mydc1) ~]# sdc-sapi /applications?name=manta | json -H 0.metadata.MANTAV 60 | 1 61 | 62 | If `MANTAV` is `2`, this is a mantav2: 63 | 64 | [root@headnode (mydc2) ~]# sdc-sapi /applications?name=manta | json -H 0.metadata.MANTAV 65 | 2 66 | 67 | 68 | ## How do I find the Node.js and Java clients for mantav2? 69 | 70 | The Node.js and Java clients for mantav2 are still under development. They are 71 | currently available in a feature branch of their respective git repositories. 72 | 73 | #### Node.js client 74 | 75 | The Node.js Manta client is developed in the 76 | [joyent/node-manta](https://github.com/TritonDataCenter/node-manta) repository. 77 | 78 | - mantav1: Currently on the ["master" branch](https://github.com/TritonDataCenter/node-manta/tree/master/) 79 | of joyent/node-manta, and published to npm as 80 | ["manta"](https://www.npmjs.com/package/manta) -- i.e. `npm install manta`. 81 | - mantav2: Currently on the ["buckets" branch](https://github.com/TritonDataCenter/node-manta/tree/buckets/) 82 | of joyent/node-manta. It is not yet published to npm. 83 | 84 | *(The intent is to eventually move mantav2 to the "master" branch and publish it 85 | to npm as "mantav2". Mantav1 dev would move to the "mantav1" branch and continue 86 | to publish to npm as "manta".)* 87 | 88 | 89 | #### Java client 90 | 91 | The Java Manta client is developed in the 92 | [joyent/java-manta](https://github.com/TritonDataCenter/java-manta) repository. 93 | 94 | - mantav1: Currently on the ["master" branch](https://github.com/TritonDataCenter/java-manta/tree/master/) 95 | of joyent/java-manta. Current release versions are 3.x. 96 | - mantav2: Currently on the ["buckets-m1" branch](https://github.com/TritonDataCenter/java-manta/tree/buckets-m1/) 97 | of joyent/java-manta. 98 | 99 | *(The intent is to eventually move mantav2 to the "master" branch and release it 100 | as a new "4.x" major version. Mantav1 dev would move to the "3.x" branch and 101 | continue to release as 3.x versions.)* 102 | 103 | 104 | ## Is mantav1 still supported? 105 | 106 | Operation of a Mantav1 per the [mantav1 Operator 107 | Guide](https://github.com/TritonDataCenter/manta/blob/mantav1/docs/operator-guide.md) 108 | remains unchanged, other than that operators should look for images named 109 | `mantav1-$servicename` rather than `manta-$servicename`. For example: 110 | 111 | ``` 112 | $ updates-imgadm list name=~mantav1- --latest 113 | UUID NAME VERSION FLAGS OS PUBLISHED 114 | ... 115 | 26515c9e-94c4-4204-99dd-d068c0c2ed3e mantav1-postgres mantav1-20200226T135432Z-gcff3bea I smartos 2020-02-26T14:08:42Z 116 | 5c8c8735-4c2c-489b-83ff-4e8bee124f63 mantav1-storage mantav1-20200304T221656Z-g1ba6beb I smartos 2020-03-04T22:21:22Z 117 | ``` 118 | 119 | There are "mantav1" branches of all the relevant repositories, from which 120 | "mantav1-$servicename" images are created for Mantav1 setup and operation. 121 | Joyent offers paid support for on premise mantav1. 122 | 123 | While mantav1 work is done to support particular customer issues, and PRs 124 | are accepted for mantav1 branches, the focus of current work is on mantav2. 125 | 126 | 127 | ## How do I migrate my Manta from mantav1 to mantav2? 128 | 129 | This work is still in development. At this time a Mantav2 deployment must 130 | start from scratch. 131 | -------------------------------------------------------------------------------- /docs/operator-guide/README.md: -------------------------------------------------------------------------------- 1 | # Manta Operator Guide 2 | 3 | *(Note: This is the operator guide for 4 | [Mantav2](https://github.com/TritonDataCenter/manta/blob/master/docs/mantav2.md). If you 5 | are operating a mantav1 deployment, please see the [Mantav1 Operator 6 | Guide](https://github.com/TritonDataCenter/manta/blob/mantav1/docs/operator-guide.md).)* 7 | 8 | This operator guide is divided into sections: 9 | 10 | 1. [Manta Architecture](./architecture.md) 11 | 2. [Deploying Manta](./deployment.md) - setting up a new Manta 12 | 3. [Operating and Maintaining Manta](./maintenance.md): performing 13 | upgrades, using Manta's alarming, metrics, and logs 14 | 4. [Migrating from Mantav1 to Mantav2](./mantav2-migration.md) 15 | 16 | * * * 17 | 18 | Manta is an internet-facing object store. The user interface to Manta is 19 | essentially: 20 | 21 | * A separate filesystem-like namespace *Directory API*, with directories and 22 | objects, accessible over HTTP. 23 | * A *Buckets API* (an experimental feature similar to S3) with objects, 24 | accessible over HTTP. 25 | * *Objects* are arbitrary-size blobs of data 26 | * Users can use standard HTTP `PUT`/`GET`/`DELETE` verbs to create, list, and 27 | remove buckets, directories, and objects. 28 | * Users can fetch arbitrary ranges of an object, but may not *modify* an object 29 | except by replacing it. 30 | 31 | Users can interact with Manta through the official Node.js CLI; the Node, or 32 | Java SDKs; curl(1); or any web browser. For more information, see the [Manta 33 | user guide](../user-guide). 34 | -------------------------------------------------------------------------------- /docs/operator-guide/architecture.md: -------------------------------------------------------------------------------- 1 | # Manta Architecture 2 | 3 | *([Up to the Manta Operator Guide front page.](./))* 4 | 5 | This section discusses the basics of the Manta architecture. 6 | 7 | 8 | ## Contents 9 | 10 | 11 | 12 | 13 | 14 | - [Design constraints](#design-constraints) 15 | - [Basic terminology](#basic-terminology) 16 | - [Manta and Triton (SDC)](#manta-and-triton-sdc) 17 | - [Components of Manta](#components-of-manta) 18 | - [Services, instances, and agents](#services-instances-and-agents) 19 | - [Manta components at a glance](#manta-components-at-a-glance) 20 | - [Consensus and internal service discovery](#consensus-and-internal-service-discovery) 21 | - [External service discovery](#external-service-discovery) 22 | - [Storage tier](#storage-tier) 23 | - [Metadata tier](#metadata-tier) 24 | - [Postgres, replication, and sharding](#postgres-replication-and-sharding) 25 | - [Other shards](#other-shards) 26 | - [Moray](#moray) 27 | - [Buckets-mdapi](#buckets-mdapi) 28 | - [Electric-moray](#electric-moray) 29 | - [Buckets-mdplacement](#buckets-mdplacement) 30 | - [The front door](#the-front-door) 31 | - [Objects and directories](#objects-and-directories) 32 | - [Multipart uploads](#multipart-uploads) 33 | - [Garbage Collection](#garbage-collection) 34 | - [Metering](#metering) 35 | - [Manta Scalability](#manta-scalability) 36 | 37 | 38 | 39 | 40 | ## Design constraints 41 | 42 | **Horizontal scalability.** It must be possible to add more hardware to scale 43 | any component within Manta without downtime. As a result of this constraint, 44 | there are multiple instances of each service. 45 | 46 | **Strong consistency.** In the face of network partitions where it's not 47 | possible to remain both consistent and available, Manta chooses consistency. So 48 | if all three datacenters in a three-DC deployment become partitioned from one 49 | another, requests may fail rather than serve potentially incorrect data. 50 | 51 | **High availability.** Manta must survive failure of any service, physical 52 | server, rack, or even an entire datacenter, assuming it's been deployed 53 | appropriately. Development installs of Manta can fit on a single system, and 54 | obviously those don't survive server failure, but several production deployments 55 | span three datacenters and survive partitioning or failure of an entire 56 | datacenter without downtime for the other two. 57 | 58 | ## Basic terminology 59 | 60 | We use **nodes** to refer to physical servers. **Compute nodes** mean the same 61 | thing they mean in Triton, which is any physical server that's not a head node. 62 | **Storage nodes** are compute nodes that are designated to store actual Manta 63 | objects. 64 | 65 | A Manta install uses: 66 | 67 | * a headnode (see "Manta and Triton" below) 68 | * one or more storage nodes to store user objects 69 | * one or more non-storage compute nodes for the other Manta services. 70 | 71 | We use the term *datacenter* (or DC) to refer to an availability zone (or AZ). 72 | Each datacenter represents a single Triton deployment (see below). Manta 73 | supports being deployed in either 1 or 3 datacenters within a single *region*, 74 | which is a group of datacenters having a high-bandwidth, low-latency network 75 | connection. 76 | 77 | 78 | ## Manta and Triton (SDC) 79 | 80 | Manta is built atop Triton (formerly known as SmartDataCenter). A 81 | three-datacenter deployment of Manta is built atop three separate Triton 82 | deployments. The presence of Manta does not change the way Triton is deployed 83 | or operated. Administrators still have AdminUI, APIs, and they're still 84 | responsible for managing the Triton services, platform versions, and the like 85 | through the normal Triton mechanisms. 86 | 87 | 88 | ## Components of Manta 89 | 90 | All user-facing Manta functionality can be divided into a few major subsystems: 91 | 92 | * The **storage tier** is responsible for storing the physical copies of user 93 | objects on disk. Storage nodes store objects as files with random uuids. So 94 | within each storage node, the objects themselves are effectively just large, 95 | opaque blobs of data. 96 | * The **metadata tier** is responsible for storing metadata about each object 97 | that's visible from the public Manta API. This metadata includes the set of 98 | storage nodes on which the physical copy is stored. 99 | 100 | 101 | In order to make all this work, there are several other pieces. For example: 102 | 103 | * The **front door** is made up of the SSL terminators, load balancers, and API 104 | servers that actually handle user HTTP requests. All user interaction with 105 | Manta happens over HTTP, so the front door handles all user-facing operations. 106 | * An **authentication cache** maintains a read-only copy of the Joyent account 107 | database. All front door requests are authenticated against this cache. 108 | * A **garbage collection** system removes objects marked for deletion. 109 | * A **consensus layer** is used to keep track of primary-secondary relationships 110 | in the metadata tier. 111 | * DNS-based **nameservices** are used to keep track of all instances of all 112 | services in the system. 113 | 114 | 115 | ## Services, instances, and agents 116 | 117 | Just like with Triton, components are divided into services, instances, and 118 | agents. Services and instances are SAPI concepts. 119 | 120 | A **service** is a group of **instances** of the same kind. For example, 121 | "webapi" is a service, and there may be multiple webapi zones. Each zone is an 122 | instance of the "webapi" service. The vast majority of Manta components are 123 | service instances, and there are several different services involved. 124 | 125 | Note: Do not confuse SAPI services with SMF services. We're talking about SAPI 126 | services here. A given SAPI instance (which is a zone) may have many *SMF* 127 | services. 128 | 129 | ### Manta components at a glance 130 | 131 | | Kind | Major subsystem | Service | Purpose | Components | 132 | | ------- | ------------------ | ------------------------ | -------------------------------------- | ------------------------------------------------------------------------------------------------------ | 133 | | Service | Consensus | nameservice | Service discovery | ZooKeeper, [binder](https://github.com/TritonDataCenter/binder) (DNS) | 134 | | Service | Front door | loadbalancer | SSL termination and load balancing | haproxy, [muppet](https://github.com/TritonDataCenter/muppet) | 135 | | Service | Front door | webapi | Manta HTTP Directory API server | [muskie](https://github.com/TritonDataCenter/manta-muskie) | 136 | | Service | Front door | buckets-api\* | Manta HTTP Buckets API server | [buckets-api](https://github.com/TritonDataCenter/manta-buckets-api) | 137 | | Service | Front door | authcache | Authentication cache | [mahi](https://github.com/TritonDataCenter/mahi) (redis) | 138 | | Service | Garbage Collection | garbage-buckets-consumer | Manta Buckets API garbage collection | [garbage-buckets-consumer (bin)](https://github.com/TritonDataCenter/manta-garbage-collector/blob/master/bin/garbage-buckets-consumer.js), [garbage-buckets-consumer (lib)](https://github.com/TritonDataCenter/manta-garbage-collector/blob/master/lib/garbage-buckets-consumer.js) | 139 | | Service | Garbage Collection | garbage-deleter | Deleting storage for objects | [garbage-deleter (bin)](https://github.com/TritonDataCenter/manta-mako/blob/master/bin/garbage-deleter.js), [garbage-deleter (lib)](https://github.com/TritonDataCenter/manta-mako/blob/master/lib/garbage-deleter.js) | 140 | | Service | Garbage Collection | garbage-dir-consumer | Manta Directory API garbage collection | [garbage-dir-consumer (bin)](https://github.com/TritonDataCenter/manta-garbage-collector/blob/master/bin/garbage-dir-consumer.js), [garbage-dir-consumer (lib)](https://github.com/TritonDataCenter/manta-garbage-collector/blob/master/lib/garbage-dir-consumer.js) | 141 | | Service | Garbage Collection | garbage-mpu-cleaner | MPU garbage collection | [garbage-mpu-cleaner (bin)](https://github.com/TritonDataCenter/manta-garbage-collector/blob/master/bin/garbage-mpu-cleaner.js), [garbage-mpu-cleaner (lib)](https://github.com/TritonDataCenter/manta-garbage-collector/blob/master/lib/garbage-mpu-cleaner.js) | 142 | | Service | Garbage Collection | garbage-uploader | Send GC instructions to storage zones | [garbage-uploader (bin)](https://github.com/TritonDataCenter/manta-garbage-collector/blob/master/bin/garbage-uploader.js), [garbage-uploader (lib)](https://github.com/TritonDataCenter/manta-garbage-collector/blob/master/lib/garbage-uploader.js) | 143 | | Service | Metadata | postgres | Directory metadata storage | postgres, [manatee](https://github.com/TritonDataCenter/manta-manatee) | 144 | | Service | Metadata | buckets-postgres\* | Buckets metadata storage | postgres, [manatee](https://github.com/TritonDataCenter/manta-manatee) | 145 | | Service | Metadata | moray | Directory key-value store | [moray](https://github.com/TritonDataCenter/moray) | 146 | | Service | Metadata | buckets-mdapi\* | Buckets key-value store | [buckets-mdapi](https://github.com/TritonDataCenter/manta-buckets-mdapi) | 147 | | Service | Metadata | electric-moray | Directory consistent hashing (sharding)| [electric-moray](https://github.com/TritonDataCenter/electric-moray) | 148 | | Service | Metadata | buckets-mdplacement\* | Buckets consistent hashing (sharding) | [buckets-mdplacement](https://github.com/TritonDataCenter/manta-buckets-mdplacement) | 149 | | Service | Metadata | reshard\* | Metadata reshard tool | [reshard](https://github.com/TritonDataCenter/manta-reshard) | 150 | | Service | Metadata | storinfo | Storage metadata cache and picker | [storinfo](https://github.com/TritonDataCenter/manta-storinfo) | 151 | | Service | Storage | storage | Object storage and capacity reporting | [mako](https://github.com/TritonDataCenter/manta-mako) (nginx), [minnow](https://github.com/TritonDataCenter/manta-minnow), [rebalancer-agent](https://github.com/TritonDataCenter/manta-rebalancer/tree/master/agent) | 152 | | Service | Storage | rebalancer | Storage zone evacuation/rebalancing | [rebalancer](https://github.com/TritonDataCenter/manta-rebalancer) | 153 | | Service | Operations | madtom | Web-based Manta monitoring | [madtom](https://github.com/TritonDataCenter/manta-madtom) | 154 | | Service | Operations | ops | Operator workspace | | 155 | 156 | \* _experimental features_ 157 | 158 | 159 | ## Consensus and internal service discovery 160 | 161 | In some sense, the heart of Manta (and Triton) is a service discovery mechanism 162 | (based on ZooKeeper) for keeping track of which service instances are running. 163 | In a nutshell, this works like this: 164 | 165 | 1. Setup: There are 3-5 "nameservice" zones deployed that form a ZooKeeper 166 | cluster. There's a "binder" DNS server in each of these zones that serves 167 | DNS requests based on the contents of the ZooKeeper data store. 168 | 2. Setup: When other zones are deployed, part of their configuration includes 169 | the IP addresses of the nameservice zones. These DNS servers are the only 170 | components that each zone knows about directly. 171 | 3. When an instance starts up (e.g., a "moray" zone), an SMF service called the 172 | *registrar* connects to the ZooKeeper cluster (using the IP addresses 173 | configured with the zone) and publishes its own IP to ZooKeeper. A moray 174 | zone for shard 1 in region "us-east" publishes its own IP under 175 | "1.moray.us-east.joyent.us". 176 | 4. When a client wants to contact the shard 1 moray, it makes a DNS request for 177 | 1.moray.us-east.joyent.us using the DNS servers in the ZooKeeper cluster. 178 | Those DNS servers returns *all* IPs that have been published for 179 | 1.moray.us-east.joyent.us. 180 | 5. If the registrar in the 1.moray zone dies, the corresponding entry in the 181 | ZooKeeper data store is automatically removed, causing that zone to fall out 182 | of DNS. Due to DNS TTLs of 60s, it may take up to a minute for clients to 183 | notice that the zone is gone. 184 | 185 | Internally, most services work this way. 186 | 187 | ## External service discovery 188 | 189 | Since we don't control Manta clients, the external service discovery system is 190 | simpler and more static. We manually configure the public 191 | `us-central.manta.mnx.io` DNS name to resolve to each of the loadbalancer 192 | public IP addresses. After a request reaches the loadbalancers, everything uses 193 | the internal service discovery mechanism described above to contact whatever 194 | other services they need. 195 | 196 | ## Storage tier 197 | 198 | The storage tier is made up of Mantis Shrimp nodes that have a great deal of 199 | of physical storage in order to store users' objects. 200 | 201 | Each storage node has an instance of the **storage** service, also called a 202 | "mako" or "shark" (as in: a *shard* of the storage tier). Inside this zone 203 | runs: 204 | 205 | * **mako**: an nginx instance that supports simple PUT/GET for objects. This is 206 | not the front door; this is used *internally* to store each copy of a user 207 | object. Objects are stored in a ZFS delegated dataset inside the storage 208 | zone, under `/manta/$account_uuid/$object_uuid`. 209 | * **minnow**: a small Node service that periodically reports storage capacity 210 | data into the metadata tier so that the front door knows how much capacity 211 | each storage node has. 212 | * **rebalancer-agent**: a small Rust service that processes object copy 213 | requests coming from the rebalancer. 214 | 215 | The **rebalancer** or **rebalancer manager** is a separate service running 216 | outside of the storage zones which orchestrates the migration of objects 217 | between storage zones. The service allows an operator to evacuate an entire 218 | storage zone in the event of hardware outage or planned replacement. 219 | 220 | 221 | ## Metadata tier 222 | 223 | The metadata tier is itself made up of three levels: 224 | 225 | * "postgres" and "buckets-postgres" zones, which run instances of the 226 | postgresql database 227 | * "moray" and "buckets-mdapi" zones, which run key-value stores on top of 228 | postgres 229 | * "electric-moray" and "buckets-mdplacement" zones, which handle sharding of 230 | metadata requests 231 | 232 | ### Postgres, replication, and sharding 233 | 234 | All object metadata is stored in PostgreSQL databases. Metadata is keyed on the 235 | object's name, and the value is a JSON document describing properties of the 236 | object including what storage nodes it's stored on. 237 | 238 | This part is particularly complicated, so pay attention! The metadata tier is 239 | **replicated for availability** and **sharded for scalability**. 240 | 241 | It's easiest to think of sharding first. Sharding means dividing the entire 242 | namespace into one or more *shards* in order to scale horizontally. So instead 243 | of storing objects A-Z in a single postgres database, we might choose two shards 244 | (A-M in shard 1, N-Z in shard 2), or three shards (A-I in shard 1, J-R in shard 245 | 2, S-Z in shard 3), and so on. Each shard is completely separate from the 246 | others. They don't overlap at all in the data that they store. The shard 247 | responsible for a given object is determined by consistent hashing on the 248 | *directory name* of the object. So the shard for "/mark/stor/foo" is determined 249 | by hashing "/mark/stor". 250 | 251 | Within each shard, we use multiple postgres instances for high availability. At 252 | any given time, there's a *primary peer* (also called the "master"), a 253 | *secondary peer* (also called the "synchronous slave"), and an *async peer* 254 | (sometimes called the "asynchronous slave"). As the names suggest, we configure 255 | synchronous replication between the primary and secondary, and asynchronous 256 | replication between the secondary and the async peer. **Synchronous** 257 | replication means that transactions must be committed on both the primary and 258 | the secondary before they can be committed to the client. **Asynchronous** 259 | replication means that the asynchronous peer may be slightly behind the other 260 | two. 261 | 262 | The idea with configuration replication in this way is that if the primary 263 | crashes, we take several steps to recover: 264 | 265 | 1. The shard is immediately marked read-only. 266 | 2. The secondary is promoted to the primary. 267 | 3. The async peer is promoted to the secondary. With the shard being read-only, 268 | it should quickly catch up. 269 | 4. Once the async peer catches up, the shard is marked read-write again. 270 | 5. When the former primary comes back online, it becomes the asynchronous peer. 271 | 272 | This allows us to quickly restore read-write service in the event of a postgres 273 | crash or an OS crash on the system hosting the primary. This process is managed 274 | by the "manatee" component, which uses ZooKeeper for leader election to 275 | determine which postgres will be the primary at any given time. 276 | 277 | It's really important to keep straight the difference between *sharding* and 278 | *replication*. Even though replication means that we have multiple postgres 279 | instances in each shard, only the primary can be used for read/write operations, 280 | so we're still limited by the capacity of a single postgres instance. That's 281 | why we have multiple shards. 282 | 283 | 284 | 285 | ### Other shards 286 | 287 | There are actually two kinds of metadata in Manta: 288 | 289 | * Object metadata, which is sharded as described above. This may be medium to 290 | high volume, depending on load. 291 | * Storage node capacity metadata, which is reported by "minnow" instances (see 292 | above) and all lives on one shard. This is extremely low-volume: a couple of 293 | writes per storage node per minute. 294 | 295 | Manta supports the **resharding** of directory object metadata, a process which 296 | would typically be used to add additional shards (for horizontal scaling of 297 | metadata capacity). This operation is handled by the reshard service which 298 | currently supports only the doubling of shards and incurs object write downtime 299 | during the hash ring update step in the process. 300 | 301 | ### Moray 302 | 303 | For each metadata shard (which we said above consists of three PostgreSQL 304 | databases), there's two or more "moray" instances. Moray is a key-value store 305 | built on top of postgres. Clients never talk to postgres directly; they always 306 | talk to Moray. (Actually, they generally talk to electric-moray, which proxies 307 | requests to Moray. See below.) Moray keeps track of the replication topology 308 | (which Postgres instances is the primary, which is the secondary, and which is 309 | the async) and directs all read/write requests to the primary Postgres instance. 310 | This way, clients don't need to know about the replication topology. 311 | 312 | Like Postgres, each Moray instance is tied to a particular shard. These are 313 | typically referred to as "1.moray", "2.moray", and so on. 314 | 315 | ### Buckets-mdapi 316 | 317 | Whereas moray handles the metadata requests for directory objects, buckets-mdapi 318 | performs the same function for buckets objects. 319 | 320 | ### Electric-moray 321 | 322 | The electric-moray service sits in front of the sharded Moray instances and 323 | directs requests to the appropriate shard. So if you try to update or fetch the 324 | metadata for `/mark/stor/foo`, electric-moray will hash `/mark/stor` to find the 325 | right shard and then proxy the request to one of the Moray instances operating 326 | that shard. 327 | 328 | ### Buckets-mdplacement 329 | 330 | Buckets-mdplacement performs the same sharding function as electric-moray for 331 | buckets objects. But unlike electric-moray, it is not on the data path of 332 | service requests. The hash ring information is cached in buckets-api upon 333 | service startup. 334 | 335 | ## The front door 336 | 337 | The front door consists of "loadbalancer", "webapi" and "buckets-api" zones. 338 | 339 | "loadbalancer" zones run haproxy for both SSL termination and load balancing 340 | across the available "webapi" and "buckets-api" instances. "haproxy" is 341 | managed by a component called "muppet" that uses the DNS-based service discovery 342 | mechanism to keep haproxy's list of backends up-to-date. 343 | 344 | "webapi" zones run the Manta-specific API server, called **muskie**. Muskie 345 | handles PUT/GET/DELETE requests to the front door, including requests to: 346 | 347 | * create and delete objects 348 | * create, list, and delete directories 349 | * create multipart uploads, upload parts, fetch multipart upload state, commit 350 | multipart uploads, and abort multipart uploads 351 | 352 | "buckets-api" zones run an alternative API server which handles S3-like 353 | PUT/GET/DELETE requests for objects with a different organization paradigm. 354 | The feature is still considered experimental, with limited support and 355 | unpublished API clients at this time. 356 | 357 | 358 | ### Objects and directories 359 | 360 | Requests for objects and directories involve: 361 | 362 | * validating the request 363 | * authenticating the user (via mahi, the auth cache) 364 | * looking up the requested object's metadata (via electric-moray or buckets- 365 | mdplacement) 366 | * authorizing the user for access to the specified resource 367 | 368 | For requests on directories and zero-byte objects, the last step is to update or 369 | return the right metadata. 370 | 371 | For write requests on objects, muskie or buckets-api then: 372 | 373 | * Constructs a set of candidate storage nodes that will be used to store the 374 | object's data, where each storage node is located in a different datacenter 375 | (in a multi-DC configuration). By default, there are two copies of the data, 376 | but users can configure this by setting the durability level with the 377 | request. 378 | * Tries to issue a PUT with 100-continue to each of the storage nodes in the 379 | candidate set. If that fails, try another set. If all sets are exhausted, 380 | fail with 503. 381 | * Once the 100-continue is received from all storage nodes, the user's data is 382 | streamed to all nodes. Upon completion, there should be a 204 response from 383 | each storage node. 384 | * Once the data is safely written to all nodes, the metadata tier is updated 385 | (using a PUT to electric-moray or to buckets-mdapi), and a 204 is returned to 386 | the client. At this point, the object's data is recorded persistently on the 387 | requested number of storage nodes, and the metadata is replicated on at least 388 | two index nodes. 389 | 390 | For read requests on objects, muskie or buckets-api instead contacts each of 391 | the storage nodes hosting the data and streams data from whichever one responds 392 | first to the client. 393 | 394 | 395 | ## Multipart uploads 396 | 397 | Multipart uploads, a.k.a. MPU, provide an alternate way for users to upload 398 | Manta objects (only for the Directory API). The user creates the multipart 399 | upload, uploads the object in parts, and exposes the object in Manta by 400 | committing the multipart upload. Generally, these operations are implemented 401 | using existing Manta constructs: 402 | 403 | * Parts are normal Manta objects, with a few key differences. Users cannot use 404 | the GET, POST or DELETE HTTP methods on parts. Additionally, all parts are 405 | co-located on the same set of storage nodes, which are selected when the 406 | multipart upload is created. 407 | * All parts for a given multipart upload are stored in a parts directory, which 408 | is a normal Manta directory. 409 | * Part directories are stored in the top-level `/$MANTA_USER/uploads` directory 410 | tree. 411 | 412 | Most of the logic for multipart uploads is performed by Muskie, but there are 413 | some additional features of the system only used for multipart uploads: 414 | 415 | * the **manta_uploads** bucket in Moray stores **finalizing records** for a 416 | given shard. A finalizing record is inserted atomically with the target 417 | object record when a multipart upload is committed. 418 | * the mako zones have a custom **mako-finalize** operation invoked by muskie 419 | when a multipart upload is committed. This operation creates the target object 420 | from the parts and subsequently deletes the parts from disk. This operation 421 | is invoked on all storage nodes that will contain the target object when the 422 | multipart upload is committed. 423 | 424 | 425 | ## Garbage Collection 426 | 427 | **Garbage collection** consists of several components in the `garbage-collector` 428 | and `storage` zones and is responsible for removing the storage used by objects 429 | which have been removed from the metadata tier. It is also responsible for 430 | removing metadata for finalized MPU uploads. 431 | 432 | When an object is deleted from the metadata tier in either the Manta Directory 433 | API or Manta Buckets API, the objects on disk are not immediately removed, nor are 434 | all references in the metadata tier itself. The original record is moved into a 435 | new deletion record which includes the information required to delete the 436 | storage backing the now-deleted object. The garbage collection system is 437 | responsible for actually performing the cleanup. 438 | 439 | Processes in the `garbage-collector` zone include: 440 | 441 | * `garbage-buckets-consumer` -- consumes deletion records from `buckets-mdapi` 442 | (created when an object is deleted by Manta Buckets API). The records found 443 | are written to local `instructions` files in the `garbage-collector` zone. 444 | 445 | * `garbage-dir-consumer` -- consumes deletion records from the 446 | `manta_fastdelete_queue` bucket (created when an object is deleted through 447 | the Manta Directory API). The records found are written to local 448 | `instructions` files in the `garbage-collector` zone. 449 | 450 | * `garbage-uploader` -- consumes the locally queued `instructions` and uploads 451 | them to the appropriate `storage` zone for processing. 452 | 453 | * `garbage-mpu-cleaner` -- consumes "finalized" MPU uploads (Manta Directory 454 | API only) and deletes the upload parts, upload directory and the finalize 455 | record itself after the upload has been finalized for some period of time 456 | (default: 5 minutes). 457 | 458 | On the `storage` zones, there's an additional component of garbage collection: 459 | 460 | * `garbage-deleter` -- consumes `instructions` that were uploaded by 461 | `garbage-uploader` and actually deletes the no-longer-needed object files in 462 | `/manta` of the storage zone. Once the storage is deleted, the completed 463 | instructions files are also deleted. 464 | 465 | Each of these services in both zones, run as their own SMF service and has their 466 | own log file in `/var/svc/log`. 467 | 468 | 469 | ## Metering 470 | 471 | **Metering** is the process of measuring how much resource each user used. It 472 | is not a full-fledged usage reporting feature at this time but the operator 473 | can still obtain the total object counts and bytes used per user by aggregating 474 | the metrics from individual storage zones. In each storage zone, the usage 475 | metrics are reported by a daily cron job that generates a `mako_rollup.out` 476 | text file under the `/var/tmp/mako_rollup` directory. 477 | 478 | 479 | ## Manta Scalability 480 | 481 | There are many dimensions to scalability. 482 | 483 | In the metadata tier: 484 | 485 | * number of objects (scalable with additional shards) 486 | * number of objects in a directory (fixed, currently at a million objects) 487 | 488 | In the storage tier: 489 | 490 | * total size of data (scalable with additional storage servers) 491 | * size of data per object (limited to the amount of storage on any single 492 | system, typically in the tens of terabytes, which is far larger than 493 | is typically practical) 494 | 495 | In terms of performance: 496 | 497 | * total bytes in or out per second (depends on network configuration) 498 | * count of concurrent requests (scalable with additional metadata shards or API 499 | servers) 500 | 501 | As described above, for most of these dimensions, Manta can be scaled 502 | horizontally by deploying more software instances (often on more hardware). For 503 | a few of these, the limits are fixed, but we expect them to be high enough for 504 | most purposes. For a few others, the limits are not known, and we've never (or 505 | rarely) run into them, but we may need to do additional work when we discover 506 | where these limits are. 507 | -------------------------------------------------------------------------------- /docs/operator-guide/deployment.md: -------------------------------------------------------------------------------- 1 | # Manta Deployment 2 | 3 | *([Up to the Manta Operator Guide front page.](./))* 4 | 5 | This section describes how to plan for and deploy a new Manta. 6 | 7 | ## Contents 8 | 9 | 10 | 11 | 12 | 13 | - [Planning a Manta deployment](#planning-a-manta-deployment) 14 | - [Choosing the number of datacenters](#choosing-the-number-of-datacenters) 15 | - [Choosing the number of metadata shards](#choosing-the-number-of-metadata-shards) 16 | - [Choosing the number of storage and non-storage compute nodes](#choosing-the-number-of-storage-and-non-storage-compute-nodes) 17 | - [Choosing how to lay out zones](#choosing-how-to-lay-out-zones) 18 | - [Example single-datacenter, multi-server configuration](#example-single-datacenter-multi-server-configuration) 19 | - [Example three-datacenter configuration](#example-three-datacenter-configuration) 20 | - [Other configurations](#other-configurations) 21 | - [Deploying Manta](#deploying-manta) 22 | - [A note on channel support](#a-note-on-channel-support) 23 | - [Step by step instructions](#step-by-step-instructions) 24 | - [Post-Deployment Steps](#post-deployment-steps) 25 | - [Prerequisites](#prerequisites) 26 | - [Set up a Manta Account](#set-up-a-manta-account) 27 | - [Test Manta from the CLI Tools](#test-manta-from-the-cli-tools) 28 | - [Networking configuration](#networking-configuration) 29 | - [manta-adm configuration](#manta-adm-configuration) 30 | 31 | 32 | 33 | 34 | # Planning a Manta deployment 35 | 36 | Before even starting to deploy Manta, you must decide: 37 | 38 | * the number of datacenters 39 | * the number of metadata shards 40 | * the number of storage and non-storage compute nodes 41 | * how to lay out the non-storage zones across the fleet 42 | 43 | ## Choosing the number of datacenters 44 | 45 | You can deploy Manta across any odd number of datacenters in the same region 46 | (i.e., having a reliable low-latency, high-bandwidth network connection among 47 | all datacenters). We've only tested one- and three-datacenter configurations. 48 | Even-numbered configurations are not supported. See "Other configurations" 49 | below for details. 50 | 51 | A single-datacenter installation can be made to survive server failure, but 52 | obviously cannot survive datacenter failure. The us-east deployment uses three 53 | datacenters. 54 | 55 | ## Choosing the number of metadata shards 56 | 57 | Recall that each metadata shard has the storage and load capacity of a single 58 | postgres instance. If you want more capacity than that, you need more shards. 59 | Shards can be added later without downtime, but it's a delicate operation. The 60 | us-east deployment uses three metadata shards, plus a separate shard for the 61 | compute and storage capacity data. 62 | 63 | We recommend at least two shards so that the compute and storage capacity 64 | information can be fully separated from the remaining shards, which would be 65 | used for metadata. 66 | 67 | ## Choosing the number of storage and non-storage compute nodes 68 | 69 | The two classes of node (storage nodes and non-storage nodes) usually have 70 | different hardware configurations. 71 | 72 | The number of storage nodes needed is a function of the expected data footprint 73 | and (secondarily) the desired compute capacity. 74 | 75 | The number of non-storage nodes required is a function of the expected load on 76 | the metadata tier. Since the point of shards is to distribute load, each 77 | shard's postgres instance should be on a separate compute node. So you want at 78 | least as many compute nodes as you will have shards. The us-east deployment 79 | distributes the other services on those same compute nodes. 80 | 81 | For information on the latest recommended production hardware, see [Joyent 82 | Manufacturing Matrix](http://eng.joyent.com/manufacturing/matrix.html) and 83 | [Joyent Manufacturing Bill of 84 | Materials](http://eng.joyent.com/manufacturing/bom.html). 85 | 86 | The us-east deployment uses older versions of the Tenderloin-A for service 87 | nodes and Mantis Shrimps for storage nodes. 88 | 89 | ## Choosing how to lay out zones 90 | 91 | Since there are so many different Manta components, and they're all deployed 92 | redundantly, there are a lot of different pieces to think about. (The 93 | production deployment in us-east has 20+ zones in *each* of the three 94 | datacenters.) So when setting up a Manta deployment, it's very important to 95 | think ahead of time about which components will run where! 96 | 97 | **The `manta-adm genconfig` tool (when used with the --from-file option) can be 98 | very helpful in laying out zones for Manta. See the `manta-adm` manual page for 99 | details.** `manta-adm genconfig --from-file` takes as input a list of physical 100 | servers and information about each one. Large deployments that use Device 42 to 101 | manage hardware inventory may find the 102 | [manta-genazconfig](https://github.com/TritonDataCenter/manta-genazconfig) tool useful for 103 | constructing the input for `manta-adm genconfig`. 104 | 105 | The most important production configurations are described below, 106 | but for reference, here are the principles to keep in mind: 107 | 108 | * **Storage** zones should only be located in dedicated storage nodes. We do 109 | not recommend co-locating them with other services. All other zones should be 110 | deployed onto non-storage compute nodes. 111 | * **Nameservice**: There must be an odd number of "nameservice" zones in order 112 | to achieve consensus, and there should be at least three of them to avoid a 113 | single point of failure. There must be at least one in each DC to survive 114 | any combination of datacenter partitions, and it's recommended that they be 115 | balanced across DCs as much as possible. 116 | * For the non-sharded, non-ops-related zones (which is everything except 117 | **moray**, **postgres**, **ops**, **madtom**), there 118 | should be at least two of each kind of zone in the entire deployment (for 119 | availability), and they should not be in the same datacenter (in order to 120 | survive a datacenter loss). For single-datacenter deployments, they should at 121 | least be on separate compute nodes. 122 | * Only one **madtom** zone is considered required. It 123 | would be good to provide more than one in separate datacenters (or at least 124 | separate compute nodes) for maintaining availability in the face of a 125 | datacenter failure. 126 | * There should only be one **ops** zone. If it's unavailable for any reason, 127 | that will only temporarily affect metering, garbage collection, and reports. 128 | * For **postgres**, there should be at least three instances in each shard. 129 | For multi-datacenter configurations, these instances should reside in 130 | different datacenters. For single-datacenter configurations, they should be 131 | on different compute nodes. (Postgres instances from different shards can be 132 | on the same compute node, though for performance reasons it would be better to 133 | avoid that.) 134 | * For **moray**, there should be at least two instances per shard in the entire 135 | deployment, and these instances should reside on separate compute nodes (and 136 | preferably separate datacenters). 137 | * **garbage-collector** zones should not be configured to poll more than 6 138 | shards. Further, they should not be co-located with instances of other 139 | CPU-intensive Manta components (e.g. loadbalancer) to avoid interference 140 | with the data path. 141 | 142 | Most of these constraints are required in order to maintain availability in the 143 | event of failure of any component, server, or datacenter. Below are some 144 | example configurations. 145 | 146 | ## Example single-datacenter, multi-server configuration 147 | 148 | On each storage node, you should deploy one "storage" zone. 149 | 150 | If you have N metadata shards, and assuming you'll be deploying 3 postgres 151 | instances in each shard, you'd ideally want to spread these over 3N compute 152 | nodes. If you combine instances from multiple shards on the same host, you'll 153 | defeat the point of splitting those into shards. If you combine instances from 154 | the same shard on the same host, you'll defeat the point of using replication 155 | for improved availability. 156 | 157 | You should deploy at least two Moray instances for each shard onto separate 158 | compute nodes. The remaining services can be spread over the compute nodes 159 | in whatever way, as long as you avoid putting two of the same thing onto the 160 | same compute node. Here's an example with two shards using six compute nodes: 161 | 162 | | CN1 | CN2 | CN3 | CN4 | CN5 | CN6 | 163 | | ------------- | -------------- | ----------------- | ----------------- | ------------ | -------------- | 164 | | postgres 1 | postgres 1 | postgres 1 | postgres 2 | postgres 2 | postgres 2 | 165 | | moray 1 | moray 1 | electric-moray | moray 2 | moray 2 | electric-moray | 166 | | nameservice | webapi | nameservice | webapi | nameservice | webapi | 167 | | ops | madtom | garbage-collector | loadbalancer | loadbalancer | loadbalancer | 168 | | storinfo | authcache | storinfo | | | authcache | 169 | 170 | In this notation, "postgres 1" and "moray 1" refer to an instance of "postgres" 171 | or "moray" for shard 1. 172 | 173 | 174 | ## Example three-datacenter configuration 175 | 176 | All three datacenters should be in the same region, meaning that they share a 177 | reliable, low-latency, high-bandwidth network connection. 178 | 179 | On each storage node, you should deploy one "storage" zone. 180 | 181 | As with the single-datacenter configuration, you'll want to spread the postgres 182 | instances for N shards across 3N compute nodes, but you'll also want to deploy 183 | at least one postgres instance in each datacenter. For four shards, we 184 | recommend the following in each datacenter: 185 | 186 | | CN1 | CN2 | CN3 | CN4 | 187 | | ---------------- | ----------------- | -------------- | ------------ | 188 | | postgres 1 | postgres 2 | postgres 3 | postgres 4 | 189 | | moray 1 | moray 2 | moray 3 | moray 4 | 190 | | nameservice | nameservice | electric-moray | authcache | 191 | | webapi | webapi | loadbalancer | loadbalancer | 192 | | ops | garbage-collector | madtom | storinfo | 193 | 194 | In this notation, "postgres 1" and "moray 1" refer to an instance of "postgres" 195 | or "moray" for shard 1. 196 | 197 | ## Other configurations 198 | 199 | For testing purposes, it's fine to deploy all of Manta on a single system. 200 | Obviously it won't survive server failure. This is not supported for a 201 | production deployment. 202 | 203 | It's not supported to run Manta in an even number of datacenters since there 204 | would be no way to maintain availability in the face of an even split. More 205 | specifically: 206 | 207 | * A two-datacenter configuration is possible but cannot survive datacenter 208 | failure or partitioning. That's because the metadata tier would require 209 | synchronous replication across two datacenters, which cannot be maintained in 210 | the face of any datacenter failure or partition. If we relax the synchronous 211 | replication constraint, then data would be lost in the event of a datacenter 212 | failure, and we'd also have no way to resolve the split-brain problem where 213 | both datacenters accept conflicting writes after the partition. 214 | * For even numbers N >= 4, we could theoretically survive datacenter failure, 215 | but any N/2 -- N/2 split would be unresolvable. You'd likely be better off 216 | dividing the same hardware into N - 1 datacenters. 217 | 218 | It's not supported to run Manta across multiple datacenters not in the same 219 | region (i.e., not having a reliable, low-latency, high-bandwidth connection 220 | between all pairs of datacenters). 221 | 222 | # Deploying Manta 223 | 224 | Before you get started for anything other than a COAL or lab deployment, be 225 | sure to read and fully understand the section on "Planning a Manta deployment" 226 | above. 227 | 228 | ## A note on channel support 229 | 230 | The two main commands used to deploy and maintain Manta, `manta-init` and 231 | `manta-adm` operate on software delivered as Triton images. These images are 232 | retrieved from `https://updates.tritondatacenter.com` by default. Both `manta-init` and 233 | `manta-adm` are "channel aware", taking `-C` options to override the Triton 234 | image channel to download images from. The default channel for a Manta 235 | deployment can be set using `sdcadm channel set ...` within a given datacenter. 236 | 237 | ## Step by step instructions 238 | 239 | These general instructions should work for anything from COAL to a 240 | multi-DC, multi-compute-node deployment. The general process is: 241 | 242 | 1. Set up Triton in each datacenter, including the headnode, all Triton 243 | services, and all compute nodes you intend to use. For easier management of 244 | hosts, we recommend that the hostname reflect the type of server and, 245 | possibly, the intended purpose of the host. For example, we use the "RA" 246 | or "RM" prefix for "Richmond-A" hosts and "MS" prefix for "Mantis Shrimp" 247 | hosts. 248 | 249 | 2. In the global zone of each Triton headnode, set up a manta deployment zone 250 | using: 251 | 252 | sdcadm post-setup common-external-nics # enable downloading service images 253 | sdcadm post-setup manta 254 | 255 | 3. In each datacenter, generate a Manta networking configuration file. 256 | 257 | a. For COAL, from the GZ, use: 258 | 259 | headnode$ /zones/$(vmadm lookup alias=manta0)/root/opt/smartdc/manta-deployment/networking/gen-coal.sh > /var/tmp/netconfig.json 260 | 261 | b. For those using the internal Joyent Engineering lab, run this from 262 | the [lab.git repo](https://mo.joyent.com/docs/lab/master/): 263 | 264 | lab.git$ node bin/genmanta.js -r RIG_NAME LAB_NAME 265 | 266 | and copy that to the headnode GZ. 267 | 268 | c. For other deployments, see "Networking configuration" below. 269 | 270 | 4. Once you've got the networking configuration file, configure networks by 271 | running this in the global zone of each Triton headnode: 272 | 273 | headnode$ ln -s /zones/$(vmadm lookup alias=manta0)/root/opt/smartdc/manta-deployment/networking /var/tmp/networking 274 | headnode$ cd /var/tmp/networking 275 | headnode$ ./manta-net.sh CONFIG_FILE 276 | 277 | This step is idempotent. Note that if you are setting up a multi-DC Manta, 278 | ensure that (1) your Triton networks have cross datacenter connectivity and 279 | routing set up and (2) the Triton firewalls allow TCP and UDP traffic cross- 280 | datacenter. 281 | 282 | 5. For multi-datacenter deployments, you must [link the datacenters within 283 | Triton](https://docs.joyent.com/private-cloud/install/headnode-installation/linked-data-centers) 284 | so that the UFDS database is replicated across all three datacenters. 285 | 286 | 6. For multi-datacenter deployments, you must [configure SAPI for 287 | multi-datacenter 288 | support](https://github.com/TritonDataCenter/sdc-sapi/blob/master/docs/index.md#multi-dc-mode). 289 | 290 | 7. If you'll be deploying a loadbalancer on any compute nodes *other* than a 291 | headnode, then you'll need to create the "external" NIC tag on those CNs. 292 | For common single-system configurations (for dev and test systems), you don't 293 | usually need to do anything for this step. For multi-CN configurations, 294 | you probably *will* need to do this. See the Triton documentation for 295 | [how to add a NIC tag to a 296 | CN](https://docs.joyent.com/sdc7/nic-tags#AssigningaNICTagtoaComputeNode). 297 | 298 | 8. In each datacenter's manta deployment zone, run the following: 299 | 300 | manta$ manta-init -s SIZE -e YOUR_EMAIL 301 | 302 | **`manta-init` must not be run concurrently in multiple datacenters.** 303 | 304 | `SIZE` must be one of "coal", "lab", or "production". `YOUR_EMAIL` is used 305 | to create an Amon contact for alarm notifications. 306 | 307 | This step runs various initialization steps, including downloading all of 308 | the zone images required to deploy Manta. This can take a while the first 309 | time you run it, so you may want to run it in a screen session. It's 310 | idempotent. 311 | 312 | 9. In each datacenter's manta deployment zone, deploy Manta components. 313 | 314 | a. In COAL, just run `manta-deploy-coal`. This step is idempotent. 315 | 316 | b. For a lab machine, just run `manta-deploy-lab`. This step is 317 | idempotent. 318 | 319 | c. For any other installation (including a multi-CN installation), you'll 320 | need to run several more steps: assign shards for storage and object 321 | metadata with "manta-shardadm"; create a hash ring with 322 | "manta-adm create-topology"; generate a "manta-adm" configuration file 323 | (see "manta-adm configuration" below); and finally run "manta-adm update 324 | config.json" to deploy those zones. Your best bet is to examine the 325 | "manta-deploy-dev" script to see how it uses these tools. See 326 | "manta-adm configuration" below for details on the input file to 327 | "manta-adm update". Each of these steps is idempotent, but the shard and 328 | hash ring must be set up before deploying any zones. 329 | 330 | 10. If desired, set up connectivity to the "ops" and 331 | "madtom" zones. See "Overview of Operating Manta" below for details. 332 | 333 | 11. For multi-datacenter deployments, set the MUSKIE\_MULTI\_DC SAPI property. 334 | This is required to enforce that object writes are distributed to multiple 335 | datacenters. In the SAPI master datacenter: 336 | 337 | headnode $ app_uuid="$(sdc-sapi /applications?name=manta | json -Ha uuid)" 338 | headnode $ echo '{ "metadata": { "MUSKIE_MULTI_DC": true } }' | \ 339 | sapiadm update "$app_uuid" 340 | 341 | Repeat the following in each datacenter. 342 | 343 | headnode $ manta-oneach -s webapi 'svcadm restart "*muskie*"' 344 | 345 | 12. If you wish to enable basic monitoring, run the following in each 346 | datacenter: 347 | 348 | manta-adm alarm config update 349 | 350 | to deploy Amon probes and probe groups shipped with Manta. This will cause 351 | alarms to be opened when parts of Manta are not functioning. Email 352 | notifications are enabled by default using the address provided to 353 | `manta-init` above. (Email notifications only work after you have 354 | configured the Amon service for sending email.) If you want to be notified 355 | about alarm events via XMPP, see "Changing alarm contact methods" below. 356 | 357 | 13. **In development environments with more than one storage zone on a single 358 | system, it may be useful to apply quotas to storage zones so that if the 359 | system fills up, there remains space in the storage pool to address the 360 | problem.** You can do this by finding the total size of the storage pool 361 | using `zfs list zones` in the global zone: 362 | 363 | # zfs list zones 364 | NAME USED AVAIL REFER MOUNTPOINT 365 | zones 77.5G 395G 612K /zones 366 | 367 | Determine how much you want to allow the storage zones to use. In this 368 | case, we'll allow the zones to use 100 GiB each, making up 300 GiB, or 75% 369 | of the available storage. Now, find the storage zones: 370 | 371 | # manta-adm show storage 372 | SERVICE SH ZONENAME GZ ADMIN IP 373 | storage 1 15711409-ca77-4204-b733-1058f14997c1 172.25.10.4 374 | storage 1 275dd052-5592-45aa-a371-5cd749dba3b1 172.25.10.4 375 | storage 1 b6d2c71f-ec3d-497f-8b0e-25f970cb2078 172.25.10.4 376 | 377 | and for each one, update the quota using `vmadm update`. You can apply a 378 | 100 GiB quota to all of the storage zones on a single-system Manta using: 379 | 380 | manta-adm show -H -o zonename storage | while read zonename; do 381 | vmadm update $zonename quota=100; done 382 | 383 | **Note:** This only prevents Manta storage zones from using more disk space 384 | than you've budgeted for them. If the rest of the system uses more than 385 | expected, you could still run out of disk space. To avoid this, make sure 386 | that all zones have quotas and the sum of quotas does not exceed the space 387 | available on the system. 388 | 389 | **Background:** Manta operators are responsible for basic monitoring of 390 | components, including monitoring disk usage to avoid components running out 391 | of disk space. Out of the box, Manta stops using storage zones that are 392 | nearly full. This mechanism relies on storage zones reporting how full they 393 | are, which they determine by dividing used space by available space. 394 | However, Manta storage zones are typically deployed without quotas, which 395 | means their available space is limited by the total space in the ZFS storage 396 | pool. This accounting is not correct when there are multiple storage zones 397 | on the same system. 398 | 399 | To make this concrete, consider a system with 400 GiB of total ZFS pool 400 | space. Suppose there are three storage zones, each using 100 GiB of space, 401 | and suppose that the rest of the system uses negligible storage space. In 402 | this case, there are 300 GiB in use overall, so there's 100 GiB available in 403 | the pool. As a result, each zone reports that it's using 100 GiB and has 404 | 100 GiB available, so it's 50% full. In reality, though, the system is 75% 405 | full. Each zone reports that it has 100 GiB free, but if we were to write 406 | just 33 GiB to each zone, the whole system would be full. 407 | 408 | This problem only affects deployments that place multiple storage zones on 409 | the same system, which is not typical outside of development. In 410 | development, the problem can be worked around by applying appropriate quotas 411 | in each zone (as described above). 412 | 413 | ## Post-Deployment Steps 414 | 415 | Once the above steps have been completed, there are a few steps you 416 | should consider doing to ensure a working deployment. 417 | 418 | ### Prerequisites 419 | 420 | If you haven't already done so, you will need to [install the Manta CLI tools](https://github.com/TritonDataCenter/node-manta#installation). 421 | 422 | ### Set up a Manta Account 423 | 424 | To test Manta with the Manta CLI tools, you will need an account 425 | configured in Triton. You can either use one of the default configured 426 | accounts or setup your own. The most common method is to test using the 427 | `poseidon` account which is created by the Manta install. 428 | 429 | In either case, you will need access to the Operations Portal. [See the 430 | instructions here](https://docs.joyent.com/private-cloud/install/headnode-installation#adding-external-access-to-adminui-and-imgapi) on how to find the IP address of the Operations Portal from 431 | your headnode. 432 | 433 | Log into the Operations Portal: 434 | 435 | * COAL users should use login `admin` and the password [you initially setup](https://github.com/TritonDataCenter/triton/blob/master/docs/developer-guide/coal-setup.md#configure-the-headnode). 436 | * Lab users will also use `admin`, but need to ask whoever 437 | provisioned your lab account for the password. 438 | 439 | Once in, follow [these instructions](https://docs.joyent.com/private-cloud/users#portalsshkeys) to add ssh keys to the account of your choice. 440 | 441 | ### Test Manta from the CLI Tools 442 | 443 | Once you have setup an account on Manta or added your ssh keys added to an 444 | existing account, you can test your Manta install with the Manta CLI 445 | tools you installed above in "Prerequisites". 446 | 447 | There are complete instructions on how to get started with the CLI 448 | tools [in the User Guide](./user-guide/#getting-started). 449 | 450 | Some things in that guide will not be as clear for users of custom deployments. 451 | 452 | 1. The biggest difference will be the setting of the `MANTA_URL` 453 | variable. You will need to find the IP address of your API 454 | endpoint. To do this from your headnode: 455 | 456 | headnode$ manta-adm show -H -o primary_ip loadbalancer 457 | 458 | Multiple addresses will be returned. Choose any one and set `MANTA_URL` 459 | to `https://$that_ip`. 460 | 2. `MANTA_USER` will be the account you setup in "Set up a Manta Account" 461 | section. 462 | 3. `MANTA_KEY_ID` will be the ssh key id you added in "Set up a Manta 463 | Account" section. 464 | 4. If the key you used is in an environment that has not installed a 465 | certificate signed by a recognized authority you might see `Error: 466 | self signed certificate` errors. To fix this, add 467 | `MANTA_TLS_INSECURE=true` to your environment or shell config. 468 | 469 | A final `~/.bashrc` or `~/.bash_profile` might look something like: 470 | 471 | export MANTA_USER=poseidon 472 | export MANTA_URL=https:// 473 | export MANTA_TLS_INSECURE=true 474 | export MANTA_KEY_ID=`ssh-keygen -l -f ~/.ssh/id_rsa.pub | awk '{print $2}' | tr -d '\n'` 475 | 476 | Lastly test the CLI tools from your development machine: 477 | 478 | $ echo "Hello, Manta" > /tmp/hello.txt 479 | $ mput -f /tmp/hello.txt ~~/stor/hello-foo 480 | .../stor/hello-foo [=======================================================>] 100% 13B 481 | $ mls ~~/stor/ 482 | hello-foo 483 | $ mget ~~/stor/hello-foo 484 | Hello, Manta 485 | 486 | 487 | ## Networking configuration 488 | 489 | The networking configuration file is a per-datacenter JSON file with several 490 | properties: 491 | 492 | | Property | Kind | Description | 493 | | ----------------- | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | 494 | | `azs` | array of strings | list of all availability zones (datacenters) participating in Manta in this region | 495 | | `this_az` | string | string (in `azs`) denoting this availability zone | 496 | | `manta_nodes` | array of strings | list of server uuid's for *all* servers participating in Manta in this AZ | 497 | | `admin` | object | describes the "admin" network in this datacenter (see below) | 498 | | `manta` | object | describes the "manta" network in this datacenter (see below) | 499 | | `nic_mappings` | object | maps each server in `manta_nodes` to an object mapping each network name to the network interface on the server that should be tagged | 500 | | `mac_mappings` | object | (deprecated) maps each server uuid from `manta_nodes` to an object mapping each network name ("admin", "manta") to the MAC address on that server over which that network should be created. | 501 | | `distribute_svcs` | boolean | control switch over boot-time networking detection performed by `manta-net.sh` (see below) | 502 | 503 | "admin" and "manta" describe the networks that are built into Manta: 504 | 505 | * `admin`: the Triton administrative network 506 | * `manta`: the Manta administrative network, used for high-volume communication 507 | between all Manta services. 508 | 509 | Each of these is an object with several properties: 510 | 511 | | Property | Kind | Description | 512 | | --------- | ------ | ---------------------------------------------------------------------- | 513 | | `network` | string | Name for the Triton network object (usually the same as the network name) | 514 | | `nic_tag` | string | NIC tag name for this network (usually the same as the network name) | 515 | 516 | Besides those two, each of these blocks has a property for the current 517 | availability zone that describes the "subnet", "gateway", "vlan_id", and 518 | "start" and "end" provisionable addresses. 519 | 520 | `nic_mappings` is a nested object structure defining the network interface to be 521 | tagged for each server defined in `manta_nodes`, and for each of Manta's 522 | required networks. See below for an example of this section of the 523 | configuration. 524 | 525 | Note: If aggregations are used, they must already exist in NAPI, and updating 526 | NIC tags on aggregations will require a reboot of the server in question. 527 | 528 | The optional boolean `distribute_svcs` gives control to the operator over the 529 | boot-time networking detection that happens each time `manta-net.sh` is executed 530 | (which determines if the global zone SMF services should be distributed). For 531 | example, an operator has enabled boot-time networking in a datacenter _after_ 532 | installing Manta, and subsequently would like to add some more storage nodes. 533 | For consistency, the operator can set `distribute_svcs` to `true` in order to 534 | force distribution of these global zone services. 535 | 536 | Note: For global zone network changes handled by boot-time networking to 537 | take effect, a reboot of the node must be performed. See 538 | [Triton's virtual networking documentation](https://docs.joyent.com/private-cloud/networks/sdn/architecture#bootstrapping-networking-state-in-the-global-zone) 539 | for more information on boot-time networking. 540 | 541 | For reference, here's an example multi-datacenter configuration with one service 542 | node (aac3c402-3047-11e3-b451-002590c57864) and one storage node 543 | (445aab6c-3048-11e3-9816-002590c3f3bc): 544 | 545 | { 546 | "this_az": "staging-1", 547 | 548 | "manta_nodes": [ 549 | "aac3c402-3047-11e3-b451-002590c57864", 550 | "445aab6c-3048-11e3-9816-002590c3f3bc" 551 | ], 552 | "azs": [ 553 | "staging-1", 554 | "staging-2", 555 | "staging-3" 556 | ], 557 | 558 | "admin": { 559 | "nic_tag": "admin", 560 | "network": "admin", 561 | "staging-1": { 562 | "subnet": "172.25.3.0/24", 563 | "gateway": "172.25.3.1" 564 | }, 565 | "staging-2": { 566 | "subnet": "172.25.4.0/24", 567 | "gateway": "172.25.4.1" 568 | }, 569 | "staging-3": { 570 | "subnet": "172.25.5.0/24", 571 | "gateway": "172.25.5.1" 572 | } 573 | }, 574 | 575 | "manta": { 576 | "nic_tag": "manta", 577 | "network": "manta", 578 | "staging-1": { 579 | "vlan_id": 3603, 580 | "subnet": "172.27.3.0/24", 581 | "start": "172.27.3.4", 582 | "end": "172.27.3.254", 583 | "gateway": "172.27.3.1" 584 | }, 585 | "staging-2": { 586 | "vlan_id": 3604, 587 | "subnet": "172.27.4.0/24", 588 | "start": "172.27.4.4", 589 | "end": "172.27.4.254", 590 | "gateway": "172.27.4.1" 591 | }, 592 | "staging-3": { 593 | "vlan_id": 3605, 594 | "subnet": "172.27.5.0/24", 595 | "start": "172.27.5.4", 596 | "end": "172.27.5.254", 597 | "gateway": "172.27.5.1" 598 | } 599 | }, 600 | 601 | "nic_mappings": { 602 | "aac3c402-3047-11e3-b451-002590c57864": { 603 | "manta": { 604 | "mac": "90:e2:ba:4b:ec:d1" 605 | } 606 | }, 607 | "445aab6c-3048-11e3-9816-002590c3f3bc": { 608 | "manta": { 609 | "mac": "90:e2:ba:4a:32:71" 610 | }, 611 | "mantanat": { 612 | "aggr": "aggr0" 613 | } 614 | } 615 | } 616 | } 617 | 618 | The deprecated `mac_mappings` can also be used in place of `nic_mappings`. Only 619 | one of `nic_mappings` or `mac_mappings` is supported per network configuration 620 | file. 621 | 622 | In a multi-datacenter configuration, this would be used to configure the 623 | "staging-1" datacenter. There would be two more configuration files, one 624 | for "staging-2" and one for "staging-3". 625 | 626 | 627 | 628 | ## manta-adm configuration 629 | 630 | "manta-adm" is the tool we use both to deploy all of the Manta zones and then 631 | to provision new zones, deprovision old zones, or reprovision old zones with a 632 | new image. "manta-adm" also has commands for viewing what's deployed, showing 633 | information about compute nodes, and more, but this section only discusses the 634 | configuration file format. 635 | 636 | A manta-adm configuration file takes the form: 637 | 638 | { 639 | "COMPUTE_NODE_UUID": { 640 | "SERVICE_NAME": { 641 | "IMAGE_UUID": COUNT_OF_ZONES 642 | }, 643 | "SHARDED_SERVICE_NAME": { 644 | "SHARD_NUMBER": { 645 | "IMAGE_UUID": COUNT_OF_ZONES 646 | }, 647 | } 648 | }, 649 | } 650 | 651 | The file specifies how many of each kind of zone should be deployed on each 652 | compute node. For most zones, the "kind" of zone is just the service name 653 | (e.g., "storage"). For sharded zones, you also have to specify the shard 654 | number. 655 | 656 | After you've run `manta-init`, you can generate a sample configuration for a 657 | single-system install using "manta-adm genconfig". Use that to give you an 658 | idea of what this looks like: 659 | 660 | $ manta-adm genconfig coal 661 | { 662 | "": { 663 | "nameservice": { 664 | "197e905a-d15d-11e3-90e2-6bf8f0ea92b3": 1 665 | }, 666 | "postgres": { 667 | "1": { 668 | "92782f28-d236-11e3-9e6c-5f7613a4df37": 2 669 | } 670 | }, 671 | "moray": { 672 | "1": { 673 | "ef659002-d15c-11e3-a5f6-4bf577839d16": 1 674 | } 675 | }, 676 | "electric-moray": { 677 | "e1043ddc-ca82-11e3-950a-ff14d493eebf": 1 678 | }, 679 | "storage": { 680 | "2306b44a-d15d-11e3-8f87-6b9768efe5ae": 2 681 | }, 682 | "authcache": { 683 | "5dff63a4-d15c-11e3-a312-5f3ea4981729": 1 684 | }, 685 | "storinfo": { 686 | "2ef81a09-ad04-445f-b4fe-1aa87ce4e54c": 1 687 | }, 688 | "webapi": { 689 | "319afbfa-d15e-11e3-9aa9-33ebf012af8f": 1 690 | }, 691 | "loadbalancer": { 692 | "7aac4c88-d15c-11e3-9ea6-dff0b07f5db1": 1 693 | }, 694 | "ops": { 695 | "ab253aae-d15d-11e3-8f58-3fb986ce12b3": 1 696 | } 697 | } 698 | } 699 | 700 | This file effectively specifies all of the Manta components except for the 701 | platforms. 702 | 703 | You can generate a configuration file that describes your current deployment 704 | with `manta-adm show -s -j`. 705 | 706 | For a coal or lab deployment, your best bet is to save the output of `manta-adm 707 | genconfig coal` or `manta-adm genconfig lab` to a file and use that. This is 708 | what the `manta-deploy-coal` and `manta-deploy-lab` scripts do, and you may as 709 | well just use those. 710 | 711 | Once you have a file like this, you can pass it to `manta-adm update`, which 712 | will show you what it will do in order to make the deployment match the 713 | configuration file, and then it will go ahead and do it. For more information, 714 | see "manta-adm help update". 715 | -------------------------------------------------------------------------------- /docs/sample-training.md: -------------------------------------------------------------------------------- 1 | ## Sample training outline 2 | 3 | This document outlines sample topics for a training course on operating Manta. 4 | 5 | * Docs: 6 | * Manta Overview: https://github.com/TritonDataCenter/manta 7 | * Manta Ops Guide: https://github.com/TritonDataCenter/manta/blob/master/docs/manta-ops.md 8 | * Manatee Users Guide: https://github.com/TritonDataCenter/manatee/blob/master/docs/user-guide.md 9 | * Manatee Troubleshooting Guide: https://github.com/TritonDataCenter/manatee/blob/master/docs/trouble-shooting.md 10 | * Architecture Review 11 | * Using Manta 12 | * Get new users set up for Manta 13 | * `m* tools`: `mmkdir`, `mrmdir`, `mls`, `mfind`, `mput`, `mget`, `msign`, `mrm`, 14 | * Basic Map/reduce patterns 15 | * Discovery/Moving around 16 | * `sdc-cnapi` 17 | * `sdc-vmapi` 18 | * `sdc-sapi` 19 | * `sdc-login` 20 | * `manta-login` - type, zonename, etc. 21 | * `manta-adm` - show, cn 22 | * Deployments 23 | * `manta-adm` - `-s -j`, edit, `update -n`, `update` 24 | * `manta-deploy` 25 | * `manta-undeploy` 26 | * `sapiadm reprovision ...` 27 | * Upgrades 28 | * Operations 29 | * Typical Zone setup 30 | * setup, `mdata:execute`, sapi_manifests, config-agent 31 | * svcs, svcadm, svccfg 32 | * Dashboards 33 | * Via ssh tunnels like: `ssh -o TCPKeepAlive=yes -N -n root@[Headnode] -L 5555:[MadtomAdminIp]:80` 34 | * Madtom: _hint: sometimes it lies_ 35 | * Marlin Dashboard 36 | * Alarms 37 | * mantamon 38 | * Known issues 39 | * Zookeeper Leader goes down 40 | * Postgres needs vacuum/analyze 41 | * Powering down Manta 42 | * Command Line Tools/Where stuff is 43 | * General 44 | * `json` 45 | * `bunyan` 46 | * Zookeeper/binder 47 | * `zkCli.sh` 48 | * `dig @localhost` 49 | * Postgres (Manatee) 50 | * `manatee-adm` 51 | * `psql moray` 52 | * Moray/Electric Moray 53 | * `getobject` 54 | * `getbucket` 55 | * Storage 56 | * `/manta/[owner]/[object_id]` 57 | * `/manta/tombstone/[date]/[object_id]` 58 | -------------------------------------------------------------------------------- /docs/user-guide/.htaccess: -------------------------------------------------------------------------------- 1 | Redirect 301 /manta/tutorial.html /manta/index.html 2 | -------------------------------------------------------------------------------- /docs/user-guide/README.md: -------------------------------------------------------------------------------- 1 | # Manta User Guide 2 | 3 | Manta is a highly scalable, distributed object storage service. This guide 4 | describes how to use the service. 5 | 6 | # Features 7 | 8 | Some features of the service include: 9 | 10 | * An HTTP API 11 | * Multiple data center replication with per object replication controls 12 | * Unlimited number of objects with no size limit per object 13 | * Read-after-write consistency 14 | 15 | 16 | # Documents in this User Guide 17 | 18 | * [Libraries / SDKs](./sdks.md) 19 | * [CLI Command Reference](./commands-reference.md) 20 | * [Storage Reference](./storage-reference.md) 21 | * [Role-Based Access Control Guide](./rbac.md) 22 | 23 | 24 | # Getting Started 25 | 26 | ## User accounts, authentication, and security 27 | 28 | To use Manta, you need a Triton account. 29 | ([Triton](https://github.com/TritonDataCenter/triton) is the cloud management platform on 30 | which a Manta runs.) If you don't already have an account, contact your 31 | administrator. Once you have signed up, you will need to add an SSH public key 32 | to your account. The SSH key is used to authenticate you with the Manta API. 33 | 34 | ## Install the Node.js Manta tools 35 | 36 | The Manta node.js SDK provides a node.js library and CLI tools for using Manta 37 | from the command-line. First install [node.js](http://nodejs.org/), then: 38 | 39 | npm install -g manta 40 | 41 | Verify that it installed successfully and is on your PATH by using `mls`, one 42 | of the Manta CLI tools: 43 | 44 | $ mls --version 45 | 5.2.1 46 | 47 | ## Setting Up Your Environment 48 | 49 | While you can specify command line switches to all of the node-manta CLI 50 | programs, it is significantly easier for you to set them globally in your 51 | environment. There are four environment variables that all Manta command-line 52 | tools look for: 53 | 54 | * `MANTA_URL` - The https API endpoint. 55 | 56 | * `MANTA_USER` - Your account login name. 57 | 58 | * `MANTA_KEY_ID` - The fingerprint of your SSH key. This can be calculated 59 | using `ssh-keygen -l ...`. For example: 60 | 61 | ``` 62 | $ ssh-keygen -l -f ~/.ssh/id_rsa.pub | awk '{print $2}' 63 | SHA256:qJjQoXlVnG940ZGWIgIrLm2lWbRFWk7nDKKzbLMtU4I 64 | ``` 65 | 66 | If the key is loaded into your ssh-agent, then `ssh-add -l` will show the 67 | fingerprint as well. 68 | 69 | * `MANTA_SUBUSER` - Optional. This is only required if using [role-based access 70 | control](./rbac.md). A sub-user of the `MANTA_USER` account with configured 71 | limited access. 72 | 73 | For example, you might have something like the following in your shell profile 74 | file (`~/.profile`, `~/.bashrc`, etc.): 75 | 76 | export MANTA_URL=https://us-east.manta.example.com 77 | export MANTA_USER=john.smith 78 | export MANTA_KEY_ID=$(ssh-keygen -l -f ~/.ssh/id_rsa.pub | awk '{print $2}') 79 | 80 | 81 | Everything works if typing `mls /$MANTA_USER/` returns the top level contents. 82 | 83 | $ mls /$MANTA_USER/ 84 | public/ 85 | stor/ 86 | 87 | The shortcut `~~` is equivalent to typing `/$MANTA_USER`. Since many operations 88 | require full Manta paths, you'll find it useful. We will use it for the 89 | remainder of this document. 90 | 91 | $ mls ~~/ 92 | public/ 93 | stor/ 94 | 95 | 96 | # CLI 97 | 98 | The command-line tools for Manta's directory-style API are generally analogs 99 | of common Unix tools: `mls` is similar to `ls`, `mmkdir` for `mkdir`, 100 | `mrm` for `rm`. See the [Command Reference](./commands-reference.md) for docs 101 | on each of these tools. 102 | 103 | ## Objects 104 | 105 | Objects are the main entity you will use. An object is non-interpreted data of 106 | any size that you read and write to the store. Objects are immutable. You cannot 107 | append to them or edit them in place. When you overwrite an object, you 108 | completely replace it. 109 | 110 | By default, objects are replicated to two physical servers, but you can specify 111 | between one and six copies, depending on your needs. You will be charged for the 112 | number of bytes you consume, so specifying one copy is half the price of two, 113 | with the trade-off being a decrease in potential durability and availability. 114 | 115 | For more complete coverage of how Manta stores objects, see the 116 | [Storage Reference](./storage-reference.md). 117 | 118 | When you write an object, you give it a name. Object names (keys) look like Unix 119 | file paths. This is how you would create an object named `~~/stor/hello-foo` 120 | that contains the data in the file "hello.txt": 121 | 122 | $ echo "Hello, Manta" > /tmp/hello.txt 123 | $ mput -f /tmp/hello.txt ~~/stor/hello-foo 124 | .../stor/hello-foo [==========================>] 100% 13B 125 | 126 | $ mget ~~/stor/hello-foo 127 | Hello, Manta 128 | 129 | The service fully supports streaming uploads, so piping a file also works: 130 | 131 | $ curl -sL http://www.gutenberg.org/ebooks/120.txt.utf-8 | \ 132 | mput -H 'content-type: text/plain' ~~/stor/treasure_island.txt 133 | 134 | In the example above, we don't have a local file, so `mput` doesn't attempt to 135 | set the MIME type. To make sure our object is properly readable by a browser, we 136 | set the HTTP `Content-Type` header explicitly. 137 | 138 | ## "\~\~/stor" and "\~\~/public" 139 | 140 | Now, about `~~/stor`. Your "namespace" is `/:login/stor`. This is where all of 141 | your data that you would like to keep private is stored. You can create any 142 | number of objects and directories in this namespace without conflicting with 143 | other users. 144 | 145 | In addition to `/:login/stor`, there is also `/:login/public`, which allows for 146 | unauthenticated reads over HTTP and HTTPS. This directory is useful for 147 | you to host world-readable files, such as media assets you would use in a CDN. 148 | 149 | ## Directories 150 | 151 | All objects can be stored in Unix-like directories. `/:login/stor` is the top 152 | level private directory and `/:login/public` your top-level public directory. 153 | You can logically think of these like `/` in Unix environments. You can create 154 | any number of directories and sub-directories, but there is a limit of one 155 | million entries in a single directory. 156 | 157 | Directories are useful when you want to logically group objects (or other 158 | directories) and be able to list them. Here are a few examples of 159 | creating, listing, and deleting directories: 160 | 161 | $ mmkdir ~~/stor/stuff 162 | 163 | Without an argument, `mls` defaults to `~~/stor`: 164 | 165 | $ mls 166 | stuff/ 167 | treasure_island.txt 168 | $ mls ~~/stor/stuff 169 | $ mls -l ~~/stor 170 | drwxr-xr-x 1 john.smith 0 May 15 17:02 stuff 171 | -rwxr-xr-x 1 john.smith 391563 May 15 16:48 treasure_island.txt 172 | $ mmkdir -p ~~/stor/stuff/foo/bar/baz 173 | $ mrmdir ~~/stor/stuff/foo/bar/baz 174 | $ mrm -r ~~/stor/stuff 175 | 176 | 177 | 180 | -------------------------------------------------------------------------------- /docs/user-guide/commands-reference.md: -------------------------------------------------------------------------------- 1 | # CLI Command Reference 2 | 3 | This document lists the command-line interface (CLI) tools available from the 4 | Manta [Node.js SDK](./sdks.md#nodejs-sdk). See the [Getting 5 | Started](./#getting-started) notes for installing. 6 | 7 | * [mls](https://github.com/TritonDataCenter/node-manta/blob/master/docs/man/mls.md) - Lists directory contents 8 | * [mput](https://github.com/TritonDataCenter/node-manta/blob/master/docs/man/mput.md) - Uploads data to an object 9 | * [mget](https://github.com/TritonDataCenter/node-manta/blob/master/docs/man/mget.md) - Downloads an object from the service 10 | * [minfo](https://github.com/TritonDataCenter/node-manta/blob/master/docs/man/minfo.md) - show HTTP headers for a Manta object 11 | * [mfind](https://github.com/TritonDataCenter/node-manta/blob/master/docs/man/mfind.md) - Walks a hierarchy to find names of objects by name, size, or type 12 | * [mmkdir](https://github.com/TritonDataCenter/node-manta/blob/master/docs/man/mmkdir.md) - Make directories 13 | * [mrm](https://github.com/TritonDataCenter/node-manta/blob/master/docs/man/mrm.md) - Remove objects or directories 14 | * [mrmdir](https://github.com/TritonDataCenter/node-manta/blob/master/docs/man/mrmdir.md) - Remove empty directories 15 | * [msign](https://github.com/TritonDataCenter/node-manta/blob/master/docs/man/msign.md) - Create a signed URL to a object stored in the service 16 | * [muntar](https://github.com/TritonDataCenter/node-manta/blob/master/docs/man/muntar.md) - Create a directory hierarchy from a tar file 17 | * [mchmod](https://github.com/TritonDataCenter/node-manta/blob/master/docs/man/mchmod.md) - Change object role tags 18 | * [mchattr](https://github.com/TritonDataCenter/node-manta/blob/master/docs/man/mchattr.md) - Change object attributes 19 | -------------------------------------------------------------------------------- /docs/user-guide/rbac.md: -------------------------------------------------------------------------------- 1 | # Role Based Access Control and Manta 2 | 3 | Manta's role-based access control (RBAC) lets you limit access to Manta objects 4 | to other members of your organization. 5 | 6 | 7 | # Overview 8 | 9 | Role Based Access Control is made up of four elements: 10 | 11 | * Roles 12 | * Users 13 | * Policies 14 | * Resources 15 | 16 | We'll take them from the bottom up to see how they fit together. 17 | 18 | **Resources** are the things you want to grant access to. 19 | In Manta resources are Manta objects. 20 | 21 | **Policies** are lists of rules that describe access to resources. 22 | The rules are written in a human readable language 23 | that describes the action that is allowed 24 | and the context in which that rule is valid. 25 | 26 | The default policy for all objects is to deny access always. 27 | Rules are written in terms of what is allowed. 28 | For example, the following rules say that 29 | getting a Manta object and listing a Manta directory is allowed: 30 | 31 | CAN getobject 32 | CAN getdirectory 33 | 34 | 35 | **Users** are login credentials that are associated with your Triton 36 | account. While each user name must be unique within an account, 37 | user names do not need to be globally unique. 38 | 39 | If there is a Triton account named `bigco` and another one named `littleco`, 40 | both can have a user named `contractor`. 41 | 42 | When making REST API requests as a user, simply specify the account in the 43 | `keyId` of the signature in the form of `account/user`. Specifically, 44 | 45 | an account keyId has this format: 46 | 47 | bigco/keys/72:b1:da:b6:3f:3e:67:40:53:ca:c9:ab:0d:c4:2a:f7 48 | 49 | whereas a subuser's a keyId looks like this: 50 | 51 | bigco/contractor/keys/a1:b2:c3:d4:e5:f6:a7:b8:c9:d0:e1:f2:a3:b4:c5:d6 52 | 53 | Manta tools use the following environment variables 54 | to make working with accounts and users easier. 55 | 56 | * `MANTA_USER` is the account owner. 57 | * `MANTA_SUBUSER` is a user within the account. 58 | 59 | **Roles** bring users, policies, and resources together. 60 | Roles define what users can do with a resource. For example, a role 61 | containing a policy with the `CAN getobject` action enables the members 62 | of the role to make `GET` object requests. 63 | 64 | To allow the access to a specific resource, 65 | you also need to associate, or tag, the resource with a role and add 66 | authorized users as members of the role. You can tag or untag roles 67 | for a resource by updating the 'role-tag' attribute value in the 68 | object metadata. See the [mchmod](https://github.com/TritonDataCenter/node-manta/blob/master/docs/man/mchmod.md) 69 | or [mchattr](https://github.com/TritonDataCenter/node-manta/blob/master/docs/man/mchattr.md) CLI reference for 70 | an example of updating object role tags or updating metadata in general. 71 | 72 | Roles can be tagged to directories as well (the only exception is the root 73 | directory of the account). Directory access allows the user to list the objects 74 | in it but does not imply permissions for objects in it. This is different from 75 | the POSIX filesystems. In other words, 76 | 77 | - Roles on the directory are not required to access objects in that directory 78 | or to list the contents of subdirectories. 79 | - Roles tagged on a directory are not automatically cascaded to the objects or 80 | subdirectories within the directory. 81 | 82 | When a user wants to access a resource, 83 | the access system checks whether the user 84 | is a default member of role(s) associated with the resource. 85 | If so, the access system checks the policies 86 | associated with the role 87 | to determine whether the user can access the resource. 88 | 89 | If a specific role is passed in the request header (e.g. `-H "Role: operator"`), 90 | the access system will evaluate the permissions for that specific role only. 91 | 92 | The account owner always has complete access to every resource in the account. 93 | 94 | 95 | # Learning More About Access Control 96 | 97 | To learn more see the main [RBAC documentation](https://docs.joyent.com/public-cloud/rbac). 98 | 99 | If you want a quick walkthrough of how access control works with Manta, 100 | see [Getting Started With Access Control](https://docs.joyent.com/public-cloud/rbac/quickstart). 101 | 102 | If you are already familiar with the key RBAC concepts, you can review the 103 | complete list of [Manta Actions](https://docs.joyent.com/public-cloud/rbac/rules#manta-actions) 104 | and start defining the rules for your RBAC policies. 105 | -------------------------------------------------------------------------------- /docs/user-guide/sdks.md: -------------------------------------------------------------------------------- 1 | # SDKs and CLI Tools 2 | 3 | There are several language packages (Software Development Kits, SDKs) for Manta. 4 | The Node.js and Java SDKs are maintained by Joyent. 5 | 6 | 7 | # Node.js SDK 8 | 9 | The Node.js SDK is used to develop and to test Manta. It is the most robust SDK 10 | available for Manta. 11 | 12 | The Node.js SDK includes a command-line interface that provides several 13 | tools that let you work with Manta much as you would work with a Unix system. 14 | See the [Getting Started tutorial](./#getting-started) for an introduction. 15 | 16 | The documentation for using the Manta node.js package in node code, see 17 | [the node-manta docs](https://github.com/TritonDataCenter/node-manta/tree/master/docs). 18 | 19 | 20 | # Java SDK 21 | 22 | The Java SDK has feature parity with the Node.js SDK. However, if new features 23 | are released it may be late to implement them. It supports all object operations 24 | and HTTP signing. 25 | 26 | The Java SDK is available as a [Maven](https://maven.apache.org/) module. To use 27 | it add the following to your project's pom.xml: 28 | 29 | 30 | com.joyent.manta 31 | java-manta-client 32 | 33 | LATEST 34 | 35 | 36 | You can find the source for Java SDK at 37 | [https://github.com/TritonDataCenter/java-manta](https://github.com/TritonDataCenter/java-manta). 38 | 39 | 40 | # Other community or unsupported SDKs 41 | 42 | ## Python SDK 43 | 44 | The Python SDK is a community-maintained package for Manta, providing 45 | a "manta" package and `mantash`, a shell that lets you work with Manta in 46 | a bash-like environment. 47 | 48 | # Mantash single commands can be run like: 49 | # mantash ls 50 | # Or you can enter the mantash interactive shell and run commands from 51 | # there. Let's do that: 52 | $ mantash 53 | [jill@us-east /jill/stor]$ ls 54 | [jill@us-east /jill/stor]$ # our stor is empty 55 | [jill@us-east /jill/stor]$ put numbers.txt ./ # upload local file 56 | [jill@us-east /jill/stor]$ ls 57 | numbers.txt 58 | [jill@us-east /jill/stor]$ cat numbers.txt 59 | one 60 | two 61 | three 62 | four 63 | 64 | You can find the Python SDK at 65 | [https://github.com/TritonDataCenter/python-manta](https://github.com/TritonDataCenter/python-manta). 66 | 67 | 68 | ## Ruby SDK 69 | 70 | The Ruby SDK is a client for communicating with Manta. It is effectively 71 | an HTTP(S) wrapper which handles required HTTP headers and performs some sanity 72 | checks. The Ruby SDK seeks to expose all of Manta's features in a thin 73 | low-abstraction client. 74 | 75 | You can find the Ruby SDK at 76 | [https://github.com/TritonDataCenter/ruby-manta](https://github.com/TritonDataCenter/ruby-manta). 77 | 78 | 79 | ## mantaRSDK 80 | 81 | The R SDK is an interactive client in the form of an R package, that exposes all 82 | of Manta's features, and supports R on Unix, Linux and Windows. 83 | 84 | You can find the mantaRSDK at 85 | [https://github.com/TritonDataCenter/mantaRSDK](https://github.com/TritonDataCenter/mantaRSDK) 86 | and Rbunyan at [https://github.com/TritonDataCenter/Rbunyan](https://github.com/TritonDataCenter/Rbunyan). 87 | 88 | 89 | ## Hadoop FileSystem Driver 90 | 91 | Hadoop can natively access Manta over the network. Most major Hadoop FileSystem 92 | features are supported. This is a community driven project, so updates to it are 93 | dependent upon community involvement. The driver is available as a stand-alone 94 | jar file that can be dropped into a Hadoop or Apache Drill installation. 95 | 96 | You can download the jar directory from the [releases 97 | page](https://github.com/TritonDataCenter/hadoop-manta/releases) 98 | on the [project's github page](https://github.com/TritonDataCenter/hadoop-manta) 99 | or directly from Maven Central. 100 | 101 | 102 | ## PHP SDK 103 | 104 | The PHP SDK supports all object operations. It is a community driven SDK, so 105 | updates to it are dependent upon community involvement. The SDK is available as 106 | a [Packagist package](https://packagist.org/packages/joyent/php-manta) and can 107 | be installed using [Composer](https://getcomposer.org/): 108 | 109 | composer require joyent/php-manta 110 | 111 | It has been tested in PHP 5.6, PHP 7.0 and HHVM. 112 | 113 | You can find the source for PHP SDK at 114 | [https://github.com/TritonDataCenter/php-manta](https://github.com/TritonDataCenter/php-manta). 115 | 116 | 117 | ## Perl Module 118 | 119 | There is a Perl module available via CPAN that supports object operations. It is 120 | a community driven module, so updates to it are dependent upon community 121 | involvement. The module is available on 122 | [CPAN](http://search.cpan.org/~andrewh/Manta-Client/). 123 | 124 | You can find the source for the Perl module at 125 | [https://github.com/TritonDataCenter/Manta-Client](https://github.com/TritonDataCenter/Manta-Client). 126 | 127 | 128 | ## Golang SDK 129 | 130 | The Go SDK for Manta is a combination SDK that provides support for Triton 131 | CloudAPI operations in addition to Manta API operations. 132 | 133 | You can find the source for the Go Manta SDK at 134 | [https://github.com/TritonDataCenter/triton-go](https://github.com/TritonDataCenter/triton-go). 135 | 136 | 137 | ## Erlang SDK 138 | 139 | The Erlang SDK for Manta is community-maintained SDK that has support for most 140 | file operations. 141 | 142 | You can find the source for the Erlang SDK at 143 | [https://github.com/TritonDataCenter/erlang-manta](https://github.com/TritonDataCenter/erlang-manta). 144 | -------------------------------------------------------------------------------- /docs/user-guide/storage-reference.md: -------------------------------------------------------------------------------- 1 | # Object Storage Reference 2 | 3 | Manta uses a REST API to read, write, and delete objects. This document assumes 4 | that you are familiar with HTTP-based REST systems, including HTTP requests, 5 | responses, status codes, and headers. 6 | 7 | If you want to start with basic information on Manta, see the 8 | [Getting Started](./#getting-started) tutorial. 9 | 10 | Unless otherwise specified, the semantics described here are stable, which means 11 | that you can expect that future updates will not change the documented behavior. 12 | You should avoid relying on behavior not specified here. 13 | 14 | 15 | # Storage Overview 16 | 17 | The storage service is based on two concepts: object and directories. 18 | 19 | * **Objects** consist of data and metadata you can read, write, and delete from 20 | the storage service. The data portion is opaque. The metadata is a set of 21 | HTTP headers that describe the object, such as `Content-Type` and 22 | `Content-MD5`. An object is identified by a name. 23 | * **Directories** are named groups of objects, as on traditional file systems. 24 | Every object belongs to a directory. 25 | The private storage directory, `/:login/stor` functions as the top level, or 26 | root directory. 27 | 28 | 29 | # Objects 30 | 31 | Objects are the primary entity you store in Manta. 32 | Objects can be of any size, including zero bytes. 33 | Objects consist of your raw, uninterpreted data, 34 | as well as the metadata (HTTP headers) returned when you retrieve an object. 35 | 36 | # Headers 37 | There are several headers for objects that control HTTP semantics in Manta. 38 | 39 | ## Content Length 40 | 41 | When you write an object, you must use one of two headers: 42 | 43 | * Use `Content-Length` if you can specify the object size in bytes. 44 | * Use `transfer-encoding: chunked` to upload objects using [HTTP chunked encoding](http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.6). 45 | 46 | Chunked encoding lets you stream an object for storage without knowing the size of the object ahead of time. 47 | By default, the maximum amount of data you can send this way is 5GB. 48 | You can use the optional `max-content-length` header to specify how much space you estimate the object requires. 49 | This estimate is only an upper bound. 50 | The system will record how much data you *actually* transferred and record that. 51 | Subsequent GET requests will return the actual size of the object. 52 | 53 | ## 100-continue Request Header 54 | 55 | You can, but are not required to, use the 56 | [`Expect: 100-continue`](http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html#sec8.2.3) 57 | header in your write requests. 58 | Using this header saves network bandwidth. 59 | If the write request would fail, the system returns an error without transferring any data. 60 | The node-manta CLI use this feature. 61 | 62 | ## Content Headers 63 | 64 | You should always specify a `Content-Type` header, 65 | which will be stored and returned back (HTTP content-negotiation will be handled). 66 | If you do not specify a content type, the default is `application/octet-stream`. 67 | 68 | If you specify a `Content-MD5` header, the system validates that the content 69 | uploaded matches the value of the header. You must encode MD5 headers in Base64, 70 | as described in [RFC 1864](https://www.ietf.org/rfc/rfc1864.txt). 71 | 72 | 73 | The `durability-level` header is a value from 1 to 6 74 | that specifies how many copies of an object the system stores. 75 | If you do not specify a durability level, the default is 2. 76 | While you can set a durability-level of 1, doing so would put your data at 77 | a higher risk of loss. 78 | 79 | ## Conditional Request Headers 80 | 81 | The system honors the standard HTTP conditional requests such as 82 | [`If-Match`](http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.24), 83 | [`If-Modified-Since`](http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.25), 84 | etc. 85 | 86 | ## CORS Headers 87 | 88 | Cross-Origin Resource Sharing [CORS](http://www.w3.org/TR/cors/) headers are 89 | supported on a per object basis. 90 | 91 | If `access-control-allow-origin` is sent on a `PUT`, 92 | it can be a comma separated 93 | list of `origin` values. 94 | When a request is sent with the `origin` header, 95 | the *list* of values of the stored `access-control-allow-origin` header is processed 96 | and only the *matching* value is returned, if any. For example: 97 | 98 | $ echo "foo" | \ 99 | mput -q -H 'access-control-allow-origin: foo.com,bar.com' /:login/public/foo 100 | $ curl -is -X HEAD -H 'origin: foo.com' http://10.2.121.5/:login/public/foo 101 | HTTP/1.1 200 OK 102 | Connection: close 103 | Etag: f7c79088-d70d-4725-b716-7b85a40ede6a 104 | Last-Modified: Fri, 17 May 2013 20:04:51 GMT 105 | access-control-allow-origin: foo.com 106 | Content-MD5: 07BzhNET7exJ6qYjitX/AA== 107 | Durability-Level: 2 108 | Content-Length: 4 109 | Content-Type: application/octet-stream 110 | Date: Fri, 17 May 2013 20:05:58 GMT 111 | Server: Manta 112 | x-request-id: 30afb030-bf2d-11e2-be7d-99e967737d07 113 | x-response-time: 7 114 | x-server-name: fef8c5b8-3483-458f-95dc-7d9172ecefd1 115 | 116 | If no `origin` header is sent, the system assumes that the request did not originate from 117 | a browser and the original list of values is echoed back. 118 | While this behavior does not conform to the CORS specification, 119 | it does allow you to administratively see 120 | what is stored on your object. 121 | 122 | `access-control-expose-headers` is supported as a list of HTTP headers that a 123 | browser will expose. This list is not interpreted by the system. 124 | 125 | `access-control-allow-methods` is supported as a list of HTTP methods that the system 126 | will honor for this request. You can only specify HTTP operations the system 127 | supports: HEAD, GET, PUT, DELETE. 128 | 129 | `access-control-max-age` is supported and uninterpreted by the system. 130 | 131 | ## Cache-Control 132 | 133 | The HTTP `cache-control` header is stored and returned by the system. 134 | This is useful for controlling how long CDNs or a web caching agent caches a version of 135 | an object. 136 | 137 | ## Custom Headers 138 | 139 | You can store custom headers with your object by prefixing them with `m-`. 140 | For example, you might use the header `m-local-user: jill` to tag an object 141 | with the name of the user who created it. 142 | 143 | You can use up to 4 KB of header data. 144 | 145 | # Directories 146 | 147 | Directories contain objects and other directories. 148 | All objects are stored at the top level or subdirectory one of the following directories: 149 | 150 | | Directory | Description | 151 | | ---------------- | ----------- | 152 | | `/:login/stor` | private object storage | 153 | | `/:login/public` | public object storage | 154 | 155 | ## Private Storage (/:login/stor) 156 | 157 | As noted above, `/:login/stor` functions as the top level, or root, directory where you store 158 | objects and create directories. 159 | Only you can read, write, and delete data here. 160 | You can create any number of directories and objects in this directory. 161 | 162 | While the system does not yet support discretionary 163 | access controls on objects or directories, you can grant access to individual 164 | objects in this namespace by using signed URLs, which are explained below. 165 | 166 | With the exception of signed URL requests, all traffic to `/:login/stor` must be 167 | made over a secure channel (TLS). 168 | 169 | ## Public Storage (/:login/public) 170 | 171 | `/:login/public` is a world-readable namespace. 172 | Only you can create and delete objects in this directory. 173 | Read access to objects in this namespace is available through HTTP and HTTPS without 174 | authorization headers. 175 | Deletions and writes to this directory must made over a secure channel. 176 | 177 | ## Working with Directories 178 | 179 | You create a directory the same way that you create an object, 180 | but you use the special header `Content-Type: application/json; type=directory`. 181 | 182 | When you retrieve a directory, 183 | the response has the `Content-Type: application/x-json-stream; type=directory` header. 184 | The body consists of a set of JSON objects separated by newlines (`\n`). 185 | Each object has a `type` field that indicates whether the JSON object specifies a 186 | directory or a storage object. 187 | 188 | Here is an example with additional newlines added for clarity. 189 | 190 | { 191 | "name": "1c1bf695-230d-490e-aec7-3b11dff8ef32", 192 | "type": "directory", 193 | "mtime": "2012-09-11T20:28:30Z" 194 | } 195 | 196 | { 197 | "name": "695d5de6-45f4-4156-b6b7-3a8d4af89391", 198 | "etag": "bdf0aa96e3bb87148be084252a059736", 199 | "size": 44, 200 | "type": "object", 201 | "mtime": "2012-09-11T20:28:31Z" 202 | } 203 | 204 | | Field | Description | 205 | | ------- | ----------- | 206 | | `type` | Either `object` or `directory`. | 207 | | `name` | The name of the object or directory. | 208 | | `mtime` | An [ISO 8601 timestamp](http://www.w3.org/TR/NOTE-datetime) of the last update time of the object or directory. | 209 | | `size` | Present only if `type` is `object`. The size of the object in bytes. | 210 | | `etag` | Present only if `type` is `object`. Used for conditional requests. | 211 | 212 | 213 | When you use an HTTP GET request to list a directory, 214 | the `result-set-size` header in the response contains the *total* number of entries in the directory. 215 | However, you will get 256 entries per request, so you will have to paginate through the result sets. 216 | You can increase the number of entries per request to 1024. 217 | Results are sorted lexicographically. 218 | 219 | To get the next page of a listing, pass in the *last* name returned in the set 220 | until the total number of entries you have processed matches `result-set-size`. 221 | 222 | 223 | You can store CORS, `cache-control` and `m-` headers on directories, as you can 224 | on objects. Currently, no data is supported on directories. 225 | 226 | 227 | # Storage System Architecture 228 | 229 | This section describes some of the design principles that guide the operation of 230 | Manta. 231 | 232 | ## Guiding Principles 233 | 234 | Several principles guide the design of the service: 235 | 236 | * From the perspective of the [CAP theorem](http://en.wikipedia.org/wiki/CAP_theorem), 237 | the system is *strongly consistent*. 238 | It chooses to be strongly consistent, at 239 | the risk of more HTTP 500 errors than an eventually consistent system. 240 | This system is engineered to minimize errors in the event of network or system 241 | failures and to recover as quickly as possible, but more errors will occur than 242 | in an eventually consistent system. However, it is possible to read the writes 243 | immediately. The distinction between a HTTP 404 response and a HTTP 500 response is very clear: 244 | A 404 response *really* means your data isn't there. 245 | A 500 response means that it might be, but there is some sort of outage. 246 | * When the system responds with an HTTP 200, you can be certain your data is 247 | durably stored on the number of servers you requested. The system is designed to 248 | *never* allow data loss or corruption. 249 | * The system is designed to be secure. All writes must be performed over a 250 | secure channel (TLS). Most reads will be as well, unless you are specifically 251 | requesting to bypass TLS for browser/web channels. 252 | 253 | ## System scale 254 | 255 | Manta is designed to support an arbitrarily large number of objects and an 256 | arbitrarily large number of directories. However, it bounds the number of 257 | objects in a single directory so that list operations can be performed 258 | efficiently. 259 | 260 | The system does not have any limit on the size of a single object, but it may 261 | return a "no space" error if the requested object size is larger than a single 262 | physical server has space for. In practice, this number will be in tens of 263 | terabytes, but network transfer times make object sizes of that magnitude 264 | unreasonable anyway. 265 | 266 | There is no default API rate limit imposed upon you, however the system reserves the 267 | right to throttle requests if necessary to protect the system. For high-volume 268 | web assets, you should use it as a content delivery network (CDN) origin. 269 | 270 | All REST APIs are modeled as streams. They are designed to let you iterate 271 | through result sets without consuming too much memory. For example, listing 272 | a directory returns newline separated JSON objects as opposed to an array or 273 | large XML document. 274 | 275 | ## Durability 276 | 277 | At a simple level, durability is a system's ability to tolerate failures without 278 | a loss of data. By default, the system stores two copies of your object and these 279 | two copies are placed in two different data centers. Distributing copies of your 280 | objects reduces the risk of data loss in the event of a failure. The system 281 | relies on ZFS RAID-Z to store your objects, so the durability is actually greater 282 | than two would imply. You can store anywhere from 1 to 6 copies. 283 | 284 | You are billed for exactly the number of bytes you consume in the system. 285 | For example, if you write a 1MB object with the default number of copies (2), 286 | you will be billed for 2MB of storage each month. When the number of copies 287 | requested is greater than one, the system ensures that *at least* two copies 288 | are placed in two different data centers, and then stripes the other copies 289 | across data centers. 290 | 291 | If any given data center is down at the time, you may have copies unbalanced with 292 | extra replicas in fewer data centers, but there will always be at least two data 293 | centers with your copy of data. This allows you to still access your data in the 294 | event of any one data center failure. 295 | -------------------------------------------------------------------------------- /package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "manta-project", 3 | "version": "1.0.1", 4 | "description": "Manta project landing repo", 5 | "private": true, 6 | "devDependencies": { 7 | "doctoc": "~1.3.0" 8 | } 9 | } 10 | -------------------------------------------------------------------------------- /tools/jr-manifest.json: -------------------------------------------------------------------------------- 1 | { 2 | "jrVersion": 1, 3 | "description": "Manta public repositories", 4 | "repoCandidateSearch": { 5 | "description": "public github.com/TritonDataCenter repos directly relevant to development of Manta", 6 | "type": "public", 7 | "includeArchived": false 8 | }, 9 | "blessedLabels": [ 10 | { 11 | "name": "meta", 12 | "type": "boolean", 13 | "description": "a repo that isn't code, but is relevant for product development" 14 | }, 15 | { 16 | "name": "release", 17 | "type": "boolean", 18 | "description": "a top-level repo for a Mantav2 release component" 19 | }, 20 | { 21 | "name": "image", 22 | "type": "string", 23 | "description": "the VM image name produced from this repo for Mantav2, if any" 24 | }, 25 | { 26 | "name": "mantaservice", 27 | "type": "string", 28 | "description": "the Mantav2 service name, if this is a top-level Manta service repo" 29 | }, 30 | { 31 | "name": "mantav1", 32 | "type": "boolean", 33 | "description": "Repositories are related to mantav1. See https://github.com/TritonDataCenter/manta/blob/master/docs/mantav2.md" 34 | }, 35 | { 36 | "name": "mantav1branch", 37 | "type": "boolean", 38 | "description": "Repositories which were branched for mantav1." 39 | }, 40 | { 41 | "name": "mantav1only", 42 | "type": "boolean", 43 | "description": "Repositories which are only needed for mantav1." 44 | } 45 | ], 46 | "defaults": { 47 | "labels": { 48 | "manta": true, 49 | "public": true 50 | } 51 | }, 52 | "repositories": [ 53 | { 54 | "name": "MANTA-3788" 55 | }, 56 | { 57 | "name": "Manta-Client" 58 | }, 59 | { 60 | "name": "aperture-config" 61 | }, 62 | { 63 | "name": "binder", 64 | "labels": { 65 | "release": true, 66 | "vm": true, 67 | "mg": "binder", 68 | "mantaservice": "nameservice", 69 | "image": "mantav2-nameservice", 70 | "mantav1": true, 71 | "mantav1branch": true 72 | } 73 | }, 74 | { 75 | "name": "catest" 76 | }, 77 | { 78 | "name": "commons-vfs-manta" 79 | }, 80 | { 81 | "name": "cosbench" 82 | }, 83 | { 84 | "name": "electric-moray", 85 | "labels": { 86 | "release": true, 87 | "vm": true, 88 | "mg": "electric-moray", 89 | "mantaservice": "electric-moray", 90 | "image": "mantav2-electric-moray", 91 | "mantav1": true, 92 | "mantav1branch": true 93 | } 94 | }, 95 | { 96 | "name": "eng", 97 | "labels": { 98 | "meta": true 99 | } 100 | }, 101 | { 102 | "name": "erlang-manta" 103 | }, 104 | { 105 | "name": "fsnotify" 106 | }, 107 | { 108 | "name": "gocommon" 109 | }, 110 | { 111 | "name": "gomanta" 112 | }, 113 | { 114 | "name": "gosign" 115 | }, 116 | { 117 | "name": "grafana" 118 | }, 119 | { 120 | "name": "hadoop-manta" 121 | }, 122 | { 123 | "name": "haproxy-1.4" 124 | }, 125 | { 126 | "name": "haproxy-1.5" 127 | }, 128 | { 129 | "name": "haproxy-1.8" 130 | }, 131 | { 132 | "name": "java-http-signature" 133 | }, 134 | { 135 | "name": "java-manta" 136 | }, 137 | { 138 | "name": "java-manta-test-harness" 139 | }, 140 | { 141 | "name": "jenkins-joylib" 142 | }, 143 | { 144 | "name": "joyent-repos" 145 | }, 146 | { 147 | "name": "kang" 148 | }, 149 | { 150 | "name": "mahi", 151 | "labels": { 152 | "release": true, 153 | "vm": true, 154 | "mg": "mahi", 155 | "mantaservice": "authcache", 156 | "image": "mantav2-authcache", 157 | "mantav1": true, 158 | "mantav1branch": true 159 | } 160 | }, 161 | { 162 | "name": "mako-gc-feeder" 163 | }, 164 | { 165 | "name": "mako-regional-report" 166 | }, 167 | { 168 | "name": "manatee" 169 | }, 170 | { 171 | "name": "manatee-state-machine" 172 | }, 173 | { 174 | "name": "manta", 175 | "labels": { 176 | "meta": true, 177 | "mantav1": true, 178 | "mantav1branch": true 179 | } 180 | }, 181 | { 182 | "name": "manta-buckets-api", 183 | "labels": { 184 | "release": true, 185 | "vm": true, 186 | "mg": "buckets-api", 187 | "mantaservice": "buckets-api", 188 | "image": "mantav2-buckets-api" 189 | } 190 | }, 191 | { 192 | "name": "manta-buckets-mdapi", 193 | "labels": { 194 | "vm": true, 195 | "release": true, 196 | "mantaservice": "buckets-mdapi", 197 | "image": "mantav2-buckets-mdapi" 198 | } 199 | }, 200 | { 201 | "name": "manta-buckets-mdplacement", 202 | "labels": { 203 | "vm": true, 204 | "release": true, 205 | "mantaservice": "buckets-mdplacement", 206 | "image": "mantav2-buckets-mdplacement" 207 | } 208 | }, 209 | { 210 | "name": "manta-chum" 211 | }, 212 | { 213 | "name": "manta-compute-bin", 214 | "labels": { 215 | "mantav1": true, 216 | "mantav1branch": true, 217 | "mantav1only": true 218 | } 219 | }, 220 | { 221 | "name": "manta-debugging-guide" 222 | }, 223 | { 224 | "name": "manta-finder" 225 | }, 226 | { 227 | "name": "manta-garbage-collector", 228 | "labels": { 229 | "release": true, 230 | "vm": true, 231 | "mantaservice": "garbage-collector", 232 | "mg": "manta-garbage-collector", 233 | "image": "mantav2-garbage-collector", 234 | "mantav1": true, 235 | "mantav1branch": true 236 | } 237 | }, 238 | { 239 | "name": "manta-genazconfig" 240 | }, 241 | { 242 | "name": "manta-hk" 243 | }, 244 | { 245 | "name": "manta-mackerel", 246 | "labels": { 247 | "release": true, 248 | "mg": "mackerel", 249 | "mantav1": true, 250 | "mantav1branch": true 251 | } 252 | }, 253 | { 254 | "name": "manta-madtom", 255 | "labels": { 256 | "release": true, 257 | "vm": true, 258 | "mantaservice": "madtom", 259 | "mg": "madtom", 260 | "image": "mantav2-madtom", 261 | "mantav1": true, 262 | "mantav1branch": true 263 | } 264 | }, 265 | { 266 | "name": "manta-mako", 267 | "labels": { 268 | "release": true, 269 | "vm": true, 270 | "mantaservice": "storage", 271 | "mg": "mako", 272 | "image": "mantav2-storage", 273 | "mantav1": true, 274 | "mantav1branch": true 275 | } 276 | }, 277 | { 278 | "name": "manta-manatee", 279 | "labels": { 280 | "release": true, 281 | "vm": true, 282 | "mantaservice": "postgres", 283 | "mg": "manta-manatee", 284 | "image": "mantav2-postgres", 285 | "mantav1": true, 286 | "mantav1branch": true 287 | } 288 | }, 289 | { 290 | "name": "manta-marlin", 291 | "labels": { 292 | "vm": true, 293 | "mantav1": true, 294 | "mantav1branch": true, 295 | "mantav1only": true 296 | } 297 | }, 298 | { 299 | "name": "manta-marlin-dashboard", 300 | "labels": { 301 | "vm": true, 302 | "mantav1": true, 303 | "mantav1branch": true, 304 | "mantav1only": true 305 | } 306 | }, 307 | { 308 | "name": "manta-mdshovel" 309 | }, 310 | { 311 | "name": "manta-mdu" 312 | }, 313 | { 314 | "name": "manta-medusa", 315 | "labels": { 316 | "vm": true, 317 | "mantav1": true, 318 | "mantav1branch": true, 319 | "mantav1only": true 320 | } 321 | }, 322 | { 323 | "name": "manta-minnow", 324 | "labels": { 325 | "release": true, 326 | "mg": "minnow", 327 | "mantav1": true, 328 | "mantav1branch": true 329 | } 330 | }, 331 | { 332 | "name": "manta-mlive" 333 | }, 334 | { 335 | "name": "manta-mola", 336 | "labels": { 337 | "release": true, 338 | "vm": true, 339 | "mg": "mola", 340 | "mantaservice": "ops", 341 | "image": "mantav2-ops", 342 | "mantav1": true, 343 | "mantav1branch": true 344 | } 345 | }, 346 | { 347 | "name": "manta-monitor" 348 | }, 349 | { 350 | "name": "manta-monitor-deploy-stack" 351 | }, 352 | { 353 | "name": "manta-mprune" 354 | }, 355 | { 356 | "name": "manta-muskie", 357 | "labels": { 358 | "release": true, 359 | "vm": true, 360 | "mg": "muskie", 361 | "image": "mantav2-webapi", 362 | "mantaservice": "webapi", 363 | "mantav1": true, 364 | "mantav1branch": true 365 | } 366 | }, 367 | { 368 | "name": "manta-physusage" 369 | }, 370 | { 371 | "name": "manta-rebalancer", 372 | "labels": { 373 | "release": true, 374 | "vm": true, 375 | "mg": "rebalancer", 376 | "mantaservice": "rebalancer", 377 | "image": "mantav2-rebalancer" 378 | } 379 | }, 380 | { 381 | "name": "manta-reshard", 382 | "labels": { 383 | "release": true, 384 | "vm": true, 385 | "mg": "manta-reshard", 386 | "mantaservice": "reshard", 387 | "image": "mantav2-reshard", 388 | "mantav1": true, 389 | "mantav1branch": true 390 | } 391 | }, 392 | { 393 | "name": "manta-scripts" 394 | }, 395 | { 396 | "name": "manta-sharkspotter" 397 | }, 398 | { 399 | "name": "manta-show-changes" 400 | }, 401 | { 402 | "name": "manta-storinfo", 403 | "labels": { 404 | "release": true, 405 | "vm": true, 406 | "mg": "storinfo", 407 | "mantaservice": "storinfo", 408 | "image": "mantav2-storinfo" 409 | } 410 | }, 411 | { 412 | "name": "manta-thoth" 413 | }, 414 | { 415 | "name": "manta-tools-deck" 416 | }, 417 | { 418 | "name": "manta-wrasse", 419 | "labels": { 420 | "vm": true, 421 | "mantav1": true, 422 | "mantav1branch": true, 423 | "mantav1only": true 424 | } 425 | }, 426 | { 427 | "name": "mantaRSDK" 428 | }, 429 | { 430 | "name": "mname-balancer" 431 | }, 432 | { 433 | "name": "moray", 434 | "labels": { 435 | "release": true, 436 | "vm": true, 437 | "mg": "moray", 438 | "mantaservice": "moray", 439 | "image": "mantav2-moray", 440 | "mantav1": true, 441 | "mantav1branch": true 442 | } 443 | }, 444 | { 445 | "name": "muppet", 446 | "labels": { 447 | "release": true, 448 | "vm": true, 449 | "mg": "muppet", 450 | "mantaservice": "loadbalancer", 451 | "image": "mantav2-loadbalancer", 452 | "mantav1": true, 453 | "mantav1branch": true 454 | } 455 | }, 456 | { 457 | "name": "node-aperture" 458 | }, 459 | { 460 | "name": "node-artedi" 461 | }, 462 | { 463 | "name": "node-buckets-mdapi" 464 | }, 465 | { 466 | "name": "node-checker" 467 | }, 468 | { 469 | "name": "node-contract", 470 | "labels": { 471 | "mantav1": true, 472 | "mantav1only": true 473 | } 474 | }, 475 | { 476 | "name": "node-crc" 477 | }, 478 | { 479 | "name": "node-cueball" 480 | }, 481 | { 482 | "name": "node-fash" 483 | }, 484 | { 485 | "name": "node-fast" 486 | }, 487 | { 488 | "name": "node-fast-messages" 489 | }, 490 | { 491 | "name": "node-haproxy-log" 492 | }, 493 | { 494 | "name": "node-hyprlofs" 495 | }, 496 | { 497 | "name": "node-libmanta" 498 | }, 499 | { 500 | "name": "node-mahi" 501 | }, 502 | { 503 | "name": "node-manta" 504 | }, 505 | { 506 | "name": "node-mname" 507 | }, 508 | { 509 | "name": "node-mname-client" 510 | }, 511 | { 512 | "name": "node-mooremachine" 513 | }, 514 | { 515 | "name": "node-moray" 516 | }, 517 | { 518 | "name": "node-moray-filter" 519 | }, 520 | { 521 | "name": "node-moray-sandbox" 522 | }, 523 | { 524 | "name": "node-pg_dump" 525 | }, 526 | { 527 | "name": "node-sdc-clients" 528 | }, 529 | { 530 | "name": "node-smartdc-auth" 531 | }, 532 | { 533 | "name": "node-spawn-async" 534 | }, 535 | { 536 | "name": "node-storinfo" 537 | }, 538 | { 539 | "name": "node-watershed" 540 | }, 541 | { 542 | "name": "node-zookeeper-client" 543 | }, 544 | { 545 | "name": "node-zsock-async", 546 | "labels": { 547 | "mantav1": true, 548 | "mantav1only": true 549 | } 550 | }, 551 | { 552 | "name": "pg_prefaulter" 553 | }, 554 | { 555 | "name": "pgsqlstat" 556 | }, 557 | { 558 | "name": "pgstatsmon", 559 | "labels": { 560 | "release": true, 561 | "vm": true, 562 | "mg": "pgstatsmon", 563 | "mantaservice": "pgstatsmon", 564 | "image": "mantav2-pgstatsmon", 565 | "mantav1": true, 566 | "mantav1branch": true 567 | } 568 | }, 569 | { 570 | "name": "postgres" 571 | }, 572 | { 573 | "name": "postgres-loadgen" 574 | }, 575 | { 576 | "name": "prometheus" 577 | }, 578 | { 579 | "name": "prr" 580 | }, 581 | { 582 | "name": "python-manta" 583 | }, 584 | { 585 | "name": "redfish-util" 586 | }, 587 | { 588 | "name": "registrar" 589 | }, 590 | { 591 | "name": "restdown-brand-remora" 592 | }, 593 | { 594 | "name": "rfd", 595 | "labels": { 596 | "meta": true 597 | } 598 | }, 599 | { 600 | "name": "rfe", 601 | "labels": { 602 | "meta": true 603 | } 604 | }, 605 | { 606 | "name": "ruby-manta" 607 | }, 608 | { 609 | "name": "rust-cueball" 610 | }, 611 | { 612 | "name": "rust-cueball-dns-resolver" 613 | }, 614 | { 615 | "name": "rust-cueball-manatee-primary-resolver" 616 | }, 617 | { 618 | "name": "rust-cueball-postgres-connection" 619 | }, 620 | { 621 | "name": "rust-cueball-static-resolver" 622 | }, 623 | { 624 | "name": "rust-cueball-tcp-stream-connection" 625 | }, 626 | { 627 | "name": "rust-fast" 628 | }, 629 | { 630 | "name": "rust-illumos-priv" 631 | }, 632 | { 633 | "name": "rust-libmanta" 634 | }, 635 | { 636 | "name": "rust-manta" 637 | }, 638 | { 639 | "name": "rust-moray" 640 | }, 641 | { 642 | "name": "rust-muppet" 643 | }, 644 | { 645 | "name": "rust-quickcheck-helpers" 646 | }, 647 | { 648 | "name": "rust-sharkspotter" 649 | }, 650 | { 651 | "name": "rust-triton-clients" 652 | }, 653 | { 654 | "name": "rust-utils" 655 | }, 656 | { 657 | "name": "sastopo2svg" 658 | }, 659 | { 660 | "name": "sdc-config-agent" 661 | }, 662 | { 663 | "name": "sdc-manatee" 664 | }, 665 | { 666 | "name": "sdc-manta", 667 | "labels": { 668 | "release": true, 669 | "vm": true, 670 | "mg": "manta-deployment", 671 | "tritonservice": "manta", 672 | "image": "mantav2-deployment", 673 | "mantav1": true, 674 | "mantav1branch": true 675 | } 676 | }, 677 | { 678 | "name": "sdcnode" 679 | }, 680 | { 681 | "name": "topo-digraph-xml" 682 | }, 683 | { 684 | "name": "triton-origin-image" 685 | }, 686 | { 687 | "name": "triton-prometheus", 688 | "labels": { 689 | "release": true, 690 | "mantaservice": "prometheus", 691 | "image": "mantav2-prometheus", 692 | "mg": "prometheus", 693 | "vm": true, 694 | "mantav1": true, 695 | "mantav1branch": true 696 | } 697 | }, 698 | { 699 | "name": "zookeeper-common" 700 | } 701 | ], 702 | "excludedRepositories": [ 703 | "Cassandra-Operator", 704 | "MALT", 705 | "Rbunyan", 706 | "UpdatablePriorityQueue.js", 707 | "acpica", 708 | "alpine-lx-brand-image-builder", 709 | "ayyaz-tests-networks", 710 | "blip", 711 | "bugview", 712 | "buildymcbuildface", 713 | "cbstatsmon", 714 | "centos-image-tests", 715 | "centos-lx-brand-image-builder", 716 | "certificate-expiry-monitor", 717 | "chef", 718 | "chyves", 719 | "cloud-init", 720 | "cloud-init-debian", 721 | "cloud-init-ubuntu", 722 | "cloud-perf-labs", 723 | "cloudapi-gql", 724 | "compute-ui-tests", 725 | "conch", 726 | "conch-docs", 727 | "conch-reporter-smartos", 728 | "conch-shell", 729 | "conch-ui", 730 | "consul", 731 | "containerpilot", 732 | "containersummit-workshop", 733 | "convertvm", 734 | "cosbench-manta", 735 | "cosbench2csv", 736 | "ctf2json", 737 | "customInsights", 738 | "daggr", 739 | "debian-image-tests", 740 | "debian-lx-brand-image-builder", 741 | "docker-machine-driver-triton", 742 | "dogpatch", 743 | "dragnet", 744 | "drone", 745 | "drone-runtime", 746 | "dtrace-runner", 747 | "dtracetools", 748 | "dtracetools-lx", 749 | "dtruss-osx", 750 | "epoll-test-suite", 751 | "eslint-config-joyent-portal", 752 | "evttool", 753 | "fabio", 754 | "fedora-kvm-image-builder", 755 | "fedora-kvm-image-tests", 756 | "fedora-lx-brand-image-builder", 757 | "firewall-logger-agent", 758 | "fishbulb", 759 | "fm-log-report", 760 | "freebsd-image-tests", 761 | "freebsd-kvm-image-builder", 762 | "freebsd-live", 763 | "freebsd-vpc", 764 | "gcc", 765 | "github-enterprise-triton-image", 766 | "go", 767 | "gosdc", 768 | "hackathon-nvtool", 769 | "hapi-triton-auth", 770 | "hapi-webconsole-nav", 771 | "hogan.js", 772 | "homebrew-tap", 773 | "httpmd5", 774 | "hwgrok", 775 | "illumos-extra", 776 | "illumos-joyent", 777 | "illumos-kvm", 778 | "illumos-kvm-cmd", 779 | "illumos-libavl", 780 | "illumos-list", 781 | "illumos-virtio", 782 | "image-converter", 783 | "imagetools", 784 | "innovator-workshop", 785 | "ipmitool", 786 | "ipxe", 787 | "java-fast-md5", 788 | "java-triton", 789 | "javascriptlint", 790 | "jenkins-terraform-triton", 791 | "jsstyle", 792 | "kafka-connect-manta", 793 | "kbmd", 794 | "keyapi", 795 | "kirby-preview-base", 796 | "knife-joyent", 797 | "kosh", 798 | "leveldown", 799 | "lib9p", 800 | "libMicro", 801 | "libfaketime", 802 | "libunwind", 803 | "linux-live", 804 | "lx-brand-image-tests", 805 | "lx-tests", 806 | "machine", 807 | "manifest-creator", 808 | "manta-archiver", 809 | "manta-fspopulate", 810 | "manta-mreq", 811 | "manta-munistats", 812 | "manta-nfs", 813 | "manta-shortener-client", 814 | "manta-storspan", 815 | "manufacturing", 816 | "mdata-client", 817 | "mdata-docs", 818 | "mdb_go", 819 | "mdb_v8", 820 | "mdb_v8_testing", 821 | "me-centos", 822 | "me-freebsd", 823 | "mesos", 824 | "mesos-dockerfiles", 825 | "mesoscon-eu-2015-hackathon", 826 | "mfsbsd", 827 | "mi-cassandra", 828 | "mi-centos", 829 | "mi-centos-7", 830 | "mi-centos-bhyve-6", 831 | "mi-centos-bhyve-7", 832 | "mi-centos-hvm", 833 | "mi-chefserver", 834 | "mi-debian-8", 835 | "mi-debian-9", 836 | "mi-debian-bhyve-8", 837 | "mi-debian-bhyve-9", 838 | "mi-debian-hvm", 839 | "mi-example", 840 | "mi-freebsd-10", 841 | "mi-hadoop", 842 | "mi-java", 843 | "mi-mongodb", 844 | "mi-nodejs", 845 | "mi-percona", 846 | "mi-percona-cluster", 847 | "mi-postgresql", 848 | "mi-riak", 849 | "mi-standard", 850 | "mi-ubuntu-hybrid", 851 | "mibe", 852 | "mill", 853 | "milld-nginx", 854 | "minecrab", 855 | "minio", 856 | "mjob-cost", 857 | "mmlog", 858 | "mock-docker-registry", 859 | "mod_usdt", 860 | "moray-etl-jsonb", 861 | "moray-test-suite", 862 | "netlink", 863 | "nginx", 864 | "nhttpsnoop", 865 | "nic-audit", 866 | "node-amon-common", 867 | "node-amon-plugins", 868 | "node-amqp", 869 | "node-amqp-joyent", 870 | "node-amqpsnoop", 871 | "node-asn1", 872 | "node-assert-plus", 873 | "node-async-hook", 874 | "node-bcrypt-pbkdf", 875 | "node-bunyan-syslog", 876 | "node-catstreams", 877 | "node-cls-hooked", 878 | "node-cmdutil", 879 | "node-codel", 880 | "node-consulite", 881 | "node-ctype", 882 | "node-debug-school", 883 | "node-debugging-methodologies", 884 | "node-dns", 885 | "node-docker-file-parser", 886 | "node-docker-registry-client", 887 | "node-eslint-plugin-joyent", 888 | "node-exeunt", 889 | "node-extsprintf", 890 | "node-fdleakcheck", 891 | "node-force10", 892 | "node-forkexec", 893 | "node-fuzzstream", 894 | "node-getopt", 895 | "node-http-signature", 896 | "node-httpstream", 897 | "node-imgadm", 898 | "node-imgmanifest", 899 | "node-in-the-industry", 900 | "node-ip6addr", 901 | "node-ipd", 902 | "node-ipmi", 903 | "node-jiramark", 904 | "node-jsprim", 905 | "node-juju", 906 | "node-krill", 907 | "node-kstat", 908 | "node-libdtrace-async", 909 | "node-lockfd", 910 | "node-lomstream", 911 | "node-macaddr", 912 | "node-manatee", 913 | "node-manta-dir-watcher", 914 | "node-manta-mutex", 915 | "node-mantafs", 916 | "node-mdata-client", 917 | "node-meta-d", 918 | "node-metajob", 919 | "node-nfs", 920 | "node-oncrpc", 921 | "node-pg-lsn", 922 | "node-pgtrigger", 923 | "node-piloted", 924 | "node-png-joyent", 925 | "node-progbar", 926 | "node-qlocker", 927 | "node-remote-tester-drone", 928 | "node-restify", 929 | "node-restify-clients", 930 | "node-restify-warden", 931 | "node-sdc-changefeed", 932 | "node-skinner", 933 | "node-smartdc", 934 | "node-snmpjs", 935 | "node-splitduplex", 936 | "node-sshpk", 937 | "node-sshpk-agent", 938 | "node-stackvis", 939 | "node-statvfs", 940 | "node-strsplit", 941 | "node-tab", 942 | "node-tar", 943 | "node-timefilter", 944 | "node-timeseries-heatmap", 945 | "node-trace-event", 946 | "node-triton", 947 | "node-triton-audit-logger", 948 | "node-triton-merci-buckets", 949 | "node-triton-metrics", 950 | "node-triton-netconfig", 951 | "node-triton-package-boilerplate", 952 | "node-triton-tags", 953 | "node-ufds", 954 | "node-uname", 955 | "node-urclient", 956 | "node-vasync", 957 | "node-verror", 958 | "node-vmadm", 959 | "node-vmapi-resolver", 960 | "node-vstream", 961 | "node-vstream-json-parser", 962 | "node-wfq", 963 | "node-whyyoulittle", 964 | "node-workflow", 965 | "node-workflow-moray-backend", 966 | "node-zfile", 967 | "node-zfs", 968 | "node-zkstream", 969 | "node-zoneid", 970 | "node-zonename", 971 | "node-zsock", 972 | "node-zuora-rest", 973 | "node-zutil", 974 | "node_zuora", 975 | "node_zuora_aqua", 976 | "nodejs-advisory-board", 977 | "ohai", 978 | "oid-docs", 979 | "openbsd-kvm-image-builder", 980 | "openbsd-kvm-image-tests", 981 | "packer", 982 | "packer-builder-triton", 983 | "pg_sample", 984 | "pglockanalyze", 985 | "php-manta", 986 | "pivy", 987 | "pkgbuild", 988 | "pkgin", 989 | "pkgin-test-suite", 990 | "pkgsrc", 991 | "pkgsrc-joyent", 992 | "pkgsrc-legacy", 993 | "pkgsrc-wiki", 994 | "pkgsrc-wip", 995 | "pkgsrc.joyent.com", 996 | "play", 997 | "portsnoop", 998 | "presto-manta", 999 | "promstat", 1000 | "prsigset", 1001 | "publicsuffix-list", 1002 | "pyserial", 1003 | "rancher", 1004 | "schemas", 1005 | "sdc-adminui", 1006 | "sdc-agents-core", 1007 | "sdc-agents-installer", 1008 | "sdc-amon", 1009 | "sdc-amon-agent", 1010 | "sdc-amon-relay", 1011 | "sdc-amonadm", 1012 | "sdc-amonredis", 1013 | "sdc-assets", 1014 | "sdc-booter", 1015 | "sdc-bunyan-serializers", 1016 | "sdc-cloudapi", 1017 | "sdc-cn-agent", 1018 | "sdc-cnapi", 1019 | "sdc-designation", 1020 | "sdc-docker", 1021 | "sdc-docker-build", 1022 | "sdc-docker-build-dl", 1023 | "sdc-dockerlogger", 1024 | "sdc-firewaller-agent", 1025 | "sdc-fwapi", 1026 | "sdc-fwrule", 1027 | "sdc-hagfish-watcher", 1028 | "sdc-hammer", 1029 | "sdc-headnode", 1030 | "sdc-hermes", 1031 | "sdc-imgapi", 1032 | "sdc-imgapi-cli", 1033 | "sdc-napi", 1034 | "sdc-napi-ufds-watcher", 1035 | "sdc-nat", 1036 | "sdc-net-agent", 1037 | "sdc-nfs", 1038 | "sdc-nfsserver", 1039 | "sdc-papi", 1040 | "sdc-portolan", 1041 | "sdc-portolan-moray", 1042 | "sdc-rabbitmq", 1043 | "sdc-sapi", 1044 | "sdc-scripts", 1045 | "sdc-sdc", 1046 | "sdc-securetoken", 1047 | "sdc-smart-login", 1048 | "sdc-system-tests", 1049 | "sdc-ufds", 1050 | "sdc-ur-agent", 1051 | "sdc-vm-agent", 1052 | "sdc-vmapi", 1053 | "sdc-vmtools", 1054 | "sdc-vmtools-lx-brand", 1055 | "sdc-vmtools-windows", 1056 | "sdc-volapi", 1057 | "sdc-wf-client", 1058 | "sdc-wf-shared", 1059 | "sdc-workflow", 1060 | "sdcadm", 1061 | "sdev-tests", 1062 | "smartmachine_cookbooks", 1063 | "smartos-changelog", 1064 | "smartos-dev-tools", 1065 | "smartos-docs", 1066 | "smartos-image-tests", 1067 | "smartos-live", 1068 | "smartos-netboot-server", 1069 | "smartos-vmtools", 1070 | "smartos_cookbooks", 1071 | "smfgen", 1072 | "smtools", 1073 | "snoopdump", 1074 | "sshpk-crypt", 1075 | "statemap", 1076 | "summit-workshop", 1077 | "syslinux", 1078 | "teamcity-manta-artifact-storage-plugin", 1079 | "terraform", 1080 | "terraform-provider-triton", 1081 | "terraform-triton-bastion", 1082 | "terraform-triton-grafana", 1083 | "terraform-triton-kafka", 1084 | "terraform-triton-presto", 1085 | "terraform-triton-prometheus", 1086 | "terraform-triton-redash", 1087 | "terraform-triton-zookeeper", 1088 | "tokio-zookeeper", 1089 | "toshstomp", 1090 | "triton", 1091 | "triton-cmon", 1092 | "triton-cmon-agent", 1093 | "triton-cns", 1094 | "triton-dcapi", 1095 | "triton-dehydrated", 1096 | "triton-docker-cli", 1097 | "triton-go", 1098 | "triton-grafana", 1099 | "triton-kbmapi", 1100 | "triton-kubernetes", 1101 | "triton-logarchiver", 1102 | "triton-mockcloud", 1103 | "triton-mockcloud-common", 1104 | "triton-report", 1105 | "triton-terraform", 1106 | "triton-tracer", 1107 | "tsg-graphql", 1108 | "ubuntu-certified-image-tests", 1109 | "ubuntu-lx-brand-image-builder", 1110 | "ufds-terror-alert", 1111 | "ursa", 1112 | "uvxbridge", 1113 | "v8plus", 1114 | "vmss", 1115 | "vnodestats", 1116 | "void-lx-brand-image-builder", 1117 | "waferlock", 1118 | "webconsole-cloudapi-client", 1119 | "webconsole-images", 1120 | "webconsole-navigation", 1121 | "webconsole-nginx", 1122 | "webconsole-node", 1123 | "webconsole-tsg", 1124 | "webrev", 1125 | "zendesk", 1126 | "zfs_snapshot_tar", 1127 | "ziploader", 1128 | "zoneinit", 1129 | "zookeeper" 1130 | ] 1131 | } 1132 | --------------------------------------------------------------------------------