├── .github └── workflows │ └── rspec.yaml ├── .gitignore ├── CODEOWNERS ├── LICENSE ├── README.md ├── file_paths.md ├── language ├── apply.md ├── calls.md ├── catalog_expressions.md ├── data-types │ ├── pcore-data-representation.md │ ├── pcore-generic-data.md │ ├── pcore-ruby-data-api.md │ └── pcore-serialization.md ├── deprecations.md ├── expression_precedence.md ├── expressions.md ├── func-api.md ├── heredoc.md ├── intro.md ├── lexical_structure.md ├── module.md ├── modus-operandi.md ├── names.md ├── parameter_scope.md ├── plugin-api.md ├── puppet-functions.md ├── resource-api │ └── README.md ├── resource_types.md ├── settings.md ├── templates.md └── types_values_variables.md ├── models ├── LanguageSpec.ecore ├── LanguageSpec.ecorediag ├── LanguageSpec.png ├── README.md └── pn.md ├── moving_modules.md ├── tasks ├── README.md ├── error.json └── task.json └── tests ├── .gitignore ├── Gemfile └── spec ├── spec_helper.rb └── tasks └── schema_spec.rb /.github/workflows/rspec.yaml: -------------------------------------------------------------------------------- 1 | name: Ruby 2 | 3 | on: 4 | push: 5 | branches: ["master"] 6 | pull_request: 7 | branches: ["master"] 8 | 9 | permissions: 10 | contents: read 11 | 12 | defaults: 13 | run: 14 | working-directory: tests 15 | 16 | jobs: 17 | test: 18 | runs-on: ubuntu-latest 19 | steps: 20 | - uses: actions/checkout@v3 21 | - name: Set up Ruby 22 | uses: ruby/setup-ruby@v1 23 | with: 24 | ruby-version: '3.1' 25 | bundler-cache: true 26 | - name: Bundle install 27 | run: bundle install 28 | - name: Run tests 29 | run: bundle exec rspec -fd 30 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | *~ 3 | /.project 4 | /.idea/ 5 | -------------------------------------------------------------------------------- /CODEOWNERS: -------------------------------------------------------------------------------- 1 | * @puppetlabs/phoenix @puppetlabs/skeletor 2 | 3 | # Notify bolt team about proposed changes to the task spec 4 | /tasks @puppetlabs/bolt 5 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | puppet-specifications - Specifications for puppet and related technologies. 2 | 3 | Copyright (C) 2015 Puppet Labs Inc 4 | 5 | Puppet Labs can be contacted at: info@puppetlabs.com 6 | 7 | Licensed under the Apache License, Version 2.0 (the "License"); 8 | you may not use this file except in compliance with the License. 9 | You may obtain a copy of the License at 10 | 11 | http://www.apache.org/licenses/LICENSE-2.0 12 | 13 | Unless required by applicable law or agreed to in writing, software 14 | distributed under the License is distributed on an "AS IS" BASIS, 15 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | See the License for the specific language governing permissions and 17 | limitations under the License. 18 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | Puppet Specifications 2 | === 3 | 4 | This repository contains specifications for [the project Puppet][1], and related technologies. 5 | 6 | Puppet Language Specification 7 | --- 8 | The Puppet Programming Language Specification is a specification of the Puppet Programming 9 | Language. The first published release of this specification specifies the language version 4. 10 | 11 | The version 4.0.0 is the first version of the specification (this to make it harmonize 12 | with the 4.0.0 release of Puppet). From that point, the intention is to keep the 13 | same specification version even if the minor version of the implementation changes (i.e. 14 | for other reasons that the specification has changed). When a specification change is made, 15 | it *may* skip several numbers to again harmonize with the Puppet implementation version number. 16 | 17 | Until Puppet 4.0 was released, there was just the "current" and "future" implementations 18 | of the language. As time goes on it will be impossible to use only those words as their meaning 19 | is relative to a particular release of puppet (the future in 3.6 is not the same as the future in 3.7, and again not the same as current in 4.0) - hence the need for a separate version of the 20 | specification. 21 | 22 | The [Puppet Project][1] is the reference implementation of the specification. 23 | 24 | Semantic Versioning 25 | --- 26 | The specifications follows semantic versioning with the following semantics: 27 | 28 | * The micro version contains corrections, clarifications of the specification. All implementation 29 | of the specification that are compliant with the same minor version are also compliant with 30 | all micro versions of the same minor version. 31 | * The minor versions contains changes that are non breaking. But an implementation that 32 | is compliant with a previous minor versions is not automatically compliant with all future 33 | minor versions for the same major version. 34 | * The major versions contains changes that are breaking. An implementation that is compliant 35 | with an earlier major version can not be compliant with a major specification change. (It may 36 | offser compliance with multiple versions of the specification via the use of feature flags). 37 | 38 | Index 39 | --- 40 | 41 | * [Introduction][2] - terminology and EBNF grammar 42 | * [Modus Operandi][3] - overview of the runtime (loading, start-point, order of execution) 43 | * [Types, Values and Variables][4] specification of types, values and variables 44 | * [Naming and Scoping][5] - specification of names, scopes, and references 45 | * [Lexical Structure][6] - specification of the textual aspects of the Puppet Language 46 | * [Expressions][7] - specification of all non catalog expressions in the language 47 | * [Catalog Expressions][8] - specification of all expressions related to catalog production 48 | * [Expression Precedence][9] - specification of the precedence of expressions / operators 49 | * [Puppet Functions][14] - functions in the puppet language 50 | * [Deprecation][10] - specification of deprecated constructs 51 | * PROPOSALS 52 | * API 53 | * [Function Ruby API][11] - the API for writing functions in Ruby 54 | * [Plugin Ruby API][12] - the API for plugins in Ruby 55 | * [Resource Type Ruby API][16] - the API for resource types in Ruby 56 | * Models 57 | * [Puppet Extended S-Expression Notation (PN)][17] - specification of the PN format 58 | * Plan Extensions 59 | * [Apply Expression][18] - an expression to capture a manifest block and apply it on remote nodes 60 | * General 61 | * [Settings and Options][13] - specification of settings and options 62 | * [Puppet Installation Layout][15] - specification of Puppet related files on disk 63 | 64 | [2]:language/intro.md 65 | [3]:language/modus-operandi.md 66 | [4]:language/types_values_variables.md 67 | [5]:language/names.md 68 | [6]:language/lexical_structure.md 69 | [7]:language/expressions.md 70 | [8]:language/catalog_expressions.md 71 | [9]:language/expression_precedence.md 72 | [10]:language/deprecations.md 73 | 74 | [11]:language/func-api.md 75 | [12]:language/plugin-api.md 76 | [13]:language/settings.md 77 | [14]:language/puppet-functions.md 78 | [15]:file_paths.md 79 | [16]:language/resource_types.md 80 | [17]:models/pn.md 81 | [18]:language/apply.md 82 | 83 | [1]:http://www.github.com/puppetlabs/puppet 84 | -------------------------------------------------------------------------------- /language/apply.md: -------------------------------------------------------------------------------- 1 | Apply Expression 2 | === 3 | 4 | An `ApplyExpression` describes a manifest block which will be compiled to a catalog and applied on remote nodes. 5 | 6 | Grammar 7 | --- 8 | 9 | ApplyExpression 10 | : 'apply' arguments = ArgumentList '{' statements? '}' 11 | ; 12 | 13 | statements 14 | # All statements in the Puppet Programming Language (not shown) except NodeDefinition, 15 | # ClassDefinition, ResourceTypeDefinition, FunctionDefinition, and exported resource collectors 16 | : ... 17 | | primary_expression 18 | 19 | primary_expression 20 | : ... # all expressions that are primary expression in the Puppet Language (not shown) 21 | | epp_render_expression 22 | 23 | See [Function Calls](expressions.md#function-calls) for `ArgumentList` grammar definitions. 24 | 25 | An `ApplyExpression` takes an argument list of zero or more arguments. The intention is that the 1st 26 | argument is required, and represents a set of remote nodes to operate on, while the second argument is a hash of 27 | options. However defining the behavior of an ApplyExpression is out of the scope of this standard. 28 | 29 | The statements of an ApplyExpression comprise those statements expected within a manifest or the body of a class, 30 | excepting statements that themselves define named objects. The scope of statements is considered to be top-scope. 31 | 32 | An ApplyExpression may return a value. 33 | 34 | ApplyExpression is allowed within a PlanDefinition (to be defined later). 35 | -------------------------------------------------------------------------------- /language/calls.md: -------------------------------------------------------------------------------- 1 | Calls 2 | === 3 | 4 | The Puppet Programming Language supports a variety of expressions / constructs that constitute a **call**. A call 5 | transfers values from the caller (argument) and matches them against the called entity's parameters, after which it 6 | either evaluates the called entity directly or enqueues it for evaluation at a later point in time. 7 | 8 | These constructs constitute a call: 9 | 10 | * **Function Call**, The [Function Calls][1] section describes the details of how functions can be 11 | called, this section includes calling/using lambda expressions. 12 | * **Template Call**, The functions `epp`, and `inline_epp` can pass arguments to template parameters; 13 | thus performing a call to the template to produce text. 14 | * **Resource Instantiation**, creation of resources, resource defaults, resource overrides, and collection. 15 | * **Instantiation**, creation of values (new objects), conversion of data types etc (Since 4.5.0). 16 | 17 | [1]: expressions.md#function-calls 18 | 19 | ### Arguments 20 | 21 | Arguments are literal values and the results of evaluating expressions. In this call to 22 | the function `with` (which relays its received arguments on to a lambda), two arguments are given 23 | to the function (`1`, and the result of `2+2`). The lambda parameter `$x` is assigned the value `1`, 24 | and the lambda parameter `$y` is assigned the value `4`. 25 | 26 | # XXX i understand this is a contrived example but it'd be 27 | # better to have something you'd actually see in a module -e0 28 | with(1, 2+2) |$x, $y | { $x + $y } 29 | 30 | ### Parameters 31 | 32 | Parameters are named and optionally typed variables. A parameter may optionally also have a default value expression, 33 | and may declare that it captures any excess arguments. (Different rules apply depending on the type of call; 34 | by-position, or by-name). 35 | 36 | Parameters are always specified in a Parameter List as shown in the following grammar. 37 | 38 | ### Parameter List Grammar 39 | 40 | 41 | ParameterList 42 | : ParameterDeclaration (',' ParameterDeclaration)* ','? 43 | ; 44 | 45 | ParameterDeclaration 46 | : type = Expression? 47 | vararg ?= '*'? 48 | name = VariableExpression ('=' default_value = Expression)? 49 | ; 50 | 51 | VariableExpression : VARIABLE ; 52 | 53 | * Parameters follow the naming rule for variables. 54 | * The type is an optional `Type` that a given argument must have to be acceptable. If no type 55 | is specified, the type defaults to `Any`. 56 | * The vararg (captures-rest) `'*'` indicates that the parameter accepts excess arguments. 57 | When the argument 58 | passing form (see pass-by-position, and pass-by-name) allows 'captures-rest', it must be placed 59 | last, and only one captures-rest parameter may be used. 60 | * The default value expression allows specification of a value to be used in case there is no 61 | argument given for that parameter (missing argument). 62 | * A default value expression must be compliant with the parameter's type (or an error is raised). 63 | * A default value expression may not introduce new variables. 64 | 65 | Argument Passing 66 | --- 67 | Argument passing is performed with either **Pass-By-Position**, or **Pass-By-Name**. Function calls and calls to lambdas 68 | are always done with Pass-By-Position. Resource creation and EPP uses Pass-By-Name. Resource defaults and resource 69 | overrides use a variant of Pass-By-Name that allows amending values with `+>` instead of just setting them with `=>`. 70 | 71 | ### Pass By Position 72 | 73 | Pass-By-Position transfers given arguments to parameters based on their position: the first (leftmost) 74 | given argument is given to the first (leftmost) defined parameter. The caller must know the 75 | order in which to present the arguments; it is not possible to direct a particular argument to 76 | a particular parameter based on the name of the parameter. 77 | 78 | In pass-by-position: 79 | 80 | * All given values (including `undef`) count as arguments with a value and these values 81 | are assigned to the parameter in the same order they are called. 82 | * Only arguments not given at all result in a parameter having a missing argument, which leads to an 83 | error (missing argument) unless the parameter has a default value expression (in which case the 84 | result of evaluating that expression is used as the parameter's value). 85 | * A Parameter List for a Pass-By-Position entity may not have a parameter with a default value 86 | expression to the left of one that requires a value. XXX could this be restated as: "all parameters with defaults must 87 | come after required parameters"? -e0 88 | * A default value expression is evaluated in the called entity's closure (which for functions 89 | is the global scope/module they are defined in, for lambdas the scope where they are defined/ 90 | module). 91 | * **A default value expression does not have access to the parameters in the parameter list**, the 92 | parameters have not yet received their values when the default value expressions are evaluated. 93 | 94 | 95 | 96 | 100 |
Note
97 | In a future version of the specification, it will be allowed to access the values 98 | of parameters to the left of a parameter's default value expression. PUP-1985. 99 |
101 | 102 | ### Pass By Name 103 | 104 | Pass-By-Name transfers given arguments to parameters based on their name; both arguments and 105 | parameters are named in this style. In resource creation, resource defaults, and resource 106 | overrides, language syntax associates argument names with values - e.g. in a resource expression: 107 | 108 | notify { 'mynotify': message => 'hello world' } 109 | 110 | the argument `message` is associated with the value 'hello world'. When calling EPP, the arguments 111 | to the template are specified in a hash. 112 | 113 | epp('the_template', { x => 10, y => 20 }) 114 | 115 | In pass-by-name: 116 | 117 | * Only given values that are not `undef` counts as arguments with a value and these values 118 | are assigned to the corresponding parameters. 119 | * Arguments not given at all, or set to `undef` results in a parameter having a missing 120 | argument, which leads to a "missing argument" error unless the parameter 121 | has a default value expression, in which case the result of evaluating that expression is used as the parameter's 122 | value. 123 | * For some types of callable entities, there may be automatic lookup/injection of missing values that 124 | can supply a default value - see the respective entities (resource, class, etc.). 125 | * A captures-rest parameter is not allowed. 126 | * **A default value expression does not have access to the parameters in the parameter list**, the 127 | order in which the parameters receive their values when the default value expression is evaluated 128 | is undefined. 129 | 130 | 131 | 132 | 139 | 143 |
Note
133 | The 3x runtime which deals with resource expression, resource defaults, 134 | resource overrides, and collections, has unspecified behavior with respect to 135 | default value expression evaluation scope - they 136 | may be able to access other parameters, but this is only by coincidence as the order of 137 | evaluation is unspecified and varies with Ruby runtime version. 138 |
140 | In a future version of the specification, it will be allowed to access the values 141 | of parameters to the left of a parameter's default value expression. PUP-1985. 142 |
144 | -------------------------------------------------------------------------------- /language/data-types/pcore-data-representation.md: -------------------------------------------------------------------------------- 1 | Pcore Data Representation 2 | === 3 | Puppet's type system (Pcore) has two kinds of data representation (in addition to being able to describe data in the Puppet Programming Language itself). 4 | 5 | * [Pcore Generic Data][1] - As a programming language specific but generic data structure (that can then be 6 | serialized using a technology of choice for that programming language). 7 | * [Pcore Serialization][2] - As a Pcore specific encoding using either a textual or binary "on the wire" format. 8 | 9 | Important Note about Circular Data 10 | --- 11 | Regular data in Puppet is never circular since to all values are immutable and thus making it 12 | impossible to create such a structure. This means that care must be taken in custom Ruby logic 13 | that is integrated with Puppet to ensure that such custom logic does not return circular data. 14 | 15 | While regular data in Puppet cannot be circular, definitions of data types can be. Consider 16 | the type `type StringTree = Array[Variant[String, StringTree]]` - when this type is serialized 17 | (as opposed to a serialization of a value of this type) it clearly needs to be able to 18 | refer back to itself recursively. 19 | 20 | There are features in Pcore data representation to handle circular type definitions. Either: 21 | 22 | * with "type_by_reference = true" makes references to data types be by name and 23 | deserialization resolves names to loadable types available when deserializing. 24 | * with "type_by_reference = false" the definitions of the referenced types are included 25 | in the serialized result and deserialization adds the included type definitions 26 | to the set of available types in the deserializing environment. 27 | * When there is a circular type definition, a reference is used with a "back pointer" 28 | expressed as a [JSON path expressions][4]. 29 | 30 | See [Pcore Ruby Data API][3] for more information about these features. 31 | 32 | Pcore Generic Data 33 | --- 34 | Converting data in memory created by a programming language into a generic representation in 35 | that programming language makes it easy to then handle serialization and deserialization 36 | and to mix such serialization with other generic serializations. An advantage of this format 37 | is that the result is humanly readable and that it can be both read and written generically 38 | (without having to have an implementation of pcore serialization). See [Pcore Generic Data][1] 39 | for the specification of this format. This is the format known as "rich data" that is used by 40 | for example, a serialized Puppet Catalog. 41 | 42 | The Ruby API for Pcore Generic Data transformation produces a `Hash` that can be transformed 43 | to JSON, YAML or other formats, and transformed back to runtime objects from such a deserialized 44 | `Hash`. 45 | 46 | Pcore Serialization 47 | --- 48 | The Pcore specific serialization support produces more compact and efficient output 49 | but sacrifices human readability. See [Pcore Serialization][2] for the specification of this format. 50 | While specific to Pcore, the currently available 51 | on-the-wire formats (encodings) are however chosen such that they can be generically 52 | read and written by most programming languages (e.g. by using JSON for textual representation, 53 | and MsgPack for binary), but requires an implementation of Pcore to transform to/from 54 | semantic data values as portions of the serialized data may be opaque. 55 | 56 | The Ruby API for Pcore Serialization typically transforms to/from some kind of IO. 57 | 58 | [1]:pcore-generic-data.md 59 | [2]:pcore-serialization.md 60 | [3]:pcore-ruby-data-api.md 61 | [4]:http://goessner.net/articles/JsonPath/index.html#e2 62 | -------------------------------------------------------------------------------- /language/data-types/pcore-generic-data.md: -------------------------------------------------------------------------------- 1 | 2 | Pcore values as Generic Data 3 | ==== 4 | Pcore's Generic Data transformation transforms data such that 5 | it complies with the Pcore `Data` type; a type that is constrained to only match 6 | values that can be directly encoded in JSON. This document specifies this format. 7 | 8 | | Pcore Type | Is Represented as | 9 | | --- | --- | 10 | | `Numeric`, `String`, `Boolean` | corresponding JSON data type (see below for details)| 11 | | `Undef` | JSON `null` | 12 | | `Array` | JSON array | 13 | | a `Hash` =~ `Data` | JSON Object (Hash) | 14 | | a `Hash` !~ `Data` | Pcore Object representation of a Hash as JSON Object | 15 | | `Runtime` | Runtime values that are not Pcore values cannot be represented | 16 | | all other | Pcore Object representation as JSON Object | 17 | 18 | A `Numeric` value (`Integer` and `Float`) are represented by their 19 | corresponding Ruby runtime types. When converted to JSON there is only one 20 | numeric JSON datatype and the resulting deserialized data type is determined 21 | by the deserializer and the language runtime. As an example, JavaScript does 22 | not have a distinction between integers and floating point values, and will also 23 | lose precision in integers because they are represented as floating point in a 24 | JavaScript runtime. 25 | 26 | A `Hash` value requires special treatment if it has keys that are illegal in JSON. 27 | (JSON only allows keys that are Strings whereas Pcore allows almost any value 28 | as a key). The alternative form is to encode such a "rich hash" as a Pcore Object 29 | with the hash entries encoded as an array of alternating key, value entries. 30 | 31 | A `Runtime` value is a representation of a runtime programming language value 32 | that is not described by any other Pcore data type. It exists primarily to be able to 33 | map Pcore data types to runtime types, and to capture and report such "alien" values. 34 | An implementation of conversion to a Generic Data Hash can make a best effort to 35 | convert (for example by using a "to string" representation) but must at all times 36 | at least produce a warning. For typical use it is recommended to make the conversion 37 | such that an error is raised when encountering a `Runtime` value. 38 | 39 | All other values are represented as Pcore Objects. A Pcore Object is represented as 40 | a `Hash` with the special key `__ptype` mapped to the data type name in `String` 41 | form. 42 | 43 | Then there are three cases for the value(s) of a data type: 44 | 45 | * If the value is of a data type that is specified to have a single `String` value representation, 46 | then the hash will have the key `__pvalue` set to the values string representation. 47 | * If the value is completely defined by the type (like the `Default` data type), there 48 | is no `__pvalue` key. 49 | * Otherwise, the attributes of the value are encoded as additional key/value entries 50 | in the same hash. 51 | 52 | ### Examples 53 | 54 | **A Regular Expression** 55 | 56 | ```puppet 57 | /.*/ 58 | ``` 59 | 60 | is converted to: 61 | 62 | ```json 63 | { 64 | "__ptype": "Regexp", 65 | "__pvalue": ".*" 66 | } 67 | ``` 68 | 69 | **A Timestamp** 70 | 71 | ```puppet 72 | Timestamp() 73 | ``` 74 | 75 | is converted to (or rather was converted to when this was written as Timestamp() means "now"): 76 | 77 | ```json 78 | { 79 | "__ptype": "Timestamp", 80 | "__pvalue": "2018-08-20T12:07:15.710381000 UTC" 81 | } 82 | ``` 83 | 84 | **A default value** 85 | 86 | ```puppet 87 | default 88 | ``` 89 | 90 | is converted to: 91 | 92 | ```json 93 | { 94 | "__ptype": "Default", 95 | } 96 | ``` 97 | 98 | **An Array** 99 | 100 | ```puppet 101 | [1, 2, 3] 102 | ``` 103 | 104 | is converted to: 105 | 106 | ```json 107 | [ 108 | 1, 109 | 2, 110 | 3 111 | ] 112 | ``` 113 | 114 | **A Hash =~ Data** 115 | 116 | ```puppet 117 | { a => 10, b => 20} 118 | ``` 119 | 120 | is converted to: 121 | 122 | ```json 123 | { 124 | "a": 10, 125 | "b": 20 126 | } 127 | ``` 128 | 129 | **A Hash !~ Data** 130 | 131 | ```puppet 132 | { 10 => a, 20 => b} 133 | ``` 134 | 135 | is converted to: 136 | 137 | ```json 138 | { 139 | "__ptype": "Hash", 140 | "__pvalue": [ 141 | 10, "a", 142 | 20, "b" 143 | ] 144 | } 145 | ``` 146 | 147 | **All values that are of Object Type** 148 | 149 | ```puppet 150 | type Car = Object[ 151 | attributes => { 152 | regnbr => String, 153 | color => String 154 | } 155 | ] 156 | Car("abc123", "red") 157 | ``` 158 | 159 | is converted to: 160 | 161 | ```json 162 | { 163 | "__ptype": "Car", 164 | "regnbr": "abc123", 165 | "color": "red" 166 | } 167 | ``` 168 | -------------------------------------------------------------------------------- /language/data-types/pcore-ruby-data-api.md: -------------------------------------------------------------------------------- 1 | The Ruby Pcore API for data 2 | === 3 | 4 | The Generic Data API 5 | === 6 | The Generic Data API can convert runtime values to generic data, and back again to 7 | runtime values. This API does not define any actual serialization, but since it 8 | guarantees that converted runtime values conform to the Data type, it is safe to 9 | simply convert the returned value to JSON or YAML (or some other format). 10 | 11 | Converting to Generic Data 12 | --- 13 | To get any data transformed to generic data: 14 | 15 | ```ruby 16 | val = "the value to convert" 17 | converted = Puppet::Pops::Serialization::ToDataConverter.convert(val, 18 | :rich_data => true, 19 | :type_by_reference => true, 20 | :symbol_as_string => false, 21 | :local_reference => false, 22 | ) 23 | ``` 24 | 25 | The options are: 26 | 27 | | Option | Meaning | 28 | | --- | --- | 29 | | `:rich_data` | If non Data should be transformed to strings or Generic Data 30 | | `:type_by_reference` | If true, types are string references otherwise the types definition 31 | | `:local_reference` | If true deduplification will take place 32 | | `:symbol_as_string` | If true, symbols are turned into strings instead of Runtime instances 33 | | `:path_prefix` | A string that is output in errors and warnings before the actual issue. 34 | 35 | The format used by Puppet's `--rich_data` option is the same as what is shown in the example above. 36 | 37 | Rich Data (`:rich_data` => true) will transform rich data to a `Hash` with the meta keys 38 | `__ptype` and `__pvalue` as shown in [Generic Data Format][1]. If the option is false, all 39 | rich data values will be transformed to strings on a best effort and a warning is issued. While 40 | the result is often readable by a human, it cannot be read back in without loss of data type. 41 | 42 | Deduplicfication (`:local_reference` => `true`) is an experimental feature that replaces duplicate 43 | entires in the result with an instance of a `LocalRef` that has a value that is a json path to 44 | the original value. 45 | 46 | Type expansion (`:type_by_reference` => `false`) is an experimental feature that instead of 47 | using a type reference string as the value of the `__ptype` key will have a full representation 48 | of that data type. This is useful when the recipient does not have prior knowledge of the data type. 49 | 50 | Replacing Symbols can be done with (`symbols_as_string` => `true`). This will transform 51 | all Ruby `Symbol` instances except `:undef` and `:default` (which have special meaning) to 52 | the corresponding string. When the option is `true` all non special symbols are 53 | transformed to instances of `Runtime['Ruby', 'Symbol']`. 54 | 55 | Converting From Generic Data 56 | --- 57 | To convert generic data back from the generic form to runtime instances: 58 | 59 | ```ruby 60 | val = Puppet::Pops::Serialization::FromDataConverter.convert(converted) 61 | ``` 62 | 63 | The method allows options to specify a loader to use (defaults to puppet's standard loader), 64 | and also if values with unresolved type references should be kept in the result or if 65 | the operation should error. 66 | 67 | | Option | Meaning | 68 | | --- | --- | 69 | | `:loader` | The loader to use, if not specified the standard loader is used 70 | | `:allow_unresolved` | If `true`, unresolved type references will not error. Defaults to `false`. 71 | 72 | Both options are for advanced usage, and you should probably only use the defaults. 73 | 74 | Pcore Serialization Ruby API 75 | === 76 | 77 | Using JSON serialization 78 | --- 79 | 80 | ### Serializing 81 | 82 | This example shows how to serialize a `value` to a json string: 83 | 84 | ```ruby 85 | io = StringIO.new 86 | writer = Puppet::Pops::Serialization::JSON::Writer.new(io) 87 | serializer = Puppet::Pops::Serialization::Serializer.new(writer, :type_by_reference => true) 88 | serializer.write(value) 89 | serializer.finish 90 | serialized_string = io.string 91 | ``` 92 | 93 | The `Serializer` can be initialized with the option `:type_by_reference` which 94 | if it is set to `true` will include type references instead of serializing the 95 | definition of used (custom) types. When set to `true` the types must be available 96 | to the loader used when deserializing. 97 | 98 | Any Ruby IO object can be given to the Writer. 99 | 100 | ### Deserializing 101 | 102 | This example shows how to deserialize the `serialized_string` as produced by 103 | the example above into the `value` it used as input: 104 | 105 | ```ruby 106 | io = StringIO.new(serialized_string) 107 | reader = Puppet::Pops::Serialization::JSON::Reader.new(io) 108 | loader = scope.compiler.loaders.find_loader(nil) # use the standard loader for the environment 109 | deserializer = Puppet::Pops::Serialization::Deserializer.new(reader, loader) 110 | value = deserializer.read() 111 | ``` 112 | 113 | Any Ruby IO object can be given to the Reader. 114 | Note that a loader is required to find the data types referenced in the serialized data. 115 | 116 | Using Other Serializers 117 | --- 118 | Other serializers work exactly the same way - as an example, to use the MsgPack based serializer 119 | the writer would be `Puppet::Pops::Serialization::MsgPack::Writer`, and the reader `Puppet::Pops::Serialization::MsgPack::Reader`, while the rest of the examples for JSON 120 | would be essentially the same (since MsgPack is binary some care must be taken to handle the 121 | resulting binary string if StringIO is used. 122 | 123 | [1]:pcore-generic-data.md 124 | -------------------------------------------------------------------------------- /language/data-types/pcore-serialization.md: -------------------------------------------------------------------------------- 1 | Pcore serialization 2 | === 3 | Pcore Serialization in contrast to the human readable [Pcore Generic Data Format][1] is 4 | designed to make the best effort to produce a compact and "machine friendly" data stream 5 | that can be processed by a streaming data format parser. 6 | It does this by using a compact representation of values that performs deduplification 7 | of complex data and by including up front counts for values containing sequences. 8 | 9 | The Pcore Serializer has two levels - a *semantic* "higher level" that handles 10 | values of all data types defined by and in Pcore, and a lower *intrinsic* 11 | reader/writer level that handles the "on-the-wire" format. The lower level 12 | is required to handle intrinsic values of the types 13 | `Boolean`, `Undef`, `String`, `Integer`, and `Float`. 14 | The lower level must also have a way to write extensions with a payload such 15 | that extensions are distinct from intrinsic values. 16 | 17 | Notably, both `Array` and `Hash` are handled as extensions because many 18 | lower level formats have restrictions on keys in hashes and there needs to 19 | be a way to encode extensions distinctly. For stream processing it is also beneficial 20 | to know the count of elements up front as opposed having to wait until an entire sequence 21 | is read before knowing the count. 22 | 23 | As an example: in the JSON reader/writer provided with Pcore all extensions 24 | are encoded as a JSON Array containing extension id, and payload. 25 | 26 | Extensions 27 | --- 28 | The following table shows the extensions used by Pcore Serialization and what the respective 29 | extension's payload is. A payload of '-' means that the extension has no payload. 30 | 31 | By design, there are no free extensions for user defined types as all reserved extensions are 32 | for the future use of Pcore Serialization. 33 | 34 | | Extension | Token | Payload 35 | | --- | --- | --- 36 | | 0x00 | INNER_TABULATION | index 37 | | 0x01 | TABULATION | index 38 | | 0x02 - 0x0F | *reserved* | 39 | | 0x10 | ARRAY_START | count of elements, sequence of each element 40 | | 0x11 | MAP_START | count of elements, sequence of elements' key, value pairs 41 | | 0x12 | PCORE_OBJECT_START | type reference, count of given attributes, each attribute 42 | | 0x13 | OBJECT_START | count of given attributes, the type, each attribute 43 | | 0x14 | SENSITIVE_START | the sensitive value 44 | | 0x15 - 0x1F | *reserved* | 45 | | 0x20 | DEFAULT | - 46 | | 0x21 | COMMENT | the comment string 47 | | 0x22 - 0x2F | *reserved* | 48 | | 0x30 | REGEXP | the regexp in string form 49 | | 0x31 | TYPE_REFERENCE | type name 50 | | 0x32 | *unused* | 51 | | 0x33 | TIME | integer seconds, integer fraction in nanosec 52 | | 0x34 | TIMESPAN | integer seconds, integer fraction in nanosec 53 | | 0x35 | VERSION | the version in string form 54 | | 0x36 | VERSION_RANGE | the version range in string form 55 | | 0x37 | BINARY | the binary in binary form (only for binary lower level) 56 | | 0x38 | BASE64 | a binary in base64 form (only for textual lower level) 57 | | 0x39 | URI | the URI in string form 58 | | 0x3A - 0xFF | *reserved* | 59 | 60 | 61 | INNER TABULATION 62 | --- 63 | Inner tabulation is provided by the reader/writer. It tabulates all strings, and if a string 64 | has already been written (and thereby given an index) the output will 65 | then contain: 66 | 67 | ``` 68 | INNER_TABLUATION 69 | ``` 70 | 71 | Note that only String values are tabulated as it would be worse to tabulate 72 | numerical values, booleans and undef/null than including them as is. 73 | 74 | TABULATION 75 | --- 76 | Tabulation of non intrinsic values are performed by the semantic layer. It will tabulate all 77 | complex values such that if a value has already been written (and therefore given an index) 78 | the output will then contain: 79 | 80 | ``` 81 | TABULATION 82 | ``` 83 | 84 | OBJECT_START vs. PCORE_OBJECT_START 85 | --- 86 | All values that are instances of a type in the `Pcore::` name space (i.e. in the type system 87 | itself), and all object values when serializing "by reference" produce `PCORE_OBJECT_START` 88 | with a payload of type in puppet language string form and count of attributes with serialized 89 | values followed by the serialized attributes. 90 | 91 | When serializing with "by reference" set to `false`, then values of non `Pcore::` name spaced types 92 | produce `OBJECT_START` with a payload of attribute count followed by the serialized type, and then the value's serialized attributes. 93 | 94 | 95 | [1]:pcore-generic-data.md 96 | -------------------------------------------------------------------------------- /language/deprecations.md: -------------------------------------------------------------------------------- 1 | New Deprecations 2 | === 3 | This contains a list of deprecations that should be added to a release prior to Puppet 4. 4 | For each issue there is a brief discussion / motivation and implementation suggestion 5 | 6 | "class" is not a valid name for a class 7 | --- 8 | The keyword "class" is specifically allowed in the grammar and then becomes a string just 9 | like any other name. The deprecation can most cleanly be done in the grammar itself and 10 | this is also the point of origin for this feature. 11 | 12 | 13 | Upper case variable names 14 | --- 15 | Variables that start with an upper case letter are problematic when they are interpolated. 16 | The current 4x implementation validates them as invalid for sake of consistent naming (all initial 17 | UC segments is a reference to type and lower case to an instance). With an uc variables something like $a::B is an error since another rules says that each segment must start with the same case. 18 | 19 | It is thus impossible to refer to this variable from the outside. (That may be a feature in its 20 | own right :-) but a very odd way of providing this. 21 | 22 | Again, the easiest deprecation is probably in the grammar. 23 | 24 | (Note that there is a difference between a lexicographical variable name and a valid name). 25 | 26 | Variables with initial underscore 27 | --- 28 | Local variables may have an initial underscore, but a qualified variable may not have an underscore 29 | in the first position of a name segment. 30 | 31 | legal: 32 | 33 | $_x 34 | $_x_ 35 | $x_::y 36 | $x::y_ 37 | 38 | illegal: 39 | 40 | $::_x 41 | $x::_y 42 | 43 | Access to non existing variables 44 | --- 45 | There is a strict mode that can be turned on for variables. It performs a throw if the flag is turned on. It could be changed to always throw, then catch it, issue warning if flag is off and return undef. 46 | 47 | To turn on strict mode: 48 | 49 | --strict_variables 50 | 51 | 52 | Access to classes using uc class name/title 53 | --- 54 | 3.x allows a reference such as Class[Foo] to mean the class named 'foo'. This is 55 | inconsistent since just a Foo is a reference to a resource type. The use of uc name 56 | as class 'title' should be deprecated. (It is found in several tests). 57 | 58 | += and -= have been discontinued 59 | --- 60 | The operators `+=` (append) and `-=` (delete) have been discontinued. They used to have 61 | their own (quirky) semantics. They are trivially replaced as follows: 62 | 63 | $a += expression 64 | # replace with 65 | $a = $a + expression 66 | 67 | $a -= expression 68 | # replace with 69 | $a = $a - expression 70 | 71 | This works because the reference to a variable on the right hand side is evaluated before the 72 | new variable is created. It also works because `+` and `-` operations for add and delete works 73 | on numbers and Collections (`Array` and `Hash`). 74 | 75 | Older Deprecations Removed in 4x. 76 | === 77 | Here is a list of old behavior / deprecations that will be removed. 78 | 79 | hyphens in names 80 | --- 81 | The ability to accept have been removed. 82 | 83 | Ruby DSL 84 | --- 85 | Support for Ruby DSL will be removed. 86 | -------------------------------------------------------------------------------- /language/expression_precedence.md: -------------------------------------------------------------------------------- 1 | Expression / Operator Precedence 2 | === 3 | 4 | From highest to lowest precedence. Default association is "left" (the others are right and 5 | non-associate). Left means the LHS is fed into the RHS, right the reverse, and non associative 6 | means that the operator can not be repeated. 7 | 8 | The L/R column denotes if the Expression is L (left; can be assigned to), R (right; produces 9 | a value), 10 | or - (not L or R; typically punctuation). This is further explained in [L- and R-Value Expressions]. 11 | 12 | [L- and R-Value Expressions]: expressions.md#l-and-r-value-expressions 13 | 14 | | Operator / Expression | L/R | Association | Explanation 15 | | --- | --- | --- | --- 16 | | `;` | - | | expression separator, resource body separator 17 | | | | - | | separator around lambda parameters 18 | | `(` | R | | expression grouping () alter precedence of contained expression 19 | | `)` | - | | - " - 20 | | `.` | - | | inline call operand / function separator 21 | | `CALL` | R | | prefixed and inline style calls 22 | | `[` | R | | array start, access operator 23 | | `]` | - | | - " - 24 | | `?` | R | | select 25 | | <| <<| | R | | collect (virtual, exported) 26 | | |> |>> | - | | end of collect (virtual, exported) 27 | | `!` | R | right | not 28 | | `-` *unary* | R | nonassoc | unary minus 29 | | `*` *unary* | R | nonassoc | unary 'splat' for unfolding array 30 | | `in` | R | | 31 | | `=~` `!~` | R | | matches, not-matches 32 | | `*` `/` `%` | R | | multiplication, division, modulo 33 | | `+` `-` | R | | addition / concat / merge, subtraction / delete 34 | | `<<` `>>` | R | | left-shift / append, right-shift 35 | | `==` `!=` | R | | equal, not-equal 36 | | `>` `>=` `<` `<=` | R | | greater, greater-or-equal, less, less-or-equal 37 | | `and` | R | | boolean and 38 | | `or` | R | | boolean or 39 | | `=` | R | right | assign 40 | | `{` | R | | block / hash start 41 | | `{` after `?` | - | | block start after `?` 42 | | `}` | - | | block / hash / selector end 43 | | `:` (in title) | - | | title terminator 44 | | `:` (case colon) | - | | case proposition-list terminator 45 | | `=>` `+>` | - | | name-value association 46 | | `,` | - | | comma, separator in lists (parameters, arrays, hashes) 47 | | L/R-value expression | L/R | | Any complete L or R value expression (see expressions) 48 | | `@` `@@` | - | | virtual, exported (starts a resource expression) 49 | | resource expression | R | | resource, resource override, resource default expressions 50 | | `->` `~>` `<-` `<~` | R | | relationship 51 | | un-parenthesized call | R | | statement like calls 52 | 53 | The precedence governs the parsing, not all combinations of *expression operator expression* are 54 | sensical, and those that are not are validated as being in error. 55 | 56 | It is of importance to understand the precedence of operators to be able to 57 | understand why a particular expression does not give the expected result, or why it produces 58 | a particular error message. 59 | -------------------------------------------------------------------------------- /language/heredoc.md: -------------------------------------------------------------------------------- 1 | Puppet Heredoc 2 | === 3 | Puppet Heredoc is a feature that allows longer sequences of verbatim text to appear in 4 | a Puppet Program. The term Heredoc is borrowed from the Ruby Language. The Puppet Heredoc 5 | has similar traits as the Heredoc in Ruby, but differs in syntax and offered extra features. 6 | 7 | Grammar 8 | --- 9 | Heredoc is done by lexical processing of the source text, and the detailed grammar for 10 | heredoc is found in [Lexical Structure][1]. 11 | 12 | [1]: lexical_structure.md#heredoc 13 | 14 | 15 | The grammar defines the `@()` operator to indicate *"here doc, until given end tag"*, e.g. 16 | 17 | $a = @(END) 18 | This is the text that gets assigned to $a. 19 | And this too. 20 | END 21 | 22 | Further, the operator allows specification of the semantics of the 23 | contained text, as follows: 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 |
@(END)no interpolation and no escapes
@("END")double quoted string semantics for interpolation and no escapes
33 | 34 | It is possible to optionally specify a **syntax** that allows validation of the heredoc text, as in this example specifying 35 | that this text should be valid JSON. 36 | 37 | $a = @(END:json) 38 | This is supposed to be valid JSON text 39 | END 40 | 41 | And it is possible to specify the processing of escaped characters. By default both 42 | the `@(END)` form as well as the `@("END")` form has no escapes and text is verbatim. It is possible to turn on all 43 | escapes, or individual escapes by ending the heredoc specification with a `/`. If followed by only a `/` all escapes are turned on, 44 | and if followed by individual escapes, only the specified escapes are turned on. 45 | In this example, the user specifies that `\t` should be turned into tab characters: 46 | 47 | $a = @(END/t) 48 | There is a tab\tbefore 'before' 49 | END 50 | 51 | 52 | These features are explained in more details in the following sections. 53 | 54 | The Heredoc Tag 55 | --- 56 | The heredoc tag is written on the form: 57 | 58 | @( [:] [/] ) 59 | 60 | 61 | The heredoc endtag may consist of any text except the delimiters (see details below). 62 | Syntax is optionally specified with a `:` (colon) followed by the name of the syntax used in the text. 63 | Escapes are optionally specified with a `/` followed by the possible escapes. 64 | 65 | Here are the details: 66 | 67 | * <endtag> is any text not containing `:`, `/`, `)` or `\r`, `\n` (This is the text that indicates the end of the text) 68 | * <syntax> is [a-z][a-zA-Z0-9_+]*, and is the name of a syntax. This is validated to not contain empty segments between 69 | `+` signs. The result is always downcased. 70 | * <escapes> zero or more of the letters t, s, r, n, L, $ where 71 | * \t is replaced by a tab character 72 | * \s is replaced by a space 73 | * \r is replaced by carriage-return 74 | * \n is replaced by a new-line 75 | * \L replaces escaped end of line (\r?\n) with nothing 76 | * \$ is replaced by a $ and prevents any interpolation 77 | * If any of the other escapes are specified \\ is replaced by one \ 78 | * if <escapes> is empty (no characters) all escapes t, s, r, n, L, and $ are turned on. 79 | * It is an error to specify any given escape character more than once in the specification. 80 | 81 | The three parts in the heredoc tag may be surrounded by whitespace for readability. The list 82 | of escapes may not contain whitepsaces. 83 | 84 | End Marker 85 | --- 86 | 87 | The end marker consists of the heredoc endtag (specified at the opening `@`) and 88 | optional operators to control trimming. 89 | 90 | The end marker must appear on a line of its own. It may have leading and 91 | trailing whitespace, but no other text (it is then not recognized as the 92 | end of the heredoc). In contrast to Ruby heredoc, there is no need to 93 | specify at the opening that the end tag may be indented. 94 | 95 | The end marker may optionally start with one of the following operators: 96 | 97 | 98 | 99 | 100 | 102 | 103 | 104 | 105 | 110 | 111 | 112 | 113 | 114 | 115 |
-indicates that trailing whitespace (including new line) 101 | is trimmed from the last line.
|indentation marker, any left white-space on each line in the text that matches 106 | the white-space to the left of the position of the `|` char 107 | is trimmed from the text on all lines in the heredoc. Typically this is a sequence of spaces. A mix of tabs and spaces 108 | may be used, but there is no tab/space conversion, the same sequence used to indent lines should be used 109 | to indent the pipe char (thus, it is not possible to trim "part of a tab").
|-combines indentation and trimming of trailing whitespace
116 | 117 | The optional start may be separated from the end marker by whitespace 118 | (but not newline). These are legal end markers: 119 | 120 | -END 121 | - END 122 | |END 123 | | END 124 | |           END 125 | |- END 126 | 127 | The sections [Indentation](#indentation), and [Trimming](#trimming) 128 | contains examples and further details. 129 | 130 | Indentation 131 | --- 132 | 133 | The position of the | marker before the end tag controls how much 134 | leading whitespace to trim from the text. 135 | 136 | 0.........1.........2.........3.........4.........5.........6 137 | $a = @(END) 138 |   This is indented 2 spaces in the source, but produces 139 |   a result flush left with the initial 'T' 140 |     This line is thus indented 2 spaces. 141 |   | END 142 | 143 | Without the leading pipe operator, the end tag may be placed anywhere on 144 | the line. This will include all leading whitespace. 145 | 146 | 0.........1.........2.........3.........4.........5.........6 147 | $a = @(END) 148 |   This is indented 2 spaces in the source, and produces 149 |   a result with left margin equal to the source file's left edge. 150 |     This line is thus indented 4 spaces. 151 |                                         END 152 | 153 | When the indentation is right of the beginning position of some lines 154 | the present left whitespace is removed, but not further adjustment is 155 | made, thus altering the relative indentation. 156 | 157 | 0.........1.........2.........3.........4.........5.........6 158 | $a = @(END) 159 |   XXX 160 |     YYY 161 |    | END 162 | 163 | Results in the string `XXX\n YYY\n` 164 | 165 | ### Tabs in the input and indentation 166 | 167 | The left trimming is based on using the whitepsace to the left of the pipe character 168 | as a pattern. Thus, a mix of space and tabs may be used, but there is no tab to/from space 169 | conversion so it is not possible to trim part of a "tab". (**Note: It is considered best practice to 170 | always use spaces for indentation**). 171 | 172 | Trimming 173 | --- 174 | 175 | It is possible to trim the trailing whitespace from the last line (the 176 | line just before the end tag) by starting the end tag with a minus '-'. 177 | When a '-' is used this is the indentation position. 178 | 179 | 0.........1.........2.........3.........4.........5.........6 180 | $a = @(END) 181 |   This line will not be terminated by a new line 182 |   -END 183 | $b = @(END) 184 |   This line will not be terminated by a new line 185 |   |- END 186 | 187 | This is equivalent to: 188 | 189 | $a =  '  This line will not be terminated by a new line' 190 | $b =  'This line will not be terminated by a new line' 191 | 192 | It is allowed to have whitespace between the - and the tag, e.g. 193 | 194 | 0.........1.........2.........3.........4.........5.........6 195 | $a = @(END) 196 |   This line will not be terminated by a new line 197 |   - END 198 | 199 | Splitting and joining long lines 200 | --- 201 | If content has very long lines, lines can be split, and then joined by escaping the 202 | new line character. 203 | 204 | 205 | 0.........1.........2.........3.........4.........5.........6 206 | $a = @(END/L) 207 |   I am a very long line of text that is difficult to work \ 208 | with. The escaped end of line joins the long line into one. 209 |  | - END 210 | 211 | Spaces allowed in the tag 212 | --- 213 | 214 | White space is allowed in the tag name. Spaces are significant between words. 215 | 216 | 0.........1.........2.........3.........4.........5.........6 217 | $a = @(Verse 8 of The Raven) 218 |   Then this ebony bird beguiling my sad fancy into smiling, 219 |   By the grave and stern decorum of the countenance it wore, 220 |   `Though thy crest be shorn and shaven, thou,' I said, `art sure no craven. 221 |   Ghastly grim and ancient raven wandering from the nightly shore - 222 |   Tell me what thy lordly name is on the Night's Plutonian shore!' 223 |   Quoth the raven, `Nevermore.' 224 |   | Verse 8 of The Raven 225 | 226 | The comparison of opening and end tag is performed by first removing any quotes from the opening 227 | tag, then trimming leading/trailing whitespace, and then comparing the endtag against the text. 228 | The endtag must be written with the same internal spacing and capitalization as in the opening tag 229 | to be recognized. 230 | 231 | Escapes 232 | --- 233 | By default escapes in the text (e.g. \n) is treated as verbatim text. The rationale for this, is that 234 | heredoc is most often used for verbatim text, where any escapes makes it very tedious to correctly mark up 235 | special characters. There are however usecases where it is important to have detailed control over escapes. 236 | 237 | The set of allowed escapes are fixed to: t, s, r, n, $, and L (as shown earlier). The 'L' is special in that it allows the 238 | end of lines to be escaped (with the effect of joining them). The 'L' automatically handles line endings with carriage return/ 239 | line feed combinations. (In case it is not obvious: You can not insert an `\L` in the text). Here is an example: 240 | 241 | $a = @(END/L) 242 | First line, \ 243 | also on first line in result 244 | |- END 245 | 246 | Produces 247 | 248 | First line, also on first line in result 249 | 250 | 251 | Without the specification of `/L` the result would have been 252 | 253 | First line, 254 | also on first line in result 255 | 256 | When an escape is on for any character, it is always possible to escape the backslash to prevent replacement from 257 | taking place. 258 | 259 | $a = @(END/L) 260 | First line, \\ 261 | on second line 262 | |- END 263 | 264 | An empty escapes specification turns on all escapes. 265 | 266 | Multiple Heredoc on the same line 267 | --- 268 | 269 | It is possible to use more than one heredoc on the same line as in this 270 | example: 271 | 272 | foo(@(FIRST), @(SECOND)) 273 |   This is the text for the first heredoc 274 |     FIRST 275 |   This is the text for the second 276 |     SECOND 277 | 278 | If however, the line is broken like this: 279 | 280 | foo(@(FIRST), 281 | @(SECOND)) 282 | 283 | Then the heredocs must appear in this order: 284 | 285 | foo(@(FIRST), 286 |   This is the text for the first heredoc 287 |   FIRST 288 | @(SECOND)) 289 |   This is the text for the second 290 |   SECOND 291 | 292 | Additional semantics 293 | --- 294 | 295 | The `@(tag)` is equivalent to a right value (i.e. a string), only that its 296 | content begins on the next line of not already consumed heredoc text. The 297 | shuffling around of the text is a purely lexical exercise. 298 | 299 | Thus, it is possible to continue the heredoc expression, e.g. with a 300 | method call. 301 | 302 | $a = @(END).upcase() 303 |   I am not shouting. At least not yet... 304 |   | END 305 | 306 | (It is not possible to continue the expression after the end-tag). 307 | 308 | 309 | Syntax Checking of Heredoc Text 310 | --- 311 | Syntax checking of a heredoc text is made possible by allowing the heredoc to be tagged with the name 312 | of a syntax/language. Here are some examples: 313 | 314 | @(END:javascript) 315 | @(END:ruby) 316 | @(END:property_file) 317 | @(END:yaml) 318 | @(END:json) 319 | @(END:myschema+json) 320 | @("END":epp) 321 | 322 | This way, the checking can take place server side before the content 323 | is (much later) used with possibly very hard to detect problems as a 324 | result. 325 | 326 | The given syntax name is given to a syntax checker implementation that is selected 327 | based on the name. The syntax name characters `+` and `.` are translated to two and one underscore 328 | character respectively. (See more about `+` below). A period is allowed since many mime types 329 | have this in their names . 330 | 331 | 332 | A `+` character is allowed as a name separator (in the style of RFC2046-Mime Media Types). 333 | The intent is to describe syntax that is encoded in formats such as xml or json where the content 334 | must be valid json, xml etc. *and* be valid in the specified dialect/"schema" (e.g. xslt+xml). 335 | Mime media types includes the use of `.` (e.g. vnd.apple.installer+xml). 336 | 337 | By convention, the most specific syntax should be placed first. 338 | When mapping syntax to a checker, each `+` defines a name segment. 339 | An attempt is first made with the full name (e.g. 'xslt+xml'), and if no 340 | such function exists, the leftmost 341 | name segment is dropped, and a new attempt is made to find a checker (e.g. 'xml'). 342 | The first found checker is given the 343 | task of validating the string. If no checker is found, the string is not validated. 344 | 345 | The full syntax string is passed to the validation function to allow 346 | mapping of segments to schema names, thus allowing once checker to check many more 347 | specific syntaxes without having to explicitly specify this. 348 | 349 | Checkers are wired into the Puppet runtime by using the binder. Puppet comes bundled 350 | with a syntax checker for 'json'. 351 | 352 | The API for extending puppet with additional checkers is currently experimental. See 353 | the source of `puppetx/syntax_checker.rb`, and `puppets/puppetlabs/syntax_checkers/json.rb` for 354 | reference. 355 | 356 | The implementation should inherit from the `Puppetx::Puppet::SyntaxChecker` and be placed 357 | in a module under `lib/puppetx/mymodule/syntax_checkers/`. A binding that wires it and makes 358 | it available for a given syntax should be placed in the module's directory 359 | `lib/puppet/bindings//default.rb` with the following content: 360 | 361 | Puppet::Bindings::newbindings('::default' do 362 | bind { 363 | name 'mysyntax' 364 | instance_of 'Puppetx::Puppet::SyntaxChecker' 365 | in_multibind 'puppetx::puppet::syntaxcheckers' 366 | to_instance 'Puppetx::::SyntaxCheckers::MySyntax' 367 | } 368 | end 369 | 370 | This binding is automatically included, and can then be used in a heredoc: 371 | 372 | @(END:mysyntax) 373 | Oh, my syntax, my beautiful syntax, 374 | please tell me when you find an errr. 375 | END 376 | -------------------------------------------------------------------------------- /language/intro.md: -------------------------------------------------------------------------------- 1 | Puppet Language Specification 2 | === 3 | This specification is a technical description of the Puppet Language. 4 | 5 | Terminology 6 | --- 7 |
8 |
Puppet Program
9 |
A Program written in the Puppet Language, also known as the "Puppet 10 | DSL"
11 | 12 |
Program
13 |
A Program written in the Puppet Language unless text refers to some other language 14 | like Ruby Program, Java Program etc. 15 |
16 | 17 |
LHS
18 |
Left Hand Side, the left operand in a binary expression.
19 | 20 |
RHS
21 |
Right Hand Side, the right operand in a binary expression.
22 | 23 |
Lexing
24 |
The act of turning source text into tokens that are recognized by a parser.
25 | 26 |
Parsing
27 |
The act of turning source text into a abstract syntax model.
28 | 29 |
Loading
30 |
The act of locating source text, parsing, and evaluating logic to the point where 31 | definitions are available (but not necessarily evaluated in full). 32 |
33 | 34 |
Evaluation
35 |
The act of carrying out the instructions in the Puppet Program being executed.
36 | 37 |
Compilation
38 |
The act of producing a Catalog by executing a Puppet Program. 39 | Also known as Catalog Building. 40 |
41 | 42 |
Declaration
43 |
The act of introducing a named and possibly typed element into a Program.
44 | 45 |
Definition
46 |
The act of assigning/setting the content/value of an element in the Program
47 | 48 |
Manifest
49 |
A file with source code in the Puppet Language. The extension .pp is used for such files. 50 |
51 |
52 | 53 | Grammar Notation 54 | --- 55 | Grammars (lexical and syntactic) are written in a variant of Extended Backus Naur Form (EBNF) with the following syntax and semantics: 56 | 57 | | syntax | meaning 58 | |------ | ------- 59 | | 'c' | the terminal character c 60 | | '\r' | the terminal ascii character CR 61 | | '\n' | the terminal ascii character NL 62 | | '\t' | the terminal ascii character TAB 63 | | '\\' | the terminal ascii character BACKSLASH 64 | | rule: | a rule name, (all uppercase rule is a terminal rule, mixed case is a regular rule) 65 | | TOKEN: | a terminal token rule 66 | | ( ) | groups elements 67 | | ? | the preceding element occurs zero or one time 68 | | * | the preceding element occurs zero or many times 69 | | + | the preceding element occurs one or many times 70 | | | | or 71 | | /re/ | a regular expression as defined by Ruby 2.0 72 | | ; | rule end 73 | | sym = | symbolic naming of rule to the right 74 | | sym += | symbolic naming of array containing iterative values from rule on right 75 | | rule<Type> | A rule call that when evaluated produces the given (runtime type) 76 | | rule <Type>: | A type safe rule that when evaluated produces the given (runtime type) 77 | 78 | The presence of `sym=` and `sym+=` does not alter the grammar, they only provide notation to 79 | be able to refer to the various parts of the rule using symbolic names. 80 | 81 | ### Grammar Example 82 | ``` 83 | Hello: 'h' 'e' 'l' 'l' 'o' ; 84 | Hi: 'h' 'i' ; 85 | NAME: /[A-Za-z]+/ ; 86 | Greeting: (Hello | Hi ) NAME '!'?; 87 | 88 | StringAccess 89 | : Expression '[' from = Int (',' to = Int)? ']' 90 | ; 91 | 92 | Int : Expression ; 93 | 94 | ASequenceOfNames 95 | : (names += NAME)+ 96 | ; 97 | 98 | ``` 99 | ### Set Algebra Notation 100 | 101 | Algebra notation is used in the specification to describe the operations on 102 | types as well as other logical constructs. Here is a summary of the set theory notations 103 | used in this specification: 104 | 105 | `X` is *member of* (the set `Y`) (can be read as `exists in`) 106 | 107 | X ∈ Y 108 | 109 | `X` is *not member of* (the set `Y`) (can be read as `not exists in`) 110 | 111 | X ∉ Y 112 | 113 | The *union* of `X` and `Y` (all that are in `X`, in `Y`, or in both) 114 | 115 | X ∪ Y 116 | 117 | The intersection of `X` and `Y` (all that are in both `X` and `Y`) 118 | 119 | X ∩ Y 120 | 121 | Implies, gives, produces (depending on context) 122 | 123 | → 124 | 125 | The empty set 126 | 127 | ∅ 128 | 129 | 130 | #### Examples: 131 | 132 | The union of the type Integer and the type Float is Numeric: 133 | 134 | Integer ∪ Float → Numeric 135 | 136 | This means that if there is a collection of objects that consist of integers and float 137 | (but no instances of any other type), then the collection is of 138 | `Collection[Numeric]` types. 139 | -------------------------------------------------------------------------------- /language/lexical_structure.md: -------------------------------------------------------------------------------- 1 | Lexical Structure 2 | === 3 | 4 | Unicode 5 | --- 6 | 7 | 8 | 9 | 16 | 17 | 28 |
Puppet Language keywords and punctuation uses only ASCII. Strings (single and double quoted), Heredoc text, and Comments can contain non ASCII:
Since Puppet 4.4.0
10 | All Puppet source code is expected to be in UTF-8. Non Ascii characters must be in UTF-8 and 11 | may appear in comments, single and double quoted strings, heredocs, and templates. 12 |
13 |

Byte order marks (BOM) are not allowed. If present an error is raised identifying the 14 | kind of byte order mark.

15 |
Before Puppet 4.4.0
18 | The implementation uses Ruby default encoding when reading 19 | Puppet source text files. 20 | Thus, while the language itself does not make use of any non ASCII characters, it is possible 21 | to include other characters in strings given that the source file is written using the Ruby 22 | runtime environments default encoding and that a Ruby version is used that supports encodings. 23 | For Puppet code running on runtimes before Puppet 4.4.0, the only platform neutral way to use non 24 | ASCII characters in the source is to use the Puppet 4x Unicode Escape mechanism \uXXXX. 25 |
26 |

Byte order marks (BOM) causes undefined behavior.

27 |
29 | 30 | Not Completely Context Free 31 | --- 32 | The Puppet Programming Language's lexical structure is not completely context free; the interpretation of the source text differs in a few cases depending on the previously seen significant token. 33 | 34 | Line Terminators 35 | --- 36 | The sequence of input characters in a Program's source file is divided into lines by recognizing 37 | *line terminators* 38 | 39 | ``` 40 | LineTerminator 41 | : '\r'? '\n' 42 | ; 43 | ``` 44 | 45 | Line Numbers and Positions on Line 46 | --- 47 | The first line is given the line number 1. The first character on a line is in position 1. 48 | 49 | Whitespace 50 | --- 51 | Whitespace between tokens is generally insignificant except for a few exceptional cases 52 | noted in the section about Punctuation TBD. REF TO PUNCTUATION SPECIAL PROCESSING. 53 | 54 | ``` 55 | WHITESPACE 56 | : /([[:blank:]]|\r\n)+/ 57 | ; 58 | ``` 59 | 60 | The regular expression `[:blank:]` matches all types of unicode spaces. This is done in order 61 | to not disrupt lexical processing when text is copy / pasted from published examples since 62 | these may contain hard spaces, narrow spaces etc. 63 | 64 | Comments 65 | --- 66 | Comments between tokens are stripped away. Comments may be written as *single line comments* or 67 | *multi line comments*. 68 | 69 | # this is a single line comment 70 | 71 | /* this is a multi- 72 | line comment. 73 | */ 74 | 75 | A single line comment inside a multi line comment (or vice versa) has no special meaning. 76 | The lexical grammar implies that comments do not occur inside of literal strings. 77 | 78 | ``` 79 | SINGLE_LINE_COMMENT 80 | : /#[^\r\n]*(\r?\n)?/ 81 | ; 82 | 83 | MULTI_LINE_COMMENT 84 | : /\/\*(.*?)\*\//m 85 | ; 86 | ``` 87 | 88 | A single line comment starts with a `#` and runs to the end of the line, or to the end of the input if there is no line ending and this is the last line. 89 | 90 | A multi line comment consists of at least `/**/`, allows embedded `/*`, but is terminated 91 | by the first occurrence of `*/`. It is not possible to escape the end of the multiline 92 | comment. 93 | 94 | 95 | 100 |
Note
96 | Documentation processing tools makes comments in certain positions appear as documentation. 97 | These tools may have additional rules regarding placement and content. Such rules 98 | are not defined by this specification. 99 |
101 | 102 | Numbers 103 | --- 104 | Numbers are recognized in decimal integer and floating point form as well as integers 105 | in octal and hexadecimal form. All numbers start with a digit. 106 | 107 | ``` 108 | NUMBER 109 | : HEX | OCTAL | DECIMAL | FLOAT 110 | ; 111 | 112 | HEX 113 | : /0[xX][0-9a-fA-F]+/ 114 | ; 115 | 116 | OCTAL 117 | : /0[0-7]+/ 118 | ; 119 | 120 | DECIMAL 121 | : /[1-9][0-9]*/ 122 | ; 123 | 124 | FLOAT 125 | : /0?\d+(\.\d+)?([eE]-?\d+)? 126 | ; 127 | ``` 128 | 129 | A number that starts with `0` and is not followed by a period `.` must be a valid octal number. 130 | All numbers containing a decimal period `.` or an exponential are interpreted as floating point 131 | numbers and all other are integers. 132 | 133 | All given numbers must be valid in their implied radix. 134 | 135 | 136 | 137 | 141 | 142 | 145 |
Note
138 | Literal numbers maintain their radix (hex, octal, or decimal) but this is lost in evaluation where 139 | all values are decimal and string formatting is required. 140 |
Future
143 | The exponent may allow a '+' in the future. 144 |
146 | 147 | 148 | Strings 149 | --- 150 | Strings of text are available in single and double quoted form. Both kinds of strings 151 | can extend over multiple lines. Heredoc is an alternative form of String (see below). 152 | 153 | 154 | 155 | 160 | 161 | 164 |
Note
156 | Line endings in strings contains the corresponding characters 157 | from the source text file - there is no transformation of \r\n into \n 158 | or vice versa. 159 |
Future
162 | This may change in the future. 163 |
165 | 166 | ### Single Quoted String 167 | 168 | A *Single Quoted String* starts and ends with `'`. The following escape sequences are supported: 169 | 170 | #### Single Quoted String Escape Sequences 171 | 172 | | sequence | result 173 | | --- | --- 174 | | `\'` | a single `'` 175 | | `\\` | a single `\` 176 | | `\` *any other* | a single `\` followed by *any other* 177 | 178 | ### Double Quoted String 179 | 180 | A *Double Quoted String* starts and ends with `"`. In addition to supporting an extended 181 | set of escape sequences, a double quoted string also supports interpolation of Puppet Language 182 | expressions. 183 | 184 | #### Double Quoted String Escape Sequences 185 | 186 | | sequence | result 187 | | --- | --- 188 | | `\"` | a single `"` 189 | | `\\` | a single `\` (and removes the escaping power of the escaped \) 190 | | `\r` | an ASCII CR 191 | | `\n` | an ASCII NL 192 | | `\t` | an ASCII TAB 193 | | `\s` | an ASCII SPACE 194 | | `\uXXXX` | a UNICODE character denoted by 4 hex digits i.e. /[0-9a-fA-F]{4}/ 195 | | `\u{XXXX}` | a UNICODE character denoted by 1-6 hex digits i.e. /[0-9a-fA-F]{1,6}/ 196 | | `\` *any other* | a single `\` followed by *any other* (removes any special meaning from *any other*) 197 | 198 | #### Double Quoted String Expression Interpolation 199 | 200 | A double quoted string is delivered using three different tokens: `DQPRE`, `DQMID`, and `DQPOST`. Any other tokens may appear between a `DQPRE` and a `DQMID`, and between a `DQMID` and a `DQPOST`. An interpolated string may consist of only a `DQPRE` and a `DQPOST`, or be optimized into a single `STRING` token if there is no interpolation. 201 | 202 | A `DQPRE` starts with `"` and is terminated by `$NAME`, or `${`. A `DQMID` starts automatically in the first non `NAME` character after the sequence `$`, `NAME`, or a `}` that balances the opening `${` and is terminated the same way as `DQPRE`. A `DQPOST` starts the same way as a `DQMID`, and is terminated by a closing `"`. 203 | 204 | A double quoted string may contain nested (complete) double quoted strings in the interpolated 205 | expressions. 206 | 207 | The lexical processing delivers a `$`, `NAME` sequence as a `VARIABLE` token (as if the user had 208 | written `${$name}`). No lexical processing is performed for interpolation using `${ }`; this 209 | is instead done as part of syntactic and semantic processing of the result. 210 | 211 | Here are some examples to illustrate: 212 | 213 | "Hello $name" 214 | #=> DQPRE('Hello '), VARIABLE(name), DQPOST('') 215 | 216 | "Hello ${name}" 217 | #=> DQPRE('Hello '), NAME('name'), DQPOST('') 218 | 219 | "Hello nbr ${1+1}, what is your name?" 220 | #=> DQPRE('Hello nor ', NUMBER(1), PLUS(+), NUMBER(1), DQPOST(', what is your name?') 221 | 222 | "Hello $name1 and $name2!" 223 | #=> DQPRE('Hello '), VARIABLE('name1'), DQMID(' and '), VARIABLE('name2'), DQPOST('!') 224 | 225 | The String Interpolation Expression is further explained in [Expressiona][1] 226 | 227 | [1]: expressions.md 228 | 229 | Regular Expressions 230 | --- 231 | A *Regular Expression* is written on the form 232 | 233 | ``` 234 | REGEXP 235 | : /[^\/\n]*\// 236 | ; 237 | ``` 238 | This means that a regular expression starts and ends with `/` and may not extend over multiple lines. 239 | 240 | The syntax of the Puppet Language regular expression is defined by the Ruby Regular Expression. 241 | The Puppet Language does not support the use of `\A`, and `\z` and does not support modifiers after 242 | the closing `/`. 243 | 244 | A *Regular Expression* is recognized in most lexical contexts but not in positions where 245 | an operator is accepted. Specifically, it is not recognized 246 | when appearing after `')'`, `']'`, `'|>>'`, `'|>'`, `NAME`, `REF`, `STRING`, `BOOLEAN`, `REGEX`, `HEREDOC`, and the string-parts of a double quoted string with interpolation. 247 | 248 | There is one ambiguity in that a Regular Expression must be allowed to appear after a `'}'` (end of a case expression option and start of a new). This clashes with constructs where `'}'` is the end of an expression that produces an *R-value* and where it is possible to divide the result. In the event the program logic required several divisions (e.g. `...} //` the source must place the 249 | second `'/'` on a new line to avoid `//` to be recognized as a regular expression (or alternatively compute a single divisor to avoid the repeated division). 250 | 251 | Bare Words, Names and References 252 | --- 253 | A *bare word* (`WORD`) is an unquoted sequence of letters and the underscore (`_`) character and intermixed 254 | hyphens `-` optionally starting with `::` and divided into multiple name-space segments starting with `::`. 255 | Each bare word segment must start with a lower case letter `a`-`z` or underscore `_`. A bare word may not end with 256 | a hyphen (`-`) or with a name-space separator (`::`). A bare word may not contain single `:` characters. 257 | 258 | A "bare word" that complies with the more restrictive rule `/(::)?[a-z]\w*(::[a-z]\w*)*/` is taken as a `NAME`. 259 | The term `QualifiedName` is used to denote a `NAME` with name-space `::` separators. 260 | 261 | In some contexts there is a difference between a `NAME` and a `WORD` (a `WORD` is for example not acceptable as 262 | the name of named elements like a function or a class). In general, when used as values both `WORD` and `NAME` evaluate 263 | to the bare word string. 264 | 265 | A sequence that starts with an upper case letter is a reference to a **type** and is never interpreted as a `WORD` or a `NAME`. 266 | The term `QualifiedReference` is used to denote such upper case sequences. In some contexts a `QualifiedReference` may 267 | be used to name an element (for example when creating a type alias as in `type MyType = SomeOtherType`). In general, a 268 | `QualifiedReference` is a reference to an existing data type like `Integer` or resource type like `File`. 269 | 270 | A character sequence may be divided into *name-space* segments. The name-space separator is `::` and it may also be 271 | used first in the name to *anchor* the name in the root/global name-space. 272 | Each name-space segment must follow the same format; all segments must start with a letter of the same case (where an underscore (`_`) is considered to be lower case). 273 | 274 | While a `WORD` may contain name space separators (`::`) the result is simply a string and any interpretation of the `::` as name space separators is up to the user of such a string. 275 | 276 | Keywords can not be used as identifiers (names) of elements, but may be used as names of attributes/properties. 277 | 278 | ### Lower Case Bare Words / NAME / Qualified Name 279 | 280 | ``` 281 | NAME 282 | : /(::)?[a-z]\w*(::[a-z]\w*)*/ 283 | ; 284 | 285 | WORD 286 | : /((?:::){0,1}(?:[a-z_](?:[\w-]*[\w])?))+/ 287 | ; 288 | 289 | ``` 290 | 291 | NAME Examples: 292 | 293 | apache::port 294 | ::apache 295 | ::apache::port 296 | 297 | WORD Examples: 298 | 299 | bare_word_string 300 | this-is-a-bare-word-string 301 | _bare_word 302 | _foo_ 303 | _Bare-word-as-it-starts-with-underscore 304 | 305 | Examples of non bare words: 306 | 307 | NotBareWord_IsATypeReference 308 | can-not-end-with-hyphen- 309 | not::OK 310 | -can-not-start-with-hyphen 311 | 312 | ### Upper Case Bare Words / REF / QualifiedReference / TypeReference 313 | 314 | ``` 315 | REF 316 | : /(::)?[A-Z]\w*(::[A-Z]\w*)*/ 317 | ; 318 | ``` 319 | 320 | Examples: 321 | 322 | File 323 | ::File 324 | Class 325 | Integer 326 | 327 | 328 | 329 | 332 |
Note
330 | Many upper case words denote built in types and these names should be considered to be reserved. 331 |
333 | 334 | 335 | Variable 336 | --- 337 | A variable in the language is always preceded by `$`. (There are special cases in double quoted 338 | string expression interpolation where a `NAME` may be taken as a variable name). 339 | 340 | ``` 341 | VARIABLE 342 | : /\$(::)?(\w+::)*\w+/ 343 | ; 344 | ``` 345 | 346 | Note that there is a difference between what is lexically recognized as a variable and a valid variable reference. The lexically recognized variable accepts the following illegal 347 | names: 348 | 349 | $0xG # Numeric variable that is not a valid decimal number 350 | $0080 # A Numeric variable may be 0 (exactly), or a decimal value that does not start with 0 351 | $Abc # Variables may not start with upper case letter 352 | 353 | The distinction between lexicographic variable and valid variable is mainly important for 354 | string interpolation. Here is an example: 355 | 356 | "Hello $00080, how are you" 357 | 358 | This is an attempt to interpolate the invalid variable `$00080` and not an interpolation of the valid variable `$0` followed by the text `'0080, how are you'`. If the latter was intended, it should be 359 | written in one of these forms: 360 | 361 | "Hello ${0}0080, how are you" 362 | "Hello ${$0}0080, how are you" 363 | 364 | See the following sections in [Expressions][1] for more information: 365 | 366 | * String Interpolation 367 | * Types, Values, and Variables 368 | 369 | Keywords 370 | --- 371 | When an *Identifier* had been identified and it is equal (in its entirety) to a keyword, the 372 | keyword token is produced instead of a `NAME` token. Keywords are case sensitive. 373 | 374 | | Literal | value 375 | | --- | --- 376 | | `false` | Boolean false 377 | | `true` | Boolean true 378 | | `undef` | The Puppet Language notion of nil / null / undefined 379 | 380 | | Keywords 381 | | --- 382 | | `and` 383 | | `case` 384 | | `class` 385 | | `default` 386 | | `define` 387 | | `else` 388 | | `elsif` 389 | | `function` 390 | | `if` 391 | | `in` 392 | | `inherits` 393 | | `node` 394 | | `or` 395 | | `type` 396 | | `unless` 397 | 398 | The semantics of these is described in [Expressions][1]. 399 | 400 | The following keywords are considered reserved for future use and should be avoided. 401 | 402 | | Reserved Words 403 | | --- 404 | | `private` 405 | | `attr` 406 | | `plan` 407 | | `apply` 408 | 409 | These names are reserved for types, and are unsuitable as identifiers for other kinds of 410 | elements: 411 | 412 | | Reserved Names / Types 413 | | --- 414 | | `any, Any` 415 | | `array, Array` 416 | | `attr, Attr` 417 | | `boolean, Boolean` 418 | | `catalogentry, catalogEntry, CatalogEntry` 419 | | `class, Class` 420 | | `collection, Collection` 421 | | `data, Data` 422 | | `default, Default` 423 | | `enum, Enum` 424 | | `float, Float` 425 | | `hash, Hash` 426 | | `integer, Integer` 427 | | `numeric, Numeric` 428 | | `object, Object` 429 | | `optional, Optional` 430 | | `scalar, Scalar` 431 | | `pattern, Pattern` 432 | | `private, Private` 433 | | `resource, Resource` 434 | | `runtime, Runtime` 435 | | `semver, SemVer` 436 | | `semverrange, SemVerRange` 437 | | `string, String` 438 | | `struct, Struct` 439 | | `timespan, Timespan` 440 | | `timestamp, Timestamp` 441 | | `tuple, Tuple` 442 | | `type, Type` 443 | | `undef, Undef` 444 | | `variant, Variant` 445 | 446 | While the lower case names are perfectly fine to use unless they are also keywords (i.e. when they 447 | have no special meaning) they should not be used as names of functions, classes, or user defined 448 | defined resource types as the name would clashes with the built in types. This occurs because lower 449 | case named definitions automatically get an upper cased type reference. 450 | 451 | 452 | Separators / Punctuation 453 | --- 454 | ``` 455 | ( ) { } [ ] ; , . | : 456 | ``` 457 | 458 | ### Special Punctuation Processing 459 | 460 | * When a `[` is preceded by `WHITESPACE` or is at the beginning of the input the delivered token is 461 | `LISTSTART`, else the token `LBRACK`. This is done to disambiguate `$a[1]` (index operation on ` 462 | $a`) from `$a [1]` (lookup of variable value `$a`, followed by an array with the value `1`), 463 | and similar ambiguities. 464 | 465 | * When a `{` is preceded by a `?` (`WHITESPACE` ignored) the delivered token is 466 | `SELBRACE` (select brace) instead of `LBRACE` to 467 | disambiguate between the clash of a general expression (a hash value) and the start of a select 468 | expression block. This is further discussed in the grammar / semantics of the language. 469 | 470 | Operators 471 | --- 472 | These are the operators of the Puppet Programming Language. They are lexicographically delivered 473 | as individual tokens. Their semantics are specified in [Expressions][1]. 474 | 475 | ``` 476 | = < > ! ? 477 | == <= >= != 478 | =~ !~ 479 | + - * / % 480 | << >> 481 | <| |> 482 | <<| |>> 483 | => +> 484 | -> <- 485 | ~> <~ 486 | @ @@ 487 | ~ 488 | ``` 489 | 490 | Heredoc 491 | --- 492 | A *Heredoc* is a lexical processing function that processes *out of band* text appearing on the lines (or if multiple heredocs are present on the same line, on the lines after the preceding 493 | heredoc), until an end marker specified by the heredoc. 494 | 495 | $a = [@(END1), @(END2)] 496 | This is the text in the first heredoc, until the end marker is seen 497 | END1 498 | This is the text in the second heredoc, until the end marker is seen 499 | END2 500 | 501 | The heredoc consists of an heredoc expression enclosed in `@( )`. The heredoc expression 502 | consists of a specification of the endtag, an optional syntax specification, and an optional 503 | specification of escape sequence processing. 504 | 505 | From a lexical perspective, the HEREDOC lexical function is recognized by: 506 | 507 | ``` 508 | HEREDOC 509 | : /@\(([^:\/\r\n\)]+)(?::[:blank:]*([a-z][a-zA-Z0-9_+]+)[:blank:]*)?(?:\/((?:\w|[$])*)[:blank:]*)?\)/ 510 | ; 511 | ``` 512 | 513 | Which is then processed by a separate heredoc processor for internal syntax: 514 | 515 | ``` 516 | HeredocExpression 517 | : '@' '(' EndTag (':' Syntax)? ('/' Escapes* )? ')' 518 | ; 519 | 520 | EndTag 521 | : DoubleQuotedEndTag | TextEndTag 522 | ; 523 | 524 | DoubleQuotedEndTag 525 | : /^"(.*)"$/ 526 | ; 527 | 528 | TextEndTag 529 | : /[^:\/\r\n\)]+/ 530 | ; 531 | 532 | Syntax 533 | : /[a-z][a-zA-Z_+]+/ 534 | ; 535 | 536 | Escapes 537 | : 't' | 'r' | 'n' | 's' | 'u' | 'L' | '$' 538 | ; 539 | ``` 540 | A recognized heredoc lexical function that does not comply with the heredoc processing rules 541 | raises an error. 542 | 543 | The text that belongs to the heredoc expression ends when a line begins with: 544 | 545 | ``` 546 | WHITESPACE? ('|' WHITESPACE?)? ('-' WHITESPACE?)? <> 547 | ``` 548 | 549 | Where `<>` denotes the `EndTag` *text* as given in the heredoc expression. The `|` is an optional marker that indicates where the left margin is, and the `-` denotes if right trimming should 550 | be performed on the last line of text. 551 | 552 | The lexical processing produces at least three tokens for a heredoc; a `HEREDOC` with a value corresponding 553 | to the Syntax expression part, followed by a token containing text positioning information, followed by a `STRING`, or `DQPRE`, `DQMID*`, `DQPOST` sequence. 554 | 555 | See [Heredoc][2] for details about the semantics. 556 | 557 | [2]: heredoc.md 558 | 559 | Template Mode 560 | --- 561 | The lexing process may be initialized in *Template Mode*. In this mode, the stream of source text starts in text/*Unquoted String* mode and allows for the source to weave logic into the text 562 | in various ways. 563 | 564 | The EPP lexical tokens are only recognized in templates. In general, the opening type tokens escape from text mode to expression mode (in various ways), and the closing type tokens returns from expression mode to text mode. 565 | 566 | ``` 567 | <% <%- <%= <%% <%# 568 | %> -%> 569 | 570 | ``` 571 | Template processing is detailed in TBD. REF TO EPP 572 | 573 | The lexical processing of EPP produces an `EPPSTART` token at the beginning of the text sequence. This 574 | token may be followed by tokens that constitute a parameter list. The lexical processing of 575 | the rest of the template is broken up into `RENDER_STRING` and `RENDER_EXPRESSION` tokens intermixed with regular tokens. 576 | 577 | Other 578 | --- 579 | All other tokens are delivered as an `OTHER` token, and will cause the grammar to issue a syntax error. 580 | -------------------------------------------------------------------------------- /language/module.md: -------------------------------------------------------------------------------- 1 | # Puppet Module Schema 2 | 3 | This document describes allowed (optional) and required files and directories in 4 | a Puppet Module Release. 5 | 6 | ## Table of contents 7 | 8 | * [What is a Puppet Module](#what-is-a-puppet-module) 9 | * [Allowed files and directories](#allowed-files-and-directories) 10 | * [Explanations and reasoning](#explanations-and-reasoning) 11 | * [Schema versioning](#schema-versioning) 12 | 13 | ## What is a Puppet Module 14 | 15 | A module in the sense of this document is a compressed tar archive. It is 16 | usually distributed via [forge.puppet.com](https://forge.puppet.com/). A module 17 | is usually developed in a git repository. Some files in the git repository are 18 | only used for development and testing. They should not be released. 19 | 20 | Common files often seen in a vcs repository that are used for development but 21 | shall not be released: 22 | 23 | `/spec`, `/Rakefile`, `/Gemfile`, `/.gitignore`, `/.github/`, `/.devcontainer`, `/Dockerfile`, `/.git` 24 | 25 | Note that above are just examples and not a complete list. The goal of this 26 | document is to provide an allowlist (*for a module release, not a VCS repo*), 27 | not a denylist. 28 | 29 | The official 30 | [Puppet documentation](https://www.puppet.com/docs/puppet/latest/modules_fundamentals.html) 31 | already explains what a module is and what it can contain. 32 | 33 | ## Allowed files and directories 34 | 35 | | Directories and Files | Purpose | 36 | |-----------------------|---------| 37 | | `/manifests/` | MAY contain Puppet code | 38 | | `/hiera.yaml` | A module MAY define a Hiera configuration for Hiera data within this module | 39 | | `/data/` | If the module has a `hiera.yaml`, the related data has to be within `/data/` | 40 | | `/templates/` | Stores [epp](https://www.puppet.com/docs/puppet/latest/lang_template_epp.html) (preferred) or [erb](https://www.puppet.com/docs/puppet/latest/lang_template_erb.html) templates | 41 | | `/files/` | Static files that Puppet code within the module will distribute | 42 | | `/examples/` | Example Puppet snippets that explain how to use the module. They can be used within acceptance tests | 43 | | `/facts.d/` | [External facts](https://www.puppet.com/docs/puppet/latest/external_facts.html) that are synced via [pluginsync](https://www.puppet.com/docs/puppet/latest/plugins_in_modules.html) | 44 | | `/lib/facter/` | MAY contain custom facts | 45 | | `/lib/puppet/type/` | Custom types | 46 | | `/lib/puppet/provider/` | Custom provider for one or multiple types | 47 | | `/lib/puppet/functions/` | Modern functions in Ruby for the new API | 48 | | `/lib/puppet/parser/functions/` | Legacy functions in Ruby | 49 | | `/lib/puppet_x/` | Custom Ruby modules to extend types, providers, functions or facts | 50 | | `/lib/augeas/lenses/` | Custom [Augeas](https://augeas.net/) lenses | 51 | | `/functions/` | MAY contain [functions written in Puppet DSL](https://www.puppet.com/docs/puppet/latest/lang_write_functions_in_puppet.html) | 52 | | `/metadata.json` | The `metadata.json` file MUST be present and MUST adhere to [Puppet's metadata](https://www.puppet.com/docs/puppet/latest/modules_metadata.html). [metadata-json-lint](https://github.com/voxpupuli/metadata-json-lint#metadata-json-lint) can be used to validate your file. | 53 | | `/README` | A README that describes what the module does. It's best practice to add a file extension like `.md`, `.rst` when a markup language is used | 54 | | `/LICENSE` | The `/LICENSE` file, with an optional file extension, SHOULD be included in the module. If the file is present, it MUST match `/metadata.json`'s license field. | 55 | | `/CHANGELOG` | A module SHOULD contain a changelog that's updated for every release. A new release SHOULD NOT alter existing changelog entries. It MAY use a file extension if a markup language is used. The [Puppet forge](https://forge.puppet.com/) supports the markdown markup language. | 56 | | `/docs/` | Directory for additional documentation | 57 | | `/REFERENCE.md` | [puppet-strings](https://www.puppet.com/docs/puppet/latest/puppet_strings.html) based documentation in markdown, updated on each release | 58 | | `/locales/` | Used for i18n support, can contain translated strings, deprecated within Puppet | 59 | | `/scripts/` | May serve static files, like `/files/` (see [PUP-11187](https://puppet.atlassian.net/browse/PUP-11187) for background) | 60 | | `/tasks/` | Contains [Tasks for Bolt](https://www.puppet.com/docs/bolt/latest/tasks.html) | 61 | | `/plans/` | Contains [Plans for Bolt](https://www.puppet.com/docs/bolt/latest/plans) | 62 | | `/types/` | Contains [type aliases](https://www.puppet.com/docs/puppet/latest/lang_type_aliases.html) | 63 | | `/bolt_plugin.json` | The file can contain metadata about [a Bolt plugin](https://www.puppet.com/docs/bolt/latest/writing_plugins.html#module-structure) | 64 | 65 | 66 | ## Mandatory files 67 | 68 | Mandatory are: 69 | * `/metadata.json` 70 | * `/README` 71 | * `/LICENSE` 72 | * `/CHANGELOG` 73 | 74 | ## Explanations and reasoning 75 | 76 | In the past, modules sometines contained a `/Modulefile`. It contained metadata 77 | about the module. The `/metadata.json` is the successor. A module can now only 78 | have a `/metadata.json`. It must not have a `/Modulefile`. 79 | 80 | The `/REFERENCE.md` file is optional. It's generated by puppet-strings. Some 81 | modules might use a different tool for documentation (and then cannot generate 82 | a `REFERENCE.md`). If a `/REFERENCE.md` is present in the release, it has to be 83 | up to date. 84 | 85 | ## Schema versioning 86 | 87 | This is version 0 of the schema. A changelog will be added when the schema is 88 | updated. 89 | 90 | A potential change is the removal of `/locales/`. The i18n support in puppet is 91 | currently deprecated, but still possible. When it's removed from Puppet and 92 | Puppetserver, the schema will be updated to reflect this. 93 | -------------------------------------------------------------------------------- /language/modus-operandi.md: -------------------------------------------------------------------------------- 1 | Modus Operandi 2 | === 3 | This chapter is about "method of operation" (modus operandi), a specification how code gets loaded, in which order, and the order of execution of a Puppet Program. 4 | 5 | Point of Entry / Boot Phase 6 | --- 7 | There are several points of entry 8 | 9 | * An agent request a catalog 10 | * A puppet apply 11 | * with code given on the command line as a string 12 | * with a reference to a file 13 | * running the code as per the configuration of puppet 14 | * The puppet configuration contains several settings that affect the entry point 15 | * the use of an External Node Classifier 16 | * the path to where environments are found 17 | * the code setting which may contain a string to execute 18 | * the staring set of manifests (one or several in one directory in an environment) 19 | * plugins that are evaluated early and that contribute to the configuration / resolution 20 | or have a direct impact on the produced catalog. 21 | * Utility commands that will evaluate a Puppet Program for the purpose of answering questions 22 | about it. 23 | 24 | ### Environment 25 | 26 | Almost all puppet applications require an Environment. An environment defines where the 27 | initial manifests are, and where modules are located. From Puppet 4.0 environments are 28 | always directory based, an environment is represented by a directory having the name of 29 | that environment and with a defined structure. The setting `environmentpath` defines where these directories are. 30 | 31 | An Environment may have its own `environment.conf` file in its root directory with 32 | overrides of default settings that apply only to this environment. 33 | 34 | *TODO: Creation of the injector should be done (or at least be able to lazily get the injector) 35 | as it depends on the boot, the environment and its module path. There may be injector specific 36 | settings for the environment. 3.7's injector is not directory environment ready.* 37 | 38 | *TODO: Loaders are configured at this point. They determine the visibility of modules from the module path, and the visibility each module has into other modules that they depend on.* 39 | 40 | ### 'code' Setting 41 | 42 | The `code` setting may contain a Puppet Program source string that is evaluated. 43 | 44 | *TODO: Is this pre-pended to the environment's set of manifests, or does this overshadow 45 | the use of environments, or is this the code for the "default environment" that does not 46 | have a directory with its content* 47 | 48 | ### Environment Manifests 49 | 50 | The manifests in the environment are placed in its `manifests` subdirectory. The `--manifest` settings 51 | can either reference a single `.pp` file, or to a directory (typically the /manifests directory). 52 | When a single file is referenced, only that manifest is used. If referencing a directory, the entire directory structure under this 53 | directory (inclusive) is recursively processed in alpha numerical order given by sorting the list of all paths. 54 | This is logically equivalent to concatenating all of the files in alphabetical order into one single file. 55 | 56 | International characters in file names should be avoided as the resulting order is platform specific (undefined) for characters outside of the ASCII range. The sort is currently not affected by the Locale, but may be in the future. 57 | 58 | ### At the end of the Boot Phase 59 | 60 | At the end of this phase, all infrastructure is configured for the evaluation of the Puppet Program that is defined by the settings, injected logic, initial manifests, and the modules on the module path. 61 | 62 | Catalog Building Phase 63 | --- 64 | The Catalog Building Phase commences after the Boot Phase. It performs an Evaluation of the initial 65 | Manifest(s) which in turn may trigger Loading and Evaluation of additional Manifests. When this initial step is finished, the Catalog Building matches the Node making the request with the 66 | set of known node definitions. The node definition that matches the current Node is then Evaluated (which in turn triggers the Loading and Evaluation of the remainder of the Puppet Program). The resulting catalog is then completed. 67 | 68 | The sub phases of the Catalog Building Phase are: 69 | 70 | * Initial Evaluation 71 | * Selection of a node definition 72 | * Evaluation of a node definition 73 | * Catalog Completion 74 | 75 | ### Initial Evaluation 76 | 77 | This is the execution of one Evaluation Phase for the purpose of establishing global values, 78 | and optionally a set of node definitions. See the Evaluation Phase for how Manifests are 79 | loaded and evaluated. 80 | 81 | *TODO: This phase defines the availability of trusted information from the node. This 82 | because logic loaded by the initial manifests may need this to establish what it 83 | wants.* 84 | 85 | ### Selection of node definition 86 | 87 | When the initial evaluation is completed, a set of global variables have been bound in top scope. 88 | Optionally there are now also a set of (unevaluated) node definition known to the Catalog Builder. 89 | 90 | A node expression has a match expression that is evaluated against information about the current node. 91 | 92 | The rules are: 93 | 94 | * *TODO: describe the node matching rules* 95 | * default node 96 | * nodes with regular expressions 97 | 98 | A node definition is always selected. This definition may be empty in which case, the initially 99 | evaluated logic and what it loads and evaluates is what defines the result. 100 | 101 | ### Evaluation of Node Definition 102 | 103 | The selected node definition is evaluated after the initial evaluation. This is the execution 104 | of one Evaluation Phase. 105 | 106 | 107 | Evaluation Phase 108 | --- 109 | The Evaluation Phase is a sub phase of the Catalog Building Phase. It is the general Load and Evaluate Loop. It behaves the same way at all times. 110 | 111 | Loading is either explicit; a path to logic to load, or symbolic; load the entity of a given 112 | type and name, which results in a search for the Manifest that should contain the wanted named 113 | entity. Both ways of loading result in the same behavior once the Manifest file to load 114 | is known. 115 | 116 | The Manifest is Lexed and Parsed into an Abstract Syntax Tree (a Puppet Program Model). The system 117 | may fail with an error if it was not able to complete this. Such a failure aborts the current transaction. Once a Puppet Program Model has been produced, this model is validated against 118 | semantic rules. This may results in a set of warnings and errors. If there are errors, these 119 | are reported and the transaction is aborted. Warnings are only reported. 120 | 121 | The next step is to add all named definitions to the set of known definitions. Named definitions 122 | are ClassDefinition, ResourceTypeDefinition, FunctionDefinition, and NodeDefinition. Their respective 123 | definitions are not evaluated at this point, the result is a set of associations between their type/name and their definition. 124 | 125 | If asked to load something where there is already a type/name => definition association, the already 126 | known definition is used. 127 | 128 | It is an error if a new load produces a type/name that already has an association with a definition 129 | (redefinition error). 130 | 131 | The body of the Manifest itself is then evaluated. As this body is the container of NamedDefinition, the evaluation of those result in a no-op (silent step over). 132 | 133 | When evaluation requires a name/type element this triggers loading of that name/type and the 134 | Evaluation Phase recurses. 135 | 136 | A name/type is required when: 137 | 138 | * A function is called (function/name => definition is needed in order to evaluate the function) 139 | * A call to `include`, `require`, `contain`, is made (class/name => definition 140 | is needed in order to evaluate the class). If the definition has already been evaluated, this 141 | is a no-op. 142 | * A queued expression is evaluated, and it requires a type/name => definition association. 143 | 144 | The evaluation of the following expressions are queued: 145 | 146 | * Resource Instantiation 147 | 148 | The evaluation of the following expressions are placed in a queue that is evaluated during 149 | the Catalog Completion Phase. 150 | 151 | * Relationship Expressions 152 | * Queries 153 | 154 | The evaluation of a Manifest continues until all expressions have been evaluated (top-down). Evaluation then continues to evaluate all queued expression. This may trigger new recursive loading, 155 | type/name => definition association, and queuing of more expressions. 156 | This evaluation continues until the evaluation queue is empty. 157 | 158 | Catalog Completion Phase 159 | --- 160 | The Catalog Completion Phase evaluates all Queries, and then all Relationship Expressions. 161 | This may in turn trigger new Evaluation Phases if anything is placed in the evaluation queue. 162 | 163 | Once there is nothing more to evaluate, and there are no unevaluated queries or relationship 164 | expressions, the catalog is finished, validated and transformed to the requested catalog format. 165 | Typically, this catalog is sent to an agent for application (synchronization of actual state with 166 | the desired state described in the produced catalog), but catalogs may be produced for 167 | other reasons (testing, documentation, etc.). 168 | 169 | *TODO - the list of things to describe:* 170 | 171 | * manifest ordering 172 | * what the queries result in 173 | * what the relationships result in 174 | 175 | Application Phase 176 | --- 177 | The Application Phase is the phase that occurs when a completed catalog is acted upon 178 | by an agent for the purpose of synchronizing its managed resources with the desired state 179 | described in the given catalog. 180 | 181 | *TODO - the list of things to describe:* 182 | 183 | * generated resources 184 | 185 | 186 | Auto Loading 187 | --- 188 | Referenced elements of the language are automatically loaded based on their name. This applies 189 | to: 190 | 191 | * classes 192 | * resource types (plugins in Ruby and user defined resources) 193 | * functions (both 3x and 4x APIs) 194 | 195 | Auto loading has the following rules: 196 | 197 | * A "definition" must be stored in a file that corresponds to its name in order for it 198 | to be automatically loaded. 199 | * Nested constructs are only visible if parent has been loaded, there is no search for elements 200 | inside of files with a name different than the file. 201 | * Auto Loading is performed from the perspective of the code that triggers the loading. 202 | 203 | The semantics of the 3x loader is that everything is visible to everything else. These semantics apply to all types of elements except functions defined using the 4x function API. 204 | 205 | The 4x function API uses the 4x loaders which restrict the visibility by using the following rules. 206 | 207 | * Elements defined by Puppet runtime (e.g. logging functions) is visible to everything else 208 | * Everything in the Puppet runtime (the core) is available to everything else 209 | * Everything defined at the environment level is visible to all modules in that environment 210 | * A module without declared dependencies sees what is generally available, and all other modules 211 | * A module with declared dependencies has visibility into what is generally available and the modules 212 | on which it depends. 213 | 214 | ### Auto Loading Details 215 | 216 | #### Roots for 3.x Loader 217 | 218 | The 3.x based autoloader will load from the following roots (in order): 219 | 220 | * The source location where gem are installed (as dictated by runtime configuration) 221 | * All modules from an environment's expanded module path (in the order resulting from expansion) 222 | * Puppet's `libdir` setting 223 | * The Ruby variable `$LOAD_PATH` 224 | 225 | #### Subdirectory per kind 3.x. Loader 226 | 227 | The loader appends `/manifests` to all `.pp` candidates, and `/lib/puppet` and a path fragment for the kind of thing being loaded for all entities being loaded that are `.rb` based. The path 228 | fragments per kind are: 229 | 230 | * `/parser/functions` - for 3.x functions (`.rb` only) 231 | * `/type` - for 3.x (resource) types (`.rb` only) 232 | 233 | #### Roots for 4.x Loader 234 | 235 | The 4.x autoloader (loading 4.x functions only in the Puppet 4.x series), will load from the following roots (search order is determined by what is loaded, its name, and the closure of the instruction causing the load to take place): 236 | 237 | * The directory corresponding to an environment (`.pp` and `.rb`). 238 | * All modules from an environment's expanded module path in the order resulting from expansion (`.pp` and `.rb`). 239 | * The directory where puppet/lib/* is loaded from (`.rb` based load only) 240 | 241 | #### Subdirectories per type of function 242 | 243 | * for `.rb` functions the fragment `/functions` is added to `puppet/lib` 244 | * for `.pp` functions the fragment `/functions` is added to the root 245 | 246 | #### Name to filename transformation 247 | 248 | The fully qualified and absolute name of the searched entity is transformed to a path fragment where each name segment in lower case becomes a path segment. 249 | 250 | #### Search for a Name 251 | 252 | A search for an entity differs in the 3.x and 4.x API. In both cases no loading will take place if the named entity is already defined. 253 | 254 | In 3.x the search starts with the most specific path (all name segments expanded). If that does not yield an existing file the search continues by dropping the most specific part of the name. This search continues until an existing file is found. That file is then loaded (if not already loaded). After the load, the wanted named entity is expected to have been created. The operation fails if that is not the case. 255 | 256 | In 4.x the search is strictly confined to the most specific path (all name segments expanded). Search is also confined to the root denoted by the first name segment (module name), or lack thereof (searches environment). 257 | 258 | #### Recursive Loading of initial Manifest 259 | 260 | Puppet will automatically load either a single `.pp` file or recursively load `.pp` files as directed by the `manifest` setting. Whatever is loaded during this initial load will be defined and available irrespective of their given name spaces. 261 | 262 | #### Loadable Paths 263 | 264 | The overall set of paths that will be searched are defined by the roots, the kind of searched entity, and files that are valid lower case representation of a name of the given kind. In addition to this set of paths, all `.pp` files appointed to by the `manifest` setting are loaded. 265 | 266 | Since all auto loadable entities must start each name segment with a letter a-z, and may be followed by letters a-z, the underscore character _, and digits 0-9; only file names matching the naming rule, and being contained in either a root, or a directory named with a name that match the naming rule will be eligible for loading. 267 | 268 | The name rule does not apply to files appointed (since or recursively) by the `manifest` setting. 269 | 270 | 271 | -------------------------------------------------------------------------------- /language/names.md: -------------------------------------------------------------------------------- 1 | Names 2 | === 3 | *Qualified Names* and *Qualified References* are used to refer to entities declared in a Puppet Program or to entities declared via plugins to the Puppet Runtime System. 4 | 5 | A **Qualified Reference** is a reference to a type; one of: 6 | 7 | * Built in types 8 | * User defined resource type 9 | * Resource type runtime plugin 10 | * A Puppet Class 11 | 12 | A **Qualified Name** is a reference to a value (of some type), the type and scope depends on which operators are being used. 13 | 14 | $x # name is a reference to the variable named 'x' 15 | sprintf("%x4", 256) # name is a reference to the function named 'sprintf' 16 | 17 | Names and References are either *simple* consisting of a single identifier, or *qualified*, consisting 18 | of a sequence of identifiers separated by `::` tokens. A qualified name/reference may start with `::` which makes the reference *absolute*. 19 | 20 | In determining the meaning of a name or reference, the context of the occurrence is used to disambiguate among the various kinds of named elements. 21 | 22 | Access control can be specified for a named element making this element only visible to a restricted set of contexts. Access control is different from scope. Access specifies the part of the program 23 | text within which the declared entity can be referenced by a qualified name/reference. 24 | 25 | Declarations 26 | --- 27 | A declaration introduces an entity into a program and introduces an identifier that can be used to refer to this entity. The ability to address an entity depends on where the declaration and 28 | reference occurs - this is specified by Access Control and Scope. 29 | 30 | A declared entity is one of the following: 31 | 32 | * built in type 33 | * plugin defined Resource type 34 | * plugin defined Function 35 | * (host) Class 36 | * Class Parameter 37 | * plugin defined Resource type parameter 38 | * Variable 39 | * Node 40 | * Resource Instance 41 | 42 | Scope of Declaration 43 | --- 44 | The scope of a declaration is the region of the program within which the entity declared by the declaration can be referred to using a simple name, provided it is visible. 45 | 46 | TODO: List of declarations and their scopes (W.I.P) 47 | 48 | * All functions are always in scope 49 | * All types are always in scope 50 | * 51 | * The scope of a variable declaration in a Class is 52 | * The scope of a Class 53 | 54 | 55 | Shadowing, Obscuring, and Overloading 56 | --- 57 | A local variable can only be referred using a simple name (never a qualified name). 58 | 59 | It is not allowed to redeclare variables or parameters in the same scope. 60 | A local scope may redeclare variables declared in outer scopes. These shadow the outer scope 61 | declaration(s). 62 | 63 | It is never allowed to declare a variable in a scope other than the current scope (i.e. it is 64 | not possible to assign a value to a variable in another scope by using a qualified name). 65 | 66 | Technically a Shadow is a new declaration of an existing entity, and Obscuring is the declaration of 67 | a name in a context in such a way that it obscures a named entity in another scope (i.e. it is not 68 | a shadow, but its existence blocks a name to resolve to what it obscures). 69 | 70 | The Puppet Programming Language does not have any obscuring constructs as there is a distinction between variable references $var, names (initial lower cased Qualified Name), and references 71 | (initial upper case Qualified Reference). 72 | 73 | Overloading is the term used to resolve the binding of declaration to definition in the 74 | presence of multiple definitions of the same name. In the Puppet Programming Language overloading 75 | takes place when loading a named entity from a search path, the first entity found 76 | on the path wins. If a named entity has already been introduced, this will block access to 77 | an entity with the same name that would otherwise have been loaded. 78 | 79 | Future versions of this specification may impose stricter rules on where a named 80 | entity may be defined; this to reduce the risk of accidental overloading. 81 | 82 | 83 | Scopes 84 | --- 85 | These constructs introduce a new scope: 86 | 87 | * The Program's top scope 88 | * A (host) Class 89 | * A user defined Resource type 90 | * A lambda 91 | * A Node 92 | 93 | ### Top Scope 94 | 95 | Top scope is created automatically when the execution of the program begins. Top scope 96 | is the program region(s) outside of classes, user defined resource types, nodes and lambdas. 97 | 98 | There is no difference between the top scope region in different source code files. There is 99 | only one top scope. 100 | 101 | ### Class Scope 102 | 103 | All Classes are singletons - there is never more than one instance of a given class. A parameterized 104 | class may only be instantiated once - there can never be two instances of the same class with 105 | different parameters in the same compilation. 106 | 107 | Parameters and Variables in a Class C is in scope for C, and in all classes inheriting C. 108 | 109 | All Parameters and Variables are by default public and are visible to all other scopes. 110 | Parameters and Variables declared private are visible only to C. 111 | 112 | ### User Defined Resource Scope 113 | 114 | A user defined resource type (just like plugin defined resource types) may be instantiated multiple 115 | times provided instances are given unique identifiers. 116 | 117 | Variables (other than those referring to the resource parameters) in a user defined resource type are always private - they are not visible to any region outside of the resource type body. 118 | 119 | A resource's parameters can not be externally referenced via variables, but can be obtained 120 | via an access expression: 121 | 122 | Resource[Type, title][param_name] 123 | 124 | The result of such an operation is evaluation order dependent; the resource must have been created, evaluated, and the parameter must have been given a value. 125 | 126 | 127 | Determining the meaning of a name 128 | --- 129 | ### In General 130 | 131 | A Qualified Name in general is a Bare Word that evaluates to a String. 132 | 133 | hello == 'hello' 134 | abc::xyz == 'abc::xyz' 135 | ::xyz == '::xyz' 136 | 137 | ### Function Name 138 | 139 | A Qualified Name is a reference to a function when: 140 | 141 | * it appears where an expression is accepted and: 142 | * it is followed by arguments in parentheses e.g. `abc()` 143 | * it is the RHS in a dot binary expression e.g. `$a.abc` 144 | * it appears in the top level body of a Program, conditional construct, define, or class, and: 145 | * the name is one of the built in statement type functions, and 146 | * is followed by one or a comma separated list of argument expressions (this is known as 147 | an un-parenthesized function call). 148 | 149 | **TODO:** Reference to where the list of statement like functions are 150 | 151 | ### Resource Type Reference 152 | 153 | A Qualified Name is a reference to a Resource Type (plugin or definition expression) when it appears 154 | as the type name in a Resource Expression. e.g. the name `file` in this example: 155 | 156 | file { title: } 157 | 158 | ### Attribute Reference 159 | 160 | A Qualified Name is a reference to an Attribute of a Resource win it appears as the LHS in 161 | an Attribute Operation (operators `=>` and `+>`) 162 | 163 | ### Name of Entity 164 | 165 | A Qualified Name is a name in Class-, and Define-expressions. It is not a name in a Node expression, there the name parts can be composed with interleaved dots, and with numbers to form a string. 166 | 167 | A Qualified Name is a reference to the parent class when used after `inherits` in a class definition. 168 | 169 | Access Control 170 | --- 171 | **NOTE: Access Control is not fully specified and has not yet been implemented.** 172 | 173 | Access control in the Puppet Programming Language consists of the keyword `private` applied to: 174 | 175 | * class 176 | * user defined resource type 177 | * parameter 178 | * variable 179 | 180 | This is explained in the following sections. 181 | 182 | ### Class 183 | 184 | A private class may only be directly used from the module where it is declared. The term 185 | *directly-used* means: 186 | 187 | * including the class 188 | * requiring the class 189 | * instantiating the class as a resource 190 | * inheriting from the class 191 | * forming a relationship with the private class 192 | 193 | The class itself is visible in the catalog, and it is possible to reference it and its parameters. 194 | Since its existence in the catalog is public so is referencing it, and its parameters. Access to a private class and its parameters will issue a warning as the intention is that it is an implementation concern of the particular module. 195 | 196 | **TODO: The rules for access to resource instances and parameters for private resources/params 197 | requires more thought - warning / error / simply not visible...** 198 | 199 | ### User Defined Resource Type 200 | 201 | A private user defined resource type may only be directly used from the module where it is declared. 202 | The term *directy-used* means: 203 | 204 | * instantiating an instance of the resource type 205 | * forming a relationship 206 | 207 | The type itself is visible, and so are the instances that are created. Access to instances of a private resource type and its parameters will issue a warning as the intention is that it is an implementation concern of the particular module. 208 | 209 | ### Parameters 210 | 211 | A private parameter may only be directly used from the module where it was declared. The term 212 | *directly-used* here means: 213 | 214 | * referencing the parameter using variable syntax 215 | * referencing the parameter via catalog lookup of type/id and then accessing its parameter 216 | 217 | Since the result is still visible in the catalog, there is nothing preventing it from being accessed using catalog/type/id lookup, but a warning will be issued in these cases. 218 | 219 | ### Variable 220 | 221 | A private variable is only available in the scope where it is declared (and inner local scopes). It is only meaningful to declare private variables in top, node and class scope (all other variables are already private/local to their respective scopes). 222 | 223 | A private variable is simply not visible to other scopes. 224 | 225 | 226 | 227 | -------------------------------------------------------------------------------- /language/parameter_scope.md: -------------------------------------------------------------------------------- 1 | Parameters 2 | === 3 | 4 | The Parameters supported in different kinds of definitions share a common 5 | grammar for parameter definitions - the `ParameterList` rule: 6 | 7 | ``` 8 | ParameterList 9 | : '(' parameters += Parameter (',' parameters += Parameter)* ','? ')' 10 | ; 11 | 12 | Parameter 13 | : type = Expression? captures_rest='*'? name = QualifiedName 14 | ('=' default_expression = Expression)? 15 | ; 16 | ``` 17 | 18 | Some features may not be semantically valid for a particular kind of definition. 19 | Classes and Defines does not currently support "captures_rest". This may change 20 | in the future. 21 | 22 | Parameter Value Binding 23 | === 24 | 25 | When arguments are given during evaluation (i.e a function call, declaration of a class or resource) the argument values are assigned to variables as specified by the definition's parameters. 26 | 27 | If a parameter is not given a value: 28 | 29 | * An automatic data binding lookup is performed. 30 | * If that did not produce a value, a specified `default_value` expression is evaluated 31 | and its result is used as the value of the corresponding variable. 32 | 33 | In all cases: 34 | 35 | * If the parameter's type does not accept the value (a given argument, value from data lookup, 36 | or value from default expression evaluation), an error is raised. 37 | 38 | The Default Expression Evaluation 39 | --- 40 | 41 | When a default value expression is evaluated, the runtime uses a special parameter scope. 42 | This is an intermediate scope that is used only while assigning argument values, performing lookups and evaluating default value expressions. Once all parameter variables have been given their value, this context ceases to exist, and the resulting variables are set in the scope that is used to evaluate the body of the definition (e.g. the body of a function). 43 | 44 | The parameter variables are give values from left to right. The parameter's value is bound to a variable that is available in the Parameter Scope. This means that default value expressions can access variables that are to the left (comes before) the parameter being processed. 45 | 46 | The Parameter Scope, in addition to the parameter variables, also provides access to the top level scope. 47 | 48 | 49 | 50 | 54 | 55 | 64 |
Access to Parameter Variables, and Variables in 'scope':
Since Puppet 4.4.0
51 | Are defined by this specification. A strict, "variable to the left may be referenced" rule 52 | applies. The rules for how to access variables in other scopes are also specified. 53 |
Before Puppet 4.4.0
56 | The behavior was undefined. In some cases variable values to the left could be accessed, 57 | in other cases it did not work. Default value expressions could in some cases 58 | reference variables in the calling scope! It could also introduce variables 59 | in the calling scope as a result of the undefined and untested behavior. 60 |
61 | For versions before Puppet 4.4.0, default value expressions should only use literal values 62 | or make references to fully qualified variables that a user knows are already evaluated. 63 |
65 | 66 | 67 | ### A default expression may reference a parameter to its left 68 | 69 | | Function | 70 | | -------- | 71 | | `function example($a = 10, $b = $a) {`
`}` 72 | 73 | 74 | | When called like this... | the parameters are set like this: | 75 | | ---- | ---- | 76 | | `example(0)` | `$a = 10`
`$b = 10` 77 | | `example(2)` | `$a = 2`
`$b = 2` 78 | | `example(2, 5)` | `$a = 2`
`$b = 5` 79 | 80 | 81 | 82 | ### It is an error to reference a variable to the right of the default expression 83 | 84 | | Function | 85 | | -------- | 86 | | `function example($a = 10, $b = $c, $c = 20) {`
`}` 87 | 88 | | When called like this... | the parameters are set like this: 89 | | --- | --- 90 | | `example(1,2,3)` | `$a = 1`
`$b = 2`
`$c = 3` 91 | | `example(1,2)` | `$a = 1`
`$b = 2`
`$c = 20` 92 | | `example(1)` | error, default expression for $b tries to illegally access not yet evaluated $c 93 | 94 | 95 | ### The evaluation of each parameter has separate match scope 96 | 97 | Numerical variables refer to the result of the last performed regular expression 98 | match. When evaluating parameters, these are reset before evaluation 99 | of each parameters' default expression. numerical variables are undef when there is no 100 | preceding match, or where such a match did not produce a matching capture for the numeral. 101 | 102 | ### Numerical variables are undef when there is no match scope 103 | 104 | | Function | 105 | | -------- | 106 | | `function example($a = $0, $b = $1) {`
`}` 107 | 108 | | When called like this... | the parameters are set like this: 109 | | --- | --- 110 | | `example()` | `$a = undef`
`$b = undef` 111 | 112 | 113 | ### A match scope for a parameter does not affect match scopes to its right 114 | 115 | | Function | 116 | | -------- | 117 | | `function example($a = ['hello' =~ /(h)(.*)/, $1, $2], $b = $1) {`
`}` 118 | 119 | | When called like this... | the parameters are set like this: 120 | | --- | --- 121 | | `example()` | `$a = [true, 'h', 'ello']`
`$b = undef` 122 | 123 | | Function | 124 | | -------- | 125 | | `function example($a=['hello' =~ /(h)(.*)/, $1, $2], $b=['hi' =~ /(h)(.*)/, $1, $2], $c=$1) {`
`}` 126 | 127 | | When called like this... | the parameters are set like this: 128 | | --- | --- 129 | | `example()` | `$a = [true, 'h', 'ello']`
`$b = [true, 'h', 'i']`
`$c = undef` 130 | 131 | 132 | ### Match scopes nest per parameter 133 | 134 | | Function | 135 | | -------- | 136 | | `function example($a = ['hi' =~ /(h)(.*)/, $1, if 'foo' =~ /f(oo)/ { $1 }, $1, $2], $b = $0) {`
`}` 137 | 138 | | When called like this... | the parameters are set like this: 139 | | --- | --- 140 | | `example()` | `$a = [true, 'h', 'oo', 'h', 'i']`
`$b = undef` 141 | 142 | 143 | Note that the top level match scope is restored at index 3, and empty again for the next parameter (`$b`). 144 | 145 | ### A Match scope from other scopes is not available: 146 | 147 | | Function | 148 | | -------- | 149 | | `'foo' =~ /(f)(o)(o)/`
`function example($a = $0 {`
`}` 150 | 151 | | When called like this... | the parameters are set like this: 152 | | --- | --- 153 | | `function caller() {`
  `'foo' =~ /(f)(o)(o)/`
  `example()`
`}`
`caller()` | `$a = undef` 154 | | `function caller() {`
  `example()`
`}`
`'foo' =~ /(f)(o)(o)/`
`caller()` | `$a = undef` 155 | 156 | 157 | ### Assignments are not allowed except in a nested scope: 158 | 159 | | These Function Definitions | All result in error 160 | | -------- | :-----: 161 | | `function example($a = $x = $10) { }` | Assignment not allowed here 162 | | `function example($a = [$x = 10]) { }` | - " - 163 | | `function example($a = $a) { }` | - " - 164 | | `function example($a = ($b = 3), $b = 5) { }` | - " - 165 | | `function example($a = 10, $b = ($a = 10)) { }` | - " - 166 | 167 | 168 | ## Assignments are allowed inside lambdas that are nested in the default expressions 169 | 170 | ``` 171 | function example( 172 | $a = [1,2,3], 173 | $b = 0, 174 | $c = $a.map |$x| { $b = $x; $b * $a.reduce |$x, $y| {$x + $y}} 175 | ) { } 176 | 177 | example() 178 | ``` 179 | 180 | results in: 181 | 182 | ``` 183 | $a => [1,2,3] 184 | $b => 0 185 | $c = [6, 12, 18] 186 | ``` 187 | 188 | ### Nested lambdas have access to enclosing match scope 189 | 190 | ``` 191 | function example($a = case "hello" { 192 | /(h)(.*)/ : { 193 | [1,2,3].map |$x| { "$x-$2" } 194 | } 195 | }) 196 | example() 197 | ``` 198 | 199 | results in: 200 | 201 | ``` 202 | $a = ["1-ello", "2-ello", "3-ello”] 203 | ``` 204 | 205 | ### Nested lambdas have access to parameter scope 206 | 207 | ``` 208 | function example($a = "hello", 209 | $b = [1,2,3].map |$x| { "$x-$a" }) 210 | } 211 | example() 212 | ``` 213 | 214 | results in: 215 | 216 | ``` 217 | $a = ["1-hello", "2-hello", "3-hello”] 218 | ``` 219 | 220 | ### Parameter scope is not available in function block 221 | 222 | ``` 223 | function example($a = "hello" =~ /.*/) { 224 | notify { test: message "Y$0es" } 225 | } 226 | example() 227 | ``` 228 | 229 | results in: A notify “test” with message “Yes” 230 | 231 | ### Accessing earlier match results can be done using the match function: 232 | 233 | ``` 234 | function example( 235 | $a = "hello".match(/(h)(.*)/), 236 | $b = $a[0], 237 | $c = $a[1] 238 | ) { } 239 | example() 240 | ``` 241 | 242 | results in: 243 | 244 | ``` 245 | $a == ['hello', 'h', 'ello'] 246 | $b == 'hello' 247 | $c == 'h' 248 | 249 | ``` 250 | 251 | ### Duplicate keys in arguments by name calls are errors: 252 | 253 | ``` 254 | define example($a) { } 255 | example { 'test': 256 | a => 10, 257 | a => 20, 258 | } 259 | ``` 260 | 261 | result: 262 | 263 | ``` 264 | error, duplicate specification of parameter $a 265 | ``` 266 | 267 | ### Defines and Classes allows access from the right in all cases: 268 | 269 | ``` 270 | define example($a, $b=$a) { } 271 | example { test: a=>10 } 272 | ``` 273 | 274 | ``` 275 | define example($a=5, $b=$a) { } 276 | example { test: a=>10 } 277 | ``` 278 | 279 | ``` 280 | define example($a=10, $b=$a) { } 281 | example { test: } 282 | ``` 283 | 284 | In all the examples, the result would be: 285 | 286 | ``` 287 | $a = 10 288 | $b = 10 289 | ``` 290 | 291 | This works the same way for `class`, with the additional rules when there is a value bound to `$a` in data binding: 292 | 293 | 1. the bound value is used when an argument was not given 294 | 2. a default expression is then not evaluated 295 | 296 | 297 | ### Access to parameters in an outer scope 298 | 299 | There are issues when referencing variables that do not exist in the parameter list, but exist in an outer scope. 300 | 301 | ### Functions 302 | 303 | Functions can only access global scope and fully qualified variables referencing class parameters for evaluated classes. Such references are subject to evaluation order and should be avoided. 304 | 305 | Functions cannot access variables in node scope. 306 | 307 | ``` 308 | class foo { 309 | $bar = '$bar in foo' 310 | } 311 | include foo 312 | 313 | $surprise = '$surprise in top scope' 314 | 315 | node default { 316 | $surprise = '$surprise in node scope' 317 | } 318 | 319 | function example($a = $surprise, $b = $foo::bar) { 320 | notice $a 321 | notice $b 322 | } 323 | example() 324 | ``` 325 | 326 | Would notice: 327 | 328 | ``` 329 | $surprise in top scope 330 | $bar in foo 331 | ``` 332 | 333 | ### Metaparameters are available in Parameter Scope 334 | 335 | ``` 336 | define example($a = $title) { } 337 | example { 'hello' : } 338 | ``` 339 | 340 | results in: 341 | 342 | ``` 343 | $a == 'hello' 344 | ``` 345 | -------------------------------------------------------------------------------- /language/plugin-api.md: -------------------------------------------------------------------------------- 1 | Plugin API 2 | === 3 | W.I.P 4 | 5 | Type System 6 | --- 7 | This section contains typical operations performed in Ruby that deals with the Puppet 8 | Type System. 9 | 10 | ### Type Algebra 11 | 12 | Compute the Puppet Type of an instance: 13 | 14 | t = Puppet::Pops::Types::TypeCalculator.infer(any_object) 15 | 16 | Obtain an instance of a type: 17 | 18 | t = Puppet::Pops::Types::TypeFactory.array_of_data() 19 | 20 | Is the instance of a particular type: 21 | 22 | t = Puppet::Pops::Types::TypeFactory.array_of_data() 23 | Puppet::Pops::Types::TypeCalculator.instance?(t, some_object) 24 | 25 | Is the inferred type assignable (compatible/ narrower) than a particular type? 26 | 27 | data_array_T = Puppet::Pops::Types::TypeFactory.array_of_data() 28 | t = Puppet::Pops::Types::TypeCalculator.infer(any_object) 29 | Puppet::Pops::Types::TypeCalculator.assignable?(data_array_T, t) 30 | 31 | ### Converting Type to/from String 32 | 33 | All types can be represented in String form. 34 | 35 | Converting to String: 36 | 37 | t.to_s 38 | 39 | Parsing a String to Type: 40 | 41 | t = Puppet::Pops::Types::TypeParser.new().parse(s) 42 | # Raises Puppet::ParseError if the string does not represent a valid type 43 | 44 | ### Iteration of Type 45 | 46 | Types that supports enumeration may not support this directly themselves. To iterate 47 | over a type's instances perform this: 48 | 49 | enumerator = Puppet::Pops::Types::TypeCalculator.enumerable(t) 50 | if enumerator 51 | enumerator.each {|x| ...} 52 | else 53 | # not an enumerable type 54 | end 55 | 56 | (The `Integer` type is enumerable, but currently no other types). 57 | 58 | #### Iteration and size 59 | 60 | The method `size`, and any method that requires knowing all entries may potentially be 61 | very expensive if all items have to be retrieved in order to compute the size. 62 | 63 | (At this point, only the `Integer` type with bound range supports iteration, and it knows its size). 64 | 65 | Function 66 | --- 67 | 68 | ### Lambda 69 | A lambda / closure is passed to a function as the very last argument. An implementation should check 70 | if the last argument is a lambda by checking if it responds to `:puppet_lambda`. 71 | 72 | if args[-1].respond_to?(:puppet_lambda) 73 | closure = args[1] 74 | else 75 | closure = nil 76 | end 77 | 78 | A call to the lambda is performed by simply passing scope and arguments to it: 79 | 80 | closure.call(scope, arg1, arg2, ...) 81 | 82 | 83 | -------------------------------------------------------------------------------- /language/puppet-functions.md: -------------------------------------------------------------------------------- 1 | Puppet Functions 2 | === 3 | 4 | Puppet Functions is simply the ability to write functions in Puppet. 5 | 6 | Grammar 7 | --- 8 | 9 | FunctionDefinition 10 | : 'function' name = NAME parameters = ParameterList? return_type = ReturnType? '{' statements += statements '}' 11 | ; 12 | 13 | ParameterList 14 | : '(' parameters += Parameter ')' 15 | ; 16 | 17 | Parameter 18 | : type=Expression? captures_rest='*'? name=QualifiedName ('=' value = Expression)? 19 | ; 20 | 21 | ReturnType 22 | : '>>' Expression 23 | ; 24 | 25 | The `Parameter` definition is the same as for Lambda. 26 | 27 | * Only `FunctionDefinition` and `Lambda` can use `captures_rest` in their `ParameterList` since they use arguments passed 'by position'. 28 | * A `captures_rest` parameter (if used), must be placed last in the list 29 | * A parameter with a default value can not be placed after one that that does not have one. 30 | * A default expression may refer to parameters defined in a parameter that is to the left of it. (once PUP-1985 is implemented - undefined otherwise) 31 | * The default expressions are evaluated in the **function's closure scope** (i.e. global scope), with 32 | the exception of parameters defined in the function list, such that: 33 | * it is an error to refer to a (non '::' anchored) variable name that appears as a parameter 34 | to the right, even if it could be resolved to a globally available value. 35 | * A `captures_rest` parameter will always receive an Array. 36 | * A `captures_rest` parameter's default can be given as a single value, or an `Array`. 37 | If a single value is given, it is automatically wrapped in an `Array`. 38 | * The type of the `captures_rest` parameter is either a non-Array type which becomes the element type 39 | of each captured element, or an Array Type that may cap the number or entries as well as specifying 40 | the element type. As a consequence, to capture a sequence of `Array[T]`, this must be specified as 41 | `Array[Array[T]]`. 42 | * Since 4.7.0 it is possible to specify the expected return type of the function. When it is specified an 43 | automatic type assertion will be made against the value produced by the function. 44 | 45 | 46 | Discussion: 47 | 48 | The above rules for `captures_rest` are motivated by the thought that the most common use is 49 | not to pass multiple arrays, and it is more convenient to write `foo(String *$rest)` then to have to write `foo(Array[String] *$rest)`, since the `*` already implies `Array`. The consequence is when using that shorthand notation is that an array of arrays must be written `Array[Array[T]]` - which is expected to be far more uncommon, than either capping the list 50 | (e.g. `foo(Array[String,1,10] *$rest)`, or passing a variable number of arrays. 51 | 52 | See [Parameter Scope][1] for more information about scoping rules an variable access in a default value expression. 53 | 54 | Lambda Support 55 | --- 56 | Functions written in the Puppet Language does not support lambdas / code blocks. While it is 57 | possible to call a puppet function with a lambda, and possibly also type a parameter as accepting a `Callable` parameter, this can not be put to any practical use since there is no support for calling the given block. It expected that this will be supported in a later version of this specification. Until then, the behavior of calling a puppet function with a code block is undefined. 58 | 59 | 60 | Function Definition - Scope & Autoloading 61 | --- 62 | 63 | ### Autoloading 64 | 65 | Namespaced Puppet Functions are auto loaded from modules and the environment when located under /functions or /functions respectively. 66 | 67 | Auto loaded puppet functions should always be namespaced; in a module, it's required to use the module name. In an environment, it's highly recommended to use the special name `environment` (i.e. not the *name* of the environment since that typically changes as code is being developed, tested, put into production and then maintained, etc.). 68 | 69 | The name of the .pp file must match the simple (non namespaced name part) of the function. Thus, a function **'testmodule::min'** in module 'testmodule' is located like this: 70 | 71 | testmodule 72 | |- functions 73 | | min.pp 74 | 75 | With the following contents in `min.pp` (note use of full namespace): 76 | 77 | function testmodule::min($a, $b) { 78 | # ... 79 | } 80 | 81 | And a function **'environment::min'** in a 'production' environment like this: 82 | 83 | production 84 | |- functions 85 | |- environment 86 | | min.pp 87 | 88 | With the following contents in `min.pp` (note use of namespace 'environment'): 89 | 90 | function environment::min($a, $b) { 91 | # ... 92 | } 93 | 94 | Nested name spaces are allowed in both modules and the environment - e.g. a function **'testmodule::math::min'** would be located like this: 95 | 96 | testmodule 97 | |- functions 98 | |- math 99 | | min.pp 100 | 101 | 102 | With the following contents in `min.pp` (note use of full namespace): 103 | 104 | function testmodule::math::min($a, $b) { 105 | # ... 106 | } 107 | 108 | And a a function **'environment::math::min'** would be located like this: 109 | 110 | production 111 | |- functions 112 | |- environment 113 | |- math 114 | | min.pp 115 | 116 | With the following contents in `min.pp` (note use of full namespace): 117 | 118 | function environment::math::min($a, $b) { 119 | # ... 120 | } 121 | 122 | In an environment, it is only possible to declare functions in the global namespace (not recommended) or in the special namespace 'environment'. Functions 123 | added in other namespaces will not be found. E.g. declaring function **'someother::min'** like this will not make it visible to the auto loader: 124 | 125 | production 126 | |- functions 127 | |- someother 128 | | min.pp 129 | 130 | Rules: 131 | 132 | * Loading is performed by mapping the fully qualified function name to a 4.x Ruby function path, and a puppet function path, then: 133 | * if the ruby path exist this 4.x Ruby function is loaded (and search stops) 134 | * if the puppet path exists this Puppet function is loaded (and search stops) 135 | * last, an attempt is made to load a 3.x Ruby function 136 | * Files containing an auto loaded function may only contain a single function (or an error is raised and evaluation stops). 137 | 138 | > Note: 139 | > 140 | > Function loading only searches a set of distinct paths based on the fully qualified name of 141 | > the function. 142 | > Contrast this with other kinds of automatic loading of classes and user defined resource types 143 | > where a search is made using a widening of the namespace, and finally reaching 144 | > a modules `manifests/init.pp`. 145 | > 146 | > **Specifically**: Autoloading a function from a module does not trigger loading of the module's 147 | > `manifests/init.pp` (nor is such initialization required to call a function from a module). 148 | > If an author of a module provides functions that require that the module's `manifests/init.pp` 149 | > is loaded, the function should include the module's class, or require that the caller first 150 | > includes the module's class). 151 | > 152 | > **Specifically**: a file `init.pp` under the `functions` directory of a module or the environment 153 | > does not have any special rules associated with it. 154 | > If that file exists it is supposed to contain a function named `::init`. 155 | > Contrast this with `manifests/init.pp` which represents the module it is in. There is no such 156 | > concept for functions. 157 | 158 | ### Defining functions in Manifests 159 | 160 | It is possible to define a Puppet Function in any manifest. Such functions will come into existence when the manifest in question is loaded for some reason other than calling the function (e.g. from 'manifests/site.pp' or when including a class). 161 | 162 | The following restrictions/rules/conventions apply on naming non-auto-loaded functions: 163 | 164 | * A function defined in a module **must** be qualified with the module's namespace 165 | * A function defined in an environment's main manifest **should** begin with the special namespace 'environment' 166 | * *except* when patching of a function is required - then the name may shadow other functions defined in the same environment, or the modules in this environment. Functions provided by the puppet runtime cannot be shadowed. A shadowed function cannot be called. 167 | 168 | Note that the term "environment's main manifest" means logic loaded from the command line (`apply -e`), the Puppet setting `code`, the code loaded from the setting `manifest` (e.g. the `manifests/site.pp` file, or a directory of manifests). 169 | 170 | The use cases for using functions defined in manifests are: 171 | 172 | * Defining several helper functions that are used locally in a class/user defined type. (Although not yet provided in the language, these functions would typically be made `private` to the module). 173 | * For patching: 174 | * To define a single word (non name-spaced) function (not recommended in general, useful for integrating code that needs such a function and where the original function's implementation is flawed/unwanted) 175 | * To override/shadow functions in modules that are flawed/unwanted. 176 | * To experiment during development 177 | 178 | > Note: In the current implementation of Puppet 4.3.x the naming restrictions on non-auto-loaded 179 | > functions are not enforced. They are expected to be enforced in some future release. 180 | 181 | [1]: parameter_scope.md 182 | -------------------------------------------------------------------------------- /language/resource_types.md: -------------------------------------------------------------------------------- 1 | # Puppet Resource Types 2 | 3 | This document describes puppet's Ruby DSL for defining resource types, aka [custom types](https://docs.puppet.com/guides/custom_types.html#types). Examples are taken from the `puppetlabs-aws` module and core puppet. 4 | 5 | The second part of the document describes issues that cause puppet resource types to leak across environments (lack of environment isolation). 6 | 7 | ## Puppet::Type.newtype method 8 | 9 | Defines a new puppet resource type and registers the type using the specified symbolic name. For example, to define a resource type named `ec2_instance`: 10 | 11 | ```ruby 12 | Puppet::Type.newtype(:ec2_instance) do 13 | # definition goes here 14 | end 15 | ``` 16 | 17 | The `newtype` method creates a new class `Puppet::Type::Ec2_instance`, which subclasses `Puppet::Type`. The block passed to the `newtype` method defines methods and variables on the `Puppet::Type::Ec2_instance` class. 18 | 19 | ## Puppet::Type.type method 20 | 21 | Defines a method for retrieving the class for a resource type, for example, when writing a provider for the `ec2_instance` type: 22 | 23 | ```ruby 24 | Puppet::Type.type(:ec2_instance).provide(:v2, ...) do 25 | # provider definition 26 | end 27 | ``` 28 | 29 | ## Puppet::Type::<name> methods 30 | 31 | Within the `Puppet::Type.newtype` block, the following DSL methods are often called, and are evaluated in the context of the `Puppet::Type::` class: 32 | 33 | ### doc string 34 | 35 | Defines the documentation string for the type, which is output when running `puppet describe `. The doc string is usually set via the `Puppet::Type` class variable, though there is also a `Puppet::Type.doc=` class method. 36 | 37 | ```ruby 38 | Puppet::Type.newtype(:ec2_instance) do 39 | @doc = 'A type representing an AWS VPC' 40 | end 41 | ``` 42 | 43 | ### namevar 44 | 45 | The type must specify a parameter to uniquely identify the resource. Conceptually this is referred to as the namevar. The type can specify multiple namevar parameters, which is referred to a composite namevar. For example, the identity of a `package` resource is based on the name of the package and provider, because you can have different types of packages with the same name, e.g. rpm and gem. 46 | 47 | Puppet will implicitly treat any parameter named `name` as a namevar. A parameter can explicitly specify that it is a namevar, described in the Parameter & Properties section below. 48 | 49 | ### title patterns 50 | 51 | Provides a way for setting attributes from the title. For example, the `file` resource sets its `path` namevar based on its `title`, stripping trailing slashes in the process. So if `title` is `/foo/bar//`, then `path` will be `/foo/bar`. 52 | 53 | ```ruby 54 | Puppet::Type.newtype(:file) do 55 | def self.title_patterns 56 | [ [ /^(.*?)\/*\Z/m, [ [ :path ] ] ] ] 57 | end 58 | end 59 | ``` 60 | 61 | A type must define a `title_patterns` method when using composite namevars so that puppet knows how to decompose the `title` into its constituent namevars, e.g. see [java-ks](https://github.com/puppetlabs/puppetlabs-java_ks/blob/abac95473a505080f60b4d0118c82d3568063da0/lib/puppet/type/java_ks.rb#L150-L176), [websphere](https://github.com/puppetlabs/puppetlabs-websphere_application_server/blob/cd2dbc59c43030db36a81f9c1ed155d5fbe4e85c/lib/puppet/type/websphere_sdk.rb#L11-L28). 62 | 63 | ### ensureable 64 | 65 | Creates an `ensure` property with acceptable values of `present` and `absent`, each of which invoke the provider's `create` and `destroy` methods, respectively: 66 | 67 | ```ruby 68 | Puppet::Type.newtype(:ec2_instance) do 69 | ensurable 70 | end 71 | ``` 72 | 73 | Ensurable can also take a block, and in that case, the type should define the allowed set of values. For example, the `package` type defines allowed values as: 74 | 75 | ```ruby 76 | Puppet::Type.newtype(:package) do 77 | ensurable do 78 | newvalue(:present) 79 | newvalue(:absent) 80 | newvalue(:purged) 81 | newvalue(:held) 82 | newvalue(:latest) 83 | newvalue(/./) 84 | end 85 | end 86 | ``` 87 | 88 | ### newproperty 89 | 90 | Defines a new property for a type. Puppet will ensure the resource's current state (as retrieved by the provider) matches the property's desired state (as expressed in the manifest). 91 | 92 | ```ruby 93 | Puppet::Type.newtype(:ec2_instance) do 94 | newproperty(:region) do 95 | end 96 | end 97 | ``` 98 | 99 | ### newparam 100 | 101 | Defines a new parameter for the type. Puppet does not enforce state for a parameter. Instead parameters specify additional information about how the provider should ensure the resource and its properties are in the correct state. 102 | 103 | ```ruby 104 | Puppet::Type.newtype(:ec2_instance) do 105 | newparam(:user_data) do 106 | end 107 | end 108 | ``` 109 | 110 | ### newmetaparam 111 | 112 | Defines a new metaparameter for a type. Puppet has several built-in metaparameters described below. It is uncommon for types to define new metaparams as these generally require puppet core changes to be useful, e.g. `noop`, `tag`, `schedule`, etc. 113 | 114 | ### validate 115 | 116 | Performs per-resource validation at catalog application time. This method is often used to validate related parameters and properties for a single resource, e.g. mutually exclusive properties or if one property requires another. One concrete example is the `file` resource requires the `source` parameter to be set if the `recurse` parameter is set to `remote`: 117 | 118 | ```ruby 119 | Puppet::Type.newtype(:file) do 120 | validate do 121 | self.fail "You cannot specify a remote recursion without a source" if !self[:source] && self[:recurse] == :remote 122 | end 123 | end 124 | ``` 125 | 126 | ### autorelations 127 | 128 | Allows a type to define automatic relationships (`before`, `subscribe`, `require`, `notify`) between each instance of the type and instances of a different type. For example, the `exec` type will autorequire any file resource whose `path` matches the exec `command`. Note the relationship will only be added to the catalog if puppet is managing both ends of the relationship. That way puppet will automatically create the file containing the command to execute, before trying to executing it, regardless of manifest order. 129 | 130 | ```ruby 131 | autorequire(:file) do 132 | reqs = [] 133 | self[:command].scan(file_regex) { |str| 134 | reqs << str 135 | } 136 | reqs 137 | end 138 | ``` 139 | 140 | Autorequires are by far the most common, though recently puppet added support for `autobefore`, `autonotify`, and `autosubscribe`. 141 | 142 | ### pre_run_check 143 | 144 | The agent will call the `pre_run_check` method for each `Puppet::Type` instance in the agent's catalog. This provides an opportunity for a resource to perform consistency checks/validation against other resources in the catalog. It differs from the `validate` method, since it is called later during catalog application, and can rely on the catalog having all generated resources. 145 | 146 | ### provider features 147 | 148 | Defines a feature for the type, which allows puppet to perform additional validation on the agent at catalog application time based on the selected provider. For example the `service` type defines an `enableable` feature: 149 | 150 | ```ruby 151 | Puppet::Type.newtype(:service) do 152 | feature :enableable 153 | end 154 | ``` 155 | 156 | And the type specifies that the `enable` property can only be managed if the selected provider supports the `enableable` feature. This validation check is performed at catalog application time, once the provider has been resolved on the agent. 157 | 158 | ```ruby 159 | Puppet::Type.newtype(:service) do 160 | newproperty(:enable, :required_features => :enableable) do 161 | end 162 | end 163 | ``` 164 | 165 | An array of features can also be specified, e.g. 166 | 167 | ```ruby 168 | newproperty(:enable, :required_features => [:green, :blue]) do 169 | end 170 | ``` 171 | 172 | A **provider** indicates it supports the feature using `has_feature`: 173 | 174 | ```ruby 175 | Puppet::Type.type(:service).provide :launchd, :parent => :base do 176 | has_feature :enableable 177 | end 178 | ``` 179 | 180 | A type can also restrict allowed parameter/property values based on provider features. For example, the `enable` property can only be set to `mask` if the provider is `maskable`: 181 | 182 | ```ruby 183 | Puppet::Type.newtype(:service) do 184 | feature :maskable 185 | 186 | newproperty(:enable, :required_features => :enableable) do 187 | ... 188 | newvalue(:mask, :event => :service_disabled, :required_features => :maskable) do 189 | provider.mask 190 | end 191 | end 192 | end 193 | ``` 194 | 195 | ### mixins 196 | 197 | Since puppet resource types are defined in ruby, you can mixin additional functionality. The AWS module uses this to create subclasses for different types of Route53 DNS records: 198 | 199 | ```ruby 200 | Puppet::Type.newtype(:route53_a_record) do 201 | extend PuppetX::Puppetlabs::Route53Record 202 | @doc = 'Type representing a Route53 DNS record.' 203 | create_properties_and_params() 204 | end 205 | ``` 206 | 207 | where the base module defines common parameters and properties for all `Route53Records`: 208 | 209 | ```ruby 210 | module PuppetX 211 | module Puppetlabs 212 | module Route53Record 213 | def create_properties_and_params 214 | ensurable 215 | newproperty(:zone) do 216 | end 217 | end 218 | end 219 | end 220 | end 221 | ``` 222 | 223 | ## Parameter & Property DSL Methods 224 | 225 | The following DSL methods are commonly used for individual parameters and properties within the body of `newparam` and `newproperty` respectively: 226 | 227 | ### desc 228 | 229 | Description of the property, output when running `puppet describe `: 230 | 231 | ```ruby 232 | Puppet::Type.newtype(:ec2_instance) do 233 | newproperty(:region) do 234 | desc 'The region in which to launch the instance.' 235 | end 236 | end 237 | ``` 238 | 239 | ### isnamevar 240 | 241 | Specifies that the parameter is the `namevar` (aka identity) for the resource: 242 | 243 | ```ruby 244 | Puppet::Type.newtype(:ec2_instance) do 245 | newparam(:name) do 246 | isnamevar 247 | end 248 | end 249 | ``` 250 | 251 | As mentioned earlier, if a type specifies a parameter named `:name`, it will automatically be the namevar, so the call to `isnamevar` is redundant, but is explicit. 252 | 253 | Alternatively, you can pass an option when calling newparam: 254 | 255 | ```ruby 256 | Puppet::Type.newtype(:ec2_instance) do 257 | newparam(:name, :namevar => true) do 258 | end 259 | end 260 | ``` 261 | 262 | Note that the namevar is necessarily a parameter, and not a property, since changing the name identifies a different resource as opposed to changing the name of an existing resource. 263 | 264 | Defining a parameter as the namevar also means it is required. Normally the namevar is automatically set to be the same as the title, unless the type overrides the `title_patterns` method, e.g. for composite namevars. 265 | 266 | ### validate 267 | 268 | Validates an individual parameter value at catalog application time. The value to validate is yielded to the block: 269 | 270 | ```ruby 271 | Puppet::Type.newtype(:ec2_instance) do 272 | newproperty(:region) do 273 | validate do |value| 274 | fail 'region should not contain spaces' if value =~ /\s/ 275 | end 276 | end 277 | end 278 | ``` 279 | 280 | Puppet parameter and properties can be multi-valued. For example, the `ec2_instance` can be given a list of `security_groups`, and each value in the list will be yielded to the `validate` method: 281 | 282 | ```ruby 283 | Puppet::Type.newtype(:ec2_instance) do 284 | newproperty(:security_groups, ...) do 285 | validate do |value| 286 | fail 'security_groups should be ...' if value !~ /something/ 287 | end 288 | end 289 | end 290 | ``` 291 | 292 | ### munge 293 | 294 | Normalizes the desired (aka `should`) property value at catalog application time. Puppet will compare the normalized value against the `current` value the provider returns to determine if the property is insync or not. There is also an `unmunge` method, less commonly used. For example, the `ec2_autoscalinggroup` defines a minimum number of instances in the group, where the value is munged from a string to an integer: 295 | 296 | ```ruby 297 | Puppet::Type.newtype(:ec2_autoscalinggroup) do 298 | newproperty(:min_size) do 299 | munge do |value| 300 | value.to_i 301 | end 302 | end 303 | end 304 | ``` 305 | 306 | Often times the logic for validation and munging is the same, e.g. try to convert a value into an integer. As a result, the `validate` logic is omitted, and validation is performed during `munge`. 307 | 308 | ### newvalue 309 | 310 | Defines an enumeration of values for a parameter or property. For example the `ec2_instance` type uses the plural form `newvalues` to define an enumeration of `instance_initiated_shutdown_behavior` values: 311 | 312 | ```ruby 313 | Puppet::Type.newtype(:ec2_instance) do 314 | newparam(:instance_initiated_shutdown_behavior) do 315 | desc 'Whether the instance stops or terminates when you initiate shutdown from the instance.' 316 | newvalues(:stop, :terminate) 317 | end 318 | end 319 | ``` 320 | 321 | The singular form `newvalue` defines one possible value and takes a block. At catalog application time, if puppet determines the property is not insync, it will call the block to "sync" that resource's property. It's common to define multiple `newvalue` blocks, where each value's block calls an appropriate provider method. 322 | 323 | ```ruby 324 | Puppet::Type.newtype(:ec2_instance) do 325 | newproperty(:ensure) do 326 | newvalue(:running) do 327 | provider.create unless provider.running? 328 | end 329 | newvalue(:stopped) do 330 | provider.stop unless provider.stopped? 331 | end 332 | end 333 | end 334 | ``` 335 | 336 | The `newvalue` and `newvalues` methods can also be passed a regex. Puppet will compare explicit symbols/strings first, and if there are no matches, compare regex's. For example, the `package` type defines: 337 | 338 | ```ruby 339 | Puppet::Type.newtype(:package) do 340 | newproperty(:ensure) do 341 | newvalue(:present) 342 | newvalue(:absent) 343 | ... 344 | newvalue(/./) 345 | end 346 | end 347 | ``` 348 | 349 | The last regex is used to match version strings, e.g `ensure => '1.2.3'` 350 | 351 | ### aliasvalue 352 | 353 | Aliases a value to be the same as an existing value. For example, the package type aliases `installed` to be the same as `present`, because it's more natural to declare that a package is `installed`: 354 | 355 | ``` 356 | Puppet::Type.newtype(:ec2_instance) do 357 | newproperty(:ensure) do 358 | newvalue(:present) 359 | aliasvalue(:installed, :present) 360 | end 361 | end 362 | ``` 363 | 364 | ### defaultto 365 | 366 | Defines the default value for a parameter. If a value is specified, then `defaultto` creates a `default` method with a block that always returns the specified value. 367 | 368 | ```ruby 369 | Puppet::Type.newtype(:ec2_instance) do 370 | newparam(:instance_initiated_shutdown_behavior) do 371 | newvalues(:stop, :terminate) 372 | defaultto :stop 373 | end 374 | end 375 | ``` 376 | 377 | If a block is specified, then the block is called to retrieve the default value at catalog application time. For example, the `filebucket` type defines a `server` parameter: 378 | 379 | ```ruby 380 | Puppet::Type.newtype(:filebucket) do 381 | newparam(:server) do 382 | desc "The server providing the remote filebucket service." 383 | defaultto { Puppet[:server] } 384 | end 385 | end 386 | ``` 387 | 388 | The default value of `Puppet[:server]` works for all agents, because the default value is resolved at catalog application time, not compilation time. 389 | 390 | ### is_to_s/should_to_s/change_to_s 391 | 392 | Overrides log messages for a property at catalog application time. Common examples are convert an id to a human readable name, sorting multi-valued attributes, redacting passwords, etc. 393 | 394 | ```ruby 395 | Puppet::Type.newtype(:user) do 396 | newproperty(:password) do 397 | def is_to_s(currentvalue) 398 | return '[old password hash redacted]' 399 | end 400 | end 401 | end 402 | ``` 403 | 404 | ## Parameter/Property Options 405 | 406 | The following options are passed to the `newparam` and `newproperty` methods to modify their behavior. These are largely hacks because puppet's predefined parameter/properties types are not well-specified or complete. For example, there isn't a `Puppet::Parameter::Integer` class. 407 | 408 | ### array_matching 409 | 410 | By default, if multiple desired (aka `should`) values are specified in a manifest, puppet will make sure the current (aka `is`) value matches at least one of the desired values. Alternatively, you can specify `array_matching => all`, and puppet will ensure that the array of current values match the desired values. 411 | 412 | By default, the `insync?` comparison is sensitive to order and duplicate values, so sometimes a type will override the `insync?` method. For example, security groups are compared set-wise, which ignores duplicates. 413 | 414 | ```ruby 415 | Puppet::Type.newtype(:ec2_instance) do 416 | newproperty(:security_groups, :array_matching => :all) do 417 | desc 'The security groups to associate the instance.' 418 | def insync?(is) 419 | is.to_set == should.to_set 420 | end 421 | end 422 | end 423 | ``` 424 | 425 | ### parent 426 | 427 | Specifies a parent class that the newly defined property/parameter should extend. For example, the `ec2_instance` type defines a `tags` property that inherits from the `PuppetX::Property::AwsTag` class: 428 | 429 | ```ruby 430 | require_relative '../../puppet_x/puppetlabs/property/tag.rb' 431 | 432 | Puppet::Type.newtype(:ec2_instance) do 433 | newproperty(:tags, :parent => PuppetX::Property::AwsTag) do 434 | desc 'The tags for the instance.' 435 | end 436 | end 437 | ``` 438 | 439 | The `PuppetX::Property::AwsTag` class is defined in helper code, so the type must require it using a relative path. The helper code defines logic for validating aws tags and how log messages are written: 440 | 441 | ```ruby 442 | module PuppetX 443 | module Property 444 | class AwsTag < Puppet::Property 445 | 446 | def format_tags(value) 447 | Hash[value.sort] 448 | end 449 | 450 | [:should_to_s, :is_to_s].each { |method| 451 | alias_method method, :format_tags 452 | } 453 | 454 | validate do |value| 455 | fail 'tags should be a Hash' unless value.is_a?(Hash) 456 | end 457 | end 458 | end 459 | end 460 | ``` 461 | 462 | ### boolean 463 | 464 | Meta-programs a predicate method for the parameter/property. For example, the file type defines a `force` parameter: 465 | 466 | ```ruby 467 | newparam(:force, :boolean => true, :parent => Puppet::Parameter::Boolean) do 468 | end 469 | ``` 470 | 471 | which defines a `Puppet::Type::File#force?` method. It's not clear who calls this method, but the pattern of specifying both `:boolean => true, :parent => Puppet::Parameter::Boolean` is copy/pasted everywhere. 472 | 473 | ```ruby 474 | irb(main):008:0> Puppet::Type.type(:file).new(:path => '/tmp/foo', :force => true).force? 475 | => true 476 | ``` 477 | 478 | # Environment Isolation Issues 479 | 480 | When the master compiles a catalog for the first time, it loads puppet resource types from that request's environment-specific modulepath. If the master then compiles a catalog using a different environment, it will use whatever type information was loaded from the previous environment. We refer to this as puppet types leaking across environments. 481 | 482 | In the future, we want to isolate types within an environment, so that you can deploy different module versions in different environments, e.g. dev, qa, prod, but still get consistent results in any given environment. 483 | 484 | This section describes issues with the way puppet resource types are defined and loaded, which prevent environment isolation. 485 | 486 | 1. The main issue is that the `Puppet::Type.newtype` method defines a class that is global in the ruby runtime. So you can't have two different versions of the type in the same ruby runtime. 487 | 488 | 1. Types often require helper code, like the `AwsTag` example, and that has the same issue as above. 489 | 490 | 1. Types use various tricks to require helper code in order to support when running on the master and when running `puppet apply`. For example: 491 | 492 | ```ruby 493 | require_relative '../../puppet_x/puppetlabs/property/tag.rb' 494 | ``` 495 | ```ruby 496 | require Pathname.new(__FILE__).dirname + '../' + 'puppet_x/puppetlabs/powershell_version' 497 | ``` 498 | ```ruby 499 | begin 500 | require "puppet_x/puppetlabs/registry" 501 | rescue LoadError => detail 502 | require Pathname.new(__FILE__).dirname + "../../" + "puppet_x/puppetlabs/registry" 503 | end 504 | ``` 505 | 506 | 1. The `Puppet::Type.newtype` method adds a `provider` parameter to the type and **loads all providers for the type**, even when the type is loaded on the master. A comment in the code says it is to [determine the default provider](https://github.com/puppetlabs/puppet/blob/4.4.1/lib/puppet/metatype/manager.rb#L115-L117), but that is no longer true. We only determine the default provider when the type is loaded on the agent, and we resolve provider confines and suitability. 507 | 508 | 1. The `title_patterns` method is called during **compilation** and causes attribute values to be set on the resource that is serialized in the catalog. For example given `file { '/foo/bar//': ensure => file }`, the catalog will contain: 509 | 510 | ```json 511 | { 512 | "type": "File", 513 | "title": "/foo/bar//", 514 | "parameters": { 515 | "path": "/foo/bar", 516 | "ensure": "file" 517 | } 518 | }, 519 | ``` 520 | 521 | There has been confusion over time about exactly what parts of a type are evaluated on the master vs agent. Part of the confusion is because the master will apply multiple settings catalogs for file/directory based-settings, e.g. `ssldir`. This ensures the files exist and have the correct permissions. However, it causes the master's ruby process to behave like an agent, so all of the validate, munge, etc methods for `Puppet::Type::File` are evaluated on the master. Also when the master is running as root, the same applies to the `Puppet::Type::User` and `Puppet::Type::Group` types so that the master can manage permissions for those file/directories. But generally speaking, the master does not call the validate, munge, etc methods. 522 | -------------------------------------------------------------------------------- /language/settings.md: -------------------------------------------------------------------------------- 1 | New Settings 2 | ============ 3 | 4 | A strict_variables setting has been added to Puppet. Setting this value true causes the future parser to raise errors when referencing unknown variables. It does not affect referencing variables explicitly set to undef. 5 | 6 | Its default is false. 7 | 8 | Note: If this setting is set true with the original parser implementation, an error will be raised because of an uncaught throw of :undefined_variable. 9 | -------------------------------------------------------------------------------- /language/templates.md: -------------------------------------------------------------------------------- 1 | # Embedded Puppet - EPP 2 | 3 | The template based text generation feature Embedded Puppet (EPP) is an integral part of the Puppet Programming Language that allows template text to contain free form text intermixed with logic expressed in the Puppet Programming Language for the purpose of producing repeating or conditional text content. 4 | 5 | EPP is to the Puppet Programming Language what ERB is to Ruby, and EPP supports the same template tags as ERB. While the two technologies are similar, they are not identical. 6 | 7 | EPP is used via two functions supplied with puppet (`epp` and `inline_epp`). This specification does not limit the set of functions that may provide different ways of invoking or using the EPP template features. 8 | 9 | ### EPP Functions 10 | 11 | EPP template text is processed by the two epp functions: 12 | 13 | * `epp(template_name, optional_args_hash)` - for evaluating an `.epp` file 14 | * `inline_epp(epp_text_string, optional_args_hash)` - for evaluating a string in EPP syntax 15 | 16 | When EPP-text is evaluated the result is a String. 17 | 18 | #### Visibility of Scoped Variables 19 | 20 | In order to encourage template reuse and foster good best practices regarding separation of 21 | concerns, the scoping rules of the EPP functions are as follows: 22 | 23 | * Both `inline_epp()` and `epp()` always provide access to the global (i.e. top/node) scope 24 | variables. 25 | * `inline_epp()` provides access to the local calling scope's variables unless an 26 | (optional) hash of variable name/value entries is given in which case these are used **instead** 27 | of having access to the local scope variables. 28 | * The `epp` function does not provide access to the calling scope's variables via short 29 | name references (unless the calling scope is also the top-scope). 30 | * If a template declares parameters that require a value to be given, these must be given in 31 | the name/value hash. 32 | 33 | In other words, an external (non-inline) template behaves as a function, and an inline 34 | template behaves as a lambda or as a function depending on if it has defined parameters, or is given variables in a hash. 35 | 36 | #### inline_epp function 37 | 38 | The `inline_epp` function is given a String containing the EPP source text. The function optionally takes a Hash of variable name to value mappings. 39 | 40 | #### epp function 41 | 42 | The `epp` function processes EPP source text in a file (the template file). 43 | The first argument to this function should be the file to to take the EPP source text 44 | from, given either as a absolute path, or as a module specific symbolic path on the form `/`. 45 | Such a symbolic reference will load `` from the given module's `templates` 46 | directory. (For example, the reference `apache/vhost.conf.epp` will load the 47 | file `/apache/templates/vhost.conf.epp`.) 48 | 49 | A module based reference enforces that the file must end with `'.epp'` while it allows that the reference is made without the `'.epp'` suffix which it then adds automatically. (For example, the reference `apache/vhost.conf`, is the same template reference as `apache/vhost.conf.epp`). 50 | 51 | For an absolute path, the path is used as given if the file exists. If the absolute path is referencing a non-existing file, the `'.epp'` suffix is appended in a second attempt to load the template. 52 | 53 | It is an error if the referenced template file does not exist. 54 | 55 | ### EPP Template Text 56 | 57 | An EPP lexical text processor (epp-lexer) starts in 'text mode', and delivers a token stream to the EPP parser. In addition to the token delivered for the Puppet Programming Language source text, 58 | the epp-lexer also delivers the following tokens: 59 | 60 | * `EPP_START` - a token that indicates the beginning of EPP text 61 | * `RENDER_STRING` - a verbatim sequence of characters 62 | 63 | Below is the EPP Specific part of the Puppet Language Grammar, where parts of the general Puppet Program grammar is shown to illustrate how EPP is wired into the language: 64 | 65 | program 66 | : statements # A non EPP Puppet Program 67 | | epp_expression # An EPP Puppet Program 68 | 69 | statements 70 | : ... # All statements in the Puppet Programming Language (structure not shown) 71 | | primary_expression 72 | 73 | primary_expression 74 | : ... # all expressions that are primary expression in the Puppet Language (not shown) 75 | | epp_render_expression 76 | 77 | epp_expression 78 | : EPP_START epp_parameters_list? statements? 79 | 80 | epp_parameters_list 81 | : '|' (parameters ','?)? '|' # parameters are the same as for a define 82 | 83 | epp_render_expression 84 | : RENDER_STRING 85 | | '<%=' expression ('%> | '-%>') 86 | | '<%=' '{' statements '}' ('%> | '-%>') 87 | 88 | 89 | The `EPP_START` token is automatically generated by the epp-lexer at the start of the EPP source text. It can be followed by an optional parameter list and optional statements if 90 | the first line of the EPP source text contains an opening `<%` that is not preceded by any text, 91 | or a `<%-` that is not preceded by any non whitespace text. In other words; it is not possible to introduce a template's parameters after any template text has been rendered. 92 | 93 | Example start of an EPP template with parameters: 94 | 95 | <%- | $x, $y, $z = 'this is a default value' | -%> 96 | 97 | All other occurrences of the EPP tokens `<%`,`<%-` or `<%=` will change the epp-lexer's mode from 98 | text-mode to puppet-mode. 99 | When in puppet-mode, the epp-lexer delivers tokens to the grammar as for any puppet program. 100 | When the epp-lexer sees an EPP-end token (`%>` or `-%>`) it switches back to text-mode and delivers any verbatim text as a `RENDER_STRING` token. The template text must end in text-mode. In other words; all EPP tags must be closed. There are also EPP-tags that do not change mode to allow rendering of EPP tags, and to introduce comments in text. 101 | 102 | When the resulting EPP program is evaluated, it is evaluated the same way as a non EPP Puppet Program is with the addition that at the end of the evaluation, the results of the RENDER_STRING expressions are concatenated into one string and returned as the result. 103 | 104 | > NOTE: 105 | > EPP tags do not nest! When in puppet-mode, it is not possible to again enter nested text-mode. 106 | > It is however allowed to call one of the EPP functions to 107 | > nest the result of evaluating another template. 108 | 109 | > NOTE: emitted whitespace text is significant 110 | > Care must be taken to not introduce emitted whitespace text between separate tags of puppet logic 111 | > intended to be one sequence of logic since that can lead to syntax errors. 112 | > As a rule of thumb; emitting whitespace (or any other 113 | > text) can only be made in places in the puppet logic where a function call is allowed. 114 | 115 | Rules: 116 | 117 | * An `epp_render_expression` produces a string result as a side effect that is concatenated at 118 | the end of the process. 119 | * Each invocation of an `epp_render_expression` has `undef` value. 120 | * For a `epp_render_expression` that contains a block expression (with multiple statements), the 121 | result of the block is rendered (the last expression). 122 | * Variables that are assigned in the template are local to the template. 123 | * A local variable introduced in the template may shadow a variable in an outer scope. 124 | * Variables in a template are immutable. 125 | 126 | This means that it is not possible to assign the result of verbatim text or to pass verbatim 127 | text as an argument in a function call. 128 | 129 | <% $a = %> text <%= $a %> 130 | 131 | This produces the result `" text "`. The first assignment to `$a` sets `$a` to `undef`, and the second will interpolate an empty string since `$a` is `undef`. 132 | 133 | <% notice ( "a", %> text <%, "b" ) %> 134 | 135 | This will notice `"a b"`, because the verbatim text produces `undef`. 136 | 137 | ### EPP Tags and their meaning 138 | 139 | | tag | description | 140 | | --- | --- | 141 | | `<%` | Switches to puppet mode. Whitespace to the left of the tag is rendered (no left trimming)| 142 | | `<%=` | Switches to puppet expression mode. (Left trimming is not possible) | 143 | | `<%%` | A literal `<%` is rendered, mode does not change | 144 | | `%%>` | A literal `%>` is rendered, mode does not change | 145 | | `<%-` | Switches to puppet mode. Whitespace text immediately preceding the tag, up to but not including a new line, is not rendered. | 146 | | `<%#` | A comment not included in the output (up to the next `%>`, or right trimming `-%>`). Continues in text mode after having skipped the comment. May optionally right trim by ending the tag with `-%>`. | 147 | | `<%#-` | Same as `<%#` but with trimming of all preceding whitespace on the same line.| 148 | | `%>` | Ends puppet mode | 149 | | `-%>` | Ends puppet mode and trims any generated trailing whitespace as well as whitespace immediately following the tag, up to, and including a newline. | 150 | 151 | 152 | Note: 153 | * Before Puppet 6.0.0 an EPP comment `<%#` always trimmed all preceding whitespace on the same line. From Puppet 6.0.0 it does not 154 | and `<%#-` should instead be used if this is wanted. 155 | * Left trimming EPP comment `<%#-` is available from Puppet 6.0.0. 156 | * If using a non EPP single line comment it consumes the entire line including text that looks like EPP tags. Such an EPP must be 157 | closed on a separate line. 158 | ``` 159 | <%-# this is a puppet language comment and it goes to new line %> and thus continues here 160 | -%> 161 | * If using non EPP multi line comment it consumes all text between `/*` and `*/` including any EPP tags. 162 | ``` 163 | <% /* this is puppet language comment, <% this is not EPP %>, and this is not a closing EPP %> */ %> 164 | ``` 165 | 166 | ### Examples 167 | 168 | $x = droid 169 | notice inline_epp(@(END)) 170 | This is the <%= $x %> you are looking for! 171 | | END 172 | 173 | Produces a notice of the string "This is the droid you are looking for!" 174 | 175 | $a = world 176 | notice inline_epp(@(END), {x => magic}) 177 | <%-| $x |-%> 178 | <% Integer[1,3].each |$count| { %> 179 | hello epp <%= $x %> <%= $a %> <%= $count %> 180 | <%- } %> 181 | |- END 182 | 183 | Produces the following output: 184 | 185 | Notice: Scope(Class[main]): 186 | hello epp magic world 1 187 | hello epp magic world 2 188 | hello epp magic world 3 189 | 190 | (In the example above `$a` resulted in `"world"` because all of the logic is in the global scope). 191 | 192 | -------------------------------------------------------------------------------- /models/LanguageSpec.ecore: -------------------------------------------------------------------------------- 1 | 2 | 6 | 7 |
8 | 9 | 10 | 11 |
12 | 13 | 15 | 16 | 17 | 19 | 20 | 21 | 22 | 23 |
24 | 25 | 26 | 27 | 28 | 29 | 30 |
31 | 32 | 34 | 35 | 36 | 37 |
38 | 39 | 41 | 42 | 43 | 44 | 45 | 46 | 48 | 49 |
50 | 51 | 52 | 54 | 56 | 57 |
58 | 59 | 60 | 61 | 62 | 63 |
64 | 65 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 |
80 | 81 | 82 | 83 | 84 | 85 |
86 | 87 | 88 | 89 | 90 |
91 | 92 | 93 | 94 | 95 | 96 | 97 |
98 | 99 | 100 | 101 | -------------------------------------------------------------------------------- /models/LanguageSpec.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/puppetlabs/puppet-specifications/2c2b639757fa41b28323374042c8480be7913805/models/LanguageSpec.png -------------------------------------------------------------------------------- /models/README.md: -------------------------------------------------------------------------------- 1 | The LanguageSpec Model 2 | === 3 | 4 | The intention of the language specification model is to be an implementation independent specification of the expectancies / requirements on a Puppet Program. 5 | 6 | Initially the model captures parser and validation expectancies. The intention is to also add Evaluation and Catalog related expectancies. 7 | 8 | The rationale for this model is that an implementation neutral assertable specification is required to ensure specification compliance. 9 | 10 | The model can be used in various ways: 11 | 12 | * Rspec or JUnit runners can translate and execute the model dynamically 13 | * Code can be generated in an implementation language - an implementation that runs the tests 14 | 15 | S-Expressions 16 | --- 17 | The AST model assertions are based on transformation of puppet AST to S-Expressions ([1]). Reference implementations of the AST model to S-expressions are / will be available in Java and Ruby. 18 | 19 | This ensures that the tree representations can be expressed without implementation concern (an implementation may not be based on the actual Puppet AST ecore model), and an implementor of a 20 | different model can then transform an implementation specific model into S-expressions in order 21 | to validate the result. 22 | 23 | [1]: http://en.wikipedia.org/wiki/S-expression 24 | 25 | Current Version of the Model 26 | --- 27 | The initial (and current) version of the model only covers expectancies of parse results 28 | and validation of parsed results. The intention is to also add assertions/expectancies for evaluation results and resulting catalog results. 29 | 30 | As an example. The model now is rich enough to expression that a statement like: 31 | 32 | $a = 1 + 1 33 | 34 | Should result in the S-expression: 35 | 36 | (= $a (+ 1 1)) 37 | 38 | (NOTE: Presently the Model to S-Expression has been a debugging tool, and this mapping needs 39 | to be formalized and reviewed. As an example, `$a` should probably be expressed as `(var a)`). 40 | 41 | The model can also describe expectations on SyntaxError and Diagnostics (validation results). 42 | 43 | Expectations can be combined (default is AND) with NOT, OR, and XOR (exclusive or). 44 | 45 | Semantics 46 | --- 47 | The execution of tests are driven by the expectancies. If a parse result is expected, then naturally 48 | the given source string must be parsed. If no diagnostics expectancies are given, then it is expected 49 | to be free from diagnostics. If diagnostics expectancies are present, the given diagnostics expectancies defines the full set of assertions applied to the produced diagnostics. 50 | 51 | When evaluation expectancies are added, the semantics are similar - if a catalog result is expected, 52 | naturally there is the need to build a catalog from the given source. A simpler evaluation expectancy (say 1 + 1 == 2) may only require a partial evaluation. 53 | 54 | (NOTE: It is the intention to add evaluation expectancies in a later revision). 55 | -------------------------------------------------------------------------------- /models/pn.md: -------------------------------------------------------------------------------- 1 | ## The Puppet Extended S-Expression Notation (PN) 2 | 3 | ### Objective 4 | 5 | The primary objective for this format is to provide a short but precise 6 | human readable format of the puppet language AST. This format is needed to write 7 | tests that verifies the parser's function. 8 | 9 | Another objective is to provide easy transport of the AST using JSON 10 | or similar data formats that are constrained to boolean, number, string, 11 | undef, array, and object data types. The PN conversion to `Data` was specifically 12 | designed with this objective in mind. 13 | 14 | ### Format 15 | A PN forms a directed acyclig graph of nodes. There are five types of nodes: 16 | 17 | * Literal: A boolean, integer, float, string, or undef 18 | * List: An ordered list of nodes 19 | * Map: An ordered map of string to node associations 20 | * Call: A named list of nodes. 21 | 22 | ### PN represented as Data 23 | 24 | A PN can represent itself as the Puppet `Data` type and will then use 25 | the following conversion rules: 26 | 27 | PN Node | Data type 28 | --------|---------- 29 | `Literal` | `Boolean`, `Integer`, `Float`, `String`, and `Undef` 30 | `List` | `Array[Data]` 31 | `Map` | `Struct['#', Array[Data]]` 32 | `Call` | `Struct['^', Tuple[String[1], Data, 1]]` 33 | 34 | The `Map` is converted into a single element `Hash` with using the key 35 | `'#'` and then an array with an even number of elements where an evenly 36 | positioned element is the key for the value at the next position. The 37 | reason for this is that JSON (and other formats) will not retain the 38 | order of a hash and in Puppet that order is significant. 39 | 40 | A `Call` is similar to a `Map` but uses the key `'^'`. The first element 41 | of the associated array is the name of the function and any subsequent 42 | elements are arguments to that function. 43 | 44 | ### PN represented as String 45 | 46 | The native textual representation of a PN is similar to Clojure. 47 | 48 | PN Node | Sample string representation 49 | --------|---------- 50 | `Literal boolean` | `true`, `false` 51 | `Literal integer` | `834`, `-123`, `0` 52 | `Literal float` | `32.28`, `-1.0`, `33.45e18` 53 | `Literal string` | `"plain"`, `"quote \\""`, `"tab \\t"`, `"return \\r"`, `"newline \\n"`, `"control \\u{14}"` 54 | `Literal undef` | `nil` 55 | `List` | `["a" "b" 32 true]` 56 | `Map` | `{:a 2 :b 3 :c true}` 57 | `Call` | `(myFunc 1 2 "b")` 58 | 59 | ### PN represented as JSON or YAML 60 | 61 | When representing PN as JSON or YAML, it must first be converted to `Data` as described in 62 | section [PN represented as Data](#pn-represented-as-data) above. 63 | 64 | Examples: 65 | 66 | PN class | PN | JSON 67 | -----|----|----- 68 | `List` | `[1, 2, "a"]` | `[1, 2, "a"]` 69 | `Map` | `{:a 2 :b 3 :c true}` | `{ "#": ["a", 2, "b", 3, "c", true] }` 70 | `Call` | `(myFunc 1 2 "b")` | `{ "^": [ "myFunc", 1, 2, "b" ] }` 71 | 72 | #### Puppet Expression transformed into PN and JSON 73 | 74 | The Abstract Syntax Tree (AST) that is the result of parsing a Puppet Expression can be transformed 75 | into a PN. 76 | 77 | Examples: 78 | 79 | Puppet Expression | PN | JSON 80 | -----|----|----- 81 | `1 + 2 * 3` | `(+ 1 (* 2 3))` | `{ "^": [ "+", 1, { "^": [ "*", 2, 3 ]} ]}` 82 | `a * (2 + 3)` | `(* (qn "a") (paren (+ 2 3)))` | `{ "^": [ "*", { "^": [ "paren", { "^": [ "+", 1, 2 ]} ]} ]}` 83 | `"hello ${var}"` | `(concat "hello " (str (var "var")))` | `{ "^": [ "concat", "hello ", { "^": [ "str", { "^": [ "var", "var" ]} ]} ]}` -------------------------------------------------------------------------------- /moving_modules.md: -------------------------------------------------------------------------------- 1 | Moving Non-Core Types & Providers 2 | ================= 3 | 4 | ## Index 5 | 6 | * [Goals](#goals) 7 | * [User Stories](#user-stories) 8 | * [Changes](#changes) 9 | * [Resource Type Taxonomy](#resource-type-taxonomy) 10 | * [Internal](#internal) 11 | * [Core](#core) 12 | * [External](#external) 13 | * [Dependencies](#dependencies) 14 | * [Packaging](#packaging) 15 | * [Technical Issues](#technical-issues) 16 | * [Module Names](#module-names) 17 | * [Builtin Types](#builtin-types) 18 | * [Environment Isolation](#environment-isolation) 19 | * [Module Dependencies](#module-dependencies) 20 | * [Module Precedence](#module-precedence) 21 | * [Agent Compatibility](#agent-compatibility) 22 | 23 | # Goals 24 | 25 | * Extract non-core types and providers from the puppet repo and move them to modules. Doing so will reduce the surface area of puppet, decrease puppet CI cycle times, while making the extracted types and providers more accessible to community members and increasing their maintainability. 26 | * Continue to bundle some modules with puppet-agent during packaging so that users have a batteries-included experience. 27 | * Decrease the amount of effort required to add new supported platforms. 28 | 29 | # User Stories 30 | 31 | ## Serverless Puppet 32 | 33 | As an admin, I want to run masterless puppet and immediately be able to manage basic resources on my system, so that I can more quickly get value from puppet. I do not want to search the forge in order to perform basic system tasks. The preinstalled modules should make sense for the local platform I'm running puppet on. On \*nix, I should be able to manage cron, and on Windows, powershell resources. 34 | 35 | ## Server-based Catalog Compilation 36 | 37 | As an admin, I should be able to compile catalogs for resources whose types are included in the locally installed puppet-agent package, so that I don't have to install modules I already have. 38 | 39 | ## Server-based Catalog Application 40 | 41 | As an admin, if catalog compilation succeeds, then I want assurance that the catalog will be applied consistently across all agent versions regardless of which preinstalled modules are on each agent. For example, if the master has a newer version of a type, then all agents in that environment should use the same version of the type (and its provider) at catalog application time. 42 | 43 | ## Module Updates 44 | 45 | As an admin, if there is a problem with a type/provider packaged with puppet-agent, I want to be able to install a newer version from the forge (for both serverless and server-based), so that I don't need to wait for Puppet to release a new puppet-agent build. 46 | 47 | ## Puppet-Agent Updates 48 | 49 | As an admin, when the puppet-agent package is updated, I want to be able to use the new versions of types and providers that come with the new package, even if that means overwriting and deleting older preinstalled versions. However, installing a new puppet-agent package should not clobber modules I've installed via puppet module tool, r10k, etc. 50 | 51 | ## Module Pinning 52 | 53 | As an admin, if I update the puppet-agent package, and it updates a preinstalled module, but the module introduces a regression, I want to easily install the older version of the module from the forge. I don't want to be forced to downgrade puppet-agent packages, because that process introduces more risk. 54 | 55 | ## Environment Isolation 56 | 57 | As an admin, I want to be able to deploy different versions of extracted modules in different environments without breaking environment isolation. 58 | 59 | ## Module Contributor 60 | 61 | As an experienced puppet user, when I fix a bug in a puppet-maintained provider I use, I want to contribute that fix back upstream with a minimum of friction, so that I don't have to carry the patch to future versions of the agent. 62 | 63 | ## Community Maintainer 64 | 65 | As a puppet community member with expertise in parts of the provider ecosystem, I want to manage the flow of fixes and contributions into the providers I care about, so that contributions and module releases can happen rapidly without requiring puppet core contributors to do a full agent release. 66 | 67 | # Changes 68 | 69 | ## puppet repo 70 | 71 | Add a new puppet setting `vendormoduledir` defaulting to: 72 | 73 | /opt/puppetlabs/puppet/vendor_modules # On *nix 74 | C:\Program Files\Puppet Labs\Puppet\puppet\vendor_modules # On Windows 75 | 76 | The autoloader will include the `vendormoduledir` in its search path with least 77 | precedence. 78 | 79 | It will be possible to set `vendormoduledir = ` so that the autoloader ignores vendored 80 | modules. For example, if you want to manage all module dependencies via Puppetfile. 81 | 82 | All of the modules in the [`External`](#external) section below will be removed from the 83 | puppet repo and extracted into modules on the forge. 84 | 85 | ## puppet-agent package 86 | 87 | Several of the external modules will be added to the puppet-agent package, e.g. 88 | `host`, and they will be installed into the `vendormoduledir`. 89 | 90 | ## puppet gem 91 | 92 | When installing puppet as a gem, external modules will not be present. But 93 | they can be installed using a Puppetfile or `puppet module install` commands 94 | such as in the provisioning step of a vagrant workflow. 95 | 96 | ## Forge 97 | 98 | We will be publishing extracted modules to the forge using the PDK and following 99 | modules best practices. 100 | 101 | # Resource Type Taxonomy 102 | 103 | ## Internal 104 | 105 | The following types are internal to Puppet and will remain as is: 106 | 107 | component 108 | schedule 109 | stage 110 | whit 111 | 112 | ## Core 113 | 114 | The following types will be left in Puppet for now. The `file`, `user`, and `group` types are needed to apply settings catalogs. The `filebucket`, `resources`, and `tidy` types know too much about puppet internals to be extracted. The `notify` type is used extensively in puppet rspec tests as it is the most basic (providerless) type. The `package` and `service` types have multiple providers for each type, which makes removal more difficult. We may extract them at a later time (TBD). 115 | 116 | exec 117 | file 118 | filebucket 119 | group 120 | notify 121 | package 122 | resources 123 | service 124 | tidy 125 | user 126 | 127 | ## External 128 | 129 | The following types and providers will be extracted from Puppet. A subset (details TBD below) will be added back to puppet-agent during packaging. 130 | 131 | Each top-level path below specifies the name of the module, e.g. `augeas`, and the files contained within each module. The modules will be installed in a new directory visible to Puppet's autoloader, so catalog compilation and application will just work without additional configuration. Puppet will prefer modules in the modulepath and pluginsync'ed lib directory over the packaged modules, so that newer versions of modules can fix bugs in packaged modules. 132 | 133 | Path Comments 134 | 135 | / (*nix) /opt/puppetlabs/puppet/vendor_modules/ 136 | (Windows) C:\Program Files\Puppet Labs\Puppet\puppet\vendor_modules 137 | 138 | augeas_core/ Depends on 'puppet/parameter/boolean' 139 | lib/puppet/feature/augeas.rb Extracted from lib/puppet/features/base.rb 140 | lib/puppet/type/augeas.rb 141 | lib/puppet/provider/augeas/augeas.rb 142 | 143 | cron_core/ Depends on 'puppet/provider/parsedfile', 144 | lib/puppet/type/cron.rb 'puppet/util/filetype' 145 | 146 | host_core/ Depends on 'puppet/property/ordered_list', 147 | lib/puppet/type/host.rb 'puppet/provider/parsedfile', 148 | lib/puppet/provider/host/parsed.rb 149 | 150 | k5login_core/ Depends on 'puppet/type/file/selcontext', 151 | lib/puppet/type/k5login.rb 'puppet/util/selinux' 152 | 153 | mailalias_core/ Depends on 'puppet/provider/parsedfile' 154 | lib/puppet/type/mailalias.rb 155 | lib/puppet/provider/mailalias/aliases.rb 156 | 157 | maillist_core/ 158 | lib/puppet/type/maillist.rb 159 | lib/puppet/provider/maillist/maillist.rb 160 | 161 | macdslocal_core/ Depends on 'puppet/provider/nameservice/directory_service', 162 | lib/puppet/type/ 'puppet/util/plist' 163 | computer.rb 164 | macauthorization.rb 165 | mcx.rb 166 | lib/puppet/provider/ 167 | computer.rb 168 | macauthorization.rb 169 | mcxcontent.rb 170 | 171 | mount_core/ Depends on 'puppet/property/boolean', 172 | lib/puppet/type/mount.rb 'puppet/provider/parsedfile' 173 | lib/puppet/provider/mount.rb 174 | lib/puppet/provider/mount/parsed.rb 175 | 176 | nagios_core/ Depends on 'puppet/provider/parsedfile' 177 | lib/puppet/external/nagios.rb 178 | lib/puppet/external/nagios/ 179 | base.rb 180 | grammer.py 181 | makefile 182 | parser.rb 183 | lib/puppet/type/ 184 | nagios_*.rb 185 | lib/puppet/provider/naginator.rb 186 | lib/puppet/util/nagios_maker.rb 187 | 188 | network_device_core/ 189 | lib/puppet/feature/telnet.rb 190 | lib/puppet/type/ 191 | router.rb 192 | interface.rb 193 | vlan.rb 194 | lib/puppet/provider/ 195 | cisco.rb 196 | interface/cisco.rb 197 | vlan/cisco.rb 198 | lib/puppet/util/ 199 | network_device.rb 200 | network_device/* 201 | 202 | scheduled_task_core/ Depends on 'puppet/util/windows' 203 | lib/puppet/type/scheduled_task.rb 204 | lib/puppet/provider/scheduled_task/win32_taskscheduler.rb 205 | lib/puppet/util/windows/taskscheduler.rb 206 | 207 | selinux_core/ Depends on 'puppet/type/file/selcontext', 208 | lib/puppet/feature/selinux.rb 'puppet/util/selinux' 209 | lib/puppet/type/ 210 | selboolean.rb 211 | selmodule.rb 212 | lib/puppet/provider/ 213 | selmodule/seboolean.rb 214 | selmodule/semodule.rb 215 | 216 | sshkeys_core/ Depends on 'puppet/provider/parsed' 217 | lib/puppet/type/ 218 | sshkey.rb 219 | ssh_authorized_key.rb 220 | lib/puppet/provider/ 221 | sshkey/parsed.rb 222 | ssh_authorized_key/parsed.rb 223 | 224 | yumrepo_core/ Depends on 'puppet/util/filetype' 225 | lib/puppet/type/yumrepo.rb 226 | lib/puppet/provider/yumrepo/inifile.rb 227 | lib/puppet/util/inifile.rb 228 | 229 | zfs_core/ 230 | lib/puppet/type/zfs.rb 231 | lib/puppet/type/zpool.rb 232 | lib/puppet/provider/zfs/zfs.rb 233 | lib/puppet/provider/zpool/zpool.rb 234 | 235 | zone_core/ Depends on 'puppet/property/list' 236 | lib/puppet/type/zone.rb 237 | lib/puppet/provider/zone/zone.rb 238 | 239 | ## Dependencies 240 | 241 | The following classes are public API used by the above modules. 242 | 243 | Puppet::Error 244 | Puppet::FileSystem 245 | Puppet::Parameter 246 | Puppet::Property 247 | Puppet::Resource 248 | Puppet::Settings 249 | Puppet::Type 250 | Puppet::Util 251 | 252 | The following classes are specific to a few different types and providers, which makes extracting them difficult: 253 | 254 | Puppet::Provider::NameService::DirectoryService Used by mac user and group providers 255 | Puppet::Type::File::SelContext Used by :file, :k5login and :sel* 256 | Puppet::Util::FileType Used by :cron, :yumrepo 257 | Puppet::Util::PList Used by mac types 258 | Puppet::Util::SELinux Used by :file, :k5login and :sel* 259 | Puppet::Util::Windows Used by various windows types 260 | 261 | # Packaging 262 | 263 | This section lists which modules will be vendored in puppet-agent: 264 | 265 | augeas 266 | cron 267 | host 268 | mount 269 | scheduled_task 270 | selinux 271 | sshkeys 272 | yumrepo 273 | zfs 274 | zone 275 | 276 | The following internal types/providers will remain in puppet: 277 | 278 | group 279 | package 280 | service 281 | user 282 | 283 | # Technical Issues 284 | 285 | ## Module Names 286 | 287 | The puppet ecosystem contains modules that have the same name as some of the 288 | core types we want to extract: `augeas`, `cron`, `mount`, `selinux`. However, 289 | there can only be one version of a module installed per-environment (unless you 290 | use tricks like adding multiple directories to the `modulepath`). To avoid 291 | naming collisions, I'm proposing we append `_core` to all of the module names 292 | that we extract, e.g. `augeas_core`. This only changes the name of the module, 293 | not the names of the types contained within. Also it doesn't eliminate a 294 | collision, just makes it less likely. 295 | 296 | ## Builtin Types 297 | 298 | The loaders register builtin types based on 299 | `Puppet::Pops::Loader::StaticLoader#BUILTIN_TYPE_NAMES`. This is an 300 | optimization since we know the types exist in puppet and we don't need to scan 301 | the filesystem based on the per-environment modulepath. Any type removed from 302 | puppet, should be removed from `BUILTIN_TYPE_NAMES`. This will have an impact 303 | on compiler performance where the timeout is not unlimited. 304 | 305 | ## Environment Isolation 306 | 307 | All of the types extracted from puppet and removed from `BUILTIN_TYPE_NAMES` 308 | above, will be subject to [environment isolation](https://puppet.com/docs/puppet/5.4/environment_isolation.html#environment-isolation) 309 | issues. Users that install newer versions of modules containing types that are 310 | also vendored, should use `puppet generate types` to ensure types in one 311 | environment don't affect other environments. This isn't really a new concern, 312 | it's just that users haven't had to `puppet generate types` for builtin types 313 | before. 314 | 315 | ## Module Dependencies 316 | 317 | The autoloader does not rely on module metadata to load types. So any module 318 | that relies on a type, should just work in Puppet 6 provided the extracted 319 | module is vendored into puppet-agent (eg `augeas`), or the user installs the module 320 | (eg `nagios`). 321 | 322 | ## Module Precedence 323 | 324 | ### Catalog Compilation 325 | 326 | It should be possible to install a new version of a module and have that take 327 | precedence over the `vendor_modules` directory during compilation (or `puppet 328 | apply`). That will require a change to the autoloader to search for types. 329 | 330 | ### Pluginsync 331 | 332 | It is difficult to make changes to builtin types, because they are not 333 | pluginsync'ed from the master to agents. As a result, you can have situations 334 | where compilation succeeds, but catalog application on an older puppet agent 335 | fails, because the old agent doesn't understand the newer `property`. For 336 | example, we had that problem when we added the `virtual` property to the 337 | `package` type. 338 | 339 | To eliminate this problem we could pluginsync vendored modules. However, it 340 | would cause agents to pluginsync types and providers they often already have, 341 | and it means the vendored modules need to be compatible with all agents that 342 | connect to the master. 343 | 344 | We've decided to make vendored modules visible to the autoloader, but not part 345 | of the modulepath. That way they are available during compilation, but won't be 346 | pluginsync'ed. 347 | 348 | ## Agent Compatibility 349 | 350 | Pre-6.0 agents always prefer pluginsynced types over its builtin and vendored 351 | versions. One exception is when running the agent via bundler due to the way 352 | bundler manages the ruby LOAD_PATH. 353 | 354 | If a new property/parameter is added to a type on the master and the manifest 355 | attempts to use that property/parameter, then old agents will not be able to 356 | apply the catalog. Therefore, it's important that users only deploy modules 357 | that are compatible with the oldest agents in the fleet, but this proposal 358 | doesn't change that. 359 | -------------------------------------------------------------------------------- /tasks/README.md: -------------------------------------------------------------------------------- 1 | # Puppet Task Spec (version 1, revision 4) 2 | 3 | The Puppet Task ecosystem allows users to execute actions on target systems. A **Task** is the smallest component of this system capturing a single action that can be taken on a target and is - in its simplest form - an executable file. It is packaged and distributed as Puppet modules which are made available to the task runner. 4 | 5 | ## Design goals 6 | 7 | - The barrier to entry for writing simple tasks should be as low as possible. Drop a script in a folder and it's runnable. 8 | - Start with a small functional core and add features as needed. 9 | 10 | ## Terminology 11 | 12 | **Task**: A task is a file and an optional metadata file that will be executed on a target. 13 | 14 | **task runner**: The application which executes tasks on target systems. This is the interface for controlling and running tasks, and is assumed to provide some sort of programmatic interface (API). In initial releases this is Orchestrator (executing via PXP) and [Bolt] (via ssh/winrm). In the future new task runners may be added. For example the puppet agent may run tasks from resources. 15 | 16 | **UI**: User interface, such as CLI tools ([Bolt], `puppet task`) or the Puppet Enterprise Console. 17 | 18 | ## Puppet Task Spec Versioning 19 | 20 | The task spec has a version and a revision. Changes that break existing tasks increment the version of the task spec. Revisions function as a minor version and are incremented for new features. This spec indicates in parentheticals the revision number for new features, for example "Running remote tasks (rev 4)." 21 | 22 | Older versions of the task runner may not support tasks that rely on newer features of the task spec. The task spec version is not intended to capture this now. Authors should use the `puppet_task_version` field in the module's metadata to document such incompatibilities for users. 23 | 24 | Task Runners should document which version and revision of the task spec they support. 25 | 26 | ### Revision history 27 | 28 | These are the revisions to this version of the task spec. 29 | 30 | **Note**: Bolt generally uses the latest version and revision of the task spec. Puppet Enterprise versions use a specific version and revision of the task spec, indicated in the PE documentation. 31 | 32 | | Revision | Changes | 33 | |----------|---------------------------| 34 | | 1 | | 35 | | 2 | Cross-platform tasks. | 36 | | 3 | Multiple files per task. | 37 | | 4 | Hide private tasks. | 38 | | 5 | Default parameter values. | 39 | | 6 | Sensitive task output | 40 | 41 | ## Task packaging and file 42 | 43 | Tasks are packaged and distributed in the `/tasks` directory of a Puppet module. Any file in the top level of that directory with a valid name is considered a task. In addition to tasks, task authors may create a metadata file for the task with the same name and a `.json` extension. 44 | 45 | ### Task name and filename 46 | 47 | A task consists of an optional metadata file and one or more implementation files. Files with the `.json` extensions are metadata files, and any other file extension (including files with no extension) is an implementation file. Implementation files do not need the executable bit set. 48 | 49 | An implementation file `.` without a corresponding metadata file `.json` is a task. 50 | 51 | A metadata file `.json` also identifies a task. If the metadata specifies an `implementations` array, then the files listed in that array are the implementations for the task. Otherwise, there must be one corresponding implementation file `.`. 52 | 53 | Task names have the same restriction as puppet type names and must match the regular expression `\A[a-z][a-z0-9\_]*\z`. The extensions `.md` and `.conf` are forbidden. Only files at the top level of the `tasks` directory matching the task name regular expression are used; all other files are ignored. 54 | 55 | The canonical name of a task is `::`. The `init` task is treated specially, and may be referred to by the shorthand ``. 56 | 57 | ### Task metadata 58 | 59 | Task metadata is stored in `/tasks` dir in the module in `.json` with `UTF-8` encoding. 60 | 61 | All fields should have reasonable defaults so that writing metadata is optional. Metadata defaults should err towards security. 62 | 63 | The preferred style of the keys should be `snake_case`. 64 | 65 | #### Options 66 | 67 | **description**: A description of what the task does.(rev 1) 68 | 69 | **puppet_task_version**: The version of this spec used.(rev 1) 70 | 71 | **supports_noop**: Default `false`. This task respects the `_noop` metaparam. If this is not set the task runner will refuse to run the task in noop.(rev 1) 72 | 73 | **input_method**: What input method to use to pass parameters to the task. Default varies, see [Input Methods](#input-methods).(rev 1) 74 | 75 | **parameters**: The parameters or input the task accepts listed with a [Puppet data type](../language/types_values_variables.md) string and optional description. Top level params names have the same restrictions as Puppet class param names and must match `\A[a-z][a-z0-9_]*\z`. Parameters may be `sensitive` in which case the task runner should hide their values in UI where possible.(rev 1) 76 | 77 | **files**: A list of file resources to be made available to the task executable on the target specified as file paths. Files must be saved in module directories that Puppet makes available via mount points: `files`, `lib`, `tasks`. File specifications ending with `/` will require the entire directory. File separator must be `/`. (rev 3) 78 | 79 | **implementations**: A list of implementation objects. An implementation object describes resources and feature requirements that must be available on a target for specified resources to be utilized. The available features are defined by the task runner; task runners should define at least the `shell`, `powershell` and `puppet-agent` features.(rev 2) 80 | 81 | **private**: A boolean to specify whether user facing tools should hide a task by default. This is useful if a task has a machine oriented interface or is intended to be used only in the context of one plan. Default is false.(rev 3) 82 | 83 | **extensions**: A hash of extensions to the task spec used by a Specific Task Runner. Each key at the top level should be the name of the Task Runner the extension is used by. Task Runners should not read extensions outside of their own namespace.(rev 3) 84 | 85 | **remote**: Default `false`. All implementation of this task operate on a remote target using the `_target` metaparam.(rev 4) 86 | 87 | **identifiers**: An arbitrary map of key-value pairs without inherent, semantic meaning to a task runner. Intended to provide identification and / or categorization of tasks that can be consumed by custom or third-party tooling. 88 | 89 | #### Example Task Metadata 90 | 91 | ```json 92 | { 93 | "description": "Description of what the task does", 94 | "parameters": { 95 | "param1" : { 96 | "description": "Description of what param1 does with option1 or option2", 97 | "type": "Enum[option1, option2]" 98 | }, 99 | "param2": { 100 | "description": "Description of what param2 does with an array of non-empty strings", 101 | "type": "Array[String[0]]", 102 | "sensitive": true 103 | }, 104 | "param3": { 105 | "description": "Description of optional param3", 106 | "type": "Optional[Integer]" 107 | }, 108 | "param4": { 109 | "description": "Description of param4 which has a default value", 110 | "type": "String", 111 | "default": "hello" 112 | } 113 | }, 114 | "implementations" : [ 115 | {"name": "foo_sh.sh", "requirements": ["shell"], "input_method": "environment"}, 116 | {"name": "foo_ps1.ps1", "requirements": ["powershell"], "files": ["my_util/files/util/win_tools.ps1"]} 117 | ], 118 | "files": ["my_module/lib/puppet_x/helpful_code.rb", "kitchensink/files/task_helpers/"], 119 | "identifiers": { 120 | "tags": ["tag1", "Tag 2", "Three"], 121 | "id": "id1", 122 | "is_special": true, 123 | "nested_hash": { 124 | "also": "works" 125 | } 126 | } 127 | } 128 | ``` 129 | 130 | A JSON schema of metadata accepted by task runners is included in [task.json](task.json). The Puppet Forge hosts a mirror at https://forgeapi.puppet.com/schemas/task.json. 131 | 132 | ## Task parameters 133 | 134 | The parameters property is used to validate the parameters to a task and generate UIs. The property names in this object are the parameter names and map to a parameters options object. 135 | 136 | If the parameters property is missing or `null` any parameter values with be accepted. Authors should not write tasks without specifying parameters. 137 | 138 | If the parameters property is empty no parameters will be accepted. 139 | 140 | Any parameter that does not specify a type will accept any value (default type is [Any](../language/types_values_variables.md#any) or [Data](../language/types_values_variables.md#data)). 141 | 142 | If a parameter type accepts `null` the task runner will accept either a `null` value or the absence of the property. Task authors must accept missing properties for nullable parameters. Task authors must not differentiate between absent properties and properties with `null` values. 143 | 144 | ### Options 145 | 146 | **type**: A Puppet Type string describing the type of value the parameter accepts. Default is `Any`.(rev 1) 147 | 148 | **description**: A string description of the parameter.(rev 1) 149 | 150 | **sensitive**: A Boolean value to identify data as sensitive. Values are masked when they appear in logs and API responses.(rev 1) 151 | 152 | **default**: A value to use if no value is passed for the parameter. This value must conform to the `type` specified for the parameter.(rev 5) 153 | 154 | ## Task implementations 155 | 156 | The implementations property is used to describe different implementations for differet targets. The value is an array of implementation options objects. For each target the task is run on the Task Runner must compare the required features of the target to it's list of available features on that target and execute the first implementation where all requirements are statisfied. 157 | 158 | ### Options 159 | 160 | **name**: The file name of the task file that contains the implementation of the task. This must be at the top level of the tasks directory of the module. In order to remain compatible with runners implementing revision 1, task names should be unique. For example, consider the task `foo` with implementations in bash and powershell. Instead of naming the executables `foo.sh` and `foo.ps1` build unique names by including extension information in the base filename: `foo_sh.sh` `foo_ps1.ps1`.(rev 2) 161 | 162 | **requirements**: A list of features the target is required to support for the implementation to be used. `shell`, `powershell` and `puppet-agent` features are added by the ssh, winrm, and pcp transports respectively.(rev 2) 163 | 164 | **input_method**: The input method to use for this implementation of the task. Default empty `[]` which will make this implementation suitable for all targets.(rev 3) 165 | 166 | **files**: files required by this implementation. These will be concatenated with the files array from the top level of the tasks, metadata.(rev 3) 167 | 168 | **remote**: This implementation is remote. Set remote on specific implementations if the task supports both normal and remote execution.(rev 4) 169 | 170 | ### Metaparameters 171 | 172 | In addition to the tasks parameters, the task runner may inject metaparameters prefixed by `_`. 173 | 174 | **_noop**: Used to implement logic in tasks to support cases where task should not perform certain actions.(rev 1) 175 | 176 | **_task**: Allow multiple task implementations to access the same executable file. The `_task` metaparameter provides the executable the task name to allow task specific logic to be implemented.(rev 2) 177 | 178 | **_installdir**: Tasks with `files` specified in the metadata will be passed the `_installdir` metaparameter to provide the file path to the expected resources.(rev 3) 179 | 180 | **_target**: Connection information for connecting to the real target when the task is running on a proxy. 181 | 182 | #### Metaparameter Examples 183 | 184 | When the task runner runs a task with `files` metadata it copies the specified files into a temporary directory on the target. The directory structure of the specified file resources will be preserved such that paths specified with the `files` metadata option will be available to tasks prefixed with `_installdir`. 185 | 186 | The task executable itself will be located within the `_installdir` at its normal module location, such as `_installdir/mymodule/tasks/init`. This allows writing tasks that can be tested locally from source by requiring dependencies by relative path. 187 | 188 | ##### Python Example 189 | ###### Metadata 190 | ```json 191 | { 192 | "files": ["multi_task/files/py_helper.py"] 193 | } 194 | ``` 195 | ###### File Resource 196 | `multi_task/files/py_helper.py` 197 | ```python 198 | def useful_python(): 199 | return dict(helper="python") 200 | ``` 201 | ###### Task 202 | ```python 203 | #!/usr/bin/env python 204 | import sys 205 | import os 206 | import json 207 | 208 | params = json.load(sys.stdin) 209 | sys.path.append(os.path.join(params['_installdir'], 'multi_task', 'files')) 210 | # Alternatively use relative path 211 | # sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'multi_task', 'files')) 212 | import py_helper 213 | 214 | print(json.dumps(py_helper.useful_python())) 215 | ``` 216 | ###### Output 217 | ``` 218 | Started on localhost... 219 | Finished on localhost: 220 | { 221 | "helper": "python" 222 | } 223 | Successful on 1 node: localhost 224 | Ran on 1 node in 0.12 seconds 225 | ``` 226 | ##### Ruby Example 227 | ###### Metadata 228 | ```json 229 | { 230 | "files": ["multi_task/files/rb_helper.rb"] 231 | } 232 | ``` 233 | ###### File Resource 234 | `multi_task/files/rb_helper.rb` 235 | ```ruby 236 | def useful_ruby 237 | { helper: "ruby" } 238 | end 239 | ``` 240 | ###### Task 241 | ```ruby 242 | #!/usr/bin/env ruby 243 | require 'json' 244 | 245 | params = JSON.parse(STDIN.read) 246 | require_relative File.join(params['_installdir'], 'multi_task', 'files', 'rb_helper.rb') 247 | # Alternatively use relative path 248 | # require_relative File.join(__dir__, '..', '..', 'multi_task', 'files', 'rb_helper.rb') 249 | 250 | puts useful_ruby.to_json 251 | ``` 252 | ###### Output 253 | ``` 254 | Started on localhost... 255 | Finished on localhost: 256 | { 257 | "helper": "ruby" 258 | } 259 | Successful on 1 node: localhost 260 | Ran on 1 node in 0.12 seconds 261 | ``` 262 | ## Task execution 263 | 264 | If the task has multiple implementation files, the `implementations` field of the metadata is used to determine which implementation is suitable for the target. Each implementation can specify `requirements`, which is an array of the required "features" to use that implementation. 265 | 266 | If the task has a single implementation file and doesn't use the `implementations` field, that implementation will be used on every target. 267 | 268 | The task implementation is copied to the target and then executed on the target by the task runner. 269 | 270 | If `files` metadata has been provided those executable resources will be copied to the target and made available to the task via the `_installdir` metaparameter. 271 | 272 | No command-line arguments are passed to the task when it is executed. 273 | 274 | The task file in the module does not need execute permissions set. 275 | 276 | The location of the task file on the target varies based on the transport used by the task runner. Task authors have no control over this path. 277 | 278 | The operating environment the task is executed in (such as environment variables set and user privilege) is also controlled by the task runner. Task authors should document any requirements they have. 279 | 280 | Task runners will handle execution on Microsoft Windows specially based on the following mapping for file extensions: 281 | 282 | - **.ps1**: powershell 283 | - **.rb**: puppet agent ruby if available 284 | - **.pp**: puppet apply 285 | 286 | Future releases of the task runner may expand this or add configuration options to the task runner around interpreter choice. 287 | 288 | ## Task Input 289 | 290 | Parameters are passed according to the specified input method. If no value is specified for a parameter and the parameter has a `default` defined in the task metadata, the task runner will use the `default` value in place of the parameter. 291 | 292 | There are a three input methods available to tasks: [stdin](#stdin), [environment](#environment-variables), and [powershell](#powershell). By default tasks without a `.ps1` extension are passed parameters with both the `stdin` and `environment` input methods. Tasks with a `.ps1` extension are passed parameters with the `powershell` input method by default. 293 | 294 | In the future we may support other formats and methods for passing params to the task. 295 | 296 | ### Stdin 297 | 298 | The parameters are passed to the task in a JSON object on `stdin`. 299 | 300 | ### Environment Variables 301 | 302 | Params are set as environment variables before the task is executed. 303 | 304 | Param names will be prefixed with `PT_` to create the environment variables in the form `PT_` (i.e. the value of the `foobar` param is set as `PT_foobar`). 305 | 306 | String parameters will not be quoted. Numerical parameters and structured objects will use their JSON representations. 307 | 308 | #### Example 309 | 310 | ##### JSON 311 | ```json 312 | { 313 | "a": 1, 314 | "b": "a string", 315 | "c": [1, 2, "3"], 316 | "d": {"x": {"y": [0]}} 317 | } 318 | ``` 319 | 320 | ##### Environment Variables 321 | ```bash 322 | PT_a=1 323 | PT_b=a string 324 | PT_c=[1, 2, "3"] 325 | PT_d={"x": {"y": [0]}} 326 | ``` 327 | 328 | ### Powershell 329 | 330 | The `powershell` input method will pass each param as a named argument to the powershell script. This is the default input method for task files ending with a `.ps1` extension. 331 | 332 | ## Output handling 333 | 334 | ### Stdout 335 | 336 | Task output is consumed on stdout and made available through the task runner API. 337 | 338 | If the output cannot be parsed as a JSON object (i.e. `{...}`), one is created by the task runner with the captured stdout stored in the `_output` key: i.e. `{"_output": "the string of output"}`. Otherwise the parsed JSON object is used. 339 | 340 | All '_' prefixed keys are reserved and should only be used as described below: 341 | - **_output**: A text summary of the job result. If the job does not return a JSON object the contents of stdout will be put in this key.(rev 1) 342 | - **_error**: Tasks can set this when they fail and the UI can more gracefully display messages from it.(rev 1) 343 | - **_sensitive**: The value of this key should be treated as sensitive by the task runner.(rev 6) 344 | 345 | ### Stderr 346 | 347 | Stderr may be captured by the task runner. 348 | 349 | In the initial release it may be temporarily stored on the target but not exposed through the task runner API. This behavior may change in future iterations of the task runner. 350 | 351 | ### Exitcode 352 | 353 | The exitcode of the task is captured and exposed by the task runner. 354 | 355 | An exitcode of 0 is considered success by the task runner unless the task result contains an `_error` key. Any non-zero exit code or a result containing an `_error` is considered a failure. 356 | 357 | If a task returns a non-zero exit code it may return a response on stdout with the `_error` object of the following form: 358 | ```json 359 | { 360 | "kind": "mytask/myerror", 361 | "msg": "task failed because", 362 | "details": { "key": "value" } 363 | } 364 | ``` 365 | 366 | otherwise a default `_error` will be generated by the task runner: 367 | ```json 368 | { 369 | "kind": "puppetlabs.tasks/task-error", 370 | "msg": "The task errored with a code 12", 371 | "details": { "exitcode": 12 } 372 | } 373 | ``` 374 | 375 | ## Errors 376 | 377 | Errors should be generated in the task_runner when the task could not be successfully executed or the results of the task could not be captured. 378 | 379 | Some examples: 380 | - **task_file_error**: The task could not be copied onto the target. 381 | - **unexecutable_task**: The task execution failed. 382 | - **unparsable_output**: The task output could not be parsed with any of the specified formatters. 383 | - **output_encoding_error**: The task output was not utf-8. 384 | 385 | The JSON schema for errors is included in [error.json](error.json). 386 | 387 | [Bolt]: https://github.com/puppetlabs/bolt 388 | 389 | ## Executing Remote Tasks(rev 4) 390 | 391 | Tasks may be written to execute on a proxy target and interact remotely with 392 | the specified target. This is useful when the target has a limited shell 393 | environment or only exposes an API. 394 | 395 | - The task runner must accept a hash of connection information for remote 396 | targets. 397 | - The task runner must add the `_target` metaparam containing a hash of 398 | connection information before executing the task on the proxy target. 399 | - The task runner should refuse to execute task implementations that do not have 400 | `remote` set on remote targets. 401 | - The task runner should refuse to execute task implementations that do have 402 | `remote` set on normal targets. 403 | - The task runner may have a default proxy target for remote targets. 404 | -------------------------------------------------------------------------------- /tasks/error.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "http://json-schema.org/draft-06/schema#", 3 | "description": "The format for errors in puppet", 4 | "type": "object", 5 | "properties" : { 6 | "kind": { 7 | "description": "the type of error", 8 | "type": "string", 9 | "pattern": "keyword/regex" 10 | }, 11 | "msg": { 12 | "description": "A human readable error message", 13 | "type": "string" 14 | }, 15 | "details": { 16 | "description": "kind specific data about the error.", 17 | "type": "object" 18 | } 19 | }, 20 | "requiredProperties": ["type", "msg"] 21 | } 22 | -------------------------------------------------------------------------------- /tasks/task.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "http://json-schema.org/draft-06/schema#", 3 | "title": "Puppet Task Metadata", 4 | "description": "The metadata format for Puppet Tasks", 5 | "type": "object", 6 | "properties": { 7 | "description": { 8 | "type": "string", 9 | "description": "A longer description(one paragraph) of how to use the task" 10 | }, 11 | "puppet_task_version": { 12 | "type": "integer", 13 | "description": "The version of this spec used", 14 | "default": 1 15 | }, 16 | "supports_noop": { 17 | "type": "boolean", 18 | "default": false, 19 | "description": "This task respects the '_noop' metaparam. If this false or absent the task runner will refuse to run this task if noop is specified." 20 | }, 21 | "remote": { 22 | "type": "boolean", 23 | "default": false, 24 | "description": "This task is capable of operating on a remote target using connection information in the '_target' metaparam." 25 | }, 26 | "input_method": { 27 | "type": "string", 28 | "enum": ["stdin", "environment", "both", "powershell"], 29 | "description": "What input method should be used to pass params to the task" 30 | }, 31 | "parameters": { 32 | "$ref": "#/definitions/paramsObject", 33 | "description": "An object mapping valid parameter names to corresponding json-schemas" 34 | }, 35 | "implementations": { 36 | "type": "array", 37 | "items": { 38 | "type": "object", 39 | "required": ["name"], 40 | "properties": { 41 | "name": { 42 | "type": "string", 43 | "description": "Name of task executable file" 44 | }, 45 | "requirements": { 46 | "type": "array", 47 | "additionalItems": { 48 | "type": "string" 49 | }, 50 | "description": "Features required on target to execute task" 51 | }, 52 | "files": { 53 | "type": "array", 54 | "additionalItems": { 55 | "type": "string" 56 | }, 57 | "description": "File resources required by task" 58 | } 59 | } 60 | }, 61 | "description": "Rules for selecting implementation resources based on features available on target" 62 | }, 63 | "files": { 64 | "type": "array", 65 | "items": { 66 | "type": "string" 67 | }, 68 | "description": "Path to file resources saved in valid module directory to be provided to task" 69 | }, 70 | "private": { 71 | "type": "boolean", 72 | "description": "Should this task appear by default in UI lists of tasks" 73 | }, 74 | "extensions": { 75 | "type": "object", 76 | "description": "Task Runner specific metadata extensions", 77 | "items": { 78 | "type": "object" 79 | } 80 | }, 81 | "identifiers": { 82 | "type": "object", 83 | "description": "An arbitrary map of key-value pairs without inherent, semantic meaning to a Task Runner" 84 | } 85 | }, 86 | "definitions": { 87 | "parameterName": { 88 | "description": "Valid names for parameter keys", 89 | "type": "string", 90 | "pattern": "^[a-z][a-z0-9_]*$" 91 | }, 92 | "paramsObject": { 93 | "type": "object", 94 | "description": "An object with restricted keys and enumData support", 95 | "propertyNames": {"$ref": "#/definitions/parameterName"}, 96 | "additionalProperties": { 97 | "type": "object", 98 | "description": "Extend Normal JSON schema to require an object and describe 'enumData' to map enum values to descriptions", 99 | "properties": { 100 | "description": { 101 | "description": "A description of the parameter", 102 | "type": "string" 103 | }, 104 | "type": { 105 | "description": "A puppet type string that describes a data type that will match the parameter value", 106 | "type": "string" 107 | }, 108 | "sensitive": { 109 | "description": "Whether the task runner should treat the parameter value as sensitive", 110 | "type": "boolean", 111 | "default": false 112 | }, 113 | "default": { 114 | "description": "The default value to pass to the task implementation if the parameter isn't provided" 115 | } 116 | } 117 | } 118 | } 119 | } 120 | } 121 | -------------------------------------------------------------------------------- /tests/.gitignore: -------------------------------------------------------------------------------- 1 | .bundle 2 | Gemfile.lock 3 | -------------------------------------------------------------------------------- /tests/Gemfile: -------------------------------------------------------------------------------- 1 | source 'https://rubygems.org' 2 | 3 | # Requires Ruby 2.4+ 4 | group :development do 5 | gem 'json_schemer', require: false 6 | gem 'rspec', require: false 7 | end 8 | -------------------------------------------------------------------------------- /tests/spec/spec_helper.rb: -------------------------------------------------------------------------------- 1 | PROJECT_ROOT = File.join(__dir__, '..', '..') 2 | -------------------------------------------------------------------------------- /tests/spec/tasks/schema_spec.rb: -------------------------------------------------------------------------------- 1 | require 'spec_helper' 2 | require 'json_schemer' 3 | require 'net/http' 4 | 5 | def read_schema_document(path) 6 | JSON.parse(File.open(path, 'rb') { |f| f.read }) 7 | end 8 | 9 | def get_uri(url) 10 | redirects = 0 11 | begin 12 | response = Net::HTTP.get_response(URI.parse(url)) 13 | url = response['location'] if response.is_a?(Net::HTTPRedirection) 14 | redirects += 1 15 | end while response.is_a?(Net::HTTPRedirection) && redirects < 10 16 | 17 | raise "Reached maximum redirects for #{url}" unless redirects < 10 18 | raise "Response from #{url} is not HTTP Status 200: #{response.message}" unless response.is_a?(Net::HTTPOK) 19 | response.body 20 | end 21 | 22 | def hyper_schema_for_schema(schema) 23 | hyper_schema_uri = schema['$schema'] 24 | hyper_schema_content = JSON.parse(get_uri(hyper_schema_uri)) 25 | JSONSchemer.schema(hyper_schema_content) 26 | end 27 | 28 | [ 29 | { name: 'Tasks', file: 'task.json'}, 30 | { name: 'Errors', file: 'error.json'} 31 | ].each do |testcase| 32 | context "#{testcase[:name]} schema" do 33 | let(:schema_path) { File.join(PROJECT_ROOT, 'tasks', testcase[:file]) } 34 | let(:schema_json) { read_schema_document(schema_path) } 35 | let(:hyper_schema) { hyper_schema_for_schema(schema_json) } 36 | 37 | it 'is a valid JSON document' do 38 | expect { schema_json }.to_not raise_error 39 | end 40 | 41 | it 'is a valid JSON Schema document' do 42 | expect(hyper_schema.valid?(schema_json)).to be(true) 43 | end 44 | end 45 | end 46 | 47 | --------------------------------------------------------------------------------