├── doc ├── managing_specs.rst ├── network_protocol.rst ├── json_serializer.rst ├── evolve_spec.rst ├── using_generator.rst ├── lang_ref.rst └── generator_ref.rst └── README.rst /doc/managing_specs.rst: -------------------------------------------------------------------------------- 1 | ************** 2 | Managing Specs 3 | ************** 4 | 5 | Here we cover several strategies for dealing with larger projects composed of 6 | many specs, routes, and user-defined types. 7 | 8 | Using Namespaces 9 | ================ 10 | 11 | Whenever possible, group related routes and their associated data types into 12 | namespaces. This organizes your API into logical groups. 13 | 14 | Code generators should translate your namespaces into logical groups in the 15 | target language. For example, the Python generator creates a separate Python 16 | module for each namespace. 17 | 18 | Splitting a Namespace Across Files 19 | ================================== 20 | 21 | If a spec is growing large and unwieldy with thousands of lines, it might make 22 | sense to split the namespace across multiple spec files. 23 | 24 | All you need to do is create multiple ``.babel`` files with the same 25 | `Namespace `_ definition. Code generators cannot 26 | distinguish between spec files--only namespaces--so no code will be affected. 27 | 28 | When splitting a namespace across multiple spec files, each file should use the 29 | namespace name as a prefix of its filename. 30 | 31 | The ``babelapi`` command-line interface makes it easy to specify multiple 32 | specs:: 33 | 34 | $ babelapi python spec1.babel spec2.babel spec3.babel output/ 35 | $ babelapi python *.babel output/ 36 | 37 | Using Header Files 38 | ================== 39 | 40 | If multiple spec files depend on the same user-defined type or alias, then you 41 | should move the common definition to a header file. 42 | 43 | See how to use `Include `_. 44 | 45 | Separating Public and Private Routes 46 | ==================================== 47 | 48 | Most services have a set of public routes that they publish for external 49 | developers, as well as a set of private routes that are intended to only be 50 | used internally by first-party apps. 51 | 52 | To use Babel for both public and private routes, we recommend splitting specs 53 | into ``{namespace}_public.babel`` and ``{namespace}_private.babel`` files. You 54 | may choose to simply use ``{namespace}.babel`` to represent the public spec. 55 | 56 | When publishing your API to third-party developers, you can simply include the 57 | public spec file. When generating code for internal use, you can use both the 58 | public and private spec files. 59 | -------------------------------------------------------------------------------- /doc/network_protocol.rst: -------------------------------------------------------------------------------- 1 | **************** 2 | Network Protocol 3 | **************** 4 | 5 | While Babel does not standardize the network protocol used to send messages 6 | between hosts, we do anticipate that HTTP will be a popular medium. This 7 | document describes how serialized Babel objects can be transmitted using 8 | HTTP based on experiences from implementing the Dropbox v2 API. 9 | 10 | Remote Procedure Calls 11 | ====================== 12 | 13 | The majority of the routes in the Dropbox API are expected to be used by 14 | non-browser clients. Clients tend to be one of the several SDKs that Dropbox 15 | maintains for popular programming languages. 16 | 17 | Because of this, it was convenient to use the body of the HTTP request and 18 | response to store JSON-serialized Babel objects. Both requests and responses 19 | set the ``Content-Type`` header to ``application/json``. 20 | 21 | Based on the ``get_account`` route defined in the running example in the `Language 22 | Reference `_, the following is a sample exchange between a client 23 | and server. 24 | 25 | An example request:: 26 | 27 | POST /users/get_account 28 | Content-Type: application/json 29 | 30 | { 31 | "account_id": "id-48sa2f0" 32 | } 33 | 34 | An example sucessful response:: 35 | 36 | 200 OK 37 | Content-Type: application/json 38 | 39 | { 40 | "account_id": "id-48sa2f0", 41 | "email": "alex@example.org", 42 | "name": "Alexander the Great" 43 | } 44 | 45 | An example error response:: 46 | 47 | 409 Conflict 48 | Content-Type: application/json 49 | 50 | { 51 | "reason": "no_account", 52 | } 53 | 54 | HTTP Endpoint 55 | The ``get_account`` route defined in the ``users`` namespace was mapped to 56 | the url ``/users/get_account``. No host information is shown intentionally. 57 | 58 | HTTP Status Code 59 | A response indicates that its body contains an object conforming to a 60 | route's error data type by returning an HTTP 409 error code. It was 61 | important to use a "protocol layer" feature to indicate if an error had 62 | occurred. 63 | 64 | Authentication 65 | There is no authentication scheme shown in the above example. One 66 | possibility is to use an ``Authorization`` header. 67 | 68 | Browser Compatibility 69 | ===================== 70 | 71 | [TODO] Encoding JSON in headers for upload and download-style routes. 72 | -------------------------------------------------------------------------------- /doc/json_serializer.rst: -------------------------------------------------------------------------------- 1 | *************** 2 | JSON Serializer 3 | *************** 4 | 5 | Code generators include a JSON serializer which will convert a target 6 | language's representation of Babel data types into JSON. This document explores 7 | how Babel data types, regardless of language, are mapped to JSON. 8 | 9 | Primitive Types 10 | =============== 11 | 12 | ========================== ==================================================== 13 | Babel Primitive JSON Representation 14 | ========================== ==================================================== 15 | Boolean Boolean 16 | Bytes String: Base64-encoded 17 | Float{32,64} Number 18 | Int{32,64}, UInt{32,64} Number 19 | List Array 20 | String String 21 | Timestamp String: Encoded using strftime() based on the 22 | Timestamp's format argument. 23 | Void Null 24 | ========================== ==================================================== 25 | 26 | Struct 27 | ====== 28 | 29 | A struct is represented as a JSON object. Each specified field has a key in the 30 | object. For example:: 31 | 32 | struct Coordinate 33 | x Int64 34 | y Int64 35 | 36 | 37 | converts to:: 38 | 39 | { 40 | "x": 1, 41 | "y": 2 42 | } 43 | 44 | If an optional (has a default or is nullable) field is not specified, the key 45 | should be omitted. For example, given the following spec:: 46 | 47 | struct SurveyAnswer 48 | age Int64 49 | name String = "John Doe" 50 | address String? 51 | 52 | If ``name`` and ``address`` are unset and ``age`` is 28, then the struct 53 | serializes to:: 54 | 55 | { 56 | "age": 28 57 | } 58 | 59 | Setting ``name`` or ``address`` to ``null`` is not a valid serialization; 60 | deserializers will raise an error. 61 | 62 | An explicit ``null`` is allowed for fields with nullable types. While it's 63 | less compact, this makes serialization easier in some languages. The previous 64 | example could therefore be represented as:: 65 | 66 | { 67 | "age": 28, 68 | "address": null 69 | } 70 | 71 | Enumerated Subtypes 72 | ------------------- 73 | 74 | A struct that enumerates subtypes serializes similarly to a regular struct, 75 | but includes a ``.tag`` key to distinguish the type. Here's an example to 76 | demonstrate:: 77 | 78 | struct A 79 | union* 80 | b B 81 | c C 82 | w Int64 83 | 84 | struct B extends A 85 | x Int64 86 | 87 | struct C extends A 88 | y Int64 89 | 90 | Serializing ``A`` when it contains a struct ``B`` (with values of ``1`` for 91 | each field) appears as:: 92 | 93 | { 94 | ".tag": "b", 95 | "w": 1, 96 | "x": 1 97 | } 98 | 99 | If the recipient receives a tag it cannot match to a type, it should fallback 100 | to the parent type if it's specified as a catch-all. 101 | 102 | For example:: 103 | 104 | { 105 | ".tag": "d", 106 | "w": 1, 107 | "z": 1 108 | } 109 | 110 | Because ``d`` is unknown, the recipient checks that struct ``A`` is a 111 | catch-all. Since it is, it deserializes the message to an ``A`` object. 112 | 113 | Union 114 | ===== 115 | 116 | Let's use the following example to illustrate how a union is serialized:: 117 | 118 | union U 119 | singularity 120 | number Int64 121 | coord Coordinate? 122 | 123 | struct Coordinate 124 | x Int64 125 | y Int64 126 | 127 | The serialization of ``U`` with tag ``singularity`` is:: 128 | 129 | { 130 | ".tag": "singularity" 131 | } 132 | 133 | The ``.tag`` key makes it easy for a recipient to immediately determine the 134 | selected union member. 135 | 136 | For a union member of primitive type (``number`` in the example), the 137 | serialization is as follows:: 138 | 139 | { 140 | ".tag": "number", 141 | "number": 42 142 | } 143 | 144 | Note that ``number`` is used as the value for ``.tag`` and as a key to hold 145 | the value. This same pattern is used for union members with types that are 146 | other unions or structs with enumerated subtypes. 147 | 148 | Union members that are structs that do no enumerate subtypes (``coord`` in the 149 | example) serialize as the struct with the addition of a ``.tag`` key. For 150 | example, the serialization of ``Coordinate`` is:: 151 | 152 | { 153 | "x": 1, 154 | "y": 2 155 | } 156 | 157 | The serialization of ``U`` with tag ``coord`` is:: 158 | 159 | { 160 | ".tag": "coord", 161 | "x": 1, 162 | "y": 2 163 | } 164 | 165 | Nullable 166 | ^^^^^^^^ 167 | 168 | Note that ``coord`` references a nullable type. If it's unset, then the 169 | serialization only includes the tag:: 170 | 171 | { 172 | ".tag": "coord" 173 | } 174 | 175 | You may notice that if ``Coordinate`` was defined to have no fields, it is 176 | impossible to differentiate between an unset value and a value of coordinate. 177 | In these cases, we prescribe that the deserializer should return a null 178 | or unset value. 179 | 180 | Compact Form 181 | ^^^^^^^^^^^^ 182 | 183 | Deserializers should support an additional representation of void union 184 | members: the tag itself as a string. For example, tag ``singularity`` could 185 | be serialized as simply:: 186 | 187 | "singularity" 188 | 189 | This is convenient for humans manually entering the argument, allowing them to 190 | avoid typing an extra layer of JSON object nesting. 191 | -------------------------------------------------------------------------------- /doc/evolve_spec.rst: -------------------------------------------------------------------------------- 1 | *************** 2 | Evolving a Spec 3 | *************** 4 | 5 | APIs are constantly evolving. In designing Babel, we sought to codify what 6 | changes are backwards incompatible, and added facilities to make maintaining 7 | compatibility easier. 8 | 9 | Background 10 | ========== 11 | 12 | The root of the problem is that when an API interface evolves, it does not 13 | evolve simultaneously for all communicating parties. This happens for a couple 14 | reasons: 15 | 16 | 1. The owner of the API does not have control over 3rd parties that have 17 | integrated their software at some point in the evolution of the 18 | interface. These integrations may never be updated making 19 | compatibility-awareness critical. 20 | 2. Even the owner of the API may roll out evolutions to their fleet of 21 | servers in stages, meaning that clusters of servers will have different 22 | understandings of the interface for windows of time. 23 | 24 | Sender-Recipient 25 | ================ 26 | 27 | When discussing interface compatibility, it's best to think in terms of a 28 | message sender and a message receiver, either of which may have a newer 29 | interface. 30 | 31 | If the sender has a newer version, we want to make sure that the recipient 32 | still understands the message it receives, and ignores the parts that it 33 | doesn't. 34 | 35 | If the recipient has a newer version, we want to make sure that it knows what 36 | to do when the sender's message is missing data. 37 | 38 | Backwards Incompatible Changes 39 | ------------------------------ 40 | 41 | * Removing a struct field 42 | * An old receiver may have application-layer dependencies on the field, 43 | which will cease to exist. 44 | * Changing the type of a struct field. 45 | * An old receiver may have application-layer dependencies on the field 46 | type. In statically typed languages deserialization will fail. 47 | * Adding a new tag to a union without a `catch-all tag `_. 48 | * We expect receivers to exhaustively handle all tags. If a new tag is 49 | returned, the receiver's handler code will be insufficient. 50 | * Changing the type of a tag with a non-Void type. 51 | * Similar to the above, if a tag changes, the old receiver's 52 | handler code will break. 53 | * Changing any of the types of a route description to an incompatible one. 54 | * When changing an arg, result, or error data type for a route, you 55 | should think about it as applying a series of operations to convert 56 | the old data type to the new one. 57 | * The change in data type is backwards incompatible if any operation 58 | is backwards incompatible. 59 | 60 | Backwards Compatible Changes 61 | ---------------------------- 62 | 63 | * Adding a new route. 64 | * Changing the name of a stuct, union, or alias. 65 | * Adding a field to a struct that is optional or has a default. 66 | * If the receiver is newer, it will either set the field to the 67 | default, or mark the field as unset, which is acceptable since the 68 | field is optional. 69 | * If the sender is newer, it will send the new field. The receiver will 70 | simply ignore the field that it does not understand. 71 | * Change the type of a tag from Void to anything else. 72 | * The older receiver will ignore information associated with the new 73 | data type and continue to present a tag with no value to the 74 | application. 75 | * Adding another tag to a union that has a catch-all specified. 76 | * The older receiver will not understand the incoming tag, and will 77 | simply set the union to its catch-all tag. The application-layer will 78 | handle this new tag through the same code path that handles the 79 | catch-all tag. 80 | 81 | Planning for Backwards Compatibility 82 | ==================================== 83 | 84 | * When defining a union that you're likely to add tags to in the 85 | future, add a `catch-all tag `_. 86 | 87 | Leader-Clients 88 | ============== 89 | 90 | We focused on senders and recipients because they illustrate the general case 91 | where any two parties may have different versions of a spec. However, your 92 | system may have an added layer of predictability where some party ("leader") is 93 | guaranteed to have the same or newer version of the spec than its "clients." 94 | 95 | It's important to note that a leader-clients relationship can be transient and 96 | opportunistic--it's important to decide if this relationship exists in your 97 | setup. 98 | 99 | The leader-client relationship comes up often: 100 | 101 | 1. A service that has an API is the "leader" for first-party or third-party 102 | clients in the wild that are accessing the service's data. The server 103 | will get a spec update, and clients will have to update their code to 104 | take advantage of the new spec. 105 | 2. Within a fleet of servers, you may have two clusters that communicate 106 | with each other, one of which receives scheduled updates before the 107 | other. 108 | 109 | A known leader can be stricter with what it receives from clients: 110 | 111 | * When the leader is acting as a recipient, it should reject any struct 112 | fields it is unaware of. It knows that the unknown fields are not because 113 | the client, acting as a sender, has a newer version of the spec. 114 | 115 | * Since a client acting as a recipient may have an older spec, it 116 | should retain the behavior of ignoring unknown fields. 117 | 118 | * If the leader is acting as a recipient, it should reject all unknown 119 | tags even if the union specifies a catch-all. 120 | * If the leader is acting as a recipient, any tag with type Void should 121 | have no associated value in the serialized message since it's not 122 | possible for a client to have converted the data type to something else. 123 | 124 | [TODO] There are more nuanced backwards compatible changes such as: A tag 125 | can be removed if the union is only sent from the server to a client. Will this 126 | level of detail just lead to errors in practice? 127 | 128 | Route Versioning 129 | ================ 130 | 131 | Building language facilities to ease route versioning has yet to be addressed. 132 | Right now, if you know you are making a backwards incompatible change, we 133 | suggest the following verbose approach: 134 | 135 | * Create a new route. 136 | * We recommend simply attaching a numerical suffix to prevent a name 137 | collision. For example, ``/get_account`` becomes ``/get_account2``. 138 | * Copy the definition of any data types that are changing in a backwards 139 | incompatible way. For example, if the response data type is undergoing an 140 | incompatible change, duplicate the response data type, give it a new 141 | name, and make the necessary modifications. 142 | * Be sure to update the route signature to reference the new data type. 143 | 144 | Future Work 145 | =========== 146 | 147 | Building in a lint checker into the ``babelapi`` command-line interface that 148 | warns if a spec change is backwards incompatible based on the revision history. 149 | This assumes that the spec file is in a version-tracking system like git or hg. 150 | -------------------------------------------------------------------------------- /doc/using_generator.rst: -------------------------------------------------------------------------------- 1 | ******************** 2 | Using Generated Code 3 | ******************** 4 | 5 | Using a generator, you can convert the structs, unions, and routes in your spec 6 | into objects in your programming language of choice. 7 | 8 | Currently, the only generator included with Babel is for `Python 9 | <#python-guide>`_. We intend to create generators for an assortment of 10 | languages including: 11 | 12 | * Java 13 | * Javascript 14 | * PHP 15 | * Ruby 16 | * Swift 17 | 18 | If you're looking to make your own generator, see 19 | `Writing a Generator `_. 20 | 21 | Compile with the CLI 22 | ==================== 23 | 24 | Compiling a spec and generating code is done using the ``babelapi`` 25 | command-line interface (CLI):: 26 | 27 | $ babelapi -h 28 | usage: babelapi [-h] [-v] generator spec [spec ...] output 29 | 30 | BabelAPI 31 | 32 | positional arguments: 33 | generator Specify the path to a generator. It must have a .babelg.py 34 | extension. 35 | spec Path to API specifications. Each must have a .babel 36 | extension. 37 | output The folder to save generated files to. 38 | 39 | optional arguments: 40 | -h, --help show this help message and exit 41 | -v, --verbose Print debugging statements. 42 | 43 | We'll compile the ``users.babel`` example from the 44 | `Language Reference `_. The first argument is the path to the 45 | Python generator which can be found in the ``babelapi`` folder:: 46 | 47 | $ babelapi generator/python/python.babelg.py users.babel . 48 | INFO:babelapi.idl:Parsing spec users.babel 49 | INFO:babelapi.compiler:Found generator at ... 50 | INFO:babelapi.compiler:Running generator ... 51 | INFO:bablesdk.generator.PythonGenerator:Copying babel_validators.py to output folder 52 | INFO:bablesdk.generator.PythonGenerator:Copying babel_serializers.py to output folder 53 | INFO:bablesdk.generator.PythonGenerator:Generating ./users.py 54 | 55 | The first argument selects the included Python code generator. ``users.babel`` 56 | is the spec file to compile. If we had another spec file, we could list it 57 | right after ``users.babel``. The ``.`` says to save the output of the code 58 | generator to the current directory. 59 | 60 | Python Guide 61 | ============ 62 | 63 | This section explains how to use the pre-packaged Python generator and work 64 | with the Python classes that have been generated from a spec. 65 | 66 | From the above section, you can generate Python using:: 67 | 68 | $ babelapi generator/python/python.babelg.py users.babel . 69 | 70 | This runs the Python generator on the ``users.babel`` spec. Its output 71 | target is ``.``, which is the current directory. A Python module is created for 72 | each declared namespace, so in this case only ``users.py`` is created. 73 | 74 | Two additional modules are copied into the target directory. The first, 75 | ``babel_validators.py``, contains classes for validating Python values against 76 | their expected Babel types. You will not need to explicitly import this module, 77 | but the auto-generated Python modules depend on it. The second, 78 | ``babel_serializers.py``, contains a ``json_encode()`` and ``json_decode()`` 79 | function. You will need to import this module to serialize your objects. 80 | 81 | In the following sections, we'll interact with the classes generated in 82 | ``users.py``. For simplicity, we'll assume we've opened a Python interpreter 83 | with the following shell command:: 84 | 85 | $ python -i users.py 86 | 87 | For non-test projects, we recommend that you set a generation target within a 88 | Python package, and use Python's import facility. 89 | 90 | Primitive Types 91 | --------------- 92 | 93 | The following table shows the mapping between a Babel `primitive type 94 | `_ and its corresponding type in Python. 95 | 96 | ========================== ============== ===================================== 97 | Primitive Python 2.x / 3 Notes 98 | ========================== ============== ===================================== 99 | Bytes bytes 100 | Boolean bool 101 | Float{32,64} float long type within range is converted. 102 | Int{32,64}, UInt{32,64} long 103 | List list 104 | String unicode / str str type is converted to unicode. 105 | Timestamp datetime 106 | ========================== ============== ===================================== 107 | 108 | Struct 109 | ------ 110 | 111 | For each struct in your spec, you will see a corresponding Python class of the 112 | same name. 113 | 114 | In our example, ``BasicAccount``, ``Account``, and ``GetAccountReq`` are all 115 | Python classes. They have an attribute (getter/setter/deleter property) for 116 | each field defined in the spec. You can instantiate these classes and specify 117 | field values either in the constructor or by assigning to an attribute:: 118 | 119 | >>> b = BasicAccount(account_id='id-48sa2f0') 120 | >>> b.email = 'alex@example.org' 121 | 122 | If you assign a value that fails validation, an exception is raised:: 123 | 124 | >>> b.email = 10 125 | Traceback (most recent call last): 126 | File "", line 1, in 127 | File "users.py", line 149, in email 128 | val = self.__email_data_type.validate(val) 129 | ... 130 | babel_data_types.ValidationError: '10' expected to be a string, got integer 131 | 132 | >>> b.email = 'bob' 133 | Traceback (most recent call last): 134 | File "", line 1, in 135 | File "users.py", line 149, in email 136 | val = self.__email_data_type.validate(val) 137 | ... 138 | babel_data_types.ValidationError: 'bob' did not match pattern '^[^@]+@[^@]+.[^@]+$' 139 | 140 | Inheritance in Babel also shows up as inheritance in Python:: 141 | 142 | >>> issubclass(Account, BasicAccount) 143 | True 144 | 145 | Accessing a required field (non-optional with no default) that has not been set 146 | raises an error:: 147 | 148 | >>> a = Account() 149 | >>> a.account_id 150 | Traceback (most recent call last): 151 | File "", line 1, in 152 | File "users.py", line 58, in account_id 153 | raise AttributeError("missing required field 'account_id'") 154 | AttributeError: missing required field 'account_id' 155 | 156 | If a field is optional and was never set, ``None`` is returned:: 157 | 158 | >>> print a.name 159 | None 160 | 161 | If a field has a default but was never set, the default is returned. 162 | 163 | Union 164 | ----- 165 | 166 | For each union in your spec, you will see a corresponding Python class of the 167 | same name. 168 | 169 | You do not use a union class's constructor directly. To select a tag with a 170 | void type, use the class attribute of the same name:: 171 | 172 | >>> GetAccountErr.no_account 173 | GetAccountErr('no_account') 174 | 175 | To select a tag with a value, use the class method of the same name and pass 176 | in an argument to serve as the value. 177 | 178 | >>> import datetime 179 | >>> Status.inactive(datetime.datetime.utcnow()) 180 | Status('inactive') 181 | 182 | The value is also validated on creation:: 183 | 184 | >>> Status.inactive('bad value') 185 | Traceback (most recent call last): 186 | File "", line 1, in 187 | File "users.py", line 121, in inactive 188 | return cls('inactive', val) 189 | ... 190 | babel_data_types.ValidationError: expected timestamp, got string 191 | 192 | To write code that handles all the tags of a union, use the ``is_[tag]()`` 193 | methods. We recommend you exhaustively check all tags, or include an else 194 | clause to ensure that all possibilities are accounted for. For tags that have 195 | values, use the ``get_[tag]()`` method to access the value:: 196 | 197 | >>> # assume that s is an instance of Status 198 | >>> if s.is_active(): 199 | ... # handle active status 200 | ... elif s.is_inactive(): 201 | ... v = s.get_inactive() 202 | ... # handle inactive status 203 | 204 | Struct With Enumerated Subtypes 205 | ------------------------------- 206 | 207 | As with regular structs, structs that enumerate subtypes have corresponding 208 | Python classes that behave identically to regular structs. 209 | 210 | The difference is apparent when a field has a data type that is a struct with 211 | enumerated subtypes. Expanding on our example from the language reference, 212 | assume the following spec:: 213 | 214 | struct Resource 215 | union* 216 | file File 217 | folder Folder 218 | 219 | path String 220 | 221 | struct File extends Resource: 222 | size UInt64 223 | 224 | struct Folder extends Resource: 225 | "No new fields." 226 | 227 | struct Response 228 | rsrc Resource 229 | 230 | If we instantiate ``Response``, the ``rsrc`` field can only be assigned a 231 | ``File`` or ``Folder`` object. It should not be assigned a ``Resource`` object. 232 | 233 | An exception to this is on deserialization. Because ``Resource`` is specified 234 | as a catch-all, it's possible when deserializing a ``Response`` to get a 235 | ``Resource`` object in the ``rsrc`` field. This indicates that the returned 236 | subtype was unknown because the recipient has an older spec than the sender. 237 | To handle catch-alls, you should use an else clause:: 238 | 239 | >>> print resp.rsrc.path # Guaranteed to work regardless of subtype 240 | >>> if isinstance(resp, File): 241 | ... # handle File 242 | ... elif isinstance(resp, Folder): 243 | ... # handle Folder 244 | ... else: 245 | ... # unknown subtype of Resource 246 | 247 | Route 248 | ----- 249 | 250 | [TODO] 251 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | ***** 2 | Babel 3 | ***** 4 | 5 | **This repository is a stub. No code is available yet. Check back soon.** 6 | 7 | Define an API once in Babel. Use code generators to translate your 8 | specification into objects and functions in the programming languages of your 9 | choice. 10 | 11 | Babel is made up of several components: 12 | 13 | 1. An interface-description language (IDL) for specifying APIs. 14 | 2. A command-line tool (``babelapi``) that takes an API specification and 15 | generator module, and generates output. 16 | 3. A Python-interface for defining new generators. 17 | 4. A JSON-compatible serialization format. 18 | 19 | Babel is in active use for the `Dropbox v2 API 20 | `_. Right now, the only available generator 21 | is for Python, but we're working on releasing the other ones we have 22 | internally: Swift, C#, Java, Go, JavaScript, and HTML documentation. 23 | 24 | * Introduction 25 | * Motivation_ 26 | * Installation_ 27 | * `Taste of Babel <#a-taste-of-babel>`_ 28 | * `Language Reference (.babel) `_ 29 | * `Choosing a Filename `_ 30 | * `Comments `_ 31 | * `Namespace `_ 32 | * `Primitive Types `_ 33 | * `Alias `_ 34 | * `Struct `_ 35 | * `Union `_ 36 | * `Nullable Type `_ 37 | * `Route `_ 38 | * `Include `_ 39 | * `Documentation `_ 40 | * `Formal Grammar `_ 41 | * `Using Generated Code `_ 42 | * `Compile with the CLI `_ 43 | * `Python Guide `_ 44 | * `Managing Specs `_ 45 | * `Using Namespaces `_ 46 | * `Splitting a Namespace Across Files `_ 47 | * `Using Header Files `_ 48 | * `Separating Public and Private Routes `_ 49 | * `Evolving a Spec `_ 50 | * `Background `_ 51 | * `Sender-Recipient `_ 52 | * `Backwards Incompatible Changes `_ 53 | * `Backwards Compatible Changes `_ 54 | * `Planning for Backwards Compatibility `_ 55 | * `Leader-Clients `_ 56 | * `Route Versioning `_ 57 | * `Writing a Generator (.babelg.py) `_ 58 | * `Using the API Object `_ 59 | * `Creating an Output File `_ 60 | * `Emit Methods `_ 61 | * `Indentation `_ 62 | * `Examples `_ 63 | * `JSON Serializer `_ 64 | * `Network Protocol `_ 65 | 66 | .. _motivation: 67 | 68 | Motivation 69 | ========== 70 | 71 | Babel was birthed at Dropbox at a time when it was becoming clear that API 72 | development needed to be scaled beyond a single team. The company was 73 | undergoing a large expansion in the number of product groups, and it wasn't 74 | scalable for the API team, which traditionally dealt with core file operations, 75 | to learn the intricacies of each product and build their APIs. 76 | 77 | Babel's chief goal is to decentralize API development and ownership at Dropbox. 78 | To be successful, it needed to do several things: 79 | 80 | **Decouple APIs from SDKS**: Dropbox has first-party clients for our mobile 81 | apps, desktop client, and website. Each of these is implemented in a different 82 | language. And we want to make onboarding easier for third-parties by providing 83 | them with SDKs. It's untenable to ask product groups that build APIs to also 84 | implement these endpoints in a half-dozen different language-specific SDKs. 85 | Without decoupling, as was the case in our v1 API, the SDKs will inevitably 86 | fall behind. Our solution is to have our SDKs automatically generated. 87 | 88 | **Improve Visibility into our APIs**: These days, APIs aren't just in the 89 | domain of engineering. Product managers, product specialists, partnerships, 90 | sales, and services groups all need to have clear and accurate specifications 91 | of our APIs. After all, APIs define Dropbox's data models and functionlaity. 92 | Before Babel, API design documents obseleted by changes during implementation 93 | were the source of truth. 94 | 95 | **Consistency and Predictability**: Consistency ranging from documentation 96 | tense to API patterns are important for making an API predictable and therefore 97 | easier to use. We needed an easy way to make and enforce patterns. 98 | 99 | **JSON**: To make consumption easier for third parties, we wanted our data 100 | types to map to JSON. For cases where serialization efficiency 101 | (space and time) are important, you can try using msgpack (alpha support 102 | available in the Python generator). It's possible also to define your own 103 | serialization scheme, but at that point, you may consider using something like 104 | `Protobuf `_. 105 | 106 | Assumptions 107 | ----------- 108 | 109 | Babel makes no assumptions about the transport layer being used to make API 110 | requests and return responses; its first use case is the Dropbox v2 API which 111 | operates over HTTP. Babel does not come with nor enforce any particular RPC 112 | framework. 113 | 114 | Babel makes some assumptions about the data types supported in target 115 | programming languages. It's assumed that there is a capacity for representing 116 | dictionaries (unordered string keys -> value), lists, numeric types, and 117 | strings. 118 | 119 | 120 | Babel assumes that a route (or API endpoint) can have its argument and 121 | result types defined without relation to each other. In other words, the 122 | type of response does not change based on the input to the endpoint. An 123 | exception to this rule is afforded for error responses. 124 | 125 | .. _installation: 126 | 127 | Installation 128 | ============ 129 | 130 | Download or clone BabelAPI, and run the following in its root directory:: 131 | 132 | $ sudo python setup.py install 133 | 134 | This will install a script ``babelapi`` to your PATH that can be run from the 135 | command line:: 136 | 137 | $ babelapi -h 138 | 139 | Alternative 140 | ----------- 141 | 142 | If you choose not to install ``babelapi`` using the method above, you will need 143 | to ensure that you have the Python packages ``ply`` and ``six``, which can be 144 | installed through ``pip``:: 145 | 146 | $ pip install ply>=3.4 six>=1.3.0 147 | 148 | If the ``babelapi`` package is in your PYTHONPATH, you can replace ``babelapi`` 149 | with ``python -m babelapi.cli`` as follows:: 150 | 151 | $ python -m babelapi.cli -h 152 | 153 | If you have the ``babelapi`` package on your machine, but did not install it or 154 | add its location to your PYTHONPATH, you can use the following:: 155 | 156 | $ PYTOHNPATH=path/to/babelapi python -m babelapi.cli -h 157 | 158 | .. taste-of-babel: 159 | 160 | A Taste of Babel 161 | ================ 162 | 163 | Here we define a hypothetical route that shows up in some form or another in 164 | APIs for web services: querying the account information for a user of a 165 | service. Our hypothetical spec lives in a file called ``users.babel``:: 166 | 167 | # We put this in the "users" namespace in anticipation that 168 | # there would be many user-account-related routes. 169 | namespace users 170 | 171 | route get_account (GetAccountReq, Account, GetAccountErr) 172 | "Get information about a specified user's account." 173 | 174 | # This struct represents the input data to the route. 175 | struct GetAccountReq 176 | account_id AccountId 177 | 178 | # We define an AccountId as being a 10-character string 179 | # once here to avoid declaring it each time. 180 | alias AccountId = String(min_length=10, max_length=10) 181 | 182 | struct Account 183 | "Information about a user's account." 184 | 185 | account_id AccountId 186 | "A unique identifier for the user's account." 187 | email String(pattern="^[^@]+@[^@]+\.[^@]+$") 188 | "The e-mail address of the user." 189 | name String(min_length=1)? 190 | "The user's full name. :val:`null` if no name was provided." 191 | status Status 192 | "The status of the account." 193 | 194 | example default "A regular user" 195 | account_id="id-48sa2f0" 196 | email="alex@example.org" 197 | name="Alexander the Great" 198 | status=active 199 | 200 | union Status 201 | active 202 | "The account is active." 203 | inactive Timestamp("%a, %d %b %Y %H:%M:%S") 204 | "The account is inactive. The value is when the account was 205 | deactivated." 206 | 207 | # This union represents the possible errors that might be returned. 208 | union GetAccountErr 209 | no_account 210 | "No account with the requested id could be found." 211 | perm_denied 212 | "Insufficient privileges to query account information." 213 | unknown* 214 | 215 | Using the Python generator, we can generate a Python module that mirrors this 216 | specification using the command-line interface. From the top-level of the 217 | ``babelapi`` folder, try:: 218 | 219 | $ babelapi generator/python/python.babelg.py . users.babel 220 | 221 | Now we can interact with the specification in Python:: 222 | 223 | $ python -i users.py 224 | >>> a = Account() 225 | >>> a.account_id = 1234 # fails type validation 226 | Traceback (most recent call last): 227 | ... 228 | babel_data_types.ValidationError: '1234' expected to be a string, got integer 229 | 230 | >>> a.account_id = '1234' # fails length validation 231 | Traceback (most recent call last): 232 | ... 233 | babel_data_types.ValidationError: '1234' must be at least 10 characters, got 4 234 | 235 | >>> a.account_id = 'id-48sa2f0' # passes validation 236 | 237 | >>> # Now we use the included JSON serializer 238 | >>> from babel_serializers import json_encode 239 | >>> a2 = Account(account_id='id-48sa2f0', name='Alexander the Great', 240 | ... email='alex@example.org', status=Status.active) 241 | >>> json_encode(GetAccountRoute.result_data_type, a2) 242 | '{"status": "active", "account_id": "id-48sa2f0", "name": "Alexander the Great", "email": "alex@example.org"}' 243 | -------------------------------------------------------------------------------- /doc/lang_ref.rst: -------------------------------------------------------------------------------- 1 | ****************** 2 | Language Reference 3 | ****************** 4 | 5 | To illustrate how to write a spec, we're going to dissect a spec that defines 6 | a hypothetical route that shows up in some form or another in most APIs for 7 | web services: querying the account information for a user. 8 | 9 | The spec should live in a file called ``users.babel``:: 10 | 11 | # We put this in the "users" namespace in anticipation that 12 | # there would be many user-account-related routes. 13 | namespace users 14 | 15 | # We define an AccountId as being a 10-character string 16 | # once here to avoid declaring it each time. 17 | alias AccountId = String(min_length=10, max_length=10) 18 | 19 | struct BasicAccount 20 | "Basic information about a user's account." 21 | 22 | account_id AccountId 23 | "A unique identifier for the user's account." 24 | email String(pattern="^[^@]+@[^@]+\.[^@]+$") 25 | "The e-mail address of the user." 26 | 27 | union Status 28 | active 29 | "The account is active." 30 | inactive Timestamp("%a, %d %b %Y %H:%M:%S") 31 | "The account is inactive. The value is when the account was 32 | deactivated." 33 | 34 | struct Account extends BasicAccount 35 | "Information about a user's account." 36 | 37 | name String(min_length=1)? 38 | "The user's full name. :val:`null` if no name was provided." 39 | status Status 40 | "The status of the account." 41 | 42 | example default 43 | "A regular user" 44 | account_id="id-48sa2f0" 45 | email="alex@example.org" 46 | name="Alexander the Great" 47 | status=active 48 | 49 | # This struct represents the input data to the route. 50 | struct GetAccountReq 51 | account_id AccountId 52 | 53 | # This union represents the possible errors that might be returned. 54 | union GetAccountErr 55 | no_account 56 | "No account with the requested id could be found." 57 | perm_denied 58 | "Insufficient privileges to query account information." 59 | unknown* 60 | 61 | route get_account (GetAccountReq, Account, GetAccountErr) 62 | "Get information about a specified user's account." 63 | 64 | Choosing a Filename 65 | =================== 66 | 67 | All specifications must have a ``.babel`` extension. We recommend that the 68 | name of the file be the same as the `namespace <#ns>`_ defined in the spec. 69 | 70 | Comments 71 | ======== 72 | 73 | Any text between a hash ``#`` and a newline is considered a comment. Comments 74 | can take up an entire line, or they can be added to the end of a line. 75 | 76 | Use comments to explain a section of the spec to a reader of the spec. Unlike 77 | `documentation <#documentation>`_ strings, comments are not accessible to 78 | generators as they are ignored by the parser. 79 | 80 | .. _ns: 81 | 82 | Namespace 83 | ========= 84 | 85 | Specs must begin with a namespace declaration as is the case here:: 86 | 87 | namespace users 88 | 89 | This logically groups all of the routes and data types in the spec file into 90 | the ``users`` namespace. A spec file must declare exactly one namespace, but 91 | multiple spec files may contribute to the same namespace. 92 | 93 | Namespaces are useful for grouping related functionality together. For example, 94 | the Dropbox API has a namespace devoted to all file operations (uploading, 95 | downloading, ...), and another namespace for all operations relevant to user 96 | accounts. 97 | 98 | Basic Types 99 | =========== 100 | 101 | In the example, ``String`` and ``Timestamp`` are basic types. Here's a 102 | table of all such types and the arguments they take: 103 | 104 | ======================= ================================= ===================== 105 | Type Arguments (**bold** are required Notes 106 | and positional) 107 | ======================= ================================= ===================== 108 | Bytes -- An array of bytes. 109 | Boolean -- 110 | Float{32,64} * min_value 111 | * max_value 112 | Int{32,64}, UInt{32,64} * min_value 113 | * max_value 114 | List * **data_type**: A primitive or Lists are homogeneous. 115 | composite type. 116 | * min_items 117 | * max_items 118 | String * min_length A unicode string. 119 | * max_length 120 | * pattern: A regular expression 121 | to be used for validation. 122 | Timestamp * **format**: Specified as a This is used by the 123 | string understood by JSON-serializer since 124 | strptime(). it has no native 125 | timestamp data type. 126 | Void -- 127 | ======================= ================================= ===================== 128 | 129 | Positional arguments (bold in the above table) are always required and appear 130 | at the beginning of an argument list:: 131 | 132 | struct ShoppingList 133 | items List(String) 134 | 135 | Keyword arguments are optional and are preceded by the argument name and an 136 | ``=``:: 137 | 138 | struct Person 139 | age UInt64(max_value=130) 140 | 141 | If no arguments are needed, the parentheses can be omitted:: 142 | 143 | struct Example 144 | number Int64 145 | string String 146 | 147 | Here are some more examples:: 148 | 149 | struct Coordinate 150 | x Int64 151 | y Int64 152 | 153 | struct Example 154 | f1 Bytes 155 | f2 Boolean 156 | f3 Float64(min_value=0) 157 | # List of primitive types 158 | f4 List(Int64) 159 | # List of user-defined types 160 | f5 List(Coordinate, max_items=10) 161 | f6 String(pattern="^[A-z]+$") 162 | f7 Timestamp("%a, %d %b %Y %H:%M:%S +0000") 163 | 164 | Mapping to a Target Language 165 | ---------------------------- 166 | 167 | Code generators map the primitive types of Babel to types in a target language. 168 | For more information, consult the appropriate guide in `Using Generated Code 169 | `_. 170 | 171 | Alias 172 | ===== 173 | 174 | Aliases let you parameterize a type once with a name and optional documentation 175 | string, and then use that name elsewhere:: 176 | 177 | alias AccountId = String(min_length=10, max_length=10) 178 | "A unique identifier for the user's account." 179 | 180 | In our example, declaring an ``AccountId`` alias makes future references to it 181 | clearer since the name provides an extra semantic hint:: 182 | 183 | struct BasicAccount 184 | "Basic information about a user's account." 185 | 186 | account_id AccountId 187 | 188 | struct GetAccountReq 189 | account_id AccountId 190 | 191 | Aliases make refactoring easier. We only need to change the definition of the 192 | ``AccountId`` alias to change it everywhere. 193 | 194 | Aliases can reference user-defined types and other aliases, and can make a type 195 | nullable. 196 | 197 | Struct 198 | ====== 199 | 200 | A struct is a user-defined type made up of fields that have their own types:: 201 | 202 | struct BasicAccount 203 | "Basic information about a user's account. 204 | 205 | This can be multi-line." 206 | 207 | account_id AccountId 208 | "A unique identifier for the user's account." 209 | email String(pattern="^[^@]+@[^@]+\.[^@]+$") 210 | "The e-mail address of the user." 211 | 212 | A struct can be documented by specifying a string immediately following the 213 | struct declaration. The string can be multiple lines, as long as each 214 | subsequent line is at least at the indentation of the starting quote. 215 | Refer to `Documentation`_ for more. 216 | 217 | After the documentation is a list of fields. Fields are formatted with the field 218 | name first followed by the field type. To provide documentation for a field, 219 | specify a string on a new indented line following the field declaration. 220 | 221 | Inheritance 222 | ----------- 223 | 224 | Using the ``extends`` keyword, a struct can declare itself a subtype of another 225 | struct, known as the supertype. The subtype inherit all the fields of the 226 | supertype:: 227 | 228 | struct Account extends BasicAccount 229 | 230 | ``Account`` inherits ``account_id`` and ``email`` from ``BasicAccount``. 231 | 232 | A feature common to object-oriented programming, a subtype may be used in place 233 | of a supertype. 234 | 235 | Composition 236 | ----------- 237 | 238 | User-defined types can be composed of other user-defined types, either 239 | structs or unions:: 240 | 241 | union Status 242 | active 243 | "The account is active." 244 | inactive Timestamp("%a, %d %b %Y %H:%M:%S") 245 | "The account is inactive. The value is when the account was 246 | deactivated." 247 | 248 | struct Account extends BasicAccount 249 | "Information about a user's account." 250 | 251 | name String(min_length=1)? 252 | "The user's full name. :val:`null` if no name was provided." 253 | status Status 254 | "The status of the account." 255 | 256 | Defaults 257 | -------- 258 | 259 | A field with a primitive type can have a default set with a ``=`` followed by 260 | a value at the end of the field declaration:: 261 | 262 | struct Example 263 | number UInt64 = 1024 264 | string String = "hello, world." 265 | 266 | Setting a default means that a field is optional. If it is not specified in a 267 | message, the receiver should not error, but instead return the default when 268 | the field is queried. The receiver should, however, track the fact that the 269 | field was unspecified, so that if the message is re-serialized the default is 270 | not present in the message. 271 | 272 | A default cannot be set for a nullable type. Nullable types implicitly have a 273 | default of ``null``. 274 | 275 | A default can be set for a field with a union data type, but only to a union 276 | member with a void type. Using the example of ``Account``, the ``status`` can 277 | be set to ``active`` by default:: 278 | 279 | struct Account extends BasicAccount 280 | "Information about a user's account." 281 | 282 | name String(min_length=1)? 283 | "The user's full name. :val:`null` if no name was provided." 284 | status Status = active 285 | "The status of the account." 286 | 287 | In practice, defaults are useful when `evolving a spec `_. 288 | 289 | Examples 290 | -------- 291 | 292 | Examples let you include realistic samples of data in definitions. This gives 293 | spec readers a concrete idea of what typical values will look like. Also, 294 | examples help demonstrate how distinct fields might interact with each other. 295 | 296 | Generators have access to examples, which is useful when automatically 297 | generating documentation. 298 | 299 | An example is declared by using the ``example`` keyword followed by a label. 300 | By convention, "default" should be used as the label name for an example that 301 | can be considered a good representation of the general case for the type:: 302 | 303 | struct Account extends BasicAccount 304 | "Information about a user's account." 305 | 306 | name String(min_length=1)? 307 | "The user's full name. :val:`null` if no name was provided." 308 | status Status 309 | "The status of the account." 310 | 311 | example default 312 | "A regular user" 313 | account_id = "id-48sa2f0" 314 | email = "alex@example.org" 315 | name = "Alexander the Great" 316 | status = active 317 | 318 | example unnamed 319 | "An anonymous user" 320 | account_id = "id-29sk2p1" 321 | email = "anony@example.org" 322 | name = null 323 | status = active 324 | 325 | Every required field (not nullable and no default) must be specified, otherwise 326 | an error will be returned. ``null`` can be used to mark that a nullable type 327 | is not present. 328 | 329 | An optional multi-line documentation string can be specified after the line 330 | declaring the example and before the example fields. 331 | 332 | When you have a set of nested types, each type defines examples for its fields 333 | with primitive types. For fields with composite types, the value of the example 334 | must be a label of an example in the target composite type. Here's an example 335 | where ``Name`` is now its own struct:: 336 | 337 | struct Account extends BasicAccount 338 | 339 | name Name 340 | 341 | example default 342 | account_id = "id-48sa2f0" 343 | email = "alex@example.org" 344 | name = default 345 | status = active 346 | 347 | example anonymous 348 | account_id = "id-29sk2p1" 349 | email = "anony@example.org" 350 | name = anonymous 351 | status = active 352 | 353 | struct Name 354 | first_name String? 355 | 356 | example default 357 | first_name = "Alexander the Great" 358 | 359 | example anonymous 360 | first_name = null 361 | 362 | As you can see, the ``anonymous`` example for ``Account`` explicitly references 363 | the ``anonymous`` example for ``Name``. 364 | 365 | Examples for unions must only specify one field, since only one union member 366 | can be selected at a time. For example:: 367 | 368 | union Owner 369 | nobody 370 | account Account 371 | organization String 372 | 373 | example default 374 | nobody = null 375 | 376 | example person 377 | account = default 378 | 379 | example group 380 | organization = "Dropbox" 381 | 382 | In the ``default`` example, notice that void tags are specified with a value of 383 | ``null``. In the ``person`` example, the ``default`` example for the 384 | ``Account`` type is referenced. 385 | 386 | Lists can be expressed with bracket notation:: 387 | 388 | struct S 389 | l1 List(String) 390 | l2 List(T) 391 | l3 List(List(T)) 392 | 393 | example default 394 | l1 = ["hello", "world"] 395 | l2 = [start, end] 396 | l3 = [[start], []] 397 | 398 | struct T 399 | i UInt64 400 | 401 | example start 402 | i = 0 403 | 404 | example end 405 | i = 42 406 | 407 | Union 408 | ===== 409 | 410 | A union in Babel is a 411 | `tagged union `_. Think of it as a 412 | type that can store one of several different possibilities at a time. Each 413 | possibility has an identifier that is called a "tag". In our example, the union 414 | ``Status`` has tags ``active`` and ``inactive``:: 415 | 416 | union Status 417 | "The status of a user's account." 418 | 419 | active 420 | "The account is active." 421 | inactive Timestamp("%a, %d %b %Y %H:%M:%S") 422 | "The account is inactive. The value is when the account was 423 | deactivated." 424 | 425 | A tag is associated with a type (``inactive`` stores a ``Timestamp``). If the 426 | type is omitted as in the case of ``active``, the type is implicitly ``Void``. 427 | 428 | The primary advantage of a union is its logical expressiveness. You'll often 429 | encounter types that are best described as choosing between a set of options. 430 | Avoid the common anti-pattern of using a struct with a nullable field for each 431 | option, and relying on your application logic to enforce that only one is set. 432 | 433 | Another advantage is that for languages that support tagged unions, the 434 | compiler can check that your application code handles all possible cases and 435 | that accesses are safe. Generators will take advantage of such features when 436 | they are available in the target language. 437 | 438 | Like a struct, a documentation string can follow the union declaration and/or 439 | follow each tag definition. 440 | 441 | Catch-all Tag 442 | ------------- 443 | 444 | By default, we consider unions to be closed. That is, for the sake of backwards 445 | compatibility, a recipient of a message should never encounter a tag that it 446 | isn't aware of. A recipient can therefore confidently handle the case where a 447 | user is ``active`` or ``inactive`` and trust that no other value will ever be 448 | encountered. 449 | 450 | Because we anticipate that this will be constricting for APIs undergoing 451 | evolution, we've introduced the notion of a catch-all tag. If a recipient 452 | receives a tag that it isn't aware of, it will default the union to the 453 | catch-all tag. 454 | 455 | The notation is simply an ``*`` that follows a tag with an omitted type, ie. 456 | its type is Void:: 457 | 458 | union GetAccountErr 459 | no_account 460 | "No account with the requested id could be found." 461 | perm_denied 462 | "Insufficient privileges to query account information." 463 | unknown* 464 | 465 | In the example above, a recipient should have written code to handle 466 | ``no_account``, ``perm_denied``, and ``unknown``. If a tag that was not 467 | previously known is received (e.g. ``bad_account``), the union will default 468 | to the ``unknown`` tag. 469 | 470 | We expect this to be especially useful for unions that represent the possible 471 | errors a route might return. Recipients in the wild may have been generated 472 | with only a subset of the current errors, but they'll continue to function 473 | appropriately as long as they handle the catch-all tag. 474 | 475 | Inheritance 476 | ----------- 477 | 478 | Using the ``extends`` keyword, a union can declare itself as a supertype of 479 | another union, known as the subtype. The supertype will have all the tags of 480 | the subtype:: 481 | 482 | union DeleteAccountError extends GetAccountError 483 | 484 | ``DeleteAccount`` inherits the tags ``no_account``, ``perm_denied``, and 485 | ``unknown`` from ``GetAccountError``. Since ``GetAccountError`` has already 486 | defined a catch-all tag, ``DeleteAccountError`` or any other supertype cannot 487 | declare another catch-all. 488 | 489 | Note that the supertype/subtype relationship created by ``extends`` between two 490 | unions is the opposite of an ``extends`` between two structs. It's stated this 491 | way to maintain the invariant that a subtype may be used in place of a 492 | supertype. Specifically, a ``GetAccountError`` can be used in place of 493 | ``DeleteAccountError`` because a handler will be prepared for all possibilities 494 | of ``GetAccountError`` since they are a subset of ``DeleteAccountError``. 495 | 496 | 497 | Struct With Enumerated Subtypes 498 | =============================== 499 | 500 | If a struct enumerates its subtypes, an instance of any subtype will satisfy 501 | the type constraint. This is useful when wanting to discriminate amongst types 502 | that are part of the same hierarchy while simultaneously being able to avoid 503 | discriminating when accessing common fields. 504 | 505 | To declare the enumeration, define a union following the documentation string 506 | of the struct if one exists. Unlike a regular union, it is unnamed. Each member 507 | of the union specifies a tag followed by the name of a subtype. The tag (known 508 | as the "type tag") is present in the serialized format to distinguish between 509 | subtypes. For example:: 510 | 511 | struct Resource 512 | "Sample doc." 513 | 514 | union 515 | file File 516 | folder Folder 517 | 518 | path String 519 | 520 | struct File extends Resource: 521 | ... 522 | 523 | struct Folder extends Resource: 524 | ... 525 | 526 | Anywhere ``Resource`` is referenced, an instance of ``File`` or ``Folder`` 527 | satisfies the type constraint. 528 | 529 | A struct that enumerates subtypes cannot inherit from any other struct. Also, 530 | type tags cannot match any field names. 531 | 532 | Catch-all 533 | --------- 534 | 535 | Similar to a union, a struct with enumerated types can be labeled as a 536 | catch-all. This is done by appending an asterix, ``*``, to the ``union``:: 537 | 538 | struct Resource 539 | "Sample doc." 540 | 541 | union* 542 | file File 543 | folder Folder 544 | 545 | path String 546 | 547 | struct File extends Resource: 548 | ... 549 | 550 | struct Folder extends Resource: 551 | ... 552 | 553 | If recipient receives a tag for a subtype that it is unaware of, it will 554 | substitute the base struct in its place if it's a catch-all. In the example 555 | above, if the subtype is a ``Symlink`` (not shown), then the recipient will 556 | return a ``Resource`` in its place. 557 | 558 | Nullable Type 559 | ============= 560 | 561 | When a type is followed by a ``?``, the type is nullable:: 562 | 563 | name String(min_length=1)? 564 | 565 | Nullable means that the type can be unspecified, ie. ``null``. Code generators 566 | should use a language's native facilities for null, 567 | `boxed types `_, 568 | and `option types `_ if possible. For 569 | languages that do not support these features, a separate function to check for 570 | the presence of a type is the preferred method. 571 | 572 | A nullable type is considered optional. If it is not specified in a message, 573 | the receiver should not error, but instead treat it as absent. 574 | 575 | Route 576 | ===== 577 | 578 | Routes correspond to your API endpoints:: 579 | 580 | route get_account (GetAccountReq, Account, GetAccountErr) 581 | "Get information about a specified user's account." 582 | 583 | The route is named ``get_account``. ``GetAccountReq`` is the data type of 584 | the request to the route. ``Account`` is the data type of a response from the 585 | route. ``GetAccountErr`` is the data type of an error response. 586 | 587 | Similar to structs and unions, a documentation string may follow the route 588 | signature. 589 | 590 | Attributes 591 | ---------- 592 | 593 | A full description of an API route tends to require vocabulary that is specific 594 | to a service. For example, the Dropbox API needs a way to specify some routes 595 | as including a binary body (uploads) for requests. Another example is specifying 596 | which routes can be used without authentication credentials. 597 | 598 | To cover this open-ended use case, routes can have an ``attrs`` section declared 599 | followed by an arbitrary set of ``key=value`` pairs:: 600 | 601 | route ping (Void, Void, Void) 602 | 603 | attrs 604 | key1 = "value1" 605 | key2 = 1234 606 | key3 = 3.14 607 | key4 = false 608 | key5 = null 609 | 610 | A value can reference a union tag with void type:: 611 | 612 | route ping (Void, Void, Void) 613 | 614 | attrs 615 | key = Letters.a 616 | 617 | union Letters 618 | a 619 | b 620 | c 621 | 622 | Code generators will populate a route object with these attributes. 623 | 624 | Deprecation 625 | ----------- 626 | 627 | You can mark a route as deprecated as follows:: 628 | 629 | route old_route (Arg, Void, Void) deprecated 630 | 631 | If the route is deprecated in favor of a newer route, use ``deprecated by`` 632 | followed by the new route's name:: 633 | 634 | route old_route (Arg, Void, Void) deprecated by new_route 635 | 636 | route new_route (NewArg, NewResult, Void) 637 | 638 | Import 639 | ====== 640 | 641 | You can refer to types and aliases in other namespaces by using the ``import`` 642 | directive. 643 | 644 | For example, we can move the definition of ``AccountId`` and ``BasicAccount`` 645 | into a file called ``common.babel``:: 646 | 647 | namespace common 648 | 649 | # We define an AccountId as being a 10-character string 650 | # once here to avoid declaring it each time. 651 | alias AccountId = String(min_length=10, max_length=10) 652 | 653 | struct BasicAccount 654 | "Basic information about a user's account." 655 | 656 | account_id AccountId 657 | "A unique identifier for the user's account." 658 | email String(pattern="^[^@]+@[^@]+\.[^@]+$") 659 | "The e-mail address of the user." 660 | 661 | Now in ``users.babel``, we add an ``import`` statement under the namespace 662 | directive as follows:: 663 | 664 | namespace users 665 | 666 | import common 667 | 668 | When referencing data types in ``common``, use the prefix ``common.``. For 669 | example, ``common.AccountId`` and ``common.BasicAccount``. 670 | 671 | .. _doc: 672 | 673 | Documentation 674 | ============= 675 | 676 | Documentation strings are an important part of specifications, which is why 677 | they can be attached to routes, structs, struct fields, unions, and union 678 | options. It's expected that most elements should be documented. It's not 679 | required only because some definitions are self-explanatory or adding 680 | documentation would be redundant, as is often the case when a struct field 681 | (with a doc) references a struct (with a doc). 682 | 683 | Documentation is accessible to generators. Code generators will inject 684 | documentation into the language objects that represent routes, structs, and 685 | unions. Generators for API documentation will find documentation strings 686 | especially useful. 687 | 688 | .. _doc-refs: 689 | 690 | References 691 | ---------- 692 | 693 | References help generators tailor documentation strings for a target 694 | programming language. 695 | 696 | References have the following format:: 697 | 698 | :tag:`value` 699 | 700 | Supported tags are ``route``, ``type``, ``field``, ``link``, and ``val``. 701 | 702 | route 703 | A reference to a route. The value should be the name of the route. Code 704 | generators should reference the class or function that represents the route. 705 | type 706 | A reference to a user-defined data type (Struct or Union). The value should 707 | be the name of the user-defined type. 708 | field 709 | A reference to a field of a struct or a tag of a union. If the field being 710 | referenced is a member of a different type than the docstring, then use the 711 | format `TypeName.field_name`. Otherwise, use just the field name as the 712 | value. 713 | link 714 | A hyperlink. The format of the value is `` ``, e.g. 715 | ``Babel Repo https://github.com/dropbox/babelapi``. Everything after the 716 | last space is considered the URI. The rest is treated as the title. For 717 | this reason, you should ensure that your URIs are 718 | `percent encoded `_. 719 | Generators should convert this to a hyperlink understood by the target 720 | language. 721 | val 722 | A value. Supported values include ``null``, ``true``, ``false``, integers, 723 | floats, and strings. Generators should convert the value to the native 724 | representation of the value for the target language. 725 | 726 | Grammar 727 | ======= 728 | 729 | Specification:: 730 | 731 | Spec ::= Namespace Import* Definition* 732 | Namespace ::= 'namespace' Identifier 733 | Import ::= 'import' Identifier 734 | Definition ::= Alias | Route | Struct | Union 735 | Alias ::= 'alias' Identifier '=' TypeRef (NL INDENT Doc DEDENT)? 736 | 737 | Struct:: 738 | 739 | Struct ::= 'struct' Identifier Inheritance? NL INDENT Doc? Subtypes? Field* Example* DEDENT 740 | Inheritance ::= 'extends' Identifier 741 | SubtypeField ::= Identifier TypeRef NL 742 | Subtypes ::= 'union' NL INDENT SubtypeField+ DEDENT 743 | Default ::= '=' Literal 744 | Field ::= Identifier TypeRef Default? (NL INDENT Doc DEDENT)? 745 | 746 | Union:: 747 | 748 | Union ::= 'union' Identifier NL INDENT (VoidTag|Tag)* DEDENT 749 | VoidTag ::= Identifier '*'? (NL INDENT Doc DEDENT)? 750 | Tag ::= Identifier TypeRef (NL INDENT Doc DEDENT)? 751 | 752 | Route:: 753 | 754 | Route ::= 'route' Identifier '(' TypeRef ',' TypeRef ',' TypeRef ')' (NL INDENT Doc DEDENT)? 755 | 756 | Type Reference:: 757 | 758 | Attributes ::= '(' (Identifier '=' (Literal | Identifier) ','?)* ')' 759 | TypeRef ::= Identifier Attributes? '?'? 760 | 761 | Primitives:: 762 | 763 | Primitive ::= 'Bytes' | 'Boolean' | 'Float32' | 'Float64' | 'Int32' 764 | | 'Int64' | 'UInt32' | 'UInt64' | 'String' | 'Timestamp' 765 | 766 | Composites:: 767 | 768 | Composite ::= 'List' 769 | 770 | Basic:: 771 | 772 | Identifier ::= (Letter | '_')? (Letter | Digit | '_')* # Should we allow trailing underscores? 773 | Letter ::= ['A'-'z'] 774 | Digit ::= ['0'-'9'] 775 | Literal :: = BoolLiteral | FloatLiteral | IntLiteral | StringLiteral 776 | BoolLiteral ::= 'true' | 'false' 777 | FloatLiteral ::= '-'? Digit* ('.' Digit+)? ('E' IntLiteral)? 778 | IntLiteral ::= '-'? Digit+ 779 | StringLiteral ::= '"' .* '"' # Not accurate 780 | Doc ::= StringLiteral # Not accurate 781 | NL = Newline 782 | INDENT = Incremental indentation 783 | DEDENT = Decremented indentation 784 | 785 | TODO: Need to add additional information about handling of NL, INDENT, DEDENT, 786 | and whitespace between tokens. Also, the attrs section of Routes and 787 | examples (+ lists). 788 | -------------------------------------------------------------------------------- /doc/generator_ref.rst: -------------------------------------------------------------------------------- 1 | ******************* 2 | Writing a Generator 3 | ******************* 4 | 5 | This document explains how to write your own generator. If you're simply 6 | looking to use an included generator, please see `Using Generated Code 7 | `_. 8 | 9 | Generators convert a spec into some other markup or code. Most commonly, a 10 | generator will target a programming language and convert a spec into classes 11 | and functions. But, generators can also create markup for things like API 12 | documentation. 13 | 14 | Generators are written as Python modules that satisfy the following 15 | conditions: 16 | 17 | 1. The filename must have a ``.babelg.py`` extension. For example, 18 | ``example.babelg.py`` 19 | 20 | 2. At least one class must exist in the module that extends the 21 | ``babelapi.generator.CodeGenerator`` class and implements the abstract 22 | ``generate()`` method. BabelAPI automatically detects subclasses and calls 23 | the ``generate()`` method. All such subclasses will be called in ASCII 24 | order. 25 | 26 | Getting Started 27 | =============== 28 | 29 | Here's a simple no-op generator:: 30 | 31 | from babelapi.generator import CodeGenerator 32 | 33 | class ExampleGenerator(CodeGenerator): 34 | def generate(self, api): 35 | pass 36 | 37 | Assuming that the generator is saved in your current directory as 38 | ``example.babelg.py`` and that our running example spec ``users.babel`` from the 39 | `Language Reference `_ is also in the current directory. you can 40 | invoke the generator with the following command:: 41 | 42 | $ babelapi example.babelg.py users.babel . 43 | 44 | Generating Output Files 45 | ======================= 46 | 47 | To create an output file, use the ``output_to_relative_path()`` method. 48 | Its only argument is the path relative to the output directory, which was 49 | specified as an argument to ``babelapi``, where the file should be created. 50 | 51 | Here's an example generator that creates an output file for each namespace. 52 | Each file is named after a respective namespace and have a ``.cpp`` extension. 53 | Each file contains a one line C++-style comment:: 54 | 55 | from babelapi.generator import CodeGenerator 56 | 57 | class ExampleGenerator(CodeGenerator): 58 | def generate(self, api): 59 | for namespace_name in api.namespaces: 60 | with self.output_to_relative_path(namespace_name + '.cpp'): 61 | self.emit('/* {} */'.format(namespace_name)) 62 | 63 | Using the API Object 64 | ==================== 65 | 66 | The ``generate`` method receives an ``api`` variable, which represents the API 67 | spec as a Python object. The object is an instance of the ``babelapi.api.Api`` 68 | class. From this object, you can access all the defined namespaces, data types, 69 | and routes. 70 | 71 | Api 72 | --- 73 | 74 | namespaces 75 | A map from namespace name to Namespace object. 76 | 77 | 78 | Namespace 79 | --------- 80 | 81 | name 82 | The name of the namespace. 83 | 84 | doc 85 | The documentation string for the namespace. This is a concatenation of the 86 | docstrings for this namespace across all spec files in the order that they 87 | were specified to `babelapi` on the command line. The string has no leading 88 | or trailing whitespace except for a newline at the end. 89 | 90 | If no documentation string exists, this is ``None``. 91 | 92 | routes 93 | A list of Route objects in alphabetical order. 94 | 95 | route_by_name 96 | A map from route name to Route object. 97 | 98 | data_types 99 | A list of user-defined DataType objects in alphabetical order. 100 | 101 | data_type_by_name 102 | A map from data type name to DataType object. 103 | 104 | aliases 105 | A list of Alias objects in alphabetical order. Aliases will only be 106 | available if the generator has set its ``preserve_aliases`` class variable 107 | to true. 108 | 109 | alias_type_by_name 110 | A map from alias name to Alias object. 111 | 112 | get_imported_namespaces(must_have_imported_data_type=False) 113 | A list of Namespace objects. A namespace is a member of this list if it is 114 | imported by the current namespace and a data type or alias is referenced 115 | from it. If you want only namespaces with aliases referenced, set the 116 | ``must_have_imported_data_type`` parameter to true. Namespaces are in ASCII 117 | order by name. 118 | 119 | get_namespaces_imported_by_route_io() 120 | A list of Namespace objects. A namespace is a member of this list if it is 121 | imported by the current namespace and has a data type from it referenced as 122 | an argument, result, or error of a route. Namespaces are in ASCII order by 123 | name. 124 | 125 | get_route_io_data_types() 126 | A list of all user-defined data types that are referenced as either an 127 | argument, result, or error of a route. If a List or Nullable data type is 128 | referenced, then the contained data type is returned assuming it's a 129 | user-defined type. 130 | 131 | linearize_data_types() 132 | Returns a list of all data types used in the namespace. Because the 133 | inheritance of data types can be modeled as a DAG, the list will be a 134 | linearization of the DAG. It's ideal to generate data types in this 135 | order so that user-defined types that reference other user-defined types 136 | are defined in the correct order. 137 | 138 | linearize_aliases() 139 | Returns a list of all aliases used in the namespace. The aliases are 140 | ordered to ensure that if they reference other aliases those aliases come 141 | earlier in the list. 142 | 143 | Route 144 | ----- 145 | 146 | name 147 | The name of the route. 148 | 149 | deprecated 150 | Set to a ``DeprecationInfo`` object if this route is deprecated. If the 151 | route was deprecated by a newer route, ``DeprecationInfo`` will have 152 | a ``by`` attribute populated with the new route. 153 | 154 | doc 155 | The documentation string for the route. 156 | 157 | arg_data_type 158 | A DataType object of the arg to the route. 159 | 160 | arg_data_type 161 | A DataType object of the result of the route. 162 | 163 | error_data_type 164 | A DataType object of the error of the route. 165 | 166 | attrs 167 | A map from string keys to values that is a direct copy of the attrs 168 | specified in the route definition. Values are limited to Python primitives 169 | (None, bool, float, int, str) and `TagRef objects <#union-tag-reference>`_. 170 | 171 | See the Python object definition for more information. 172 | 173 | DataType 174 | -------- 175 | 176 | name 177 | The name of the data type. 178 | 179 | See ``babelapi.data_type`` for all primitive type definitions and their 180 | attributes. 181 | 182 | Struct 183 | ------ 184 | 185 | name 186 | The name of the struct. 187 | 188 | namespace 189 | The namespace the struct was defined in. 190 | 191 | doc 192 | The documentation string for the struct. 193 | 194 | fields 195 | A list of StructField objects defined by this struct. Does not include any 196 | inherited fields. 197 | 198 | all_fields 199 | A list of StructField objects including inherited fields. Required fields 200 | come before optional fields. 201 | 202 | all_required_fields 203 | A list of StructField objects required fields. Includes inherited fields. 204 | 205 | all_optional_fields 206 | A list of StructField objects for optional fields. Includes inherited 207 | fields. Optional fields are those that have defaults, or have a data type 208 | that is nullable. 209 | 210 | parent_type 211 | If it exists, it points to a DataType object (another struct) that this 212 | struct inherits from. 213 | 214 | has_documented_type_or_fields(include_inherited_fields=False) 215 | Returns whether this type, or any of its fields, are documented. 216 | 217 | Use this when deciding whether to create a block of documentation for 218 | this type. 219 | 220 | has_documented_fields(include_inherited_fields=False) 221 | Returns whether at least one field is documented. 222 | 223 | get_all_subtypes_with_tags() 224 | Unlike other enumerated-subtypes-related functionality, this method returns 225 | not just direct subtypes, but all subtypes of this struct. The tag of each 226 | subtype is the tag of the enumerated subtype from which it descended. 227 | 228 | The return value is a list of tuples representing subtypes. Each tuple has 229 | two items. First, the type tag to be used for the subtype. Second, a 230 | ``Struct`` object representing the subtype. 231 | 232 | Use this when you need to generate a lookup table for a root struct that 233 | maps a generated class representing a subtype to the tag it needs in the 234 | serialized format. 235 | 236 | Raises an error if the struct doesn't enumerate subtypes. 237 | 238 | get_enumerated_subtypes() 239 | Returns a list of subtype fields. Each field has a ``name`` attribute which 240 | is the tag for the subtype. Each field also has a ``data_type`` attribute 241 | that is a ``Struct`` object representing the subtype. 242 | 243 | Raises an error if the struct doesn't enumerate subtypes. 244 | 245 | has_enumerated_subtypes() 246 | Returns whether this struct enumerates its subtypes. 247 | 248 | is_catch_all() 249 | Indicates whether this struct should be used in the event that none of its 250 | known enumerated subtypes match a received type tag. 251 | 252 | Raises an error if the struct doesn't enumerate subtypes. 253 | 254 | is_member_of_enumerated_subtypes_tree() 255 | Returns true if this struct enumerates subtypes or if its parent does. 256 | Structs that are members of trees must be able to be serialized without 257 | their inherited fields. 258 | 259 | get_examples() 260 | Returns an `OrderedDict 261 | `_ 262 | mapping labels to ``Example`` objects. 263 | 264 | StructField 265 | ----------- 266 | 267 | name 268 | The name of the field. 269 | 270 | doc 271 | The documentation string for the field. 272 | 273 | data_type 274 | The DataType of the field. 275 | 276 | has_default 277 | Whether this field has a default if it is unset. 278 | 279 | default 280 | The default for this field. Errors if no default is defined. 281 | 282 | The Python type of the default depends on the data type of the field. The 283 | following table shows the mapping: 284 | 285 | ========================== ============ ============ 286 | Primitive Python 2.x Python 3.x 287 | ========================== ============ ============ 288 | Bytes str bytes 289 | Boolean bool bool 290 | Float{32,64} float float 291 | Int{32,64}, UInt{32,64} long int 292 | List list list 293 | String unicode str 294 | Timestamp str str 295 | ========================== ============ ============ 296 | 297 | If the data type of a field is a union, its default can be a `TagRef 298 | object <#union-tag-reference>`_. No defaults are supported for structs. 299 | 300 | Union 301 | ----- 302 | 303 | name 304 | The name of the union. 305 | 306 | namespace 307 | The namespace the struct was defined in. 308 | 309 | doc 310 | The documentation string for the union. 311 | 312 | fields 313 | A list of UnionField objects defined by this union. Does not include any 314 | inherited fields. 315 | 316 | all_fields 317 | A list of all UnionField objects that make up the union. Required fields 318 | come before optional fields. 319 | 320 | parent_type 321 | If it exists, it points to a DataType object (another union) that this 322 | union inherits from. 323 | 324 | catch_all_field 325 | A UnionField object representing the catch-all field. 326 | 327 | has_documented_type_or_fields(include_inherited_fields=False) 328 | Returns whether this type, or any of its fields, are documented. 329 | 330 | Use this when deciding whether to create a block of documentation for 331 | this type. 332 | 333 | has_documented_fields(include_inherited_fields=False) 334 | Returns whether at least one field is documented. 335 | 336 | get_examples() 337 | Returns an `OrderedDict 338 | `_ 339 | mapping labels to ``Example`` objects. 340 | 341 | UnionField 342 | ---------- 343 | 344 | name 345 | The name of the field. 346 | 347 | doc 348 | The documentation string for the field. 349 | 350 | data_type 351 | The DataType of the field. 352 | 353 | catch_all 354 | A boolean indicating whether this field is the catch-all for the union. 355 | 356 | Alias 357 | ----- 358 | 359 | name 360 | The target name. 361 | 362 | data_type 363 | The DataType referenced by the alias as the source. 364 | 365 | doc 366 | The documentation string for the alias. 367 | 368 | Example 369 | ------- 370 | 371 | label 372 | The label for the example defined in the spec. 373 | 374 | text 375 | A textual description of the example that follows the label in the spec. 376 | Is ``None`` if no text was provided. 377 | 378 | example 379 | A JSON representation of the example that is generated based on the example 380 | defined in the spec. 381 | 382 | .. _emit_methods: 383 | 384 | Emit*() Methods 385 | =============== 386 | 387 | There are several ``emit*()`` methods included in a ``CodeGenerator`` that each 388 | serve a different purpose. 389 | 390 | ``emit(s='')`` 391 | Adds indentation, then the input string, and lastly a newline to the output 392 | buffer. If ``s`` is an empty string (default) then an empty line is created 393 | with no indentation. 394 | 395 | ``emit_wrapped_text(s, prefix='', initial_prefix='', subsequent_prefix='', width=80, break_long_words=False, break_on_hyphens=False)`` 396 | Adds the input string to the output buffer with indentation and wrapping. 397 | The wrapping is performed by the ``textwrap.fill`` Python library 398 | function. 399 | 400 | ``prefix`` is prepended to every line of the wrapped string. 401 | ``initial_prefix`` is prepended to the first line of the wrapped string 402 | ``subsequent_prefix`` is prepended to every line after the first. 403 | On a line, ``prefix`` will always come before ``initial_prefix`` and 404 | ``subsequent_prefix``. ``width`` is the target width of each line including 405 | indentation and prefixes. 406 | 407 | If true, ``break_long_words`` breaks words longer than width. If false, 408 | those words will not be broken, and some lines might be longer 409 | than width. If true, ``break_on_hyphens`` allows breaking hyphenated words; 410 | wrapping will occur preferably on whitespaces and right after the hyphen 411 | in compound words. 412 | 413 | ``emit_raw(s)`` 414 | Adds the input string to the output buffer. The string must end in a 415 | newline. It may contain any number of newline characters. No indentation is 416 | generated. 417 | 418 | Indentation 419 | =========== 420 | 421 | The ``babelapi.generator.CodeGenerator`` class provides a context 422 | manager for adding incremental indentation. Here's an example:: 423 | 424 | from babelapi.generator import CodeGenerator 425 | 426 | class ExampleGenerator(CodeGenerator): 427 | def generate(self, api): 428 | with self.output_to_relative_path('ex_indent.out'): 429 | with self.indent() 430 | self.emit('hello') 431 | self._output_world() 432 | def _output_world(self): 433 | with self.indent(): 434 | self.emit('world') 435 | 436 | The contents of ``ex_indent.out`` is:: 437 | 438 | hello 439 | world 440 | 441 | Indentation is always four spaces. We plan to make this customizable in the 442 | future. 443 | 444 | Helpers for Code Generation 445 | =========================== 446 | 447 | ``generate_multiline_list(items, before='', after='', delim=('(', ')'), compact=True, sep=',', skip_last_sep=False)`` 448 | Given a list of items, emits one item per line. This is convenient for 449 | function prototypes and invocations, as well as for instantiating arrays, 450 | sets, and maps in some languages. 451 | 452 | ``items`` is the list of strings that make up the list. ``before`` is the 453 | string that comes before the list of items. ``after`` is the string that 454 | follows the list of items. The first element of ``delim`` is added 455 | immediately following ``before``, and the second element is added 456 | prior to ``after``. 457 | 458 | If ``compact`` is true, the enclosing parentheses are on the same lines as 459 | the first and last list item. 460 | 461 | ``sep`` is the string that follows each list item when compact is true. If 462 | compact is false, the separator is omitted for the last item. 463 | ``skip_last_sep`` indicates whether the last line should have a trailing 464 | separator. This parameter only applies when ``compact`` is false. 465 | 466 | ``block(before='', after='', delim=('{','}'), dent=None, allman=False)`` 467 | A context manager that emits configurable lines before and after an 468 | indented block of text. This is convenient for class and function 469 | definitions in some languages. 470 | 471 | ``before`` is the string to be output in the first line which is not 472 | indented. ``after`` is the string to be output in the last line which is 473 | also not indented. The first element of ``delim`` is added immediately 474 | following ``before`` and a space. The second element is added prior to a 475 | space and then ``after``. ``dent`` is the amount to indent the block. If 476 | none, the default indentation increment is used. ``allman`` indicates 477 | whether to use ``Allman`` style indentation instead of the default ``K&R`` 478 | style. For more about indent styles see `Wikipedia 479 | `_. 480 | 481 | ``process_doc(doc, handler)`` 482 | Helper for parsing documentation `references `_ in 483 | Babel docstrings and replacing them with more suitable annotations for the 484 | target language. 485 | 486 | ``doc`` is the docstring to scan for references. ``handler`` is a function 487 | you define with the following signature: `(tag: str, value: str) -> str`. 488 | ``handler`` will be called for every reference found in the docstring with 489 | the tag and value parsed for you. The returned string will be substituted 490 | in the docstring for the reference. 491 | 492 | Generator Instance Variables 493 | ============================ 494 | 495 | logger 496 | This is an instance of the `logging.Logger 497 | `_ class 498 | from the Python standard library. Messages written to the logger will be 499 | output to standard error as the generator runs. 500 | 501 | target_folder_path 502 | The path to the output folder. Use this when the 503 | ``output_to_relative_path`` method is insufficient for your purposes. 504 | 505 | Data Type Classification Helpers 506 | ================================ 507 | 508 | ``babelapi.data_type`` includes functions for classifying data types. These are 509 | useful when generators need to discriminate between types. The following are 510 | available:: 511 | 512 | is_binary_type(data_type) 513 | is_boolean_type(data_type) 514 | is_composite_type(data_type) 515 | is_integer_type(data_type) 516 | is_float_type(data_type) 517 | is_list_type(data_type) 518 | is_nullable_type(data_type) 519 | is_numeric_type(data_type) 520 | is_primitive_type(data_type) 521 | is_string_type(data_type) 522 | is_struct_type(data_type) 523 | is_timestamp_type(data_type) 524 | is_union_type(data_type) 525 | is_user_defined_type(data_type) 526 | is_void_type(data_type) 527 | 528 | There is also an ``unwrap_nullable(data_type)`` function that takes a 529 | ``Nullable`` object and returns the type that it wraps. If the argument is not 530 | a ``Nullable``, then it's returned unmodified. Similarly, 531 | ``unwrap_aliases(data_type)`` takes an ``Alias`` object and returns the type 532 | that it wraps. There might be multiple levels of aliases wrapping the type. 533 | 534 | The ``unwrap(data_type)`` function will return the underlying type once all 535 | wrapping ``Nullable`` and ``Alias`` objects have been removed. Note that an 536 | ``Alias`` can wrap a ``Nullable`` and a ``Nullable`` can wrap an ``Alias``. 537 | 538 | Union Tag Reference 539 | =================== 540 | 541 | Tag references can occur in two instances. First, as the default of a struct 542 | field with a union data type. Second, as the value of a route attribute. 543 | References are limited to members with void type. 544 | 545 | TagRef 546 | ------ 547 | 548 | union_data_type 549 | The Union object that is the data type of the field. 550 | 551 | tag_name 552 | The name of the union member with void type that is the field default. 553 | 554 | To check for a default value that is a ``TagRef``, use ``is_tag_ref(val)`` 555 | which can be imported from ``babelapi.data_type``. 556 | 557 | Command-Line Arguments 558 | ====================== 559 | 560 | Generators can receive arguments from the command-line. A ``--`` is used to 561 | separate arguments to the ``babelapi`` program and the generator. For example:: 562 | 563 | $ babelapi generator/python/python.babelg spec.babel . -- -h 564 | usage: python-generator [-h] [-r ROUTE_METHOD] 565 | 566 | optional arguments: 567 | -h, --help show this help message and exit 568 | -r ROUTE_METHOD, --route-method ROUTE_METHOD 569 | A string used to construct the location of a Python 570 | method for a given route; use {ns} as a placeholder 571 | for namespace name and {route} for the route name. 572 | This is used to translate Babel doc references to 573 | routes to references in Python docstrings. 574 | 575 | The above prints the help string specific to the included Python generator. 576 | 577 | Command-line parsing relies on Python's `argparse module 578 | `_ so familiarity with it 579 | is helpful. 580 | 581 | To define a command-line parser for a generator, assign an `Argument Parser 582 | `_ 583 | object to the ``cmdline_parser`` class variable of your generator. Set the 584 | ``prog`` keyword to the name of your generator, otherwise, the help string 585 | will claim to be for ``babelapi``. 586 | 587 | The ``generate`` method will have access to an ``args`` instance variable with 588 | an `argparse.Namespace object 589 | `_ 590 | holding the parsed command-line arguments. 591 | 592 | Here's a minimal example:: 593 | 594 | import argparse 595 | from babelapi.generator import CodeGenerator 596 | 597 | _cmdline_parser = argparse.ArgumentParser(prog='example') 598 | _cmdline_parser.add_argument('-v', '--verbose', action='store_true', 599 | help='Prints to stdout.') 600 | 601 | class ExampleGenerator(CodeGenerator): 602 | 603 | cmdline_parser = _cmdline_parser 604 | 605 | def generate(self, api): 606 | if self.args.verbose: 607 | print 'Running in verbose mode' 608 | 609 | Examples 610 | ======== 611 | 612 | The following examples can all be found in the ``babelapi/example/generator`` 613 | folder. 614 | 615 | Example 1: List All Namespaces 616 | ------------------------------ 617 | 618 | We'll create a generator ``ex1.babelg.py`` that generates a file called 619 | ``ex1.out``. Each line in the file will be the name of a defined namespace:: 620 | 621 | from babelapi.generator import CodeGenerator 622 | 623 | class ExampleGenerator(CodeGenerator): 624 | def generate(self, api): 625 | """Generates a file that lists each namespace.""" 626 | with self.output_to_relative_path('ex1.out'): 627 | for namespace in api.namespaces.values(): 628 | self.emit(namespace.name) 629 | 630 | We use ``output_to_relative_path()`` a member of ``CodeGenerator`` to specify 631 | where the output of our ``emit*()`` calls go (See more emit_methods_). 632 | 633 | Run the generator from the root of the BabelAPI folder using the example specs 634 | we've provided:: 635 | 636 | $ babelapi example/generator/ex1/ex1.babelg.py example/api/dbx-core/*.babel output/ex1 637 | 638 | Now examine the contents of the output:: 639 | 640 | $ cat example/generator/ex1/ex1.out 641 | files 642 | users 643 | 644 | Example 2: A Python module for each Namespace 645 | --------------------------------------------- 646 | 647 | Now we'll create a Python module for each namespace. Each module will define 648 | a ``noop()`` function:: 649 | 650 | from babelapi.generator import CodeGenerator 651 | 652 | class ExamplePythonGenerator(CodeGenerator): 653 | def generate(self, api): 654 | """Generates a module for each namespace.""" 655 | for namespace in api.namespaces.values(): 656 | # One module per namespace is created. The module takes the name 657 | # of the namespace. 658 | with self.output_to_relative_path('{}.py'.format(namespace.name)): 659 | self._generate_namespace_module(namespace) 660 | 661 | def _generate_namespace_module(self, namespace): 662 | self.emit('def noop():') 663 | with self.indent(): 664 | self.emit('pass') 665 | 666 | Note how we used the ``self.indent()`` context manager to increase the 667 | indentation level by a default 4 spaces. If you want to use tabs instead, 668 | set the ``tabs_for_indents`` class variable of your extended CodeGenerator 669 | class to ``True``. 670 | 671 | Run the generator from the root of the BabelAPI folder using the example specs 672 | we've provided:: 673 | 674 | $ babelapi example/generator/ex2/ex2.babelg.py example/api/dbx-core/*.babel output/ex2 675 | 676 | Now examine the contents of the output:: 677 | 678 | $ cat output/ex2/files.py 679 | def noop(): 680 | pass 681 | $ cat output/ex2/users.py 682 | def noop(): 683 | pass 684 | 685 | Example 3: Define Python Classes for Structs 686 | -------------------------------------------- 687 | 688 | As a more advanced example, we'll define a generator that makes a Python class 689 | for each struct in our specification. We'll extend from 690 | ``MonolingualCodeGenerator``, which enforces that a ``lang`` class variable is 691 | declared:: 692 | 693 | from babelapi.data_type import is_struct_type 694 | from babelapi.generator import CodeGeneratorMonolingual 695 | from babelapi.lang.python import PythonTargetLanguage 696 | 697 | class ExamplePythonGenerator(CodeGeneratorMonolingual): 698 | 699 | # PythonTargetLanguage has helper methods for formatting class, obj 700 | # and variable names (some languages use underscores to separate words, 701 | # others use camelcase). 702 | lang = PythonTargetLanguage() 703 | 704 | def generate(self, api): 705 | """Generates a module for each namespace.""" 706 | for namespace in api.namespaces.values(): 707 | # One module per namespace is created. The module takes the name 708 | # of the namespace. 709 | with self.output_to_relative_path('{}.py'.format(namespace.name)): 710 | self._generate_namespace_module(namespace) 711 | 712 | def _generate_namespace_module(self, namespace): 713 | for data_type in namespace.linearize_data_types(): 714 | if not is_struct_type(data_type): 715 | # Only handle user-defined structs (avoid unions and primitives) 716 | continue 717 | 718 | # Define a class for each struct 719 | class_def = 'class {}(object):'.format(self.lang.format_class(data_type.name)) 720 | self.emit(class_def) 721 | 722 | with self.indent(): 723 | if data_type.doc: 724 | self.emit('"""') 725 | self.emit_wrapped_text(data_type.doc) 726 | self.emit('"""') 727 | 728 | self.emit() 729 | 730 | # Define constructor to take each field 731 | args = ['self'] 732 | for field in data_type.fields: 733 | args.append(self.lang.format_variable(field.name)) 734 | self.generate_multiline_list(args, 'def __init__', ':') 735 | 736 | with self.indent(): 737 | if data_type.fields: 738 | self.emit() 739 | # Body of init should assign all init vars 740 | for field in data_type.fields: 741 | if field.doc: 742 | self.emit_wrapped_text(field.doc, '# ', '# ') 743 | member_name = self.lang.format_variable(field.name) 744 | self.emit('self.{0} = {0}'.format(member_name)) 745 | else: 746 | self.emit('pass') 747 | self.emit() 748 | --------------------------------------------------------------------------------