├── README.md ├── draft └── logging-with-objects.md ├── neo ├── custom-ucd-inspector.png ├── neo-csv-paper.md ├── neo-json-paper.md ├── neo-unicode.md ├── stamp.md └── ztimestamp.md └── zinc ├── zinc-sso-paper.md └── zinc-websockets-paper.md /README.md: -------------------------------------------------------------------------------- 1 | # Documentation 2 | 3 | This repository holds various documentation files in Markdown format. 4 | 5 | 6 | -------------------------------------------------------------------------------- /draft/logging-with-objects.md: -------------------------------------------------------------------------------- 1 | # Logging With Objects 2 | 3 | *Sven Van Caekenberghe* June 9, 2014 4 | 5 | Logging, generating some form of output during the execution of a computer program, for monitoring or for later study, is useful everywhere and is thus a prime candidate for a framework. It seems however that it is hard to reach a consensus in this area, each developer having their own reasons for not liking some existing framework and/or preferring their own solution. Maybe stripping logging down to its bare essentials is a solution. 6 | 7 | Most existing logging frameworks are based on strings being the central thing being produced as log output by a running program. They differ in what extra attributes they add, like tags, severity levels, and so on. They also enforce a particular way the logging is actually done, thus creating an intrusive dependency. 8 | 9 | This document describes the design of the 'Logging With Objects' micro framework. 10 | 11 | ## Logging Objects 12 | 13 | What if we allowed and even encouraged **any object** to be a thing that can be logged ? This would allow each developer to chose the exact information they want to log. The framework could provide some useful super classes to inherit from, but should not require them. 14 | 15 | Although any object can be used as log object, there are some semantics that should be respected: 16 | 17 | - log objects should be **static or constant**, so that they or their sub objects do not change after they have been send off to the log 18 | - log objects with their sub objects should consist of **a single, separate graph** that can serialised, the single graph definition depending on the serialisation framework being used 19 | - log objects should have **a main string representation** suitable for classic textual logging 20 | - there should be some **communality among all log object types** to make later processing or analysis possible, inheritance, traits or shared protocols can be used for this 21 | - it might be useful to be able to sort log objects in the order in which they were generated 22 | - it might be useful to be able to uniquely identify log objects, even globally 23 | - it might be useful to support equality and hashing 24 | 25 | Conceptually, it is best to think of log objects as representing *something that has happened or that is about to happen*. Hence, the term log event can be used as well. Here are some examples: 26 | 27 | - UserLoggedIn 28 | - UserLoggedOut 29 | - WebPageRendered 30 | - RestCallCompleted 31 | - PrintJobScheduled 32 | - PrintJobCompleted 33 | - ServerErrorOccurred 34 | - AboutToSaveToFile 35 | - DidSaveToFile 36 | 37 | Obviously, some namespace prefixes have to be added. Maybe an Event, LogEvent or Signal suffix is useful as well. 38 | 39 | Note how these names automatically suggest useful attributes to add: username, time to execute, web page or REST call URI, filename. 40 | 41 | Here are some attributes that are quite common, though none of them should be required: 42 | 43 | - timestamp 44 | - ID, UUID, GUID 45 | - process/thread ID 46 | - username 47 | - session ID 48 | - package/application tag 49 | - server ID 50 | - severity level or classification 51 | 52 | The value types of these attributes can be important as well, but a more loose definition gives more flexibility. 53 | 54 | Note how in most cases, in a well structured object oriented program, *you will already have objects that can become useful parts of log objects*: 55 | 56 | - request/response 57 | - call/command 58 | - action 59 | - announcement 60 | - event 61 | 62 | Performance is an issue that can be solved by reusing objects that are already in the flow of computation. Remember to make sure that they remain constant. If critical, a local flag could prevent log object creation altogether. 63 | 64 | The key idea is to *include clean, structured information* that can then be useful when looking at the log events. This structured information is precisely what is almost completely lost, or very hard to recover, when logging just strings. 65 | 66 | The framework should provide in a couple of base classes for quickly building your own log event objects. But it would be even better to define a couple of traits or protocols describing useful attribute sets for log events. 67 | 68 | ## Logging 69 | 70 | The single thing that a running program should do is **create log objects and send them off to some central destination** that collects them in the order they arrive. 71 | 72 | The framework should not directly define how log objects are send off, nor how to find this central destination. The scope of being a central destination could be per application, image, server, or even across servers. Smaller scopes could be aggregated into large ones. 73 | 74 | The only requirement is that log objects are created and added to a log, in order. The addition can be triggered by a #log, #emit or #signal message. It is up to the application developer to add convenience methods to make this easy. Automatic fill in of as much attributes as possible, based on context, will make the actual logging code lighter. 75 | 76 | ## Processing Logs 77 | 78 | What happens with logs should be totally separate from the way the logs are generated. Obviously, the details of the processing are linked to the attributes of the log objects. 79 | 80 | Since we insist on logging objects, the main goal is to keep them somehow in their natural state. An obvious approach is to keep the last N log objects in memory. Another approach is to serialise them as such to external persistent storage. FUEL, STON or JSON, but also any database, come to mind. 81 | 82 | **Processing logs thus becomes the same as working with any collection of more or less uniform objects**. The standard collection API and tools can be used. 83 | 84 | Any framework functionality offered here will add requirements to log objects, in terms of the attributes that they should provide. Traits are ideal to define these requirements. 85 | 86 | ## Announcements 87 | 88 | The announcement framework (another micro framework) can be used to distribute log events from the producers to the actual consumers. Announcements are a mechanism, not a requirement as such. 89 | 90 | ## Notifications 91 | 92 | Another possible mechanism for log event producing code to send off objects is to use notifications, a subclass of exception. To collect them, a standard exception handler could then be used. Notifications are a mechanism, not a requirement as such. 93 | 94 | ## Conclusion 95 | 96 | Adding logging to your application or framework according to the idea of 'Logging With Objects' should help in removing the lock in anxiety of such a wide cross cutting concern. You can define your own log objects (maybe conforming to a couple of minimal protocols) and select your own mechanism to send the off to the log. Just connect your log object stream to the framework's post processing tools and your are done. 97 | 98 | To make it easier and/or to serve as examples, you can reuse some log object base classes or some of the transfer mechanisms offered. -------------------------------------------------------------------------------- /neo/custom-ucd-inspector.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/svenvc/docs/a3caf5650d8ca64337b888b0d3c275d650f7e2c4/neo/custom-ucd-inspector.png -------------------------------------------------------------------------------- /neo/neo-csv-paper.md: -------------------------------------------------------------------------------- 1 | # CSV 2 | 3 | *Sven Van Caekenberghe* 4 | 5 | *June 2012* 6 | 7 | *(This is a draft)* 8 | 9 | 10 | CSV (Comma Separated Values) is a popular data-interchange format. 11 | NeoCSV is an elegant and efficient standalone Smalltalk framework to 12 | read and write CSV converting to or from Smalltalk objects. 13 | 14 | 15 | ## An introduction to CSV 16 | 17 | 18 | CSV is a lightweight text-based de facto standard for human-readable tabular data interchange. 19 | Essentially, the key characteristics are that CSV (or more generally, delimiter separated text data): 20 | 21 | - is text based (ASCII, Latin1, Unicode) 22 | - consists of records, 1 per line (any line ending convention) 23 | - where records consist of fields separated by a delimiter (comma, tab, semicolon) 24 | - where every record has the same number of fields 25 | - where fields can be quoted should they contain separators or line endings 26 | 27 | Here are some relevant links: 28 | 29 | - 30 | - 31 | 32 | Note that there is not one single offical standard specification. 33 | 34 | 35 | ## NeoCSV 36 | 37 | 38 | The NeoCSV framework contains a reader (NeoCSVReader) and a writer (NeoCSVWriter) 39 | to parse respectively generate delimiter separated text data to or from Smalltalk objects. 40 | The goals of this project are: 41 | 42 | - to be standalone (have no dependencies and have little requirements) 43 | - to be small, elegant and understandable 44 | - to be efficient (both in time and space) 45 | - to be flexible and non-intrusive 46 | 47 | To use either the reader or the writer, 48 | you instanciate them on a character stream and use standard stream access messages. 49 | Here are two examples: 50 | 51 | (NeoCSVReader on: '1,2,3\4,5,6\7,8,9' withCRs readStream) upToEnd. 52 | 53 | String streamContents: [ :stream | 54 | (NeoCSVWriter on: stream) 55 | nextPutAll: #( (x y z) (10 20 30) (40 50 60) (70 80 90) ) ]. 56 | 57 | 58 | ## Generic Mode 59 | 60 | 61 | NeoCSV can operate in generic mode without any further customization. 62 | While writing, 63 | 64 | - record objects should respond to the #do: protocol 65 | - fields are always send #asString and quoted 66 | - CRLF line ending is used 67 | 68 | While reading, 69 | 70 | - records become arrays 71 | - fields remain strings 72 | - any line ending is accepted 73 | - both quoted and unquoted fields are allowed 74 | 75 | The standard delimiter is a comma character. 76 | Quoting is always done using a double quote character. 77 | A double quote character inside a field will be escaped by repeating it. 78 | Field separators and line endings are allowed inside a quoted field. 79 | All whitespace is significant. 80 | 81 | 82 | ## Customizing NeoCSVWriter 83 | 84 | 85 | Any character can be used as field separator, for example: 86 | 87 | neoCSVWriter separator: Character tab 88 | 89 | or 90 | 91 | neoCSVWriter separator: $; 92 | 93 | Likewise, any of the 3 common line end conventions can be set: 94 | 95 | neoCSVWriter lineEndConvention: #cr 96 | 97 | There are 3 ways a field can be written (in increasing order of efficiency): 98 | 99 | - quoted - converting it with #asString and quoting it (the default) 100 | - raw - converting it with #asString but not quoting it 101 | - object - not quoting it and using #printOn: directly on the output stream 102 | 103 | Obviously, when disabling qouting, you have to be sure your values do not contain embedded separators or line endings. 104 | If you are writing arrays of numbers for example, this would be the fastest way to do it: 105 | 106 | neoCSVWriter 107 | fieldWriter: #object; 108 | nextPutAll: #( (100 200 300) (400 500 600) (700 800 900) ) 109 | 110 | The fieldWriter option applies to all fields. 111 | 112 | If your data is in the form of regular domain level objects it would be wasteful 113 | to convert them to arrays just for writing them as CSV. 114 | NeoCSV has a non-intrusive option to map your domain object's fields. 115 | You add field specifications based on accessors. 116 | This is how you would write an array of Points. 117 | 118 | neoCSVWriter 119 | nextPut: #(x y); 120 | addFields: #(x y); 121 | nextPutAll: { 1@2. 3@4. 5@6 } 122 | 123 | Note how we first write the header (before customizing the writer). 124 | The addField: and addFields: methods arrange for the specified accessors 125 | to be performed on the incoming objects to produce values that will be written by the fieldWriter. 126 | Additionally, there is protocol to specify different field writing behavior per field, 127 | using addQuotedField:, addRawField: and addObjectField:. 128 | To specify different field writers for an array (actually an SequenceableCollection subclass), 129 | you can use the methods first, second, third, ... as accessors. 130 | 131 | 132 | ## Customizing NeoCSVReader 133 | 134 | 135 | The parser is flexible and forgiving. 136 | Any line ending will do, quoted and non-quoted fields are allowed. 137 | 138 | Any character can be used as field separator, for example: 139 | 140 | neoCSVReader separator: Character tab 141 | 142 | or 143 | 144 | neoCSVReader separator: $; 145 | 146 | NeoCSVReader will produce records that are instances of its recordClass, which defaults to Array. 147 | All fields are always read as Strings. 148 | If you want, you can specify converters for each field, to convert them to integers or floats, 149 | any other object. Here is an example: 150 | 151 | neoCSVReader 152 | addIntegerField; 153 | addFloatField; 154 | addField; 155 | addFieldConverter: [ :string | Date fromString: string ]; 156 | upToEnd. 157 | 158 | Here we specify 4 fields: an integer, a float, a string and a date field. 159 | Field convertions specified this way only work on indexable record classes, like Array. 160 | 161 | In many cases you will probably want your data to be returned as one of your domain objects. 162 | It would be wasteful to first create arrays and then convert all those. 163 | NeoCSV has non-intrusive options to create instances of your own object classes and 164 | to convert and set fields on them directly. 165 | This is done by specifying accessors and converters. 166 | Here is an example for reading Associations of Floats. 167 | 168 | (NeoCSVReader on: '1.5,2.2\4.5,6\7.8,9.1' withCRs readStream) 169 | recordClass: Association; 170 | addFloatField: #key: ; 171 | addFloatField: #value: ; 172 | upToEnd. 173 | 174 | For each field you give the mutator accessor to use, as well as an implicit or explicit conversion block. 175 | 176 | One thing that it will enforce is that all records have an equal number of fields. 177 | When there are no field accessors or converters, the field count will be set automatically after the first record is read. 178 | If you want you could set it upfront. 179 | When there are field accessors or converters, the field count will be set to the number of specified fields. 180 | -------------------------------------------------------------------------------- /neo/neo-json-paper.md: -------------------------------------------------------------------------------- 1 | # JSON 2 | 3 | *Sven Van Caekenberghe* 4 | 5 | *June 2012* 6 | 7 | *(This is a draft)* 8 | 9 | 10 | JSON (JavaScript Object Notation) is a popular data-interchange format. 11 | NeoJSON is an elegant and efficient standalone Smalltalk framework to 12 | read and write JSON converting to or from Smalltalk objects. 13 | 14 | 15 | ## An introduction to JSON 16 | 17 | 18 | JSON is a lightweight text-based open standard designed for human-readable data interchange. 19 | It was derived from the JavaScript scripting language for representing simple data structures and associative arrays, called objects. 20 | Despite its relationship to JavaScript, it is language-independent, with parsers available for many languages. 21 | 22 | Here are some relevant links: 23 | 24 | - 25 | - 26 | - 27 | 28 | There are only a couple of primitive types in JSON: 29 | 30 | - numbers (integer or floating point) 31 | - strings 32 | - the boolean constants true and false 33 | - null 34 | 35 | Only two composite types exist: 36 | 37 | - lists (an ordered sequenece of values) 38 | - maps (an unordered associative array, mapping string property names to values) 39 | 40 | That is really all there is to it. No options or additions are defined in the standard. 41 | 42 | 43 | ## NeoJSON 44 | 45 | 46 | The NeoJSON framework contains a reader (NeoJSONReader) and a writer (NeoJSONWriter) 47 | to parse respectively generate JSON to or from Smalltalk objects. 48 | The goals of this project are: 49 | 50 | - to be standalone (have no dependencies and have little requirements) 51 | - to be small, elegant and understandable 52 | - to be efficient (both in time and space) 53 | - to be flexible and non-intrusive 54 | 55 | Compared to other Smalltalk JSON frameworks, NeoJSON has 56 | 57 | - less dependencies and little requirements 58 | - can be more efficient (be faster and use less memory) 59 | - allows for the use of schemas and mappings 60 | 61 | 62 | ## Primitives 63 | 64 | Obviously, the primitive types are mapped to corresponding Smalltalk classes. 65 | While reading: 66 | 67 | - numbers become Integers or Floats 68 | - strings become Strings 69 | - booleans become Booleans 70 | - null become nil 71 | 72 | While writing 73 | 74 | - Numbers are converted to floats, except for Integers that become integers 75 | - Strings and subclasses become strings 76 | - Booleans become booleans 77 | - nil becomes null 78 | 79 | 80 | ## Generic Mode 81 | 82 | 83 | NeoJSON can operate in a generic mode that requires no further configuration. 84 | While reading: 85 | 86 | - maps become instances of mapClass, Dictionary by default 87 | - lists become instance of listClass, Array by default 88 | 89 | The reader can be customized to use a different mapClass or listClass. 90 | There is also an option to convert all map keys to symbols, which is off by default. 91 | While writing: 92 | 93 | - Dictionary and SmallDictionary become maps 94 | - all other Collection classes become lists 95 | - all other Objects are rejected 96 | 97 | Here are some examples writing in generic mode: 98 | 99 | NeoJSONWriter toString: #(1 2 3). 100 | 101 | NeoJSONWriter toString: { Float pi. true. false. 'string' }. 102 | 103 | NeoJSONWriter toStringPretty: (Dictionary new at: #x put: 1; at: #y put: 2; yourself). 104 | 105 | NeoJSONWriter can output either in a compact format (the default) or in a pretty printed format. 106 | And these are some examples reading in generic mode: 107 | 108 | NeoJSONReader fromString: ' [ 1,2,3 ] '. 109 | 110 | NeoJSONReader fromString: ' [ 3.14159, true, false, null, "string" ] '. 111 | 112 | NeoJSONReader fromString: ' { "x" : 1, "y" : 2 } '. 113 | 114 | 115 | In order to use the generic mode, you have to convert your domain objects to and from 116 | Dictionaries and SequenceableCollections. This is realatively easy but not very efficient, 117 | depending on the use case. 118 | 119 | 120 | ## Schemas and Mappings 121 | 122 | 123 | NeoJSON allows for the optional specification of schemas and mappings to be used 124 | when writing and/or when reading. 125 | A NeoJSONMapper holds a number of schemas. 126 | Each schema is identified by either a class or a symbol. 127 | Each schema specifies a mapping, an object that will help in doing the actual reading or writing. 128 | 129 | The most common mapping deals with objects that define a number of named properties or attributes. 130 | These can be defined based on instance variables (optionally derived by reflection), 131 | accessors (getter/setter pairs) or even blocks. 132 | Such an object mapping is identified by a Smalltalk class, which is also used to create new instances. 133 | Each property mapping can have an optional value schema to be used recursively 134 | when reading and/or writing property values. 135 | 136 | The less common custom mapping holds a generic reader and/or writer block to deal with 137 | special cases such as specific collection types with an optional schema for the elements, 138 | or a direct mapping of semi primitive types such as Date or DateAndTime. 139 | 140 | A mapping can be specified explicitely on a mapper, or can be resolved using the #neoJsonMapping: class method. 141 | 142 | Here are some examples of mappings: 143 | 144 | mapper mapAllInstVarsFor: Point. 145 | 146 | mapper for: TestObject do: [ :mapping | 147 | mapping mapInstVars: #(id name). 148 | (mapping mapInstVar: #timestamp to: 'created-at') valueSchema: DateAndTime. 149 | (mapping mapInstVar: #points) valueSchema: #ArrayOfPoints. 150 | (mapping mapInstVar: #bytes) valueSchema: ByteArray ]. 151 | 152 | mapper for: DateAndTime customDo: [ :mapping | 153 | mapping decoder: [ :string | DateAndTime fromString: string ]. 154 | mapping encoder: [ :dateAndTime | dateAndTime printString ] ]. 155 | 156 | mapper for: #ArrayOfPoints customDo: [ :mapping | 157 | mapping listOfElementSchema: Point ]. 158 | 159 | mapper for: #DictionaryOfPoints customDo: [ :mapping | 160 | mapping mapWithValueSchema: Point ]. 161 | 162 | mapper for: ByteArray customDo: [ :mapping | 163 | mapping listOfType: ByteArray ] 164 | 165 | The classes NeoJSONReader and NeoJSONWriter are subclasses of NeoJSONMapper. 166 | When writing, mappings are used when arbitrary objects are seen. 167 | For example, in order to be able to write an array of points, you could do as follows: 168 | 169 | String streamContents: [ :stream | 170 | (NeoJSONWriter on: stream) 171 | prettyPrint: true; 172 | mapInstVarsFor: Point; 173 | nextPut: (Array with: 1@3 with: -1@3) ]. 174 | 175 | Collections are handled automatically, like in the generic case. 176 | When reading, a mapping is used as a binding or an explicit type specifying what Smalltalk objects that you want to read. 177 | Here is a very simple case, reading a map as a point: 178 | 179 | (NeoJSONReader on: ' { "x" : 1, "y" : 2 } ' readStream) 180 | mapInstVarsFor: Point; 181 | nextAs: Point. 182 | 183 | Since JSON lacks a universal way to specify the class of an object/map, 184 | we have to specify the target schema that we want to use as an argument to #nextAs:. 185 | 186 | With custom mappings, it is possible to 187 | - define the schema of the elements of a list 188 | - define the schema of the elements of a list as well as the class of the list 189 | - define the schema of the values of a map 190 | In fact, NeoJSONCustomMapping can be extended to implement even more specialized mappings. 191 | 192 | Finally, here is a more complex example, reading a list of maps as an array of points: 193 | 194 | (NeoJSONReader on: '[ { "x" : 1, "y" : 2 }, { "x" : 3, "y" : 4 } ]' readStream) 195 | mapInstVarsFor: Point; 196 | for: #ArrayOfPoints customDo: [ :mapping | 197 | mapping listOfElementSchema: Point ]; 198 | nextAs: #ArrayOfPoints. 199 | 200 | NeoJSON deals efficiently with mappings: the minimal amount of intermediary structures are created, 201 | which is quite different from the generic case. 202 | 203 | 204 | ## Internals 205 | 206 | 207 | On modern hardware, NeoJSON can write or read in the tens of thousands of small objects per second. 208 | Several benchmarks are included in the unit tests package. 209 | -------------------------------------------------------------------------------- /neo/neo-unicode.md: -------------------------------------------------------------------------------- 1 | # Neo-Unicode 2 | 3 | Neo Unicode is a repository with experimental, proof of concept code, that aims to improve 4 | [Pharo](http://www.pharo.org)'s [Unicode](https://en.wikipedia.org/wiki/Unicode) support. 5 | This is a work in progress that is not yet suitable for public consumption. 6 | 7 | The project consists of 2 packages, Neo-Unicode-Core and Neo-Unicode-Tests 8 | in the http://mc.stfx.eu/Neo repository. There is no Metacello configuration. 9 | 10 | _Sven Van Caekenberghe (December 2015)_ 11 | 12 | 13 | ## The current situation 14 | 15 | In Pharo we have Unicode strings as a sequence of Unicode characters, 16 | each defined/identified by a code point (out of 10.000s covering all languages). 17 | 18 | To encode Unicode for external representation as bytes, we use UTF-8 like the rest of the modern world. 19 | 20 | You can read more about the current situation in the first part of the [Character Encoding and Resource Meta Description](http://files.pharo.org/books/enterprisepharo/book/Zinc-Encoding-Meta/Zinc-Encoding-Meta.html) chapter of the Enterprise Pharo book. 21 | 22 | All in all, the current situation is OK for day to day practical use. 23 | 24 | 25 | ## What is the problem ? What is missing ? 26 | 27 | The world is a complex place and the [Unicode](http://www.unicode.org) standard 28 | tries to cover all possible languages, situations, usages and interpretations, combined with backward compatibility. 29 | For 1000s of pages, all kinds of exceptions and special rules are described. 30 | Obviously we do not have enough support for all of this. 31 | 32 | We do not need all of Unicode, nor do we need to implement all edge cases, but we need more. 33 | In particular, we need to support normalization, casing and collation/sorting. 34 | 35 | 36 | ## The Unicode Character Database 37 | 38 | A first step for better Unicode support is to work with the actual data defined by the standard. 39 | The main element is the Unicode Character Database, which lists details for each of approximately 30.000 code points. 40 | 41 | **NeoUnicodeCharacterData** holds this information per code point. 42 | The database is loaded in Pharo from an external URL, on demand, and then cached. 43 | 44 | NeoUnicodeCharacterData database 45 | 46 | An extension method provides access to this data from a character. 47 | 48 | $é unicodeCharacterData. 49 | $e unicodeCharacterData. 50 | $1 unicodeCharacterData. 51 | 52 | A GT Inspector extension helps in interpreting this data, 53 | as it is stored in a way that favours compactness more than readability. 54 | 55 | ![Image](https://raw.githubusercontent.com/svenvc/docs/master/neo/custom-ucd-inspector.png) 56 | 57 | See the class and method comments, as well as the unit tests for more information. 58 | Obviously, correctly interpreting all these properties is very complex. 59 | 60 | 61 | ## Normalization 62 | 63 | Some characters or letters can be written in more than one way. 64 | 65 | The simplest example in a common language (the French letter é) is 66 | 67 | LATIN SMALL LETTER E WITH ACUTE [U+00E9] 68 | 69 | which can also be written as 70 | 71 | LATIN SMALL LETTER E [U+0065] followed by COMBINING ACUTE ACCENT [U+0301] 72 | 73 | The former being a composed normal form, the latter a decomposed normal form. 74 | 75 | Not only diacritical marks (accents) lead to two representations, various special characters might do too. 76 | For example, the ellipsis character of 3 dots can be considered, under a certain interpretation, 77 | to be equivalent to 3 separate dots in a row. 78 | 79 | HORIZONTAL ELLIPSIS [U+2026] 80 | FULL STOP [U+002E] followed by FULL STOP [U+002E] followed by FULL STOP [U+002E] 81 | 82 | Some characters have more than 2 representations. The Ångström symbol Å can be written in 3 ways ! 83 | 84 | ANGSTROM SIGN [U+212B] 85 | LATIN CAPITAL LETTER A WITH RING ABOVE [U+00C5] 86 | LATIN CAPITAL LETTER A [U+0041] followed by COMBINING RING ABOVE [U+030A] 87 | 88 | Normalization is needed to properly compare strings. 89 | There are different normalization forms, NFC, NFD, NFKC and NFKD, each with specific rules. 90 | 91 | **NeoUnicodeNormalizer** is a first and simplified implementation of this operation, 92 | using the Unicode Character Database. 93 | 94 | NeoUnicodeNormalizer new decomposeString: 'les élèves Français'. 95 | NeoUnicodeNormalizer new composeString: 'les e´le`ves Franc ̧is'. 96 | NeoUnicodeNormalizer new decomposeString: 'Düsseldorf Königsallee'. 97 | NeoUnicodeNormalizer new composeString: 'Du¨sseldorf Ko¨nigsallee'. 98 | 99 | (Note: copy/paste of the above decomposed strings won't work, 100 | generate the decomposed strings yourself in Pharo using #decomposeString:) 101 | 102 | 103 | ## Casing 104 | 105 | Recognizing lower, upper and title case and converting between them is defined by Unicode as well. 106 | The Unicode Character Database can be used for these operations. 107 | 108 | **NeoUnicodeCaser** is a simple tool to do these conversions, 109 | using the Unicode Character Database. 110 | 111 | NeoUnicodeCaser new case: #uppercase string: 'abc'. 112 | NeoUnicodeCaser new case: #lowercase string: 'ABC'. 113 | NeoUnicodeCaser new case: #titlecase string: 'abc'. 114 | 115 | 116 | ## Collation/Sorting 117 | 118 | Nothing yet. 119 | 120 | 121 | ## Legacy Notes 122 | 123 | The current **Unicode** class seems to be capable of reading an older version of UnicodeData.txt 124 | but uses it only for casing. 125 | 126 | The current **CombinedChar** class seems to be capable of reading an older version of UnicodeData.txt 127 | but uses it only combining composable characters. 128 | 129 | Neither holds on to the database as it is pretty big, but they do hold onto quite some data. 130 | This could probably be optimized in the future. 131 | 132 | 133 | ## References 134 | 135 | Some links to useful documents: 136 | - http://www.unicode.org/Public/UNIDATA/ 137 | - http://www.unicode.org/reports/tr15/ 138 | - http://www.unicode.org/versions/Unicode8.0.0/ch03.pdf (3.11) 139 | - ftp://ftp.unicode.org/Public/3.0-Update/UnicodeCharacterDatabase-3.0.0.html 140 | - http://unicode.org/faq/normalization.html 141 | - http://www.unicode.org/reports/tr44/ 142 | - http://blog.golang.org/strings 143 | - http://blog.golang.org/normalization 144 | - http://docs.oracle.com/javase/7/docs/api/java/text/Normalizer.html 145 | - https://developer.apple.com/swift/blog/?id=30 146 | - http://useyourloaf.com/blog/swift-string-cheat-sheet.html 147 | - https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swift_Programming_Language/StringsAndCharacters.html 148 | - https://www.mikeash.com/pyblog/friday-qa-2015-11-06-why-is-swifts-string-api-so-hard.html 149 | - http://graphemica.com/ 150 | - http://www.charbase.com 151 | - http://www.fileformat.info/info/unicode/index.htm 152 | - https://www.azabani.com/pages/gbu/ 153 | 154 | 155 | 156 | -------------------------------------------------------------------------------- /neo/stamp.md: -------------------------------------------------------------------------------- 1 | # Stamp 2 | 3 | *Sven Van Caekenberghe* 4 | 5 | *August 2013* 6 | 7 | *(Work in progress)* 8 | 9 | Stamp is an implementation of [STOMP (Simple (or Streaming) Text Oriented Message Protocol)](http://en.wikipedia.org/wiki/Streaming_Text_Oriented_Messaging_Protocol) for [Pharo](http://www.pharo.org), a protocol to interact with message-oriented middleware (MOM). 10 | 11 | More specifically, Stamp implements [STOMP 1.2](http://stomp.github.io/stomp-specification-1.2.html) and was tested against [RabbitMQ 3.1](http://www.rabbitmq.com). Other message-oriented middleware implementations accessible through STOMP include [Apache ActiveMQ](http://activemq.apache.org), [Glassfish Open MQ](http://mq.java.net) and [Fuse Message Broker based on Active MQ](http://fusesource.com/products/enterprise-activemq/) - but these have not yet been tested. 12 | 13 | [Messaging middleware](http://en.wikipedia.org/wiki/Message-oriented_middleware) is an important technology for building scaleable and flexible enterprise software architectures. 14 | 15 | ## Installation 16 | 17 | The MIT licensed [source code for Stamp](http://www.smalltalkhub.com/#!/~SvenVanCaekenberghe/Stamp) lives on [SmalltalkHub](http://www.smalltalkhub.com) and can be loaded using Metacello. 18 | 19 | Gofer it 20 | smalltalkhubUser: 'SvenVanCaekenberghe' project: 'Stamp'; 21 | configurationOf: 'Stamp'; 22 | loadVersion: #bleedingEdge. 23 | 24 | Unit tests and benchmarks are included. 25 | 26 | ## Usage 27 | 28 | To be able to use Stamp, you need a running [RabbitMQ](http://www.rabbitmq.com/download.html) instance with the [STOMP plugin](http://www.rabbitmq.com/stomp.html) enabled. It is helpful to also install the [management plugin](http://www.rabbitmq.com/management.html) which exports a web interface. 29 | 30 | Whether you are sending or receiving messages, whether you are a conceptual producer or consumer, you are always a client of the messaging middleware. The key object to use is thus called StampClient. 31 | 32 | To connect, you need to specify a host (defaults to localhost), port (defaults to 61613), login and passcode. 33 | 34 | | client | client := StampClient new. client login: 'guest'. client passcode: 'guest'. Use the RabbitMQ web management interface to configure the default virtual host, login and passcode (implicit login/passcode can be configured as well). There is a #debug: true option to enable Transcript logging of the wire protocol. 35 | 36 | Using #open and #close you actually connect and disconnect. 37 | 38 | Although STOMP resembles HTTP quite a bit, it is fundamentally different. HTTP is a synchronous request/response protocol, while STOMP is asynchronous without the concept of request/response pairs. 39 | 40 | The base concept in the STOMP wire protocol is called a frame, sent or received. Stamp has some convenience methods that hide frames, but they will show up. The implementation matches each frame type to a different class, as subclasses of StampFrame. 41 | 42 | There is a convenience method to send a plain text message, which means that you put it on a queue. 43 | 44 | client sendText: 'Hello World!' to: 'test-queue'. 45 | 46 | Another convenience method allows a client to subscribe or listen to a queue. 47 | 48 | client subscribeTo: 'test-queue'. 49 | 50 | Next, you will have to actually wait for incoming messages. 51 | 52 | client readMessage. 53 | 54 | Note that #readMessage will eventually time out. To listen in a convenient loop, there is a control structure helper. 55 | 56 | client runWith: [ :message | 57 | "Do something with message" ] 58 | 59 | The block will be invoked for each incoming message while ConnectionTimedOut will be ignored by looping. To exit the loop, you should signal ConnectedClosed. In any case, at the end the client will be closed. 60 | 61 | In general though, the convenience methods will not be enough because often extra parameters have to be set. For this, Stamp uses its frame objects. 62 | 63 | For example, sending a message uses a StampSendFrame. 64 | 65 | | sendFrame | 66 | sendFrame := StampSendFrame new. 67 | sendFrame destination: 'test-queue'. 68 | sendFrame text: 'Hello there'. 69 | sendFrame persistent: true. 70 | sendFrame replyTo: '/temp-queue/greetings'. 71 | client write: sendFrame. 72 | 73 | Similarly, subscribing is done with a StampSubscribeFrame. 74 | 75 | | subscribeFrame | 76 | subscribeFrame := StampSubscribeFrame new. 77 | subscribeFrame destination: 'test-queue'. 78 | subscribeFrame id: client nextId. 79 | subscribeFrame clientIndividualAck. 80 | client writeWithReceipt: subscribeFrame. 81 | 82 | And the messages that you receive are instances of StampMessageFrame with a body and contentType, contentLength and other meta data. 83 | 84 | The frame objects allow access to important meta data, needed to set the necessary semantics for proper message middleware usage. 85 | 86 | In particular, the current Stamp implementation supports the following: 87 | 88 | - ack and nack on received messages 89 | - auto, client and client individual acknowledgments 90 | - receipts when sending messages 91 | - transactions 92 | - persistent queues 93 | - durable subscriptions 94 | - temporary reply queues for RPC 95 | - AMPQ headers and queues 96 | - heartbeats 97 | - arbitrary content types and meta data 98 | 99 | Please refer to the unit tests for examples. In particular, #testSimpleRpc, #testSimpleRpcCounter and #testSimpleWorkQueue illustrate the main high level patterns. 100 | 101 | Stamp is strictly single threaded by design and thus has no locks to protect itself. A single loop can handle incoming messages, sending outgoing messages as necessary. 102 | 103 | Stamp itself is still evolving. Any feedback is welcome. 104 | ## Acknowledgement 105 | 106 | This project was inspired by Göran Krampe's older [StompProtocol](http://www.squeaksource.com/StompProtocol.html) implementation. After making a number changes and commits to this code base, I felt I was diverging so much and so fundamentally from the original implementation that I had to fork it. Stamp started as a toy project developed in private but gradually matured to a point where it is reasonably complete, stands on its own and would be useful for others. -------------------------------------------------------------------------------- /neo/ztimestamp.md: -------------------------------------------------------------------------------- 1 | # ZTimestamp 2 | 3 | I am ZTimestamp. 4 | I am a Magnitude. 5 | I represent a point in time, a combination of a date and a time. 6 | 7 | I am an alternative for DateAndTime and TimeStamp. 8 | I have second precision and live in the UTC/GMT/Zulu timezone. 9 | I use ISO/International conventions and protocols only. 10 | I support some essential arithmetic. 11 | 12 | I have an efficient internal representation: 13 | 14 | - jnd - the julian day number 15 | - secs - the number of seconds since midnight, the beginning of the day 16 | 17 | Examples: 18 | 19 | ZTimestamp now. 20 | ZTimestamp fromString: '1969-07-20T20:17:40Z'. 21 | 22 | I am somewhat compatible with existing, standard Chronology objects. 23 | I correctly parse representations with a timezone designator 24 | and can print a representation in arbitrary timezones. 25 | 26 | (ZTimestamp fromString: DateAndTime now truncated printString) localPrintString. 27 | 28 | # Where ? 29 | 30 | Use ConfigurationOfZTimestamp in http://mc.stfx.eu/Neo to load me. 31 | 32 | # Related 33 | 34 | I come with a number of related tools: 35 | ZTimestampFormat (example based date/time formatting and parsing), 36 | ZTimestampSNTPClient and ZTimestampPreciseSNTPClient (check clock synchronization). 37 | 38 | -------------------------------------------------------------------------------- /zinc/zinc-sso-paper.md: -------------------------------------------------------------------------------- 1 | # Zinc SSO 2 | 3 | *Jan van de Sandt & Sven Van Caekenberghe* 4 | 5 | *December 2012 - Updated Januari 2013* 6 | 7 | *(This is a draft)* 8 | 9 | Zinc SSO is a library/framework providing tools in the realm of authorization and authentication, 10 | more specifically offering implementations of client side OAuth & OpenID. 11 | 12 | OAuth allows (web) applications to access resources on behalf of resource owners (typically users). 13 | At the same time, OAuth allows resource owners to grant access to their resources 14 | without actually sharing their credentials (typically username/password). 15 | 16 | A common use case is to use OAuth to enable web applications to delegate 17 | authentication to (often well-known) external service providers. 18 | Your users will be happy because they can reuse their existing account, 19 | you will be happy because you have less to implement and worry about. 20 | 21 | Zinc SSO is an addon to [Zinc HTTP Components](http://zn.stfx.eu). 22 | 23 | *(Please note that Zinc-SSO is currently in development. This is beta code.)* 24 | 25 | # OAuth 26 | 27 | See also [http://en.wikipedia.org/wiki/Oauth](http://en.wikipedia.org/wiki/Oauth). 28 | 29 | ## Installation 30 | 31 | Zinc-SSO depends on [Zinc HTTP Components](http://zn.stfx.eu) and 32 | [Zodiac](http://zdc.stfx.eu) and uses [NeoJSON](http://stfx.eu/neojson) 33 | and XML Support as well. 34 | The easiest way to load the code is by using ConfigurationOfZincHTTPComponents. 35 | 36 | Gofer it 37 | url: 'http://mc.stfx.eu/ZincHTTPComponents'; 38 | package: 'ConfigurationOfZincHTTPComponents'; 39 | load. 40 | 41 | ConfigurationOfZincHTTPComponents project latestVersion load: 'SSO'. 42 | 43 | Note that since Zodiac is needed, you will need the SSL Plugin for your VM. 44 | The dependencies are easiest resolved in Pharo 2.0. 45 | 46 | For Pharo 8 use... 47 | ``` 48 | Metacello new 49 | repository: 'github://svenvc/zinc/repository'; 50 | baseline: 'ZincHTTPComponents'; 51 | load: 'SSO'. 52 | ``` 53 | 54 | ## Zinc-SSO OAuth Support 55 | 56 | With OAuth an application can ask a user permission to access the user's data on a third party system. 57 | For example a Seaside application can ask a user permission to read its data on Twitter. If an application 58 | only asks for permission to view the user profile data than you use OAuth just for single sign-on and user 59 | identification functionality. It is also possible to ask for additional authorizations, for example for 60 | permission to create new Tweets for a user. In this case you can do a lot more than just single sign-on. 61 | 62 | Currently there are two versions of OAuth in active use. Version 1.0a and version 2.0, see [wikipedia] 63 | (http://en.wikipedia.org/wiki/OAuth) 64 | for a list of OAuth service providers and the versions they support. The two versions offer the same 65 | kind of functionality but are technically very different. Version 2.0 is really simple to use and just 66 | relies on https for security. Version 1.0a is more complex and uses a digital signature to verify the 67 | integrity of a http request. 68 | 69 | # OAuth 1.0a Support 70 | 71 | A well known service provider that uses version 1.0a is Twitter. We will use Twitter as an example 72 | here but things will work the same for other service providers that support this version. 73 | 74 | Twitter documentation: https://dev.twitter.com/docs/auth/implementing-sign-twitter 75 | 76 | The first thing that you need to do is to register your application with Twitter. You need 77 | to provide a name and a description of your application and you must specify what kind of 78 | access you need. In the case of Twitter you can choose between readonly access or write 79 | access. For single sign-on readonly access is sufficient. Twitter will generate a consumer 80 | key and a consumer secret string. We need these two values in the requests we send to twitter. 81 | The key will be one of the parameters, the secret is used to create the digital signature for 82 | the requests and must never be used as a request parameter. 83 | 84 | In Zinc-SSO the consumer key and secret are stored in a ZnOAuth1ConsumerData instance. This 85 | object also holds the urls for the required API calls. The url's for Twitter are hardcoded 86 | in a class method: 87 | 88 | consumerData := ZnOAuth1ConsumerData newForTwitterAuthentication 89 | consumer: 'YgnhZtjas8ccdVdyZ1QGBA'; 90 | consumerSecret: '--the-secret--'; 91 | yourself. 92 | 93 | Now you can put a “Signin with Twitter” button or link in your web application. In the code 94 | that gets called you should perform the following steps: 95 | 96 | Step 1: Get a request token – Before we can redirect the user to the Twitter signin page 97 | we need to get a request token. An important parameter for the API call to get this token 98 | is the callbackUrl. This is the url that the user will be redirected back to after the signin. 99 | 100 | service := ZnOAuth1Service new 101 | providerAccount: consumerData ; 102 | yourself. 103 | requestToken := service getRequestTokenFor: 'http://my-domain/sso-callback'. 104 | 105 | The #getRequestTokenFor: will create a signed OAuth request which includes your consumer 106 | key and will send the request to Twitter. If all goes well Twitter will respond with a 107 | request token. The token consists of a value and a secret. We need to store this token 108 | in our session. Later when the user is redirected back to our app we need this token. 109 | 110 | Step 2: Redirect the user to the signin page of the service provider 111 | 112 | redirectUrl := service loginUrlFor: requestToken 113 | 114 | ZnOAuth1Service>>#loginUrl: will answer a ZnUrl object to which we should redirect the user. 115 | 116 | Step 3: Convert the request token into an access token 117 | 118 | In our code that gets called when the user is redirected back to us we need to make a second 119 | API call to convert the request token into an access token. If the user signed in successfully 120 | and allowed our application access the request should contain two parameters: 121 | oauth_token and oauth_verifier. The oauth_token should be equal to the value of the 122 | requestToken and the oauth_verifier is needed to get the access token: 123 | 124 | handleOAuth1Callback: request 125 | 126 | requestToken := self session at: 'oauth-req-token' 127 | ifAbsent: [ self error: 'Invalid callback – no request token' ]. 128 | 129 | oauthToken := request uri queryAt: 'oauth_token'. 130 | oauthVerifier := request uri queryAt: 'oauth_verifier'. 131 | 132 | (oauthToken isNil or: [ oauthVerifier isNil]) 133 | ifTrue: [ self error: 'Invalid request' ]. 134 | 135 | oauthToken = requestToken value 136 | ifFalse: [ self error: 'Invalid request' ]. 137 | 138 | accessToken := self twitterOAuth1Service getAccessToken: requestToken verifier: oauthVerifier. 139 | 140 | Now we have an accessToken the requestToken is no longer needed. We can use the accessToken to retrieve 141 | information about the user. We can do that immediately and forget about the accessToken or we can store 142 | this accessToken somewhere and do this at a later time. The time that an accessToken stays valid is 143 | service provider specific. 144 | 145 | The way to get information about the user is also service provider specific. Twitter recommends that 146 | you use the verify_credentials API call to get basic user information. The class 147 | ZnOAuth1TwitterUserAccess contains a method that wraps this Twitter API call. 148 | 149 | userData := ZnOAuth1TwitterUserAccess new 150 | oauth1Service: self twitterOAuth1Service ; 151 | accessToken: accessToken ; 152 | accountVerifyCredentials. 153 | 154 | The #accountVerifyCredentials method answers a Dictionary with information about the user. 155 | See the [Twitter documentation](https://dev.twitter.com/docs/api/1.1/get/account/verify_credentials) 156 | for the exact contents. 157 | 158 | 159 | ## OAuth Providers 160 | 161 | There are demos available for Google, Twitter, Microsoft and Facebook accounts. 162 | 163 | ## Demos 164 | 165 | Setting up OAuth and OpenID takes some work, which why two online demos are available: 166 | 167 | - [http://sso.stfx.eu] (http://sso.stfx.eu) 168 | - [http://sso.doit.st] (http://sso.doit.st) 169 | 170 | The first demo is running on top of Zinc only, 171 | the second demo uses Seaside for its user interface. 172 | Both use the same Zinc-SSO-OAuth-Core code. 173 | 174 | ## Getting Started 175 | 176 | In order to get started with your own setup, 177 | you will first have to register your 178 | application with the providers that you want to use. 179 | During that registration you have to specify the domain of your web application. 180 | 181 | After registration you will get two crucial elements of information: 182 | 183 | - a client id or key 184 | - a secret 185 | 186 | Combined with your callback URL/URI these are used as parameters: 187 | 188 | ZnOAuth2ConsumerData 189 | key: '531369976180189' 190 | secret: '16ag98834ec8925787a75a8141a98810' 191 | redirectUrl: 'http://my-app.my-domain.com/sso-google-callback' 192 | 193 | The ZnOAuth2ConsumerData is then used to parameterize a ZnOAuth2Session. 194 | The ZnOAuth2Session does two things: 195 | 196 | - generate the URL to request authentication from the service provider 197 | - handle the authentication callback from the service provider to your web app 198 | 199 | If all goes well, the callback will contain enough information, called user data, 200 | for you to accept that the user holds that account. 201 | You can trust that the service provider did its normal work to authenticate the login. 202 | 203 | *** 204 | 205 | Lots more to understand and explain ;-) 206 | -------------------------------------------------------------------------------- /zinc/zinc-websockets-paper.md: -------------------------------------------------------------------------------- 1 | # WebSockets 2 | 3 | *Sven Van Caekenberghe* 4 | 5 | *September 2012 - Updated January 2013* 6 | 7 | *(This is a draft)* 8 | 9 | 10 | The WebSocket protocol defines a full-duplex single socket connection 11 | over which messages can be sent between a client and a server. 12 | It simplifies much of the complexity around bi-directional web communication and connection management. 13 | WebSocket represents the next evolutionary step in web communication compared to Comet and Ajax. 14 | 15 | 16 | ## An Introduction to WebSockets 17 | 18 | 19 | HTTP, one of the main technologies of the internet, 20 | defines a communication protocol between a client and a server 21 | where the initiative of the communication lies with the client 22 | and each interaction consists of a client request and a server response. 23 | When correctly implemented and used, HTTP is enormously scaleable and very flexible. 24 | 25 | With the arrival of advanced Web applications mimicking regular 26 | desktop applications with rich user interfaces, as well as mobile Web applications, 27 | it became clear that HTTP was not suitable or not a great fit for two use cases: 28 | 29 | - when the server wants to take the initiative and send the client a message 30 | - when the client wants to send (many) (possibly asynchronous) short messages with little overhead 31 | 32 | In the HTTP protocol, the server cannot take the initiative to send a message, 33 | the only workaround is for the client to do some form of polling. 34 | For short messages, the HTTP protocol adds quite a lot of overhead in the form of meta data headers. 35 | For many applications, the response (and the delay waiting for it) are not needed. 36 | Previously, Comet and Ajax were used as (partial) solutions to these use cases. 37 | 38 | The WebSocket protocol defines a reliable communication channel between two equal parties, 39 | typically, but not necessarily, a Web client and a Web server, 40 | over which asynchronous messages can be send with very little overhead. 41 | Messages can be any String or ByteArray. Overhead is just a couple of bytes. 42 | There is no such thing as a direct reply or a synchronous confirmation. 43 | 44 | Using WebSockets, a server can notify a client instantly of interesting events, 45 | and clients can quickly send small notifications to a server, possibly multiplexing 46 | many virtual communications channels over a single network socket. 47 | 48 | 49 | ## The WebSocket Protocol 50 | 51 | 52 | Zinc WebSockets implements RFC 6455 , 53 | not any of the previous development versions. 54 | For an introduction to, both and 55 | are good starting points. 56 | 57 | As a protocol, WebSocket starts with an initial setup handshake that is based on HTTP. 58 | The initiative for setting up a WebSocket lies with the client, 59 | who is sending a so called connection upgrade request. 60 | The upgrade request contains a couple of special HTTP headers. 61 | The server begins as a regular HTTP server accepting the connection upgrade request. 62 | When the request is conform the specifications, 63 | a 101 switching protocols response is sent. 64 | This response also contains a couple of special HTTP headers. 65 | From that point on, the HTTP conversation over the network socket stops 66 | and the WebSocket protocol begins. 67 | 68 | WebSocket messages consist of one or more frames with minimal encoding. 69 | Behind the scenes, a number of control frames are used to properly close the WebSocket and 70 | to manage keeping alive the connection using ping and pong frames. 71 | 72 | 73 | ## Source Code 74 | 75 | 76 | The code implementing Zinc WebSockets resides in a single package called 'Zinc-WebSocket-Core' in the 77 | Zinc HTTP Components repositories. There is also an accompanying 'Zinc-WebSocket-Tests' package 78 | containing the unit tests. The ConfigurationOfZincHTTPComponents has a group called 'WebSocket' that 79 | you can load separately. 80 | ```Smalltalk 81 | ConfigurationOfZincHTTPComponents project latestVersion load: 'WebSocket' 82 | ``` 83 | 84 | ## Using Client Side WebSockets 85 | 86 | 87 | An endpoint for a WebSocket is specified using a URL 88 | 89 | ws://www.example.com:8088/my-app 90 | 91 | Two new schemes are defined, ws:// for regular WebSockets and wss:// for the secure (TLS/SSL) variant. 92 | The host:port and path specification should be familiar. 93 | 94 | Zinc WebSockets supports the usage of client side WebSockets of both the regular 95 | and secure variants (the secure variant requires Zodiac TLS/SSL). 96 | The API is really simple, once you open the socket, 97 | you use #sendMessage: and #readMessage: and finally #close. 98 | 99 | Here is a client side example taking to a public echo service: 100 | 101 | | webSocket | 102 | webSocket := ZnWebSocket to: 'ws://echo.websocket.org'. 103 | [ webSocket 104 | sendMessage: 'Pharo Smalltalk using Zinc WebSockets !'; 105 | readMessage ] ensure: [ webSocket close ]. 106 | 107 | Note that #readMessage: is blocking. 108 | It always returns a complete String or ByteArray, possible assembled out of multiple frames. 109 | Inside #readMessage: control frames will be handled automagically. 110 | Reading and sending are completely separate and independent. 111 | 112 | For sending very large messages, there are #sendTextFrames: and #sendByteFrames: that take 113 | a collection of Strings or ByteArrays to be sent as different frames of the same message. 114 | At the other end, these will be joined together and seen as a single message. 115 | 116 | In any non-trivial application, you will have to add your own encoding and decoding to messages. 117 | In many cases, JSON will be the obvious choice as the client end is often JavaScript. 118 | A modern, standalone JSON parser and writer is NeoJSON. 119 | 120 | To use secure web sockets, just use the proper URL scheme wss:// as in the following example: 121 | 122 | | webSocket | 123 | webSocket := ZnWebSocket to: 'wss://echo.websocket.org'. 124 | [ webSocket 125 | sendMessage: 'Pharo Smalltalk using Zinc WebSockets & Zodiac !'; 126 | readMessage ] ensure: [ webSocket close ]. 127 | 128 | Of course, your image has to contain Zodiac and your VM needs access to the proper plugin. 129 | That should not be a problem with the lastest Pharo 1.4 and 2.0 releases. 130 | 131 | 132 | ## Using Server Side WebSockets 133 | 134 | 135 | Since the WebSocket protocol starts off as HTTP, 136 | it is logical that a ZnServer with a special delegate is the starting point. 137 | ZnWebSocketDelegate implements the standard #handleRequest: 138 | to check if the incoming request is a valid WebSocket connection upgrade request. 139 | If so, the matching 101 switching protocols response is constructed and sent. 140 | From that moment on, the network socket stream is handed over to a new, 141 | server side ZnWebSocket object. 142 | 143 | ZnWebSocketDelegate has two properties. 144 | An optional prefix implements a specific path, like /my-ws-app. 145 | The required handler is any object implementing #value: 146 | with the new web socket as argument. 147 | 148 | Let's implement the echo service that we connected to as a client in the previous subsection. 149 | In essence, we should go in a loop, reading a message and sending it back. 150 | Here is the code: 151 | 152 | ZnServer startDefaultOn: 1701. 153 | ZnServer default delegate: (ZnWebSocketDelegate handler: 154 | [ :webSocket | 155 | [ | message | 156 | message := webSocket readMessage. 157 | webSocket sendMessage: message ] repeat ]). 158 | 159 | We start a default server on port 1701 and replace its delegate with a ZnWebSocketDelegate. 160 | The ZnWebSocketDelegate will pass each correct web socket request on to its handler. 161 | In this example, a block is used as handler. 162 | The handler is given a new connected ZnWebSocket instance. 163 | For the echo service, we go into a repeat loop, reading a message and sending it back. 164 | 165 | Finally, you can stop the server using 166 | 167 | ZnServer stopDefault. 168 | 169 | Although the code above works, it will eventually encounter two NetworkErrors: 170 | 171 | - ConnectionTimedOut 172 | - ConnectionClosed (or its more specific subclass ZnWebSocketClosed) 173 | 174 | The #readMessage call blocks on the socket stream waiting for input until its timeout expires, 175 | which will be signalled with a ConnectionTimedOut exception. 176 | In most applications, you should just keep on reading, 177 | essentially ignoring the timeout for an infinite wait on incoming messages. 178 | 179 | This behaviour is implemented in the ZnWebSocket>>#runWith: convenience method: 180 | it enters a loop reading messages and passing them to a block, 181 | continuing on timeouts. This simplifies our example: 182 | 183 | ZnServer startDefaultOn: 1701. 184 | ZnServer default delegate: (ZnWebSocketDelegate handler: 185 | [ :webSocket | 186 | webSocket runWith: [ :message | 187 | message := webSocket readMessage. 188 | webSocket sendMessage: message ] ]). 189 | 190 | That leaves us with the problem of ConnectionClosed. 191 | This exception can occur at the lowest level when the underlying network connection closes unexpectedly, 192 | or at the WebSocket protocol level when the other end sends a close frame. 193 | In either case we have to deal with it as a server. 194 | In our trivial echo example, we can catch and ignore any ConnectionClosed exception. 195 | 196 | There is a handy shortcut method on the class side of ZnWebSocket that 197 | helps to quickly set up a server implementing a WebSocket service. 198 | 199 | ZnWebSocket 200 | startServerOn: 8080 201 | do: [ :webSocket | 202 | [ 203 | webSocket runWith: [ :message | 204 | message := webSocket readMessage. 205 | webSocket sendMessage: message ] ] 206 | on: ConnectionClosed 207 | do: [ self crLog: 'Ignoring connection close, done' ] ]. 208 | 209 | Don't forget to inspect the above code so that you have a reference to the server to close it, 210 | as this will not be the default server. 211 | 212 | Although using a block as handler is convenient, for non-trivial examples 213 | a regular object implementing #value: will probably be better. 214 | You can find such an implementation in ZnWebSocketEchoHandler. 215 | 216 | The current process (thread) as spawned by the server can be used freely by the handler code, 217 | for as long as the web socket connection lasts. 218 | The responsibility for closing the connection lies with the handler, 219 | although a close from the other side will be handled correctly. 220 | 221 | To test our echo service, you could connect to it using a client side web socket, 222 | like we did in the previous subsection. This is what the unit test ZnWebSocketTests>>#testEcho does. 223 | Another solution is to run some JavaScript code in a web browser. 224 | You can find the necessary HTML page containing JavaScript code invoking the echo service on 225 | the class side of ZnWebSocketEchoHandler. 226 | The following setup will serve this code: 227 | 228 | ZnServer startDefaultOn: 1701. 229 | ZnServer default logToTranscript. 230 | ZnServer default delegate 231 | map: 'ws-echo-client-remote' 232 | to: [ :request | ZnResponse ok: (ZnEntity html: ZnWebSocketEchoHandler clientHtmlRemote) ]; 233 | map: 'ws-echo-client' 234 | to: [ :request | ZnResponse ok: (ZnEntity html: ZnWebSocketEchoHandler clientHtml) ]; 235 | map: 'ws-echo' 236 | to: (ZnWebSocketDelegate map: 'ws-echo' to: ZnWebSocketEchoHandler new). 237 | 238 | Now, you can try the following URLs: 239 | 240 | - 241 | - 242 | 243 | The first one will connect to ws://echo.websocket.org as a reference, 244 | the second one will connect to our implementation at ws://localhost:1701/ws-echo. 245 | 246 | Another simple example is available in ZnWebSocketStatusHandler where 247 | a couple of Smalltalk image statistics are emitted every second for an efficient live view in your browser. 248 | In this scenario, the server accepts each incoming web socket connection and starts streaming to it, 249 | not interested in any incoming messages. Here is the core loop: 250 | 251 | ZnWebSocketStatusHandler>>value: webSocket 252 | [ 253 | self crLog: 'Started status streaming'. 254 | [ 255 | webSocket sendMessage: self status. 256 | 1 second asDelay wait. 257 | webSocket isConnected ] whileTrue ] 258 | on: ConnectionClosed 259 | do: [ self crLog: 'Ignoring connection close' ]. 260 | self crLog: 'Stopping status streaming' 261 | 262 | The last example, ZnWebSocketChatroomHandler, implements the core logic of a chatroom: 263 | clients can send messages to the server who distributes them to all connected clients. 264 | In this case, the handler has to manage a collection of all connected client web sockets. 265 | Here is the core loop: 266 | 267 | ZnWebSocketChatroomHandler>>value: webSocket 268 | [ 269 | self register: webSocket. 270 | webSocket runWith: [ :message | 271 | self crLog: 'Received message: ', message printString. 272 | self distributeMessage: message ] ] 273 | on: ConnectionClosed 274 | do: [ 275 | self crLog: 'Connection close, cleaning up'. 276 | self unregister: webSocket ] 277 | 278 | Distributing the message is as simple as iterating over each client (ignoring some details): 279 | 280 | ZnWebSocketChatroomHandler>>distributeMessage: message 281 | clientWebSockets do: [ :each | 282 | each sendMessage: message ]. 283 | 284 | Here is code to setup all examples: 285 | 286 | ZnServer startDefaultOn: 1701. 287 | ZnServer default logToTranscript. 288 | ZnServer default delegate 289 | map: 'ws-echo-client-remote' 290 | to: [ :request | ZnResponse ok: (ZnEntity html: ZnWebSocketEchoHandler clientHtmlRemote) ]; 291 | map: 'ws-echo-client' 292 | to: [ :request | ZnResponse ok: (ZnEntity html: ZnWebSocketEchoHandler clientHtml) ]; 293 | map: 'ws-echo' 294 | to: (ZnWebSocketDelegate map: 'ws-echo' to: ZnWebSocketEchoHandler new); 295 | map: 'ws-chatroom-client' 296 | to: [ :request | ZnResponse ok: (ZnEntity html: ZnWebSocketChatroomHandler clientHtml) ]; 297 | map: 'ws-chatroom' 298 | to: (ZnWebSocketDelegate map: 'ws-chatroom' to: ZnWebSocketChatroomHandler new); 299 | map: 'ws-status-client' 300 | to: [ :request | ZnResponse ok: (ZnEntity html: ZnWebSocketStatusHandler clientHtml) ]; 301 | map: 'ws-status' 302 | to: (ZnWebSocketDelegate map: 'ws-status' to: ZnWebSocketStatusHandler new). 303 | 304 | Vist any of the following URLs: 305 | 306 | - 307 | - 308 | - 309 | 310 | Inside your Smalltalk image, you can also send chat messages, much like a moderator: 311 | 312 | (ZnServer default delegate prefixMap at: 'ws-chatroom') 313 | handler distributeMessage: 'moderator>>No trolling please!'. 314 | 315 | 316 | ## A Quick Tour of the Implementation 317 | 318 | 319 | All code resides in the 'Zinc-WebSocket-Core' package. 320 | The wire level protocol, the encoding and decoding of frames can be found in ZnWebSocketFrame. 321 | The key methods are #writeOn: and #readFrom: as well as the instance creation protocol. 322 | Together with the testing protocol and #printOn: these should give enough information to understand the implementation. 323 | 324 | ZnWebSocket implements the protocol above frames, either from a server or a client perspective. 325 | The key methods are #readMessage and #readFrame, sending is quite simple. 326 | Client side setup can be found on the class side of ZnWebSocket. 327 | Server side handling of the setup is implemented in ZnWebSocketDelegate. 328 | 329 | Two exceptions, ZnWebSocketFailed and ZnWebSocketClosed 330 | and a shared ZnWebSocketUtils class round out the core code. 331 | 332 | 333 | ## Live Demo 334 | 335 | 336 | There is a live demo available with the basic Zinc-WebSocket demos: echo, status & chatroom. 337 | 338 | 339 | 340 | Have a look at ZnWebSocketDelegate class>>#installExamplesInServer: as a starting point to learn how this demo was set up. 341 | 342 | Setting up a production demo is complicated by the fact that most proxies and load balancers, most notable market leader Apache, do not (yet) deal correctly with the WebSocket protocol. It is thus easiest to organize things so that your client talk directly to your Smalltalk image. 343 | 344 | 345 | The implementation of Zinc WebSockets as an add-on to Zinc HTTP Components 346 | was made possible in part through financial backing by Andy Burnett of 347 | [Knowinnovation Inc.](http://www.knowinnovation.com) and [ESUG](http://www.esug.org). 348 | --------------------------------------------------------------------------------