├── LICENSE
├── bin
└── cmd.js
├── church.md
├── example
├── basics
│ ├── classic0.js
│ ├── classic1.js
│ ├── concat.js
│ ├── consume0.js
│ ├── consume1.js
│ ├── consume2.js
│ ├── lines.js
│ ├── message.txt
│ ├── read0.js
│ ├── read1.js
│ ├── read2.js
│ ├── through.js
│ ├── transform0.js
│ ├── write0.js
│ ├── writing0.js
│ └── writing1.js
├── dnode
│ ├── crazy
│ │ ├── client.js
│ │ └── server.js
│ └── simple
│ │ ├── client.js
│ │ └── server.js
├── html-streams
│ ├── browser.js
│ ├── build.sh
│ ├── package.json
│ ├── render.js
│ ├── server.js
│ └── static
│ │ ├── index.html
│ │ ├── row.html
│ │ └── style.css
├── roll_your_own_socketio
│ ├── index.html
│ ├── main.js
│ └── server.js
├── scuttlebutt
│ ├── client.js
│ ├── model.js
│ ├── package.json
│ └── server.js
└── writable.js
├── package.json
└── readme.markdown
/LICENSE:
--------------------------------------------------------------------------------
1 | Creative Commons Attribution 3.0 Unported
2 |
3 | License
4 |
5 | THE WORK (AS DEFINED BELOW) IS PROVIDED UNDER THE TERMS OF THIS CREATIVE COMMONS
6 | PUBLIC LICENSE ("CCPL" OR "LICENSE"). THE WORK IS PROTECTED BY COPYRIGHT AND/OR
7 | OTHER APPLICABLE LAW. ANY USE OF THE WORK OTHER THAN AS AUTHORIZED UNDER THIS
8 | LICENSE OR COPYRIGHT LAW IS PROHIBITED.
9 |
10 | BY EXERCISING ANY RIGHTS TO THE WORK PROVIDED HERE, YOU ACCEPT AND AGREE TO BE
11 | BOUND BY THE TERMS OF THIS LICENSE. TO THE EXTENT THIS LICENSE MAY BE CONSIDERED
12 | TO BE A CONTRACT, THE LICENSOR GRANTS YOU THE RIGHTS CONTAINED HERE IN
13 | CONSIDERATION OF YOUR ACCEPTANCE OF SUCH TERMS AND CONDITIONS.
14 |
15 | 1. Definitions
16 |
17 | "Adaptation" means a work based upon the Work, or upon the Work and other
18 | pre-existing works, such as a translation, adaptation, derivative work,
19 | arrangement of music or other alterations of a literary or artistic work, or
20 | phonogram or performance and includes cinematographic adaptations or any other
21 | form in which the Work may be recast, transformed, or adapted including in any
22 | form recognizably derived from the original, except that a work that constitutes
23 | a Collection will not be considered an Adaptation for the purpose of this
24 | License. For the avoidance of doubt, where the Work is a musical work,
25 | performance or phonogram, the synchronization of the Work in timed-relation with
26 | a moving image ("synching") will be considered an Adaptation for the purpose of
27 | this License. "Collection" means a collection of literary or artistic works,
28 | such as encyclopedias and anthologies, or performances, phonograms or
29 | broadcasts, or other works or subject matter other than works listed in Section
30 | 1(f) below, which, by reason of the selection and arrangement of their contents,
31 | constitute intellectual creations, in which the Work is included in its entirety
32 | in unmodified form along with one or more other contributions, each constituting
33 | separate and independent works in themselves, which together are assembled into
34 | a collective whole. A work that constitutes a Collection will not be considered
35 | an Adaptation (as defined above) for the purposes of this License. "Distribute"
36 | means to make available to the public the original and copies of the Work or
37 | Adaptation, as appropriate, through sale or other transfer of ownership.
38 | "Licensor" means the individual, individuals, entity or entities that offer(s)
39 | the Work under the terms of this License. "Original Author" means, in the case
40 | of a literary or artistic work, the individual, individuals, entity or entities
41 | who created the Work or if no individual or entity can be identified, the
42 | publisher; and in addition (i) in the case of a performance the actors, singers,
43 | musicians, dancers, and other persons who act, sing, deliver, declaim, play in,
44 | interpret or otherwise perform literary or artistic works or expressions of
45 | folklore; (ii) in the case of a phonogram the producer being the person or legal
46 | entity who first fixes the sounds of a performance or other sounds; and, (iii)
47 | in the case of broadcasts, the organization that transmits the broadcast. "Work"
48 | means the literary and/or artistic work offered under the terms of this License
49 | including without limitation any production in the literary, scientific and
50 | artistic domain, whatever may be the mode or form of its expression including
51 | digital form, such as a book, pamphlet and other writing; a lecture, address,
52 | sermon or other work of the same nature; a dramatic or dramatico-musical work; a
53 | choreographic work or entertainment in dumb show; a musical composition with or
54 | without words; a cinematographic work to which are assimilated works expressed
55 | by a process analogous to cinematography; a work of drawing, painting,
56 | architecture, sculpture, engraving or lithography; a photographic work to which
57 | are assimilated works expressed by a process analogous to photography; a work of
58 | applied art; an illustration, map, plan, sketch or three-dimensional work
59 | relative to geography, topography, architecture or science; a performance; a
60 | broadcast; a phonogram; a compilation of data to the extent it is protected as a
61 | copyrightable work; or a work performed by a variety or circus performer to the
62 | extent it is not otherwise considered a literary or artistic work. "You" means
63 | an individual or entity exercising rights under this License who has not
64 | previously violated the terms of this License with respect to the Work, or who
65 | has received express permission from the Licensor to exercise rights under this
66 | License despite a previous violation. "Publicly Perform" means to perform public
67 | recitations of the Work and to communicate to the public those public
68 | recitations, by any means or process, including by wire or wireless means or
69 | public digital performances; to make available to the public Works in such a way
70 | that members of the public may access these Works from a place and at a place
71 | individually chosen by them; to perform the Work to the public by any means or
72 | process and the communication to the public of the performances of the Work,
73 | including by public digital performance; to broadcast and rebroadcast the Work
74 | by any means including signs, sounds or images. "Reproduce" means to make copies
75 | of the Work by any means including without limitation by sound or visual
76 | recordings and the right of fixation and reproducing fixations of the Work,
77 | including storage of a protected performance or phonogram in digital form or
78 | other electronic medium. 2. Fair Dealing Rights. Nothing in this License is
79 | intended to reduce, limit, or restrict any uses free from copyright or rights
80 | arising from limitations or exceptions that are provided for in connection with
81 | the copyright protection under copyright law or other applicable laws.
82 |
83 | 3. License Grant. Subject to the terms and conditions of this License, Licensor
84 | hereby grants You a worldwide, royalty-free, non-exclusive, perpetual (for the
85 | duration of the applicable copyright) license to exercise the rights in the Work
86 | as stated below:
87 |
88 | to Reproduce the Work, to incorporate the Work into one or more Collections, and
89 | to Reproduce the Work as incorporated in the Collections; to create and
90 | Reproduce Adaptations provided that any such Adaptation, including any
91 | translation in any medium, takes reasonable steps to clearly label, demarcate or
92 | otherwise identify that changes were made to the original Work. For example, a
93 | translation could be marked "The original work was translated from English to
94 | Spanish," or a modification could indicate "The original work has been
95 | modified."; to Distribute and Publicly Perform the Work including as
96 | incorporated in Collections; and, to Distribute and Publicly Perform
97 | Adaptations. For the avoidance of doubt:
98 |
99 | Non-waivable Compulsory License Schemes. In those jurisdictions in which the
100 | right to collect royalties through any statutory or compulsory licensing scheme
101 | cannot be waived, the Licensor reserves the exclusive right to collect such
102 | royalties for any exercise by You of the rights granted under this License;
103 | Waivable Compulsory License Schemes. In those jurisdictions in which the right
104 | to collect royalties through any statutory or compulsory licensing scheme can be
105 | waived, the Licensor waives the exclusive right to collect such royalties for
106 | any exercise by You of the rights granted under this License; and, Voluntary
107 | License Schemes. The Licensor waives the right to collect royalties, whether
108 | individually or, in the event that the Licensor is a member of a collecting
109 | society that administers voluntary licensing schemes, via that society, from any
110 | exercise by You of the rights granted under this License. The above rights may
111 | be exercised in all media and formats whether now known or hereafter devised.
112 | The above rights include the right to make such modifications as are technically
113 | necessary to exercise the rights in other media and formats. Subject to Section
114 | 8(f), all rights not expressly granted by Licensor are hereby reserved.
115 |
116 | 4. Restrictions. The license granted in Section 3 above is expressly made
117 | subject to and limited by the following restrictions:
118 |
119 | You may Distribute or Publicly Perform the Work only under the terms of this
120 | License. You must include a copy of, or the Uniform Resource Identifier (URI)
121 | for, this License with every copy of the Work You Distribute or Publicly
122 | Perform. You may not offer or impose any terms on the Work that restrict the
123 | terms of this License or the ability of the recipient of the Work to exercise
124 | the rights granted to that recipient under the terms of the License. You may not
125 | sublicense the Work. You must keep intact all notices that refer to this License
126 | and to the disclaimer of warranties with every copy of the Work You Distribute
127 | or Publicly Perform. When You Distribute or Publicly Perform the Work, You may
128 | not impose any effective technological measures on the Work that restrict the
129 | ability of a recipient of the Work from You to exercise the rights granted to
130 | that recipient under the terms of the License. This Section 4(a) applies to the
131 | Work as incorporated in a Collection, but this does not require the Collection
132 | apart from the Work itself to be made subject to the terms of this License. If
133 | You create a Collection, upon notice from any Licensor You must, to the extent
134 | practicable, remove from the Collection any credit as required by Section 4(b),
135 | as requested. If You create an Adaptation, upon notice from any Licensor You
136 | must, to the extent practicable, remove from the Adaptation any credit as
137 | required by Section 4(b), as requested. If You Distribute, or Publicly Perform
138 | the Work or any Adaptations or Collections, You must, unless a request has been
139 | made pursuant to Section 4(a), keep intact all copyright notices for the Work
140 | and provide, reasonable to the medium or means You are utilizing: (i) the name
141 | of the Original Author (or pseudonym, if applicable) if supplied, and/or if the
142 | Original Author and/or Licensor designate another party or parties (e.g., a
143 | sponsor institute, publishing entity, journal) for attribution ("Attribution
144 | Parties") in Licensor's copyright notice, terms of service or by other
145 | reasonable means, the name of such party or parties; (ii) the title of the Work
146 | if supplied; (iii) to the extent reasonably practicable, the URI, if any, that
147 | Licensor specifies to be associated with the Work, unless such URI does not
148 | refer to the copyright notice or licensing information for the Work; and (iv) ,
149 | consistent with Section 3(b), in the case of an Adaptation, a credit identifying
150 | the use of the Work in the Adaptation (e.g., "French translation of the Work by
151 | Original Author," or "Screenplay based on original Work by Original Author").
152 | The credit required by this Section 4 (b) may be implemented in any reasonable
153 | manner; provided, however, that in the case of a Adaptation or Collection, at a
154 | minimum such credit will appear, if a credit for all contributing authors of the
155 | Adaptation or Collection appears, then as part of these credits and in a manner
156 | at least as prominent as the credits for the other contributing authors. For the
157 | avoidance of doubt, You may only use the credit required by this Section for the
158 | purpose of attribution in the manner set out above and, by exercising Your
159 | rights under this License, You may not implicitly or explicitly assert or imply
160 | any connection with, sponsorship or endorsement by the Original Author, Licensor
161 | and/or Attribution Parties, as appropriate, of You or Your use of the Work,
162 | without the separate, express prior written permission of the Original Author,
163 | Licensor and/or Attribution Parties. Except as otherwise agreed in writing by
164 | the Licensor or as may be otherwise permitted by applicable law, if You
165 | Reproduce, Distribute or Publicly Perform the Work either by itself or as part
166 | of any Adaptations or Collections, You must not distort, mutilate, modify or
167 | take other derogatory action in relation to the Work which would be prejudicial
168 | to the Original Author's honor or reputation. Licensor agrees that in those
169 | jurisdictions (e.g. Japan), in which any exercise of the right granted in
170 | Section 3(b) of this License (the right to make Adaptations) would be deemed to
171 | be a distortion, mutilation, modification or other derogatory action prejudicial
172 | to the Original Author's honor and reputation, the Licensor will waive or not
173 | assert, as appropriate, this Section, to the fullest extent permitted by the
174 | applicable national law, to enable You to reasonably exercise Your right under
175 | Section 3(b) of this License (right to make Adaptations) but not otherwise. 5.
176 | Representations, Warranties and Disclaimer
177 |
178 | UNLESS OTHERWISE MUTUALLY AGREED TO BY THE PARTIES IN WRITING, LICENSOR OFFERS
179 | THE WORK AS-IS AND MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING
180 | THE WORK, EXPRESS, IMPLIED, STATUTORY OR OTHERWISE, INCLUDING, WITHOUT
181 | LIMITATION, WARRANTIES OF TITLE, MERCHANTIBILITY, FITNESS FOR A PARTICULAR
182 | PURPOSE, NONINFRINGEMENT, OR THE ABSENCE OF LATENT OR OTHER DEFECTS, ACCURACY,
183 | OR THE PRESENCE OF ABSENCE OF ERRORS, WHETHER OR NOT DISCOVERABLE. SOME
184 | JURISDICTIONS DO NOT ALLOW THE EXCLUSION OF IMPLIED WARRANTIES, SO SUCH
185 | EXCLUSION MAY NOT APPLY TO YOU.
186 |
187 | 6. Limitation on Liability. EXCEPT TO THE EXTENT REQUIRED BY APPLICABLE LAW, IN
188 | NO EVENT WILL LICENSOR BE LIABLE TO YOU ON ANY LEGAL THEORY FOR ANY SPECIAL,
189 | INCIDENTAL, CONSEQUENTIAL, PUNITIVE OR EXEMPLARY DAMAGES ARISING OUT OF THIS
190 | LICENSE OR THE USE OF THE WORK, EVEN IF LICENSOR HAS BEEN ADVISED OF THE
191 | POSSIBILITY OF SUCH DAMAGES.
192 |
193 | 7. Termination
194 |
195 | This License and the rights granted hereunder will terminate automatically upon
196 | any breach by You of the terms of this License. Individuals or entities who have
197 | received Adaptations or Collections from You under this License, however, will
198 | not have their licenses terminated provided such individuals or entities remain
199 | in full compliance with those licenses. Sections 1, 2, 5, 6, 7, and 8 will
200 | survive any termination of this License. Subject to the above terms and
201 | conditions, the license granted here is perpetual (for the duration of the
202 | applicable copyright in the Work). Notwithstanding the above, Licensor reserves
203 | the right to release the Work under different license terms or to stop
204 | distributing the Work at any time; provided, however that any such election will
205 | not serve to withdraw this License (or any other license that has been, or is
206 | required to be, granted under the terms of this License), and this License will
207 | continue in full force and effect unless terminated as stated above. 8.
208 | Miscellaneous
209 |
210 | Each time You Distribute or Publicly Perform the Work or a Collection, the
211 | Licensor offers to the recipient a license to the Work on the same terms and
212 | conditions as the license granted to You under this License. Each time You
213 | Distribute or Publicly Perform an Adaptation, Licensor offers to the recipient a
214 | license to the original Work on the same terms and conditions as the license
215 | granted to You under this License. If any provision of this License is invalid
216 | or unenforceable under applicable law, it shall not affect the validity or
217 | enforceability of the remainder of the terms of this License, and without
218 | further action by the parties to this agreement, such provision shall be
219 | reformed to the minimum extent necessary to make such provision valid and
220 | enforceable. No term or provision of this License shall be deemed waived and no
221 | breach consented to unless such waiver or consent shall be in writing and signed
222 | by the party to be charged with such waiver or consent. This License constitutes
223 | the entire agreement between the parties with respect to the Work licensed here.
224 | There are no understandings, agreements or representations with respect to the
225 | Work not specified here. Licensor shall not be bound by any additional
226 | provisions that may appear in any communication from You. This License may not
227 | be modified without the mutual written agreement of the Licensor and You. The
228 | rights granted under, and the subject matter referenced, in this License were
229 | drafted utilizing the terminology of the Berne Convention for the Protection of
230 | Literary and Artistic Works (as amended on September 28, 1979), the Rome
231 | Convention of 1961, the WIPO Copyright Treaty of 1996, the WIPO Performances and
232 | Phonograms Treaty of 1996 and the Universal Copyright Convention (as revised on
233 | July 24, 1971). These rights and subject matter take effect in the relevant
234 | jurisdiction in which the License terms are sought to be enforced according to
235 | the corresponding provisions of the implementation of those treaty provisions in
236 | the applicable national law. If the standard suite of rights granted under
237 | applicable copyright law includes additional rights not granted under this
238 | License, such additional rights are deemed to be included in the License; this
239 | License is not intended to restrict the license of any rights under applicable
240 | law. Creative Commons Notice
241 |
242 | Creative Commons is not a party to this License, and makes no warranty
243 | whatsoever in connection with the Work. Creative Commons will not be liable to
244 | You or any party on any legal theory for any damages whatsoever, including
245 | without limitation any general, special, incidental or consequential damages
246 | arising in connection to this license. Notwithstanding the foregoing two (2)
247 | sentences, if Creative Commons has expressly identified itself as the Licensor
248 | hereunder, it shall have all rights and obligations of Licensor.
249 |
250 | Except for the limited purpose of indicating to the public that the Work is
251 | licensed under the CCPL, Creative Commons does not authorize the use by either
252 | party of the trademark "Creative Commons" or any related trademark or logo of
253 | Creative Commons without the prior written consent of Creative Commons. Any
254 | permitted use will be in compliance with Creative Commons' then-current
255 | trademark usage guidelines, as may be published on its website or otherwise made
256 | available upon request from time to time. For the avoidance of doubt, this
257 | trademark restriction does not form part of this License.
258 |
259 | Creative Commons may be contacted at http://creativecommons.org/.
260 |
--------------------------------------------------------------------------------
/bin/cmd.js:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env node
2 |
3 | var fs = require('fs');
4 | var pager = require('default-pager');
5 |
6 | fs.createReadStream(__dirname + '/../readme.markdown').pipe(pager());
7 |
--------------------------------------------------------------------------------
/church.md:
--------------------------------------------------------------------------------
1 | ### Michael O. Church: archived articles
2 |
3 | - [What is spaghetti code? August 15, 2012.](https://web.archive.org/web/20150910093715/https://michaelochurch.wordpress.com/2012/08/15/what-is-spaghetti-code/)
4 | - [Functional programs rarely rot. December 6, 2012.](https://web.archive.org/web/20151221082937/https://michaelochurch.wordpress.com/2012/12/06/functional-programs-rarely-rot/)
5 |
--------------------------------------------------------------------------------
/example/basics/classic0.js:
--------------------------------------------------------------------------------
1 | var Stream = require('stream');
2 | var stream = new Stream;
3 | stream.readable = true;
4 |
5 | var c = 64;
6 | var iv = setInterval(function () {
7 | if (++c >= 75) {
8 | clearInterval(iv);
9 | stream.emit('end');
10 | }
11 | else stream.emit('data', String.fromCharCode(c));
12 | }, 100);
13 |
14 | stream.pipe(process.stdout);
15 |
--------------------------------------------------------------------------------
/example/basics/classic1.js:
--------------------------------------------------------------------------------
1 | process.stdin.on('data', function (buf) {
2 | console.log(buf);
3 | });
4 | process.stdin.on('end', function () {
5 | console.log('__END__');
6 | });
7 |
--------------------------------------------------------------------------------
/example/basics/concat.js:
--------------------------------------------------------------------------------
1 | var concat = require('concat-stream');
2 | process.stdin.pipe(concat(function (body) {
3 | console.dir(JSON.parse(body));
4 | }));
5 |
--------------------------------------------------------------------------------
/example/basics/consume0.js:
--------------------------------------------------------------------------------
1 | process.stdin.on('readable', function () {
2 | var buf = process.stdin.read();
3 | console.dir(buf);
4 | });
5 |
--------------------------------------------------------------------------------
/example/basics/consume1.js:
--------------------------------------------------------------------------------
1 | process.stdin.on('readable', function () {
2 | var buf = process.stdin.read(3);
3 | console.dir(buf);
4 | });
5 |
--------------------------------------------------------------------------------
/example/basics/consume2.js:
--------------------------------------------------------------------------------
1 | process.stdin.on('readable', function () {
2 | var buf = process.stdin.read(3);
3 | console.dir(buf);
4 | process.stdin.read(0);
5 | });
6 |
--------------------------------------------------------------------------------
/example/basics/lines.js:
--------------------------------------------------------------------------------
1 | var offset = 0;
2 |
3 | process.stdin.on('readable', function () {
4 | var buf = process.stdin.read();
5 | if (!buf) return;
6 | for (; offset < buf.length; offset++) {
7 | if (buf[offset] === 0x0a) {
8 | console.dir(buf.slice(0, offset).toString());
9 | buf = buf.slice(offset + 1);
10 | offset = 0;
11 | process.stdin.unshift(buf);
12 | return;
13 | }
14 | }
15 | process.stdin.unshift(buf);
16 | });
17 |
--------------------------------------------------------------------------------
/example/basics/message.txt:
--------------------------------------------------------------------------------
1 | beep boop
2 |
--------------------------------------------------------------------------------
/example/basics/read0.js:
--------------------------------------------------------------------------------
1 | var Readable = require('stream').Readable;
2 |
3 | var rs = Readable();
4 | rs.push('beep ');
5 | rs.push('boop\n');
6 | rs.push(null);
7 |
8 | rs.pipe(process.stdout);
9 |
--------------------------------------------------------------------------------
/example/basics/read1.js:
--------------------------------------------------------------------------------
1 | var Readable = require('stream').Readable;
2 | var rs = Readable();
3 |
4 | var c = 97;
5 | rs._read = function () {
6 | rs.push(String.fromCharCode(c++));
7 | if (c > 'z'.charCodeAt(0)) rs.push(null);
8 | };
9 |
10 | rs.pipe(process.stdout);
11 |
--------------------------------------------------------------------------------
/example/basics/read2.js:
--------------------------------------------------------------------------------
1 | var Readable = require('stream').Readable;
2 | var rs = Readable();
3 |
4 | var count = 0;
5 | var c = 97 - 1;
6 |
7 | rs._read = function () {
8 | count++;
9 | if (c >= 'z'.charCodeAt(0)) return rs.push(null);
10 |
11 | setTimeout(function () {
12 | rs.push(String.fromCharCode(++c));
13 | }, 100);
14 | };
15 |
16 | rs.pipe(process.stdout);
17 |
18 | process.on('exit', function () {
19 | console.error('\n_read() called ' + count + ' times');
20 | });
21 | process.stdout.on('error', process.exit);
22 |
--------------------------------------------------------------------------------
/example/basics/through.js:
--------------------------------------------------------------------------------
1 | var through = require('through');
2 | process.stdin.pipe(through(write, end));
3 |
4 | function write (buf) {
5 | console.log(buf);
6 | }
7 | function end () {
8 | console.log('__END__');
9 | }
10 |
--------------------------------------------------------------------------------
/example/basics/transform0.js:
--------------------------------------------------------------------------------
1 | var Transform = require('stream').Transform;
2 |
3 | var ts = Transform();
4 | ts._transform = function(chunk, enc, next) {
5 | var data = chunk.toString();
6 | data = data.toUpperCase();
7 | this.push(data); // send the transformed data on its way
8 | next();
9 | }
10 |
11 | process.on('exit', function() {
12 | console.error('\nYou cut me off!');
13 | });
14 | process.stdout.on('error', process.exit);
15 |
16 | process.stdin.pipe(ts).pipe(process.stdout);
17 |
--------------------------------------------------------------------------------
/example/basics/write0.js:
--------------------------------------------------------------------------------
1 | var Writable = require('stream').Writable;
2 | var ws = Writable({ objectMode: true });
3 | ws._write = function (chunk, enc, next) {
4 | console.dir(chunk);
5 | next();
6 | };
7 |
8 | process.stdin.pipe(ws);
9 |
--------------------------------------------------------------------------------
/example/basics/writing0.js:
--------------------------------------------------------------------------------
1 | process.stdout.write('beep ');
2 |
--------------------------------------------------------------------------------
/example/basics/writing1.js:
--------------------------------------------------------------------------------
1 | var fs = require('fs');
2 | var ws = fs.createWriteStream('message.txt');
3 |
4 | ws.write('beep ');
5 |
6 | setTimeout(function () {
7 | ws.end('boop\n');
8 | }, 1000);
9 |
--------------------------------------------------------------------------------
/example/dnode/crazy/client.js:
--------------------------------------------------------------------------------
1 | var dnode = require('dnode');
2 | var net = require('net');
3 |
4 | var d = dnode();
5 | d.on('remote', function (remote) {
6 | remote.transform('beep', function (cb) {
7 | cb(10, function (s) {
8 | console.log('beep:10 => ' + s);
9 | d.end();
10 | });
11 | });
12 | });
13 |
14 | var c = net.connect(5004);
15 | c.pipe(d).pipe(c);
16 |
--------------------------------------------------------------------------------
/example/dnode/crazy/server.js:
--------------------------------------------------------------------------------
1 | var dnode = require('dnode');
2 | var net = require('net');
3 |
4 | var server = net.createServer(function (c) {
5 | var d = dnode({
6 | transform : function (s, cb) {
7 | cb(function (n, fn) {
8 | var oo = Array(n+1).join('o');
9 | fn(s.replace(/[aeiou]{2,}/, oo).toUpperCase());
10 | });
11 | }
12 | });
13 | c.pipe(d).pipe(c);
14 | });
15 |
16 | server.listen(5004);
17 |
--------------------------------------------------------------------------------
/example/dnode/simple/client.js:
--------------------------------------------------------------------------------
1 | var dnode = require('dnode');
2 | var net = require('net');
3 |
4 | var d = dnode();
5 | d.on('remote', function (remote) {
6 | remote.transform('beep', function (s) {
7 | console.log('beep => ' + s);
8 | d.end();
9 | });
10 | });
11 |
12 | var c = net.connect(5004);
13 | c.pipe(d).pipe(c);
14 |
--------------------------------------------------------------------------------
/example/dnode/simple/server.js:
--------------------------------------------------------------------------------
1 | var dnode = require('dnode');
2 | var net = require('net');
3 |
4 | var server = net.createServer(function (c) {
5 | var d = dnode({
6 | transform : function (s, cb) {
7 | cb(s.replace(/[aeiou]{2,}/, 'oo').toUpperCase())
8 | }
9 | });
10 | c.pipe(d).pipe(c);
11 | });
12 |
13 | server.listen(5004);
14 |
--------------------------------------------------------------------------------
/example/html-streams/browser.js:
--------------------------------------------------------------------------------
1 | var through = require('through');
2 | var render = require('./render');
3 |
4 | var shoe = require('shoe');
5 | var stream = shoe('/sock');
6 |
7 | var rows = document.querySelector('#rows');
8 | stream.pipe(render()).pipe(through(function (html) {
9 | rows.innerHTML += html;
10 | }));
11 |
--------------------------------------------------------------------------------
/example/html-streams/build.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 | browserify -t brfs browser.js > static/bundle.js
3 |
--------------------------------------------------------------------------------
/example/html-streams/package.json:
--------------------------------------------------------------------------------
1 | {
2 | "name": "html-streams-example",
3 | "version": "0.0.0",
4 | "dependencies": {
5 | "hyperglue": "~1.0.0",
6 | "slice-file": "~0.2.1",
7 | "through": "~2.3.4",
8 | "ecstatic": "~4.1.4",
9 | "shoe": "~0.0.10",
10 | "browserify": "~2.14.2",
11 | "brfs": "~0.2.2",
12 | "hyperstream": "~0.1.4"
13 | }
14 | }
15 |
--------------------------------------------------------------------------------
/example/html-streams/render.js:
--------------------------------------------------------------------------------
1 | var through = require('through');
2 | var hyperglue = require('hyperglue');
3 | var fs = require('fs');
4 | var html = fs.readFileSync(__dirname + '/static/row.html');
5 |
6 | module.exports = function () {
7 | return through(function (line) {
8 | try { var row = JSON.parse(line) }
9 | catch (err) { return this.emit('error', err) }
10 |
11 | this.queue(hyperglue(html, {
12 | '.who': row.who,
13 | '.message': row.message
14 | }).outerHTML);
15 | });
16 | };
17 |
--------------------------------------------------------------------------------
/example/html-streams/server.js:
--------------------------------------------------------------------------------
1 | var http = require('http');
2 | var fs = require('fs');
3 | var hyperstream = require('hyperstream');
4 | var ecstatic = require('ecstatic')(__dirname + '/static');
5 |
6 | var sliceFile = require('slice-file');
7 | var sf = sliceFile(__dirname + '/data.txt');
8 |
9 | var render = require('./render');
10 |
11 | var server = http.createServer(function (req, res) {
12 | if (req.url === '/') {
13 | var hs = hyperstream({
14 | '#rows': sf.slice(-5).pipe(render())
15 | });
16 | hs.pipe(res);
17 | fs.createReadStream(__dirname + '/static/index.html').pipe(hs);
18 | }
19 | else ecstatic(req, res)
20 | });
21 | server.listen(8000);
22 |
23 | var shoe = require('shoe');
24 | var sock = shoe(function (stream) {
25 | sf.follow(-1,0).pipe(stream);
26 | });
27 | sock.install(server, '/sock');
28 |
--------------------------------------------------------------------------------
/example/html-streams/static/index.html:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
5 |
--------------------------------------------------------------------------------
/example/html-streams/static/style.css:
--------------------------------------------------------------------------------
1 | h1 {
2 | color: rgb(60,60,150);
3 | }
4 |
5 | .row {
6 | background-color: rgb(200,200,230);
7 | padding: 20px;
8 | width: 500px;
9 | border-radius: 5px;
10 | margin-bottom: 10px;
11 | }
12 |
13 | .row .who {
14 | color: rgb(60,60,150);
15 | }
16 |
--------------------------------------------------------------------------------
/example/roll_your_own_socketio/index.html:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
--------------------------------------------------------------------------------
/example/roll_your_own_socketio/main.js:
--------------------------------------------------------------------------------
1 | var shoe = require('shoe');
2 | var emitStream = require('emit-stream');
3 | var JSONStream = require('JSONStream');
4 |
5 | var parser = JSONStream.parse([true]);
6 | var stream = parser.pipe(shoe('/sock')).pipe(parser);
7 | var ev = emitStream(stream);
8 |
9 | ev.on('lower', function (msg) {
10 | var div = document.createElement('div');
11 | div.textContent = msg.toLowerCase();
12 | document.body.appendChild(div);
13 | });
14 |
15 | ev.on('upper', function (msg) {
16 | var div = document.createElement('div');
17 | div.textContent = msg.toUpperCase();
18 | document.body.appendChild(div);
19 | });
20 |
--------------------------------------------------------------------------------
/example/roll_your_own_socketio/server.js:
--------------------------------------------------------------------------------
1 | var http = require('http');
2 | var EventEmitter = require('events').EventEmitter;
3 |
4 | var server = http.createServer(require('ecstatic')(__dirname));
5 | server.listen(8080);
6 |
7 | var shoe = require('shoe');
8 | var emitStream = require('emit-stream');
9 | var JSONStream = require('JSONStream');
10 |
11 | var sock = shoe(function (stream) {
12 | var ev = new EventEmitter;
13 | emitStream(ev)
14 | .pipe(JSONStream.stringify())
15 | .pipe(stream)
16 | ;
17 |
18 | var intervals = [];
19 |
20 | intervals.push(setInterval(function () {
21 | ev.emit('upper', 'abc');
22 | }, 500));
23 |
24 | intervals.push(setInterval(function () {
25 | ev.emit('lower', 'def');
26 | }, 300));
27 |
28 | stream.on('end', function () {
29 | intervals.forEach(clearInterval);
30 | });
31 |
32 | });
33 | sock.install(server, '/sock');
34 |
--------------------------------------------------------------------------------
/example/scuttlebutt/client.js:
--------------------------------------------------------------------------------
1 | var Model = require('scuttlebutt/model');
2 | var net = require('net');
3 |
4 | var m = new Model;
5 | var s = m.createStream();
6 |
7 | s.pipe(net.connect(8888, 'localhost')).pipe(s);
8 |
9 | m.on('update', function cb (key) {
10 | // wait until we've gotten at least one count value from the network
11 | if (key !== 'count') return;
12 | m.removeListener('update', cb);
13 |
14 | setInterval(function () {
15 | m.set('count', Number(m.get('count')) + 1);
16 | }, 100);
17 | });
18 |
19 | m.on('update', function (key, value) {
20 | console.log(key + ' = ' + value);
21 | });
22 |
--------------------------------------------------------------------------------
/example/scuttlebutt/model.js:
--------------------------------------------------------------------------------
1 | var Model = require('scuttlebutt/model');
2 | var am = new Model;
3 | var as = am.createStream();
4 |
5 | var bm = new Model;
6 | var bs = bm.createStream();
7 |
8 | var cm = new Model;
9 | var cs = cm.createStream();
10 |
11 | var dm = new Model;
12 | var ds = dm.createStream();
13 |
14 | var em = new Model;
15 | var es = em.createStream();
16 |
17 | as.pipe(bs).pipe(as);
18 | bs.pipe(cs).pipe(bs);
19 | bs.pipe(ds).pipe(bs);
20 | ds.pipe(es).pipe(ds);
21 |
22 | em.on('update', function (key, value, source) {
23 | console.log(key + ' => ' + value + ' from ' + source);
24 | });
25 |
26 | am.set('x', 555);
27 |
28 | setTimeout(function () {
29 | console.log('em.x=' + em.get('x'));
30 | }, 100);
31 |
--------------------------------------------------------------------------------
/example/scuttlebutt/package.json:
--------------------------------------------------------------------------------
1 | {
2 | "name" : "stream-handbook-scuttlebutt-example",
3 | "private" : true,
4 | "version" : "0.0.0",
5 | "dependencies" : {
6 | "scuttlebutt" : "~3.3.2"
7 | }
8 | }
9 |
--------------------------------------------------------------------------------
/example/scuttlebutt/server.js:
--------------------------------------------------------------------------------
1 | var Model = require('scuttlebutt/model');
2 | var net = require('net');
3 |
4 | var m = new Model;
5 | m.set('count', '0');
6 | m.on('update', function (key, value) {
7 | console.log(key + ' = ' + m.get('count'));
8 | });
9 |
10 | var server = net.createServer(function (stream) {
11 | stream.pipe(m.createStream()).pipe(stream);
12 | });
13 | server.listen(8888);
14 |
15 | setInterval(function () {
16 | m.set('count', Number(m.get('count')) + 1);
17 | }, 320);
18 |
--------------------------------------------------------------------------------
/example/writable.js:
--------------------------------------------------------------------------------
1 | var Stream = require('stream');
2 | var s = new Stream;
3 | s.writable = true;
4 |
5 | var bytes = 0;
6 |
7 | s.write = function (buf) {
8 | bytes += buf.length;
9 | };
10 |
11 | s.end = function (buf) {
12 | if (arguments.length) s.write(buf);
13 |
14 | s.writable = false;
15 | console.log(bytes + ' bytes written');
16 | };
17 |
18 | s.destroy = function () {
19 | s.writable = false;
20 | };
21 |
22 | var fs = require('fs');
23 | fs.createReadStream('/etc/passwd').pipe(s);
24 |
--------------------------------------------------------------------------------
/package.json:
--------------------------------------------------------------------------------
1 | {
2 | "name": "stream-handbook",
3 | "version": "1.0.0",
4 | "description": "how to write node programs with streams",
5 | "repository": {
6 | "type": "git",
7 | "url": "git://github.com/substack/stream-handbook.git"
8 | },
9 | "homepage": "https://github.com/substack/stream-handbook",
10 | "keywords": [
11 | "documentation",
12 | "guide",
13 | "handbook",
14 | "stream",
15 | "streams"
16 | ],
17 | "author": {
18 | "name": "James Halliday",
19 | "email": "mail@substack.net",
20 | "url": "http://substack.net"
21 | },
22 | "license": "cc-by-3.0",
23 | "dependencies": {
24 | "default-pager": "^1.0.1"
25 | },
26 | "bin": {
27 | "stream-handbook": "bin/cmd.js"
28 | }
29 | }
30 |
--------------------------------------------------------------------------------
/readme.markdown:
--------------------------------------------------------------------------------
1 | # Stream-handbook
2 |
3 | This document covers the basics of how to write [node.js](http://nodejs.org/)
4 | programs with [streams](http://nodejs.org/docs/latest/api/stream.html).
5 | You also could read a **[chinese edition](https://github.com/jabez128/stream-handbook)**.
6 |
7 | [](http://creativecommons.org/licenses/by/3.0/)
8 |
9 | # Node Packaged Manuscript
10 |
11 | You can install this handbook with npm. Just do:
12 |
13 | ``` js
14 | npm install -g stream-handbook
15 | ```
16 |
17 | Now you will have a `stream-handbook` command that will open this readme file in
18 | your `$PAGER`. Otherwise, you may continue reading this document as you are
19 | presently doing.
20 |
21 | # Table of contents
22 |
23 | - [introduction](#Introduction)
24 | - [why you should use streams](#why-you-should-use-streams)
25 | - [basics](#basics)
26 | - [pipe](#pipe)
27 | - [readable streams](#readable-streams)
28 | - [writable streams](#writable-streams)
29 | - [transform](#transform)
30 | - [duplex](#duplex)
31 | - [classic streams](#classic-streams)
32 | - [read more](#read-more)
33 | - [built-in streams](#built-in-streams)
34 | - [process](#process)
35 | - [child_process.spawn()](#child_processspawn)
36 | - [fs](#fs)
37 | - [net](#net)
38 | - [http](#http)
39 | - [zlib](#zlib)
40 | - [control streams](#control-streams)
41 | - [meta streams](#meta-streams)
42 | - [state streams](#state-streams)
43 | - [http streams](#http-streams)
44 | - [io streams](#io-streams)
45 | - [parser streams](#parser-streams)
46 | - [browser streams](#browser-streams)
47 | - [html streams](#html-streams)
48 | - [audio streams](#audio-streams)
49 | - [rpc streams](#rpc-streams)
50 | - [test streams](#test-streams)
51 | - [power combos](#power-combos)
52 |
53 | # Introduction
54 |
55 | ```
56 | "We should have some ways of connecting programs like garden hose--screw in
57 | another segment when it becomes necessary to massage data in
58 | another way. This is the way of IO also."
59 | ```
60 |
61 | [Doug McIlroy. October 11, 1964](https://www.bell-labs.com/usr/dmr/www/mdmpipe.html)
62 |
63 | 
64 |
65 | ***
66 |
67 | Streams come to us from the
68 | [earliest days of Unix](http://www.youtube.com/watch?v=tc4ROCJYbm0)
69 | and have proven themselves over the decades as a dependable way to compose large
70 | systems out of small components that
71 | [do one thing well](http://www.faqs.org/docs/artu/ch01s06.html).
72 | In Unix, streams are implemented by the shell with `|` pipes.
73 | In node, the built-in
74 | [stream module](https://nodejs.org/docs/latest/api/stream.html)
75 | is used by the core libraries and can also be used by user-space modules.
76 | Similar to Unix, the node stream module's primary composition operator is called
77 | `.pipe()` and you get a backpressure mechanism for free to throttle writes for
78 | slow consumers.
79 |
80 | Streams can help to
81 | [separate your concerns](http://www.c2.com/cgi/wiki?SeparationOfConcerns)
82 | because they restrict the implementation surface area into a consistent
83 | interface that can be
84 | [reused](http://www.faqs.org/docs/artu/ch01s06.html#id2877537).
85 | You can then plug the output of one stream to the input of another and
86 | [use libraries](http://npmjs.org) that operate abstractly on streams to
87 | institute higher-level flow control.
88 |
89 | Streams are an important component of
90 | [small-program design](https://web.archive.org/web/20150910093715/https://michaelochurch.wordpress.com/2012/08/15/what-is-spaghetti-code/) (link updated thanks to https://github.com/substack/stream-handbook/pull/107)
91 | and [unix philosophy](http://www.faqs.org/docs/artu/ch01s06.html)
92 | but there are many other important abstractions worth considering.
93 | Just remember that [technical debt](http://c2.com/cgi/wiki?TechnicalDebt)
94 | is the enemy and to seek the best abstractions for the problem at hand.
95 |
96 | 
97 |
98 | ***
99 |
100 | # Why You Should Use Streams
101 |
102 | I/O in node is asynchronous, so interacting with the disk and network involves
103 | passing callbacks to functions. You might be tempted to write code that serves
104 | up a file from disk like this:
105 |
106 | ``` js
107 | var http = require('http');
108 | var fs = require('fs');
109 |
110 | var server = http.createServer(function (req, res) {
111 | fs.readFile(__dirname + '/data.txt', function (err, data) {
112 | res.end(data);
113 | });
114 | });
115 | server.listen(8000);
116 | ```
117 |
118 | This code works but it's bulky and buffers up the entire `data.txt` file into
119 | memory for every request before writing the result back to clients. If
120 | `data.txt` is very large, your program could start eating a lot of memory as it
121 | serves lots of users concurrently, particularly for users on slow connections.
122 |
123 | The user experience is poor too because users will need to wait for the whole
124 | file to be buffered into memory on your server before they can start receiving
125 | any contents.
126 |
127 | Luckily both of the `(req, res)` arguments are streams, which means we can write
128 | this in a much better way using `fs.createReadStream()` instead of
129 | `fs.readFile()`:
130 |
131 | ``` js
132 | var http = require('http');
133 | var fs = require('fs');
134 |
135 | var server = http.createServer(function (req, res) {
136 | var stream = fs.createReadStream(__dirname + '/data.txt');
137 | stream.pipe(res);
138 | });
139 | server.listen(8000);
140 | ```
141 |
142 | Here `.pipe()` takes care of listening for `'data'`, `'error'` and `'end'` events from the
143 | `fs.createReadStream()`. This code is not only cleaner, but now the `data.txt`
144 | file will be written to clients one chunk at a time immediately as each chunk
145 | is received from the disk.
146 |
147 | Using `.pipe()` has other benefits too, like handling backpressure automatically
148 | so that node won't buffer chunks into memory needlessly when the remote client
149 | is on a really slow or high-latency connection.
150 |
151 | Want compression? There are streaming modules for that too!
152 |
153 | ``` js
154 | var http = require('http');
155 | var fs = require('fs');
156 | var oppressor = require('oppressor');
157 |
158 | var server = http.createServer(function (req, res) {
159 | var stream = fs.createReadStream(__dirname + '/data.txt');
160 | stream.pipe(oppressor(req)).pipe(res);
161 | });
162 | server.listen(8000);
163 | ```
164 |
165 | Now our file is compressed for browsers that support gzip or deflate! We can
166 | just let [oppressor](https://github.com/substack/oppressor) handle all that
167 | content-encoding stuff.
168 |
169 | Don't forget to listen `'error'` event if you want to catch it
170 | ``` js
171 | var server = http.createServer(function (req, res) {
172 | var stream = fs.createReadStream(__dirname + '/data.txt');
173 | stream.pipe(res);
174 | stream.on('error', function(err){
175 | res.statusCode = 500;
176 | res.end('Internal Server Error');
177 | });
178 | });
179 | ```
180 |
181 | Once you learn the stream api, you can just snap together these streaming
182 | modules like lego bricks or garden hoses instead of having to remember how to push
183 | data through wonky non-streaming custom APIs.
184 |
185 | Streams make programming in node simple, elegant, and composable.
186 |
187 | # Basics
188 |
189 | There are 5 kinds of streams: readable, writable, transform, duplex, and
190 | "classic".
191 |
192 | ## Pipe
193 |
194 | All the different types of streams use `.pipe()` to pair inputs with outputs.
195 |
196 | `.pipe()` is just a function that takes a readable source stream `src` and hooks
197 | the output to a destination writable stream `dst`:
198 |
199 | ``` js
200 | src.pipe(dst)
201 | ```
202 |
203 | `.pipe(dst)` returns `dst` so that you can chain together multiple `.pipe()`
204 | calls together:
205 |
206 | ``` js
207 | a.pipe(b).pipe(c).pipe(d)
208 | ```
209 | which is the same as:
210 |
211 | ``` js
212 | a.pipe(b);
213 | b.pipe(c);
214 | c.pipe(d);
215 | ```
216 |
217 | This is very much like what you might do on the command-line to pipe programs
218 | together:
219 |
220 | ``` js
221 | a | b | c | d
222 | ```
223 |
224 | except in node instead of the shell!
225 |
226 | ## Readable Streams
227 |
228 | Readable streams produce data that can be fed into a writable, transform, or
229 | duplex stream by calling `.pipe()`:
230 |
231 | ``` js
232 | readableStream.pipe(dst)
233 | ```
234 |
235 | ### Creating a Readable Stream
236 |
237 | Let's make a readable stream!
238 |
239 | https://github.com/dmitriz/stream-handbook/blob/master/example/basics/read0.js
240 |
241 | ``` js
242 | var Readable = require('stream').Readable;
243 |
244 | var rs = new Readable;
245 | // see https://nodejs.org/api/stream.html#stream_readable_push_chunk_encoding
246 | rs._read = function () {};
247 | rs.push('beep ');
248 | rs.push('boop\n');
249 | rs.push(null);
250 |
251 | rs.pipe(process.stdout);
252 | ```
253 |
254 | ``` js
255 | $ node read0.js
256 | beep boop
257 | ```
258 |
259 | `rs.push(null)` tells the consumer that `rs` is done outputting data.
260 |
261 | Note here that we pushed content to the readable stream `rs` before piping to
262 | `process.stdout`, but the complete message was still written.
263 |
264 | This is because when you `.push()` to a readable stream, the chunks you push are
265 | buffered until a consumer is ready to read them.
266 |
267 | However, it would be even better in many circumstances if we could avoid
268 | buffering data altogether and only generate the data when the consumer asks for
269 | it.
270 |
271 | We can push chunks on-demand by defining a `._read` function:
272 |
273 | https://github.com/dmitriz/stream-handbook/blob/master/example/basics/read1.js
274 |
275 | ``` js
276 | var Readable = require('stream').Readable;
277 | var rs = Readable();
278 |
279 | var c = 97;
280 | rs._read = function () {
281 | rs.push(String.fromCharCode(c++));
282 | if (c > 'z'.charCodeAt(0)) rs.push(null);
283 | };
284 |
285 | rs.pipe(process.stdout);
286 | ```
287 |
288 | ``` js
289 | $ node read1.js
290 | abcdefghijklmnopqrstuvwxyz
291 | ```
292 |
293 | Here we push the letters `'a'` through `'z'`, inclusive, but only when the
294 | consumer is ready to read them.
295 |
296 | The `_read` function will also get a provisional `size` parameter as its first
297 | argument that specifies how many bytes the consumer wants to read, but your
298 | readable stream can ignore the `size` if it wants.
299 |
300 | Note that you can also use `util.inherits()` to subclass a Readable stream, but
301 | that approach doesn't lend itself very well to comprehensible examples.
302 |
303 | To show that our `_read` function is only being called when the consumer
304 | requests, we can modify our readable stream code slightly to add a delay:
305 |
306 | https://github.com/dmitriz/stream-handbook/blob/master/example/basics/read2.js
307 |
308 | ``` js
309 | var Readable = require('stream').Readable;
310 | var rs = Readable();
311 |
312 | var count = 0;
313 | var c = 97 - 1;
314 |
315 | rs._read = function () {
316 | count++;
317 | if (c >= 'z'.charCodeAt(0)) return rs.push(null);
318 |
319 | setTimeout(function () {
320 | rs.push(String.fromCharCode(++c));
321 | }, 100);
322 | };
323 |
324 | rs.pipe(process.stdout);
325 |
326 | process.on('exit', function () {
327 | console.error('\n_read() called ' + count + ' times');
328 | });
329 | process.stdout.on('error', process.exit);
330 | ```
331 |
332 | Running this program we can see that `_read()` is only called 5 times when we
333 | only request 5 bytes of output:
334 |
335 | ``` js
336 | $ node read2.js | head -c5
337 | abcde
338 | _read() called 5 times
339 | ```
340 |
341 | The setTimeout delay is necessary because the operating system requires some
342 | time to send us the relevant signals to close the pipe.
343 |
344 | The `process.stdout.on('error', fn)` handler is also necessary because the
345 | operating system will send a SIGPIPE to our process when `head` is no longer
346 | interested in our program's output, which gets emitted as an EPIPE error on
347 | `process.stdout`.
348 |
349 | These extra complications are necessary when interfacing with the external
350 | operating system pipes but are automatic when we interface directly with node
351 | streams the whole time.
352 |
353 | If you want to create a readable stream that pushes arbitrary values instead of
354 | just strings and buffers, make sure to create your readable stream with
355 | `Readable({ objectMode: true })`.
356 |
357 | ### Consuming a Readable Stream
358 |
359 | Most of the time it's much easier to just pipe a readable stream into another
360 | kind of stream or a stream created with a module like
361 | [through](https://npmjs.org/package/through)
362 | or [concat-stream](https://npmjs.org/package/concat-stream),
363 | but occasionally it might be useful to consume a readable stream directly.
364 |
365 | https://github.com/dmitriz/stream-handbook/blob/master/example/basics/consume0.js
366 |
367 | ``` js
368 | process.stdin.on('readable', function () {
369 | var buf = process.stdin.read();
370 | console.dir(buf);
371 | });
372 | ```
373 |
374 | ``` sh
375 | $ (echo abc; sleep 1; echo def; sleep 1; echo ghi) | node consume0.js
376 |
377 |
378 |
379 | null
380 | ```
381 |
382 | When data is available, the `'readable'` event fires and you can call `.read()`
383 | to fetch some data from the buffer.
384 |
385 | When the stream is finished, `.read()` returns `null` because there are no more
386 | bytes to fetch.
387 |
388 | You can also tell `.read(n)` to return `n` bytes of data. Reading a number of
389 | bytes is merely advisory and does not work for object streams, but all of the
390 | core streams support it.
391 |
392 | Here's an example of using `.read(n)` to buffer stdin into 3-byte chunks:
393 |
394 | ``` js
395 | process.stdin.on('readable', function () {
396 | var buf = process.stdin.read(3);
397 | console.dir(buf);
398 | });
399 | ```
400 |
401 | Running this example gives us incomplete data!
402 |
403 | https://github.com/dmitriz/stream-handbook/blob/master/example/basics/consume1.js
404 |
405 |
406 | ``` sh
407 | $ (echo abc; sleep 1; echo def; sleep 1; echo ghi) | node consume1.js
408 |
409 |
410 |
411 | ```
412 |
413 | This is because there is extra data left in internal buffers and we need to give
414 | node a "kick" to tell it that we are interested in more data past the 3 bytes
415 | that we've already read. A simple `.read(0)` will do this:
416 |
417 | https://github.com/dmitriz/stream-handbook/blob/master/example/basics/consume2.js
418 |
419 |
420 | ``` js
421 | process.stdin.on('readable', function () {
422 | var buf = process.stdin.read(3);
423 | console.dir(buf);
424 | process.stdin.read(0);
425 | });
426 | ```
427 |
428 | Now our code works as expected in 3-byte chunks!
429 |
430 | ``` sh
431 | $ (echo abc; sleep 1; echo def; sleep 1; echo ghi) | node consume2.js
432 |
433 |
434 |
435 |
436 | ```
437 |
438 | You can also use `.unshift()` to put back data so that the same read logic will
439 | fire when `.read()` gives you more data than you wanted.
440 |
441 | Using `.unshift()` prevents us from making unnecessary buffer copies. Here we
442 | can build a readable parser to split on newlines:
443 |
444 | https://github.com/dmitriz/stream-handbook/blob/master/example/basics/lines.js
445 |
446 | ``` js
447 | var offset = 0;
448 |
449 | process.stdin.on('readable', function () {
450 | var buf = process.stdin.read();
451 | if (!buf) return;
452 | for (; offset < buf.length; offset++) {
453 | if (buf[offset] === 0x0a) {
454 | console.dir(buf.slice(0, offset).toString());
455 | buf = buf.slice(offset + 1);
456 | offset = 0;
457 | process.stdin.unshift(buf);
458 | return;
459 | }
460 | }
461 | console.dir(buf.toString());
462 | });
463 | ```
464 |
465 | ```
466 | $ tail -n +50000 /usr/share/dict/american-english | head -n10 | node lines.js
467 | 'hearties'
468 | 'heartiest'
469 | 'heartily'
470 | 'heartiness'
471 | 'heartiness\'s'
472 | 'heartland'
473 | 'heartland\'s'
474 | 'heartlands'
475 | 'heartless'
476 | 'heartlessly'
477 | ```
478 |
479 | However, there are modules on npm such as
480 | [split](https://npmjs.org/package/split) that you should use instead of rolling
481 | your own line-parsing logic.
482 |
483 | ## Writable Streams
484 |
485 | A writable stream is a stream you can `.pipe()` to but not from:
486 |
487 | ``` js
488 | src.pipe(writableStream)
489 | ```
490 |
491 | ### Creating a Writable Stream
492 |
493 | Just define a `._write(chunk, enc, next)` function and then you can pipe a
494 | readable stream in:
495 |
496 | https://github.com/dmitriz/stream-handbook/blob/master/example/basics/write0.js
497 |
498 | ``` js
499 | var Writable = require('stream').Writable;
500 | var ws = new Writable();
501 | ws._write = function (chunk, enc, next) {
502 | console.dir(chunk);
503 | next();
504 | };
505 |
506 | process.stdin.pipe(ws);
507 | ```
508 |
509 | ``` sh
510 | $ (echo beep; sleep 1; echo boop) | node write0.js
511 |
512 |
513 | ```
514 |
515 | The first argument, `chunk` is the data that is written by the producer.
516 |
517 | The second argument `enc` is a string with the string encoding, but only when
518 | `opts.decodeString` is `false` and you've been written a string.
519 |
520 | The third argument, `next(err)` is the callback that tells the consumer that
521 | they can write more data. You can optionally pass an error object `err`, which
522 | emits an `'error'` event on the stream instance.
523 |
524 | If the readable stream you're piping from writes strings, they will be converted
525 | into `Buffer`s unless you create your writable stream with
526 | `Writable({ decodeStrings: false })`.
527 |
528 | If the readable stream you're piping from writes objects, create your writable
529 | stream with `Writable({ objectMode: true })`.
530 |
531 | ### Writing to a Writable Stream
532 |
533 | To write to a writable stream, just call `.write(data)` with the `data` you want
534 | to write!
535 |
536 | ``` js
537 | process.stdout.write('beep boop\n');
538 | ```
539 |
540 | To tell the destination writable stream that you're done writing, just call
541 | `.end()`. You can also give `.end(data)` some `data` to write before ending:
542 |
543 | https://github.com/dmitriz/stream-handbook/blob/master/example/basics/write1.js
544 |
545 | ``` js
546 | var fs = require('fs');
547 | var ws = fs.createWriteStream('message.txt');
548 |
549 | ws.write('beep ');
550 |
551 | setTimeout(function () {
552 | ws.end('boop\n');
553 | }, 1000);
554 | ```
555 |
556 | ``` sh
557 | $ node writing1.js
558 | $ cat message.txt
559 | beep boop
560 | ```
561 |
562 | If you care about high water marks and buffering, `.write()` returns false when
563 | there is more data than the `opts.highWaterMark` option passed to `Writable()`
564 | in the incoming buffer.
565 |
566 | If you want to wait for the buffer to empty again, listen for a `'drain'` event.
567 |
568 | ## Transform
569 |
570 | Transform streams are a certain type of duplex stream (both readable and writable).
571 | The distinction is that in Transform streams, the output is in some way calculated
572 | from the input.
573 |
574 | You might also hear transform streams referred to as "through streams".
575 |
576 | ``` js
577 | a.pipe(transformStream).pipe(b);
578 | ```
579 |
580 | Transform streams are one-way, requiring a read before a write.
581 |
582 | ### creating a transform stream
583 | Transform streams require a `._transform(chunk, enc, next)` function. For example, to make a simple transform stream that capitalizes text passed to it, you would write:
584 |
585 | https://github.com/dmitriz/stream-handbook/blob/master/example/basics/transform0.js
586 |
587 | ``` js
588 | var Transform = require('stream').Transform;
589 |
590 | var ts = Transform();
591 | ts._transform = function(chunk, enc, next) {
592 | var data = chunk.toString();
593 | data = data.toUpperCase();
594 | this.push(data); // send the transformed data on its way
595 | next();
596 | }
597 |
598 | process.on('exit', function() {
599 | console.error('\nYou cut me off!');
600 | });
601 | process.stdout.on('error', process.exit);
602 |
603 | process.stdin.pipe(ts).pipe(process.stdout);
604 | ```
605 |
606 | ```
607 | $ (echo hello world this is a test) | node transform0.js | head -c11
608 | HELLO WORLD
609 | You cut me off!
610 | ```
611 |
612 | The first argument `chunk` is the chunk of data provided by the input stream for processing.
613 |
614 | The second argument `enc` is the encoding type of the incoming string, if applicable.
615 |
616 | The third argument `next(err)` should be called when you are done processing the incoming chunk and are ready for more.
617 |
618 | Optionally transform streams can also be given a `._flush(chunk, enc, next)` function. This function is called after all of the input chunks have been read, but before the final `end` signal is emitted signalling that the transform is complete.
619 |
620 | ## Duplex
621 |
622 | Duplex streams are readable/writable and both ends of the stream engage
623 | in a two-way interaction, sending back and forth messages like a telephone. An
624 | rpc exchange is a good example of a duplex stream. Any time you see something
625 | like:
626 |
627 | ``` js
628 | a.pipe(b).pipe(a)
629 | ```
630 |
631 | you're probably dealing with a duplex stream.
632 |
633 | ## Classic Streams
634 |
635 | Classic streams are the old interface that first appeared in node 0.4.
636 | You will probably encounter this style of stream for a long time so it's good to
637 | know how they work.
638 |
639 | Whenever a stream has a `"data"` listener registered, it switches into
640 | `"classic"` mode and behaves according to the old API.
641 |
642 | ### Classic Readable Streams
643 |
644 | Classic readable streams are just event emitters that emit `"data"` events when
645 | they have data for their consumers and emit `"end"` events when they are done
646 | producing data for their consumers.
647 |
648 | `.pipe()` checks whether a classic stream is readable by checking the truthiness
649 | of `stream.readable`.
650 |
651 | Here is a super simple readable stream that prints `A` through `J`, inclusive:
652 |
653 | ``` js
654 | var Stream = require('stream');
655 | var stream = new Stream;
656 | stream.readable = true;
657 |
658 | var c = 64;
659 | var iv = setInterval(function () {
660 | if (++c >= 75) {
661 | clearInterval(iv);
662 | stream.emit('end');
663 | }
664 | else stream.emit('data', String.fromCharCode(c));
665 | }, 100);
666 |
667 | stream.pipe(process.stdout);
668 | ```
669 |
670 | ```
671 | $ node classic0.js
672 | ABCDEFGHIJ
673 | ```
674 |
675 | To read from a classic readable stream, you register `"data"` and `"end"`
676 | listeners. Here's an example reading from `process.stdin` using the old readable
677 | stream style:
678 |
679 | ``` js
680 | process.stdin.on('data', function (buf) {
681 | console.log(buf);
682 | });
683 | process.stdin.on('end', function () {
684 | console.log('__END__');
685 | });
686 | ```
687 |
688 | ``` sh
689 | $ (echo beep; sleep 1; echo boop) | node classic1.js
690 |
691 |
692 | __END__
693 | ```
694 |
695 | Note that whenever you register a `"data"` listener, you put the stream into
696 | compatibility mode so you lose the benefits of the new streams2 api.
697 |
698 | You should pretty much never register `"data"` and `"end"` handlers yourself
699 | anymore. If you need to interact with legacy streams, use libraries that you can
700 | `.pipe()` to instead where possible.
701 |
702 | For example, you can use [through](https://npmjs.org/package/through)
703 | to avoid setting up explicit `"data"` and `"end"` listeners:
704 |
705 | ``` js
706 | var through = require('through');
707 | process.stdin.pipe(through(write, end));
708 |
709 | function write (buf) {
710 | console.log(buf);
711 | }
712 | function end () {
713 | console.log('__END__');
714 | }
715 | ```
716 |
717 | ``` sh
718 | $ (echo beep; sleep 1; echo boop) | node through.js
719 |
720 |
721 | __END__
722 | ```
723 |
724 | or use [concat-stream](https://npmjs.org/package/concat-stream) to buffer up an
725 | entire stream's contents:
726 |
727 | ``` js
728 | var concat = require('concat-stream');
729 | process.stdin.pipe(concat(function (body) {
730 | console.log(JSON.parse(body));
731 | }));
732 | ```
733 |
734 | ``` sh
735 | $ echo '{"beep":"boop"}' | node concat.js
736 | { beep: 'boop' }
737 | ```
738 |
739 | Classic readable streams have `.pause()` and `.resume()` logic for provisionally
740 | pausing a stream, but this was merely advisory. If you are going to use
741 | `.pause()` and `.resume()` with classic readable streams, you should use
742 | [through](https://npmjs.org/package/through) to handle buffering instead of
743 | writing that yourself.
744 |
745 | ### Classic Writable Streams
746 |
747 | Classic writable streams are very simple. Just define `.write(buf)`, `.end(buf)`
748 | and `.destroy()`.
749 |
750 | `.end(buf)` may or may not get a `buf`, but node people will expect `stream.end(buf)`
751 | to mean `stream.write(buf); stream.end()` and you shouldn't violate their
752 | expectations.
753 |
754 | ## Read More
755 |
756 | * [Core stream documentation](http://nodejs.org/docs/latest/api/stream.html#stream_stream)
757 | * [Node Streams: How do they work?](http://maxogden.com/node-streams)
758 | * [Why streams are awesome](https://web.archive.org/web/20130125124400/http://blog.dump.ly/post/19819897856/why-node-js-streams-are-awesome)
759 | * [Hacker News: Why Node.js streams are awesome](https://news.ycombinator.com/item?id=3752340)
760 | * [The Power of NodeJS Streams and the event-stream Module](http://thlorenz.com/blog/the-power-of-nodejs-streams-and-the-event-stream-module/)
761 |
762 | ***
763 |
764 | # Built-in Streams
765 |
766 | These streams are built into node itself.
767 |
768 | ## Process
769 |
770 | ### [process.stdin](https://nodejs.org/docs/latest/api/process.html#process_process_stdin)
771 |
772 | This readable stream contains the standard system input stream for your program.
773 |
774 | It is paused by default but the first time you refer to it `.resume()` will be
775 | called implicitly on the
776 | [next tick](https://nodejs.org/docs/latest/api/process.html#process_process_nexttick_callback_arg).
777 |
778 | If process.stdin is a tty (check with
779 | [`tty.isatty()`](https://nodejs.org/docs/latest/api/tty.html#tty_tty_isatty_fd))
780 | then input events will be line-buffered. You can turn off line-buffering by
781 | calling `process.stdin.setRawMode(true)` BUT the default handlers for key
782 | combinations such as `^C` and `^D` will be removed.
783 |
784 | ### [process.stdout](https://nodejs.org/api/process.html#process_process_stdout)
785 |
786 | This writable stream contains the standard system output stream for your program.
787 |
788 | `write` to it if you want to send data to stdout
789 |
790 | ### [process.stderr](https://nodejs.org/api/process.html#process_process_stderr)
791 |
792 | This writable stream contains the standard system error stream for your program.
793 |
794 | `write` to it if you want to send data to stderr
795 |
796 | ## child_process.spawn()
797 |
798 | ## fs
799 |
800 | ### fs.createReadStream()
801 |
802 | ### fs.createWriteStream()
803 |
804 | ## net
805 |
806 | ### [net.connect()](https://nodejs.org/docs/latest/api/net.html#net_net_connect_options_connectlistener)
807 |
808 | This function returns a [duplex stream] that connects over tcp to a remote
809 | host.
810 |
811 | You can start writing to the stream right away and the writes will be buffered
812 | until the `'connect'` event fires.
813 |
814 | ### net.createServer()
815 |
816 | ## http
817 |
818 | ### http.createServer()
819 |
820 | Create a new instance of `http.server`. Can create with optional callback
821 | function, `http.createServer([requestListener])`, which is called with every
822 | new client request. The `http.server` object created is an
823 | [EventEmitter](https://nodejs.org/api/events.html#events_class_events_eventemitter),
824 | This means bew request handlers can be bound to an existing server object using
825 | `.on()`.
826 |
827 | ### http.request()
828 |
829 | Returns an instance of the writable stream `ClientRequest`. May be passed with
830 | an optional callback. This callback will be used as a one time listener for the
831 | [`response`](https://nodejs.org/api/http.html#http_event_response) event.
832 |
833 | `http.request()` must be explicitly ended with a call to `req.end()`.
834 |
835 | ## zlib
836 |
837 | ### zlib.createGzip()
838 |
839 | ### zlib.createGunzip()
840 |
841 | ### zlib.createDeflate()
842 |
843 | ### zlib.createInflate()
844 |
845 | ***
846 |
847 | # Control Streams
848 |
849 | ## [through](https://github.com/dominictarr/through)
850 |
851 | ## [from](https://github.com/dominictarr/from)
852 |
853 | ## [pause-stream](https://github.com/dominictarr/pause-stream)
854 | ## [pipe-io](https://github.com/coderaiser/pipe-io)
855 |
856 | Create pipe between streams and adds callback wich would
857 | be called once whenever everything is done, or error occures.
858 |
859 | Pipes is a great and very fast way to connect a couple
860 | streams with each other, but error handling could be
861 | very annoying. If you forget to handle error on one of them
862 | your application would just crash with no call stask.
863 |
864 | Here is example of creating a `pipe` between two files with
865 | all cases taken into account. One would be read and second written.
866 |
867 | ```js
868 | var fs = require('fs'),
869 | read = fs.createReadStream('README.md'),
870 | write = fs.createWriteStream('README2.md'),
871 |
872 | open = function(msg) {
873 | read.pipe(write);
874 | },
875 | finish = function() {
876 | console.log('done');
877 | done();
878 | },
879 | error = function(e) {
880 | e && console.error(e.message);
881 | done();
882 | },
883 |
884 | done = function() {
885 | read.removeListener('error', error);
886 | write.removeListener('error', error);
887 | write.removeListener('open', open);
888 | write.removeListener('finish', finish);
889 | }
890 |
891 | read.on('error', e);
892 | write.on('error', e);
893 |
894 | write.on('open', open);
895 | write.on('finish', finish);
896 | ```
897 | `pipe-io` helps write less code with same stability:
898 |
899 | ```js
900 | var fs = require('fs'),
901 | pipe = require('pipe-io'),
902 | read = fs.createReadStream('README.md'),
903 | write = fs.createWriteStream('README2.md'),
904 | error = function(e) {
905 | e && console.error(e.message);
906 | return e;
907 | }
908 |
909 | pipe([read, write], function(e) {
910 | error(e) || console.log('done');
911 | });
912 | ```
913 |
914 | ## [concat-stream](https://github.com/maxogden/node-concat-stream)
915 |
916 | concat-stream will buffer up stream contents into a single buffer.
917 | `concat(cb)` just takes a single callback `cb(body)` with the buffered
918 | `body` when the stream has finished.
919 |
920 | For example, in this program, the concat callback fires with the body string
921 | `"beep boop"` once `cs.end()` is called.
922 | The program takes the body and upper-cases it, printing `BEEP BOOP.`
923 |
924 | ``` js
925 | var concat = require('concat-stream');
926 |
927 | var cs = concat(function (body) {
928 | console.log(body.toUpperCase());
929 | });
930 | cs.write('beep ');
931 | cs.write('boop.');
932 | cs.end();
933 | ```
934 |
935 | ``` sh
936 | $ node concat.js
937 | BEEP BOOP.
938 | ```
939 |
940 | Here's an example usage of concat-stream that will parse incoming url-encoded
941 | form data and reply with a stringified JSON version of the form parameters:
942 |
943 | ``` js
944 | var http = require('http');
945 | var qs = require('querystring');
946 | var concat = require('concat-stream');
947 |
948 | var server = http.createServer(function (req, res) {
949 | req.pipe(concat(function (body) {
950 | var params = qs.parse(body.toString());
951 | res.end(JSON.stringify(params) + '\n');
952 | }));
953 | });
954 | server.listen(5005);
955 | ```
956 |
957 | ``` sh
958 | $ curl -X POST -d 'beep=boop&dinosaur=trex' http://localhost:5005
959 | {"beep":"boop","dinosaur":"trex"}
960 | ```
961 |
962 | ## [duplex](https://github.com/dominictarr/duplex)
963 |
964 | ## [duplexer](https://github.com/Raynos/duplexer)
965 |
966 | ## [duplexer2](https://github.com/deoxxa/duplexer2)
967 |
968 | ## [emit-stream](https://github.com/substack/emit-stream)
969 |
970 | ## [invert-stream](https://github.com/dominictarr/invert-stream)
971 |
972 | ## [map-stream](https://github.com/dominictarr/map-stream)
973 |
974 | ## [remote-events](https://github.com/dominictarr/remote-events)
975 |
976 | ## [buffer-stream](https://github.com/Raynos/buffer-stream)
977 |
978 | ## [event-stream](https://github.com/dominictarr/event-stream)
979 |
980 | ## [auth-stream](https://github.com/Raynos/auth-stream)
981 |
982 | ## [fork-stream](https://github.com/deoxxa/fork-stream)
983 |
984 | ## [combine-stream](https://github.com/deoxxa/combine-stream)
985 |
986 | ## [ternary-stream](https://github.com/robrich/ternary-stream)
987 |
988 | ## [ordered-through](https://github.com/juliangruber/ordered-through)
989 |
990 | ***
991 |
992 | # meta streams
993 |
994 | ## [mux-demux](https://github.com/dominictarr/mux-demux)
995 |
996 | ## [stream-router](https://github.com/Raynos/stream-router)
997 |
998 | ## [multi-channel-mdm](https://github.com/Raynos/multi-channel-mdm)
999 |
1000 | ***
1001 |
1002 | # state streams
1003 |
1004 | ## [crdt](https://github.com/dominictarr/crdt)
1005 |
1006 | ## [fsm-flow](https://github.com/ccortezia/fsm-flow)
1007 |
1008 | ## [delta-stream](https://github.com/Raynos/delta-stream)
1009 |
1010 | ## [scuttlebutt](https://github.com/dominictarr/scuttlebutt)
1011 |
1012 | [scuttlebutt](https://github.com/dominictarr/scuttlebutt) can be used for
1013 | peer-to-peer state synchronization with a mesh topology where nodes might only
1014 | be connected through intermediaries and there is no node with an authoritative
1015 | version of all the data.
1016 |
1017 | The kind of distributed peer-to-peer network that
1018 | [scuttlebutt](https://github.com/dominictarr/scuttlebutt) provides is especially
1019 | useful when nodes on different sides of network barriers need to share and
1020 | update the same state. An example of this kind of network might be browser
1021 | clients that send messages through an http server to each other and backend
1022 | processes that the browsers can't directly connect to. Another use-case might be
1023 | systems that span internal networks since IPv4 addresses are scarce.
1024 |
1025 | [scuttlebutt](https://github.com/dominictarr/scuttlebutt) uses a
1026 | [gossip protocol](https://en.wikipedia.org/wiki/Gossip_protocol)
1027 | to pass messages between connected nodes so that state across all the nodes will
1028 | [eventually converge](https://en.wikipedia.org/wiki/Eventual_consistency)
1029 | on the same value everywhere.
1030 |
1031 | Using the `scuttlebutt/model` interface, we can create some nodes and pipe them
1032 | to each other to create whichever sort of network we want:
1033 |
1034 | ``` js
1035 | var Model = require('scuttlebutt/model');
1036 | var am = new Model;
1037 | var as = am.createStream();
1038 |
1039 | var bm = new Model;
1040 | var bs = bm.createStream();
1041 |
1042 | var cm = new Model;
1043 | var cs = cm.createStream();
1044 |
1045 | var dm = new Model;
1046 | var ds = dm.createStream();
1047 |
1048 | var em = new Model;
1049 | var es = em.createStream();
1050 |
1051 | as.pipe(bs).pipe(as);
1052 | bs.pipe(cs).pipe(bs);
1053 | bs.pipe(ds).pipe(bs);
1054 | ds.pipe(es).pipe(ds);
1055 |
1056 | em.on('update', function (key, value, source) {
1057 | console.log(key + ' => ' + value + ' from ' + source);
1058 | });
1059 |
1060 | am.set('x', 555);
1061 | ```
1062 |
1063 | The network we've created is an undirected graph that looks like:
1064 |
1065 | ```
1066 | a <-> b <-> c
1067 | ^
1068 | |
1069 | v
1070 | d <-> e
1071 | ```
1072 |
1073 | Note that nodes `a` and `e` aren't directly connected, but when we run this
1074 | script:
1075 |
1076 | ``` sh
1077 | $ node model.js
1078 | x => 555 from 1347857300518
1079 | ```
1080 |
1081 | the value that node `a` set finds its way to node `e` by way of nodes `b` and
1082 | `d`. Here all the nodes are in the same process but because
1083 | [scuttlebutt](https://github.com/dominictarr/scuttlebutt) uses a
1084 | simple streaming interface, the nodes can be placed on any process or server and
1085 | connected with any streaming transport that can handle string data.
1086 |
1087 | Next we can make a more realistic example that connects over the network and
1088 | increments a counter variable.
1089 |
1090 | Here's the server which will set the initial `count` value to 0 and `count ++`
1091 | every 320 milliseconds, printing all updates to count:
1092 |
1093 | ``` js
1094 | var Model = require('scuttlebutt/model');
1095 | var net = require('net');
1096 |
1097 | var m = new Model;
1098 | m.set('count', '0');
1099 | m.on('update', function (key, value) {
1100 | console.log(key + ' = ' + m.get('count'));
1101 | });
1102 |
1103 | var server = net.createServer(function (stream) {
1104 | stream.pipe(m.createStream()).pipe(stream);
1105 | });
1106 | server.listen(8888);
1107 |
1108 | setInterval(function () {
1109 | m.set('count', Number(m.get('count')) + 1);
1110 | }, 320);
1111 | ```
1112 |
1113 | Now we can make a client that connects to this server, updates the count on an
1114 | interval, and prints all the updates it receives:
1115 |
1116 | ``` js
1117 | var Model = require('scuttlebutt/model');
1118 | var net = require('net');
1119 |
1120 | var m = new Model;
1121 | var s = m.createStream();
1122 |
1123 | s.pipe(net.connect(8888, 'localhost')).pipe(s);
1124 |
1125 | m.on('update', function cb (key) {
1126 | // wait until we've gotten at least one count value from the network
1127 | if (key !== 'count') return;
1128 | m.removeListener('update', cb);
1129 |
1130 | setInterval(function () {
1131 | m.set('count', Number(m.get('count')) + 1);
1132 | }, 100);
1133 | });
1134 |
1135 | m.on('update', function (key, value) {
1136 | console.log(key + ' = ' + value);
1137 | });
1138 | ```
1139 |
1140 | The client is slightly trickier since it should wait until it has an update from
1141 | somebody else to start updating the counter itself or else its counter would be
1142 | zeroed.
1143 |
1144 | Once we get the server and some clients running we should see a sequence like this:
1145 |
1146 | ``` js
1147 | count = 183
1148 | count = 184
1149 | count = 185
1150 | count = 186
1151 | count = 187
1152 | count = 188
1153 | count = 189
1154 | ```
1155 |
1156 | Occasionally on some of the nodes we might see a sequence with repeated values like:
1157 |
1158 | ``` js
1159 | count = 147
1160 | count = 148
1161 | count = 149
1162 | count = 149
1163 | count = 150
1164 | count = 151
1165 | ```
1166 |
1167 | These values are due to
1168 | [scuttlebutt's](https://github.com/dominictarr/scuttlebutt)
1169 | history-based conflict resolution algorithm which is hard at work ensuring that the state of the system across all nodes is eventually consistent.
1170 |
1171 | Note that the server in this example is just another node with the same
1172 | privileges as the clients connected to it. The terms "client" and "server" here
1173 | don't affect how the state synchronization proceeds, just who initiates the
1174 | connection. Protocols with this property are often called symmetric protocols.
1175 | See [dnode](https://github.com/substack/dnode) for another example of a
1176 | symmetric protocol.
1177 |
1178 | ## [append-only](http://github.com/Raynos/append-only)
1179 |
1180 | ***
1181 |
1182 | # http streams
1183 |
1184 | ## [request](https://github.com/mikeal/request)
1185 |
1186 | ## [oppressor](https://github.com/substack/oppressor)
1187 |
1188 | ## [response-stream](https://github.com/substack/response-stream)
1189 |
1190 | ***
1191 |
1192 | # io streams
1193 |
1194 | ## [reconnect](https://github.com/dominictarr/reconnect)
1195 |
1196 | ## [kv](https://github.com/dominictarr/kv)
1197 |
1198 | ## [discovery-network](https://github.com/Raynos/discovery-network)
1199 |
1200 | ***
1201 |
1202 | # parser streams
1203 |
1204 | ## [tar](https://github.com/creationix/node-tar)
1205 |
1206 | ## [trumpet](https://github.com/substack/node-trumpet)
1207 |
1208 | ## [JSONStream](https://github.com/dominictarr/JSONStream)
1209 |
1210 | Use this module to parse and stringify json data from streams.
1211 |
1212 | If you need to pass a large json collection through a slow connection or you
1213 | have a json object that will populate slowly this module will let you parse data
1214 | incrementally as it arrives.
1215 |
1216 | ## [json-scrape](https://github.com/substack/json-scrape)
1217 |
1218 | ## [stream-serializer](https://github.com/dominictarr/stream-serializer)
1219 |
1220 | ***
1221 |
1222 | # browser streams
1223 |
1224 | ## [shoe](https://github.com/substack/shoe)
1225 |
1226 | ## [domnode](https://github.com/maxogden/domnode)
1227 |
1228 | ## [sorta](https://github.com/substack/sorta)
1229 |
1230 | ## [graph-stream](https://github.com/substack/graph-stream)
1231 |
1232 | ## [arrow-keys](https://github.com/Raynos/arrow-keys)
1233 |
1234 | ## [attribute](https://github.com/Raynos/attribute)
1235 |
1236 | ## [data-bind](https://github.com/travis4all/data-bind)
1237 |
1238 | ***
1239 |
1240 | # html streams
1241 |
1242 | ## [hyperstream](https://github.com/substack/hyperstream)
1243 |
1244 |
1245 | # audio streams
1246 |
1247 | ## [baudio](https://github.com/substack/baudio)
1248 |
1249 | # rpc streams
1250 |
1251 | ## [dnode](https://github.com/substack/dnode)
1252 |
1253 | [dnode](https://github.com/substack/dnode) lets you call remote functions
1254 | through any kind of stream.
1255 |
1256 | Here's a basic dnode server:
1257 |
1258 | ``` js
1259 | var dnode = require('dnode');
1260 | var net = require('net');
1261 |
1262 | var server = net.createServer(function (c) {
1263 | var d = dnode({
1264 | transform : function (s, cb) {
1265 | cb(s.replace(/[aeiou]{2,}/, 'oo').toUpperCase())
1266 | }
1267 | });
1268 | c.pipe(d).pipe(c);
1269 | });
1270 |
1271 | server.listen(5004);
1272 | ```
1273 |
1274 | then you can hack up a simple client that calls the server's `.transform()`
1275 | function:
1276 |
1277 | ``` js
1278 | var dnode = require('dnode');
1279 | var net = require('net');
1280 |
1281 | var d = dnode();
1282 | d.on('remote', function (remote) {
1283 | remote.transform('beep', function (s) {
1284 | console.log('beep => ' + s);
1285 | d.end();
1286 | });
1287 | });
1288 |
1289 | var c = net.connect(5004);
1290 | c.pipe(d).pipe(c);
1291 | ```
1292 |
1293 | Fire up the server, then when you run the client you should see:
1294 |
1295 | ```
1296 | $ node client.js
1297 | beep => BOOP
1298 | ```
1299 |
1300 | The client sent `'beep'` to the server's `transform()` function and the server
1301 | called the client's callback with the result, neat!
1302 |
1303 | The streaming interface that dnode provides here is a duplex stream since both
1304 | the client and server are piped to each other (`c.pipe(d).pipe(c)`) with
1305 | requests and responses coming from both sides.
1306 |
1307 | The craziness of dnode begins when you start to pass function arguments to
1308 | stubbed callbacks. Here's an updated version of the previous server with a
1309 | multi-stage callback passing dance:
1310 |
1311 | ``` js
1312 | var dnode = require('dnode');
1313 | var net = require('net');
1314 |
1315 | var server = net.createServer(function (c) {
1316 | var d = dnode({
1317 | transform : function (s, cb) {
1318 | cb(function (n, fn) {
1319 | var oo = Array(n+1).join('o');
1320 | fn(s.replace(/[aeiou]{2,}/, oo).toUpperCase());
1321 | });
1322 | }
1323 | });
1324 | c.pipe(d).pipe(c);
1325 | });
1326 |
1327 | server.listen(5004);
1328 | ```
1329 |
1330 | Here's the updated client:
1331 |
1332 | ``` js
1333 | var dnode = require('dnode');
1334 | var net = require('net');
1335 |
1336 | var d = dnode();
1337 | d.on('remote', function (remote) {
1338 | remote.transform('beep', function (cb) {
1339 | cb(10, function (s) {
1340 | console.log('beep:10 => ' + s);
1341 | d.end();
1342 | });
1343 | });
1344 | });
1345 |
1346 | var c = net.connect(5004);
1347 | c.pipe(d).pipe(c);
1348 | ```
1349 |
1350 | After we spin up the server, when we run the client now we get:
1351 |
1352 | ``` sh
1353 | $ node client.js
1354 | beep:10 => BOOOOOOOOOOP
1355 | ```
1356 |
1357 | It just works!™
1358 |
1359 | The basic idea is that you just put functions in objects and you call them from
1360 | the other side of a stream and the functions will be stubbed out on the other
1361 | end to do a round-trip back to the side that had the original function in the
1362 | first place. The best thing is that when you pass functions to a stubbed
1363 | function as arguments, those functions get stubbed out on the *other* side!
1364 |
1365 | This approach of stubbing function arguments recursively shall henceforth be
1366 | known as the "turtles all the way down" gambit. The return values of any of your
1367 | functions will be ignored and only enumerable properties on objects will be
1368 | sent, json-style.
1369 |
1370 | It's turtles all the way down!
1371 |
1372 | 
1373 |
1374 | Since dnode works in node or on the browser over any stream it's easy to call
1375 | functions defined anywhere and especially useful when paired up with
1376 | [mux-demux](https://github.com/dominictarr/mux-demux) to multiplex an rpc stream
1377 | for control alongside some bulk data streams.
1378 |
1379 | ## [rpc-stream](https://github.com/dominictarr/rpc-stream)
1380 |
1381 | ***
1382 |
1383 | # test streams
1384 |
1385 | ## [tap](https://github.com/isaacs/node-tap)
1386 |
1387 | ## [stream-spec](https://github.com/dominictarr/stream-spec)
1388 |
1389 | ## [random-stream](https://github.com/bpostlethwaite/random-stream)
1390 | ***
1391 |
1392 | # power combos
1393 |
1394 | ## mississippi
1395 |
1396 | The [`mississippi` stream utility collection](https://github.com/maxogden/mississippi) demonstrates how to write better steam code using the modules [`pump`](https://npmjs.org/pump), [`stream-each`](https://npmjs.org/stream-each), [`pumpify`](https://npmjs.org/pumpify), [`duplexify`](https://npmjs.org/duplexify), [`through2`](https://npmjs.org/through2), [`concat-stream`](https://npmjs.org/concat-stream) and [`end-of-stream`](https://npmjs.org/end-of-stream).
1397 |
1398 | ## distributed partition-tolerant chat
1399 |
1400 | The [append-only](http://github.com/Raynos/append-only) module can give us a
1401 | convenient append-only array on top of
1402 | [scuttlebutt](https://github.com/dominictarr/scuttlebutt)
1403 | which makes it really easy to write an eventually-consistent, distributed chat
1404 | that can replicate with other nodes and survive network partitions.
1405 |
1406 | TODO: the rest
1407 |
1408 | ## roll your own socket.io
1409 |
1410 | We can build a socket.io-style event emitter api over streams using some of the
1411 | libraries mentioned earlier in this document.
1412 |
1413 | First we can use [shoe](http://github.com/substack/shoe)
1414 | to create a new websocket handler server-side and
1415 | [emit-stream](https://github.com/substack/emit-stream)
1416 | to turn an event emitter into a stream that emits objects.
1417 | The object stream can then be fed into
1418 | [JSONStream](https://github.com/dominictarr/JSONStream)
1419 | to serialize the objects and from there the serialized stream can be piped into
1420 | the remote browser.
1421 |
1422 | ``` js
1423 | var EventEmitter = require('events').EventEmitter;
1424 | var shoe = require('shoe');
1425 | var emitStream = require('emit-stream');
1426 | var JSONStream = require('JSONStream');
1427 |
1428 | var sock = shoe(function (stream) {
1429 | var ev = new EventEmitter;
1430 | emitStream(ev)
1431 | .pipe(JSONStream.stringify())
1432 | .pipe(stream)
1433 | ;
1434 | ...
1435 | });
1436 | ```
1437 |
1438 | Inside the shoe callback we can emit events to the `ev` function. Here we'll
1439 | just emit different kinds of events on intervals:
1440 |
1441 | ``` js
1442 | var intervals = [];
1443 |
1444 | intervals.push(setInterval(function () {
1445 | ev.emit('upper', 'abc');
1446 | }, 500));
1447 |
1448 | intervals.push(setInterval(function () {
1449 | ev.emit('lower', 'def');
1450 | }, 300));
1451 |
1452 | stream.on('end', function () {
1453 | intervals.forEach(clearInterval);
1454 | });
1455 | ```
1456 |
1457 | Finally the shoe instance just needs to be bound to an http server:
1458 |
1459 | ``` js
1460 | var http = require('http');
1461 | var server = http.createServer(require('ecstatic')(__dirname));
1462 | server.listen(8080);
1463 |
1464 | sock.install(server, '/sock');
1465 | ```
1466 |
1467 | Meanwhile on the browser side of things just parse the json shoe stream and pass
1468 | the resulting object stream to `emitStream()`. `emitStream()` just returns an
1469 | event emitter that emits the server-side events:
1470 |
1471 | ``` js
1472 | var shoe = require('shoe');
1473 | var emitStream = require('emit-stream');
1474 | var JSONStream = require('JSONStream');
1475 |
1476 | var parser = JSONStream.parse([true]);
1477 | var stream = parser.pipe(shoe('/sock')).pipe(parser);
1478 | var ev = emitStream(stream);
1479 |
1480 | ev.on('lower', function (msg) {
1481 | var div = document.createElement('div');
1482 | div.textContent = msg.toLowerCase();
1483 | document.body.appendChild(div);
1484 | });
1485 |
1486 | ev.on('upper', function (msg) {
1487 | var div = document.createElement('div');
1488 | div.textContent = msg.toUpperCase();
1489 | document.body.appendChild(div);
1490 | });
1491 | ```
1492 |
1493 | Use [browserify](https://github.com/substack/node-browserify) to build this
1494 | browser source code so that you can `require()` all these nifty modules
1495 | browser-side:
1496 |
1497 | ```
1498 | $ browserify main.js -o bundle.js
1499 | ```
1500 |
1501 | Then drop a `` into some html and open it up
1502 | in a browser to see server-side events streamed through to the browser side of
1503 | things.
1504 |
1505 | With this streaming approach you can rely more on tiny reusable components that
1506 | only need to know how to talk streams. Instead of routing messages through a
1507 | global event system socket.io-style, you can focus more on breaking up your
1508 | application into tinier units of functionality that can do exactly one thing
1509 | well.
1510 |
1511 | For instance you can trivially swap out JSONStream in this example for
1512 | [stream-serializer](https://github.com/dominictarr/stream-serializer)
1513 | to get a different take on serialization with a different set of tradeoffs.
1514 | You could bolt layers over top of shoe to handle
1515 | [reconnections](https://github.com/dominictarr/reconnect) or heartbeats
1516 | using simple streaming interfaces.
1517 | You could even add a stream into the chain to use namespaced events with
1518 | [eventemitter2](https://npmjs.org/package/eventemitter2) instead of the
1519 | EventEmitter in core.
1520 |
1521 | If you want some different streams that act in different ways it would likewise
1522 | be pretty simple to run the shoe stream in this example through mux-demux to
1523 | create separate channels for each different kind of stream that you need.
1524 |
1525 | As the requirements of your system evolve over time, you can swap out each of
1526 | these streaming pieces as necessary without as many of the all-or-nothing risks
1527 | that more opinionated framework approaches necessarily entail.
1528 |
1529 | ## html streams for the browser and the server
1530 |
1531 | We can use some streaming modules to reuse the same html rendering logic for the
1532 | client and the server! This approach is indexable, SEO-friendly, and gives us
1533 | realtime updates.
1534 |
1535 | Our renderer takes lines of json as input and returns html strings as its
1536 | output. Text, the universal interface!
1537 |
1538 | render.js:
1539 |
1540 | ``` js
1541 | var through = require('through');
1542 | var hyperglue = require('hyperglue');
1543 | var fs = require('fs');
1544 | var html = fs.readFileSync(__dirname + '/static/row.html', 'utf8');
1545 |
1546 | module.exports = function () {
1547 | return through(function (line) {
1548 | try { var row = JSON.parse(line) }
1549 | catch (err) { return this.emit('error', err) }
1550 |
1551 | this.queue(hyperglue(html, {
1552 | '.who': row.who,
1553 | '.message': row.message
1554 | }).outerHTML);
1555 | });
1556 | };
1557 | ```
1558 |
1559 | We can use [brfs](http://github.com/substack/brfs) to inline the
1560 | `fs.readFileSync()` call for browser code
1561 | and [hyperglue](https://github.com/substack/hyperglue) to update html based on
1562 | css selectors. You don't need to use hyperglue necessarily here; anything that
1563 | can return a string with html in it will work.
1564 |
1565 | The `row.html` used is just a really simple stub thing:
1566 |
1567 | row.html:
1568 |
1569 | ``` html
1570 |
1571 |
1572 |
1573 |
1574 | ```
1575 |
1576 | The server will just use [slice-file](https://github.com/substack/slice-file) to
1577 | keep everything simple. [slice-file](https://github.com/substack/slice-file) is
1578 | little more than a glorified `tail/tail -f` api but the interfaces map well to
1579 | databases with regular results plus a changes feed like couchdb.
1580 |
1581 | server.js:
1582 |
1583 | ``` js
1584 | var http = require('http');
1585 | var fs = require('fs');
1586 | var hyperstream = require('hyperstream');
1587 | var ecstatic = require('ecstatic')(__dirname + '/static');
1588 |
1589 | var sliceFile = require('slice-file');
1590 | var sf = sliceFile(__dirname + '/data.txt');
1591 |
1592 | var render = require('./render');
1593 |
1594 | var server = http.createServer(function (req, res) {
1595 | if (req.url === '/') {
1596 | var hs = hyperstream({
1597 | '#rows': sf.slice(-5).pipe(render())
1598 | });
1599 | hs.pipe(res);
1600 | fs.createReadStream(__dirname + '/static/index.html').pipe(hs);
1601 | }
1602 | else ecstatic(req, res)
1603 | });
1604 | server.listen(8000);
1605 |
1606 | var shoe = require('shoe');
1607 | var sock = shoe(function (stream) {
1608 | sf.follow(-1,0).pipe(stream);
1609 | });
1610 | sock.install(server, '/sock');
1611 | ```
1612 |
1613 | The first part of the server handles the `/` route and streams the last 5 lines
1614 | from `data.txt` into the `#rows` div.
1615 |
1616 | The second part of the server handles realtime updates to `#rows` using
1617 | [shoe](http://github.com/substack/shoe), a simple streaming websocket polyfill.
1618 |
1619 | Next we can write some simple browser code to populate the realtime updates
1620 | from [shoe](http://github.com/substack/shoe) into the `#rows` div:
1621 |
1622 | ``` js
1623 | var through = require('through');
1624 | var render = require('./render');
1625 |
1626 | var shoe = require('shoe');
1627 | var stream = shoe('/sock');
1628 |
1629 | var rows = document.querySelector('#rows');
1630 | stream.pipe(render()).pipe(through(function (html) {
1631 | rows.innerHTML += html;
1632 | }));
1633 | ```
1634 |
1635 | Just compile with [browserify](http://browserify.org) and
1636 | [brfs](http://github.com/substack/brfs):
1637 |
1638 | ``` js
1639 | $ browserify -t brfs browser.js > static/bundle.js
1640 | ```
1641 |
1642 | And that's it! Now we can populate `data.txt` with some silly data:
1643 |
1644 | ```
1645 | $ echo '{"who":"substack","message":"beep boop."}' >> data.txt
1646 | $ echo '{"who":"zoltar","message":"COWER PUNY HUMANS"}' >> data.txt
1647 | ```
1648 |
1649 | then spin up the server:
1650 |
1651 | ``` sh
1652 | $ node server.js
1653 | ```
1654 |
1655 | then navigate to `localhost:8000` where we will see our content. If we add some
1656 | more content:
1657 |
1658 | ``` sh
1659 | $ echo '{"who":"substack","message":"oh hello."}' >> data.txt
1660 | $ echo '{"who":"zoltar","message":"HEAR ME!"}' >> data.txt
1661 | ```
1662 |
1663 | then the page updates automatically with the realtime updates, hooray!
1664 |
1665 | We're now using exactly the same rendering logic on both the client and the
1666 | server to serve up SEO-friendly, indexable realtime content. Hooray!
1667 |
--------------------------------------------------------------------------------