├── LICENSE └── codelab.md /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2015 Florian Loitsch 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | 23 | -------------------------------------------------------------------------------- /codelab.md: -------------------------------------------------------------------------------- 1 | Talk-to-me 2 | ========== 3 | 4 | In this code lab, you build an IRC bot that produces random sentences based on 5 | knowledge it learned from "reading" existing documents. 6 | 7 | Step 0 8 | ------ 9 | Install Dart and a decent editor. For exploration an IDE with code completion 10 | is recommended. IntelliJ is a good choice. 11 | 12 | Since we build an IRC bot, you will need an IRC client. Also, 13 | a local IRC server is great for debugging. While not 14 | strictly necessary it makes debugging much easier. There are lots of options, 15 | but a Google search seems to recommend ngircd. It's most likely total overkill 16 | for this code lab, but works fine. Make sure to compile and run it with 17 | debugging support: 18 | 19 | ```bash 20 | ./configure --prefix=$PWD/out --enable-sniffer --enable-debug 21 | make && make install 22 | out/sbin/ngircd -n -s 23 | ``` 24 | 25 | If you have a smaller option that doesn't require any compilation feel free to 26 | reach out to me so I can update this step. 27 | 28 | Optionally clone `https://github.com/floitsch/dart-irc-codelab.git` to get the 29 | complete sources of this code lab. The git repository has different branches 30 | for every step. 31 | 32 | Step 1 - Hello world 33 | ------ 34 | 35 | In this step, we create an skeleton app of Dart. We will keep this project 36 | really simple, and only work with one file. This means that we don't need to go 37 | through any project wizard. Just open a new file `main.dart` (or any other 38 | name you prefer) and open it. (You can also create a console-application 39 | skeleton and clear the code in the main file). 40 | 41 | Add the following lines to it: 42 | 43 | ```dart 44 | main() { 45 | print("Hello world"); 46 | } 47 | ``` 48 | 49 | Congratulations: you just wrote a complete Dart application. Run it either 50 | through your IDE or with `dart --checked main.dart`. The `--checked` flag is not 51 | necessary but strongly recommended when developing. It dynamically checks the 52 | types in the program. Currently your program doesn't have any yet, so let's 53 | change that: 54 | 55 | ```dart 56 | void main() { 57 | print("Hello world"); 58 | } 59 | ``` 60 | 61 | So far the dynamic checker doesn't need to work hard, but over time we will add 62 | more functions and types. As you can see, types are _optional_ in Dart. You can 63 | write them, but don't need to. We recommend to write types on function 64 | boundaries (but you can write them more frequently or not at all). 65 | 66 | Step 2 - Connect to the server 67 | ------ 68 | 69 | In this step, we connect to an IRC server. Our client will be really dumb and 70 | only support a tiny subset of IRC commands. The complete spec is in 71 | [RFC 2812](https://tools.ietf.org/html/rfc2812); a summary can be found 72 | [here](http://blog.initprogram.com/2010/10/14/a-quick-basic-primer-on-the-irc-protocol/). 73 | 74 | Let's start with connecting a socket to the server. If possible connect to a 75 | local server first, before you connect to a public external server. 76 | 77 | [Sockets](https://api.dartlang.org/apidocs/channels/stable/dartdoc-viewer/dart:io.Socket) 78 | are part of the IO-library. As such, we have to import them first. Add the 79 | following import clause to the beginning of your file: 80 | 81 | ```dart 82 | import 'dart:io'; 83 | ``` 84 | 85 | This imports all IO classes and functions into the namespace of this library. We 86 | can use the static `connect` method of `Socket` to connect to a server: 87 | `Socket.connect(host, port)`. In Dart, IO operations are asynchronous, which 88 | means that this call won't block, but yields immediately. Instead, it returns a 89 | [Future](https://api.dartlang.org/apidocs/channels/stable/dartdoc-viewer/dart:async.Future) 90 | of the result. Futures (aka "promise" or "eventual") are objects that represent 91 | values that haven't been computed yet. Once the value is available, the future 92 | invokes callbacks that have been waiting for the value. Let's give it a try: 93 | 94 | ```dart 95 | void main() { 96 | var future = Socket.connect("localhost", 6667); 97 | // Now register a callback: 98 | future.then((socket) { 99 | // The socket is now available. 100 | print("Connected"); 101 | socket.destroy(); // Shuts down the socket in both directions. 102 | }); 103 | print("Callback has been registered, but hasn't been called yet"); 104 | } 105 | ``` 106 | 107 | Here, we have our first closure of the program: the argument to `then` is a 108 | one-argument closure. Contrary to static functions, closures can not declare 109 | their return type (but can have type arguments for their arguments). They 110 | cannot be named, either. 111 | 112 | Before running this program, make sure that you have some server running on 113 | localhost 6667. You can, for example, start a local IRC server, or run 114 | netcat (`nc -l 6667`), otherwise you will see the following error message: 115 | 116 | Unhandled exception: 117 | Uncaught Error: SocketException: OS Error: Connection refused, errno = 111, address = localhost, port = 52933 118 | 119 | This is a good opportunity to point out that, throughout this code lab, we will 120 | ignore errors. A real-world program would need to be much more careful where 121 | and when it needs to catch uncaught exceptions. 122 | 123 | Assuming that everything went well we should be connected now. Let's 124 | authenticate and say "hello": 125 | 126 | ```dart 127 | /// Given a connected [socket] runs the IRC bot. 128 | void handleIrcSocket(Socket socket) { 129 | 130 | void authenticate() { 131 | var nick = "myBot"; // <=== Replace with your bot name. Try to be unique. 132 | socket.write('NICK $nick\r\n'); 133 | socket.write('USER username 8 * :$nick\r\n'); 134 | } 135 | 136 | authenticate(); 137 | socket.write('JOIN ##dart-irc-codelab\r\n'); 138 | socket.write('PRIVMSG ##dart-irc-codelab :Hello world\r\n'); 139 | socket.write('QUIT\r\n'); 140 | socket.destroy(); 141 | } 142 | 143 | void main() { 144 | Socket.connect("localhost", 6667) // No need for the temporary variable. 145 | .then(handleIrcSocket); 146 | } 147 | ``` 148 | 149 | Lot's of things are happening here: 150 | 151 | * We moved the IRC code into a `handleIrcSocket` function which we give as 152 | argument to the `then` of the future. Note that we don't need to create an 153 | anonymous closure, and can just reference the static function. 154 | 155 | * The `handleIrcSocket` function has a nested function `authenticate`. This is 156 | just to make the code easier to read. We could just inline the function body. 157 | 158 | * Inside `authenticate` we send the IRC `NICK` and `USER` commands. For this, we 159 | need a nick and a username. For simplicity we use the same name here. The 160 | variable `nick` holds that string, and is spliced into the command strings 161 | using string interpolation. String interpolation is just a simple nice way of 162 | concatenating strings. 163 | 164 | * Once we are authenticated, we join the `##dart-irc-codelab` channel and 165 | send a message to it. Note that, in theory, joining the channel is not 166 | always necessary. However, many servers (including freenode) disable messages 167 | from the outside by default. 168 | 169 | * So far, we don't listen to anything from the server. This is clearly not a 170 | good idea and something we need to fix. This will also fix another issue with 171 | the code: shutting down the socket (in both directions) after having sent our 172 | messages is very aggressive. Don't be surprised if your message doesn't make 173 | it to the IRC channel. 174 | 175 | * Messages to IRC servers must be terminated with `\r\n`. This is something that 176 | is extremely easy to forget, so we will create a helper function for it. 177 | 178 | Step 3 - Handle Server messages 179 | ------ 180 | 181 | The most important server-message we have to handle to is the `PING` message. As 182 | expected, the client has to respond with a `PONG`. Let's add this functionality: 183 | 184 | ```dart 185 | import 'dart:convert'; 186 | 187 | ... 188 | 189 | /// Sends a message to the IRC server. 190 | /// 191 | /// The message is automatically terminated with a `\r\n`. 192 | void writeln(String message) { 193 | socket.write('$message\r\n'); 194 | } 195 | 196 | void handleServerLine(String line) { 197 | print("from server: $line"); 198 | if (line.startsWith("PING")) { 199 | writeln("PONG ${line.substring("PING ".length)}"); 200 | } 201 | } 202 | 203 | socket 204 | .transform(UTF8.decoder) 205 | .transform(new LineSplitter()) 206 | .listen(handleServerLine, 207 | onDone: socket.close); 208 | 209 | authenticate(); 210 | writeln('JOIN ##dart-irc-codelab'); 211 | writeln('PRIVMSG ##dart-irc-codelab :Hello world'); 212 | writeln('QUIT'); 213 | ``` 214 | 215 | Listening to a socket can be done with the `listen` function. However, this 216 | would give us the bytes that are received by the socket. We want to have the 217 | UTF8 lines that are sent by the server. Sockets implement the 218 | [Stream](https://api.dartlang.org/apidocs/channels/stable/dartdoc-viewer/dart:async.Stream) 219 | interface and support lots of methods that help to deal with the incoming data. 220 | One of them is `transform` which takes the input and transforms it to a 221 | different representation. This is a more powerful version of the well known 222 | `map` functionality. 223 | 224 | Dart's core libraries already come with convenient conversion transformers. They 225 | are located in the `dart:convert` library. After transforming the incoming data, 226 | first to UTF8, then to individual lines, we listen to the data. 227 | Normal data (the lines) is sent to the 228 | `handleServerMessage` closure, and we pass in a named argument `onDone` which 229 | is invoked when the server shut down the connection. At that moment we simply 230 | invoke the torn-off closure `socket.close`. The closure `socket.close` is bound 231 | to the socket it came from, and will correctly close the sending part of the 232 | socket. Since we are sending a `QUIT` message after our `Hello world` message 233 | the server will furthermore close the receiving part of the connection. We can 234 | thus remove the `socket.destroy()` line we had in the previous step. 235 | 236 | If you feel (slightly) adventurous, you can try your bot in the wild now. 237 | Use your IRC client to connect to chat.freenode.net and join the 238 | `##dart-irc-codelab` channel. Make sure, you have picked a unique name and 239 | change the `localhost` line to `chat.freenode.net`. Then run your program again. 240 | You should see a hello-world message (hopefully from your bot) on the public 241 | channel. If not, you should see an indication of why the server rejected your 242 | requests. If you don't see the message and don't get anything useful from the 243 | server, check that the program doesn't contain the `socket.destroy` line 244 | anymore. 245 | 246 | So far the bot only handles the `PING` line from the server. We are only 247 | interested in one other type of messages: `PRIVMSG`s from other clients. The 248 | easiest way to deal with them is to use a regular expression. 249 | 250 | ```dart 251 | final RegExp ircMessageRegExp = 252 | new RegExp(r":([^!]+)!([^ ]+) PRIVMSG ([^ ]+) :(.*)"); 253 | 254 | ... 255 | 256 | void handleMessage(String msgNick, 257 | String server, 258 | String channel, 259 | String msg) { 260 | print("$msgNick: $msg"); 261 | } 262 | 263 | void handleServerLine(String line) { 264 | if (line.startsWith("PING")) { 265 | writeln("PONG ${line.substring("PING ".length)}"); 266 | return; 267 | } 268 | var match = ircMessageRegExp.firstMatch(line); 269 | if (match != null) { 270 | handleMessage(match[1], match[2], match[3], match[4]); 271 | return; 272 | } 273 | print("from server: $line"); 274 | } 275 | ``` 276 | 277 | In order to test the receipt of these messages we have to disable the `QUIT` 278 | command we send to the server. Otherwise the bot won't have the time to 279 | receive interesting messages. You can just delete that line now. We will 280 | deal with quitting in the next step. 281 | 282 | Step 4 - Respond to IRC messages 283 | ------ 284 | 285 | In this step we make the IRC bot interactive. Now, that the bot understands 286 | messages that are sent to a channel we can make it respond to them. For 287 | now we just want to be able to ask our bot to quit. 288 | 289 | ```dart 290 | void handleMessage(String msgNick, 291 | String server, 292 | String channel, 293 | String msg) { 294 | if (msg.startsWith("$nick:")) { 295 | // Direct message to us. 296 | var text = msg.substring(msg.indexOf(":") + 1).trim(); 297 | if (text == "please leave") { 298 | print("Leaving by request of $msgNick"); 299 | writeln("QUIT"); 300 | return; 301 | } 302 | } 303 | print("$msgNick: $msg"); 304 | } 305 | ``` 306 | 307 | This also requires to move the `nick` variable out of the `authenticate` 308 | function: 309 | 310 | ```dart 311 | void handleIrcSocket(Socket socket) { 312 | final nick = "myBot"; // <=== Replace with your bot name. Try to be unique. 313 | ... 314 | ``` 315 | 316 | With this addition we can properly leave the server by simply sending a nice irc 317 | message to the bot: `myBot: please leave`. 318 | 319 | At this point in the code lab we will switch to generating random sentences. We 320 | will come back to the irc-bot once we have something interesting to say. If you 321 | are interested you can of course experiment with a few other commands. Some easy 322 | ones are "echo ", "what's the time?", or "how long till dinner?". 323 | 324 | Step 5 - Trigrams 325 | ------ 326 | 327 | In this step we start the logic to generate random sentences. It is based on 328 | simplified Markov chains. This approach has been inspired by 329 | [a similar implementation in Python] 330 | (https://charlesleifer.com/blog/building-markov-chain-irc-bot-python-and-redis/). 331 | The idea is to create a set of all word-trigrams that exist in some given 332 | real-world documents. A sentence-generator uses these trigrams to build random 333 | sentences. Given two words the sentence-generator finds a random 334 | trigram that starts with these two words and adds the third word to the 335 | sentence. It then repeats the process with the new last two words, until it 336 | encounters a terminating ".". 337 | 338 | Let's start with extracting all trigrams from a document. For the next steps 339 | we don't need to run the irc bot, so let's just rename the old `main` to 340 | `runIrcBot`. We will invoke that function later, when we have the sentence 341 | generator ready. 342 | 343 | This time we want to handle command-line arguments, so let's add a new 344 | main that accepts them as argument. 345 | 346 | ```dart 347 | void runIrcBot() { 348 | Socket.connect("localhost", 6667) 349 | .then(handleIrcSocket); 350 | } 351 | 352 | void main(List arguments) { 353 | print(arguments); 354 | } 355 | ``` 356 | 357 | We want to analyze existing documents to farm them for trigrams. 358 | [Gutenberg](http://gutenberg.org) is a good resource for out-of-copyright work 359 | that is perfect for this task. I used 360 | [Alice in Wonderland](http://www.gutenberg.org/cache/epub/11/pg11.txt), and 361 | [The US constitution](http://www.gutenberg.org/cache/epub/5/pg5.txt) in this 362 | code lab, but there are many other interesting documents. 363 | 364 | Download these (or other books) and change your setup so that your program is 365 | invoked with these books as arguments. You should have an output similar to 366 | this one: 367 | 368 | $ dart --checked main.dart constitution.txt alice.txt 369 | [constitution.txt, alice.txt] 370 | 371 | For simplicity, we store the trigrams in a table that maps 2-word strings to 372 | all possible third words. Since there is lots of associated code with this 373 | collection we encapsulate it in a class. 374 | 375 | ```dart 376 | class SentenceGenerator { 377 | final _db = new Map>(); 378 | 379 | void addBook(String fileName) { 380 | print("TODO: add book $fileName"); 381 | } 382 | } 383 | 384 | void main(arguments) { 385 | var generator = new SentenceGenerator(); 386 | arguments.forEach(generator.addBook); 387 | } 388 | ``` 389 | 390 | Note the "\_" in `_db` field name. It means that this field is only visible 391 | within the same library. The same mechanism also works for classes, methods or 392 | static functions. The moment an identifier starts with an "\_" it is private to 393 | the current library. 394 | 395 | Since our code lab is small and privacy protection is completely unnecessary we 396 | will not create any other private symbols. 397 | 398 | We use a very crude way of updating the database when we get a new book: 399 | 400 | ```dart 401 | void addBook(String fileName) { 402 | var content = new File(fileName).readAsStringSync(); 403 | 404 | // Make sure the content terminates with a ".". 405 | if (!content.endsWith(".")) content += "."; 406 | 407 | var words = content 408 | .replaceAll("\n", " ") // Treat new lines as if they were spaces. 409 | .replaceAll("\r", "") // Discard "\r". 410 | .replaceAll(".", " .") // Add space before ".", to simplify splitting. 411 | .split(" ") 412 | .where((String word) => word != ""); 413 | 414 | var preprevious = null; 415 | var previous = null; 416 | for (String current in words) { 417 | if (preprevious != null) { 418 | // We have a trigram. 419 | // Concatenate the first two words and use it as a key. If this key 420 | // doesn't have a corresponding set yet, create it. Then add the 421 | // third word into the set. 422 | _db.putIfAbsent("$preprevious $previous", () => new Set()) 423 | .add(current); 424 | } 425 | 426 | preprevious = previous; 427 | previous = current; 428 | } 429 | } 430 | ``` 431 | 432 | This code simply runs through all words and adds them as trigrams to the 433 | database. For example the sentence "My hovercraft is full of eels." will add the 434 | following trigrams to the database: 435 | 436 | "My hovercraft" -> "is" 437 | "hovercraft is" -> "full" 438 | "is full" -> "of" 439 | "full of" -> "eels" 440 | "of eels" -> "." 441 | 442 | The values (on the right) are sets. This becomes important when seeing new 443 | trigrams that have the same first two words. For example adding the 444 | sentence "A hovercraft is an aircraft." to the database would yield: 445 | 446 | "My hovercraft" -> "is" 447 | "hovercraft is" -> "full", "an" // <= two trigrams with the same first words. 448 | "is full" -> "of" 449 | "full of" -> "eels" 450 | "of eels" -> "." 451 | "A hovercraft" -> "is" 452 | "is an" -> "aircraft" 453 | "an aircraft" -> "." 454 | 455 | A markov chain would count the occurrences to provide better guesses, but here 456 | we simply collect a set of possible trigrams. 457 | 458 | Step 6 - Random sentences 459 | ------ 460 | 461 | In this step we add support for generating random sentences. As a starting point 462 | we select a random pair of words from the database and use it as the beginning 463 | of the sentence. We then follow possible sequences until we reach a ".". 464 | 465 | For this step we need a random number generator. Dart provides an implementation 466 | in the `dart:math` library. Import that library and store a final generator 467 | as final field in the `SentenceGenerator` class: 468 | 469 | ```dart 470 | import 'dart:io'; 471 | import 'dart:convert'; 472 | import 'dart:math'; 473 | ... 474 | class SentenceGenerator { 475 | final _db = new Map>(); 476 | final rng = new Random(); 477 | ... 478 | ``` 479 | 480 | We also add a few helper functions to make the sentence generation easier: 481 | 482 | ```dart 483 | int get keyCount => _db.length; 484 | 485 | String pickRandomPair() => _db.keys.elementAt(rng.nextInt(keyCount)); 486 | 487 | String pickRandomThirdWord(String firstWord, String secondWord) { 488 | var key = "$firstWord $secondWord"; 489 | var possibleSequences = _db[key]; 490 | return possibleSequences.elementAt(rng.nextInt(possibleSequences.length)); 491 | } 492 | ``` 493 | 494 | The first function `keyCount` is in fact a getter. That is, it is used as if it 495 | was a field. This can be seen in the second function `pickRandomPair` where the 496 | `keyCount` getter is used to provide a range to the random number generator. 497 | 498 | Since these functions are very small and fit on one line we use the "`=>`" 499 | notation for them. This notation is just syntactic sugar for a function that 500 | returns one expression. We could write these two helpers as follows without 501 | any semantic difference: 502 | 503 | ```dart 504 | int get keyCount { return _db.length; } 505 | 506 | String pickRandomPair() { return _db.keys.elementAt(rng.nextInt(keyCount)); } 507 | ``` 508 | 509 | Given these helper functions we can generate a full sentence quite easily: 510 | 511 | ```dart 512 | String generateRandomSentence() { 513 | var start = pickRandomPair(); 514 | var startingWords = start.split(" "); 515 | var preprevious = startingWords[0]; 516 | var previous = startingWords[1]; 517 | var sentence = [preprevious, previous]; 518 | var current; 519 | do { 520 | current = pickRandomThirdWord(preprevious, previous); 521 | sentence.add(current); 522 | preprevious = previous; 523 | previous = current; 524 | } while (current != "."); 525 | return sentence.join(" "); 526 | } 527 | 528 | ... 529 | 530 | void main(arguments) { 531 | var generator = new SentenceGenerator(); 532 | arguments.forEach(generator.addBook); 533 | print(generator.generateRandomSentence()); 534 | } 535 | ``` 536 | 537 | Give it a try. You should get some reasonable sentences. For example: 538 | 539 | $ dart --checked main.dart constitution.txt alice.txt 540 | dreadfully puzzled by the first question, you know I'm mad?' said Alice . 541 | 542 | $ dart --checked main.dart constitution.txt alice.txt 543 | jury had a bone in his confusion he bit a large arm-chair at one corner of 544 | it: for she could not make out what she was saying, and the Acceptance of 545 | Congress, lay any Duty of Tonnage, keep Troops, or Ships of War in time of life . 546 | 547 | $ dart --checked main.dart constitution.txt alice.txt 548 | an announcement goes out in a confused way, 'Prizes! Prizes!' Alice had no 549 | very clear notion how delightful it will be When they take us up and rubbed 550 | its eyes: then it chuckled . 551 | 552 | Step 7 - Chat bot 553 | ------ 554 | 555 | In this step, we combine the sentence generator with the IRC bot. Whenever we 556 | ask the bot to "talk to me" it should generate a new random sentence and 557 | display it. 558 | 559 | At this point we need to launch the IRC bot again (with a generator as 560 | argument). Furthermore, we need to add support for a new command in 561 | `handleMessage`. These changes touch lots of different parts of the program, but 562 | are all relatively minor. 563 | 564 | ```dart 565 | void handleIrcSocket(Socket socket, SentenceGenerator sentenceGenerator) { 566 | 567 | ... 568 | 569 | void say(String message) { 570 | if (message.length > 120) { 571 | // IRC doesn't like it when lines are too long. 572 | message = message.substring(0, 120); 573 | } 574 | writeln('PRIVMSG ##dart-irc-codelab :$message'); 575 | } 576 | 577 | void handleMessage(...) { 578 | if (msg.startsWith("$nick:")) { 579 | // Direct message to us. 580 | var text = msg.substring(msg.indexOf(":") + 1).trim(); 581 | switch (text) { 582 | case "please leave": 583 | print("Leaving by request of $msgNick"); 584 | writeln("QUIT"); 585 | return; 586 | case "talk to me": 587 | say(sentenceGenerator.generateRandomSentence()); 588 | return; 589 | } 590 | } 591 | print("$msgNick: $msg"); 592 | } 593 | 594 | ... 595 | 596 | void runIrcBot(SentenceGenerator generator) { 597 | Socket.connect("localhost", 6667) 598 | .then((socket) => handleIrcSocket(socket, generator)); 599 | } 600 | 601 | ... 602 | 603 | void main(arguments) { 604 | var generator = new SentenceGenerator(); 605 | arguments.forEach(generator.addBook); 606 | runIrcBot(generator); 607 | } 608 | ``` 609 | 610 | The hardest part here is to pass the generator from the main function to the 611 | irc bot. We could have simplified our live by setting a static variable, but 612 | in general avoiding static state is a good idea. 613 | 614 | Step 8 - Completing Sentences 615 | ------ 616 | 617 | In this step, we modify the generator to accept a few words as starting 618 | suggestions. We want to start a sentence and let the generator finish it. 619 | 620 | Handling the command from IRC happens easily in the `handleMessage` function: 621 | 622 | ```dart 623 | void handleMessage(...) { 624 | if (msg.startsWith("$nick:")) { 625 | // Direct message to us. 626 | var text = msg.substring(msg.indexOf(":") + 1).trim(); 627 | switch (text) { 628 | ... 629 | default: 630 | if (text.startsWith("finish: ")) { 631 | var start = text.substring("finish: ".length); 632 | var sentence = sentenceGenerator.finishSentence(start); 633 | say(sentence == null ? "Unable to comply." : sentence); 634 | return; 635 | } 636 | } 637 | ... 638 | ``` 639 | 640 | This will currently crash dynamically, because we haven't implemented the 641 | `finishSentence` method yet. Note, that the VM still runs the code, and only 642 | fails dynamically when it encounters the line that contains the method call. 643 | However, the editor (or `dartanalyzer`) warns you statically that there is a 644 | likely problem at this location. 645 | 646 | In order to implement the missing `finishSentence` function we first split the 647 | `generateRandomSentence` function so that it accepts a beginning of a sentence: 648 | 649 | ```dart 650 | String generateRandomSentence() { 651 | var start = pickRandomPair(); 652 | var startingWords = start.split(" "); 653 | return generateSentenceStartingWith(startingWords[0], startingWords[1]); 654 | } 655 | 656 | String generateSentenceStartingWith(String preprevious, String previous) { 657 | var sentence = [preprevious, previous]; 658 | var current; 659 | ... 660 | } 661 | ``` 662 | 663 | Now let's add the `finishSentence` function. Since our database is not very big 664 | we can't assume that we can continue from the last two words. The 665 | `finishSentence` function therefore iteratively drops the last word until it 666 | can finish a sentence, or until too few words are left. In the latter case it 667 | returns null. 668 | 669 | ```dart 670 | String finishSentence(String start) { 671 | // This function has local types, to show the differences between List and 672 | // Iterable. 673 | 674 | List words = start.split(" "); 675 | // By reversing the list we don't need to deal with the length that much. 676 | // It also allows to show a few more Iterable functions. 677 | Iterable reversedRemaining = words.reversed; 678 | while (reversedRemaining.length >= 2) { 679 | String secondToLast = reversedRemaining.elementAt(1); 680 | String last = reversedRemaining.first; 681 | String leadPair = "$secondToLast $last"; 682 | if (_db.containsKey(leadPair)) { 683 | // If the leadPair is in the database, it means that we have data to 684 | // continue from these two words. 685 | String beginning = reversedRemaining 686 | .skip(2) // 'last' and 'secondToLast' are already handled. 687 | .toList() // Iterable does not have `reversed`. 688 | .reversed // These are the remaining words. 689 | .join(" "); // Join them to have the beginning of the sentence. 690 | String end = generateSentenceStartingWith(secondToLast, last); 691 | return "$beginning $end"; 692 | } 693 | // We weren't able to continue from the last two words. Drop one, and try 694 | // again. 695 | reversedRemaining = reversedRemaining.skip(1); 696 | } 697 | return null; 698 | } 699 | ``` 700 | 701 | This function makes heavy use of 702 | [Iterables](https://api.dartlang.org/apidocs/channels/stable/dartdoc-viewer/dart:core.Iterable). 703 | Iterables are one of the most important and powerful data-types in Dart. They 704 | represent a (potentially infinite) sequence of values. Since they are so 705 | important, they have lots of methods to work with their data. For example, it 706 | features ways to filter the data (`where`, `skip`, `take`), to transform it 707 | (`map`, `reduce`, `fold`), or to aggregate it (`toList`, `toSet`, `any`, 708 | `every`). 709 | 710 | It is important to note that Iterables are _lazy_ in that all methods that 711 | return themselves an Iterable don't do any work until something iterates over 712 | the returned Iterable. Even then, they only do the work on the items that are 713 | requested. On the one hand, this makes it possible to apply these methods on 714 | infinite Iterables, and to chain methods without fear of allocating intermediate 715 | storage. On the other hand, one can accidentally execute the same 716 | transforming or filtering function multiple times. 717 | 718 | ```dart 719 | var list = [1, 2, 3]; 720 | list.map((x) => print(x)); // Doesn't do anything. 721 | var mappedIterable = list.map((x) { print(x); return x + 1; }); 722 | mappedIterable.forEach((x) { /* ignore */ }); // prints 1, 2, 3 723 | mappedIterable.forEach((x) { /* ignore */ }); // prints 1, 2, 3 again. 724 | ``` 725 | 726 | If one wants to use an Iterable multiple times, but doesn't want to execute the 727 | filtering or mapping functions multiple times, one should use `toList` to store 728 | the result in a List (which implements Iterable). 729 | 730 | 731 | 732 | Step 9 - Iterables of Sentences 733 | ------ 734 | 735 | We just discovered the powerful and omnipresent Iterables of Dart. In this 736 | step, we add a method to the sentence-generator that returns an Iterable of 737 | sentences. 738 | 739 | Creating an Iterable is surprisingly easy: 740 | 741 | ```dart 742 | /// Returns an Iterable of sentences. 743 | /// 744 | /// If the optional named argument [startingWith] is not provided or `null`, 745 | /// the Iterable contains random sentences. Otherwise it contains sentences 746 | /// starting with the given prefix (as if produced by [finishSentence]). 747 | Iterable generateSentences({String startingWith}) sync* { 748 | while (true) { 749 | if (startingWith == null) { 750 | // No optional argument given, or it was null. 751 | yield generateRandomSentence(); 752 | } else { 753 | yield finishSentence(startingWith); 754 | } 755 | } 756 | } 757 | ``` 758 | 759 | Note: this implementation is inefficient, since it calls `finishSentence` 760 | with the same prefix over and over again. A more efficient solution would do the 761 | prefix computation once, end then call `generateSentenceStartingWith` directly. 762 | 763 | This function has a named argument `startingWith`. This allows us to 764 | call the function either with or without the desired prefix. Just after the 765 | named argument we have a crucial token: the `sync*` modifier of the function. 766 | 767 | Function bodies that have this modifier are rewritten in such a way that they 768 | return an Iterable, and can provide values at `yield` points. Internally, the 769 | VM creates a state machine that keeps track of where it is. Whenever the 770 | returned Iterable requests a new item, the VM advances in the state machine 771 | until it encounters another `yield`. 772 | 773 | To illustrate how the resulting Iterable can be used, let's modify the `finish` 774 | command to filter sentences that are longer than 120 characters. 775 | 776 | ```dart 777 | default: 778 | if (text.startsWith("finish: ")) { 779 | var start = text.substring("finish: ".length); 780 | var sentence = sentenceGenerator 781 | .generateSentences(startingWith: start) 782 | .take(10000) // Make sure we don't run forever. 783 | .where((sentence) => sentence != null) 784 | .firstWhere((sentence) => sentence.length < 120, 785 | orElse: () => null); 786 | say(sentence == null ? "Unable to comply." : sentence); 787 | return; 788 | } 789 | ``` 790 | 791 | Make sure to test your implementation and have some fun. Here are some 792 | results I got out of my bot: 793 | 794 | myBot: finish: Alice was very 795 | Alice was very fond of pretending to be treated with respect . 796 | 797 | myBot: finish: The Congress shall have 798 | The Congress shall have somebody to talk about wasting IT . 799 | myBot: finish: The Congress shall have 800 | The Congress shall have somebody to talk nonsense . 801 | myBot: finish: The Congress shall have 802 | The Congress shall have Power, by and with almost no restrictions whatsoever . 803 | myBot: finish: The Congress may 804 | The Congress may from time to be treated with respect . 805 | 806 | Step 10 - Restructuring (optional) 807 | ------- 808 | 809 | Over time, programs grow, and even our toy example starts to get to a point 810 | where a little bit more structure would help. One of Dart's strengths is to 811 | grow nicely from small to bigger applications. An important and necessary 812 | feature for bigger application is to be able to create independent libraries. 813 | Dart has a public package-management system ([pub](http://pub.dartlang.org)), 814 | but in this code lab we will only split the program into libraries and part 815 | files. 816 | 817 | Our bot can be nicely split into two parts: the sentence-generator, and the 818 | IRC protocol handler. Let's start by creating a separate library for the 819 | sentence generator. Take the `SentenceGenerator` class and move it into a new 820 | file `sentence_generator.dart`. On the top of the file add your copyright 821 | header, a library declarative, and the imports that are required for the 822 | generator. 823 | 824 | ```dart 825 | // Copyright (c) 2015, the Dart project authors. Please see the AUTHORS file 826 | // for details. All rights reserved. Use of this source code is governed by a 827 | // BSD-style license that can be found in the LICENSE file. 828 | 829 | library dartlang.codelab.irc.sentence_generator; 830 | 831 | import 'dart:io' show File; 832 | import 'dart:math' show Random; 833 | 834 | class SentenceGenerator { 835 | ... 836 | } 837 | ``` 838 | 839 | While not necessary, we also used the opportunity to restrict the symbols that 840 | are shown by the imported libraries. 841 | 842 | To illustrate the use of part-files we move the IRC code into it's own part 843 | file `irc.dart`. 844 | 845 | ```dart 846 | // Copyright (c) 2015, the Dart project authors. Please see the AUTHORS file 847 | // for details. All rights reserved. Use of this source code is governed by a 848 | // BSD-style license that can be found in the LICENSE file. 849 | 850 | part of dartlang.codelab.irc; 851 | 852 | final RegExp ircMessageRegExp = 853 | new RegExp(r":([^!]+)!([^ ]+) PRIVMSG ([^ ]+) :(.*)"); 854 | 855 | ... 856 | 857 | void runIrcBot(SentenceGenerator generator) { 858 | Socket.connect("localhost", 6667) 859 | .then((socket) => handleIrcSocket(socket, generator)); 860 | } 861 | ``` 862 | 863 | 864 | In the `main.dart` file we now need to import the `sentence_generator.dart` 865 | library and include the part file: 866 | 867 | ```dart 868 | // Copyright (c) 2015, the Dart project authors. Please see the AUTHORS file 869 | // for details. All rights reserved. Use of this source code is governed by a 870 | // BSD-style license that can be found in the LICENSE file. 871 | 872 | library dartlang.codelab.irc; 873 | 874 | import 'dart:io' show Socket; 875 | import 'dart:convert' show UTF8, LineSplitter; 876 | 877 | import 'sentence_generator.dart' show SentenceGenerator; 878 | 879 | part 'irc.dart'; 880 | 881 | void main(arguments) { 882 | var generator = new SentenceGenerator(); 883 | arguments.forEach(generator.addBook); 884 | runIrcBot(generator); 885 | } 886 | ``` 887 | 888 | Step 11 - What next? 889 | ------- 890 | 891 | There are still lots of fun opportunities to improve this code, but if you want 892 | give IRC bots a break, here are some suggestions: 893 | 894 | * Read some of the [articles](http://dartlang.org/articles) on `dartlang.org`. 895 | * Do another code lab, for example 896 | [darrrt badge](https://www.dartlang.org/codelabs/darrrt/), or 897 | [server side code lab](https://www.dartlang.org/codelabs/server/). 898 | * Learn more about Dart from the 899 | [Dart tutorials](https://www.dartlang.org/docs/tutorials/) 900 | * Check out some of the [samples](https://www.dartlang.org/samples/) 901 | --------------------------------------------------------------------------------