├── .gitignore ├── README.md ├── config └── config.exs ├── lib └── complex.ex ├── mix.exs ├── priv └── video_screenshot.png ├── src ├── complex.c ├── erl_comm.c └── port.c └── test ├── complex_test.exs └── test_helper.exs /.gitignore: -------------------------------------------------------------------------------- 1 | /_build 2 | /deps 3 | erl_crash.dump 4 | *.ez 5 | priv 6 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | Complex 2 | ======= 3 | 4 | Example of ports as in http://erlang.org/doc/tutorial/c_port.html. This is an 5 | [ElixirSips](http://elixirsips.com) episode (194), and below you'll find the 6 | video and the script for the episode. Just playing around with fun stuff :) 7 | 8 | [![player_image](./priv/video_screenshot.png)](http://elixirsips.com/episodes/194_interoperability_ports.html) 9 | 10 | ## Episode 194: Interoperability - Ports 11 | 12 | You will eventually want to interoperate with another system. We've already 13 | played a little bit with ports, but this is an example of writing a C port that 14 | can talk directly to your Elixir system. For now, I'm mostly just going to 15 | rebuild what is in the Erlang tutorial for ports, but with Elixir in mind. 16 | 17 | ### Project 18 | 19 | We'll start a new project: 20 | 21 | ```sh 22 | mix new complex 23 | cd complex 24 | ``` 25 | 26 | ```sh 27 | vim lib/complex.ex 28 | ``` 29 | 30 | OK, so now it's worth talking a little bit about how ports work. Basically, we 31 | spin up a process inside our operating system and we send and receive messages 32 | with that process using ports. We've done this before with existing UNIX 33 | programs using standard in and standard out, but now we're going to do it with a 34 | custom protocol. 35 | 36 | ```elixir 37 | defmodule Complex do 38 | def start(external_program) do 39 | spawn(__MODULE__, :init, [external_program]) 40 | end 41 | end 42 | ``` 43 | 44 | OK, so we're going to call start with a string that represents an external 45 | program to run, and this will spawn a process that calls this module's `init` 46 | function with that argument. Nothing weird is happening yet. 47 | 48 | ```elixir 49 | defmodule Complex do 50 | #... 51 | def init(external_program) do 52 | Process.register(self, :complex) 53 | Process.flag(:trap_exit, true) 54 | port = Port.open({:spawn, external_program}, [packet: 2]) 55 | loop(port) 56 | end 57 | end 58 | ``` 59 | 60 | Now the init function just registers this process as the atom `complex` and says 61 | it wants to trap exits. Then it opens up a port. This port spawns an external 62 | program, which is the binary whose name is the string we passed in to 'start'. 63 | We'll use a 2 byte length indicator to communicate between the two systems. The 64 | Erlang VM will automatically add this indicator when it's talking to the C 65 | program, but we'll have to handle this bit explicitly from the C side. 66 | 67 | Finally, we just loop on the port we opened. 68 | 69 | Now, before we move on I want to show you the C code that we'll be calling out 70 | to. It's intentionally simple, and again this is just the example used in the 71 | erlang user's guide for the C Port tutorial. We support 2 functions, `foo` and 72 | `bar`. I think you can understand the C code even if you aren't a whiz with C. 73 | 74 | ```sh 75 | mkdir src 76 | vim src/complex.c 77 | ``` 78 | 79 | ```c 80 | /* complex.c */ 81 | 82 | int foo(int x) { 83 | return x+1; 84 | } 85 | 86 | int bar(int y) { 87 | return y*2; 88 | } 89 | ``` 90 | 91 | Alright, so you've got that to keep in mind as what we're integrating with. 92 | 93 | 94 | Next, we'll implement the Elixir side of our API. We'll provide 2 functions, 95 | `foo` and `bar`, that take an integer argument and pass it on to the port, 96 | returning whatever it returns. 97 | 98 | ```elixir 99 | defmodule Complex do 100 | #... 101 | def foo(x) do 102 | call_port({:foo, x}) 103 | end 104 | 105 | def bar(y) do 106 | call_port({:bar, y}) 107 | end 108 | end 109 | ``` 110 | 111 | Then comes `call_port`: 112 | 113 | ```elixir 114 | defmodule Complex do 115 | #... 116 | def call_port(message) do 117 | send(:complex, {:call, self, message}) 118 | receive do 119 | {:complex, result} -> result 120 | end 121 | end 122 | end 123 | ``` 124 | 125 | This just sends a 3-tuple to the Elixir process we spawned and registered under 126 | the name `complex`. The second element in the 3-tuple is our pid. It then 127 | blocks until our process receives a message that matches the 2-tuple pattern. 128 | Finally, it returns the result when it receives it. 129 | 130 | Now we can get to the loop itself. 131 | 132 | ```elixir 133 | defmodule Complex do 134 | #... 135 | def loop(port) do 136 | receive do 137 | {:call, caller, message} -> 138 | send(port, {self, {:command, encode(message)}}) 139 | receive do 140 | {^port, {:data, data}} -> 141 | send(caller, {:complex, decode(data)}) 142 | end 143 | loop(port) 144 | end 145 | end 146 | ``` 147 | 148 | OK, so this is the core piece that wires this process together with the port. 149 | If we receive a message that tells us to make a call, we send the port a 150 | 2-tuple. The first element is our pid, and the second is a 2-tuple whose first 151 | element is the atom `command` and whose second element is the message, encoded. 152 | We then block to receive data from the port. This is sent as a 2-tuple whose 153 | first element is the atom `data`, inside a 2-tuple whose first element is the 154 | port we opened, so we can be sure the message is coming from where we're 155 | expecting it to come from. Once we've received this message from the port, we 156 | send the caller the result, decoding it on the way out. The elixir component 157 | that used our API will just deal with nicely formatted elixir data, and all of 158 | the encoding and decoding lives in this piece of code. 159 | 160 | Now that I've talked through all of that, we're going to add a couple of things 161 | that allow us to explicitly stop the port or handle a crash from the C code 162 | moderately gracefully: 163 | 164 | 165 | ```elixir 166 | defmodule Complex do 167 | #... 168 | def loop(port) do 169 | receive do 170 | {:call, caller, message} -> 171 | send(port, {self, {:command, encode(message)}}) 172 | receive do 173 | {^port, {:data, data}} -> 174 | send(caller, {:complex, decode(data)}) 175 | end 176 | loop(port) 177 | :stop -> 178 | send(port, {self, :close}) 179 | receive do 180 | {^port, :closed} -> 181 | exit(:normal) 182 | end 183 | {'EXIT', ^port, _reason} -> 184 | exit(:port_terminated) 185 | end 186 | end 187 | ``` 188 | 189 | Now all that's left is to define the `encode` and `decode` functions. 190 | 191 | ```elixir 192 | defmodule Complex do 193 | #... 194 | # OK so we'll encode the 'foo' command as 1, and the 'bar' command as 2. It's 195 | # important to realize we're going to be sending raw bytes here, so this is 196 | # the bit of the code that will break down for values greater than 255. 197 | def encode({:foo, x}), do: [1, x] 198 | def encode({:bar, y}), do: [2, y] 199 | 200 | # We'll decode messages that come back - they're just a list of 1 byte, and we 201 | # want to treat that like the integer that it is in this case. 202 | def decode([int]), do: int 203 | end 204 | ``` 205 | 206 | Finally we'll add the `stop` function that tells the process to close the port, 207 | which we've already supported in the loop: 208 | 209 | ```elixir 210 | def stop do 211 | send(:complex, :stop) 212 | end 213 | ``` 214 | 215 | Alright, all that code put together looks like this: 216 | 217 | ```elixir 218 | defmodule Complex do 219 | def start(external_program) do 220 | spawn(__MODULE__, :init, [external_program]) 221 | end 222 | 223 | def stop do 224 | send(:complex, :stop) 225 | end 226 | 227 | def foo(x) do 228 | call_port({:foo, x}) 229 | end 230 | 231 | def bar(y) do 232 | call_port({:bar, y}) 233 | end 234 | 235 | def init(external_program) do 236 | Process.register(self, :complex) 237 | Process.flag(:trap_exit, true) 238 | port = Port.open({:spawn, external_program}, [packet: 2]) 239 | loop(port) 240 | end 241 | 242 | 243 | def call_port(message) do 244 | send(:complex, {:call, self, message}) 245 | receive do 246 | {:complex, result} -> result 247 | end 248 | end 249 | 250 | def loop(port) do 251 | receive do 252 | {:call, caller, message} -> 253 | send(port, {self, {:command, encode(message)}}) 254 | receive do 255 | {^port, {:data, data}} -> 256 | send(caller, {:complex, decode(data)}) 257 | end 258 | loop(port) 259 | :stop -> 260 | send(port, {self, :close}) 261 | receive do 262 | {^port, :closed} -> 263 | exit(:normal) 264 | end 265 | {'EXIT', ^port, _reason} -> 266 | exit(:port_terminated) 267 | end 268 | end 269 | 270 | def encode({:foo, x}), do: [1, x] 271 | def encode({:bar, y}), do: [2, y] 272 | 273 | def decode([int]), do: int 274 | end 275 | ``` 276 | 277 | Now we've defined a couple of C functions but that's not enough to actually 278 | communicate with our system here - there's no concept of dealing with standard 279 | in and standard out, which is what we're going to use to wire these things 280 | together. For this, we have a couple of utilities that are also provided in the 281 | erlang interoperability tutorial. We'll talk through them fairly quickly: 282 | 283 | ```sh 284 | vim src/erl_comm.c 285 | ``` 286 | 287 | ```c 288 | /* erl_comm.c */ 289 | 290 | typedef unsigned char byte; 291 | 292 | read_cmd(byte *buf) 293 | { 294 | int len; 295 | 296 | // if we can't read 2 bytes we freak out 297 | if (read_exact(buf, 2) != 2) 298 | return(-1); 299 | // those 2 bytes tell us how long this message is. We then read the message 300 | // This is the bit where I said the elixir side has it easy but the c code has 301 | // to be explicit. 302 | len = (buf[0] << 8) | buf[1]; 303 | return read_exact(buf, len); 304 | } 305 | 306 | // when we write a message out, we write the 2 byte length indicator 307 | // as well, so that the erlang program's knowledge that we're using 308 | // 2 byte length indicators is satisfied. 309 | write_cmd(byte *buf, int len) 310 | { 311 | byte li; 312 | 313 | li = (len >> 8) & 0xff; 314 | write_exact(&li, 1); 315 | 316 | li = len & 0xff; 317 | write_exact(&li, 1); 318 | 319 | return write_exact(buf, len); 320 | } 321 | 322 | // This is just a function to read a certain number of bytes into a buffer from 323 | // standard input (0) 324 | read_exact(byte *buf, int len) 325 | { 326 | int i, got=0; 327 | 328 | do { 329 | if ((i = read(0, buf+got, len-got)) <= 0) 330 | return(i); 331 | got += i; 332 | } while (got 0) { 373 | // Once we've gotten a command, we know the function is determined by the 374 | // first byte and its argument is the second byte 375 | fn = buf[0]; 376 | arg = buf[1]; 377 | 378 | if (fn == 1) { 379 | // We said that 1 was foo 380 | res = foo(arg); 381 | } else if (fn == 2) { 382 | // and 2 was bar 383 | res = bar(arg); 384 | } 385 | 386 | // We set the response value in the first byte of our buffer 387 | buf[0] = res; 388 | // and we write that byte back to the elixir program over standard out 389 | write_cmd(buf, 1); 390 | } 391 | } 392 | ``` 393 | 394 | OK, so that's the whole shebang. But does it work? We can build it with gcc: 395 | 396 | ```sh 397 | mkdir priv 398 | gcc -o priv/extprg src/complex.c src/erl_comm.c src/port.c 399 | ``` 400 | 401 | That will output it to the priv directory as a file named "extprg". We can then 402 | spin up iex and try it out: 403 | 404 | ```sh 405 | iex -S mix 406 | ``` 407 | 408 | ``` 409 | Complex.start("priv/extprg") 410 | Complex.foo(1) 411 | Complex.bar(5) 412 | ``` 413 | 414 | And since we know that this is using 1 byte for communicating the result, we can 415 | do terrible things like send a valid command that cannot possibly receive a 416 | valid response! 417 | 418 | ``` 419 | Complex.foo(255) 420 | ``` 421 | 422 | Right, so foo increments by 1. But then we get a single byte. So this 423 | is...right...sort of. Ditto for bar: 424 | 425 | ``` 426 | Complex.bar(128) 427 | ``` 428 | 429 | And of course if we try to call a function with an integer that can't be 430 | represented by a single byte, bad things will happen: 431 | 432 | ``` 433 | iex(5)> Complex.foo(256) 434 | 435 | 19:59:08.259 [error] Bad value on output port 'priv/extprg' 436 | ``` 437 | 438 | Hey look we crashed everything! Don't make your port return a value that's 439 | outside of the range of acceptable values you specified when spawning it! 440 | 441 | If we start back up, we can see the program is running with ps: 442 | 443 | ((( do it ))) 444 | ((( then in a new terminal tab ))) 445 | 446 | ```sh 447 | ps ax|grep extprg 448 | ``` 449 | 450 | Now if we tell the module to stop the port, this OS process will go away: 451 | 452 | ``` 453 | Complex.stop 454 | ``` 455 | 456 | ```sh 457 | ps ax|grep extprg 458 | ``` 459 | 460 | ### Summary 461 | 462 | Anyway, that's it. Today we saw how you can build a C program to interoperate 463 | with your Elixir system, and we saw how you can make things go horribly wrong if 464 | you say it'll operate in one particular way and it doesn't, with respect to 465 | sending back 1 byte responses. I have personally used this facility to have 466 | erlang manage some C code that was using opencv to resize images, because 467 | writing a webserver in C to front for that function was pretty horrible. I hope 468 | you enjoyed it - see you soon! 469 | 470 | ### Resources 471 | 472 | - [Erlang Tutorial on C ports](http://erlang.org/doc/tutorial/c_port.html) 473 | -------------------------------------------------------------------------------- /config/config.exs: -------------------------------------------------------------------------------- 1 | # This file is responsible for configuring your application 2 | # and its dependencies with the aid of the Mix.Config module. 3 | use Mix.Config 4 | 5 | # This configuration is loaded before any dependency and is restricted 6 | # to this project. If another project depends on this project, this 7 | # file won't be loaded nor affect the parent project. For this reason, 8 | # if you want to provide default values for your application for third- 9 | # party users, it should be done in your mix.exs file. 10 | 11 | # Sample configuration: 12 | # 13 | # config :logger, :console, 14 | # level: :info, 15 | # format: "$date $time [$level] $metadata$message\n", 16 | # metadata: [:user_id] 17 | 18 | # It is also possible to import configuration files, relative to this 19 | # directory. For example, you can emulate configuration per environment 20 | # by uncommenting the line below and defining dev.exs, test.exs and such. 21 | # Configuration from the imported file will override the ones defined 22 | # here (which is why it is important to import them last). 23 | # 24 | # import_config "#{Mix.env}.exs" 25 | -------------------------------------------------------------------------------- /lib/complex.ex: -------------------------------------------------------------------------------- 1 | defmodule Complex do 2 | ## Public API 3 | def start(external_program) do 4 | spawn(__MODULE__, :init, [external_program]) 5 | end 6 | 7 | def stop do 8 | send(:complex, :stop) 9 | end 10 | 11 | def foo(x) do 12 | call_port({:foo, x}) 13 | end 14 | 15 | def bar(y) do 16 | call_port({:bar, y}) 17 | end 18 | 19 | def call_port(message) do 20 | send(:complex, {:call, self, message}) 21 | receive do 22 | {:complex, result} -> result 23 | end 24 | end 25 | 26 | ## Server API 27 | 28 | def init(external_program) do 29 | Process.register(self, :complex) 30 | Process.flag(:trap_exit, true) 31 | port = Port.open({:spawn, external_program}, [packet: 2]) 32 | loop(port) 33 | end 34 | 35 | ## Private bits 36 | def loop(port) do 37 | receive do 38 | {:call, caller, message} -> 39 | send(port, {self, {:command, encode(message)}}) 40 | receive do 41 | {^port, {:data, data}} -> 42 | send(caller, {:complex, decode(data)}) 43 | end 44 | loop(port) 45 | :stop -> 46 | send(port, {self, :close}) 47 | receive do 48 | {^port, :closed} -> 49 | exit(:normal) 50 | end 51 | {'EXIT', ^port, _reason} -> 52 | exit(:port_terminated) 53 | end 54 | end 55 | 56 | def encode({:foo, x}), do: [1, x] 57 | def encode({:bar, y}), do: [2, y] 58 | 59 | def decode([int]), do: int 60 | end 61 | -------------------------------------------------------------------------------- /mix.exs: -------------------------------------------------------------------------------- 1 | defmodule Complex.Mixfile do 2 | use Mix.Project 3 | 4 | def project do 5 | [app: :complex, 6 | version: "0.0.1", 7 | elixir: "~> 1.0", 8 | build_embedded: Mix.env == :prod, 9 | start_permanent: Mix.env == :prod, 10 | deps: deps] 11 | end 12 | 13 | # Configuration for the OTP application 14 | # 15 | # Type `mix help compile.app` for more information 16 | def application do 17 | [applications: [:logger]] 18 | end 19 | 20 | # Dependencies can be Hex packages: 21 | # 22 | # {:mydep, "~> 0.3.0"} 23 | # 24 | # Or git/path repositories: 25 | # 26 | # {:mydep, git: "https://github.com/elixir-lang/mydep.git", tag: "0.1.0"} 27 | # 28 | # Type `mix help deps` for more examples and options 29 | defp deps do 30 | [] 31 | end 32 | end 33 | -------------------------------------------------------------------------------- /priv/video_screenshot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knewter/complex/df7d0fc162ae03822b917b2a92c64c685bd14445/priv/video_screenshot.png -------------------------------------------------------------------------------- /src/complex.c: -------------------------------------------------------------------------------- 1 | int foo(int x) { 2 | return x+1; 3 | } 4 | 5 | int bar(int y) { 6 | return y*2; 7 | } 8 | -------------------------------------------------------------------------------- /src/erl_comm.c: -------------------------------------------------------------------------------- 1 | typedef unsigned char byte; 2 | 3 | read_cmd(byte *buf) 4 | { 5 | int len; 6 | 7 | // if we can't read 2 bytes we freak out 8 | if (read_exact(buf, 2) != 2) 9 | return(-1); 10 | // those 2 bytes tell us how long this message is. We then read the message 11 | // This is the bit where I said the elixir side has it easy but the c code has 12 | // to be explicit. 13 | len = (buf[0] << 8) | buf[1]; 14 | return read_exact(buf, len); 15 | } 16 | 17 | // when we write a message out, we write the 2 byte length indicator 18 | // as well, so that the erlang program's knowledge that we're using 19 | // 2 byte length indicators is satisfied. 20 | write_cmd(byte *buf, int len) 21 | { 22 | byte li; 23 | 24 | li = (len >> 8) & 0xff; 25 | write_exact(&li, 1); 26 | 27 | li = len & 0xff; 28 | write_exact(&li, 1); 29 | 30 | return write_exact(buf, len); 31 | } 32 | 33 | // This is just a function to read a certain number of bytes into a buffer from 34 | // standard input (0) 35 | read_exact(byte *buf, int len) 36 | { 37 | int i, got=0; 38 | 39 | do { 40 | if ((i = read(0, buf+got, len-got)) <= 0) 41 | return(i); 42 | got += i; 43 | } while (got 0) { 9 | // Once we've gotten a command, we know the function is determined by the 10 | // first byte and its argument is the second byte 11 | fn = buf[0]; 12 | arg = buf[1]; 13 | 14 | if (fn == 1) { 15 | // We said that 1 was foo 16 | res = foo(arg); 17 | } else if (fn == 2) { 18 | // and 2 was bar 19 | res = bar(arg); 20 | } 21 | 22 | // We set the response value in the first byte of our buffer 23 | buf[0] = res; 24 | // and we write that byte back to the elixir program over standard out 25 | write_cmd(buf, 1); 26 | } 27 | } 28 | -------------------------------------------------------------------------------- /test/complex_test.exs: -------------------------------------------------------------------------------- 1 | defmodule ComplexTest do 2 | use ExUnit.Case 3 | 4 | test "the truth" do 5 | assert 1 + 1 == 2 6 | end 7 | end 8 | -------------------------------------------------------------------------------- /test/test_helper.exs: -------------------------------------------------------------------------------- 1 | ExUnit.start() 2 | --------------------------------------------------------------------------------