├── .gitignore ├── .travis.yml ├── Makefile ├── README.md ├── include ├── builtins.lfe └── operators.lfe ├── lfe.config ├── python ├── lfe │ ├── decoders.py │ ├── encoders.py │ ├── erlang.py │ ├── init.py │ ├── logger.py │ ├── numpysupl.py │ └── obj.py └── requirements.txt ├── rebar.config ├── rebar.lock ├── resources ├── images │ ├── Python-logo-notext-small.png │ └── Python-logo-notext.png └── make │ ├── common.mk │ ├── dist.mk │ └── python.mk ├── src ├── py-app.lfe ├── py-config.lfe ├── py-logger.lfe ├── py-sched.lfe ├── py-sup.lfe ├── py-util.lfe ├── py.app.src └── py.lfe └── test ├── unit-py-tests.lfe └── unit-py-util-tests.lfe /.gitignore: -------------------------------------------------------------------------------- 1 | _build 2 | *.sublime-project 3 | *.sublime-workspace 4 | *.beam 5 | .eunit 6 | debug-* 7 | ebin/* 8 | *.dump 9 | .rebar 10 | python/.venv 11 | __pycache__ 12 | log/* 13 | python/get-pip.py 14 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: erlang 2 | before_script: "make get-lfetool" 3 | script: "make check" 4 | notifications: 5 | #irc: "irc.freenode.org#YOUR-PROJECT-CHANNEL" 6 | recipients: 7 | #- YOU@YOUR.DOMAIN 8 | otp_release: 9 | - 17.1 10 | - R16B03 11 | - R15B03 12 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | PROJECT=py 2 | 3 | all: compile python 4 | 5 | include resources/make/common.mk 6 | include resources/make/dist.mk 7 | include resources/make/python.mk 8 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # py 2 | 3 | *Distributed Python for the Erlang Ecosystem* 4 | 5 | 6 | 7 | 8 | ## Table of Contents 9 | 10 | * [Introduction](#introduction-) 11 | * [Requirements](#requirements-) 12 | * [Setup](#setup-) 13 | * [API Usage](#api-usage-) 14 | * [Metadata](#metadata-) 15 | * [Module Level](#module-level-) 16 | * [Calling Functions](#calling-functions-) 17 | * [Module Constants](#module-constants-) 18 | * [Objects](#objects-) 19 | * [Instantiation](#instantiation-) 20 | * [Calling Methods](#calling-methods-) 21 | * [Attribute Values](#attribute-values-) 22 | * [Operations on Objects](#operations-on-objects-) 23 | * [Experimental Dotted Name Support](#experimental-dotted-name-support-) 24 | * [ErlPort Pass-Through](#erlport-pass-through-) 25 | * [Builtins](#builtins-) 26 | * [Operators](#operators-) 27 | * [Non-Python Additions](#non-python-additions-) 28 | * [Missing Functions](#missing-functions-) 29 | * [Erlang](#erlang-) 30 | * [Architecture](#architecture-) 31 | * [Overview](#overview-) 32 | * [Erlang Components](#erlang-components-) 33 | * [Python Components](#python-components-) 34 | * [Controlling the Python Servers](#controlling-the-python-servers-) 35 | * Erlang Configuration 36 | * Python Configuration 37 | * Start, Stop, and Restart 38 | * Dynamically Adding More Python Servers 39 | * Automatic Restarts 40 | * Python Server Schedulers 41 | * [Executing Code in Parallel](#executing-code-in-parallel-) 42 | * Identical Calls 43 | * Scatter/Gather 44 | * [Distributed Python](#distributed-python-) 45 | * Starting Remote Python Servers 46 | * Executing Python on Remote Servers 47 | 48 | ## Introduction [↟](#table-of-contents) 49 | 50 | This project provides two key features: 51 | 52 | 1. An easier interface to Python, wrapping 53 | [ErlPort](http://erlport.org/docs/python.html) calls. This lets you do the 54 | following very easily: 55 | * Make module-level calls 56 | * Get module-level constants 57 | * Instantiate objects 58 | * Call object methods 59 | * Get object attributes 60 | * Call builtins and operators with convenient wrappers 61 | 1. A means of running Python in a simple distributed context using all the 62 | well-known strengths of Erlang (fault-tolerance, scalability, 63 | concurrency, soft real-time, etc.). 64 | 65 | What LFE py is not: 66 | 67 | * A PaaS; if that's what you're interested in, please take a look at 68 | [CloudI](http://cloudi.org/). 69 | * A framework for pipelining jobs across distributed data (e.g., mapreduce). 70 | For that, see the [Disco project](http://discoproject.org/). 71 | * A language-agnostic, general-purpose ports server. LFE py is, in fact, 72 | built on one of those: [ErlPort](http://erlport.org/)! 73 | 74 | LFE py was originally part of the 75 | [lsci project](https://github.com/lfex/lsci), but was split out due to the 76 | ErlPort/Python-wrapping code being generally useful for all sorts of projects, 77 | not just scientific computing in Erlang/LFE. 78 | This bit of background should give further insight 79 | into the use cases LFE py was intended to address: scientific and (more 80 | recently) general computing in Python from the Erlang VM, with a focus on 81 | interactive workflows common in academic, research, and startup R&D 82 | environments. Just the sort of thing that 83 | [IPython](http://ipython.org/) excels at, minus the GUIs :-) 84 | 85 | 86 | ## Requirements [↟](#table-of-contents) 87 | 88 | To use py, you need the following: 89 | 90 | * [lfetool](http://docs.lfe.io/quick-start/1.html) and [rebar](https://github.com/rebar/rebar) 91 | (used by ``make`` targets to automatically set ``ERL_LIBS`` for deps) 92 | * [Python 3](https://www.python.org/downloads/) 93 | * ``wget`` (used to download ``get-pip.py``) 94 | 95 | 96 | ## Setup [↟](#table-of-contents) 97 | 98 | For now, just run it from a git clone: 99 | 100 | ```bash 101 | $ git clone git@github.com:lfex/py.git 102 | $ cd py 103 | $ make 104 | ``` 105 | 106 | Activate the Python virtualenv that was created by the ``make`` command you 107 | just ran. Then start up the LFE REPL: 108 | 109 | ```bash 110 | $ . ./python/.venv/bin/activate 111 | $ make repl-no-deps 112 | ``` 113 | 114 | Note that the ``repl`` and ``repl-no-deps`` make targets automatically start up 115 | the py (and thus ErlPort) Erlang Python server. If you run the REPL without 116 | these ``make`` targets, you'll need to manually start things: 117 | 118 | ```bash 119 | $ lfetool repl lfe -s py 120 | ``` 121 | 122 | 123 | ## API Usage [↟](#table-of-contents) 124 | 125 | Below we show some basic usage of py from both LFE and Erlang. In a 126 | separate section a list of docs are linked showing detailed usage of wrapped 127 | libraries. 128 | 129 | 130 | ### Metadata [↟](#table-of-contents) 131 | 132 | First things first: let's make sure that you have the appropriate versions 133 | of things -- in particular, let's confirm that you're running Python 3: 134 | 135 | ```cl 136 | > (py-util:get-versions) 137 | (#(erlang "17") 138 | #(emulator "6.2") 139 | #(driver-version "3.1") 140 | #(lfe "0.9.0") 141 | #(erlport "0.9.8") 142 | #(py "0.0.1") 143 | #(python 144 | ("3.4.2 (v3.4.2:ab2c023a9432, Oct 5 2014, 20:42:22)" 145 | "[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]"))) 146 | ``` 147 | 148 | 149 | ### Module Level [↟](#table-of-contents) 150 | 151 | The following sub-sections describe module-level operations. 152 | 153 | 154 | #### Calling Functions [↟](#table-of-contents) 155 | 156 | ```cl 157 | > (py:func 'os 'getcwd) 158 | "/Users/yourname/lab/erlang/py" 159 | > (py:func 'datetime.datetime 'now) 160 | #("datetime" #(2014 12 23 16 57 11 693773 undefined)) 161 | ``` 162 | 163 | Note that strings in arguments need to be converted to binary: 164 | 165 | ```cl 166 | > (py:func 'os.path 'isfile '(#b("/tmp"))) 167 | false 168 | > (py:func 'os.path 'isdir '(#b("/tmp"))) 169 | true 170 | ``` 171 | 172 | Keyword arguments are passed as proplists, e.g., 173 | ``'(#(key1 val1) #(key2 val2))``. In the next example we'll pass a string (as 174 | a binary) which represents a binary number. We'll give ``int`` the keyword of 175 | ``base``, since we're not going to use the default decimal base (10): 176 | 177 | ```cl 178 | (py:func 'builtins 'int '(#b("101010")) '(#(base 2))) 179 | 42 180 | ``` 181 | 182 | 183 | #### Module Constants [↟](#table-of-contents) 184 | 185 | ```cl 186 | > (py:const 'math 'pi) 187 | 3.141592653589793 188 | ``` 189 | 190 | Optionally, you may provide a type: 191 | 192 | ```cl 193 | > (py:const 'math 'pi 'float) 194 | 3.141592653589793 195 | > (py:const 'math 'pi 'int) 196 | 3 197 | > (py:const 'math 'pi 'str) 198 | "3.141592653589793" 199 | ``` 200 | 201 | 202 | ### Objects [↟](#table-of-contents) 203 | 204 | The following sections describe how to work with Python objects. 205 | 206 | 207 | #### Instantiation [↟](#table-of-contents) 208 | 209 | With no arguments passed to the constructor: 210 | 211 | ```cl 212 | > (py:init 'builtins 'dict) 213 | #("dict" ()) 214 | > (py:init 'collections 'UserDict) 215 | #("UserDict" ()) 216 | ``` 217 | 218 | With args: 219 | 220 | ```cl 221 | > (py:init 'datetime 'date '(1923 4 2)) 222 | #("date" #(1923 4 1)) 223 | ``` 224 | 225 | 226 | #### Calling Methods [↟](#table-of-contents) 227 | 228 | To call a method, we need an object. Let's return to the date example 229 | above: 230 | 231 | ```cl 232 | > (set now (py:func 'datetime.datetime 'now)) 233 | #("datetime" #(2014 12 23 23 14 37 677463 undefined)) 234 | ``` 235 | 236 | The tuple representing a date time object has been saved as the ``now`` 237 | variable in the REPL. Let's call some methods: 238 | 239 | ```cl 240 | > (py:method now 'strftime '(#b("%Y.%m.%d %H:%M:%S"))) 241 | "2014.12.23 23:14:37" 242 | ``` 243 | 244 | #### Attribute Values [↟](#table-of-contents) 245 | 246 | Continuing with that same object: 247 | 248 | ```cl 249 | > (py:attr now 'year) 250 | 2014 251 | > (py:attr now 'microsecond) 252 | 677463 253 | ``` 254 | 255 | 256 | #### Operations on Objects [↟](#table-of-contents) 257 | 258 | Let's get another time ... and give our other variable a better name: 259 | 260 | ```cl 261 | > (set later (py:func 'datetime.datetime 'now)) 262 | #("datetime" #(2014 12 23 23 21 25 714474 undefined)) 263 | > (set earlier now) 264 | #("datetime" #(2014 12 23 23 14 37 677463 undefined)) 265 | ``` 266 | 267 | Let's use the two objects in a calculation: 268 | 269 | ```cl 270 | > (set diff (py:sub later earlier)) 271 | #("timedelta" 0 408 37011) 272 | > (py:attr diff 'seconds) 273 | 408 274 | ``` 275 | 276 | We can get that in minutes in LFE/Erlang: 277 | 278 | ```cl 279 | > (/ (py:attr diff 'seconds) 60) 280 | 6.8 281 | ``` 282 | 283 | 284 | ### Experimental Dotted Name Support [↟](#table-of-contents) 285 | 286 | There is currently tentative support for dotted names in the following 287 | calls: 288 | 289 | * ``(py:const ...)`` 290 | * ``(py:func ...)`` 291 | * ``(py:init ...)`` 292 | 293 | Examples: 294 | 295 | ```cl 296 | > (py:const 'math.pi) 297 | 3.141592653589793 298 | ``` 299 | 300 | ```cl 301 | > (py:func 'datetime.datetime.now) 302 | #("datetime" #(2014 12 29 0 47 30 334180 undefined)) 303 | > (py:func 'math.pow '(2 16)) 304 | 65536.0 305 | > (py:func 'builtins.int '(#b("101010")) '(#(base 2))) 306 | 42 307 | ``` 308 | 309 | ```cl 310 | > (py:init 'collections.UserDict) 311 | #("UserDict" ()) 312 | > (py:init 'collections.UserDict '() '(#("a" 1) #("b" 2))) 313 | #("UserDict" (#("b" 2) #("a" 1))) 314 | ``` 315 | 316 | Though this is offered, it isn't really encouraged, since there will 317 | necessarily be inconsistencies in usage. Dotted notation can be used with 318 | ``const``, ``func``, and ``init`` but not with objects (i.e., not with 319 | ``method`` and ``attr``). This mixing of styles could get confusing and 320 | you may think you have a bug in your code when, in fact, you just can't use 321 | dotted names with instantiated objects (since in LFE it's just a variable 322 | name, not an actual object). 323 | 324 | Finally, with this code in place, many more function calls are incurred 325 | regardless of whether dotted notation is used. For this reason and the 326 | discouragement against use above, this feature is on the short list for 327 | getting axed. Don't count on it being around ... 328 | 329 | 330 | ### ErlPort Pass-Through [↟](#table-of-contents) 331 | 332 | If for any reason you would like to skip the LFE py wrappers and call directly 333 | to ErlPort, you may do so: 334 | 335 | ```cl 336 | > (py:pycall 'datetime 'datetime.now) 337 | ("datetime" #(2014 12 25 20 44 4 673150 undefined)) 338 | > (py:pycall 'datetime 'datetime '(1923 4 2 0 0 0)) 339 | #("datetime" #(1923 4 2 0 0 0 0 undefined)) 340 | ``` 341 | 342 | These make direct calls to ErlPort's ``python:call`` function, but supply the 343 | required Python server ``pid`` behind the scenes. 344 | 345 | 346 | ### Builtins [↟](#table-of-contents) 347 | 348 | In several of the examples above, we made calls to the ``builtins`` module 349 | like so: 350 | 351 | ```cl 352 | > (py:init 'builtins 'dict) 353 | #("dict" ()) 354 | > (py:func 'builtins 'int '(#b("101010")) '(#(base 2))) 355 | 42 356 | ``` 357 | 358 | LFE py actually provides wrappers for these, making such calls much easier. 359 | 360 | ```cl 361 | > (py:dict) 362 | #("dict" ()) 363 | > (py:dict '(#("a" 1) #("b" 2))) 364 | #("dict" (#("b" 2) #("a" 1))) 365 | > (py:int #b("101010") '(#(base 2))) 366 | 42 367 | ``` 368 | 369 | More examples: 370 | 371 | ```cl 372 | > (py:any '(true true false false false true)) 373 | true 374 | > (py:all '(true true false false false true)) 375 | false 376 | > (py:all '(true true true)) 377 | true 378 | > (py:pow 6 42) 379 | 481229803398374426442198455156736 380 | > (py:round 0.666666667 5) 381 | 0.66667 382 | > (py:range 7 42) 383 | #($erlport.opaque python 384 | #B(128 2 99 95 95 98 117 105 108 ...)) 385 | > (py:len (py:range 7 42)) 386 | 35 387 | > (py:pylist (py:range 7 42)) 388 | (7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 389 | 31 32 33 34 35 36 ...) 390 | > (erlang:length (py:pylist (py:range 7 42))) 391 | 35 392 | ``` 393 | 394 | 395 | ### Operators [↟](#table-of-contents) 396 | 397 | It will often be the case that you want to operate on the Python data 398 | structures obtained via the LFE py calls directly in Python, without 399 | translating them into LFE/Erlang. The Python ``operator`` module is wrapped 400 | for your convenience in these cases. 401 | 402 | Examples: 403 | 404 | ```cl 405 | > (py:add 37 5) 406 | 42 407 | > (py:mul 7 6) 408 | 42 409 | > (py:sub -108 -150) 410 | 42 411 | > (py:truediv 462 11) 412 | 42.0 413 | > (py:floordiv 462 11) 414 | 42 415 | ``` 416 | 417 | Equality: 418 | 419 | ```cl 420 | > (py:gt 7 6) 421 | true 422 | > (py:le 7 6) 423 | false 424 | > (py:eq 42 42) 425 | true 426 | ``` 427 | 428 | Bitwise operations: 429 | 430 | ```cl 431 | > (py:and- 60 13) 432 | 12 433 | > (py:or- 60 13) 434 | 61 435 | > (py:xor 60 13) 436 | 49 437 | > (py:inv 60) 438 | -61 439 | > (py:rshift 60 2) 440 | 15 441 | > (py:lshift 60 2) 442 | 240 443 | ``` 444 | 445 | 446 | #### Non-Python Additions [↟](#table-of-contents) 447 | 448 | So as not to stomp on the LFE function ``(list ...)``, the Python ``list`` 449 | builtin has been aliased to the ``pylist`` function, e.g.: 450 | 451 | ```cl 452 | > (py:pylist) 453 | () 454 | > (py:pylist '(1 2 3 4)) 455 | (1 2 3 4) 456 | ``` 457 | 458 | ``(py:dir ...)`` and ``(py:vars ...)`` return elided lists, so you won't see 459 | complete results that are longer than 28 elements. If you wish to see 460 | everything, you may call ``(py:pdir)`` and ``(py:pvars)``, respectively. 461 | 462 | ``(py:repr)`` provides wrapping for the Python builtin ``repr``. If you would 463 | like to see a representation of the pickled Python data in LFE, you may use 464 | the ``(py:prepr)`` function. 465 | 466 | 467 | ### Missing Functions [↟](#table-of-contents) 468 | 469 | Any Python function that does in-place modification of objects is not included. 470 | LFE py will eventually provide analogs for in-place functions that return a new 471 | data set or object. 472 | 473 | 474 | ### Erlang [↟](#table-of-contents) 475 | 476 | We can, of course, do all of this from Erlang: 477 | 478 | ```bash 479 | $ make shell-no-deps 480 | ``` 481 | 482 | ```erlang 483 | 1> 'py-util':'get-versions'(). 484 | [{erlang,"17"}, 485 | {emulator,"6.2"}, 486 | {'driver-version',"3.1"}, 487 | {lfe,"0.9.0"}, 488 | {'lfe-py',"0.0.1"}, 489 | {python,["3.4.2 (v3.4.2:ab2c023a9432, Oct 5 2014, 20:42:22)", 490 | "[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]"]}] 491 | 2> py:func(os, getcwd). 492 | "/Users/oubiwann/Dropbox/lab/erlang/py" 493 | 3> py:func('datetime.datetime', now). 494 | {"datetime",{2014,12,25,23,16,14,979696,undefined}} 495 | 4> py:func('os.path', isfile, [<<"/tmp">>]). 496 | false 497 | 5> py:func('os.path', isdir, [<<"/tmp">>]). 498 | true 499 | 6> py:func(builtins, int, [<<"101010">>], [{base, 2}]). 500 | 42 501 | 7> py:const(math, pi). 502 | 3.141592653589793 503 | 8> py:const(math, pi, float). 504 | 3.141592653589793 505 | 9> py:const(math, pi, int). 506 | 3 507 | 10> py:const(math, pi, str). 508 | "3.141592653589793" 509 | 11> py:init(builtins, dict). 510 | {"dict",[]} 511 | 12> py:init(collections, 'UserDict'). 512 | {"UserDict",[]} 513 | 13> py:init(datetime, date, [1923, 4, 2]). 514 | {"date",{1923,4,2}} 515 | ``` 516 | 517 | ```erlang 518 | 14> Now = py:func('datetime.datetime', now). 519 | {"datetime",{2014,12,25,23,16,57,146812,undefined}} 520 | 15> py:method(Now, strftime, [<<"%Y.%m.d %H:%M:%S">>]). 521 | "2014.12.d 23:16:57" 522 | 16> py:attr(Now, year). 523 | 2014 524 | 17> py:attr(Now, microsecond). 525 | 146812 526 | ``` 527 | 528 | ```erlang 529 | 18> Later = py:func('datetime.datetime', now). 530 | {"datetime",{2014,12,25,23,19,51,934212,undefined}} 531 | 19> Earlier = Now. 532 | {"datetime",{2014,12,25,23,16,57,146812,undefined}} 533 | 20> Diff = py:sub(Later, Earlier). 534 | {"timedelta",{0,174,787400}} 535 | 21> py:attr(Diff, seconds). 536 | 174 537 | 22> py:attr(Diff, seconds) / 60. 538 | 2.9 539 | ``` 540 | 541 | ```erlang 542 | 23> py:pycall(datetime, 'datetime.now'). 543 | {"datetime",{2014,12,25,23,24,46,525495,undefined}} 544 | 24> py:pycall(datetime, datetime, [1923, 4, 2, 0, 0, 0]). 545 | {"datetime",{1923,4,2,0,0,0,0,undefined}} 546 | ``` 547 | 548 | ```erlang 549 | 25> py:dict(). 550 | {"dict",[]} 551 | 26> py:dict([{"a", 1}, {"b", 2}]). 552 | {"dict",[{"b",2},{"a",1}]} 553 | 27> py:int(<<"101010">>, [{base, 2}]). 554 | 42 555 | 28> py:any([true, true, false, false, false, true]). 556 | true 557 | 29> py:all([true, true, false, false, false, true]). 558 | false 559 | 30> py:all([true, true, true]). 560 | true 561 | 31> py:pow(6, 42). 562 | 481229803398374426442198455156736 563 | 32> py:round(0.666666667, 5). 564 | 0.66667 565 | 33> py:range(7, 42). 566 | {'$erlport.opaque',python, 567 | <<128,2,99,95,95,98,117,105,108,...>>} 568 | 34> py:len(py:range(7, 42)). 569 | 35 570 | 35> py:pylist(py:range(7, 42)). 571 | [7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26, 572 | 27,28,29,30,31,32,33,34,35|...] 573 | 36> length(py:pylist(py:range(7, 42))). 574 | 35 575 | ``` 576 | 577 | ```erlang 578 | 37> py:add(37, 5). 579 | 42 580 | 38> py:mul(7, 6). 581 | 42 582 | 39> py:sub(-108, -150). 583 | 42 584 | 40> py:truediv(462, 11). 585 | 42.0 586 | 41> py:floordiv(462, 11). 587 | 42 588 | 42> py:gt(7, 6). 589 | true 590 | 43> py:le(7, 6). 591 | false 592 | 44> py:eq(42, 42). 593 | true 594 | 45> py:'and-'(60, 13). 595 | 12 596 | 46> py:'or-'(60, 13). 597 | 61 598 | 47> py:'xor'(60, 13). 599 | 49 600 | 48> py:inv(60). 601 | -61 602 | 49> py:rshift(60, 2). 603 | 15 604 | 50> py:lshift(60, 2). 605 | 240 606 | ``` 607 | 608 | ```erlang 609 | 51> py:pylist(). 610 | [] 611 | 52> py:pylist([1, 2, 3, 4]). 612 | [1,2,3,4] 613 | ``` 614 | 615 | 616 | ## Architecture [↟](#table-of-contents) 617 | 618 | 619 | ### Overview [↟](#table-of-contents) 620 | 621 | Here is a high-level diagram of the LFE py architecture: 622 | 623 | ``` 624 | +-------------------------------------------+ 625 | | | 626 | | +-----------+ +-----------+ +-----------+ | 627 | | | py Worker | | py Worker | | py Worker | | 628 | | +------+----+ +-----+-----+ +-----+-----+ | 629 | | | | | | 630 | | | | | | 631 | | | +----+---+ | | 632 | | | | | | | 633 | | +-------+ py+sup +---------+ | 634 | | | | | 635 | | +----+---+ | 636 | | | | 637 | | +----+---+ | 638 | | | | | 639 | | | py+app | | 640 | | | | | 641 | | +--------+ | 642 | | | 643 | | LFE py | 644 | +---------------------+---------------------+ 645 | | LFE | ErlPort | 646 | +---------------------+---------------------+ 647 | | Erlang/OTP | 648 | +-------------------------------------------+ 649 | ``` 650 | 651 | Each py Worker is actually a wrapper for an ErlPort ``gen_server`` which starts 652 | up Python interpreter. LFE py is only designed to work with Python 3. Both 653 | ErlPort and it provide Python 3 modules for use in the interpreters started 654 | by the workers. They have the following conceptual structure: 655 | 656 | ``` 657 | +----------------+ 658 | | | 659 | | +----------+ | 660 | | | Encoders | | 661 | | +----------+ | 662 | | +----------+ | 663 | | | Decoders | | 664 | | +----------+ | 665 | | | 666 | | LFE py lib | 667 | +----------------+ 668 | | ErlPort lib | 669 | +----------------+ 670 | | Python 3 | 671 | +----------------+ 672 | ``` 673 | 674 | As depicted above, when the LFE py/ErlPort Python server starts, it brings up 675 | a Python 3 interpreter. LFE py configures the ``PYTHONPATH`` for 676 | ErlPort so that the custom encoder, decoder, and object helper Python modules 677 | are available for use by all Python calls issued to the workers. 678 | 679 | 680 | ### Erlang Components [↟](#table-of-contents) 681 | 682 | Working our way up from the diagram, here are references for Erlang/OTP 683 | components of LFE py: 684 | 685 | * [Erlang/OTP](http://learnyousomeerlang.com/what-is-otp) 686 | * [Erlang/OTP apps](http://learnyousomeerlang.com/building-applications-with-otp) 687 | * [How ErlPort works](http://erlport.org/docs/#id3) 688 | * [LFE](http://en.wikipedia.org/wiki/LFE_%28programming_language%29) 689 | * [py-app](https://github.com/lfex/py/blob/master/src/py-app.lfe) 690 | * [py-app](https://github.com/lfex/py/blob/master/src/py-sup.lfe) 691 | * [py Workers](https://github.com/lfex/py/blob/master/src/py.lfe#L7) 692 | 693 | 694 | ### Python Components [↟](#table-of-contents) 695 | 696 | And here are references for the Python components in LFE py: 697 | 698 | * [Python 3](https://docs.python.org/3/) 699 | * [ErlPort Python library](https://github.com/hdima/erlport/tree/master/priv/python3/erlport) 700 | * [LFE py Python library](https://github.com/lfex/py/tree/master/python/lfe) 701 | * [py Encoders](https://github.com/lfex/py/blob/master/python/lfe/encoders.py) 702 | * [py Decoders](https://github.com/lfex/py/blob/master/python/lfe/decoders.py) 703 | 704 | 705 | ## Controlling the Python Servers [↟](#table-of-contents) 706 | 707 | ### Erlang Configuration [↟](#table-of-contents) 708 | 709 | ### Python Configuration [↟](#table-of-contents) 710 | 711 | ### Start, Stop, and Restart [↟](#table-of-contents) 712 | 713 | ### Dynamically Adding More Python Servers [↟](#table-of-contents) 714 | 715 | ### Automatic Restarts [↟](#table-of-contents) 716 | 717 | ### Python Server Schedulers [↟](#table-of-contents) 718 | 719 | 720 | ## Executing Code in Parallel [↟](#table-of-contents) 721 | 722 | ### Identical Calls [↟](#table-of-contents) 723 | 724 | ### Scatter/Gather [↟](#table-of-contents) 725 | 726 | 727 | ## Distributed Python [↟](#table-of-contents) 728 | 729 | ### Starting Remote Python Servers [↟](#table-of-contents) 730 | 731 | ### Executing Python on Remote Servers [↟](#table-of-contents) 732 | -------------------------------------------------------------------------------- /include/builtins.lfe: -------------------------------------------------------------------------------- 1 | ;;;; This inlcude wraps functions provided by the Python builtins module. 2 | ;;;; 3 | ;;;; The following functions are provided in Python by this module: 4 | ;;;; https://docs.python.org/3.4/library/functions.html#built-in-funcs 5 | ;;;; 6 | (eval-when-compile 7 | 8 | (defun get-builtin-funcs () 9 | '((abs 1) 10 | (all 1) 11 | (any 1) 12 | (ascii 1) 13 | (bin 1) 14 | (bool 0) (bool 1) 15 | (bytearray 0) (bytearray 1) (bytearray 2) (bytearray 3) 16 | (bytes 0) (bytes 1) (bytes 2) (bytes 3) 17 | (callable 1) 18 | (chr 1) 19 | (compile 3) 20 | (complex 0) (complex 1) (complex 2) 21 | (dict 0) 22 | (dir 0) (dir 1) 23 | (divmod 2) 24 | (enumerate 1) (enumerate 2) 25 | (file 1) (file 2) (file 3) 26 | (filter 2) 27 | (float 0) (float 1) 28 | (format 1) (format 2) 29 | (frozenset 0) (frozenset 1) 30 | (getattr 2) (getattr 3) 31 | (globals 0) 32 | (hasattr 2) 33 | (hash 1) 34 | (help 0) (help 1) 35 | (hex 1) 36 | (id 1) 37 | (int 1) 38 | (isinstance 2) 39 | (issubclass 2) 40 | (iter 1) (iter 2) 41 | (len 1) 42 | (locals 0) 43 | (map 2) (map 3) 44 | (max 1) (max 2) (max 3) (max 4) 45 | (memoryview 1) 46 | (min 1) (min 2) (min 3) (min 4) 47 | (next 1) (next 2) 48 | (object 0) 49 | (oct 1) 50 | (open 1) 51 | (ord 1) 52 | (pow 2) (pow 3) 53 | (print 1) 54 | (range 1) (range 2) (range 3) 55 | (repr 1) 56 | (reversed 1) 57 | (round 1) (round 2) 58 | (set 0) (set 1) 59 | (slice 1) (slice 2) (slice 3) 60 | (sorted 1) (sorted 2) (sorted 3) 61 | (str 1) 62 | (sum 1) (sum 2) 63 | (super 0) (super 1) (super 2) 64 | (tuple 0) (tuple 1) 65 | (type 1) (type 3) 66 | (vars 0) (vars 1) 67 | (zip 1))) 68 | 69 | (defun get-kwarg-builtin-funcs () 70 | '((compile 4) 71 | (int 2) 72 | (open 2) 73 | (print 2) 74 | (property 1) 75 | (str 2)))) 76 | 77 | (defmacro generate-builtins-api () 78 | `(progn ,@(py-util:make-funcs (get-builtin-funcs) 'builtins))) 79 | 80 | (defmacro generate-kwarg-builtins-api () 81 | `(progn ,@(py-util:make-kwarg-funcs (get-kwarg-builtin-funcs) 'builtins))) 82 | 83 | (generate-builtins-api) 84 | (generate-kwarg-builtins-api) 85 | 86 | (defun loaded-builtins () 87 | "This is just a dummy function for display purposes when including from the 88 | REPL (the last function loaded has its name printed in stdout). 89 | This function needs to be the last one in this include." 90 | 'ok) 91 | -------------------------------------------------------------------------------- /include/operators.lfe: -------------------------------------------------------------------------------- 1 | ;;;; This inlcude wraps functions provided by the Python operator module. 2 | ;;;; 3 | ;;;; The following functions are provided in Python by this module: 4 | ;;;; https://docs.python.org/3.4/library/operator.html 5 | ;;;; 6 | (eval-when-compile 7 | 8 | (defun get-operator-funcs () 9 | '(;; Equality 10 | (lt 2) 11 | (le 2) 12 | (eq 2) 13 | (ne 2) 14 | (ge 2) 15 | (gt 2) 16 | ;; Logic 17 | (not- 1) 18 | (truth 1) 19 | (is- 2) 20 | (is-not 2) 21 | ;; Arithmetic 22 | (add 2) 23 | (floordiv 2) 24 | (index 1) 25 | (mod 2) 26 | (mul 2) 27 | (neg 1) 28 | (pos 1) 29 | (sub 2) 30 | (truediv 2) 31 | ;; Bitwise 32 | (and- 2) 33 | (inv 1) 34 | (invert 1) 35 | (lshift 2) 36 | (or- 2) 37 | (rshift 2) 38 | (xor 2) 39 | ;; Sequences 40 | (concat 2) 41 | (contains 2) 42 | (countOf 2) 43 | (getitem 2) 44 | (indexOf 2) 45 | (length_hint 1) (length_hint 2) 46 | ;; Misc 47 | (attrgetter 1) 48 | (itemgetter 1) 49 | (methodcaller 1) (methodcaller 2)))) 50 | 51 | (defmacro generate-operators-api () 52 | `(progn ,@(py-util:make-funcs (get-operator-funcs) 'operator))) 53 | 54 | (generate-operators-api) 55 | 56 | (defun loaded-operators () 57 | "This is just a dummy function for display purposes when including from the 58 | REPL (the last function loaded has its name printed in stdout). 59 | This function needs to be the last one in this include." 60 | 'ok) 61 | -------------------------------------------------------------------------------- /lfe.config: -------------------------------------------------------------------------------- 1 | #(project 2 | (#(meta 3 | (#(name py) 4 | #(description "Distributed Python for the Erlang Ecosystem") 5 | #(version "0.0.5") 6 | #(keywords ("LFE" "Lisp" "Python" "Distributed" "Interop")) 7 | #(maintainers 8 | ((#(name "Duncan McGreggor") #(email "oubiwann@gmail.com")))) 9 | #(repos 10 | (#(github "lfex/py"))))) 11 | #(app 12 | (#(mod #(py ())))))) 13 | 14 | #(logging ( 15 | #(log-level info) 16 | #(backend lager) 17 | #(colored true) 18 | #(colors (#(timestamp (color green)) 19 | #(process (color cyan)) 20 | #(date (color green)) 21 | #(time (color green)) 22 | #(modfunc (color yellow)) 23 | #(message (color green)) 24 | #(debug (color greenb)) 25 | #(info (color blue)) 26 | #(notice (color cyan)) 27 | #(warning (color yellow)) 28 | #(error (color red)) 29 | #(critical (color yellowb)) 30 | #(alert (color magentab)) 31 | #(emergency (color redb)))) 32 | #(options (#(lager_console_backend ( 33 | info 34 | #(logjam-formatter 35 | (date " " time " [" pid "] [" severity "] " message "\n")))))))) 36 | 37 | -------------------------------------------------------------------------------- /python/lfe/decoders.py: -------------------------------------------------------------------------------- 1 | import collections 2 | from datetime import date, datetime, time, timedelta 3 | 4 | from cytoolz.functoolz import compose 5 | 6 | from lfe import erlang, logger 7 | 8 | 9 | def dicts(value): 10 | dict_names = ["dict", "UserDict", "OrderedDict", "defaultdict", "ChainMap"] 11 | dict_types = [dict, collections.UserDict, collections.OrderedDict, 12 | collections.defaultdict, collections.ChainMap] 13 | if (isinstance(value, list) 14 | and len(value) == 2 15 | and any([value[0] == x for x in dict_types])): 16 | index = dict_names.index(value[0]) 17 | value = dict_types[index](*value[1]) 18 | return value 19 | 20 | 21 | def dates(value): 22 | if (isinstance(value, tuple) 23 | and len(value) == 2 24 | and value[0] == erlang.List("date")): 25 | (year, month, day) = value[1] 26 | value = date(year, month, day) 27 | return value 28 | 29 | 30 | def datetimes(value): 31 | if (isinstance(value, tuple) 32 | and len(value) == 2 33 | and value[0] == erlang.List("datetime")): 34 | (year, month, day, hour, minute, second, micro, tz) = value[1] 35 | value = datetime(year, month, day, hour, minute, second, micro, tz) 36 | return value 37 | 38 | 39 | def times(value): 40 | if (isinstance(value, tuple) 41 | and len(value) == 2 42 | and value[0] == erlang.List("time")): 43 | (hour, minute, second, micro, tz) = value[1] 44 | value = time(hour, minute, second, micro, tz) 45 | return value 46 | 47 | 48 | def timedeltas(value): 49 | if (isinstance(value, tuple) 50 | and len(value) == 2 51 | and value[0] == erlang.List("timedelta")): 52 | (days, seconds, micros) = value[1] 53 | value = timedelta(days, seconds, micros) 54 | return value 55 | 56 | 57 | def decode(value): 58 | return compose(dicts, 59 | dates, 60 | datetimes, 61 | times, 62 | timedeltas)(value) 63 | -------------------------------------------------------------------------------- /python/lfe/encoders.py: -------------------------------------------------------------------------------- 1 | import collections 2 | from datetime import date, datetime, time, timedelta 3 | 4 | from cytoolz.functoolz import compose 5 | 6 | from lfe import logger 7 | 8 | 9 | def dicts(value): 10 | dict_types = [dict, collections.UserDict, collections.OrderedDict, 11 | collections.defaultdict, collections.ChainMap] 12 | if any([isinstance(value, x) for x in dict_types]): 13 | value = (type(value).__name__, [x for x in value.items()]) 14 | return value 15 | 16 | 17 | def dates(value): 18 | if isinstance(value, date): 19 | value = ("date", (value.year, value.month, value.day)) 20 | return value 21 | 22 | 23 | def datetimes(value): 24 | if isinstance(value, datetime): 25 | value = ("datetime", (value.year, value.month, value.day, 26 | value.hour, value.minute, value.second, 27 | value.microsecond, value.tzinfo)) 28 | return value 29 | 30 | 31 | def times(value): 32 | if isinstance(value, time): 33 | value = ("time", (value.hour, value.minute, value.second, 34 | value.microsecond, value.tzinfo)) 35 | return value 36 | 37 | 38 | def timedeltas(value): 39 | if isinstance(value, timedelta): 40 | value = ("timedelta", (value.days, value.seconds, value.microseconds)) 41 | return value 42 | 43 | 44 | def encode(value): 45 | return compose(dicts, 46 | dates, 47 | datetimes, 48 | times, 49 | timedeltas)(value) 50 | -------------------------------------------------------------------------------- /python/lfe/erlang.py: -------------------------------------------------------------------------------- 1 | from erlport.erlterms import List as ErlPortList 2 | 3 | 4 | class List(ErlPortList): 5 | 6 | def __init__(self, data): 7 | if isinstance(data, str): 8 | data = self.convert_string(data) 9 | super().__init__(data) 10 | 11 | @staticmethod 12 | def convert_string(string): 13 | return [ord(x) for x in string] 14 | -------------------------------------------------------------------------------- /python/lfe/init.py: -------------------------------------------------------------------------------- 1 | from erlport.erlang import set_encoder, set_decoder 2 | 3 | from lfe import decoders, encoders, logger, obj 4 | 5 | 6 | def setup(): 7 | logger.info("Setting up LFE Python ...") 8 | obj.init() 9 | setup_encoders() 10 | setup_decoders() 11 | 12 | 13 | def setup_encoders(): 14 | set_encoder(encoders.encode) 15 | 16 | 17 | def setup_decoders(): 18 | set_decoder(decoders.decode) 19 | -------------------------------------------------------------------------------- /python/lfe/logger.py: -------------------------------------------------------------------------------- 1 | import logging 2 | 3 | 4 | logging.basicConfig( 5 | format='[%(asctime)s] [lfe:py] [%(levelname)s] %(message)s', 6 | datefmt='%Y-%m-%d %H:%M', 7 | level=logging.WARN) 8 | 9 | 10 | warn = logging.warning 11 | info = logging.info 12 | debug = logging.debug 13 | err = logging.error 14 | crit = logging.critical 15 | -------------------------------------------------------------------------------- /python/lfe/numpysupl.py: -------------------------------------------------------------------------------- 1 | """ 2 | Supplemental functions for NumPy 3 | """ 4 | import numpy as np 5 | 6 | 7 | def norm1d(array): 8 | shifted = array - (array.min()) 9 | return shifted / shifted.max() 10 | 11 | 12 | def scale1d(array, min=0, max=1): 13 | scaled = max * norm1d(array) 14 | return scaled + min 15 | 16 | 17 | class PolynomialLinearModel: 18 | "A convenience class for creating polynomial linear models." 19 | def __init__(self, xs, ys, degree): 20 | (self.xs, self.ys) = (xs, ys) 21 | self.degree = degree 22 | self.y_mean = self.get_y_mean() 23 | (self.results, self.model, self.ys_predicted, 24 | self.ss_tot, self.ss_reg, self.ss_res, 25 | self.r_squared) = (None, None, None, None, None, None, None) 26 | (self.coeffs, self.residuals, self.rank, 27 | self.singular_values, self.rcond) = (None, None, None, None, None) 28 | self.polyfit() 29 | 30 | def polyfit(self): 31 | (self.coeffs, self.residuals, self.rank, 32 | self.singular_values, self.rcond) = np.polyfit( 33 | self.xs, self.ys, self.degree, full=True) 34 | self.model = np.poly1d(self.coeffs) 35 | self.ys_predicted = self.model(self.xs) 36 | self.ss_tot = self.get_ss_tot() 37 | self.ss_reg = self.get_ss_reg() 38 | self.ss_res = self.get_ss_res() 39 | self.r_squared = self.get_r_squared() 40 | self.results = { 41 | "coeffs": self.coeffs.tolist(), 42 | "residuals": self.residuals.tolist(), 43 | "rank": self.rank, 44 | "singular-values": self.singular_values.tolist(), 45 | "rcond": float(self.rcond), 46 | "y-mean": float(self.y_mean), 47 | "ss-tot": float(self.ss_tot), 48 | "ss-reg": float(self.ss_reg), 49 | "ss-res": float(self.ss_res), 50 | "r-squared": float(self.r_squared)} 51 | 52 | def predict(self, xs): 53 | return self.model(xs) 54 | 55 | def get_y_mean(self): 56 | return self.ys.sum() / self.ys.size 57 | 58 | def get_ss_tot(self): 59 | return ((self.ys - self.get_y_mean()) ** 2).sum() 60 | 61 | def get_ss_reg(self): 62 | return ((self.ys_predicted - self.get_y_mean()) ** 2).sum() 63 | 64 | def get_ss_res(self): 65 | return ((self.ys - self.ys_predicted) ** 2).sum() 66 | 67 | def get_r_squared(self): 68 | return 1 - self.get_ss_res() / self.get_ss_tot() 69 | 70 | def __str__(self): 71 | return str(self.results) 72 | 73 | def __repr__(self): 74 | return self.__str__() 75 | -------------------------------------------------------------------------------- /python/lfe/obj.py: -------------------------------------------------------------------------------- 1 | import importlib 2 | 3 | from lfe import decoders, erlang, logger 4 | 5 | 6 | def init(): 7 | "Set up module in ErlPort Python namespace." 8 | 9 | 10 | def echo(*args, **kwargs): 11 | "Useful for debugging module-level issues." 12 | return (args, kwargs) 13 | 14 | 15 | def get_module(modname): 16 | modname = modname.decode("utf-8") 17 | try: 18 | module = importlib.import_module(modname) 19 | except ImportError as err: 20 | parts = modname.split(".") 21 | modname = parts[0] 22 | submod = parts[1] 23 | try: 24 | parent_module = importlib.import_module(modname) 25 | module = attr(parent_module, bytes(submod, "utf-8")) 26 | except: 27 | raise(err) 28 | return module 29 | 30 | 31 | def attr(obj, attr_name): 32 | return getattr(obj, attr_name.decode("utf-8").replace("-", "_")) 33 | 34 | 35 | def const(modname, attr_name): 36 | return attr(get_module(modname), attr_name) 37 | 38 | 39 | def kwargs_keys_to_strings(kwargs): 40 | return [(x.decode("utf-8"), y) for (x, y) in kwargs] 41 | 42 | 43 | def decode_args(arg_list): 44 | new_args = [] 45 | for arg in arg_list: 46 | if isinstance(arg, bytes): 47 | arg = arg.decode("utf-8") 48 | arg = decoders.decode(arg) 49 | new_args.append(arg) 50 | return new_args 51 | 52 | 53 | def decode_kwargs(kwarg_list): 54 | new_kwarg_list = [] 55 | for (key, val) in kwarg_list: 56 | if isinstance(val, bytes): 57 | val = val.decode("utf-8") 58 | val = decoders.decode(val) 59 | new_kwarg_list.append([key, val]) 60 | return dict(kwargs_keys_to_strings(new_kwarg_list)) 61 | 62 | 63 | def call_method(obj, funcname, args=[], kwarg_list=None): 64 | if len(kwarg_list) == 0: 65 | kwarg_list = [] 66 | kwargs = decode_kwargs(kwarg_list) 67 | args = decode_args(args) 68 | return attr(obj, funcname)(*args, **kwargs) 69 | 70 | 71 | def call_func(modname, funcname, args=[], kwarg_list=None): 72 | module = get_module(modname) 73 | return call_method(module, funcname, args, kwarg_list) 74 | 75 | 76 | def call_callable(func_obj, args=[], kwarg_list=None): 77 | return call_method(func_obj, b"__call__", args, kwarg_list) 78 | -------------------------------------------------------------------------------- /python/requirements.txt: -------------------------------------------------------------------------------- 1 | cytoolz 2 | -------------------------------------------------------------------------------- /rebar.config: -------------------------------------------------------------------------------- 1 | {erl_opts, [debug_info, {src_dirs, ["test"]}]}. 2 | 3 | {lfe_first_files, ["src/py-util.lfe"]}. 4 | 5 | {deps, [ 6 | {lutil, ".*", {git, "git://github.com/lfex/lutil.git", {tag, "0.8.0"}}}, 7 | {erlport, ".*", {git, "git://github.com/hdima/erlport.git", 8 | "ae6b8802b3883f9f350921b779d0e91784de5112"}}, 9 | {logjam, ".*", {git, "git://github.com/lfex/logjam.git", {tag, "0.4.0"}}} 10 | ]}. 11 | 12 | {plugins, [ 13 | {'lfe-compile', {git, "https://github.com/lfe-rebar3/compile.git", {tag, "0.3.0"}}} 14 | ]}. 15 | 16 | {provider_hooks, [{pre, [{compile, {lfe, compile}}]}]}. 17 | 18 | {profiles, [ 19 | {dev, [ 20 | {plugins, [ 21 | {'lfe-version', {git, "https://github.com/lfe-rebar3/version.git", {tag, "0.3.0"}}}, 22 | {'lfe-clean', {git, "https://github.com/lfe-rebar3/clean.git", {tag, "0.2.0"}}} 23 | ]} 24 | ]}, 25 | {test, [ 26 | {eunit_compile_opts, [ 27 | {src_dirs, ["test", "src"]} 28 | ]}, 29 | {deps, [ 30 | {ltest, ".*", {git, "git://github.com/lfex/ltest.git", {tag, "0.8.0"}}}]} 31 | ]}, 32 | {docs, [ 33 | {plugins, [ 34 | {lodox, {git, "https://github.com/quasiquoting/lodox.git", {tag, "0.12.10"}}} 35 | ]}, 36 | {lodox, 37 | [{apps, 38 | [{exemplar, 39 | [{'source-uri', 40 | "https://github.com/lfex/exemplar/blob/{version}/{filepath}#L{line}"}, 41 | {'output-path', "docs/master/current/api"}]}]} 42 | ]} 43 | ]} 44 | ]}. 45 | 46 | -------------------------------------------------------------------------------- /rebar.lock: -------------------------------------------------------------------------------- 1 | [{<<"clj">>, 2 | {git,"git://github.com/lfex/clj.git", 3 | {ref,"b935818364ff9b2715345a050c97db81289ae628"}}, 4 | 1}, 5 | {<<"color">>, 6 | {git,"git://github.com/julianduque/erlang-color", 7 | {ref,"e60f6302f795220f5f0bee86391ffee5274edec0"}}, 8 | 1}, 9 | {<<"erlport">>, 10 | {git,"git://github.com/hdima/erlport.git", 11 | {ref,"ae6b8802b3883f9f350921b779d0e91784de5112"}}, 12 | 0}, 13 | {<<"goldrush">>, 14 | {git,"git://github.com/DeadZen/goldrush.git", 15 | {ref,"212299233c7e7eb63a97be2777e1c05ebaa58dbe"}}, 16 | 2}, 17 | {<<"kla">>, 18 | {git,"git://github.com/lfex/kla.git", 19 | {ref,"a8afc4f673809d89b6c4a374b972f54ef6b39bde"}}, 20 | 2}, 21 | {<<"lager">>, 22 | {git,"git://github.com/basho/lager.git", 23 | {ref,"b2cb2735713e3021e0761623ff595d53a545438e"}}, 24 | 1}, 25 | {<<"lcfg">>, 26 | {git,"git://github.com/lfex/lcfg.git", 27 | {ref,"9c647739744de0332539b71692abc84df5d486a9"}}, 28 | 1}, 29 | {<<"lfe">>, 30 | {git,"git://github.com/rvirding/lfe.git", 31 | {ref,"b84e9a8a1db6ffdd0cfe593fc8ad440ef72a5511"}}, 32 | 1}, 33 | {<<"logjam">>, 34 | {git,"git://github.com/lfex/logjam.git", 35 | {ref,"568cc1d517795f98d19ffb46ca3b3ee5eab655dd"}}, 36 | 0}, 37 | {<<"ltest">>, 38 | {git,"git://github.com/lfex/ltest.git", 39 | {ref,"a35a91b93ab391e95f1e3f9ca254e9982f3eea2f"}}, 40 | 2}, 41 | {<<"lutil">>, 42 | {git,"git://github.com/lfex/lutil.git", 43 | {ref,"ebc8523047c53764c7daca6be187600fd381cbf9"}}, 44 | 0}]. 45 | -------------------------------------------------------------------------------- /resources/images/Python-logo-notext-small.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lfex/py/f7e55faa08139900e036c13948b8a4c04408c028/resources/images/Python-logo-notext-small.png -------------------------------------------------------------------------------- /resources/images/Python-logo-notext.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lfex/py/f7e55faa08139900e036c13948b8a4c04408c028/resources/images/Python-logo-notext.png -------------------------------------------------------------------------------- /resources/make/common.mk: -------------------------------------------------------------------------------- 1 | ifeq ($(shell which erl),) 2 | $(error Can't find Erlang executable 'erl') 3 | exit 1 4 | endif 5 | OS := $(shell uname -s) 6 | ifeq ($(OS),Linux) 7 | HOST=$(HOSTNAME) 8 | endif 9 | ifeq ($(OS),Darwin) 10 | HOST = $(shell scutil --get ComputerName) 11 | endif 12 | 13 | LIB = $(PROJECT) 14 | BIN_DIR = ./bin 15 | SOURCE_DIR = ./src 16 | OUT_DIR = ./ebin 17 | LFE=_build/default/lib/lfe/bin/lfe 18 | 19 | $(BIN_DIR): 20 | mkdir -p $(BIN_DIR) 21 | 22 | get-version: 23 | @echo "Get LFE py version:" 24 | @$(LFE) \ 25 | -eval '(lfe_io:format "~p~n" (list (py-util:get-py-version)))' -noshell -s erlang halt 26 | 27 | get-versions: 28 | @echo "Erlang/OTP, LFE, & library versions:" 29 | @$(LFE) \ 30 | -eval '(progn (py:start) (lfe_io:format "~p~n" (list (py-util:get-versions))))' -noshell -s erlang halt 31 | 32 | clean-ebin: 33 | @echo "Cleaning ebin dir ..." 34 | @rm -f $(OUT_DIR)/*.beam 35 | 36 | proj-compile: clean-ebin 37 | @echo "Compiling project code and dependencies ..." 38 | @rebar3 compile 39 | 40 | compile-tests: clean-eunit 41 | @PATH=$(SCRIPT_PATH) ERL_LIBS=$(ERL_LIBS) $(LFETOOL) tests build 42 | 43 | repl: proj-compile 44 | @which clear >/dev/null 2>&1 && clear || printf "\033c" 45 | @echo "Starting an LFE REPL ..." 46 | @$(LFE) -s $(PROJECT) 47 | 48 | clean: clean-ebin 49 | @echo "Cleaning project build dir ..." 50 | @rm -rf ebin/* _build/default/lib/py/ebin/* 51 | 52 | check: 53 | @rebar3 as tests eunit 54 | 55 | check-travis: check 56 | -------------------------------------------------------------------------------- /resources/make/dist.mk: -------------------------------------------------------------------------------- 1 | NODENAME=$(shell echo "$(PROJECT)"|sed -e 's/-//g') 2 | RUN_DIR=./run 3 | LOG_DIR=./log 4 | 5 | compile: proj-compile 6 | 7 | run-dir: 8 | @mkdir -p $(RUN_DIR) 9 | 10 | log-dir: 11 | @mkdir -p $(LOG_DIR) 12 | 13 | run: 14 | @@ERL_LIBS=$(ERL_LIBS) PATH=$(SCRIPT_PATH) \ 15 | run_erl -daemon ./run/ ./log/ \ 16 | "erl -pa ebin -config ${config-./dev} -s \'$(PROJECT)\'" 17 | 18 | dev: 19 | @echo "Running OTP app in the foreground ..." 20 | @ERL_LIBS=$(ERL_LIBS) PATH=$(SCRIPT_PATH) lfe \ 21 | -eval "application:start('$(PROJECT)')" 22 | 23 | dev-named: 24 | @echo "Running OTP app in the foreground ..." 25 | @ERL_LIBS=$(ERL_LIBS) PATH=$(SCRIPT_PATH) lfe \ 26 | -sname repl@${HOST} -setcookie `cat ~/.erlang.cookie` \ 27 | -eval "application:start('$(PROJECT)')" 28 | 29 | run-named: dev-named 30 | 31 | prod: 32 | @echo "Running OTP app in the background ..." 33 | @ERL_LIBS=$(ERL_LIBS) PATH=$(SCRIPT_PATH) lfe \ 34 | -sname ${NODENAME}@${HOST} -setcookie `cat ~/.erlang.cookie` \ 35 | -eval "application:start('$(PROJECT)')" \ 36 | -noshell -detached 37 | 38 | daemon: prod 39 | 40 | stop: 41 | @echo "Stopping OTP app ..." 42 | @ERL_LIBS=$(ERL_LIBS) PATH=$(SCRIPT_PATH) lfe \ 43 | -sname controller@${HOST} -setcookie `cat ~/.erlang.cookie` \ 44 | -eval "rpc:call('${NODENAME}@${HOST}', init, stop, [])" \ 45 | -noshell -s erlang halt 46 | 47 | list-nodes: 48 | @echo "Getting list of running OTP nodes ..." 49 | @echo 50 | @ERL_LIBS=$(ERL_LIBS) PATH=$(SCRIPT_PATH) lfe \ 51 | -sname controller@${HOST} -setcookie `cat ~/.erlang.cookie` \ 52 | -eval 'io:format("~p~n",[element(2,net_adm:names())]).' \ 53 | -noshell -s erlang halt 54 | -------------------------------------------------------------------------------- /resources/make/python.mk: -------------------------------------------------------------------------------- 1 | ERL_PORT_LIB=deps/erlport/priv/python3 2 | LIB=python 3 | VENV=$(LIB)/.venv 4 | REQS=$(LIB)/requirements.txt 5 | GET_PIP=$(LIB)/get-pip.py 6 | 7 | venv: 8 | python3 -m venv --without-pip $(VENV) 9 | 10 | $(GET_PIP): 11 | wget -O $(GET_PIP) https://bootstrap.pypa.io/get-pip.py 12 | . $(VENV)/bin/activate && \ 13 | python3 $(GET_PIP) 14 | 15 | get-py-deps: $(GET_PIP) 16 | . $(VENV)/bin/activate && \ 17 | pip install -r $(REQS) 18 | 19 | python: venv get-py-deps 20 | 21 | interp: 22 | @. $(VENV)/bin/activate && \ 23 | PYTHONPATH=$(LIB):$(ERL_PORT_LIB) \ 24 | python3 25 | -------------------------------------------------------------------------------- /src/py-app.lfe: -------------------------------------------------------------------------------- 1 | (defmodule py-app 2 | (behaviour application) 3 | (export all)) 4 | 5 | (defun start () 6 | (start 'normal '())) 7 | 8 | (defun start (_type _args) 9 | (let ((result (py-sup:start_link))) 10 | (case result 11 | (`#(ok ,pid) 12 | result) 13 | (_ 14 | `#(error ,result))))) 15 | 16 | (defun stop () 17 | (stop '())) 18 | 19 | (defun stop (_state) 20 | 'ok) 21 | -------------------------------------------------------------------------------- /src/py-config.lfe: -------------------------------------------------------------------------------- 1 | (defmodule py-config 2 | (export all)) 3 | 4 | (defun app-name () "" 'py) 5 | 6 | (defun get-loaded-apps () 7 | (proplists:get_keys 8 | (application:loaded_applications))) 9 | 10 | (defun loaded? () 11 | (lists:any 12 | (lambda (x) (== x (app-name))) 13 | (get-loaded-apps))) 14 | 15 | (defun load-config () 16 | (if (loaded?) 17 | 'already-loaded 18 | (application:load (app-name)))) 19 | 20 | (defun get (key) 21 | (load-config) 22 | (let ((result (application:get_env (app-name) key))) 23 | (case result 24 | (`#(ok ,data) 25 | data) 26 | (_ result)))) 27 | 28 | (defun get-python-path () 29 | (get 'python-path)) 30 | 31 | (defun get-max-restarts () 32 | (get 'erlport-max-restarts)) 33 | 34 | (defun get-restart-threshold () 35 | (get 'erlport-restart-threshold)) 36 | 37 | (defun get-shutdown-timeout () 38 | (get 'erlport-shutdown-timeout)) 39 | 40 | (defun call-scheduler () 41 | (call (get 'scheduler-mod) (get 'scheduler-func))) 42 | 43 | (defun get-worker-count () 44 | (get 'worker-count)) 45 | 46 | (defun get-log-level () 47 | (get 'log-level)) 48 | 49 | (defun get-worker-on-start () 50 | (get 'worker-on-start)) 51 | -------------------------------------------------------------------------------- /src/py-logger.lfe: -------------------------------------------------------------------------------- 1 | (defmodule py-logger 2 | (export all)) 3 | 4 | (include-lib "logjam/include/logjam.lfe") 5 | 6 | (defun setup () 7 | (logjam:setup)) 8 | -------------------------------------------------------------------------------- /src/py-sched.lfe: -------------------------------------------------------------------------------- 1 | ;;;; Schedulers for selecting the Python server on which to execute the next 2 | ;;;; call. 3 | ;;;; 4 | ;;;; Note there is an implicit "protocol" in this module (as well as the 5 | ;;;; py-sup module functions that are called here, e.g., (get-child-* ...)): 6 | ;;;; 7 | ;;;; * All calls that get children return a list of 2-tuples where the 8 | ;;;; first element is the data queried and the second element is the 9 | ;;;; pid of the child (python server process). This makes meaningful 10 | ;;;; sorting trivial. 11 | ;;;; 12 | ;;;; * Sorting should place the desired data point in the 1st position 13 | ;;;; of the resultant list. 14 | ;;;; 15 | ;;;; * If the above are followed, the function (select-pid data), where 16 | ;;;; data is the list of tuples, should work without issue. 17 | ;;;; 18 | ;;;; * The function (get-next-pid) calls the configured scheduler and it 19 | ;;;; expects that the scheduler will return the name for a process, which 20 | ;;;; it then uses to lookup the PID. 21 | ;;;; 22 | (defmodule py-sched 23 | (export all)) 24 | 25 | (defun get-next-pid () 26 | (let ((pid (py-config:call-scheduler))) 27 | (logjam:debug (MODULE) 'get-next-pid "Got pid: ~p" `(,pid)) 28 | (whereis pid))) 29 | 30 | (defun get-first () 31 | "Always get the first PID. 32 | 33 | Don't even bother with the others. Ever. King of imbecile schedulers." 34 | (select-pid (py-sup:get-children))) 35 | 36 | (defun random () 37 | "Select a random Python server PID." 38 | (let ((index (random:uniform (py-config:get-worker-count)))) 39 | (select-pid `(,(lists:nth index (py-sup:get-children)))))) 40 | 41 | (defun least-reductions () 42 | "Select the Python server PID with the least reductions." 43 | (select-pid (lists:sort (py-sup:get-child-reductions)))) 44 | 45 | (defun least-messages-in () 46 | "Select the Python server PID with the least messages in." 47 | (select-pid (lists:sort (py-sup:get-child-messages-in)))) 48 | 49 | (defun youngest () 50 | "Select the Python server PID that has been running for the shortest time." 51 | (select-pid (lists:reverse (lists:sort (py-sup:get-child-start-time))))) 52 | 53 | (defun oldest () 54 | "Select the Python server PID that has been running for the longest time." 55 | (select-pid (lists:sort (py-sup:get-child-start-time)))) 56 | 57 | (defun select-pid (data) 58 | (element 2 (car data))) 59 | -------------------------------------------------------------------------------- /src/py-sup.lfe: -------------------------------------------------------------------------------- 1 | (defmodule py-sup 2 | (behaviour supervisor) 3 | (export all)) 4 | 5 | (defun start_link () 6 | (supervisor:start_link 7 | `#(local ,(MODULE)) 8 | (MODULE) 9 | '())) 10 | 11 | (defun init (_) 12 | `#(ok #(,(get-supervisor-spec) 13 | ,(get-children-specs)))) 14 | 15 | (defun get-supervisor-spec () 16 | `#(one_for_one ,(py-config:get-max-restarts) 17 | ,(py-config:get-restart-threshold))) 18 | 19 | (defun get-children-specs () 20 | (lists:map #'get-child-spec/1 (py-util:get-worker-names))) 21 | 22 | (defun get-child-spec () 23 | (get-child-spec 'py)) 24 | 25 | (defun get-child-spec (child-id) 26 | `#(,child-id #(py start_link (,child-id)) 27 | permanent 28 | ,(py-config:get-shutdown-timeout) 29 | worker 30 | (py))) 31 | 32 | (defun get-pid () 33 | (let ((pid (erlang:whereis (MODULE)))) 34 | (sys:statistics pid 'true) 35 | pid)) 36 | 37 | (defun add-server (child-id) 38 | (add-server (get-pid) child-id)) 39 | 40 | (defun add-server (sup-pid child-id) 41 | (supervisor:start_child sup-pid (get-child-spec child-id))) 42 | 43 | (defun get-status () 44 | ;; We want to get statisitics here, too -- so use (get-pid) which checks 45 | ;; to make sure that the supervisor process has statisics enabled 46 | (sys:get_status (get-pid))) 47 | 48 | (defun get-state () 49 | (sys:get_state (MODULE))) 50 | 51 | (defun get-children () 52 | (lists:map 53 | (match-lambda ((`#(,name ,pid ,_ ,_)) 54 | `#(,pid ,name))) 55 | (supervisor:which_children (MODULE)))) 56 | 57 | (defun get-child-pids () 58 | (lists:map 59 | (lambda (x) 60 | (element 2 x)) 61 | (get-children))) 62 | 63 | (defun get-stats () 64 | (let ((`#(ok ,stats) (sys:statistics (get-pid) 'get))) 65 | (++ `(#(,(MODULE) ,stats)) 66 | (get-child-stats)))) 67 | 68 | (defun get-child-stats () 69 | (lists:map 70 | (match-lambda ((`#(,_ ,name)) 71 | (sys:statistics name 'true) 72 | (let ((`#(ok ,stats) (sys:statistics name 'get))) 73 | `#(,name ,stats)))) 74 | (get-children))) 75 | 76 | (defun get-child-stats (key) 77 | (lists:map 78 | (match-lambda ((`#(,name ,stats)) 79 | `#(,(proplists:get_value key stats) ,name))) 80 | (get-child-stats))) 81 | 82 | (defun get-child-reductions () 83 | (get-child-stats 'reductions)) 84 | 85 | (defun get-child-messages-in () 86 | (get-child-stats 'messages_in)) 87 | 88 | (defun get-child-start-time () 89 | (get-child-stats 'start_time)) 90 | -------------------------------------------------------------------------------- /src/py-util.lfe: -------------------------------------------------------------------------------- 1 | (defmodule py-util 2 | (export all)) 3 | 4 | (defun get-erlport-version () 5 | (lutil:get-app-version 'erlport)) 6 | 7 | (defun get-py-version () 8 | (lutil:get-app-version 'py)) 9 | 10 | (defun get-python-version () 11 | (lists:map #'string:strip/1 12 | (string:tokens (py:pycall 'sys 'version.__str__ '()) 13 | "\n"))) 14 | 15 | (defun get-versions () 16 | (++ (lutil:get-versions) 17 | `(#(erlport ,(get-erlport-version)) 18 | #(py ,(get-py-version)) 19 | #(python ,(get-python-version))))) 20 | 21 | (defun proplist->binary (proplist) 22 | "Convert all the keys to binary." 23 | (lists:map 24 | (match-lambda 25 | ((`#(,key ,value)) (when (is_atom key)) 26 | `(,(atom_to_binary key 'latin1) ,value)) 27 | ((`#(,key ,value)) (when (is_list key)) 28 | `(,(list_to_binary key) ,value))) 29 | proplist)) 30 | 31 | (defun make-func 32 | ((`(,lfe-func-name ,func-arity) mod) 33 | (let ((py-func-name (kla:replace-dash lfe-func-name)) 34 | (func-args (kla:make-args func-arity))) 35 | `(defun ,lfe-func-name ,func-args 36 | (py:pycall ',mod ',py-func-name (list ,@func-args)))))) 37 | 38 | (defun make-funcs (func-list mod) 39 | (lists:map 40 | (lambda (x) 41 | (make-func x mod)) 42 | func-list)) 43 | 44 | (defun make-kwarg-func 45 | ((`(,lfe-func-name ,func-arity) mod) 46 | (let* ((py-func-name (kla:replace-dash lfe-func-name)) 47 | (func-args (kla:make-args func-arity)) 48 | (`#(,args (,kwargs)) (split-last func-args))) 49 | `(defun ,lfe-func-name ,func-args 50 | (py:func ',mod ',py-func-name (list ,@args) ,kwargs))))) 51 | 52 | (defun make-kwarg-funcs (func-list mod) 53 | (lists:map 54 | (lambda (x) 55 | (make-kwarg-func x mod)) 56 | func-list)) 57 | 58 | (defun split-last (all-args) 59 | (lists:split (- (length all-args) 1) all-args)) 60 | 61 | (defun get-worker-names () 62 | (lists:map 63 | (lambda (x) 64 | (list_to_atom (++ "py-" (integer_to_list x)))) 65 | (lists:seq 1 (py-config:get-worker-count)))) 66 | 67 | (defun split-dotted (dotted-name) 68 | (let* ((parts (string:tokens (atom_to_list dotted-name) ".")) 69 | (`#(,first (,last)) (split-last parts))) 70 | (lists:map 71 | #'list_to_atom/1 72 | `(,(string:join first ".") 73 | ,last)))) 74 | 75 | (defun dotted? (name) 76 | (if (> (length (string:tokens (atom_to_list name) ".")) 1) 77 | 'true 78 | 'false)) 79 | -------------------------------------------------------------------------------- /src/py.app.src: -------------------------------------------------------------------------------- 1 | %% -*- erlang -*- 2 | {application, py, 3 | [%% A quick description of the application. 4 | {description, "Distributed Python for the Erlang Ecosystem"}, 5 | 6 | %% The version of the application 7 | {vsn, "0.0.5"}, 8 | 9 | %% All modules used by the application. 10 | {modules, 11 | ['py-app', 12 | 'py-config', 13 | 'py-logger', 14 | 'py-sched', 15 | 'py-sup', 16 | 'py-util', 17 | py 18 | ]}, 19 | 20 | %% All of the registered names the application uses. This can be ignored. 21 | {registered, ['py-sup']}, 22 | 23 | %% Applications that are to be started prior to this one. This can be ignored 24 | %% leave it alone unless you understand it well and let the .rel files in 25 | %% your release handle this. 26 | {applications, [kernel, stdlib]}, 27 | {mod, {'py-app', []}}, 28 | %% OTP application loader will load, but not start, included apps. Again 29 | %% this can be ignored as well. To load but not start an application it 30 | %% is easier to include it in the .rel file followed by the atom 'none' 31 | {included_applications, []}, 32 | 33 | %% configuration parameters similar to those in the config file specified 34 | %% on the command line. can be fetched with application:get_env 35 | {env, [%% LFE py Configuration 36 | {'log-level', error}, 37 | %% ErlPort Configuration 38 | {'python-path', "./python:./deps/erlport/priv/python3"}, 39 | %% Supervisor Configuration 40 | {'erlport-max-restarts', 3}, 41 | {'erlport-restart-threshold', 1}, 42 | {'erlport-shutdown-timeout', 2000}, 43 | %% Python Server Configuration 44 | {'worker-count', 3}, 45 | {'worker-on-start', {py, 'on-startworker'}}, 46 | %% Python Server Scheduler Configuration 47 | {'scheduler-mod', 'py-sched'}, 48 | {'scheduler-func', 'least-reductions'} 49 | ]} 50 | ]}. 51 | -------------------------------------------------------------------------------- /src/py.lfe: -------------------------------------------------------------------------------- 1 | (defmodule py 2 | (export all)) 3 | 4 | (include-lib "py/include/builtins.lfe") 5 | (include-lib "py/include/operators.lfe") 6 | 7 | ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; 8 | ;;; Python server functions 9 | ;;; 10 | (defun start_link () 11 | (start_link '(py))) 12 | 13 | (defun start_link (child-id) 14 | (let* ((python-path (py-config:get-python-path)) 15 | (options `(#(python_path ,python-path))) 16 | (result (python:start_link `#(local ,child-id) options)) 17 | (`#(,mod ,func) (py-config:get-worker-on-start))) 18 | (call mod func child-id) 19 | result)) 20 | 21 | (defun start () 22 | (py-logger:setup) 23 | (application:start 'py) 24 | #(ok started)) 25 | 26 | (defun stop () 27 | (application:stop 'py) 28 | #(ok stopped)) 29 | 30 | (defun restart () 31 | (stop) 32 | (start) 33 | #(ok restarted)) 34 | 35 | (defun on-startworker (proc-name) 36 | "Initialize the Python components, but don't use the scheduler 37 | to get the pid, since the supervisor hasn't finished yet." 38 | (python:call (erlang:whereis proc-name) 'lfe 'init.setup '())) 39 | 40 | (defun get-sup-pid () 41 | (py-sup:get-pid)) 42 | 43 | (defun get-python-pids () 44 | (py-sup:get-child-pids)) 45 | 46 | (defun add-server (child-id) 47 | "Add another Python ErlPort server to the supervision tree." 48 | (py-sup:add-server child-id)) 49 | 50 | ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; 51 | ;;; Call functions 52 | ;;; 53 | 54 | ;; ErlPort Calls 55 | ;; 56 | (defun pycall (mod func) 57 | (pycall mod func '())) 58 | 59 | (defun pycall (mod func args) 60 | (let ((pid (py-sched:get-next-pid))) 61 | ;; XXX add a lager DEBUG call here for sanity-checking schedulers 62 | (python:call pid mod func args))) 63 | 64 | ;; Creating Python class instances 65 | ;; 66 | (defun init (mod-class) 67 | (apply #'init/2 (py-util:split-dotted mod-class))) 68 | 69 | (defun init 70 | ((mod-class args) (when (is_list args)) 71 | (apply #'init/4 (++ (py-util:split-dotted mod-class) 72 | `(,args ())))) 73 | ((module class) 74 | (init module class '() '()))) 75 | 76 | (defun init 77 | ((mod-class args kwargs) (when (is_list args)) 78 | (apply #'init/4 (++ (py-util:split-dotted mod-class) 79 | `(,args ,kwargs)))) 80 | ((module class args) 81 | (init module class args '()))) 82 | 83 | (defun init (module class args kwargs) 84 | (func module class args kwargs)) 85 | 86 | ;; Python object and module constants 87 | ;; 88 | (defun const (mod-attr) 89 | (apply #'const/2 (py-util:split-dotted mod-attr))) 90 | 91 | (defun const 92 | ((mod attr-name) (when (is_atom mod)) 93 | (let ((attr (atom_to_binary attr-name 'latin1))) 94 | ;; Now call to the 'const' function in the Python module 'lfe.obj' 95 | (pycall 'lfe 'obj.const `(,mod ,attr)))) 96 | ((obj type) 97 | (method obj (list_to_atom (++ "__" 98 | (atom_to_list type) 99 | "__"))))) 100 | 101 | 102 | (defun const (mod func type) 103 | (pycall mod (list_to_atom (++ (atom_to_list func) 104 | "." 105 | "__" 106 | (atom_to_list type) 107 | "__")))) 108 | 109 | ;; Python object attributes 110 | ;; 111 | (defun attr 112 | ((obj attr-name) (when (is_list attr-name)) 113 | (attr obj (list_to_atom attr-name))) 114 | ((obj attr-name) (when (is_atom attr-name)) 115 | (let* ((pid (py-sched:get-next-pid)) 116 | (attr (atom_to_binary attr-name 'latin1))) 117 | ;; Now call to the 'attr' function in the Python module 'lfe.obj' 118 | (pycall 'lfe 'obj.attr `(,obj ,attr))))) 119 | 120 | ;; Python method calls 121 | ;; 122 | (defun method (obj method-name) 123 | (method obj method-name '() '())) 124 | 125 | (defun method (obj method-name args) 126 | (method obj method-name args '())) 127 | 128 | (defun method (obj method-name args kwargs) 129 | (general-call obj method-name args kwargs 'obj.call_method)) 130 | 131 | (defun call-dotten (func-name)) 132 | 133 | ;; Python module function and function object calls 134 | ;; 135 | (defun func 136 | ((func-name) (when (is_atom func-name)) 137 | (if (py-util:dotted? func-name) 138 | (apply #'func/2 (py-util:split-dotted func-name)) 139 | (func func-name '() '()))) 140 | ((callable) 141 | (func callable '() '()))) 142 | 143 | (defun func 144 | ((func-name args) (when (andalso (is_atom func-name) (is_list args))) 145 | (if (py-util:dotted? func-name) 146 | (apply #'func/4 (++ (py-util:split-dotted func-name) 147 | `(,args ()))) 148 | (func func-name args '()))) 149 | ((module func-name) (when (is_atom module)) 150 | (func module func-name '() '())) 151 | ((callable args) 152 | (func callable args '()))) 153 | 154 | (defun func 155 | ((func-name args raw-kwargs) (when (andalso (is_atom func-name) 156 | (is_list args))) 157 | (if (py-util:dotted? func-name) 158 | (apply #'func/4 (++ (py-util:split-dotted func-name) 159 | `(,args ,raw-kwargs))) 160 | (let ((kwargs (py-util:proplist->binary raw-kwargs))) 161 | ;; Now call to the 'call_callable' function in the Python 162 | ;; module 'lfe.obj' 163 | (pycall 'lfe 'obj.call_callable `(,func-name ,args ,kwargs))))) 164 | ((module func-name args) (when (is_atom module)) 165 | (func module func-name args '())) 166 | ((callable args raw-kwargs) 167 | (let ((kwargs (py-util:proplist->binary raw-kwargs))) 168 | (pycall 'lfe 'obj.call_callable `(,callable ,args ,kwargs))))) 169 | 170 | (defun func (module func-name args kwargs) 171 | ;; Now call to the 'call_func' function in the Python module 'lfe.obj' 172 | (general-call (atom_to_binary module 'latin1) 173 | func-name 174 | args 175 | kwargs 176 | 'obj.call_func)) 177 | 178 | (defun general-call (obj attr-name args raw-kwargs type) 179 | (let* ((attr (atom_to_binary attr-name 'latin1)) 180 | (kwargs (py-util:proplist->binary raw-kwargs))) 181 | (pycall 'lfe type `(,obj ,attr ,args ,kwargs)))) 182 | 183 | ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; 184 | ;;; Wrappers for Builtins 185 | ;;; 186 | (defun dict (proplist) 187 | (func 'builtins 'dict '() proplist)) 188 | 189 | (defun pylist () 190 | (pycall 'builtins 'list '())) 191 | 192 | (defun pylist (data) 193 | (pycall 'builtins 'list `(,data))) 194 | 195 | ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; 196 | ;;; Convenience Functions 197 | ;;; 198 | (defun pdir (obj) 199 | (lfe_io:format "~p~n" 200 | `(,(pycall 'builtins 'dir `(,obj))))) 201 | 202 | (defun pvars (obj) 203 | (lfe_io:format "~p~n" 204 | `(,(pycall 'builtins 'vars `(,obj))))) 205 | 206 | (defun ptype (obj) 207 | (let* ((class (attr obj '__class__)) 208 | (repr (pycall 'builtins 'repr `(,class)))) 209 | (list_to_atom (cadr (string:tokens repr "'"))))) 210 | 211 | (defun prepr 212 | ((`#(,opaque ,lang ,data)) 213 | (io:format "#(~s ~s~n #B(~ts))~n" 214 | `(,opaque ,lang ,data)))) 215 | -------------------------------------------------------------------------------- /test/unit-py-tests.lfe: -------------------------------------------------------------------------------- 1 | (defmodule unit-py-tests 2 | (behaviour ltest-unit) 3 | (export all)) 4 | 5 | (include-lib "ltest/include/ltest-macros.lfe") 6 | 7 | (deftest placeholder 8 | (is-equal 9 | 1 1)) 10 | -------------------------------------------------------------------------------- /test/unit-py-util-tests.lfe: -------------------------------------------------------------------------------- 1 | (defmodule unit-py-util-tests 2 | (behaviour ltest-unit) 3 | (export all)) 4 | 5 | (include-lib "ltest/include/ltest-macros.lfe") 6 | 7 | (deftest proplist->binary 8 | (let ((data '(#(a 1) #(b 2) #(c 3))) 9 | (expected '((#b(97) 1) (#b(98) 2) (#b(99) 3)))) 10 | (is-equal 11 | expected 12 | (py-util:proplist->binary data)))) 13 | --------------------------------------------------------------------------------