├── .gitignore
├── .idea
├── .name
├── encodings.xml
├── jsLinters
│ └── jshint.xml
└── vcs.xml
├── .travis.yml
├── AUTHORS
├── ChangeLog
├── LICENSE
├── MANIFEST.in
├── README.md
├── papa
├── __init__.py
├── server
│ ├── __init__.py
│ ├── papa_socket.py
│ ├── proc.py
│ └── values.py
└── utils.py
├── setup.py
└── tests
├── __init__.py
└── executables
├── echo_client.py
├── echo_server.py
└── write_three_lines.py
/.gitignore:
--------------------------------------------------------------------------------
1 | *.pyc
2 | __pycache__
3 | build
4 | /papa.egg-info
5 | dist
6 | /env??
7 | /.idea/*
8 | !/.idea/.name
9 | !/.idea/gjslint.conf
10 | !/.idea/jsLinters/
11 | !/.idea/dictionaries/
12 | !/.idea/other.xml
13 | !/.idea/testrunner.xml
14 | !/.idea/encodings.xml
15 | !/.idea/vcs.xml
16 |
--------------------------------------------------------------------------------
/.idea/.name:
--------------------------------------------------------------------------------
1 | papa
--------------------------------------------------------------------------------
/.idea/encodings.xml:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
--------------------------------------------------------------------------------
/.idea/jsLinters/jshint.xml:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
10 |
11 |
12 |
13 |
14 |
15 |
16 |
17 |
18 |
--------------------------------------------------------------------------------
/.idea/vcs.xml:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
--------------------------------------------------------------------------------
/.travis.yml:
--------------------------------------------------------------------------------
1 | language: python
2 | python:
3 | - "2.6"
4 | - "2.7"
5 | - "3.3"
6 | - "3.4"
7 | install: pip install .
8 | script: nosetests
9 |
--------------------------------------------------------------------------------
/AUTHORS:
--------------------------------------------------------------------------------
1 | Scott Maxwell
2 |
--------------------------------------------------------------------------------
/ChangeLog:
--------------------------------------------------------------------------------
1 | CHANGES
2 | =======
3 |
4 | 0.9.3
5 | -----
6 |
7 | * Enhanced docs
8 |
9 | 0.9.0
10 | -----
11 |
12 | * Initial release
13 |
--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
1 | The MIT License (MIT)
2 |
3 | Copyright (c) 2014 Scott Maxwell
4 |
5 | Permission is hereby granted, free of charge, to any person obtaining a copy
6 | of this software and associated documentation files (the "Software"), to deal
7 | in the Software without restriction, including without limitation the rights
8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9 | copies of the Software, and to permit persons to whom the Software is
10 | furnished to do so, subject to the following conditions:
11 |
12 | The above copyright notice and this permission notice shall be included in all
13 | copies or substantial portions of the Software.
14 |
15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21 | SOFTWARE.
22 |
--------------------------------------------------------------------------------
/MANIFEST.in:
--------------------------------------------------------------------------------
1 | include MANIFEST.in
2 | include AUTHORS
3 | include ChangeLog
4 | include LICENSE
5 | include README.md
6 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | Summary
2 | =======
3 |
4 | **papa** is a process kernel. It contains both a client library and a server
5 | component for creating sockets and launching processes from a stable parent
6 | process.
7 |
8 | Dependencies
9 | ============
10 |
11 | Papa has no external dependencies, and it never will! It has been tested under the following Python versions:
12 |
13 | - 2.6
14 | - 2.7
15 | - 3.2
16 | - 3.3
17 | - 3.4
18 |
19 |
20 | Installation
21 | ============
22 |
23 | $> pip install papa
24 |
25 |
26 | Purpose
27 | =======
28 |
29 | Sometimes you want to be able to start a process and have it survive on its own,
30 | but you still want to be able to capture the output. You could daemonize it
31 | and pipe the output to files, but that is a pain and lacks flexibility when it
32 | comes to handling the output.
33 |
34 | Process managers such as circus and supervisor are very good for starting and
35 | stopping processes, and for ensuring that they are automatically restarted when
36 | they die. However, if you need to restart the process manager, all of their
37 | managed processes must be brought down as well. In this day of zero downtime,
38 | that is no longer okay.
39 |
40 | Papa is a process kernel. It has extremely limited functionality and it has zero
41 | external dependencies. If I've done my job right, you should never need to
42 | upgrade the papa package. There will probably be a few bug fixes before it is
43 | really "done", but the design goal was to create something that did NOT do
44 | everything, but only did the bare minimum required. The big process managers can
45 | add the remaining features.
46 |
47 | Papa has 3 types of things it manages:
48 |
49 | - Sockets
50 | - Values
51 | - Processes
52 |
53 | Here is what papa does:
54 |
55 | - Create sockets and close sockets
56 | - Set, get and clear named values
57 | - Start processes and capture their stdout/stderr
58 | - Allow you to retrieve the stdout/stderr of the processes started by papa
59 | - Pass socket file descriptors and port numbers to processes as they start
60 |
61 | Here is what it does NOT do:
62 |
63 | - Stop processes
64 | - Send signals to processes
65 | - Restart processes
66 | - Communicate with processes in any way other than to capture their output
67 |
68 |
69 | Sockets
70 | =======
71 |
72 | By managing sockets, papa can manage interprocess communication. Just create a
73 | socket in papa and then pass the file descriptor to your process to use it.
74 | See the [Circus docs](http://circus.readthedocs.org/en/0.11.1/for-ops/sockets/)
75 | for a very good description of why this is so useful.
76 |
77 | Papa can create Unix, INET and INET6 sockets. By default it will create an INET
78 | TCP socket on an OS-assigned port.
79 |
80 | You can pass either the file descriptor (fileno) or the port of a socket to a
81 | process by including a pattern like this in the process arguments:
82 |
83 | - `$(socket.my_awesome_socket_name.fileno)`
84 | - `$(socket.my_awesome_socket_name.port)`
85 |
86 |
87 | Values
88 | ======
89 |
90 | Papa has a very simple name/value pair storage. This works much like environment
91 | variables. The values must be text, so if you want to store a complex structure,
92 | you will need to encode and decode with something like the
93 | [JSON module](https://docs.python.org/3/library/json.html).
94 |
95 | The primary purpose of this facility is to store state information for your
96 | process that will survive between restarts. For instance, a process manager can
97 | store the current state that all of its managed processes are supposed to be in.
98 | Then if the process manager is restarted, it can restore its internal state,
99 | then go about checking to see if anything on the machine has changed. Are all
100 | processes that should be running actually running?
101 |
102 |
103 | Processes
104 | =========
105 |
106 | Processes can be started with or without output management. You can specify a
107 | maximum size for output to be cached. Each started process has a management
108 | thread in the Papa kernel watching its state and capturing output if necessary.
109 |
110 |
111 | A Note on Naming (Namespacing)
112 | ==============================
113 |
114 | Sockets, values and processes all have unique names. A name can only represent
115 | one item per class. So you could have an "aack" socket, an "aack" value and an
116 | "aack" process, but you cannot have two "aack" processes.
117 |
118 | All of the monitoring commands support a final asterix as a wildcard. So you
119 | can get a list of sockets whose names match "uwsgi*" and you would get any
120 | socket that starts with "uwsgi".
121 |
122 | One good naming scheme is to prefix all names with the name of your own
123 | application. So, for instance, the Circus process manager can prefix all names
124 | with "circus." and the Supervisor process manager can prefix all names with
125 | "supervisor.". If you write your own simple process manager, just prefix it with
126 | "tweeter." or "facebooklet." or whatever your project is called.
127 |
128 | If you need to have multiple copies of something, put a number after a dot
129 | for each of those as well. For instance, if you are starting 3 waitress
130 | instances in circus, call them `circus.waitress.1`, `circus.waitress.2`, and
131 | `circus.waitress.3`. That way you can query for all processes named `circus.*`
132 | to see all processes managed by circus, or query for `circus.waitress.*` to
133 | see all waitress processes managed by circus.
134 |
135 |
136 | Starting the kernel
137 | ===================
138 |
139 | There are two ways to start the kernel. You can run it as a process, or you can
140 | just try to access it from the client library and allow it to autostart. The
141 | client library uses a lock to ensure that multiple threads do not start the
142 | server at the same time but there is currently no protection against multiple
143 | processes doing so.
144 |
145 | By default, the papa kernel process will communicate over port 20202. You can
146 | change this by specifying a different port number or a path. By specifying a
147 | path, a Unix socket will be used instead.
148 |
149 | If you are going to be creating papa client instances in many places in your
150 | code, you may want to just call `papa.set_default_port` or `papa.set_default_path`
151 | once when your application is starting and then just instantiate the Papa object
152 | with no parameters.
153 |
154 |
155 | Telnet interface
156 | ================
157 |
158 | Papa has been designed so that you can communicate with the process kernel
159 | entirely without code. Just start the Papa server, then do this:
160 |
161 | telnet localhost 20202
162 |
163 | You should get a welcome message and a prompt. Type "help" to get help. Type
164 | "help process" to get help on the process command.
165 |
166 | The most useful commands from a monitoring standpoint are:
167 |
168 | - list sockets
169 | - list processes
170 | - list values
171 |
172 | All of these can by used with no arguments, or can be followed by a list of
173 | names, including wildcards. For instance, to see all of the values in the
174 | circus and supervisor namespaces, do this:
175 |
176 | list values circus.* supervisor.*
177 |
178 | You can abbreviate every command as short as you like. So "l p" means
179 | "list processes" and "h l p" means "help list processes"
180 |
181 |
182 | Creating a Connection
183 | =====================
184 |
185 | You can create either long-lived or short-lived connections to the Papa kernel.
186 | If you want to have a long-lived connection, just create a Papa object to
187 | connect and close it when done, like this:
188 |
189 | class MyObject(object):
190 | def __init__(self):
191 | self.papa = Papa()
192 |
193 | def start_stuff(self):
194 | self.papa.make_socket('uwsgi')
195 | self.papa.make_process('uwsgi', 'env/bin/uwsgi', args=('--ini', 'uwsgi.ini', '--socket', 'fd://$(socket.uwsgi.fileno)'), working_dir='/Users/aackbar/awesome', env=os.environ)
196 | self.papa.make_process('http_receiver', sys.executable, args=('http.py', '$(socket.uwsgi.port)'), working_dir='/Users/aackbar/awesome', env=os.environ)
197 |
198 | def close(self):
199 | self.papa.close()
200 |
201 | If you want to just fire off a few commands and leave, it is better to use the
202 | `with` mechanism like this:
203 |
204 | from papa import Papa
205 |
206 | with Papa() as p:
207 | print(p.list_sockets())
208 | print(p.make_socket('uwsgi', port=8080))
209 | print(p.list_sockets())
210 | print(p.make_process('uwsgi', 'env/bin/uwsgi', args=('--ini', 'uwsgi.ini', '--socket', 'fd://$(socket.uwsgi.fileno)'), working_dir='/Users/aackbar/awesome', env=os.environ))
211 | print(p.make_process('http_receiver', sys.executable, args=('http.py', '$(socket.uwsgi.port)'), working_dir='/Users/aackbar/awesome', env=os.environ))
212 | print(p.list_processes())
213 |
214 | This will make a new connection, do a bunch of work, then close the connection.
215 |
216 |
217 | Socket Commands
218 | ===============
219 |
220 | There are 3 socket commands.
221 |
222 | `p.list_sockets(*args)`
223 | ------------------
224 |
225 | The `sockets` command takes a list of socket names to get info about. All of
226 | these are valid:
227 |
228 | - `p.list_sockets()`
229 | - `p.list_sockets('circus.*')`
230 | - `p.list_sockets('circus.uwsgi', 'circus.nginx.*', 'circus.logger')`
231 |
232 | A `dict` is returned with socket names as keys and socket details as values.
233 |
234 | `p.make_socket(name, host=None, port=None, family=None, socket_type=None, backlog=None, path=None, umask=None, interface=None, reuseport=None)`
235 | -----------------------------------------------------------------------------------------------------------------------------------------------
236 |
237 | All parameters are optional except for the name. To create a standard TCP socket
238 | on port 8080, you can do this:
239 |
240 | p.make_socket('circus.uwsgi', port=8080)
241 |
242 | To make a Unix socket, do this:
243 |
244 | p.make_socket('circus.uwsgi', path='/tmp/uwsgi.sock')
245 |
246 | A path for a Unix socket must be an absolute path or `make_socket` will raise a
247 | `papa.Error` exception.
248 |
249 | You can also leave out the path and port to create a standard TCP socket with an
250 | OS-assigned port. This is really handy when you do not care what port is used.
251 |
252 | If you call `make_socket` with the name of a socket that already exists, papa
253 | will return the original socket if all parameters match, or raise a `papa.Error`
254 | exception if some parameters differ.
255 |
256 | See the `make_sockets` method of the Papa object for other parameters.
257 |
258 | `p.remove_sockets(*args)`
259 | -----------------------
260 |
261 | The `remove_sockets` command also takes a list of socket names. All of these are
262 | valid:
263 |
264 | - `p.remove_sockets('circus.*')`
265 | - `p.remove_sockets('circus.uwsgi', 'circus.nginx.*', 'circus.logger')`
266 |
267 | Removing a socket will prevent any future processes from using it, but any
268 | processes that were already started using the file descriptor of the socket will
269 | continue to use the copy they inherited.
270 |
271 |
272 | Value Commands
273 | ==============
274 |
275 | There are 4 value commands.
276 |
277 | `p.list_values(*args)`
278 | -----------------
279 |
280 | The `list_values` command takes a list of values to retrieve. All of these are
281 | valid:
282 |
283 | - `p.list_values()`
284 | - `p.list_values('circus.*')`
285 | - `p.list_list_values('circus.uwsgi', 'circus.nginx.*', 'circus.logger')`
286 |
287 | A `dict` will be returned with all matching names and values.
288 |
289 | `p.set(name, value=None)`
290 | -------------------------
291 |
292 | To set a value, do this:
293 |
294 | p.set('circus.uswgi', value)
295 |
296 | You can clear a single value by setting it to `None`.
297 |
298 | `p.get(name)`
299 | -------------
300 |
301 | To retrieve a value, do this:
302 |
303 | value = p.get('circus.uwsgi')
304 |
305 | If no value is stored by that name, `None` will be returned.
306 |
307 | `p.remove_values(*args)`
308 | ----------------
309 |
310 | To remove a value or values, do something like this:
311 |
312 | - `p.remove_values('circus.*')`
313 | - `p.remove_values('circus.uwsgi', 'circus.nginx.*', 'circus.logger')`
314 |
315 | You cannot remove all variables so passing no names or passing `*` will raise
316 | a `papa.Error` exception.
317 |
318 |
319 | Process Commands
320 | ================
321 |
322 | There are 4 process commands:
323 |
324 | `p.list_processes(*args)`
325 | --------------------
326 |
327 | The `list_processes` command takes a list of process names to get info about.
328 | All of these are valid:
329 |
330 | - `p.list_processes()`
331 | - `p.list_processes('circus.*')`
332 | - `p.list_processes('circus.uwsgi', 'circus.nginx.*', 'circus.logger')`
333 |
334 | A `dict` is returned with process names as keys and process details as values.
335 |
336 | `p.make_process(name, executable, args=None, env=None, working_dir=None, uid=None, gid=None, rlimits=None, stdout=None, stderr=None, bufsize=None, watch_immediately=None)`
337 | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
338 |
339 | Every process must have a unique `name` and an `executable`. All other
340 | parameters are optional. The `make_process` method returns a `dict` that
341 | contains the pid of the process.
342 |
343 | The `args` parameter should be a tuple of command-line arguments. If you have
344 | only one argument, papa conveniently supports passing that as a string.
345 |
346 | You will probably want to pass `working_dir`. If you do not, the working
347 | directory will be that of the papa kernel process.
348 |
349 | By default, stdout and stderr are captured so that you can retrieve them
350 | with the `watch` command. By default, the `bufsize` for the output is 1MB.
351 |
352 | Valid values for `stdout` and `stderr` are `papa.DEVNULL` and `papa.PIPE` (the
353 | default). You can also pass `papa.STDOUT` to `stderr` to merge the streams.
354 |
355 | If you pass `bufsize=0`, not output will be recorded. Otherwise, bufsize can be
356 | the number of bytes, or a number followed by 'k', 'm' or 'g'. If you want a
357 | 2 MB buffer, you can pass `bufsize='2m'`, for instance. If you do not retrieve
358 | the output quicky enough and the buffer overflows, older data is removed to make
359 | room.
360 |
361 | If you specify `uid`, it can be either the numeric id of the user or the
362 | username string. Likewise, `gid` can be either the numeric group id or the
363 | group name string.
364 |
365 | If you want to specify `rlimits`, pass a `dict` with rlimit names and numeric
366 | values. Valid rlimit names can be found in the `resources` module. Leave off the
367 | `RLIMIT_` prefix. On my system, valid names are `as`, `core`, `cpu`, `data`,
368 | `fsize`, `memlock`, `nofile`, `nproc`, `rss`, and `stack`.
369 |
370 | rlimit={'cpu': 2, 'nofile': 1024}
371 |
372 | The `env` parameter also takes a `dict` with names and values. A useful trick is
373 | to do `env=os.environ` to copy your environment to the new process.
374 |
375 | If you want to run a Python application and you wish to use the same Python
376 | executable as your client application, a useful trick is to pass `sys.executable`
377 | as the `executable` and the path to the Python script as the first element of your
378 | `args` tuple. If you have no other args, just pass the path as a string to
379 | `args`.
380 |
381 | p.make_process('write3', sys.executable, args='executables/write_three_lines.py', working_dir=here, uid=os.environ['LOGNAME'], env=os.environ)
382 |
383 | The final argument that needs mention is `watch_immediately`. If you pass `True`
384 | for this, papa will make the process and return a `Watcher`. This is effectively
385 | the same as doing `p.make_process(name, ...)` followed immediately by
386 | `p.watch(name)`, but it has one fewer round-trip communication with the kernel.
387 | If all you want to do is launch an application and monitor its output, this is
388 | a good way to go.
389 |
390 | `p.remove_processes(*args)`
391 | --------------------------------
392 |
393 | If you do not care about retrieving the output or the exit code for a process,
394 | you can use `remove_processes` to tell the papa kernel to close the output
395 | buffers and automatically remove the process from the process list when it
396 | exits.
397 |
398 | - `p.remove_processes('circus.logger')`
399 | - `p.remove_processes('circus.uwsgi', 'circus.nginx.*', 'circus.logger')`
400 |
401 | `p.watch_processes(*args)`
402 | ----------------
403 |
404 | The `watch_processes` command returns a `Watcher` object for the specified process or
405 | processes. That object uses a separate socket to retrieve the output of
406 | the processes it is watching.
407 |
408 | *Optimization Note:* Actually, it hijacks the socket of your `Papa`
409 | object. If you issue any other commands to the `Papa` object that require a
410 | connection to the kernel, the `Papa` object will silently create a new socket and
411 | connect up for the additional commands. If you close the `Watcher` and the `Papa`
412 | object has not already created a new connection, the socket will be returned to
413 | the `Papa` object. So if you launch an application, use `watch` to grab all of
414 | its output until it closes, then use the `set` command to update your saved
415 | status, all of that can occur with a single connection.
416 |
417 |
418 | The Watcher object
419 | ==================
420 |
421 | When you use `watch_processes` or when you do `make_process` with
422 | `watch_immediately=True`, you get back a `Watcher` object.
423 |
424 | You can use watchers manually or with a context manager. Here is an example
425 | without a context manager:
426 |
427 | class MyLogger(object):
428 | def __init__(self, watcher):
429 | self.watcher = watcher
430 |
431 | def save_stuff(self):
432 | if self.watcher and self.watcher.ready:
433 | out, err, closed = self.watcher.read()
434 | ... save it ...
435 | self.watcher.acknowledge() # remove it from the buffer
436 |
437 | def close(self):
438 | self.watcher.close()
439 |
440 | If you are running your logger in a separate thread anyway, you might want to
441 | just use a context manager, like this:
442 |
443 | with p.watch_processes('aack') as watcher:
444 | while watcher:
445 | out, err, closed = watcher.read() # block until something arrives
446 | ... save it ...
447 | watcher.acknowledge() # remove it from the buffer
448 |
449 | The `Watcher` object has a `fileno` method, so it can be used with `select.select`,
450 | like this:
451 |
452 | watchers = []
453 |
454 | watchers.append(p.watch_processes('circus.uwsgi'))
455 | watchers.append(p.watch_processes('circus.nginx'))
456 | watchers.append(p.watch_processes('circus.mongos.*'))
457 |
458 | while watchers:
459 | ready_watchers = select.select(watchers, [], [])[0] # wait for one of these
460 | for watcher in ready_watchers: # iterate through all that are ready
461 | out, err, closed = watcher.read()
462 | ... save it ...
463 | watcher.acknowledge()
464 | if not watcher: # if it is done, remove this watcher from the list
465 | watcher.close()
466 | del watchers[watcher]
467 |
468 | Of course, in the above example it would have been even more efficient to just
469 | use a single watcher, like this:
470 |
471 | with p.watch_processes('circus.uwsgi', 'circus.nginx', 'circus.mongos.*') as watcher:
472 | while watcher:
473 | out, err, closed = watcher.read()
474 | ... save it ...
475 | # watcher.acknowledge() - no need since watcher.read will do it for us
476 |
477 | `w.ready`
478 | ---------
479 |
480 | This property is `True` if the `Watcher` has data available to read on the socket.
481 |
482 | `w.read()`
483 | ----------
484 |
485 | Read will grab all waiting output from the `Watcher` and return a `tuple` of
486 | `(out, err, closed)`. Each of these is an array of `papa.ProcessOutput` objects.
487 | An output object is actually a `namedtuple` with 3 values: `name`, `timestamp`,
488 | and `data`.
489 |
490 | The `name` element is the `name` of the process. The `timestamp` is a `float` of
491 | when the data was captured by the papa kernel. The `data` is a binary string if
492 | found in either the `out` or `err` array. It is the exit code if found in the
493 | `closed` array. Using all of these elements, you can write proper timestamps
494 | into your logs, even if data was captured by the papa kernel minutes, hours or
495 | days earlier.
496 |
497 | The `read` method will block if no data is ready to read. If you do not want to
498 | block, use either the `ready` property or a mechanism such as `select.select`
499 | before calling `read`.
500 |
501 | `w.acknowledge()`
502 | -----------------
503 |
504 | Just because your have read output from a process, the papa kernel cannot know
505 | that you successfully logged it. Maybe you crashed or were shutdown before you
506 | had the chance. So the papa kernel will hold onto the data until you acknowledge
507 | receipt. This can be done either by calling `acknowledge`, or by doing a
508 | subsequent `read` or a `close`.
509 |
510 | `w.close()`
511 | -----------
512 |
513 | When you are done with a `Watcher`, be sure to close it. That will release the
514 | socket and potentially even return the socket back to the original `Papa` object.
515 | It will also send off a final `acknowledge` if necessary.
516 |
517 | If you use a context manager, the `close` happens automatically.
518 |
519 | `if watcher:`
520 | -------------
521 |
522 | A boolean check on the `Watcher` object will return `True` if it is still
523 | active and `False` if it has received and acknowledged a close message from all
524 | processes it is monitoring.
525 |
526 | WARNING: There should be only one
527 | ---------------------------------
528 |
529 | You will get very screwy results if you have multiple watchers for the same
530 | process. Each will get the available data, then acknowledge receipt at some
531 | point, removing that data from the queue. Based on timing, both will get
532 | overlapping results, but neither is likely to get everything.
533 |
534 |
535 | Shutting Down
536 | =============
537 |
538 | Papa is meant to be a long-lived process and it is meant to be usable by
539 | multiple client apps. If you would like to shut Papa down, you can try
540 | `p.exit_if_idle()`. This call will only exit Papa if there are no processes,
541 | sockets or values. So if your app cleaned everything up and no other app is
542 | using Papa, `exit_if_idle` will allow Papa to die. It will return `True` if
543 | Papa has indicated that it will exit when the connection closes.
544 |
545 | If you want to do a complete cleanup, kill all of your processes however you
546 | like, then do:
547 |
548 | p.remove_processes('myapp.*')
549 | p.remove_sockets('myapp.*')
550 | p.remove_values('myapp.*')
551 | if p.exit_if_idle():
552 | print('Papa says it will shutdown!')
553 |
554 | WARNING: If another process connects to Papa before the connection is closed,
555 | Papa will remain open. The `exit_if_idle` command will drop the connection if
556 | it returns True so this is a very narrow window of opportunity for failure.
557 |
--------------------------------------------------------------------------------
/papa/__init__.py:
--------------------------------------------------------------------------------
1 | import os.path
2 | import socket
3 | from threading import Lock
4 | from time import time, sleep
5 | from subprocess import PIPE, STDOUT
6 | from collections import namedtuple
7 | import logging
8 | import select
9 | from papa.utils import string_type, recv_with_retry, send_with_retry
10 |
11 | try:
12 | from subprocess import DEVNULL
13 | except ImportError:
14 | DEVNULL = -3
15 |
16 | __author__ = 'Scott Maxwell'
17 | __all__ = ['Papa', 'DEBUG_MODE_NONE', 'DEBUG_MODE_THREAD', 'DEBUG_MODE_PROCESS']
18 |
19 | log = logging.getLogger('papa.client')
20 | ProcessOutput = namedtuple('ProcessOutput', 'name timestamp data')
21 |
22 |
23 | def wrap_trailing_slash(value):
24 | value = str(value)
25 | if value.endswith('\\'):
26 | value = '"{0}"'.format(value)
27 | return value
28 |
29 |
30 | def append_if_not_none(container, **kwargs):
31 | for key, value in kwargs.items():
32 | if value is not None:
33 | value = wrap_trailing_slash(value)
34 | container.append('{0}={1}'.format(key, value))
35 |
36 |
37 | class Watcher(object):
38 | def __init__(self, papa_object):
39 | self.papa_object = papa_object
40 | self.connection = papa_object.connection
41 | self.exit_code = {}
42 | self._fileno = self.connection.sock.fileno()
43 | self._need_ack = False
44 |
45 | def __enter__(self):
46 | return self
47 |
48 | # noinspection PyUnusedLocal
49 | def __exit__(self, exc_type, exc_val, exc_tb):
50 | self.close()
51 |
52 | def __bool__(self):
53 | return self.connection is not None
54 |
55 | def __len__(self):
56 | return 1 if self.connection is not None else 0
57 |
58 | def fileno(self):
59 | return self.connection.sock.fileno()
60 |
61 | @property
62 | def ready(self):
63 | try:
64 | select.select([self], [], [], 0)
65 | return True
66 | except Exception:
67 | return False
68 |
69 | def read(self):
70 | self.acknowledge()
71 | reply = {'out': [], 'err': [], 'closed': []}
72 | if self.connection:
73 | line = b''
74 | while True:
75 | line = self.connection.get_one_line_response(b'] ')
76 | split = line.split(':')
77 | if len(split) < 4:
78 | break
79 | result_type, name, timestamp, data = split
80 | data = int(data)
81 | if result_type == 'closed':
82 | self.exit_code[name] = data
83 | else:
84 | data = self.connection.read_bytes(data + 1)[:-1]
85 | result = ProcessOutput(name, float(timestamp), data)
86 | reply[result_type].append(result)
87 | self._need_ack = line == '] '
88 | if not self._need_ack:
89 | self.connection.read_bytes(2)
90 | if not self.papa_object.connection:
91 | self.papa_object.connection = self.connection
92 | else:
93 | self.connection.close()
94 | self.connection = None
95 | return None
96 | return reply['out'], reply['err'], reply['closed']
97 |
98 | def acknowledge(self):
99 | if self._need_ack:
100 | send_with_retry(self.connection.sock, b'\n')
101 | self._need_ack = False
102 |
103 | def close(self):
104 | if self.connection:
105 | # if the server is waiting for an ack, we can
106 | if self._need_ack:
107 | send_with_retry(self.connection.sock, b'q\n')
108 | self._need_ack = False
109 | self.connection.get_full_response()
110 |
111 | # we can only recover the connection if we were able to send
112 | # the quit ack. otherwise close the connection and let the
113 | # socket die
114 | if not self.papa_object.connection:
115 | self.papa_object.connection = self.connection
116 | else:
117 | self.connection.close()
118 | else:
119 | self.connection.close()
120 | self.connection = None
121 |
122 |
123 | class ClientCommandConnection(object):
124 | def __init__(self, family, location):
125 | self.family = family
126 | self.location = location
127 | self.sock = self._attempt_to_connect()
128 | self.data = b''
129 |
130 | def _attempt_to_connect(self):
131 | # Try to connect to an existing Papa
132 | sock = socket.socket(self.family, socket.SOCK_STREAM)
133 | if self.family == socket.AF_INET:
134 | sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
135 | try:
136 | sock.connect(self.location)
137 | return sock
138 | except Exception:
139 | sock.close()
140 | raise
141 |
142 | def send_command(self, command):
143 | if isinstance(command, list):
144 | command = ' '.join(c.replace(' ', '\ ').replace('\n', '\ ') for c in command if c)
145 | command = b(command)
146 | send_with_retry(self.sock, command + b'\n')
147 |
148 | def do_command(self, command):
149 | self.send_command(command)
150 | return self.get_full_response()
151 |
152 | def get_full_response(self):
153 | data = self.data
154 | self.data = b''
155 | # noinspection PyTypeChecker
156 | while not data.endswith(b'\n> '):
157 | new_data = recv_with_retry(self.sock)
158 | if not new_data:
159 | raise utils.Error('Lost connection')
160 | data += new_data
161 |
162 | data = s(data[:-3])
163 | # noinspection PyTypeChecker
164 | if data.startswith('Error:'):
165 | raise utils.Error(data[7:])
166 | return data
167 |
168 | def get_one_line_response(self, alternate_terminator=None):
169 | data = self.data
170 | while b'\n' not in data:
171 | if alternate_terminator and data.endswith(alternate_terminator):
172 | break
173 | new_data = recv_with_retry(self.sock)
174 | if not new_data:
175 | raise utils.Error('Lost connection')
176 | data += new_data
177 |
178 | if data.startswith(b'Error:'):
179 | self.data = data
180 | return self.get_full_response()
181 |
182 | data, self.data = data.partition(b'\n')[::2]
183 | return s(data)
184 |
185 | def read_bytes(self, size):
186 | data = self.data
187 | while len(data) < size:
188 | new_data = recv_with_retry(self.sock, size - len(data))
189 | if not new_data:
190 | raise utils.Error('Lost connection')
191 | data += new_data
192 |
193 | data, self.data = data[:size], data[size:]
194 | return data
195 |
196 | def push_newline(self):
197 | self.data = b'\n' + self.data
198 |
199 | def close(self):
200 | return self.sock.close()
201 |
202 |
203 | class Papa(object):
204 | _debug_mode = False
205 | _single_connection_mode = False
206 | _default_port_or_path = 20202
207 | _default_connection_timeout = 10
208 |
209 | spawn_lock = Lock()
210 | spawned = False
211 |
212 | def __init__(self, port_or_path=None, connection_timeout=None):
213 | port_or_path = port_or_path or self._default_port_or_path
214 | self.port_or_path = port_or_path
215 | self.connection_timeout = connection_timeout or self._default_connection_timeout
216 | if isinstance(port_or_path, str):
217 | if not hasattr(socket, 'AF_UNIX'):
218 | raise NotImplementedError('Unix sockets are not supported on'
219 | ' this platform')
220 | if not os.path.isabs(port_or_path):
221 | raise utils.Error('Path to Unix socket must be absolute.')
222 | self.family = socket.AF_UNIX
223 | self.location = port_or_path
224 | else:
225 | self.family = socket.AF_INET
226 | self.location = ('127.0.0.1', port_or_path)
227 |
228 | # Try to connect to an existing Papa
229 | self.connection = None
230 | self.t = None
231 | self._connect(True)
232 |
233 | def __enter__(self):
234 | return self
235 |
236 | # noinspection PyUnusedLocal
237 | def __exit__(self, exc_type, exc_val, exc_tb):
238 | self.close()
239 | if self._single_connection_mode and self.t and not exc_type:
240 | self.t.join()
241 | Papa.spawned = False
242 |
243 | def _connect(self, allow_papa_spawn=False):
244 | try:
245 | self.connection = ClientCommandConnection(self.family, self.location)
246 | self.connection.get_full_response()
247 | except Exception:
248 | try_until = time() + self.connection_timeout
249 | while time() < try_until:
250 | if allow_papa_spawn and not Papa.spawned:
251 | self._spawn_papa_server()
252 | try:
253 | self.connection = ClientCommandConnection(self.family, self.location)
254 | self.connection.get_full_response()
255 | break
256 | except Exception:
257 | sleep(.1)
258 | if not self.connection:
259 | message = 'Could not connect to Papa in {0} seconds'.format(self.connection_timeout)
260 | log.error(message)
261 | raise utils.Error(message)
262 |
263 | def _spawn_papa_server(self):
264 | with Papa.spawn_lock:
265 | if not Papa.spawned:
266 | if self._debug_mode:
267 | from papa.server import socket_server
268 | from threading import Thread
269 | t = Thread(target=socket_server, args=(self.port_or_path, self._single_connection_mode))
270 | t.daemon = True
271 | t.start()
272 | self.t = t
273 | else:
274 | from papa.server import daemonize_server
275 | log.info('Daemonizing Papa')
276 | daemonize_server(self.port_or_path, fix_title=True)
277 | Papa.spawned = True
278 |
279 | def _attempt_to_connect(self):
280 | # Try to connect to an existing Papa
281 | sock = socket.socket(self.family, socket.SOCK_STREAM)
282 | if self.family == socket.AF_INET:
283 | sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
284 | sock.connect(self.location)
285 | return sock
286 |
287 | def _send_command(self, command):
288 | if not self.connection:
289 | self._connect()
290 | self.connection.send_command(command)
291 |
292 | def _do_command(self, command):
293 | if not self.connection:
294 | self._connect()
295 | return self.connection.do_command(command)
296 |
297 | @staticmethod
298 | def _make_socket_dict(socket_info):
299 | name, args = socket_info.partition(' ')[::2]
300 | args = dict(item.partition('=')[::2] for item in args.split(' '))
301 | for key in ('backlog', 'port', 'fileno'):
302 | if key in args:
303 | args[key] = int(args[key])
304 | return name, args
305 |
306 | def fileno(self):
307 | return self.connection.sock.fileno() if self.connection else None
308 |
309 | def list_sockets(self, *args):
310 | result = self._do_command(['l', 's'] + list(args))
311 | if not result:
312 | return {}
313 | # noinspection PyTypeChecker
314 | return dict(self._make_socket_dict(item) for item in result.split('\n'))
315 |
316 | def make_socket(self, name, host=None, port=None,
317 | family=None, socket_type=None,
318 | backlog=None, path=None, umask=None,
319 | interface=None, reuseport=None):
320 | if not name:
321 | raise utils.Error('Socket requires a name')
322 | command = ['m', 's', name]
323 | if family is not None:
324 | try:
325 | family_name = utils.valid_families_by_number[family]
326 | command.append('family={0}'.format(family_name))
327 | except KeyError:
328 | raise utils.Error('Invalid socket family')
329 | if socket_type is not None:
330 | try:
331 | type_name = utils.valid_types_by_number[socket_type]
332 | command.append('type={0}'.format(type_name))
333 | except KeyError:
334 | raise utils.Error('Invalid socket type')
335 | append_if_not_none(command, backlog=backlog)
336 | if path:
337 | if not path[0] == '/' or path[-1] in '/\\':
338 | raise utils.Error('Socket path must be absolute to a file')
339 | command.append('path={0}'.format(path))
340 | append_if_not_none(command, umask=umask)
341 | else:
342 | append_if_not_none(command, host=host, port=port, interface=interface)
343 | if reuseport:
344 | command.append('reuseport=1')
345 | return self._make_socket_dict(self._do_command(command))[1]
346 |
347 | def remove_sockets(self, *args):
348 | self._do_command(['r', 's'] + list(args))
349 | return True
350 |
351 | def list_values(self, *args):
352 | result = self._do_command(['l', 'v'] + list(args))
353 | if not result:
354 | return {}
355 | # noinspection PyTypeChecker
356 | return dict(item.partition(' ')[::2] for item in result.split('\n'))
357 |
358 | def set(self, name, value=None):
359 | command = ['set', name]
360 | if value:
361 | command.append(value)
362 | self._do_command(command)
363 |
364 | def get(self, name):
365 | result = self._do_command(['get', name])
366 | return result or None # do it this way so that '' becomes None
367 |
368 | def remove_values(self, *args):
369 | self._do_command(['r', 'v'] + list(args))
370 | return True
371 |
372 | @staticmethod
373 | def _make_process_dict(socket_info):
374 | name, arg_string = socket_info.partition(' ')[::2]
375 | args = {}
376 | last_key = None
377 | for item in arg_string.split(' '):
378 | key, delim, value = item.partition('=')
379 | if not delim and last_key:
380 | args[last_key] += ' ' + key
381 | else:
382 | last_key = key
383 | if key == 'pid':
384 | value = int(value)
385 | elif key == 'started':
386 | value = float(value)
387 | elif key in ('running', 'shell'):
388 | value = value == 'True'
389 | args[key] = value
390 | return name, args
391 |
392 | def list_processes(self, *args):
393 | result = self._do_command(['l', 'p'] + list(args))
394 | if not result:
395 | return {}
396 | # noinspection PyTypeChecker
397 | return dict(self._make_process_dict(item) for item in result.split('\n'))
398 |
399 | def make_process(self, name, executable=None, args=None, env=None, working_dir=None, uid=None, gid=None, rlimits=None, stdout=None, stderr=None, bufsize=None, watch_immediately=None):
400 | command = ['m', 'p', name]
401 | append_if_not_none(command, working_dir=working_dir, uid=uid, gid=gid, bufsize=bufsize)
402 | if watch_immediately:
403 | command.append('watch=1')
404 | if bufsize != 0:
405 | if stdout is not None:
406 | if stdout == DEVNULL:
407 | command.append('stdout=0')
408 | elif stdout != PIPE:
409 | raise utils.Error('stdout must be DEVNULL or PIPE')
410 | if stderr is not None:
411 | if stderr == DEVNULL:
412 | command.append('stderr=0')
413 | elif stderr == STDOUT:
414 | command.append('stderr=stdout')
415 | elif stderr != PIPE:
416 | raise utils.Error('stderr must be DEVNULL, PIPE or STDOUT')
417 | if env:
418 | for key, value in env.items():
419 | command.append('env.{0}={1}'.format(key, wrap_trailing_slash(value)))
420 | if rlimits:
421 | for key, value in rlimits.items():
422 | command.append('rlimit.{0}={1}'.format(key.lower(), value))
423 | if executable:
424 | command.append(executable)
425 | if args:
426 | if isinstance(args, string_type):
427 | command.append(args)
428 | else:
429 | try:
430 | command.extend(args)
431 | except TypeError:
432 | command.append(str(args))
433 | if watch_immediately:
434 | return self._do_watch(command)
435 | return self._make_process_dict(self._do_command(command))[1]
436 |
437 | def remove_processes(self, *args):
438 | self._do_command(['r', 'p'] + list(args))
439 | return True
440 |
441 | def watch_processes(self, *args):
442 | return self._do_watch(['w', 'p'] + list(args))
443 |
444 | def exit_if_idle(self):
445 | return self._do_command('exit-if-idle').startswith('Exiting')
446 |
447 | def _do_watch(self, command):
448 | self._send_command(command)
449 | self.connection.get_one_line_response()
450 | watcher = Watcher(self)
451 | self.connection = None
452 | return watcher
453 |
454 | def close(self):
455 | if self.connection:
456 | self.connection.close()
457 | self.connection = None
458 |
459 | @classmethod
460 | def set_debug_mode(cls, mode=True, quit_when_connection_closed=False):
461 | cls._debug_mode = mode
462 | if quit_when_connection_closed:
463 | cls._single_connection_mode = True
464 |
465 | @classmethod
466 | def set_default_port(cls, port):
467 | cls._default_port_or_path = port
468 |
469 | @classmethod
470 | def set_default_path(cls, path):
471 | cls._default_port_or_path = path
472 |
473 | @classmethod
474 | def set_default_connection_timeout(cls, connection_timeout):
475 | cls._default_connection_timeout = connection_timeout
476 |
477 |
478 | def set_debug_mode(mode=True, quit_when_connection_closed=False):
479 | return Papa.set_debug_mode(mode, quit_when_connection_closed)
480 |
481 |
482 | def set_default_port(port):
483 | return Papa.set_default_port(port)
484 |
485 |
486 | def set_default_path(path):
487 | return Papa.set_default_path(path)
488 |
489 |
490 | def set_default_connection_timeout(connection_timeout):
491 | return Papa.set_default_connection_timeout(connection_timeout)
492 |
493 |
494 | from papa import utils
495 | s = utils.cast_string
496 | b = utils.cast_bytes
497 | Error = utils.Error
498 |
499 | if __name__ == '__main__':
500 | set_debug_mode(quit_when_connection_closed=True)
501 | p = Papa()
502 | # print('Sockets: {0}'.format(p.list_sockets()))
503 | # print('Socket uwsgi: {0}'.format(p.make_socket('uwsgi', interface='eth0')))
504 | # print('Sockets: {0}'.format(p.list_sockets()))
505 | # print('Socket chaussette: {0}'.format(p.make_socket('chaussette', path='/tmp/chaussette.sock')))
506 | # print('Sockets: {0}'.format(p.list_sockets()))
507 | # print('Socket uwsgi6: {0}'.format(p.make_socket('uwsgi6', family=socket.AF_INET6)))
508 | # print('Sockets: {0}'.format(p.list_sockets()))
509 | # try:
510 | # print('Socket chaussette: {0}'.format(p.make_socket('chaussette')))
511 | # except Exception as e:
512 | # print('Caught exception: {0}'.format(e))
513 | # print('Close chaussette: {0}'.format(p.remove_sockets('chaussette')))
514 | # print('Socket chaussette: {0}'.format(p.make_socket('chaussette')))
515 | # print('Sockets: {0}'.format(p.list_sockets()))
516 | print('Processes: {0}'.format(p.list_processes()))
517 | # print('Values: {0}'.format(p.list_values()))
518 | # print('Set aack: {0}'.format(p.set('aack', 'bar')))
519 | # print('Get aack: {0}'.format(p.get('aack')))
520 | # print('Get bar: {0}'.format(p.get('bar')))
521 | # print('Values: {0}'.format(p.list_values()))
522 | # print('Set bar: {0}'.format(p.set('bar', 'barry')))
523 | # print('Get bar: {0}'.format(p.get('bar')))
524 | # print('Values: {0}'.format(p.list_values()))
525 | # print('Set bar: {0}'.format(p.set('bar')))
526 | # print('Get bar: {0}'.format(p.get('bar')))
527 | # print('Values: {0}'.format(p.list_values()))
528 | # print('Set aack: {0}'.format(p.set('aack')))
529 | p.close()
530 | print('Killed papa')
531 |
--------------------------------------------------------------------------------
/papa/server/__init__.py:
--------------------------------------------------------------------------------
1 | import os
2 | import sys
3 | import socket
4 | from threading import Thread, Lock
5 | import logging
6 | import resource
7 | import papa
8 | from papa.utils import Error, cast_bytes, cast_string, recv_with_retry, send_with_retry
9 | from papa.server import papa_socket, values, proc
10 | import atexit
11 | try:
12 | # noinspection PyPackageRequirements
13 | from setproctitle import setproctitle
14 | except ImportError:
15 | setproctitle = None
16 |
17 | __author__ = 'Scott Maxwell'
18 |
19 | try:
20 | from argparse import ArgumentParser
21 | except ImportError:
22 | from optparse import OptionParser
23 |
24 | class ArgumentParser(OptionParser):
25 | def add_argument(self, *args, **kwargs):
26 | return self.add_option(*args, **kwargs)
27 |
28 | # noinspection PyShadowingNames
29 | def parse_args(self, args=None, values=None):
30 | return OptionParser.parse_args(self, args, values)[0]
31 |
32 | log = logging.getLogger('papa.server')
33 |
34 |
35 | class CloseSocket(Exception):
36 | def __init__(self, final_message=None):
37 | self.final_message = final_message
38 | super(CloseSocket, self).__init__(self)
39 |
40 |
41 | # noinspection PyUnusedLocal
42 | def quit_command(sock, args, instance):
43 | """Close the client socket"""
44 | raise CloseSocket('ok\n')
45 |
46 |
47 | # noinspection PyUnusedLocal
48 | def exit_if_idle_command(sock, args, instance):
49 | """Exit papa if there are no processes, sockets or values"""
50 | instance_globals = instance['globals']
51 | if not is_idle(instance_globals):
52 | return "not idle"
53 | instance_globals['exit_if_idle'] = True
54 | raise CloseSocket('Exiting papa!\n> ')
55 |
56 |
57 | # noinspection PyUnusedLocal
58 | def help_command(sock, args, instance):
59 | """Show help info"""
60 | if args:
61 | try:
62 | help_for = lookup_command(args, allow_partials=True)
63 | except Error as e:
64 | return '{0}\n'.format(e)
65 | return help_for['__doc__'].strip() if isinstance(help_for, dict) else help_for.__doc__
66 | return """Possible commands are:
67 | make socket - Create a socket to be used by processes
68 | remove sockets - Close and remove sockets by name or file number
69 | list sockets - List sockets by name or file number
70 | -----------------------------------------------------
71 | make process - Launch a process
72 | remove processes - Stop recording the output of processes by name or PID
73 | list processes - List processes by name or PID
74 | watch processes - Start receiving the output of a processes by name or PID
75 | -----------------------------------------------------
76 | set - Set a named value
77 | get - Get a named value
78 | list values - List values by name
79 | remove processes - Remove values by name
80 | -----------------------------------------------------
81 | quit - Close the client session
82 | exit-if-idle Exit papa if there are no processes, sockets or values
83 | help - Type "help " for more information
84 |
85 | NOTE: All of these commands may be abbreviated. Type at least one character of
86 | each word. The rest are optional.
87 |
88 | After a 'watch' command, just enter '-' and a return to receive more output.
89 | """
90 |
91 |
92 | remove_doc = """
93 | Remove the socket, value or process output channel.
94 |
95 | You can remove process output channels by name or PID
96 | Examples:
97 | remove process 3698
98 | remove process nginx
99 |
100 | You can remove sockets by name or file number
101 | Examples:
102 | remove sockets uwsgi
103 | remove socket 10
104 |
105 | You can remove values by name
106 | Examples:
107 | remove values uwsgi.*
108 | remove value aack
109 |
110 | All commands can be abbreviated as much as you like, so the above can also be:
111 | r process 3698
112 | rem proc nginx
113 | r sockets uwsgi
114 | r s 10
115 | r v uwsgi.*
116 | re val aack
117 | """
118 |
119 | list_doc = """
120 | List sockets, processes or values.
121 |
122 | You can list processes by name or PID
123 | Examples:
124 | list process 3698
125 | list processes nginx.*
126 |
127 | You can list sockets by name or file number
128 | Examples:
129 | list socket 10
130 | list sockets uwsgi.*
131 |
132 | You can list values by name
133 | Examples:
134 | list values uwsgi.*
135 |
136 | All commands can be abbreviated as much as you like, so the above can also be:
137 | l process 3698
138 | lis proc nginx.*
139 | l s 10
140 | l sockets uwsgi.*
141 | li val uwsgi.*
142 | """
143 |
144 | make_doc = """
145 | Make a new socket or process.
146 |
147 | Do 'help make process' or 'help make socket' for details.
148 | """
149 |
150 | watch_doc = """
151 | Watch processes.
152 |
153 | You can watch processes by name or PID
154 | Examples:
155 | watch processes 3698
156 | watch processes nginx.*
157 |
158 | All commands can be abbreviated as much as you like, so the above can also be:
159 | w process 3698
160 | wat proc nginx.*
161 | """
162 |
163 |
164 | top_level_commands = {
165 | 'list': {
166 | 'sockets': papa_socket.sockets_command,
167 | 'processes': proc.processes_command,
168 | 'values': values.values_command,
169 | '__doc__': list_doc
170 | },
171 | 'make': {
172 | 'socket': papa_socket.socket_command,
173 | 'process': proc.process_command,
174 | '__doc__': make_doc
175 | },
176 | 'remove': {
177 | 'sockets': papa_socket.close_socket_command,
178 | 'processes': proc.close_output_command,
179 | 'values': values.remove_command,
180 | '__doc__': remove_doc
181 | },
182 | 'watch': {
183 | 'processes': proc.watch_command,
184 | '__doc__': watch_doc
185 | },
186 | 'set': values.set_command,
187 | 'get': values.get_command,
188 | 'quit': quit_command,
189 | 'exit-if-idle': exit_if_idle_command,
190 | 'help': help_command,
191 | }
192 |
193 |
194 | def lookup_command(args, commands=top_level_commands, primary_command=None, allow_partials=False):
195 | cmd = args.pop(0).lower()
196 | if cmd not in commands:
197 | for item in sorted(commands):
198 | if item.startswith(cmd):
199 | cmd = item
200 | if cmd == 'exit-if-idle':
201 | raise Error('You cannot abbreviate "exit-if-idle"')
202 | break
203 | else:
204 | if primary_command:
205 | raise Error('Bad "{0}" command. The following word must be one of: {1}'.format(primary_command, ', '.join(sorted(command for command in commands if not command.startswith('__')))))
206 | else:
207 | raise Error('Unknown command "{0}"'.format(cmd))
208 |
209 | result = commands[cmd]
210 | if isinstance(result, dict):
211 | if args:
212 | if primary_command:
213 | cmd = ' '.join((primary_command, cmd))
214 | return lookup_command(args, result, cmd, allow_partials)
215 | if not allow_partials:
216 | raise Error('"{0}" must be followed by one of: {1}'.format(cmd, ', '.join(sorted(command for command in result if not command.startswith('__')))))
217 | return result
218 |
219 |
220 | class ServerCommandConnection(object):
221 | def __init__(self, sock):
222 | self.sock = sock
223 | self.data = b''
224 |
225 | def readline(self):
226 | while not b'\n' in self.data:
227 | new_data = recv_with_retry(self.sock)
228 | if not new_data:
229 | raise socket.error('done')
230 | self.data += new_data
231 |
232 | one_line, self.data = self.data.partition(b'\n')[::2]
233 | return cast_string(one_line).strip()
234 |
235 |
236 | def chat_with_a_client(sock, addr, instance_globals, container):
237 | connection = ServerCommandConnection(sock)
238 | instance = {'globals': instance_globals, 'connection': connection}
239 | try:
240 | sock.send(b'Papa is home. Type "help" for commands.\n> ')
241 |
242 | while True:
243 | one_line = connection.readline()
244 | args = []
245 | acc = ''
246 | if one_line:
247 | for arg in one_line.split(' '):
248 | if arg:
249 | if arg[-1] == '\\':
250 | acc += arg[:-1] + ' '
251 | else:
252 | acc += arg
253 | args.append(acc.strip())
254 | acc = ''
255 | if acc:
256 | args.append(acc)
257 |
258 | try:
259 | command = lookup_command(args)
260 | except Error as e:
261 | reply = 'Error: {0}\n'.format(e)
262 | else:
263 | try:
264 | reply = command(sock, args, instance) or '\n'
265 | except CloseSocket as e:
266 | if e.final_message:
267 | send_with_retry(sock, cast_bytes(e.final_message))
268 | break
269 | except papa.utils.Error as e:
270 | reply = 'Error: {0}\n'.format(e)
271 | except Exception as e:
272 | reply = 'Error: {0}\n'.format(e)
273 |
274 | if reply[-1] != '\n':
275 | reply += '\n> '
276 | else:
277 | reply += '> '
278 | reply = cast_bytes(reply)
279 | else:
280 | reply = b'> '
281 | send_with_retry(sock, reply)
282 | except socket.error:
283 | pass
284 |
285 | try:
286 | sock.close()
287 | except socket.error:
288 | pass
289 |
290 | if container:
291 | thread_object = container[0]
292 | instance_globals['active_threads'].remove(thread_object)
293 | instance_globals['inactive_threads'].append((addr, thread_object))
294 |
295 |
296 | def cleanup(instance_globals):
297 | if 'lock' in instance_globals:
298 | papa_socket.cleanup(instance_globals)
299 |
300 |
301 | def is_idle(instance_globals):
302 | with instance_globals['lock']:
303 | return not instance_globals['processes']\
304 | and not instance_globals['sockets']['by_name']\
305 | and not instance_globals['sockets']['by_path']\
306 | and not instance_globals['values']
307 |
308 |
309 | def socket_server(port_or_path, single_socket_mode=False):
310 | instance_globals = {
311 | 'processes': {},
312 | 'sockets': {'by_name': {}, 'by_path': {}},
313 | 'values': {},
314 | 'active_threads': [],
315 | 'inactive_threads': [],
316 | 'lock': Lock(),
317 | 'exit_if_idle': False,
318 | }
319 | def local_cleanup():
320 | cleanup(instance_globals)
321 | atexit.register(local_cleanup)
322 | try:
323 | if isinstance(port_or_path, str):
324 | try:
325 | os.unlink(port_or_path)
326 | except OSError:
327 | if os.path.exists(port_or_path):
328 | raise
329 | s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
330 | try:
331 | s.bind(port_or_path)
332 | except socket.error as e:
333 | raise Error('Bind failed: {0}'.format(e))
334 | else:
335 | location = ('127.0.0.1', port_or_path)
336 | s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
337 | s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
338 | try:
339 | s.bind(location)
340 | except socket.error as e:
341 | raise Error('Bind failed or port {0}: {1}'.format(port_or_path, e))
342 | except Exception as e:
343 | log.exception(e)
344 | sys.exit(1)
345 |
346 | s.listen(5)
347 | log.info('Listening')
348 | while True:
349 | try:
350 | sock, addr = s.accept()
351 | log.info('Started client session with %s', addr)
352 | container = []
353 | instance_globals['exit_if_idle'] = False
354 | t = Thread(target=chat_with_a_client, args=(sock, addr, instance_globals, container))
355 | container.append(t)
356 | instance_globals['active_threads'].append(t)
357 | t.daemon = True
358 | t.start()
359 | s.settimeout(.5)
360 | except socket.timeout:
361 | pass
362 | while instance_globals['inactive_threads']:
363 | addr, t = instance_globals['inactive_threads'].pop()
364 | t.join()
365 | log.info('Closed client session with %s', addr)
366 | if not instance_globals['active_threads']:
367 | if single_socket_mode:
368 | break
369 | if instance_globals['exit_if_idle'] and is_idle(instance_globals):
370 | log.info('Exiting due to exit_if_idle request')
371 | break
372 | s.settimeout(None)
373 | s.close()
374 | papa_socket.cleanup(instance_globals)
375 | try:
376 | # noinspection PyUnresolvedReferences
377 | atexit.unregister(local_cleanup)
378 | except AttributeError:
379 | del instance_globals['lock']
380 |
381 |
382 | def daemonize_server(port_or_path, fix_title=False):
383 | process_id = os.fork()
384 | if process_id < 0:
385 | raise Error('Unable to fork')
386 | elif process_id != 0:
387 | return
388 |
389 | # noinspection PyNoneFunctionAssignment,PyArgumentList
390 | process_id = os.setsid()
391 | if process_id == -1:
392 | sys.exit(1)
393 |
394 | for fd in range(3, resource.getrlimit(resource.RLIMIT_NOFILE)[0]):
395 | try:
396 | os.close(fd)
397 | except OSError:
398 | pass
399 |
400 | devnull = os.devnull if hasattr(os, 'devnull') else '/dev/null'
401 | devnull_fd = os.open(devnull, os.O_RDWR)
402 | for fd in range(3):
403 | # noinspection PyTypeChecker
404 | os.dup2(devnull_fd, fd)
405 |
406 | os.umask(0o27)
407 | os.chdir('/')
408 | if fix_title and setproctitle is not None:
409 | # noinspection PyCallingNonCallable
410 | setproctitle('papa daemon from %s' % os.path.basename(sys.argv[0]))
411 | socket_server(port_or_path)
412 |
413 |
414 | def main():
415 | parser = ArgumentParser('papa', description='A simple parent process for sockets and other processes')
416 | parser.add_argument('-d', '--debug', action='store_true', help='run in debug mode')
417 | parser.add_argument('-u', '--unix-socket', help='path to unix socket to bind')
418 | parser.add_argument('-p', '--port', default=20202, type=int, help='port to bind on localhost (default 20202)')
419 | parser.add_argument('--daemonize', action='store_true', help='daemonize the papa server')
420 | args = parser.parse_args()
421 |
422 | logging.basicConfig(level=logging.INFO if args.debug else logging.ERROR)
423 | if args.daemonize:
424 | daemonize_server(args.unix_socket or args.port)
425 | else:
426 | try:
427 | socket_server(args.unix_socket or args.port)
428 | except Exception as e:
429 | log.exception(e)
430 |
431 | if __name__ == '__main__':
432 | main()
433 |
--------------------------------------------------------------------------------
/papa/server/papa_socket.py:
--------------------------------------------------------------------------------
1 | import os
2 | import os.path
3 | import socket
4 | import logging
5 | from papa import utils, Error
6 | from papa.utils import extract_name_value_pairs, wildcard_iter
7 |
8 | __author__ = 'Scott Maxwell'
9 |
10 | log = logging.getLogger('papa.server')
11 |
12 | if hasattr(socket, 'AF_UNIX'):
13 | unix_socket = socket.AF_UNIX
14 | else:
15 | unix_socket = None
16 |
17 |
18 | class PapaSocket(object):
19 |
20 | # noinspection PyShadowingBuiltins
21 | def __init__(self, name, instance, family=None, type='stream',
22 | backlog=5, path=None, umask=None,
23 | host=None, port=0, interface=None, reuseport=False):
24 |
25 | if path and unix_socket is None:
26 | raise NotImplemented('Unix sockets are not supported on this system')
27 |
28 | instance_globals = instance['globals']
29 | self._sockets_by_name = instance_globals['sockets']['by_name']
30 | self._sockets_by_path = instance_globals['sockets']['by_path']
31 | self.name = name
32 | if family:
33 | self.family = utils.valid_families[family]
34 | else:
35 | self.family = unix_socket if path else socket.AF_INET
36 | self.socket_type = utils.valid_types[type]
37 | self.backlog = int(backlog)
38 | self.path = self.umask = None
39 | self.host = self.port = self.interface = self.reuseport = None
40 | self.socket = None
41 |
42 | if self.family == unix_socket:
43 | if not path or not os.path.isabs(path):
44 | raise utils.Error('Absolute path required for Unix sockets')
45 | self.path = path
46 | self.umask = None if umask is None else int(umask)
47 | else:
48 | self.port = int(port)
49 | self.interface = interface
50 |
51 | if host:
52 | self.host = self._host = host
53 | target_length = 4 if self.family == socket.AF_INET6 else 2
54 | for info in socket.getaddrinfo(host, self.port):
55 | if len(info[-1]) == target_length:
56 | self._host = info[-1][0]
57 | break
58 | else:
59 | if self.family == socket.AF_INET6:
60 | self.host = '::' if interface else '::1'
61 | else:
62 | self.host = '0.0.0.0' if interface else '127.0.0.1'
63 | self._host = self.host
64 |
65 | self.reuseport = reuseport if reuseport and hasattr(socket, 'SO_REUSEPORT') else False
66 |
67 | def __str__(self):
68 | data = [self.name,
69 | 'family={0}'.format(utils.valid_families_by_number[self.family]),
70 | 'type={0}'.format(utils.valid_types_by_number[self.socket_type])]
71 | if self.backlog is not None:
72 | data.append('backlog={0}'.format(self.backlog))
73 | if self.path is not None:
74 | data.append('path={0}'.format(self.path))
75 | if self.umask is not None:
76 | data.append('umask={0}'.format(self.umask))
77 | if self.host is not None:
78 | data.append('host={0}'.format(self.host))
79 | if self.port is not None:
80 | data.append('port={0}'.format(self.port))
81 | if self.interface is not None:
82 | data.append('interface={0}'.format(self.interface))
83 | if self.reuseport:
84 | data.append('reuseport={0}'.format(self.reuseport))
85 | if self.socket:
86 | data.append('fileno={0}'.format(self.socket.fileno()))
87 | return ' '.join(data)
88 |
89 | def __eq__(self, other):
90 | # compare all but reuseport, since reuseport might change state on
91 | # socket start
92 | return (
93 | self.name == other.name and
94 | self.family == other.family and
95 | self.socket_type == other.socket_type and
96 | self.backlog == other.backlog and
97 | self.path == other.path and
98 | self.umask == other.umask and
99 | self.host == other.host and
100 | (self.port == other.port or not self.port) and
101 | self.interface == other.interface
102 | )
103 |
104 | def start(self):
105 | existing = self._sockets_by_name.get(self.name)
106 | if existing:
107 | if self == existing:
108 | self.socket = existing.socket
109 | self.port = existing.port
110 | else:
111 | raise utils.Error('Socket for {0} has already been created - {1}'.format(self.name, str(existing)))
112 | else:
113 | if self.family == unix_socket:
114 | if self.path in self._sockets_by_path:
115 | raise utils.Error('Socket for {0} has already been created'.format(self.path))
116 | try:
117 | os.unlink(self.path)
118 | except OSError:
119 | if os.path.exists(self.path):
120 | raise
121 | s = socket.socket(self.family, self.socket_type)
122 | try:
123 | if self.umask is None:
124 | s.bind(self.path)
125 | else:
126 | old_mask = os.umask(self.umask)
127 | s.bind(self.path)
128 | os.umask(old_mask)
129 | except socket.error as e:
130 | raise utils.Error('Bind failed: {0}'.format(e))
131 | self._sockets_by_path[self.path] = self
132 | else:
133 | s = socket.socket(self.family, self.socket_type)
134 | s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
135 | if self.interface:
136 | import IN
137 | if hasattr(IN, 'SO_BINDTODEVICE'):
138 | s.setsockopt(socket.SOL_SOCKET, IN.SO_BINDTODEVICE,
139 | self.interface + '\0')
140 | try:
141 | s.bind((self._host, self.port))
142 | except socket.error as e:
143 | raise utils.Error('Bind failed on {0}:{1}: {2}'.format(self.host, self.port, e))
144 | if not self.port:
145 | self.port = s.getsockname()[1]
146 |
147 | if self.reuseport:
148 | try:
149 | s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
150 | s.close()
151 | s = None
152 | except socket.error:
153 | self.reuseport = False
154 | # noinspection PyUnresolvedReferences
155 | if s:
156 | s.listen(self.backlog)
157 | try:
158 | s.set_inheritable(True)
159 | except Exception:
160 | pass
161 | self.socket = s
162 | self._sockets_by_name[self.name] = self
163 | log.info('Created socket %s', self)
164 | return self
165 |
166 | def clone_for_reuseport(self):
167 | s = socket.socket(self.family, self.socket_type)
168 | s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
169 | if self.interface:
170 | import IN
171 | if hasattr(IN, 'SO_BINDTODEVICE'):
172 | s.setsockopt(socket.SOL_SOCKET, IN.SO_BINDTODEVICE,
173 | self.interface + '\0')
174 | try:
175 | s.bind((self._host, self.port))
176 | except socket.error as e:
177 | raise utils.Error('Bind failed on {0}:{1}: {2}'.format(self.host, self.port, e))
178 |
179 | s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
180 | s.listen(self.backlog)
181 | try:
182 | s.set_inheritable(True)
183 | except Exception:
184 | pass
185 | return s
186 |
187 | def close(self):
188 | if self.socket:
189 | self.socket.close()
190 | log.info('Closed socket %s', self)
191 | if self.path:
192 | del self._sockets_by_path[self.path]
193 | try:
194 | os.unlink(self.path)
195 | except Exception:
196 | pass
197 | del self._sockets_by_name[self.name]
198 |
199 |
200 | # noinspection PyUnusedLocal
201 | def socket_command(sock, args, instance):
202 | """Create a socket to be used by processes.
203 | You need to specify a name, followed by name=value pairs for the connection
204 | options. The name must not contain spaces.
205 |
206 | Family and type options are:
207 | family - should be unix, inet, inet6 (default is unix if path is specified,
208 | of inet if no path)
209 | type - should be stream, dgram, raw, rdm or seqpacket (default is stream)
210 | backlog - specifies the listen backlog (default is 5)
211 |
212 | Options for family=unix
213 | path - must be an absolute path (required)
214 | umask - override the current umask when creating the socket file
215 |
216 | Options for family=inet or family=inet6
217 | port - if left out, the system will assign a port
218 | interface - only bind to a single ethernet adaptor
219 | host - you will usually want the default, which will be 127.0.0.1 if no
220 | interface it specified and 0.0.0.0 otherwise
221 | reuseport - on systems that support it, papa will create and bind a new
222 | socket for each process that uses this socket
223 |
224 | The url must start with "tcp:", "udp:" or "unix:".
225 | Examples:
226 | make socket uwsgi port=8080
227 | make socket chaussette path=/tmp/chaussette.sock
228 | """
229 | if not args:
230 | raise Error('Socket requires a name')
231 | name = args.pop(0)
232 | kwargs = extract_name_value_pairs(args)
233 | p = PapaSocket(name, instance, **kwargs)
234 | with instance['globals']['lock']:
235 | return str(p.start())
236 |
237 |
238 | # noinspection PyUnusedLocal
239 | def close_socket_command(sock, args, instance):
240 | """Close and remove socket or sockets
241 |
242 | You can remove sockets by name or file number
243 | Examples:
244 | remove sockets uwsgi
245 | remove socket 10
246 | """
247 | instance_globals = instance['globals']
248 | with instance_globals['lock']:
249 | for name, p in wildcard_iter(instance_globals['sockets']['by_name'], args, required=True):
250 | p.close()
251 |
252 |
253 | # noinspection PyUnusedLocal
254 | def sockets_command(sock, args, instance):
255 | """List active sockets.
256 |
257 | You can list sockets by name or file number
258 | Examples:
259 | list socket 10
260 | list socket uwsgi.*
261 | """
262 | instance_globals = instance['globals']
263 | with instance_globals['lock']:
264 | return '\n'.join(sorted('{0}'.format(s) for _, s in wildcard_iter(instance_globals['sockets']['by_name'], args)))
265 |
266 |
267 | def cleanup(instance_globals):
268 | with instance_globals['lock']:
269 | for p in list(instance_globals['sockets']['by_name'].values()):
270 | p.close()
271 |
272 |
273 | def find_socket(name, instance):
274 | instance_globals = instance['globals']
275 | return instance_globals['sockets']['by_name'][name]
276 |
--------------------------------------------------------------------------------
/papa/server/proc.py:
--------------------------------------------------------------------------------
1 | import os
2 | import sys
3 | import logging
4 | import ctypes
5 | import select
6 | import socket
7 | import fcntl
8 | from time import time
9 | from papa import utils, Error
10 | from papa.utils import extract_name_value_pairs, wildcard_iter, cast_bytes, \
11 | send_with_retry
12 | from papa.server.papa_socket import find_socket
13 | from subprocess import Popen, PIPE, STDOUT
14 | from threading import Thread, Lock
15 | from collections import deque, namedtuple
16 |
17 | try:
18 | import pwd
19 | except ImportError:
20 | pwd = None
21 |
22 | try:
23 | import grp
24 | except ImportError:
25 | grp = None
26 |
27 | try:
28 | import resource
29 | except ImportError:
30 | resource = None
31 |
32 | try:
33 | # noinspection PyUnresolvedReferences,PyUnboundLocalVariable
34 | FileNotFoundError
35 | except NameError as e:
36 | # noinspection PyShadowingBuiltins
37 | FileNotFoundError = OSError
38 |
39 | __author__ = 'Scott Maxwell'
40 |
41 | log = logging.getLogger('papa.server')
42 |
43 |
44 | def convert_size_string_to_bytes(s):
45 | try:
46 | return int(s)
47 | except ValueError:
48 | return int(s[:-1]) * {'g': 1073741824, 'm': 1048576, 'k': 1024}[s[-1].lower()]
49 |
50 |
51 | class OutputQueue(object):
52 | Item = namedtuple('Item', 'type timestamp data')
53 | STDOUT = 0
54 | STDERR = 1
55 | CLOSED = -1
56 |
57 | def __init__(self, bufsize=1048576):
58 | self.lock = Lock()
59 | self.bufsize = bufsize
60 | self.q = deque()
61 | self._used = 0
62 | self._closed = False
63 |
64 | def add(self, output_type, data=None):
65 | if not self._closed:
66 | with self.lock:
67 | if not self._closed:
68 | data_tuple = OutputQueue.Item(output_type, time(), data)
69 | if output_type != OutputQueue.CLOSED and data:
70 | if len(data) >= self.bufsize:
71 | self.q.clear()
72 | self._used = len(data)
73 | else:
74 | self._used += len(data)
75 | while self._used > self.bufsize:
76 | first = self.q.popleft()
77 | self._used -= len(first.data)
78 | self.q.append(data_tuple)
79 |
80 | def retrieve(self):
81 | if self.q:
82 | with self.lock:
83 | if self.q:
84 | l = list(self.q)
85 | return l[-1].timestamp, l
86 | return 0, None
87 |
88 | def remove(self, timestamp):
89 | with self.lock:
90 | q = self.q
91 | while q and q[0].timestamp <= timestamp:
92 | item = q.popleft()
93 | if self._used:
94 | self._used -= len(item.data)
95 |
96 | def close(self):
97 | with self.lock:
98 | self.bufsize = 0
99 | self.q = deque()
100 | self._used = 0
101 | self._closed = True
102 |
103 | def __len__(self):
104 | return len(self.q)
105 |
106 |
107 | class Process(object):
108 | """Wraps a process.
109 |
110 | Options:
111 |
112 | - **name**: the process name. Multiple processes can share the same name.
113 |
114 | - **args**: the arguments for the command to run. Can be a list or
115 | a string. If **args** is a string, it's splitted using
116 | :func:`shlex.split`. Defaults to None.
117 |
118 | - **working_dir**: the working directory to run the command in. If
119 | not provided, will default to the current working directory.
120 |
121 | - **shell**: if *True*, will run the command in the shell
122 | environment. *False* by default. **warning: this is a
123 | security hazard**.
124 |
125 | - **uid**: if given, is the user id or name the command should run
126 | with. The current uid is the default.
127 |
128 | - **gid**: if given, is the group id or name the command should run
129 | with. The current gid is the default.
130 |
131 | - **env**: a mapping containing the environment variables the command
132 | will run with. Optional.
133 |
134 | - **rlimits**: a mapping containing rlimit names and values that will
135 | be set before the command runs.
136 | """
137 | def __init__(self, name, args, env, rlimits, instance,
138 | working_dir=None, shell=False, uid=None, gid=None,
139 | stdout=1, stderr=1, bufsize='1m'):
140 |
141 | self.instance = instance
142 | instance_globals = instance['globals']
143 | self._processes = instance_globals['processes']
144 |
145 | self.name = name
146 | self.args = args
147 | self.env = env
148 | self.rlimits = rlimits
149 | self.working_dir = working_dir
150 | self.shell = shell
151 | self.bufsize = convert_size_string_to_bytes(bufsize)
152 |
153 | self.pid = 0
154 | self.running = False
155 | self.started = 0
156 |
157 | if self.bufsize:
158 | self.out = int(stdout)
159 | self.err = stderr if stderr == 'stdout' else int(stderr)
160 | else:
161 | self.out = self.err = 0
162 |
163 | if uid:
164 | if pwd:
165 | try:
166 | self.uid = int(uid)
167 | self.username = pwd.getpwuid(self.uid).pw_name
168 | except KeyError:
169 | raise utils.Error('%r is not a valid user id' % uid)
170 | except ValueError:
171 | try:
172 | self.username = uid
173 | self.uid = pwd.getpwnam(uid).pw_uid
174 | except KeyError:
175 | raise utils.Error('%r is not a valid user name' % uid)
176 | else:
177 | raise utils.Error('uid is not supported on this platform')
178 | else:
179 | self.username = None
180 | self.uid = None
181 |
182 | if gid:
183 | if grp:
184 | try:
185 | self.gid = int(gid)
186 | grp.getgrgid(self.gid)
187 | except (KeyError, OverflowError):
188 | raise utils.Error('No such group: %r' % gid)
189 | except ValueError:
190 | try:
191 | self.gid = grp.getgrnam(gid).gr_gid
192 | except KeyError:
193 | raise utils.Error('No such group: %r' % gid)
194 | else:
195 | raise utils.Error('gid is not supported on this platform')
196 | elif self.uid:
197 | self.gid = pwd.getpwuid(self.uid).pw_gid
198 | else:
199 | self.gid = None
200 |
201 | # sockets created before fork, should be let go after.
202 | self._worker = None
203 | self._thread = None
204 | self._output = None
205 | self._auto_close = False
206 |
207 | def __eq__(self, other):
208 | return (
209 | self.name == other.name and
210 | self.args == other.args and
211 | self.env == other.env and
212 | self.rlimits == other.rlimits and
213 | self.working_dir == other.working_dir and
214 | self.shell == other.shell and
215 | self.out == other.out and
216 | self.err == other.err and
217 | self.bufsize == other.bufsize and
218 | self.uid == other.uid and
219 | self.gid == other.gid
220 | )
221 |
222 | def spawn(self):
223 | existing = self._processes.get(self.name)
224 | if existing:
225 | if self == existing:
226 | self.pid = existing.pid
227 | self.running = existing.running
228 | self.started = existing.started
229 | else:
230 | raise utils.Error('Process for {0} has already been created - {1}'.format(self.name, str(existing)))
231 | else:
232 | managed_sockets = []
233 | fixed_args = []
234 | self.started = time()
235 | for arg in self.args:
236 | if '$(socket.' in arg:
237 | start = arg.find('$(socket.') + 9
238 | end = arg.find(')', start)
239 | if end == -1:
240 | raise utils.Error('Process for {0} argument starts with "$(socket." but has no closing parenthesis'.format(self.name))
241 | socket_and_part = arg[start:end]
242 | socket_name, part = socket_and_part.rpartition('.')[::2]
243 | if not part or part not in ('port', 'fileno'):
244 | raise utils.Error('You forgot to specify either ".port" or ".fileno" after the name')
245 | try:
246 | s = find_socket(socket_name, self.instance)
247 | except Exception:
248 | raise utils.Error('Socket {0} not found'.format(socket_name))
249 | if part == 'port':
250 | replacement = s.port
251 | elif s.reuseport:
252 | sock = s.clone_for_reuseport()
253 | managed_sockets.append(sock)
254 | replacement = sock.fileno()
255 | else:
256 | replacement = s.socket.fileno()
257 | arg = '{0}{1}{2}'.format(arg[:start - 9], replacement, arg[end + 1:])
258 | fixed_args.append(arg)
259 |
260 | if not fixed_args:
261 | raise utils.Error('No command')
262 |
263 | def preexec():
264 | streams = [sys.stdin]
265 | if not self.out:
266 | streams.append(sys.stdout)
267 | if not self.err:
268 | streams.append(sys.stderr)
269 | for stream in streams:
270 | if hasattr(stream, 'fileno'):
271 | try:
272 | stream.flush()
273 | devnull = os.open(os.devnull, os.O_RDWR)
274 | # noinspection PyTypeChecker
275 | os.dup2(devnull, stream.fileno())
276 | # noinspection PyTypeChecker
277 | os.close(devnull)
278 | except IOError:
279 | # some streams, like stdin - might be already closed.
280 | pass
281 |
282 | # noinspection PyArgumentList
283 | os.setsid()
284 |
285 | if resource:
286 | for limit, value in self.rlimits.items():
287 | resource.setrlimit(limit, (value, value))
288 |
289 | if self.gid:
290 | try:
291 | # noinspection PyTypeChecker
292 | os.setgid(self.gid)
293 | except OverflowError:
294 | if not ctypes:
295 | raise
296 | # versions of python < 2.6.2 don't manage unsigned int for
297 | # groups like on osx or fedora
298 | os.setgid(-ctypes.c_int(-self.gid).value)
299 |
300 | if self.username is not None:
301 | try:
302 | # noinspection PyTypeChecker
303 | os.initgroups(self.username, self.gid)
304 | except (OSError, AttributeError):
305 | # not support on Mac or 2.6
306 | pass
307 |
308 | if self.uid:
309 | # noinspection PyTypeChecker
310 | os.setuid(self.uid)
311 |
312 | extra = {}
313 | if self.out:
314 | extra['stdout'] = PIPE
315 |
316 | if self.err:
317 | if self.err == 'stdout':
318 | extra['stderr'] = STDOUT
319 | else:
320 | extra['stderr'] = PIPE
321 |
322 | try:
323 | self._worker = Popen(fixed_args, preexec_fn=preexec,
324 | close_fds=False, shell=self.shell,
325 | cwd=self.working_dir, env=self.env, bufsize=-1,
326 | **extra)
327 | except FileNotFoundError as file_not_found_exception:
328 | if not os.path.exists(fixed_args[0]):
329 | raise utils.Error('Bad command - {0}'.format(file_not_found_exception))
330 | if self.working_dir and not os.path.isdir(self.working_dir):
331 | raise utils.Error('Bad working_dir - {0}'.format(file_not_found_exception))
332 | raise
333 | # let go of sockets created only for self.worker to inherit
334 | for sock in managed_sockets:
335 | sock.close()
336 | self._processes[self.name] = self
337 | self.pid = self._worker.pid
338 | self._output = OutputQueue(self.bufsize)
339 | log.info('Created process %s', self)
340 |
341 | self.running = True
342 | self._thread = Thread(target=self._watch)
343 | self._thread.daemon = True
344 | self._thread.start()
345 |
346 | return self
347 |
348 | def _watch(self):
349 | pipes = []
350 | stdout = self._worker.stdout
351 | stderr = self._worker.stderr
352 | output = self._output
353 | if self.out:
354 | pipes.append(stdout)
355 | if self.err and self.err != 'stdout':
356 | pipes.append(stderr)
357 | if pipes:
358 | for pipe in pipes:
359 | fd = pipe.fileno()
360 | fl = fcntl.fcntl(fd, fcntl.F_GETFL)
361 | fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)
362 | data = True
363 | while data and not self._auto_close:
364 | out = select.select(pipes, [], [])[0]
365 | for p in out:
366 | data = p.read()
367 | if data:
368 | output.add(OutputQueue.STDOUT if p == stdout else OutputQueue.STDERR, data)
369 |
370 | if stdout:
371 | stdout.close()
372 | if stderr:
373 | stderr.close()
374 | out = self._worker.wait()
375 | self.running = False
376 | if self._auto_close:
377 | instance_globals = self.instance['globals']
378 | with instance_globals['lock']:
379 | log.info('Removed process %s', self)
380 | instance_globals['processes'].pop(self.name, None)
381 | else:
382 | output.add(OutputQueue.CLOSED, out)
383 |
384 | def __str__(self):
385 | result = ['{0} pid={1} running={2} started={3}'.format(self.name, self.pid, self.running, self.started)]
386 | if self.uid:
387 | result.append('uid={0}'.format(self.uid))
388 | if self.gid:
389 | result.append('gid={0}'.format(self.gid))
390 | if self.shell:
391 | result.append('shell=True')
392 | # if self.env:
393 | # result.extend('env.{0}={1}'.format(key, value) for key, value in self.env.items())
394 | if self.args:
395 | result.append('args={0}'.format(' '.join(self.args)))
396 | return ' '.join(result)
397 |
398 | def watch(self):
399 | # noinspection PyTypeChecker
400 | return self._output.retrieve()
401 |
402 | def remove_output(self, timestamp):
403 | self._output.remove(timestamp)
404 |
405 | def close_output(self):
406 | self._output.close()
407 | self._auto_close = True
408 | if not self.running:
409 | self.instance['globals']['processes'].pop(self.name, None)
410 |
411 |
412 | # noinspection PyUnusedLocal
413 | def process_command(sock, args, instance):
414 | """Create a process.
415 | You need to specify a name, followed by name=value pairs for the process
416 | options, followed by the command and args to execute. The name must not contain
417 | spaces.
418 |
419 | Process options are:
420 | uid - the username or user ID to use when starting the process
421 | gid - the group name or group ID to use when starting the process
422 | working_dir - must be an absolute path if specified
423 | output - size of each output buffer (default is 1m)
424 |
425 | You can also specify environment variables by prefixing the name with 'env.' and
426 | rlimits by prefixing the name with 'rlimit.'
427 |
428 | Examples:
429 | make process sf uid=1001 gid=2000 working_dir=/sf/bin/ output=1m /sf/bin/uwsgi --ini uwsgi-live.ini --socket fd://27 --stats 127.0.0.1:8090
430 | make process nginx /usr/local/nginx/sbin/nginx
431 | """
432 | if not args:
433 | raise Error('Process requires a name')
434 | name = args.pop(0)
435 | env = {}
436 | rlimits = {}
437 | kwargs = {}
438 | for key, value in extract_name_value_pairs(args).items():
439 | if key.startswith('env.'):
440 | env[key[4:]] = value
441 | elif key.startswith('rlimit.'):
442 | key = key[7:]
443 | try:
444 | rlimits[getattr(resource, 'RLIMIT_%s' % key.upper())] = int(value)
445 | except AttributeError:
446 | raise utils.Error('Unknown rlimit "%s"' % key)
447 | except ValueError:
448 | raise utils.Error('The rlimit value for "%s" must be an integer, not "%s"' % (key, value))
449 | else:
450 | kwargs[key] = value
451 | watch = int(kwargs.pop('watch', 0))
452 | p = Process(name, args, env, rlimits, instance, **kwargs)
453 | with instance['globals']['lock']:
454 | result = p.spawn()
455 | if watch:
456 | send_with_retry(sock, cast_bytes('{0}\n'.format(result)))
457 | return _do_watch(sock, {name: {'p': result, 't': 0, 'closed': False}}, instance)
458 |
459 | return str(result)
460 |
461 |
462 | # noinspection PyUnusedLocal
463 | def processes_command(sock, args, instance):
464 | """List active processes.
465 |
466 | You can list processes by name or PID
467 | Examples:
468 | list process 3698
469 | list processes nginx.*
470 | """
471 | instance_globals = instance['globals']
472 | with instance_globals['lock']:
473 | return '\n'.join(sorted('{0}'.format(proc) for _, proc in wildcard_iter(instance_globals['processes'], args)))
474 |
475 |
476 | # noinspection PyUnusedLocal
477 | def close_output_command(sock, args, instance):
478 | """Close the process output channels and automatically remove the process from
479 | the list on completion.
480 |
481 | You can remove processes by name or PID
482 | Examples:
483 | remove processes uwsgi
484 | remove process 10
485 | """
486 | """Close the process output channels and automatically remove the process when done"""
487 | instance_globals = instance['globals']
488 | with instance_globals['lock']:
489 | for name, p in wildcard_iter(instance_globals['processes'], args, required=True):
490 | p.close_output()
491 |
492 |
493 | def watch_command(sock, args, instance):
494 | """Watch a process"""
495 | instance_globals = instance['globals']
496 | all_processes = instance_globals['processes']
497 | with instance_globals['lock']:
498 | procs = dict((name, {'p': proc, 't': 0, 'closed': False}) for name, proc in wildcard_iter(all_processes, args, True))
499 | if not procs:
500 | raise utils.Error('Nothing to watch')
501 | send_with_retry(sock, cast_bytes('Watching {0}\n'.format(len(procs))))
502 | return _do_watch(sock, procs, instance)
503 |
504 |
505 | if hasattr(select, 'poll'):
506 | class Poller(object):
507 | def __init__(self, sock):
508 | self.p = select.poll()
509 | self.p.register(sock.fileno(), select.POLLHUP)
510 |
511 | def poll(self, timeout):
512 | return self.p.poll(timeout * 1000)
513 |
514 | else:
515 | class Poller(object):
516 | def __init__(self, sock):
517 | self.sock = sock
518 |
519 | def poll(self, timeout):
520 | if not select.select([self.sock], [], [], timeout)[0]:
521 | self.sock.setblocking(0)
522 | try:
523 | b = self.sock.recv(1)
524 | if not b:
525 | return True
526 | except socket.error:
527 | pass
528 | self.sock.setblocking(1)
529 |
530 |
531 | def _do_watch(sock, procs, instance):
532 | instance_globals = instance['globals']
533 | all_processes = instance_globals['processes']
534 | connection = instance['connection']
535 | poller = Poller(sock)
536 | delay = .1
537 | while True:
538 | data = []
539 | for name, proc in procs.items():
540 | t, l = proc['p'].watch()
541 | if l:
542 | for item in l:
543 | if item.type == OutputQueue.CLOSED:
544 | data.append(cast_bytes('closed:{0}:{1}:{2}'.format(name, item.timestamp, item.data)))
545 | proc['closed'] = True
546 | else:
547 | data.append(cast_bytes('{0}:{1}:{2}:{3}'.format('out' if item.type == OutputQueue.STDOUT else 'err', name, item.timestamp, len(item.data))))
548 | data.append(item.data)
549 | proc['t'] = t
550 | if data:
551 | delay = .05
552 | data.append(b'] ')
553 | out = b'\n'.join(data)
554 | send_with_retry(sock, out)
555 |
556 | one_line = connection.readline().lower()
557 | closed = []
558 | for name, proc in procs.items():
559 | t = proc['t']
560 | if t:
561 | proc['p'].remove_output(t)
562 | if proc['closed']:
563 | closed.append(name)
564 | if closed:
565 | with instance_globals['lock']:
566 | for name in closed:
567 | closed_proc = procs.pop(name, None)
568 | if closed_proc and 'p' in closed_proc:
569 | log.info('Removed process %s', closed_proc['p'])
570 | all_processes.pop(name, None)
571 | if not procs:
572 | return 'Nothing left to watch'
573 | if one_line == 'q':
574 | return 'Stopped watching'
575 | else:
576 | if poller.poll(delay):
577 | return 'Client closed connection'
578 | if delay < 1.0:
579 | delay += .05
580 |
--------------------------------------------------------------------------------
/papa/server/values.py:
--------------------------------------------------------------------------------
1 | from papa.utils import wildcard_iter, Error
2 |
3 | __author__ = 'Scott Maxwell'
4 |
5 |
6 | # noinspection PyUnusedLocal
7 | def values_command(sock, args, instance):
8 | """Return all values stored in Papa"""
9 | instance_globals = instance['globals']
10 | with instance_globals['lock']:
11 | return '\n'.join(sorted('{0} {1}'.format(key, value) for key, value in wildcard_iter(instance_globals['values'], args)))
12 |
13 |
14 | # noinspection PyUnusedLocal
15 | def set_command(sock, args, instance):
16 | """Set or clear a named value. Pass no value to clear.
17 |
18 | Examples:
19 | set count 5
20 | set count
21 | """
22 | instance_globals = instance['globals']
23 | values = instance_globals['values']
24 | with instance_globals['lock']:
25 | if not args or args == ['*']:
26 | raise Error('Value requires a name')
27 | name = args.pop(0)
28 | if args:
29 | values[name] = ' '.join(args)
30 | else:
31 | values.pop(name, None)
32 |
33 |
34 | # noinspection PyUnusedLocal
35 | def remove_command(sock, args, instance):
36 | """Remove a named value or set of values. You cannot 'remove *'.
37 |
38 | Examples:
39 | remove count
40 | remove circus.*
41 | """
42 | if not args or args == ['*']:
43 | raise Error('You cannot remove all variables')
44 | instance_globals = instance['globals']
45 | values = instance_globals['values']
46 | with instance_globals['lock']:
47 | for name, _ in wildcard_iter(values, args):
48 | del values[name]
49 |
50 |
51 | # noinspection PyUnusedLocal
52 | def get_command(sock, args, instance):
53 | """Get a named value.
54 |
55 | Example:
56 | get count
57 | """
58 | if not args:
59 | raise Error('Value requires a name')
60 | instance_globals = instance['globals']
61 | with instance_globals['lock']:
62 | return instance_globals['values'].get(args[0])
63 |
--------------------------------------------------------------------------------
/papa/utils.py:
--------------------------------------------------------------------------------
1 | import socket
2 | import sys
3 | import select
4 |
5 | __author__ = 'Scott Maxwell'
6 |
7 | valid_families = dict((name[3:].lower(), getattr(socket, name))
8 | for name in dir(socket)
9 | if name.startswith('AF_') and getattr(socket, name))
10 | valid_families_by_number = dict((value, name)
11 | for name, value in valid_families.items())
12 | valid_types = dict((name[5:].lower(), getattr(socket, name))
13 | for name in dir(socket)
14 | if name.startswith('SOCK_'))
15 | valid_types_by_number = dict((value, name)
16 | for name, value in valid_types.items())
17 |
18 | PY2 = sys.version_info[0] < 3
19 |
20 |
21 | class Error(RuntimeError):
22 | pass
23 |
24 |
25 | if PY2:
26 | def cast_bytes(s, encoding='utf8'):
27 | """cast unicode or bytes to bytes"""
28 | # noinspection PyUnresolvedReferences
29 | if isinstance(s, unicode):
30 | return s.encode(encoding)
31 | return str(s)
32 |
33 | # noinspection PyUnusedLocal
34 | def cast_unicode(s, encoding='utf8', errors='replace'):
35 | """cast bytes or unicode to unicode.
36 | errors options are strict, ignore or replace"""
37 | # noinspection PyUnresolvedReferences
38 | if isinstance(s, unicode):
39 | return s
40 | # noinspection PyUnresolvedReferences
41 | return str(s).decode(encoding)
42 |
43 | # noinspection PyUnusedLocal
44 | def cast_string(s, errors='replace'):
45 | # noinspection PyUnresolvedReferences
46 | return s if isinstance(s, basestring) else str(s)
47 |
48 | # noinspection PyUnresolvedReferences
49 | string_type = basestring
50 |
51 | else:
52 | def cast_bytes(s, encoding='utf8'): # NOQA
53 | """cast unicode or bytes to bytes"""
54 | if isinstance(s, bytes):
55 | return s
56 | return str(s).encode(encoding)
57 |
58 | def cast_unicode(s, encoding='utf8', errors='replace'): # NOQA
59 | """cast bytes or unicode to unicode.
60 | errors options are strict, ignore or replace"""
61 | if isinstance(s, bytes):
62 | return s.decode(encoding, errors=errors)
63 | return str(s)
64 |
65 | cast_string = cast_unicode
66 | string_type = str
67 |
68 |
69 | def extract_name_value_pairs(args):
70 | var_dict = {}
71 | while args and '=' in args[0]:
72 | arg = args.pop(0)
73 | name, value = arg.partition('=')[::2]
74 | if value[0] == '"' and value[-1] == '"':
75 | value = value[1:-1]
76 | var_dict[name] = value
77 | return var_dict
78 |
79 |
80 | def wildcard_iter(d, matches, required=False):
81 | if not matches or matches == '*':
82 | matched = set(d.keys())
83 | else:
84 | matched = set()
85 | for match in matches:
86 | if match and match[-1] == '*':
87 | match = match[:-1]
88 | if not match:
89 | matched = set(d.keys())
90 | else:
91 | for name in d.keys():
92 | if name.startswith(match):
93 | matched.add(name)
94 | elif match in d:
95 | matched.add(match)
96 | elif required:
97 | raise Error('{0} not found'.format(match))
98 | for name in matched:
99 | yield name, d[name]
100 |
101 |
102 | def recv_with_retry(sock, size=1024):
103 | while True:
104 | try:
105 | return sock.recv(size)
106 | except socket.error as e:
107 | if e.errno == 35:
108 | select.select([sock], [], [])
109 | else:
110 | raise
111 |
112 |
113 | def send_with_retry(sock, data):
114 | while data:
115 | try:
116 | sent = sock.send(data)
117 | if sent:
118 | data = data[sent:]
119 | except socket.error as e:
120 | if e.errno == 35:
121 | select.select([], [sock], [])
122 | else:
123 | raise
124 |
--------------------------------------------------------------------------------
/setup.py:
--------------------------------------------------------------------------------
1 | import sys
2 |
3 | try:
4 | from setuptools import setup
5 | except ImportError:
6 | from distutils.core import setup
7 |
8 | try:
9 | # noinspection PyPackageRequirements,PyUnresolvedReferences
10 | from pypandoc import convert
11 | read_md = lambda f: convert(f, 'rst')
12 | except ImportError:
13 | if 'dist' in sys.argv or 'sdist' in sys.argv:
14 | print("warning: pypandoc module not found, could not convert Markdown to RST")
15 | read_md = lambda f: open(f, 'r').read()
16 |
17 | setup(
18 | name="papa",
19 | version="1.0.6",
20 | packages=["papa", "papa.server", "tests"],
21 | author="Scott Maxwell",
22 | author_email="scott@codecobblers.com",
23 | maintainer="Scott Maxwell",
24 | url="https://github.com/scottkmaxwell/papa",
25 | description="Simple socket and process kernel",
26 | long_description=read_md('README.md'),
27 | license="MIT",
28 | classifiers=["Development Status :: 5 - Production/Stable",
29 | "Environment :: Console",
30 | "Intended Audience :: Developers",
31 | "License :: OSI Approved :: MIT License",
32 | "Operating System :: MacOS :: MacOS X",
33 | "Operating System :: POSIX :: Linux",
34 | "Operating System :: POSIX :: BSD :: FreeBSD",
35 | "Programming Language :: Python",
36 | "Programming Language :: Python :: 2.6",
37 | "Programming Language :: Python :: 2.7",
38 | "Programming Language :: Python :: 3",
39 | "Programming Language :: Python :: 3.3",
40 | "Programming Language :: Python :: 3.4",
41 | "Topic :: Software Development"],
42 | tests_require=['unittest2'] if sys.version_info[:2] == (2, 6) else [],
43 | test_suite="tests",
44 | entry_points="""\
45 | [console_scripts]
46 | papa = papa.server:main
47 | """,
48 | zip_safe=True
49 | )
50 |
--------------------------------------------------------------------------------
/tests/__init__.py:
--------------------------------------------------------------------------------
1 | from random import randint
2 | import os
3 | import sys
4 | import os.path
5 | import socket
6 | from time import sleep
7 | try:
8 | # noinspection PyPackageRequirements
9 | import unittest2 as unittest
10 | except ImportError:
11 | import unittest
12 | import select
13 | import papa
14 | from papa.server.papa_socket import unix_socket
15 | from papa.utils import cast_bytes
16 | from tempfile import gettempdir
17 | import logging
18 |
19 | logging.basicConfig()
20 | import inspect
21 |
22 | isdebugging = False
23 | for frame in inspect.stack():
24 | if frame[1].endswith("pydevd.py"):
25 | isdebugging = True
26 | break
27 |
28 | here = os.path.dirname(os.path.realpath(__file__))
29 |
30 |
31 | class SocketTest(unittest.TestCase):
32 | def setUp(self):
33 | papa.set_debug_mode(quit_when_connection_closed=True)
34 |
35 | def check_subset(self, expected, result):
36 | for key, value in expected.items():
37 | self.assertIn(key, result)
38 | self.assertEqual(value, result[key])
39 |
40 | def test_inet(self):
41 | with papa.Papa() as p:
42 | expected = {'type': 'stream', 'family': 'inet', 'backlog': 5, 'host': '127.0.0.1'}
43 | reply = p.make_socket('inet_sock')
44 | self.check_subset(expected, reply)
45 | self.assertIn('port', reply)
46 | p.remove_sockets('inet_sock')
47 | self.assertDictEqual({}, p.list_sockets())
48 | reply = p.make_socket('inet_sock', reuseport=True)
49 | self.check_subset(expected, reply)
50 | self.assertIn('port', reply)
51 |
52 | def test_inet_interface(self):
53 | with papa.Papa() as p:
54 | expected = {'type': 'stream', 'interface': 'eth0', 'family': 'inet', 'backlog': 5, 'host': '0.0.0.0'}
55 | self.assertDictEqual({}, p.list_sockets())
56 |
57 | reply = p.make_socket('interface_socket', interface='eth0')
58 | self.assertIn('port', reply)
59 | self.assertIn('fileno', reply)
60 | expected['port'] = reply['port']
61 | expected['fileno'] = reply['fileno']
62 | self.assertDictEqual(expected, reply)
63 | self.assertDictEqual({'interface_socket': expected}, p.list_sockets())
64 | p.remove_sockets('interface_socket')
65 | self.assertDictEqual({}, p.list_sockets())
66 | reply = p.make_socket('interface_socket', interface='eth0', port=expected['port'])
67 | self.assertDictEqual(expected, reply)
68 | self.assertDictEqual({'interface_socket': expected}, p.list_sockets())
69 |
70 | def test_inet6(self):
71 | with papa.Papa() as p:
72 | expected = {'type': 'stream', 'family': 'inet6', 'backlog': 5, 'host': '::1'}
73 | reply = p.make_socket('inet6_sock', family=socket.AF_INET6)
74 | self.check_subset(expected, reply)
75 | self.assertIn('port', reply)
76 | p.remove_sockets('inet6_sock')
77 | self.assertDictEqual({}, p.list_sockets())
78 | reply = p.make_socket('inet6_sock', family=socket.AF_INET6, reuseport=True)
79 | self.check_subset(expected, reply)
80 | self.assertIn('port', reply)
81 |
82 | def test_inet6_interface(self):
83 | with papa.Papa() as p:
84 | expected = {'type': 'stream', 'interface': 'eth0', 'family': 'inet6', 'backlog': 5, 'host': '::'}
85 | self.assertDictEqual({}, p.list_sockets())
86 |
87 | reply = p.make_socket('interface_socket', family=socket.AF_INET6, interface='eth0')
88 | self.assertIn('port', reply)
89 | self.assertIn('fileno', reply)
90 | expected['port'] = reply['port']
91 | expected['fileno'] = reply['fileno']
92 | self.assertDictEqual(expected, reply)
93 | self.assertDictEqual({'interface_socket': expected}, p.list_sockets())
94 |
95 | @unittest.skipIf(unix_socket is None, 'Unix socket not supported on this platform')
96 | def test_file_socket(self):
97 | with papa.Papa() as p:
98 | path = os.path.join(gettempdir(), 'tst.sock')
99 | expected = {'path': path, 'backlog': 5, 'type': 'stream', 'family': 'unix'}
100 |
101 | reply = p.make_socket('fsock', path=path)
102 | self.assertIn('fileno', reply)
103 | expected['fileno'] = reply['fileno']
104 | self.assertDictEqual(expected, reply)
105 | self.assertDictEqual({'fsock': expected}, p.list_sockets())
106 |
107 | self.assertRaises(papa.Error, p.make_socket, 'fsock', path='path')
108 |
109 | def test_already_exists(self):
110 | with papa.Papa() as p:
111 | reply = p.make_socket('exists_sock')
112 | self.assertDictEqual(reply, p.make_socket('exists_sock'))
113 | self.assertRaises(papa.Error, p.make_socket, 'exists_sock', family=socket.AF_INET6)
114 |
115 | def test_wildcard(self):
116 | with papa.Papa() as p:
117 | expected = {'type': 'stream', 'family': 'inet', 'backlog': 5, 'host': '127.0.0.1'}
118 |
119 | reply = p.make_socket('inet.0')
120 | self.check_subset(expected, reply)
121 | self.assertIn('port', reply)
122 |
123 | reply = p.make_socket('inet.1')
124 | self.check_subset(expected, reply)
125 | self.assertIn('port', reply)
126 |
127 | reply = p.make_socket('other')
128 | self.check_subset(expected, reply)
129 | self.assertIn('port', reply)
130 |
131 | reply = p.list_sockets('inet.*')
132 | self.assertEqual(2, len(list(reply.keys())))
133 | self.assertEqual(['inet.0', 'inet.1'], sorted(reply.keys()))
134 |
135 | reply = p.list_sockets('other')
136 | self.assertEqual(1, len(list(reply.keys())))
137 | self.assertEqual(['other'], list(reply.keys()))
138 |
139 | reply = p.list_sockets('not_there')
140 | self.assertEqual({}, reply)
141 |
142 | reply = p.list_sockets('other', 'inet.1')
143 | self.assertEqual(2, len(list(reply.keys())))
144 | self.assertEqual(['inet.1', 'other'], sorted(reply.keys()))
145 |
146 | reply = p.list_sockets('other', 'inet*')
147 | self.assertEqual(3, len(list(reply.keys())))
148 | self.assertEqual(['inet.0', 'inet.1', 'other'], sorted(reply.keys()))
149 |
150 | reply = p.list_sockets('*')
151 | self.assertEqual(3, len(list(reply.keys())))
152 | self.assertEqual(['inet.0', 'inet.1', 'other'], sorted(reply.keys()))
153 |
154 | reply = p.list_sockets()
155 | self.assertEqual(3, len(list(reply.keys())))
156 | self.assertEqual(['inet.0', 'inet.1', 'other'], sorted(reply.keys()))
157 |
158 |
159 | class ValueTest(unittest.TestCase):
160 | def setUp(self):
161 | papa.set_debug_mode(quit_when_connection_closed=True)
162 |
163 | def test_value(self):
164 | with papa.Papa() as p:
165 | self.assertEqual(None, p.get('aack'))
166 | self.assertDictEqual({}, p.list_values())
167 |
168 | p.set('aack', 'bar')
169 | self.assertEqual('bar', p.get('aack'))
170 | self.assertDictEqual({'aack': 'bar'}, p.list_values())
171 |
172 | p.set('aack2', 'barry')
173 | self.assertEqual('barry', p.get('aack2'))
174 | self.assertDictEqual({'aack': 'bar', 'aack2': 'barry'}, p.list_values())
175 |
176 | p.set('aack3', 'larry')
177 | self.assertEqual('larry', p.get('aack3'))
178 | self.assertDictEqual({'aack': 'bar', 'aack2': 'barry', 'aack3': 'larry'}, p.list_values())
179 |
180 | p.set('bar', 'aack')
181 | self.assertEqual('aack', p.get('bar'))
182 | self.assertDictEqual({'aack': 'bar', 'aack2': 'barry', 'aack3': 'larry', 'bar': 'aack'}, p.list_values())
183 |
184 | self.assertDictEqual({'aack': 'bar', 'aack2': 'barry', 'aack3': 'larry', 'bar': 'aack'}, p.list_values('*'))
185 | self.assertDictEqual({'aack': 'bar', 'aack2': 'barry', 'aack3': 'larry'}, p.list_values('aack*'))
186 | self.assertDictEqual({'bar': 'aack'}, p.list_values('b*'))
187 | self.assertDictEqual({'aack2': 'barry', 'bar': 'aack'}, p.list_values('aack2', 'b*'))
188 |
189 | p.set('aack')
190 | self.assertEqual(None, p.get('aack'))
191 | self.assertDictEqual({'aack2': 'barry', 'aack3': 'larry'}, p.list_values('a*'))
192 |
193 | p.remove_values('aack*')
194 | self.assertDictEqual({'bar': 'aack'}, p.list_values())
195 |
196 | def test_wildcard_clear(self):
197 | with papa.Papa() as p:
198 | self.assertRaises(papa.Error, p.remove_values)
199 | self.assertRaises(papa.Error, p.remove_values, '*')
200 |
201 |
202 | class ProcessTest(unittest.TestCase):
203 | def setUp(self):
204 | papa.set_debug_mode(quit_when_connection_closed=True)
205 |
206 | @staticmethod
207 | def _merge_lines(output):
208 | if len(output) > 1:
209 | by_name = {}
210 | merged_lines = False
211 | for line in output:
212 | by_name.setdefault(line.name, []).append(line)
213 | for named_lines in by_name.values():
214 | line_number = 1
215 | while line_number < len(named_lines):
216 | line = named_lines[line_number]
217 | prev = named_lines[line_number - 1]
218 | if line.timestamp - prev.timestamp > .05:
219 | line_number += 1
220 | else:
221 | named_lines[line_number - 1] = papa.ProcessOutput(prev.name, prev.timestamp, prev.data + line.data)
222 | del named_lines[line_number]
223 | merged_lines = True
224 | if merged_lines:
225 | output = sorted((item for items in by_name.values() for item in items), key=lambda x: x.timestamp)
226 | return output
227 |
228 | def gather_output(self, watcher):
229 | out = []
230 | err = []
231 | close = []
232 | while watcher:
233 | reply = watcher.read()
234 | if reply:
235 | out.extend(reply[0])
236 | err.extend(reply[1])
237 | close.extend(reply[2])
238 | return self._merge_lines(out), self._merge_lines(err), close
239 |
240 | if isdebugging:
241 | _non_debug_gather_output = gather_output
242 |
243 | def _filter_list(self, output):
244 | remove = []
245 | for line_number, line in enumerate(output):
246 | if line.data.startswith(b'pydev debugger: process ') and b'is connecting' in line.data:
247 | if line.data.endswith(b'is connecting\n\n'):
248 | remove.append(line_number)
249 | else:
250 | end = line.data.find(b'is connecting\n\n')
251 | output[line_number] = papa.ProcessOutput(line.name, line.timestamp, line.data[end + 14:])
252 | for line_number in reversed(remove):
253 | del output[line_number]
254 |
255 | def gather_output(self, watcher):
256 | out, err, close = self._non_debug_gather_output(watcher)
257 | self._filter_list(out)
258 | self._filter_list(err)
259 | return out, err, close
260 |
261 | def test_process_with_out_and_err(self):
262 | with papa.Papa() as p:
263 | self.assertDictEqual({}, p.list_processes())
264 | reply1 = p.make_process('write3', sys.executable, args='executables/write_three_lines.py', working_dir=here, uid=os.environ['LOGNAME'], env=os.environ)
265 | self.assertIn('pid', reply1)
266 | self.assertIsInstance(reply1['pid'], int)
267 | reply = p.list_processes()
268 | self.assertEqual(1, len(list(reply.keys())))
269 | self.assertEqual('write3', list(reply.keys())[0])
270 | self.assertIn('pid', list(reply.values())[0])
271 |
272 | reply2 = p.make_process('write3', sys.executable, args='executables/write_three_lines.py', working_dir=here, uid=os.environ['LOGNAME'], env=os.environ)
273 | self.assertEqual(reply1['pid'], reply2['pid'])
274 |
275 | self.assertRaises(papa.Error, p.watch_processes, 'not_there')
276 |
277 | with p.watch_processes('write*') as w:
278 | select.select([w], [], [])
279 | self.assertTrue(w.ready)
280 | out, err, close = self.gather_output(w)
281 | exit_code = w.exit_code['write3']
282 | self.assertEqual(3, len(out))
283 | self.assertEqual(1, len(err))
284 | self.assertEqual(1, len(close))
285 | self.assertEqual('write3', out[0].name)
286 | self.assertEqual('write3', out[1].name)
287 | self.assertEqual('write3', out[2].name)
288 | self.assertEqual('write3', err[0].name)
289 | self.assertEqual('write3', close[0].name)
290 | self.assertLess(out[0].timestamp, out[1].timestamp)
291 | self.assertLess(out[1].timestamp, out[2].timestamp)
292 | self.assertLessEqual(out[2].timestamp, err[0].timestamp)
293 | self.assertLessEqual(out[2].timestamp, close[0].timestamp)
294 | self.assertLessEqual(err[0].timestamp, close[0].timestamp)
295 | self.assertEqual(b'Version: ' + cast_bytes(sys.version.partition(' ')[0]) + b'\n', out[0].data)
296 | self.assertEqual(b'Executable: ' + cast_bytes(sys.executable) + b'\n', out[1].data)
297 | self.assertEqual(b'Args: \n', out[2].data)
298 | self.assertEqual(b'done', err[0].data)
299 | self.assertEqual(0, close[0].data)
300 | self.assertEqual(0, exit_code)
301 | self.assertDictEqual({}, p.list_processes())
302 |
303 | def test_process_with_none_executable(self):
304 | with papa.Papa() as p:
305 | self.assertDictEqual({}, p.list_processes())
306 | reply1 = p.make_process('write3', None, args=[sys.executable, 'executables/write_three_lines.py'], working_dir=here, uid=os.environ['LOGNAME'], env=os.environ)
307 | self.assertIn('pid', reply1)
308 | self.assertIsInstance(reply1['pid'], int)
309 | reply = p.list_processes()
310 | self.assertEqual(1, len(list(reply.keys())))
311 | self.assertEqual('write3', list(reply.keys())[0])
312 | self.assertIn('pid', list(reply.values())[0])
313 |
314 | reply2 = p.make_process('write3', sys.executable, args='executables/write_three_lines.py', working_dir=here, uid=os.environ['LOGNAME'], env=os.environ)
315 | self.assertEqual(reply1['pid'], reply2['pid'])
316 |
317 | self.assertRaises(papa.Error, p.watch_processes, 'not_there')
318 |
319 | with p.watch_processes('write*') as w:
320 | select.select([w], [], [])
321 | self.assertTrue(w.ready)
322 | out, err, close = self.gather_output(w)
323 | exit_code = w.exit_code['write3']
324 | self.assertEqual(3, len(out))
325 | self.assertEqual(1, len(err))
326 | self.assertEqual(1, len(close))
327 | self.assertEqual('write3', out[0].name)
328 | self.assertEqual('write3', out[1].name)
329 | self.assertEqual('write3', out[2].name)
330 | self.assertEqual('write3', err[0].name)
331 | self.assertEqual('write3', close[0].name)
332 | self.assertLess(out[0].timestamp, out[1].timestamp)
333 | self.assertLess(out[1].timestamp, out[2].timestamp)
334 | self.assertLessEqual(out[2].timestamp, err[0].timestamp)
335 | self.assertLessEqual(out[2].timestamp, close[0].timestamp)
336 | self.assertLessEqual(err[0].timestamp, close[0].timestamp)
337 | self.assertEqual(b'Version: ' + cast_bytes(sys.version.partition(' ')[0]) + b'\n', out[0].data)
338 | self.assertEqual(b'Executable: ' + cast_bytes(sys.executable) + b'\n', out[1].data)
339 | self.assertEqual(b'Args: \n', out[2].data)
340 | self.assertEqual(b'done', err[0].data)
341 | self.assertEqual(0, close[0].data)
342 | self.assertEqual(0, exit_code)
343 | self.assertDictEqual({}, p.list_processes())
344 |
345 | def test_process_with_watch_immediately(self):
346 | with papa.Papa() as p:
347 | self.assertDictEqual({}, p.list_processes())
348 | with p.make_process('write3', sys.executable, args='executables/write_three_lines.py', working_dir=here, uid=os.environ['LOGNAME'], env=os.environ, watch_immediately=True) as w:
349 | out, err, close = self.gather_output(w)
350 | exit_code = w.exit_code['write3']
351 | self.assertEqual(3, len(out))
352 | self.assertEqual(1, len(err))
353 | self.assertEqual(1, len(close))
354 | self.assertEqual('write3', out[0].name)
355 | self.assertEqual('write3', out[1].name)
356 | self.assertEqual('write3', out[2].name)
357 | self.assertEqual('write3', err[0].name)
358 | self.assertEqual('write3', close[0].name)
359 | self.assertLess(out[0].timestamp, out[1].timestamp)
360 | self.assertLess(out[1].timestamp, out[2].timestamp)
361 | self.assertLessEqual(out[2].timestamp, err[0].timestamp)
362 | self.assertLessEqual(out[2].timestamp, close[0].timestamp)
363 | self.assertLessEqual(err[0].timestamp, close[0].timestamp)
364 | self.assertEqual(b'Version: ' + cast_bytes(sys.version.partition(' ')[0]) + b'\n', out[0].data)
365 | self.assertEqual(b'Executable: ' + cast_bytes(sys.executable) + b'\n', out[1].data)
366 | self.assertEqual(b'Args: \n', out[2].data)
367 | self.assertEqual(b'done', err[0].data)
368 | self.assertEqual(0, close[0].data)
369 | self.assertEqual(0, exit_code)
370 | self.assertDictEqual({}, p.list_processes())
371 |
372 | def test_process_with_err_redirected_to_out(self):
373 | with papa.Papa() as p:
374 | self.assertDictEqual({}, p.list_processes())
375 | reply1 = p.make_process('write3', sys.executable, args='executables/write_three_lines.py', working_dir=here, uid=os.environ['LOGNAME'], env=os.environ, stderr=papa.STDOUT)
376 | self.assertIn('pid', reply1)
377 | self.assertIsInstance(reply1['pid'], int)
378 | reply = p.list_processes()
379 | self.assertEqual(1, len(list(reply.keys())))
380 | self.assertEqual('write3', list(reply.keys())[0])
381 | self.assertIn('pid', list(reply.values())[0])
382 |
383 | with p.watch_processes('write*') as w:
384 | out, err, close = self.gather_output(w)
385 | exit_code = w.exit_code['write3']
386 | self.assertEqual(3, len(out))
387 | self.assertEqual(0, len(err))
388 | self.assertEqual(1, len(close))
389 | self.assertEqual('write3', out[0].name)
390 | self.assertEqual('write3', out[1].name)
391 | self.assertEqual('write3', out[2].name)
392 | self.assertEqual('write3', close[0].name)
393 | self.assertLess(out[0].timestamp, out[1].timestamp)
394 | self.assertLess(out[1].timestamp, out[2].timestamp)
395 | self.assertLessEqual(out[2].timestamp, close[0].timestamp)
396 | self.assertEqual(b'Version: ' + cast_bytes(sys.version.partition(' ')[0]) + b'\n', out[0].data)
397 | self.assertEqual(b'Executable: ' + cast_bytes(sys.executable) + b'\n', out[1].data)
398 | self.assertEqual(b'Args: \ndone', out[2].data)
399 | self.assertEqual(0, close[0].data)
400 | self.assertEqual(0, exit_code)
401 | self.assertDictEqual({}, p.list_processes())
402 |
403 | def test_process_with_no_out(self):
404 | with papa.Papa() as p:
405 | self.assertDictEqual({}, p.list_processes())
406 | reply1 = p.make_process('write3', sys.executable, args='executables/write_three_lines.py', working_dir=here, uid=os.environ['LOGNAME'], env=os.environ, stdout=papa.DEVNULL)
407 | self.assertIn('pid', reply1)
408 | self.assertIsInstance(reply1['pid'], int)
409 | reply = p.list_processes()
410 | self.assertEqual(1, len(list(reply.keys())))
411 | self.assertEqual('write3', list(reply.keys())[0])
412 | self.assertIn('pid', list(reply.values())[0])
413 |
414 | with p.watch_processes('write*') as w:
415 | out, err, close = self.gather_output(w)
416 | exit_code = w.exit_code['write3']
417 | self.assertEqual(0, len(out))
418 | self.assertEqual(1, len(err))
419 | self.assertEqual(1, len(close))
420 | self.assertEqual('write3', err[0].name)
421 | self.assertEqual('write3', close[0].name)
422 | self.assertLessEqual(err[0].timestamp, close[0].timestamp)
423 | self.assertEqual(b'done', err[0].data)
424 | self.assertEqual(0, close[0].data)
425 | self.assertEqual(0, exit_code)
426 | self.assertDictEqual({}, p.list_processes())
427 |
428 | def test_process_with_no_buffer(self):
429 | with papa.Papa() as p:
430 | self.assertDictEqual({}, p.list_processes())
431 | reply1 = p.make_process('write3', sys.executable, args='executables/write_three_lines.py', working_dir=here, uid=os.environ['LOGNAME'], env=os.environ, bufsize=0)
432 | self.assertIn('pid', reply1)
433 | self.assertIsInstance(reply1['pid'], int)
434 | reply = p.list_processes()
435 | self.assertEqual(1, len(list(reply.keys())))
436 | self.assertEqual('write3', list(reply.keys())[0])
437 | self.assertIn('pid', list(reply.values())[0])
438 |
439 | with p.watch_processes('write*') as w:
440 | out, err, close = self.gather_output(w)
441 | exit_code = w.exit_code['write3']
442 | self.assertEqual(0, len(out))
443 | self.assertEqual(0, len(err))
444 | self.assertEqual(1, len(close))
445 | self.assertEqual('write3', close[0].name)
446 | self.assertEqual(0, close[0].data)
447 | self.assertEqual(0, exit_code)
448 | self.assertDictEqual({}, p.list_processes())
449 |
450 | def test_two_list_processes_full_output(self):
451 | with papa.Papa() as p:
452 | self.assertDictEqual({}, p.list_processes())
453 | reply1 = p.make_process('write3.0', sys.executable, args='executables/write_three_lines.py', working_dir=here, uid=os.environ['LOGNAME'], env=os.environ)
454 | self.assertIn('pid', reply1)
455 | self.assertIsInstance(reply1['pid'], int)
456 |
457 | reply2 = p.make_process('write3.1', sys.executable, args='executables/write_three_lines.py', working_dir=here, uid=os.environ['LOGNAME'], env=os.environ)
458 | self.assertIn('pid', reply2)
459 | self.assertIsInstance(reply2['pid'], int)
460 |
461 | reply = p.list_processes()
462 | self.assertEqual(2, len(list(reply.keys())))
463 | self.assertEqual(['write3.0', 'write3.1'], sorted(reply.keys()))
464 | self.assertIn('pid', list(reply.values())[0])
465 | self.assertIn('pid', list(reply.values())[1])
466 | self.assertNotEqual(list(reply.values())[0]['pid'], list(reply.values())[1]['pid'])
467 |
468 | with p.watch_processes('write3.*') as w:
469 | select.select([w], [], [])
470 | self.assertTrue(w.ready)
471 | out, err, close = self.gather_output(w)
472 | exit_code0 = w.exit_code['write3.0']
473 | exit_code1 = w.exit_code['write3.1']
474 | self.assertEqual(6, len(out))
475 | self.assertEqual(2, len(err))
476 | self.assertEqual(2, len(close))
477 | self.assertEqual(3, len([item for item in out if item.name == 'write3.0']))
478 | self.assertEqual(3, len([item for item in out if item.name == 'write3.1']))
479 | self.assertEqual(1, len([item for item in err if item.name == 'write3.0']))
480 | self.assertEqual(1, len([item for item in err if item.name == 'write3.1']))
481 | self.assertEqual(1, len([item for item in close if item.name == 'write3.0']))
482 | self.assertEqual(1, len([item for item in close if item.name == 'write3.1']))
483 | self.assertEqual(b'done', err[0].data)
484 | self.assertEqual(b'done', err[1].data)
485 | self.assertEqual(0, close[0].data)
486 | self.assertEqual(0, exit_code0)
487 | self.assertEqual(0, close[1].data)
488 | self.assertEqual(0, exit_code1)
489 | self.assertDictEqual({}, p.list_processes())
490 |
491 | def test_two_list_processes_wait_for_one_to_close(self):
492 | with papa.Papa() as p:
493 | f = p.fileno()
494 | self.assertDictEqual({}, p.list_processes())
495 | reply1 = p.make_process('write3.0', sys.executable, args='executables/write_three_lines.py', working_dir=here, uid=os.environ['LOGNAME'], env=os.environ)
496 | self.assertIn('pid', reply1)
497 | self.assertIsInstance(reply1['pid'], int)
498 |
499 | sleep(.2)
500 | reply2 = p.make_process('write3.1', sys.executable, args='executables/write_three_lines.py', working_dir=here, uid=os.environ['LOGNAME'], env=os.environ)
501 | self.assertIn('pid', reply2)
502 | self.assertIsInstance(reply2['pid'], int)
503 |
504 | reply = p.list_processes()
505 | self.assertEqual(2, len(list(reply.keys())))
506 | self.assertEqual(['write3.0', 'write3.1'], sorted(reply.keys()))
507 | self.assertIn('pid', list(reply.values())[0])
508 | self.assertIn('pid', list(reply.values())[1])
509 | self.assertNotEqual(list(reply.values())[0]['pid'], list(reply.values())[1]['pid'])
510 |
511 | with p.watch_processes('write3.*') as w:
512 | while True:
513 | out, err, close = w.read()
514 | if close:
515 | break
516 | reply = p.list_processes()
517 | self.assertEqual(1, len(list(reply.keys())))
518 | self.assertEqual('write3.1', list(reply.keys())[0])
519 | self.assertEqual(f, p.fileno())
520 |
521 | def test_multiple_watchers(self):
522 | with papa.Papa() as p:
523 | f = p.fileno()
524 | self.assertDictEqual({}, p.list_processes())
525 | reply1 = p.make_process('write3.0', sys.executable, args='executables/write_three_lines.py', working_dir=here, uid=os.environ['LOGNAME'], env=os.environ)
526 | self.assertIn('pid', reply1)
527 | self.assertIsInstance(reply1['pid'], int)
528 |
529 | reply2 = p.make_process('write3.1', sys.executable, args='executables/write_three_lines.py', working_dir=here, uid=os.environ['LOGNAME'], env=os.environ)
530 | self.assertIn('pid', reply2)
531 | self.assertIsInstance(reply2['pid'], int)
532 |
533 | reply = p.list_processes()
534 | self.assertEqual(2, len(list(reply.keys())))
535 | self.assertEqual(['write3.0', 'write3.1'], sorted(reply.keys()))
536 | self.assertIn('pid', list(reply.values())[0])
537 | self.assertIn('pid', list(reply.values())[1])
538 | self.assertNotEqual(list(reply.values())[0]['pid'], list(reply.values())[1]['pid'])
539 |
540 | w1 = p.watch_processes('write3.0')
541 | self.assertEqual(f, w1.fileno())
542 |
543 | w2 = p.watch_processes('write3.1')
544 | self.assertNotEqual(f, w2.fileno())
545 |
546 | p.set('p1', 't1')
547 | self.assertNotEqual(f, p.fileno())
548 | self.assertNotEqual(p.fileno(), w1.fileno())
549 | self.assertNotEqual(p.fileno(), w2.fileno())
550 |
551 | out1, err1, close1 = self.gather_output(w1)
552 | out2, err2, close2 = self.gather_output(w2)
553 |
554 | w1.close()
555 | w2.close()
556 |
557 | self.assertEqual(3, len(out1))
558 | self.assertEqual(1, len(err1))
559 | self.assertEqual(1, len(close1))
560 | self.assertEqual('write3.0', out1[0].name)
561 | self.assertEqual('write3.0', out1[1].name)
562 | self.assertEqual('write3.0', out1[2].name)
563 | self.assertEqual('write3.0', err1[0].name)
564 | self.assertEqual('write3.0', close1[0].name)
565 | self.assertLess(out1[0].timestamp, out1[1].timestamp)
566 | self.assertLess(out1[1].timestamp, out1[2].timestamp)
567 | self.assertLessEqual(out1[2].timestamp, err1[0].timestamp)
568 | self.assertLessEqual(out1[2].timestamp, close1[0].timestamp)
569 | self.assertLessEqual(err1[0].timestamp, close1[0].timestamp)
570 | self.assertEqual(b'Version: ' + cast_bytes(sys.version.partition(' ')[0]) + b'\n', out1[0].data)
571 | self.assertEqual(b'Executable: ' + cast_bytes(sys.executable) + b'\n', out1[1].data)
572 | self.assertEqual(b'Args: \n', out1[2].data)
573 | self.assertEqual(b'done', err1[0].data)
574 | self.assertEqual(0, close1[0].data)
575 |
576 | self.assertEqual(3, len(out2))
577 | self.assertEqual(1, len(err2))
578 | self.assertEqual(1, len(close2))
579 | self.assertEqual('write3.1', out2[0].name)
580 | self.assertEqual('write3.1', out2[1].name)
581 | self.assertEqual('write3.1', out2[2].name)
582 | self.assertEqual('write3.1', err2[0].name)
583 | self.assertEqual('write3.1', close2[0].name)
584 | self.assertLess(out2[0].timestamp, out2[1].timestamp)
585 | self.assertLess(out2[1].timestamp, out2[2].timestamp)
586 | self.assertLessEqual(out2[2].timestamp, err2[0].timestamp)
587 | self.assertLessEqual(out2[2].timestamp, close2[0].timestamp)
588 | self.assertLessEqual(err2[0].timestamp, close2[0].timestamp)
589 | self.assertEqual(b'Version: ' + cast_bytes(sys.version.partition(' ')[0]) + b'\n', out2[0].data)
590 | self.assertEqual(b'Executable: ' + cast_bytes(sys.executable) + b'\n', out2[1].data)
591 | self.assertEqual(b'Args: \n', out2[2].data)
592 | self.assertEqual(b'done', err2[0].data)
593 | self.assertEqual(0, close2[0].data)
594 | self.assertDictEqual({}, p.list_processes())
595 |
596 | self.assertEqual('t1', p.get('p1'))
597 |
598 | def test_process_with_small_buffer(self):
599 | with papa.Papa() as p:
600 | self.assertDictEqual({}, p.list_processes())
601 | reply1 = p.make_process('write3', sys.executable, args='executables/write_three_lines.py', working_dir=here, uid=os.environ['LOGNAME'], env=os.environ, bufsize=14)
602 | self.assertIn('pid', reply1)
603 | self.assertIsInstance(reply1['pid'], int)
604 | reply = p.list_processes()
605 | self.assertEqual(1, len(list(reply.keys())))
606 | self.assertEqual('write3', list(reply.keys())[0])
607 | self.assertIn('pid', list(reply.values())[0])
608 | sleep(.7)
609 |
610 | with p.watch_processes('write*') as w:
611 | select.select([w], [], [])
612 | self.assertTrue(w.ready)
613 | out, err, close = self.gather_output(w)
614 | exit_code = w.exit_code['write3']
615 | self.assertEqual(1, len(out))
616 | self.assertEqual(1, len(err))
617 | self.assertEqual(1, len(close))
618 | self.assertEqual('write3', out[0].name)
619 | self.assertEqual('write3', err[0].name)
620 | self.assertEqual('write3', close[0].name)
621 | self.assertLessEqual(out[0].timestamp, err[0].timestamp)
622 | self.assertLessEqual(out[0].timestamp, close[0].timestamp)
623 | self.assertLessEqual(err[0].timestamp, close[0].timestamp)
624 | self.assertEqual(b'Args: \n', out[0].data)
625 | self.assertEqual(b'done', err[0].data)
626 | self.assertEqual(0, close[0].data)
627 | self.assertEqual(0, exit_code)
628 | self.assertDictEqual({}, p.list_processes())
629 |
630 | def test_one_process_two_parallel_watchers(self):
631 | with papa.Papa() as p:
632 | self.assertDictEqual({}, p.list_processes())
633 | reply = p.make_process('write3', sys.executable, args='executables/write_three_lines.py', working_dir=here, uid=os.environ['LOGNAME'], env=os.environ)
634 | self.assertIn('pid', reply)
635 | self.assertIn('running', reply)
636 | self.assertIsInstance(reply['pid'], int)
637 | self.assertIsInstance(reply['running'], bool)
638 |
639 | w1 = p.watch_processes('write*')
640 | w2 = p.watch_processes('write*')
641 | out1, err1, close1 = w1.read()
642 | if isdebugging and not out1 and err1 and err1[0].data.startswith('pydev debugger'):
643 | out1, err1, close1 = w1.read()
644 | out2, err2, close2 = w2.read()
645 | self.assertEqual(out1[0], out2[0])
646 | w1.close()
647 | w2.close()
648 |
649 | def test_one_process_two_serial_watchers(self):
650 | with papa.Papa() as p:
651 | self.assertDictEqual({}, p.list_processes())
652 | reply = p.make_process('write3', sys.executable, args='executables/write_three_lines.py', working_dir=here, uid=os.environ['LOGNAME'], env=os.environ)
653 | self.assertIn('pid', reply)
654 | self.assertIn('running', reply)
655 | self.assertIsInstance(reply['pid'], int)
656 | self.assertIsInstance(reply['running'], bool)
657 |
658 | with p.watch_processes('write*') as w:
659 | out1, err1, close1 = w.read()
660 | if isdebugging and not out1 and err1 and err1[0].data.startswith('pydev debugger'):
661 | out1, err1, close1 = w.read()
662 | with p.watch_processes('write*') as w:
663 | out2, err2, close2 = self.gather_output(w)
664 | self.assertLess(out1[0].timestamp, out2[0].timestamp)
665 |
666 | def test_echo_server_with_normal_socket(self):
667 | with papa.Papa() as p:
668 | reply = p.make_socket('echo_socket')
669 | self.assertIn('port', reply)
670 | self.assertIn('fileno', reply)
671 | port = reply['port']
672 |
673 | reply = p.make_process('echo1', sys.executable, args=('executables/echo_server.py', '$(socket.echo_socket.fileno)'), working_dir=here)
674 | self.assertIn('pid', reply)
675 |
676 | s = socket.socket()
677 | s.connect(('127.0.0.1', port))
678 |
679 | s.send(b'test\n')
680 | msg = b''
681 | while len(msg) < 5:
682 | msg += s.recv(5)
683 | self.assertEqual(b'test\n', msg)
684 |
685 | s.send(b'and do some more\n')
686 | msg = b''
687 | while len(msg) < 17:
688 | msg += s.recv(17)
689 | self.assertEqual(b'and do some more\n', msg)
690 |
691 | s.close()
692 | with p.watch_processes('echo*') as w:
693 | out, err, close = self.gather_output(w)
694 | self.assertEqual(b'test\nand do some more\n', out[0].data)
695 |
696 | def test_echo_server_with_echo_client(self):
697 | with papa.Papa() as p:
698 | reply = p.make_socket('echo_socket')
699 | self.assertIn('port', reply)
700 | self.assertIn('fileno', reply)
701 |
702 | reply = p.make_process('echo.server', sys.executable, args=('executables/echo_server.py', '$(socket.echo_socket.fileno)'), working_dir=here)
703 | self.assertIn('pid', reply)
704 |
705 | reply = p.make_process('echo.client', sys.executable, args=('executables/echo_client.py', '$(socket.echo_socket.port)'), working_dir=here)
706 | self.assertIn('pid', reply)
707 |
708 | with p.watch_processes('echo.*') as w:
709 | out, err, close = self.gather_output(w)
710 | self.assertEqual(2, len(out))
711 | self.assertEqual(2, len(close))
712 | self.assertEqual(b'howdy\n', out[0].data)
713 | self.assertEqual(b'howdy\n', out[1].data)
714 | self.assertIn(out[0].name, ('echo.client', 'echo.server'))
715 | self.assertIn(out[1].name, ('echo.client', 'echo.server'))
716 | self.assertNotEqual(out[0].name, out[1].name)
717 |
718 | def test_echo_server_with_reuseport(self):
719 | with papa.Papa() as p:
720 | reply = p.make_socket('echo_socket', reuseport=True)
721 | self.assertIn('port', reply)
722 | port = reply['port']
723 |
724 | reply = p.make_process('echo1', sys.executable, args=('executables/echo_server.py', '$(socket.echo_socket.fileno)'), working_dir=here)
725 | self.assertIn('pid', reply)
726 |
727 | s = socket.socket()
728 | s.connect(('127.0.0.1', port))
729 |
730 | s.send(b'test\n')
731 | msg = b''
732 | while len(msg) < 5:
733 | msg += s.recv(5)
734 | self.assertEqual(b'test\n', msg)
735 |
736 | s.send(b'and do some more\n')
737 | msg = b''
738 | while len(msg) < 17:
739 | msg += s.recv(17)
740 | self.assertEqual(b'and do some more\n', msg)
741 |
742 | s.close()
743 | with p.watch_processes('echo*') as w:
744 | out, err, close = self.gather_output(w)
745 | self.assertEqual(b'test\nand do some more\n', out[0].data)
746 |
747 | def test_process_with_close_output_late(self):
748 | with papa.Papa() as p:
749 | self.assertDictEqual({}, p.list_processes())
750 | socket_reply = p.make_socket('echo_socket')
751 | p.make_process('echo', sys.executable, args=('executables/echo_server.py', '$(socket.echo_socket.fileno)'), working_dir=here)
752 |
753 | s = socket.socket()
754 | s.connect(('127.0.0.1', socket_reply['port']))
755 |
756 | s.send(b'test\n')
757 | msg = b''
758 | while len(msg) < 5:
759 | msg += s.recv(5)
760 | self.assertEqual(b'test\n', msg)
761 | s.close()
762 |
763 | sleep(.4)
764 | p.remove_processes('echo')
765 | self.assertDictEqual({}, p.list_processes())
766 |
767 | def test_process_with_close_output_early(self):
768 | with papa.Papa() as p:
769 | self.assertDictEqual({}, p.list_processes())
770 | socket_reply = p.make_socket('echo_socket')
771 | p.make_process('echo', sys.executable, args=('executables/echo_server.py', '$(socket.echo_socket.fileno)'), working_dir=here)
772 | p.remove_processes('echo')
773 |
774 | s = socket.socket()
775 | s.connect(('127.0.0.1', socket_reply['port']))
776 |
777 | s.send(b'test\n')
778 | msg = b''
779 | while len(msg) < 5:
780 | msg += s.recv(5)
781 | self.assertEqual(b'test\n', msg)
782 | s.close()
783 |
784 | sleep(.4)
785 | self.assertDictEqual({}, p.list_processes())
786 |
787 | def test_bad_socket_reference(self):
788 | with papa.Papa() as p:
789 | self.assertRaises(papa.Error, p.make_process, 'bad', sys.executable, args=('executables/echo_server.py', '$(socket.echo_socket.fileno)'), working_dir=here)
790 |
791 | def test_bad_process(self):
792 | with papa.Papa() as p:
793 | self.assertRaises(papa.Error, p.make_process, 'bad', sys.executable + '-blah')
794 |
795 | def test_bad_working_dir(self):
796 | with papa.Papa() as p:
797 | self.assertRaises(papa.Error, p.make_process, 'bad', sys.executable, working_dir=here + '-blah')
798 |
799 | if __name__ == '__main__':
800 | papa.set_default_port(randint(20000, 21000))
801 | unittest.main()
802 |
--------------------------------------------------------------------------------
/tests/executables/echo_client.py:
--------------------------------------------------------------------------------
1 | import sys
2 | import socket
3 | from papa.utils import cast_string, send_with_retry, recv_with_retry
4 |
5 | __author__ = 'Scott Maxwell'
6 |
7 | if len(sys.argv) != 2:
8 | sys.stderr.write('Need one port number\n')
9 | sys.exit(1)
10 |
11 | sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
12 | sock.connect(('127.0.0.1', int(sys.argv[1])))
13 |
14 | send_with_retry(sock, b'howdy\n')
15 | data = recv_with_retry(sock)
16 | sys.stdout.write(cast_string(data))
17 |
18 | sock.close()
19 |
--------------------------------------------------------------------------------
/tests/executables/echo_server.py:
--------------------------------------------------------------------------------
1 | import sys
2 | import socket
3 | from papa.utils import cast_string, send_with_retry, recv_with_retry
4 |
5 | __author__ = 'Scott Maxwell'
6 |
7 | if len(sys.argv) != 2:
8 | sys.stderr.write('Need one file descriptor\n')
9 | sys.exit(1)
10 |
11 | listen_socket = socket.fromfd(int(sys.argv[1]), socket.AF_INET, socket.SOCK_STREAM)
12 | sock, address = listen_socket.accept()
13 |
14 | while True:
15 | data = recv_with_retry(sock)
16 | if data:
17 | send_with_retry(sock, data)
18 | sys.stdout.write(cast_string(data))
19 | else:
20 | break
21 |
22 | sock.close()
23 |
--------------------------------------------------------------------------------
/tests/executables/write_three_lines.py:
--------------------------------------------------------------------------------
1 | import sys
2 | from time import sleep
3 |
4 | __author__ = 'Scott Maxwell'
5 |
6 | print('Version: %s' % sys.version.partition(' ')[0])
7 | sys.stdout.flush()
8 | sleep(.1)
9 |
10 | print('Executable: %s' % sys.executable)
11 | sys.stdout.flush()
12 | sleep(.1)
13 |
14 | print('Args: %s' % ' '.join(sys.argv[1:]))
15 | sys.stdout.flush()
16 |
17 | sys.stderr.write('done')
18 |
--------------------------------------------------------------------------------