├── .gitignore ├── LICENSE ├── README.md ├── docs ├── FAME_background.md ├── VMconfig.md ├── desktop │ ├── ExecCard.desktop │ ├── WS Cycle.desktop │ ├── cleanup.desktop │ ├── client.desktop │ ├── node01.desktop │ ├── node02.desktop │ ├── server.desktop │ ├── virt-manager.desktop │ └── wscycle ├── images │ ├── Element.jpg │ ├── FEE block.png │ ├── Gen-Z on black.png │ ├── Gen-Z on white.jpg │ ├── HPE logo.png │ ├── IVSHMEM block.png │ ├── IVSHMSG block.png │ ├── M-logo green.jpg │ ├── M-logo white.jpg │ └── StarTrek.jpg ├── ivshmem-spec.txt └── setup-scripts │ ├── README.md │ ├── run-host-genz-emul-env.sh │ ├── setup-guest-genz-emul-env.sh │ └── setup-host-genz-emul-env.sh ├── ivshmsg_client.py ├── ivshmsg_server.py └── ivshmsg_twisted ├── commander.py ├── famez_requests.py ├── ivshmsg_eventfd.py ├── ivshmsg_mailbox.py ├── ivshmsg_sendrecv.py ├── twisted_client.py ├── twisted_restapi.py └── twisted_server.py /.gitignore: -------------------------------------------------------------------------------- 1 | *.exe 2 | *.pyc 3 | *.swp 4 | 5 | *.mk 6 | *.o.* 7 | 8 | junk* 9 | nohup.out 10 | 11 | __pycache__ 12 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | GNU GENERAL PUBLIC LICENSE 2 | Version 2, June 1991 3 | 4 | Copyright (C) 1989, 1991 Free Software Foundation, Inc., 5 | 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA 6 | Everyone is permitted to copy and distribute verbatim copies 7 | of this license document, but changing it is not allowed. 8 | 9 | Preamble 10 | 11 | The licenses for most software are designed to take away your 12 | freedom to share and change it. By contrast, the GNU General Public 13 | License is intended to guarantee your freedom to share and change free 14 | software--to make sure the software is free for all its users. This 15 | General Public License applies to most of the Free Software 16 | Foundation's software and to any other program whose authors commit to 17 | using it. (Some other Free Software Foundation software is covered by 18 | the GNU Lesser General Public License instead.) You can apply it to 19 | your programs, too. 20 | 21 | When we speak of free software, we are referring to freedom, not 22 | price. Our General Public Licenses are designed to make sure that you 23 | have the freedom to distribute copies of free software (and charge for 24 | this service if you wish), that you receive source code or can get it 25 | if you want it, that you can change the software or use pieces of it 26 | in new free programs; and that you know you can do these things. 27 | 28 | To protect your rights, we need to make restrictions that forbid 29 | anyone to deny you these rights or to ask you to surrender the rights. 30 | These restrictions translate to certain responsibilities for you if you 31 | distribute copies of the software, or if you modify it. 32 | 33 | For example, if you distribute copies of such a program, whether 34 | gratis or for a fee, you must give the recipients all the rights that 35 | you have. You must make sure that they, too, receive or can get the 36 | source code. And you must show them these terms so they know their 37 | rights. 38 | 39 | We protect your rights with two steps: (1) copyright the software, and 40 | (2) offer you this license which gives you legal permission to copy, 41 | distribute and/or modify the software. 42 | 43 | Also, for each author's protection and ours, we want to make certain 44 | that everyone understands that there is no warranty for this free 45 | software. If the software is modified by someone else and passed on, we 46 | want its recipients to know that what they have is not the original, so 47 | that any problems introduced by others will not reflect on the original 48 | authors' reputations. 49 | 50 | Finally, any free program is threatened constantly by software 51 | patents. We wish to avoid the danger that redistributors of a free 52 | program will individually obtain patent licenses, in effect making the 53 | program proprietary. To prevent this, we have made it clear that any 54 | patent must be licensed for everyone's free use or not licensed at all. 55 | 56 | The precise terms and conditions for copying, distribution and 57 | modification follow. 58 | 59 | GNU GENERAL PUBLIC LICENSE 60 | TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 61 | 62 | 0. This License applies to any program or other work which contains 63 | a notice placed by the copyright holder saying it may be distributed 64 | under the terms of this General Public License. The "Program", below, 65 | refers to any such program or work, and a "work based on the Program" 66 | means either the Program or any derivative work under copyright law: 67 | that is to say, a work containing the Program or a portion of it, 68 | either verbatim or with modifications and/or translated into another 69 | language. (Hereinafter, translation is included without limitation in 70 | the term "modification".) Each licensee is addressed as "you". 71 | 72 | Activities other than copying, distribution and modification are not 73 | covered by this License; they are outside its scope. The act of 74 | running the Program is not restricted, and the output from the Program 75 | is covered only if its contents constitute a work based on the 76 | Program (independent of having been made by running the Program). 77 | Whether that is true depends on what the Program does. 78 | 79 | 1. You may copy and distribute verbatim copies of the Program's 80 | source code as you receive it, in any medium, provided that you 81 | conspicuously and appropriately publish on each copy an appropriate 82 | copyright notice and disclaimer of warranty; keep intact all the 83 | notices that refer to this License and to the absence of any warranty; 84 | and give any other recipients of the Program a copy of this License 85 | along with the Program. 86 | 87 | You may charge a fee for the physical act of transferring a copy, and 88 | you may at your option offer warranty protection in exchange for a fee. 89 | 90 | 2. You may modify your copy or copies of the Program or any portion 91 | of it, thus forming a work based on the Program, and copy and 92 | distribute such modifications or work under the terms of Section 1 93 | above, provided that you also meet all of these conditions: 94 | 95 | a) You must cause the modified files to carry prominent notices 96 | stating that you changed the files and the date of any change. 97 | 98 | b) You must cause any work that you distribute or publish, that in 99 | whole or in part contains or is derived from the Program or any 100 | part thereof, to be licensed as a whole at no charge to all third 101 | parties under the terms of this License. 102 | 103 | c) If the modified program normally reads commands interactively 104 | when run, you must cause it, when started running for such 105 | interactive use in the most ordinary way, to print or display an 106 | announcement including an appropriate copyright notice and a 107 | notice that there is no warranty (or else, saying that you provide 108 | a warranty) and that users may redistribute the program under 109 | these conditions, and telling the user how to view a copy of this 110 | License. (Exception: if the Program itself is interactive but 111 | does not normally print such an announcement, your work based on 112 | the Program is not required to print an announcement.) 113 | 114 | These requirements apply to the modified work as a whole. If 115 | identifiable sections of that work are not derived from the Program, 116 | and can be reasonably considered independent and separate works in 117 | themselves, then this License, and its terms, do not apply to those 118 | sections when you distribute them as separate works. But when you 119 | distribute the same sections as part of a whole which is a work based 120 | on the Program, the distribution of the whole must be on the terms of 121 | this License, whose permissions for other licensees extend to the 122 | entire whole, and thus to each and every part regardless of who wrote it. 123 | 124 | Thus, it is not the intent of this section to claim rights or contest 125 | your rights to work written entirely by you; rather, the intent is to 126 | exercise the right to control the distribution of derivative or 127 | collective works based on the Program. 128 | 129 | In addition, mere aggregation of another work not based on the Program 130 | with the Program (or with a work based on the Program) on a volume of 131 | a storage or distribution medium does not bring the other work under 132 | the scope of this License. 133 | 134 | 3. You may copy and distribute the Program (or a work based on it, 135 | under Section 2) in object code or executable form under the terms of 136 | Sections 1 and 2 above provided that you also do one of the following: 137 | 138 | a) Accompany it with the complete corresponding machine-readable 139 | source code, which must be distributed under the terms of Sections 140 | 1 and 2 above on a medium customarily used for software interchange; or, 141 | 142 | b) Accompany it with a written offer, valid for at least three 143 | years, to give any third party, for a charge no more than your 144 | cost of physically performing source distribution, a complete 145 | machine-readable copy of the corresponding source code, to be 146 | distributed under the terms of Sections 1 and 2 above on a medium 147 | customarily used for software interchange; or, 148 | 149 | c) Accompany it with the information you received as to the offer 150 | to distribute corresponding source code. (This alternative is 151 | allowed only for noncommercial distribution and only if you 152 | received the program in object code or executable form with such 153 | an offer, in accord with Subsection b above.) 154 | 155 | The source code for a work means the preferred form of the work for 156 | making modifications to it. For an executable work, complete source 157 | code means all the source code for all modules it contains, plus any 158 | associated interface definition files, plus the scripts used to 159 | control compilation and installation of the executable. However, as a 160 | special exception, the source code distributed need not include 161 | anything that is normally distributed (in either source or binary 162 | form) with the major components (compiler, kernel, and so on) of the 163 | operating system on which the executable runs, unless that component 164 | itself accompanies the executable. 165 | 166 | If distribution of executable or object code is made by offering 167 | access to copy from a designated place, then offering equivalent 168 | access to copy the source code from the same place counts as 169 | distribution of the source code, even though third parties are not 170 | compelled to copy the source along with the object code. 171 | 172 | 4. You may not copy, modify, sublicense, or distribute the Program 173 | except as expressly provided under this License. Any attempt 174 | otherwise to copy, modify, sublicense or distribute the Program is 175 | void, and will automatically terminate your rights under this License. 176 | However, parties who have received copies, or rights, from you under 177 | this License will not have their licenses terminated so long as such 178 | parties remain in full compliance. 179 | 180 | 5. You are not required to accept this License, since you have not 181 | signed it. However, nothing else grants you permission to modify or 182 | distribute the Program or its derivative works. These actions are 183 | prohibited by law if you do not accept this License. Therefore, by 184 | modifying or distributing the Program (or any work based on the 185 | Program), you indicate your acceptance of this License to do so, and 186 | all its terms and conditions for copying, distributing or modifying 187 | the Program or works based on it. 188 | 189 | 6. Each time you redistribute the Program (or any work based on the 190 | Program), the recipient automatically receives a license from the 191 | original licensor to copy, distribute or modify the Program subject to 192 | these terms and conditions. You may not impose any further 193 | restrictions on the recipients' exercise of the rights granted herein. 194 | You are not responsible for enforcing compliance by third parties to 195 | this License. 196 | 197 | 7. If, as a consequence of a court judgment or allegation of patent 198 | infringement or for any other reason (not limited to patent issues), 199 | conditions are imposed on you (whether by court order, agreement or 200 | otherwise) that contradict the conditions of this License, they do not 201 | excuse you from the conditions of this License. If you cannot 202 | distribute so as to satisfy simultaneously your obligations under this 203 | License and any other pertinent obligations, then as a consequence you 204 | may not distribute the Program at all. For example, if a patent 205 | license would not permit royalty-free redistribution of the Program by 206 | all those who receive copies directly or indirectly through you, then 207 | the only way you could satisfy both it and this License would be to 208 | refrain entirely from distribution of the Program. 209 | 210 | If any portion of this section is held invalid or unenforceable under 211 | any particular circumstance, the balance of the section is intended to 212 | apply and the section as a whole is intended to apply in other 213 | circumstances. 214 | 215 | It is not the purpose of this section to induce you to infringe any 216 | patents or other property right claims or to contest validity of any 217 | such claims; this section has the sole purpose of protecting the 218 | integrity of the free software distribution system, which is 219 | implemented by public license practices. Many people have made 220 | generous contributions to the wide range of software distributed 221 | through that system in reliance on consistent application of that 222 | system; it is up to the author/donor to decide if he or she is willing 223 | to distribute software through any other system and a licensee cannot 224 | impose that choice. 225 | 226 | This section is intended to make thoroughly clear what is believed to 227 | be a consequence of the rest of this License. 228 | 229 | 8. If the distribution and/or use of the Program is restricted in 230 | certain countries either by patents or by copyrighted interfaces, the 231 | original copyright holder who places the Program under this License 232 | may add an explicit geographical distribution limitation excluding 233 | those countries, so that distribution is permitted only in or among 234 | countries not thus excluded. In such case, this License incorporates 235 | the limitation as if written in the body of this License. 236 | 237 | 9. The Free Software Foundation may publish revised and/or new versions 238 | of the General Public License from time to time. Such new versions will 239 | be similar in spirit to the present version, but may differ in detail to 240 | address new problems or concerns. 241 | 242 | Each version is given a distinguishing version number. If the Program 243 | specifies a version number of this License which applies to it and "any 244 | later version", you have the option of following the terms and conditions 245 | either of that version or of any later version published by the Free 246 | Software Foundation. If the Program does not specify a version number of 247 | this License, you may choose any version ever published by the Free Software 248 | Foundation. 249 | 250 | 10. If you wish to incorporate parts of the Program into other free 251 | programs whose distribution conditions are different, write to the author 252 | to ask for permission. For software which is copyrighted by the Free 253 | Software Foundation, write to the Free Software Foundation; we sometimes 254 | make exceptions for this. Our decision will be guided by the two goals 255 | of preserving the free status of all derivatives of our free software and 256 | of promoting the sharing and reuse of software generally. 257 | 258 | NO WARRANTY 259 | 260 | 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY 261 | FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN 262 | OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES 263 | PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED 264 | OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF 265 | MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS 266 | TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE 267 | PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, 268 | REPAIR OR CORRECTION. 269 | 270 | 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING 271 | WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR 272 | REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, 273 | INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING 274 | OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED 275 | TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY 276 | YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER 277 | PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE 278 | POSSIBILITY OF SUCH DAMAGES. 279 | 280 | END OF TERMS AND CONDITIONS 281 | 282 | How to Apply These Terms to Your New Programs 283 | 284 | If you develop a new program, and you want it to be of the greatest 285 | possible use to the public, the best way to achieve this is to make it 286 | free software which everyone can redistribute and change under these terms. 287 | 288 | To do so, attach the following notices to the program. It is safest 289 | to attach them to the start of each source file to most effectively 290 | convey the exclusion of warranty; and each file should have at least 291 | the "copyright" line and a pointer to where the full notice is found. 292 | 293 | 294 | Copyright (C) 295 | 296 | This program is free software; you can redistribute it and/or modify 297 | it under the terms of the GNU General Public License as published by 298 | the Free Software Foundation; either version 2 of the License, or 299 | (at your option) any later version. 300 | 301 | This program is distributed in the hope that it will be useful, 302 | but WITHOUT ANY WARRANTY; without even the implied warranty of 303 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 304 | GNU General Public License for more details. 305 | 306 | You should have received a copy of the GNU General Public License along 307 | with this program; if not, write to the Free Software Foundation, Inc., 308 | 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. 309 | 310 | Also add information on how to contact you by electronic and paper mail. 311 | 312 | If the program is interactive, make it output a short notice like this 313 | when it starts in an interactive mode: 314 | 315 | Gnomovision version 69, Copyright (C) year name of author 316 | Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. 317 | This is free software, and you are welcome to redistribute it 318 | under certain conditions; type `show c' for details. 319 | 320 | The hypothetical commands `show w' and `show c' should show the appropriate 321 | parts of the General Public License. Of course, the commands you use may 322 | be called something other than `show w' and `show c'; they could even be 323 | mouse-clicks or menu items--whatever suits your program. 324 | 325 | You should also get your employer (if you work as a programmer) or your 326 | school, if any, to sign a "copyright disclaimer" for the program, if 327 | necessary. Here is a sample; alter the names: 328 | 329 | Yoyodyne, Inc., hereby disclaims all copyright interest in the program 330 | `Gnomovision' (which makes passes at compilers) written by James Hacker. 331 | 332 | , 1 April 1989 333 | Ty Coon, President of Vice 334 | 335 | This General Public License does not permit incorporating your program into 336 | proprietary programs. If your program is a subroutine library, you may 337 | consider it more useful to permit linking proprietary applications with the 338 | library. If this is what you want to do, use the GNU Lesser General 339 | Public License instead of this License. 340 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | [Gen-Z is a new memory-semantic fabric](https://genzconsortium.org/) created 2 | as the glue for constructing exascale computing. It is an open specification 3 | evolved from the fabric used in 4 | [The Machine from Hewlett Packard Enterprise](https://www.hpe.com/TheMachine). 5 | Such fabrics allow "wide-area" connectivity of computing resources such as CPU, 6 | GPU, memory (legacy and persistent) and other devices via a memory-semantic 7 | programming model. 8 | 9 | The Gen-Z spec and working groups are evolving the standard and early 10 | hardware is beginning to appear. However there is not an open "platform" 11 | on which to develop system software. The success of QEMU and IVSHMEM as 12 | [an emulated development platform for The Machine](docs/FAME_background.md) 13 | suggests an extended use should be considered. 14 | 15 | ### Beyond IVSHMEM - a rudimentary fabric 16 | 17 | QEMU has another feature of interest in a multi-actor messaging environment 18 | like that of Gen-Z. By applying a slightly different stanza, the IVSHMEM 19 | virtual PCI device is enabled to send and handle interrupts in a 20 | "mailbox/doorbell" setup. An interrupt to the virtual PCI device is generated 21 | from an "event notification" issued to the QEMU process by a similarly 22 | configured peer QEMU. But how are these peers connected? 23 | 24 | The scheme starts with a separate program delivered with QEMU. [ 25 | ```/usr/bin/ivshmem-server ```]( 26 | https://github.com/qemu/qemu/blob/master/docs/specs/ivshmem-spec.txt) 27 | establishes a UNIX-domain socket and must be started before any properly 28 | configured QEMU VMs. A new QEMU process starts by connecting to the socket 29 | and receiving its own set of event channels, as well as those of all other 30 | peers. The mechanism in each guest OS is that writing a "doorbell" register 31 | will signal the QEMU into an event against another QEMU. The receiving QEMU 32 | transforms that event into a PCI interrupt for its guest OS. 33 | 34 | ```ivshmem-server``` only informs each QEMU of its other peers; it does not 35 | participate in further peer-to-peer communcation. A backing file must also 36 | be specified to ivshmem-server for use as a message mailbox. Obviously the 37 | guests/clients must agree on the use of the mailbox file. Standard 38 | ivshmem-server never touches the file contents. 39 | 40 | ![alt text][IVSHMSG] 41 | 42 | [IVSHMSG]: https://github.com/linux-genz/F.E.E./blob/master/docs/images/IVSHMSG%20block.png "Figure 1" 43 | 44 | The final use case above is QEMU guest-to-guest communication over the "IVSHMSG 45 | doorbell/mailbox fabric". OS-to-OS communication will involve a (new) guest 46 | kernel driver and other abstractions to hide the mechanics of IVSHMSG. 47 | This IVSHMSG shim can serve as the foundation for higher-level protocols. 48 | 49 | ## Gen-Z Emulation on top of IVSHMSG 50 | 51 | If the guest OS driver emulates a simple Gen-Z bridge, a great deal of 52 | "pure Gen-Z" software development can be done on this simple platform. 53 | Certain Gen-Z primitive operations for discovery and crawlout 54 | would also be abetted by intelligence "in the fabric". In fact, that 55 | intelligence could live in the ivshmem-server process if it were 56 | extended to participate in actual messaging. 57 | 58 | Modifying the existing ```ivshmem-server`` C program is not a simple challenge. 59 | Written within the QEMU build framework, it is not standalone source code. 60 | C is a also limited for higher-level data constructs anticipated for a Gen-Z 61 | emulation. Finally, it seems unlikely such changes would be accepted upstream. 62 | 63 | F.E.E. is a rewrite of ivshmem-server in Python using Twisted 64 | as the network-handling framework. ```ivshmsg_server.py``` is run in place of 65 | ```ivshmem-server```. It correctly serves real QEMU processes as well as 66 | the stock QEMU ``ivshmem-client``, a test program that comes with QEMU. 67 | 68 | ![alt text][EMERGEN-Z] 69 | 70 | [EMERGEN-Z]: https://github.com/linux-genz/F.E.E./blob/master/docs/images/FEE%20block.png "Figure 2" 71 | 72 | A new feature for the Python version is server participation 73 | in the doorbell/mailbox messaging to serve as fabric intelligence 74 | (ie, a smart switch). 75 | 76 | ___ 77 | 78 | ## Running the Python rewrites 79 | 80 | As ivshmsg_server.py was being created, it was tested with the QEMU 81 | ```/usr/bin/ivshmem-client```. As might be expected, there is now an 82 | ivshmsg_client.py rewrite. It has an expanded command set and over 83 | time its use as a monitor/debugger/injector will certainly grow. 84 | 85 | To use these programs as a simple chat framework you don't even need QEMU. 86 | 87 | ### ivshmsg_server.py 88 | 1. Clone this repo 89 | 1. Install python3 packages ```twisted``` and ```klein``` (names will vary by distro) 90 | 1. In one terminal window run './ivshmsg_server.py'. 91 | 1. Type "help" and hit return. 92 | 93 | By default it creates /tmp/ivshmsg_socket to which up to clients attach, 94 | and /dev/shm/ivshmsg_mailbox which is shared among all clients for messaging. 95 | The window expresses an interactive interface, the command set is quite simple 96 | at the moment. Try "dump". This will get you a visual representation of a 97 | 14-port switch. The port number is the raw messaging source/destination 98 | of attached clients. The soft-switch itself is ID 15; there is no ID 0. 99 | 100 | In general, ivshmsg_server.py could be used for many more messaging 101 | protocols beyond Gen-Z. However, it currently has a "personality" 102 | that interprets some of the Link level messages at a semantic level. 103 | That's best seen with a debugger client. 104 | 105 | ### ivshmsg_client.py 106 | 107 | 1. In a second (or more) terminal window(s) run 'ivshmsg_client.py'. 108 | 109 | You'll see them get added in the server window if default values were used 110 | in the server invocation. Each client is assigned a random "port". 111 | 112 | 1. In one of the clients, hit return, then type "help". Play with sending messages to the other client(s) or the server. 113 | 114 | Hit "d" or "dump" to see local information. 115 | 116 | Try "ping", as in "ping NN" or "ping " 117 | 118 | "Send" is the next message, and now we have an overly complicated chatroom. 119 | 120 | Try "Link" or "Link Peer-Attributes" 121 | 122 | Finally try "RFC", followed by "dump". It should look different :-) 123 | 124 | ## Generating and connecting VMs 125 | 126 | When you get the Python programs playing together (above), you'll be ready 127 | for another story [which is going to be told here.](docs/VMconfig.md) 128 | 129 | Once you have VMs configured and connected as "Driverless QEMU" actors, 130 | [proceed to the prototype Gen-Z subsystem repo](http://github.com/linux-genz/EmerGen-Z/). 131 | 132 | ___ 133 | 134 | ## Bugs and Performance 135 | 136 | As the QEMU docs say, "(IVSHMSG) is simple and fragile" and sometimes 137 | * A QEMU session will lose connection with the server, hang, die, or otherwise need a restart. 138 | * There's a pseudo-hardware holdoff/interlock timeout in the guest kernel drivers which can cause a VM core to go into RCU stall which usually leads to a virtual panic. 139 | * RARELY do you have to restart ivshmsg_server.py, but if you have to restart the QEMUs anyhow, it never hurts. 140 | * Debugging under Python Twisted will make you go blind. 141 | 142 | Most of these problems only show up in a stripped-down, dedicated speed run. 143 | On "average" hardware the setup can reach 20k messages/second between 144 | two VMs (100-bytes messages, or 2Mb/sec) sustained over minutes of time. 145 | 146 | Data rates are typically MUCH smaller when doing the type of programming 147 | for which F.E.E. is really intended: bridge-based inter-actor fabric 148 | management. It's been stable enough to promote generation of a 149 | prototype Gen-Z subsystem for the kernel. 150 | 151 | [Read all about that at this repo](http://github.com/linux-genz/EmerGen-Z/). 152 | 153 | ## Backlog 154 | 155 | A few things the primary author has thought about or heard... 156 | 157 | * A severe documentation review and refresh would help drive adoption. 158 | * Extract the Gen-Z "personality" into a more modular plugin basis for re-use of F.E.E. for other fabrics. 159 | * Move to shared code as the interactive command engine (or at least coalesce it more) as part of those extracted personalities. 160 | * More intelligence in the switch 161 | * Particularly around fabric management 162 | * "True bridging", right now the clients cheat and talk directly to each other 163 | * Error injection 164 | * More switches 165 | -------------------------------------------------------------------------------- /docs/FAME_background.md: -------------------------------------------------------------------------------- 1 | ## FAME Background 2 | 3 | The current Gen-Z situation is similar to the challenge faced with The Machine. Actual hardware was many months away and the specifications were still mildly in flux. However we wanted to start development on the system software required to manage the resources and that demanded a "suitable" development platform. 4 | 5 | The Machine consists of up to 40 nodes of an SoC running independent instances of Linux. All nodes share a 160 TB fabric-attached memory (FAM) global address space via the Gen-Z precursor fabric. QEMU/KVM provided the basis for a "suitable" development platform. A single node can be represented by a single VM. The QEMU feature Inter-VM Shared Memory (IVSHMEM) presents a file in the host operating system as physical address space in a VM. If all VMs use the same backing store, you get ["Fabric-Attached Memory Emulation" or FAME](https://github.com/FabricAttachedMemory/Emulation). That project also makes bootable disk images and configures a libvirt network to run a complete setup. 6 | 7 | ![alt text][IVSHMEM] 8 | 9 | [IVSHMEM]: https://github.com/linux-genz/F.E.E./blob/master/docs/images/IVSHMEM%20block.png "Figure 1" 10 | 11 | 12 | ### QEMU Configuration under FAME 13 | 14 | When QEMU is invoked with an IVSHMEM configuration, a new PCI device appears in the VM. The size/space of the file is represented as physical address space behind BAR2 of that device. To configure a VM for IVSHMEM/FAME, first allocate the file somewhere (such as /home/rocky/FAME/FAM of 32G), then start QEMU with the added stanza 15 | 16 | ``` 17 | -object memory-backend-file,mem-path=/home/rocky/FAME/FAM,size=32G,id=FAM,share=on -device ivshmem-plain,memdev=FAM 18 | ``` 19 | or add these lines to the end of a libvirt XML domain declaration for a VM: 20 | ```XML 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | ``` 29 | From such a VM configured with a 32G IVSHMEM file: 30 | ```bash 31 | rocky@node02 $ lspci -v 32 | : 33 | 00:09.0 RAM memory: Red Hat, Inc Inter-VM shared memory (rev 01) 34 | Subsystem: Red Hat, Inc QEMU Virtual Machine 35 | Flags: fast devsel 36 | Memory at fc05a000 (32-bit, non-prefetchable) [size=256] 37 | Memory at 800000000 (64-bit, prefetchable) [size=32G] 38 | Kernel modules: virtio_pci 39 | ``` 40 | The precise base address for the 32G space may vary depending on other VM settings. All of this is handled by the setup script of the [FAME project](https://github.com/FabricAttachedMemory/Emulation). Be sure and read the wiki there about [the difference between emulation and simulation](https://github.com/FabricAttachedMemory/Emulation/wiki/Emulation-and-Simulation) and [the full FAME setup](https://github.com/FabricAttachedMemory/Emulation/wiki/Emulation-via-Virtual-Machines). 41 | -------------------------------------------------------------------------------- /docs/VMconfig.md: -------------------------------------------------------------------------------- 1 | ## Create QEMU VM(s) for Linux guest OS 2 | 3 | These are offered as suggestion, not officially supported directives. YMMV 4 | so please create an issue if you have problems or a better answer. 5 | All three methods assume proper installation of QEMU, libvirtd, and other 6 | tools and utilities. 7 | 8 | ### Method 1: Use your favorite/current method 9 | 10 | If you have a preferred method, use it, then alter the invocation of the 11 | virtual machines to invoke IVSHMSG. Either add the following stanza to 12 | a qemu command line: 13 | 14 | ``` 15 | -chardev socket,id=IVSHMSG,path=/tmp/ivshmsg_socket -device ivshmem-doorbell,chardev=IVSHMSG,vectors=16 16 | ``` 17 | or add this to a libvirt domain XML file, at the end: 18 | ``` 19 | 20 | 21 | 22 | 23 | 24 | 25 | ``` 26 | 27 | The security configuration of some distros will not allow QEMU to open a 28 | socket in /tmp. "path" may need to be changed. You may also need to extend the very first line of the XML to specify a namespace needed to accept the command-line directives: 29 | 30 | ``` 31 | 32 | ``` 33 | 34 | ### Method 2: qemu-img and virt-install 35 | 36 | Two references were selected and mixed/matched to get a working invocation: 37 | 38 | https://raymii.org/s/articles/virt-install_introduction_and_copy_paste_distro_install_commands.html#virt-install 39 | 40 | https://docs.openstack.org/image-guide/virt-install.html 41 | 42 | A different graphics option was chosen to allow headless invocation. The 43 | virtual console can be opened with "virsh console <VMNAME>". The 44 | following stanzas were replaced: 45 | 46 | --graphics none --console pty,target_type=serial 47 | --extra-args 'console=ttyS0,115200n8 serial' 48 | 49 | 1. qemu-img create -f qcow2 ./TARGET.qcow2 4G 50 | 1. If you want to use a different virtual network than "default", create and activate it now. Then specify it in the virt-install command. 51 | 1. virt-install --name <VMNAME> --virt-type kvm --vcpus 1 --ram 1024 \ 52 | --disk path=./TARGET.qcow2 \ 53 | --network network=default \ 54 | --graphics spice --video qxl --channel spicevmc \ 55 | --os-type linux --os-variant auto \ 56 | --location <DISTRO-DEPENDENT> 57 | 58 | Debian: --location http://ftp.us.debian.org/debian/dists/testing/main/installer-amd64/ 59 | 60 | "testing" is used to get a kernel recent enough for the EmerGen-Z build (must be 4.13 or later, "stretch" is just 4.9). 61 | 62 | Ubuntu Bionic: --location http://us.archive.ubuntu.com/ubuntu/dists/bionic/main/installer-amd64/ 63 | 64 | CentOS: --location http://www.gtlib.gatech.edu/pub/centos/7/os/x86_64/ 65 | 66 | 1. Alter the VM invocation as explained in "Method 1" 67 | 68 | ### Method 3: FAME 69 | 70 | If your host OS is a Debian distro (or derivative like Ubuntu) you can use 71 | the [emulated development platform for The Machine](FAME_background.md). 72 | It will do all the necessary things: 73 | 74 | * Set up a dedicted virtual network (no need to use default) 75 | * Build and customize multiple VMs 76 | * Use correct startup for IVSHMSG 77 | * Optionally use IVSHMEM as Fabric Attached Memory (FAM) 78 | 79 | First export these variables. Yes, FAME_FAM should be empty because it's 80 | not needed for the fabric emulation of this project. 81 | 82 | ``` 83 | FAME_DIR=/some/where/useful # Under $HOME is fine 84 | FAME_FAM= 85 | FAME_IVSHMSG=yes 86 | FAME_HOSTBASE=genz # Optional, default is "node" 87 | FAME_USER=genz # Optional, default is "l4mdc" 88 | ``` 89 | 90 | Then you can run the `emulation_configure.bash` script per the documentation 91 | on that project. Note that FAME_IVSHMSG is ***not*** documented there. 92 | If you use FAM, it will appear as another virtual PCI device in the 93 | guest OS. The Librarian can find it from there. 94 | 95 | Review the $FAME_DIR/${FAME_HOSTBASE}_env.sh to see the pathname chosen 96 | for the Unix domain socket. 97 | 98 | ## Useful packages to install 99 | Most (all?) of these packages get installed by FAME. This list is for 100 | those making VMs by another method. 101 | 102 | * build-essential 103 | * git 104 | * haveged (eliminates the sshd startup delay from insufficient entropy) 105 | * linux-headers-4.xx 106 | * sudo 107 | 108 | ## Running Linux guests with IVSHMSG kernel modules 109 | 110 | While a QEMU process makes the network connection to ivshmsg_server.py, it's 111 | the guest OS inside QEMU where the messaging endpoints take place. 112 | 113 | 1. Start ivshmsg_server.py. If you used FAME to create your VMs, use the 114 | --socketpath value that matches FAME's assignment. 115 | 1. Start the VM(s). 116 | 1. Log in to the VM. Look for the IVSHMSG pseudo-device with *sudo lspci -v*: 117 | 1. In the VM, git clone two repos into one directory (so that it contains 118 | two directories when finished): 119 | 1. cd && mkdir github && cd github 120 | 1. git clone https://github.com/linux-genz/EmerGen-Z.git 121 | 1. git clone https://github.com/?????/executivecardboard.git 122 | 1. cd EmerGen-Z/subsystem 123 | 1. *make modules_install* which should create and install one kernel module, 124 | `genz.ko` 125 | 1. cd ../shim_bridge 126 | 1. *make modules_install* which should create and install two kernel modules, 127 | `ivshmsg.ko` and `emergenz_bridge.ko` 128 | 1. *sudo modprobe ivshmsg.ko verbose=2* dmesg output should indicate the driver 129 | found and attached to the IVSHMSG pseudo-device. 130 | 1. *sudo insmod emergenz_bridge.ko verbose=2* dmesg output should show the 131 | driver bound to the ivshmsg driver. There should also be a new device file 132 | /dev/genz_bridge_xx where xx matches the PCI pseudo-device address in lspci. 133 | 134 | ## Quick messaging tests 135 | 136 | 1. In another host window, run *ivshmsg_client.py --socketpath ....*. Note its 137 | IVSHMSG ID in the server monitor window, or execute "dump" in the client. 138 | 1. On the VM, echo "C:hello there" > /dev/genz_bridge_xx, where "C" is the 139 | IVHSHMSG client number of the client. 140 | 141 | -------------------------------------------------------------------------------- /docs/desktop/ExecCard.desktop: -------------------------------------------------------------------------------- 1 | [Desktop Entry] 2 | Version=1.0 3 | Type=Application 4 | Comment=Point Chrome at the monitor GUI 5 | Exec=chromium-browser --no-proxy-server http://127.0.0.1:8080/ 6 | Icon=display-im6.q16 7 | Path= 8 | Terminal=false 9 | StartupNotify=false 10 | Name=2. Executive Cardboard 11 | -------------------------------------------------------------------------------- /docs/desktop/WS Cycle.desktop: -------------------------------------------------------------------------------- 1 | [Desktop Entry] 2 | Version=1.0 3 | Type=Application 4 | Name=WS Cycle 5 | Comment=Unattended station 6 | Exec=/bin/bash -c "$HOME/Documents/wscycle" 7 | Icon= 8 | Path= 9 | Terminal=false 10 | StartupNotify=false 11 | -------------------------------------------------------------------------------- /docs/desktop/cleanup.desktop: -------------------------------------------------------------------------------- 1 | [Desktop Entry] 2 | Version=1.0 3 | Type=Application 4 | Comment=Minor cleanup, stop everything first 5 | Exec=/bin/bash -c "killall ivshmsg_client.py; killall ivshmsg_server.py; rm -f /$HOME/FAME/node_socket" 6 | Icon=user-trash 7 | Terminal=true 8 | StartupNotify=false 9 | Name=99. Cleanup 10 | Path= 11 | -------------------------------------------------------------------------------- /docs/desktop/client.desktop: -------------------------------------------------------------------------------- 1 | [Desktop Entry] 2 | Version=1.0 3 | Type=Application 4 | Comment= 5 | Exec=xfce4-terminal --geometry=90x18-1-1 -e 'bash -c "$HOME/github/F.E.E./ivshmsg_client.py --socket $HOME/FAME/node_socket || sleep 5"' 6 | Icon=utilities-system-monitor 7 | Terminal=true 8 | StartupNotify=false 9 | Name=3. Gen-Z Fabric Debugger 10 | Path= 11 | -------------------------------------------------------------------------------- /docs/desktop/node01.desktop: -------------------------------------------------------------------------------- 1 | [Desktop Entry] 2 | Version=1.0 3 | Type=Application 4 | Comment=Launch an ssh session to node01 5 | Exec=xfce4-terminal --geometry=+130+1 -e 'bash -c "ssh node01 || sleep 5"' 6 | Icon=virt-viewer 7 | Terminal=false 8 | StartupNotify=false 9 | Name=5. ssh to Node 01 10 | Path= 11 | -------------------------------------------------------------------------------- /docs/desktop/node02.desktop: -------------------------------------------------------------------------------- 1 | [Desktop Entry] 2 | Version=1.0 3 | Type=Application 4 | Comment=Launch an ssh session to node02 5 | Exec=xfce4-terminal --geometry=+130-1 -e 'bash -c "ssh node02 || sleep 5"' 6 | Icon=virt-viewer 7 | Terminal=false 8 | StartupNotify=false 9 | Name=6. ssh to Node 02 10 | Path= 11 | -------------------------------------------------------------------------------- /docs/desktop/server.desktop: -------------------------------------------------------------------------------- 1 | [Desktop Entry] 2 | Version=1.0 3 | Type=Application 4 | Comment=Launch in a termulator. DO ME FIRST! 5 | Exec=xfce4-terminal --geometry=90x26-1+1 -e 'bash -c "$HOME/github/F.E.E./ivshmsg_server.py --socket $HOME/FAME/node_socket || sleep 5"' 6 | Icon=network-server 7 | Terminal=false 8 | StartupNotify=false 9 | Name=1. Gen-Z Fabric Switch 10 | Path= 11 | -------------------------------------------------------------------------------- /docs/desktop/virt-manager.desktop: -------------------------------------------------------------------------------- 1 | [Desktop Entry] 2 | Name[as]=ভাৰ্চুৱেল ডিভাইচ ব্যৱস্থাপক 3 | Name[bg]=Мениджър на виртуални машини 4 | Name[bn_IN]=ভার্চুয়াল মেশিন ম্যানেজার 5 | Name[bs]=Upravljač virtualnog računala 6 | Name[ca]=Eina de gestió de màquines virtuals 7 | Name[cs]=Správce virtuálních strojů 8 | Name[da]=Administration for virtuel maskine 9 | Name[de]=Virtuelle Maschinenverwaltung 10 | Name[en_GB]=Virtual Machine Manager 11 | Name[es]=Gestor de máquinas virtuales 12 | Name[fi]=Virtuaalikoneiden hallitsin 13 | Name[fr]=Gestionnaire de machine virtuelle 14 | Name[gu]=વર્ચ્યુઅલ મશીન વ્યવસ્થાપક 15 | Name[hi]=वर्चुअल मशीन प्रबंधक 16 | Name[hr]=Upravljač virtualnog računala 17 | Name[hu]=Virtuális Gép vezérlőpult 18 | Name[it]=Virtual Machine Manager 19 | Name[ja]=仮想マシンマネージャー 20 | Name[kn]=ವರ್ಚುವಲ್‌ ಗಣಕ ವ್ಯವಸ್ಥಾಪಕ 21 | Name[ko]=가상 머신 관리자 22 | Name[ml]=വിര്‍ച്ച്വല്‍ മഷീന്‍ മാനേജര്‍ 23 | Name[mr]=वर्च्युअल मशीन व्यवस्थापक 24 | Name[ms]=Pengurus Mesin Maya 25 | Name[nl]=Virtuele machine beheerder 26 | Name[or]=ଆଭାସୀ ମେସିନ ପରିଚାଳକ 27 | Name[pa]=ਵਰਚੁਅਲ ਮਸ਼ੀਨ ਮੈਨੇਜਰ 28 | Name[pl]=Menedżer maszyn wirtualnych 29 | Name[pt]=Gestor de Máquinas Virtuais 30 | Name[pt_BR]=Gerenciador de Máquinas Virtuais 31 | Name[ro]=Manager maşină virtuală 32 | Name[ru]=Менеджер виртуальных машин 33 | Name[sk]=Správca virtuálnych počítačov 34 | Name[sr]=Управник виртуелне машине 35 | Name[sr@latin]=Upravnik virtuelne mašine 36 | Name[sv]=Administrerare av virtuella maskiner 37 | Name[ta]=மெய்நிகர் கணினி மேலாளர் 38 | Name[te]=వర్చ్యువల్ కంప్యూటరు నిర్వహణాధికారి 39 | Name[tr]=Sanal Makine Yöneticisi 40 | Name[uk]=Інструмент керування віртуальними машинами 41 | Name[zh_CN]=虚拟系统管理器 42 | Name[zh_TW]=虛擬機管理員 43 | Comment=Manage virtual machines 44 | Icon=virt-manager 45 | Exec=virt-manager 46 | Type=Application 47 | Terminal=false 48 | Categories=System; 49 | Name=4. VM Manager 50 | StartupNotify=false 51 | -------------------------------------------------------------------------------- /docs/desktop/wscycle: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Cycle through workspaces, assumes they are laid in in 1x4 (horizontal) grid. 4 | 5 | which xdotool 6 | [ $? -ne 0 ] && echo "Install xdotool" >&2 && exit 1 7 | 8 | TO=${1:-10} # seconds 9 | 10 | set -u 11 | 12 | [ $TO -lt 5 -o $TO -gt 30 ] && TO=10 13 | 14 | export PROG=`basename $0` 15 | N=`pgrep --count $PROG` 16 | echo $N to kill 17 | if [ $N -gt 1 ]; then 18 | nohup zenity --info --title 'Done' --text "Killing $N" & 19 | while pkill --oldest $PROG; do sleep 1 ; done 20 | exit 0 21 | fi 22 | 23 | while true; do 24 | xdotool key ctrl+alt+Right 25 | sleep $TO 26 | done 27 | 28 | exit 0 29 | -------------------------------------------------------------------------------- /docs/images/Element.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/linux-genz/F.E.E./0f7352f33d4f8054576d2b94a0605e181250d3b4/docs/images/Element.jpg -------------------------------------------------------------------------------- /docs/images/FEE block.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/linux-genz/F.E.E./0f7352f33d4f8054576d2b94a0605e181250d3b4/docs/images/FEE block.png -------------------------------------------------------------------------------- /docs/images/Gen-Z on black.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/linux-genz/F.E.E./0f7352f33d4f8054576d2b94a0605e181250d3b4/docs/images/Gen-Z on black.png -------------------------------------------------------------------------------- /docs/images/Gen-Z on white.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/linux-genz/F.E.E./0f7352f33d4f8054576d2b94a0605e181250d3b4/docs/images/Gen-Z on white.jpg -------------------------------------------------------------------------------- /docs/images/HPE logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/linux-genz/F.E.E./0f7352f33d4f8054576d2b94a0605e181250d3b4/docs/images/HPE logo.png -------------------------------------------------------------------------------- /docs/images/IVSHMEM block.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/linux-genz/F.E.E./0f7352f33d4f8054576d2b94a0605e181250d3b4/docs/images/IVSHMEM block.png -------------------------------------------------------------------------------- /docs/images/IVSHMSG block.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/linux-genz/F.E.E./0f7352f33d4f8054576d2b94a0605e181250d3b4/docs/images/IVSHMSG block.png -------------------------------------------------------------------------------- /docs/images/M-logo green.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/linux-genz/F.E.E./0f7352f33d4f8054576d2b94a0605e181250d3b4/docs/images/M-logo green.jpg -------------------------------------------------------------------------------- /docs/images/M-logo white.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/linux-genz/F.E.E./0f7352f33d4f8054576d2b94a0605e181250d3b4/docs/images/M-logo white.jpg -------------------------------------------------------------------------------- /docs/images/StarTrek.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/linux-genz/F.E.E./0f7352f33d4f8054576d2b94a0605e181250d3b4/docs/images/StarTrek.jpg -------------------------------------------------------------------------------- /docs/ivshmem-spec.txt: -------------------------------------------------------------------------------- 1 | = Device Specification for Inter-VM shared memory device = 2 | 3 | The Inter-VM shared memory device (ivshmem) is designed to share a 4 | memory region between multiple QEMU processes running different guests 5 | and the host. In order for all guests to be able to pick up the 6 | shared memory area, it is modeled by QEMU as a PCI device exposing 7 | said memory to the guest as a PCI BAR. 8 | 9 | The device can use a shared memory object on the host directly, or it 10 | can obtain one from an ivshmem server. 11 | 12 | In the latter case, the device can additionally interrupt its peers, and 13 | get interrupted by its peers. 14 | 15 | 16 | == Configuring the ivshmem PCI device == 17 | 18 | There are two basic configurations: 19 | 20 | - Just shared memory: -device ivshmem-plain,memdev=HMB,... 21 | 22 | This uses host memory backend HMB. It should have option "share" 23 | set. 24 | 25 | - Shared memory plus interrupts: -device ivshmem,chardev=CHR,vectors=N,... 26 | 27 | An ivshmem server must already be running on the host. The device 28 | connects to the server's UNIX domain socket via character device 29 | CHR. 30 | 31 | Each peer gets assigned a unique ID by the server. IDs must be 32 | between 0 and 65535. 33 | 34 | Interrupts are message-signaled (MSI-X). vectors=N configures the 35 | number of vectors to use. 36 | 37 | For more details on ivshmem device properties, see The QEMU Emulator 38 | User Documentation (qemu-doc.*). 39 | 40 | 41 | == The ivshmem PCI device's guest interface == 42 | 43 | The device has vendor ID 1af4, device ID 1110, revision 1. Before 44 | QEMU 2.6.0, it had revision 0. 45 | 46 | === PCI BARs === 47 | 48 | The ivshmem PCI device has two or three BARs: 49 | 50 | - BAR0 holds device registers (256 Byte MMIO) 51 | - BAR1 holds MSI-X table and PBA (only ivshmem-doorbell) 52 | - BAR2 maps the shared memory object 53 | 54 | There are two ways to use this device: 55 | 56 | - If you only need the shared memory part, BAR2 suffices. This way, 57 | you have access to the shared memory in the guest and can use it as 58 | you see fit. Memnic, for example, uses ivshmem this way from guest 59 | user space (see http://dpdk.org/browse/memnic). 60 | 61 | - If you additionally need the capability for peers to interrupt each 62 | other, you need BAR0 and BAR1. You will most likely want to write a 63 | kernel driver to handle interrupts. Requires the device to be 64 | configured for interrupts, obviously. 65 | 66 | Before QEMU 2.6.0, BAR2 can initially be invalid if the device is 67 | configured for interrupts. It becomes safely accessible only after 68 | the ivshmem server provided the shared memory. These devices have PCI 69 | revision 0 rather than 1. Guest software should wait for the 70 | IVPosition register (described below) to become non-negative before 71 | accessing BAR2. 72 | 73 | Revision 0 of the device is not capable to tell guest software whether 74 | it is configured for interrupts. 75 | 76 | === PCI device registers === 77 | 78 | BAR 0 contains the following registers: 79 | 80 | Offset Size Access On reset Function 81 | 0 4 read/write 0 Interrupt Mask 82 | bit 0: peer interrupt (rev 0) 83 | reserved (rev 1) 84 | bit 1..31: reserved 85 | 4 4 read/write 0 Interrupt Status 86 | bit 0: peer interrupt (rev 0) 87 | reserved (rev 1) 88 | bit 1..31: reserved 89 | 8 4 read-only 0 or ID IVPosition 90 | 12 4 write-only N/A Doorbell 91 | bit 0..15: vector 92 | bit 16..31: peer ID 93 | 16 240 none N/A reserved 94 | 95 | Software should only access the registers as specified in column 96 | "Access". Reserved bits should be ignored on read, and preserved on 97 | write. 98 | 99 | In revision 0 of the device, Interrupt Status and Mask Register 100 | together control the legacy INTx interrupt when the device has no 101 | MSI-X capability: INTx is asserted when the bit-wise AND of Status and 102 | Mask is non-zero and the device has no MSI-X capability. Interrupt 103 | Status Register bit 0 becomes 1 when an interrupt request from a peer 104 | is received. Reading the register clears it. 105 | 106 | IVPosition Register: if the device is not configured for interrupts, 107 | this is zero. Else, it is the device's ID (between 0 and 65535). 108 | 109 | Before QEMU 2.6.0, the register may read -1 for a short while after 110 | reset. These devices have PCI revision 0 rather than 1. 111 | 112 | There is no good way for software to find out whether the device is 113 | configured for interrupts. A positive IVPosition means interrupts, 114 | but zero could be either. 115 | 116 | Doorbell Register: writing this register requests to interrupt a peer. 117 | The written value's high 16 bits are the ID of the peer to interrupt, 118 | and its low 16 bits select an interrupt vector. 119 | 120 | If the device is not configured for interrupts, the write is ignored. 121 | 122 | If the interrupt hasn't completed setup, the write is ignored. The 123 | device is not capable to tell guest software whether setup is 124 | complete. Interrupts can regress to this state on migration. 125 | 126 | If the peer with the requested ID isn't connected, or it has fewer 127 | interrupt vectors connected, the write is ignored. The device is not 128 | capable to tell guest software what peers are connected, or how many 129 | interrupt vectors are connected. 130 | 131 | The peer's interrupt for this vector then becomes pending. There is 132 | no way for software to clear the pending bit, and a polling mode of 133 | operation is therefore impossible. 134 | 135 | If the peer is a revision 0 device without MSI-X capability, its 136 | Interrupt Status register is set to 1. This asserts INTx unless 137 | masked by the Interrupt Mask register. The device is not capable to 138 | communicate the interrupt vector to guest software then. 139 | 140 | With multiple MSI-X vectors, different vectors can be used to indicate 141 | different events have occurred. The semantics of interrupt vectors 142 | are left to the application. 143 | 144 | 145 | == Interrupt infrastructure == 146 | 147 | When configured for interrupts, the peers share eventfd objects in 148 | addition to shared memory. The shared resources are managed by an 149 | ivshmem server. 150 | 151 | === The ivshmem server === 152 | 153 | The server listens on a UNIX domain socket. 154 | 155 | For each new client that connects to the server, the server 156 | - picks an ID, 157 | - creates eventfd file descriptors for the interrupt vectors, 158 | - sends the ID and the file descriptor for the shared memory to the 159 | new client, 160 | - sends connect notifications for the new client to the other clients 161 | (these contain file descriptors for sending interrupts), 162 | - sends connect notifications for the other clients to the new client, 163 | and 164 | - sends interrupt setup messages to the new client (these contain file 165 | descriptors for receiving interrupts). 166 | 167 | The first client to connect to the server receives ID zero. 168 | 169 | When a client disconnects from the server, the server sends disconnect 170 | notifications to the other clients. 171 | 172 | The next section describes the protocol in detail. 173 | 174 | If the server terminates without sending disconnect notifications for 175 | its connected clients, the clients can elect to continue. They can 176 | communicate with each other normally, but won't receive disconnect 177 | notification on disconnect, and no new clients can connect. There is 178 | no way for the clients to connect to a restarted server. The device 179 | is not capable to tell guest software whether the server is still up. 180 | 181 | Example server code is in contrib/ivshmem-server/. Not to be used in 182 | production. It assumes all clients use the same number of interrupt 183 | vectors. 184 | 185 | A standalone client is in contrib/ivshmem-client/. It can be useful 186 | for debugging. 187 | 188 | === The ivshmem Client-Server Protocol === 189 | 190 | An ivshmem device configured for interrupts connects to an ivshmem 191 | server. This section details the protocol between the two. 192 | 193 | The connection is one-way: the server sends messages to the client. 194 | Each message consists of a single 8 byte little-endian signed number, 195 | and may be accompanied by a file descriptor via SCM_RIGHTS. Both 196 | client and server close the connection on error. 197 | 198 | Note: QEMU currently doesn't close the connection right on error, but 199 | only when the character device is destroyed. 200 | 201 | On connect, the server sends the following messages in order: 202 | 203 | 1. The protocol version number, currently zero. The client should 204 | close the connection on receipt of versions it can't handle. 205 | 206 | 2. The client's ID. This is unique among all clients of this server. 207 | IDs must be between 0 and 65535, because the Doorbell register 208 | provides only 16 bits for them. 209 | 210 | 3. The number -1, accompanied by the file descriptor for the shared 211 | memory. 212 | 213 | 4. Connect notifications for existing other clients, if any. This is 214 | a peer ID (number between 0 and 65535 other than the client's ID), 215 | repeated N times. Each repetition is accompanied by one file 216 | descriptor. These are for interrupting the peer with that ID using 217 | vector 0,..,N-1, in order. If the client is configured for fewer 218 | vectors, it closes the extra file descriptors. If it is configured 219 | for more, the extra vectors remain unconnected. 220 | 221 | 5. Interrupt setup. This is the client's own ID, repeated N times. 222 | Each repetition is accompanied by one file descriptor. These are 223 | for receiving interrupts from peers using vector 0,..,N-1, in 224 | order. If the client is configured for fewer vectors, it closes 225 | the extra file descriptors. If it is configured for more, the 226 | extra vectors remain unconnected. 227 | 228 | From then on, the server sends these kinds of messages: 229 | 230 | 6. Connection / disconnection notification. This is a peer ID. 231 | 232 | - If the number comes with a file descriptor, it's a connection 233 | notification, exactly like in step 4. 234 | 235 | - Else, it's a disconnection notification for the peer with that ID. 236 | 237 | Known bugs: 238 | 239 | * The protocol changed incompatibly in QEMU 2.5. Before, messages 240 | were native endian long, and there was no version number. 241 | 242 | * The protocol is poorly designed. 243 | 244 | === The ivshmem Client-Client Protocol === 245 | 246 | An ivshmem device configured for interrupts receives eventfd file 247 | descriptors for interrupting peers and getting interrupted by peers 248 | from the server, as explained in the previous section. 249 | 250 | To interrupt a peer, the device writes the 8-byte integer 1 in native 251 | byte order to the respective file descriptor. 252 | 253 | To receive an interrupt, the device reads and discards as many 8-byte 254 | integers as it can. 255 | -------------------------------------------------------------------------------- /docs/setup-scripts/README.md: -------------------------------------------------------------------------------- 1 | These steps were created for HPE Discover November 2018 in Madrid for the 2 | "Hack Shack" hands-on development presentation. They are written assuming your 3 | laptop is running a recent version of Windows. The instructions should also work 4 | on a MacOS laptop. 5 | 6 | 1. Reboot your Windows system and Enable VT & VTd in BIOS (so that Virtualbox can use VT technology in VM) 7 | 1. [Download VirtualBox]( https://download.virtualbox.org/virtualbox/5.2.22/VirtualBox-5.2.22-126460-Win.exe) 8 | 1. Download a Debian-based install ISO: 9 | 1. [Ubuntu 18.04.1 desktop](http://releases.ubuntu.com/18.04.1/ubuntu-18.04.1-desktop-amd64.iso) 10 | 1. [Debian Stretch 9.x with "amd64" via http option](https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-9.3.0-amd64-netinst.iso) 11 | 1. Install Virtualbox 5.2.22 for Windows using the VirtualBox-5.2.22-126460-Win.exe 12 | 1. Use the Linux ISO to install a virtualbox VM. Create a disk image of at least 25GB so that there is enough space to create the 2 VMs 13 | 1. Boot the Linux image and login. THis is your QEMU host for the F.E.E. VMs. 14 | 1. Start up a BASH (terminal) and then pull the F.E.E. setup scripts: 15 | 1. wget https://raw.githubusercontent.com/linux-genz/F.E.E./master/docs/setup-scripts/setup-host-genz-emul-env.sh 16 | 1. bash -c setup-host-genz-emul-env.sh 17 | 1. logout of the QEMU host and log in to enable participation in new groups: 18 | 1. run "groups" and verify that you are in these groups: libvirt and libvirt-qemu 19 | 1. Now pull and run the next setup script from this directory: 20 | 1. github/F.E.E./docs/setup-scripts/run-host-genz-emul-env.sh 21 | 22 | -------------------------------------------------------------------------------- /docs/setup-scripts/run-host-genz-emul-env.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash -x 2 | # run the Gen-Z Fabric Emulation Environment (F.E.E.) on Ubuntu 18.04.1 3 | # your mileage will vary on other distros 4 | 5 | # run this script after running setup-host-genz-emul-env.sh 6 | 7 | # Copyright 2018 Hewlett Packard Enterprise Development LP 8 | # Author: Bill Hayes 9 | 10 | # This program is free software: you can redistribute it and/or modify 11 | # it under the terms of the GNU General Public License, version 2 as 12 | # published by the Free Software Foundation. 13 | 14 | # This program is distributed in the hope that it will be useful, 15 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 16 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 17 | # GNU General Public License for more details. 18 | 19 | # You should have received a copy of the GNU General Public License along 20 | # with this program. If not, write to the Free Software Foundation, Inc., 21 | # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA 22 | 23 | # setup the virtual machines that were created by the FAME tool in virsh/libvirt/etc 24 | # NOTE: You must belong to group libvirt-qemu (see logout at end of setup-genz-emul-env.sh script) 25 | cd 26 | cd FAME 27 | 28 | # source the FAME variables 29 | . node_env.sh 30 | 31 | # create a virsh definition of the FAME VM's 32 | ./node_virsh.sh define 33 | 34 | # add entries in /etc/hosts for nodeXX (if they don't exist) 35 | if ! grep -q node01 /etc/hosts; then 36 | echo "192.168.42.1 node01" | sudo tee -a /etc/hosts 37 | echo "192.168.42.2 node02" | sudo tee -a /etc/hosts 38 | fi 39 | 40 | # setup the SSH keys (that FAME used in VM's) on this host system (if they are not setup) 41 | if [ ! -f .ssh/config ]; then 42 | cp $HOME/github/Emulation/templates/id_rsa.nophrase* $HOME/.ssh 43 | chmod 600 $HOME/.ssh/*rsa* 44 | cat > $HOME/.ssh/config << EOF 45 | ConnectTimeout 5 46 | StrictHostKeyChecking No 47 | AddKeysToAgent yes 48 | 49 | Host node* 50 | User l4mdc 51 | IdentityFile ~/.ssh/id_rsa.nophrase 52 | EOF 53 | rm -f $HOME/.ssh/known_hosts 54 | fi 55 | 56 | # start ivshmsg_server.py in another terminal window 57 | # directly stolen from: https://github.com/linux-genz/F.E.E./blob/master/docs/desktop/server.desktop 58 | # suggestion: type 'help' inside the ivshmsg_server.py terminal window 59 | gnome-terminal -e 'bash -c "$HOME/github/F.E.E./ivshmsg_server.py --socket $HOME/FAME/node_socket || sleep 5"' 60 | 61 | # let the ivshmsg_server.py get running so that it is listening for network connections 62 | sleep 10 63 | 64 | # start ivshmsg_client.py in another terminal window 65 | # directly stolen from: https://github.com/linux-genz/F.E.E./blob/master/docs/desktop/client.desktop 66 | # suggestion: type 'help' inside the ivshmsg_client.py terminal window 67 | gnome-terminal -e 'bash -c "$HOME/github/F.E.E./ivshmsg_client.py --socket $HOME/FAME/node_socket || sleep 5"' 68 | 69 | # start the VM's 70 | sudo virsh start node01 71 | sudo virsh start node02 72 | 73 | # let the VM's start running before trying to ssh to them 74 | sleep 30 75 | 76 | # setup the EmerGen-Z environment in each of VM's 77 | # for more info see https://github.com/linux-genz/F.E.E./blob/master/docs/VMconfig.md#running-linux-guests-with-ivshmsg-kernel-modules 78 | scp $HOME/github/F.E.E./docs/setup-scripts/setup-guest-genz-emul-env.sh node01:. 79 | scp $HOME/github/F.E.E./docs/setup-scripts/setup-guest-genz-emul-env.sh node02:. 80 | ssh node01 ./setup-guest-genz-emul-env.sh 81 | ssh node02 ./setup-guest-genz-emul-env.sh 82 | 83 | # setup interactive ssh sessions to the VM's 84 | gnome-terminal -e 'bash -c "ssh node01 || sleep 5"' 85 | gnome-terminal -e 'bash -c "ssh node02 || sleep 5"' 86 | 87 | -------------------------------------------------------------------------------- /docs/setup-scripts/setup-guest-genz-emul-env.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash -x 2 | # setup the EmerGen-Z on Debian Stretch VM (as setup by FAME) 3 | # for more info see https://github.com/linux-genz/F.E.E./blob/master/docs/VMconfig.md#running-linux-guests-with-ivshmsg-kernel-modules 4 | 5 | # this script is usually used by run-host-genz-emul-env.sh 6 | 7 | # Copyright 2018 Hewlett Packard Enterprise Development LP 8 | # Author: Bill Hayes 9 | 10 | # This program is free software: you can redistribute it and/or modify 11 | # it under the terms of the GNU General Public License, version 2 as 12 | # published by the Free Software Foundation. 13 | 14 | # This program is distributed in the hope that it will be useful, 15 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 16 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 17 | # GNU General Public License for more details. 18 | 19 | # You should have received a copy of the GNU General Public License along 20 | # with this program. If not, write to the Free Software Foundation, Inc., 21 | # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA 22 | 23 | # install git 24 | sudo apt-get -y update 25 | sudo apt-get -y install git-core 26 | 27 | # create github directory in the user's home directory 28 | cd 29 | mkdir github 30 | cd github 31 | 32 | # pull the EmerGen-Z source code (via git) 33 | git clone https://github.com/linux-genz/EmerGen-Z.git 34 | 35 | # build the EmerGen-Z drivers 36 | cd EmerGen-Z/subsystem 37 | make modules_install 38 | cd ../shim_bridge 39 | make modules_install 40 | # drivers have been installed here: ls -lR /lib/modules/`uname -r`/genz/ 41 | 42 | # load the EmerGen-Z drivers (and watch the Gen-Z state in ivshmsg_server.py) 43 | sudo modprobe genz verbose=2 44 | sudo modprobe genz_fee_bridge verbose=2 45 | -------------------------------------------------------------------------------- /docs/setup-scripts/setup-host-genz-emul-env.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash -x 2 | # setup Gen-Z Fabric Emulation Environment (F.E.E.) on Ubuntu 18.04.1 3 | # your mileage will vary on other distros 4 | 5 | # this script will NOT setup Executive Cardboard right now 6 | 7 | # run this script first, then run-host-genz-emul-env.sh (which uses setup-guest-genz-emul-env.sh) 8 | 9 | # Copyright 2018 Hewlett Packard Enterprise Development LP 10 | # Author: Bill Hayes 11 | 12 | # This program is free software: you can redistribute it and/or modify 13 | # it under the terms of the GNU General Public License, version 2 as 14 | # published by the Free Software Foundation. 15 | 16 | # This program is distributed in the hope that it will be useful, 17 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 18 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 19 | # GNU General Public License for more details. 20 | 21 | # You should have received a copy of the GNU General Public License along 22 | # with this program. If not, write to the Free Software Foundation, Inc., 23 | # 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA 24 | 25 | # install additional SW (qemu/kvm, git, python libraries) on top of Ubuntu 18.04.1 needed for Gen-Z F.E.E. & FAME 26 | sudo apt-get -y update 27 | sudo apt-get -y upgrade 28 | sudo apt-get -y dist-upgrade 29 | sudo apt-get -y install git qemu-kvm libvirt-clients libvirt-daemon-system bridge-utils virt-manager virt-viewer vmdebootstrap python3 python3-daemonize python3-attr python3-twisted python3-klein 30 | 31 | # enable this user to use virsh/libvirt (this seems to need a reboot to activate) 32 | sudo usermod -a -G libvirt-qemu $USER 33 | 34 | # create the FAME and github directories in the user's home directory 35 | cd 36 | mkdir {FAME,github} 37 | 38 | # get the Gen-Z F.E.E. & FAME tools on the host system 39 | cd github 40 | git clone https://github.com/FabricAttachedMemory/Emulation.git 41 | git clone https://github.com/linux-genz/F.E.E..git 42 | 43 | # build the VM's using FAME that will support Gen-Z F.E.E. and run EmerGen-Z 44 | # For the full set of options see https://github.com/linux-genz/F.E.E./blob/master/docs/VMconfig.md#method-3-fame 45 | cd Emulation 46 | 47 | # for more into see https://github.com/linux-genz/F.E.E./blob/master/docs/VMconfig.md#method-3-fame 48 | export FAME_DIR=$HOME/FAME 49 | export FAME_FAM= 50 | export FAME_IVSHMSG=yes 51 | 52 | # emulation_configure.bash frequently asks questions default to 'yes' 53 | yes yes | ./emulation_configure.bash 2 54 | 55 | # setup done, now logout to fully join the libvirt-qemu group, then run run-host-genz-emul-env.sh 56 | echo "setup done, now logout to fully join the libvirt-qemu group, then run $HOME/github/F.E.E./docs/setup-scripts/run-host-genz-emul-env.sh" 57 | 58 | -------------------------------------------------------------------------------- /ivshmsg_client.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | 3 | # This work is licensed under the terms of the GNU GPL, version 2 or 4 | # (at your option) any later version. See the LICENSE file in the 5 | # top-level directory. 6 | 7 | # Rocky Craig 8 | 9 | # This is modelled on the ivshmem-client program that comes with QEMU. 10 | 11 | import argparse 12 | import grp 13 | import os 14 | import sys 15 | 16 | from ivshmsg_twisted.twisted_client import FactoryIVSHMSGClient 17 | 18 | ########################################################################### 19 | 20 | 21 | def parse_cmdline(cmdline_args): 22 | '''cmdline_args does NOT lead with the program name. Single-letter 23 | arguments reflect the stock QEMU "ivshmem-client".''' 24 | parser = argparse.ArgumentParser( 25 | description='IVSHMSG client files', 26 | epilog='Options reflect those in the QEMU "ivshmem-client".' 27 | ) 28 | parser.add_argument('-?', action='help') # -h and --help are built in 29 | parser.add_argument('--socketpath', '-S', metavar='/path/to/socket', 30 | help='Absolute path to UNIX domain socket created by the server', 31 | default='/tmp/ivshmsg_socket' 32 | ) 33 | parser.add_argument('--verbose', '-v', 34 | help='Specify multiple times to increase verbosity', 35 | default=0, 36 | action='count' 37 | ) 38 | args = parser.parse_args(cmdline_args) 39 | 40 | # Idiot checking. 41 | assert os.path.exists(args.socketpath), \ 42 | 'No socket %s (have you started ivshmsg_server?)' % args.socketpath 43 | 44 | return args 45 | 46 | ########################################################################### 47 | # MAIN 48 | 49 | 50 | def forever(cmdline_args=None): 51 | if cmdline_args is None: 52 | cmdline_args = sys.argv[1:] # When being explicit, strip prog name 53 | try: 54 | args = parse_cmdline(cmdline_args) 55 | except Exception as e: 56 | raise SystemExit(str(e)) 57 | 58 | client = FactoryIVSHMSGClient(args) 59 | client.run() 60 | 61 | ########################################################################### 62 | 63 | 64 | if __name__ == '__main__': 65 | forever() 66 | 67 | -------------------------------------------------------------------------------- /ivshmsg_server.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | 3 | # This work is licensed under the terms of the GNU GPL, version 2 or 4 | # (at your option) any later version. See the LICENSE file in the 5 | # top-level directory. 6 | 7 | # Rocky Craig 8 | 9 | import argparse 10 | import grp 11 | import os 12 | import sys 13 | 14 | from ivshmsg_twisted.twisted_server import FactoryIVSHMSGServer 15 | 16 | ########################################################################### 17 | 18 | 19 | def parse_cmdline(cmdline_args): 20 | '''cmdline_args does NOT lead with the program name. Single-letter 21 | arguments reflect the stock QEMU "ivshmem-server".''' 22 | parser = argparse.ArgumentParser( 23 | description='IVSHMSG server files and vector counts', 24 | epilog='Options reflect those in the QEMU "ivshmem-server".' 25 | ) 26 | parser.add_argument('-?', action='help') # -h and --help are built in 27 | parser.add_argument('--daemon', '-D', 28 | help='Run in background, log to file (default: foreground/stdout)', 29 | # The twisted module expectes the attribute 'foreground'... 30 | dest='foreground', 31 | action='store_false', # ...so reverse the polarity, Scotty 32 | default=True 33 | ) 34 | parser.add_argument('--logfile', '-L', metavar='', 35 | help='Pathname of logfile for use in daemon mode', 36 | default='/tmp/ivshmsg_log' 37 | ) 38 | parser.add_argument('--mailbox', '-M', metavar='', 39 | help='Name of mailbox that exists in POSIX shared memory', 40 | default='ivshmsg_mailbox' 41 | ) 42 | parser.add_argument('--nClients', '-n', metavar='', 43 | help='Serve up to this number of clients (max=14)', 44 | type=int, 45 | default=14 46 | ) 47 | parser.add_argument('--norecycle', 48 | dest='recycle', # By default, DO recycle FDs, do not... 49 | help='Use QEMU legacy mechanism of new FDs on respawn; known to crash surviving sessions', 50 | action='store_false', 51 | default=True 52 | ) 53 | parser.add_argument('--silent', '-s', 54 | help='Do NOT participate in EventFDs/mailbox as another peer', 55 | action='store_true', 56 | default=False 57 | ) 58 | parser.add_argument('--socketpath', '-S', metavar='/path/to/socket', 59 | help='Absolute path to UNIX domain socket (will be created)', 60 | default='/tmp/ivshmsg_socket' 61 | ) 62 | parser.add_argument('--verbose', '-v', 63 | help='Specify multiple times to increase verbosity', 64 | default=0, 65 | action='count' 66 | ) 67 | parser.add_argument('--noPFM', 68 | dest='smart', 69 | help='Suppress rudimentary fabric management for clients', 70 | action='store_false', 71 | default=True 72 | ) 73 | 74 | # Generate the object and postprocess some of the fields. 75 | args = parser.parse_args(cmdline_args) 76 | assert 1 <= args.nClients <= 62, 'nClients is out of range 1 - 62' 77 | assert not (args.silent and args.smart), \ 78 | 'Silent/smart are mutually exclusive' 79 | assert not '/' in args.mailbox, 'mailbox cannot have slashes' 80 | assert not os.path.exists(args.socketpath), 'Remove %s' % args.socketpath 81 | 82 | return args 83 | 84 | ########################################################################### 85 | # MAIN 86 | 87 | 88 | def forever(cmdline_args=None): 89 | if cmdline_args is None: 90 | cmdline_args = sys.argv[1:] # When being explicit, strip prog name 91 | try: 92 | args = parse_cmdline(cmdline_args) 93 | except Exception as e: 94 | raise SystemExit(str(e)) 95 | 96 | if not args.foreground: 97 | raise NotImplementedError('Gotta run it in the foreground for now') 98 | # FIXME: clean up logging for twistd. The following is non-optimal. 99 | from daemonize import Daemonize 100 | if args.verbose: 101 | print(Daemonize.__doc__) # The website is WRONG 102 | d = Daemonize('ivshmsg_server', '/dev/null', None, auto_close_fds=None) 103 | d.start() 104 | server = FactoryIVSHMSGServer(args) 105 | server.run() 106 | 107 | ########################################################################### 108 | 109 | 110 | if __name__ == '__main__': 111 | forever() 112 | 113 | -------------------------------------------------------------------------------- /ivshmsg_twisted/commander.py: -------------------------------------------------------------------------------- 1 | ########################################################################### 2 | # Serve as a gateway between one protocol (stdin) and the other(IVSHMSG). 3 | # The protocol object must be established before (like in a factory) 4 | # and must contain a "doCommand(self, cmdline)" method. Then a line 5 | # completion (ie, typing something and hitting ENTER) will invoke doCommand. 6 | # Invocation: 7 | # 8 | # class MyFactory(BaseFactory): 9 | # : 10 | # def buildProtocol(self, addr): 11 | # protobj = MyProtocol(....) 12 | # Commander(protobj) 13 | # return protobj 14 | # 15 | # Yeah, a mixin or implementer is probably the right way. Manana. 16 | # This approach hides everything in here. 17 | # An error in here tends to not make it to stdout/err and usually 18 | # severs the network connection established by the protocol object. 19 | 20 | import os 21 | import sys 22 | 23 | from twisted.internet import reactor as TIreactor 24 | 25 | from twisted.internet.stdio import StandardIO # internally hooks into "reactor" 26 | 27 | from twisted.protocols.basic import LineReceiver 28 | 29 | class _proxyCommander(LineReceiver): 30 | 31 | delimiter = os.linesep.encode('ascii') # Override for LineReceiver 32 | 33 | _once = None 34 | 35 | def __init__(self, commProto): 36 | if self._once is not None: 37 | return 38 | self.__class__._once = self 39 | self.commProto = commProto 40 | self.jfdi = getattr(commProto, 'doCommand', self.doCommandDefault) 41 | print('Command processing is ready...', file=sys.stderr) 42 | 43 | def connectionMade(self): # First contact, kick the machinery 44 | self.lineReceived(b'') # FIXME: direct write to stdin? 45 | 46 | def connectionLost(self, reason): 47 | if self.jfdi == self.doCommandDefault: # doing it to myself 48 | TIreactor.stop() 49 | 50 | def _issue_prompt(self): 51 | try: 52 | nodename = self.commProto.promptname 53 | except AttributeError as e: 54 | nodename = 'cmd' 55 | tmp = '%s> ' % nodename 56 | self.prompt = tmp.encode() 57 | self.transport.write(self.prompt) 58 | 59 | def doCommandDefault(self, cmdline): 60 | '''The default command line processor: help and quit.''' 61 | if not cmdline.strip(): 62 | return True # "Keep going" 63 | elems = cmdline.split() 64 | cmd = elems.pop(0) 65 | if cmd in ('h', 'help', 'l', 'list') or '?' in cmd: 66 | print('h[elp]\n\tThis message') 67 | print('q[uit]\n\tJust do it') 68 | elif cmd.startswith('q'): 69 | self.transport.loseConnection() 70 | return True 71 | 72 | def lineReceived(self, line): 73 | line = line.decode().strip() 74 | args = line.split() 75 | ok = True 76 | cmd = args.pop(0) if len(args) else '' 77 | if cmd: 78 | try: 79 | ok = self.jfdi(cmd, args) 80 | except NotImplementedError as e: 81 | print(str(e)) 82 | ok = False 83 | except Exception as e: 84 | ok = False 85 | print('Error: %s' % str(e), file=sys.stderr) 86 | 87 | if not ok: 88 | if cmd: 89 | if cmd in ('q', 'quit'): 90 | self.transport.loseConnection() 91 | return 92 | print('Unrecognized command "%s", try "help"' % cmd) 93 | 94 | # Do this each time cuz client's is not known at __init__ time. 95 | self._issue_prompt() 96 | 97 | ########################################################################### 98 | 99 | 100 | def Commander(protobj): 101 | return StandardIO(_proxyCommander(protobj)) 102 | -------------------------------------------------------------------------------- /ivshmsg_twisted/famez_requests.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | 3 | # Common routines for both server (switch) and clients which drill down 4 | # on messages retrieved by parsing and generating a response. 5 | 6 | import attr 7 | import os 8 | import functools 9 | import sys 10 | 11 | from collections import OrderedDict 12 | from pprint import pprint 13 | 14 | try: 15 | from ivshmsg_mailbox import IVSHMSG_MailBox as MB 16 | except ImportError as e: 17 | from .ivshmsg_mailbox import IVSHMSG_MailBox as MB 18 | 19 | def PRINT(*args): 20 | print(*args, file=_stdtrace) 21 | 22 | def PPRINT(*args): 23 | pprint(*args, stream=_stdtrace) 24 | 25 | ########################################################################### 26 | # Better than __slots__ although maybe this should be the precursor for 27 | # making this entire file a class. It's all items that cover the gamut 28 | # of requests. "this" is the object with SID0, CID0, LinkState and cclass. 29 | 30 | 31 | ResponseObject = attr.make_class('ResponseObject', 32 | ['this', 'proxy', 'from_id', 'to_doorbell', 33 | 'logmsg', 'stdtrace', 'verbose']) 34 | 35 | ########################################################################### 36 | # Create a subroutine name out of the elements passed in. Return 37 | # the remainder. Start with the least-specific construct. 38 | 39 | 40 | def _unprocessed(client, *args, **kwargs): 41 | return False 42 | if client.SI.verbose: 43 | _logmsg('NOOP', args, kwargs) 44 | return False 45 | 46 | 47 | def chelsea(elements, verbose=0): 48 | entry = '' # They begin with a leading '_', wait for it... 49 | G = globals() 50 | for i, e in enumerate(elements): 51 | e = e.replace('-', '_') # Such as 'Link CTL Peer-Attribute' 52 | entry += '_%s' % e 53 | if verbose > 1: 54 | print('Looking for %s()...' % entry, end='', file=_stdtrace) 55 | if entry in G: 56 | args = elements[i + 1:] 57 | if verbose > 1: 58 | PRINT('found it->%s' % str(args)) 59 | return G[entry], args 60 | if verbose > 1: 61 | PRINT('NOPE') 62 | return _unprocessed, elements 63 | 64 | ########################################################################### 65 | 66 | 67 | def CSV2dict(oneCSVstr): 68 | kv = {} 69 | elems = oneCSVstr.strip().split(',') 70 | for e in elems: 71 | try: 72 | KeV = e.strip().split('=') 73 | kv[KeV[0].strip()] = KeV[1].strip() 74 | except Exception as e: 75 | continue 76 | return kv 77 | 78 | ########################################################################### 79 | # Here instead of ivshmsg_mailbox to manage the tag. Can be called as a 80 | # "discussion initiator" usually from the REPL interpreters, or as a 81 | # response to a received command from the callbacks. 82 | 83 | _next_tag = 1 # Gen-Z tag field 84 | 85 | _tagged = OrderedDict() # By tag, just store receiver now 86 | 87 | _tracker = 0 # EmerGen-Z addenda to watch conversations 88 | 89 | _TRACKER_TOKEN = '!EZT=' 90 | 91 | def send_payload(payload, from_id, to_doorbell, reset_tracker=False, 92 | tag=None, tagCID=0, tagSID=0): 93 | global _next_tag, _tracker 94 | 95 | # PRINT('Send "%s" from %d to %s' % (payload, from_id, vars(to_doorbell))) 96 | 97 | if tag is not None: # zero-length string can trigger this 98 | payload += ',Tag=%d' % _next_tag 99 | _tagged[str(_next_tag)] = '%d.%d!%s|%s' % ( 100 | tagCID, tagSID, payload, tag) 101 | _next_tag += 1 102 | 103 | # Put the tracker on the end where it's easier to find 104 | if reset_tracker: 105 | _tracker = 0 106 | _tracker += 1 107 | payload += '%s%d' % (_TRACKER_TOKEN, _tracker) 108 | 109 | ret = MB.fill(from_id, payload) # True == no timeout, no stomp 110 | to_doorbell.ring() 111 | return ret 112 | 113 | ########################################################################### 114 | # Gen-Z 1.0 "6.8 Standalone Acknowledgment" 115 | # Received by server/switch 116 | 117 | 118 | def _Standalone_Acknowledgment(response_receiver, args): 119 | retval = True 120 | tag = False 121 | try: 122 | kv = CSV2dict(args[0]) 123 | stamp, tag = _tagged[kv['Tag']].split('|') 124 | del _tagged[kv['Tag']] 125 | tag = tag.strip() 126 | kv = CSV2dict(tag) 127 | except KeyError as e: 128 | _stdtrace('UNTAGGING %d:%s FAILED' % 129 | (response_receiver.from_id, response_receiver.nodename)) 130 | retval = False 131 | kv = {} 132 | 133 | afterACK = kv.get('AfterACK', False) 134 | if afterACK: 135 | send_payload(afterACK, 136 | response_receiver.from_id, 137 | response_receiver.to_doorbell) 138 | 139 | if _tagged: 140 | PRINT('Outstanding tags:') 141 | PPRINT(_tagged) 142 | return 'dump' 143 | 144 | 145 | def _send_SA(RO, tag, reason): 146 | payload = 'Standalone Acknowledgment Tag=%s,Reason=%s' % (tag, reason) 147 | return send_payload(payload, RO.from_id, RO.to_doorbell) 148 | 149 | ########################################################################### 150 | # Gen-Z 1.0 "11.11 Link CTL" subfield 151 | # Sent by clients 152 | 153 | 154 | def send_LinkACK(RO, details, nack=False): 155 | if nack: 156 | payload = 'Link CTL NAK %s' % details 157 | else: 158 | payload = 'Link CTL ACK %s' % details 159 | return send_payload(payload, RO.from_id, RO.to_doorbell) 160 | 161 | ########################################################################### 162 | # Gen-Z 1.0 "6.10.1 P2P Core..." 163 | # Received by client, only really expecting RFC data 164 | 165 | 166 | def _CTL_Write(RO, args): 167 | kv = CSV2dict(args[0]) 168 | if int(kv['Space']) != 0: 169 | return False 170 | RO.this.CID0 = int(kv['CID']) 171 | RO.this.SID0 = int(kv['SID']) 172 | RO.this.PFMCID0 = int(kv['PFMCID']) 173 | RO.this.PFMSID0 = int(kv['PFMSID']) 174 | RO.this.linkattrs['State'] = 'configured' 175 | return _send_SA(RO, kv['Tag'], 'OK') 176 | 177 | ########################################################################### 178 | # Gen-Z 1.0 "11.6 Link RFC" 179 | # Received by switch 180 | 181 | 182 | def _Link_RFC(RO, args): 183 | if not RO.this.isPFM: 184 | _logmsg('I am not a manager') 185 | return False 186 | try: 187 | kv = CSV2dict(args[0]) 188 | delay = kv['TTC'].lower() 189 | except (IndexError, KeyError) as e: 190 | _logmsg('%d: Link RFC missing TTC' % RO.from_id) 191 | return False 192 | if not 'us' in delay: # greater than cycle time of this server 193 | _logmsg('Delay %s is too long, dropping request' % delay) 194 | return False 195 | payload = 'CTL-Write Space=0,PFMCID=%d,PFMSID=%d,CID=%d,SID=%d' % ( 196 | RO.this.CID0, RO.this.SID0, RO.proxy.CID0, RO.proxy.SID0) 197 | return send_payload(payload, RO.from_id, RO.to_doorbell, 198 | tag='AfterACK=Link CTL Peer-Attribute', 199 | tagCID=RO.this.CID0, tagSID=RO.this.SID0) 200 | 201 | ########################################################################### 202 | # Gen-Z 1.0 "11.11 Link CTL" 203 | # Entered on both client and server responses. 204 | 205 | 206 | def _Link_CTL(RO, args): 207 | '''Subelements should be empty.''' 208 | arg0 = args[0] if len(args) else '' 209 | if len(args) == 1: 210 | if arg0 == 'Peer-Attribute': 211 | attrs = 'cclass=%s,CID0=%d,SID0=%d' % ( 212 | RO.this.cclass, RO.this.CID0, RO.this.SID0) 213 | return send_LinkACK(RO, attrs) 214 | 215 | if arg0 == 'ACK' and len(args) == 2: 216 | # Update the local proxy values, ASS-U-ME it's peerattrs 217 | # FIXME: correlation ala _tagged? How do I know it's peer attrs? 218 | # FIXME: add a key to the response... 219 | if RO.proxy is None: 220 | RO.this.peerattrs = CSV2dict(args[1]) 221 | else: 222 | RO.proxy.peerattrs = CSV2dict(args[1]) 223 | return 'dump' 224 | 225 | if arg0 == 'NAK': 226 | # FIXME: do I track the sender ala _tagged and deal with it? 227 | PRINT('Got a NAK, not sure what to do with it.') 228 | return False 229 | 230 | _logmsg('Got %s from %d' % (str(args), RO.from_id)) 231 | return False 232 | 233 | ########################################################################### 234 | # Finally a home 235 | 236 | 237 | def _ping(RO, args): 238 | return send_payload('pong', RO.from_id, RO.to_doorbell) 239 | 240 | 241 | def _dump(RO, args): 242 | return 'dump' # Technically "True", but with baggage 243 | 244 | 245 | ########################################################################### 246 | # Chained from EventReader callback in twisted_[client|server].py. 247 | # Command streams are case-sensitive, read the spec. 248 | # Return True if successfully parsed and processed. 249 | 250 | _logmsg = None 251 | _stdtrace = None 252 | 253 | 254 | def handle_request(request, requester_name, response_object): 255 | global _logmsg, _stdtrace, _tracker 256 | 257 | assert isinstance(response_object, ResponseObject), 'Bad response object' 258 | if _logmsg is None: 259 | _logmsg = response_object.logmsg # FIXME: logger.logger... 260 | _stdtrace = response_object.stdtrace 261 | 262 | elements = request.split(_TRACKER_TOKEN) 263 | payload = elements.pop(0) 264 | trace = '\n%10s -> "%s"' % (requester_name, payload) 265 | EZT = int(elements[0]) if elements else False 266 | if EZT: 267 | trace += ' (%d)' % EZT 268 | _tracker = EZT 269 | PRINT(trace) 270 | 271 | elements = payload.split() 272 | try: 273 | handler, args = chelsea(elements, response_object.verbose) 274 | return handler(response_object, args) 275 | except KeyError as e: 276 | _logmsg('KeyError: %s' % str(e)) 277 | except Exception as e: 278 | _logmsg(str(e)) 279 | return False 280 | -------------------------------------------------------------------------------- /ivshmsg_twisted/ivshmsg_eventfd.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | 3 | # This work is licensed under the terms of the GNU GPL, version 2 or 4 | # (at your option) any later version. See the LICENSE file in the 5 | # top-level directory. 6 | 7 | # Rocky Craig 8 | 9 | # Routine names here mirror those in qemu/contrib/ivshmem-[client|server]. 10 | # The IVSHMEM communications protocol (now christened IVSHMSG) is based on 11 | # 8-byte integers and an optional 4-byte file descriptor. Twisted 12 | # transport.sendFileDescriptor gets the packing sizes wrong for IVSHMSG 13 | # so do it right. 14 | 15 | import errno 16 | import os 17 | import struct 18 | import sys 19 | 20 | from twisted.internet import reactor as TIreactor # should be same everywhere 21 | from twisted.internet.interfaces import IReadDescriptor 22 | 23 | from zope.interface import implementer 24 | 25 | ########################################################################### 26 | # See qemu/util/event_notifier-posix.c for routine names and models; only 27 | # the ones used by ivshmem-server.c are recreated here. Also see 28 | # https://sgros-students.blogspot.com/2013/05/calling-eventfd-from-python.html 29 | 30 | from ctypes import cdll 31 | 32 | class IVSHMSG_Event_Notifier(object): # Probably overkill 33 | 34 | # /usr/include/x86_64-linux-gnu/bits/eventfd.h 35 | EFD_SEMAPHORE = 0o00000001 36 | EFD_CLOEXEC = 0o02000000 37 | EFD_NONBLOCK = 0o00004000 38 | 39 | _libc = cdll.LoadLibrary('libc.so.6') 40 | 41 | def __init__(self, init_val=0, active=False, valid_eventfd = -1): 42 | '''valid_eventfd is from client; server always makes a new one.''' 43 | self.cbdata = None 44 | if valid_eventfd >= 0: 45 | self.rfd = self.wfd = valid_eventfd 46 | return 47 | self.rfd = self.wfd = self._libc.eventfd( 48 | init_val, self.EFD_NONBLOCK | self.EFD_CLOEXEC) 49 | assert self.rfd >= 0, 'eventfd() failed' 50 | if active: 51 | self.incr() 52 | 53 | def incr(self, delta=1): 54 | '''Corresponds to set() in C code, this is more descriptive.''' 55 | delta = int(delta) 56 | assert delta > 0, 'delta must be positive' 57 | bval = struct.pack('Q', delta) 58 | while True: 59 | try: 60 | return os.write(self.wfd, bval) == len(bval) 61 | except InterruptedError as e: # handled interally at 3.5 62 | continue 63 | except OSError as e: 64 | if e.errno == errno.EINTR: 65 | continue 66 | if e.errno == errno.EAGAIN: # would block 67 | return False 68 | raise 69 | 70 | def ring(self): 71 | '''An alias that helps legibility of IVSHMSG code.''' 72 | self.incr() 73 | 74 | def reset(self): 75 | '''Without EFD_SEMAPHORE, reset if non-zero, else EAGAIN (NONBLOCK).''' 76 | while True: 77 | try: 78 | junk = os.read(self.rfd, 8) # reset 79 | if len(junk) == 8: 80 | return True, struct.unpack('Q', junk) 81 | return False, None 82 | except InterruptedError as e: # handled interally at 3.5 83 | continue 84 | except OSError as e: 85 | if e.errno == errno.EINTR: 86 | continue 87 | if e.errno == errno.EAGAIN: # would block 88 | return False, None 89 | raise 90 | 91 | def get_fd(self): # I'd love to hear this story... 92 | return self.rfd 93 | 94 | def cleanup(self): 95 | try: 96 | os.close(self.wfd) 97 | except Exception as e: 98 | pass 99 | self.rfd = self.wfd = -1 100 | 101 | 102 | def ivshmsg_event_notifier_list(list_or_count): 103 | '''Polymorphic. If list_or_count is an integer, create that many event 104 | objects with new fds. If it's a list of ints, assume they are fds 105 | and create a list of objects re-using those ints.''' 106 | if isinstance(list_or_count, int): 107 | return [ IVSHMSG_Event_Notifier() 108 | for _ in range(list_or_count) ] 109 | if isinstance(list_or_count, (list, tuple)): 110 | return [ IVSHMSG_Event_Notifier(valid_eventfd=fd) 111 | for fd in list_or_count ] 112 | 113 | 114 | ########################################################################### 115 | # https://stackoverflow.com/questions/28449455/integrating-hid-access-with-evdev-on-linux-with-python-twisted 116 | 117 | 118 | @implementer(IReadDescriptor) 119 | class EventfdReader(object): 120 | 121 | def __init__(self, eventobj, callback, cbdata): 122 | '''cbdata is usually "self" from the caller.''' 123 | assert isinstance(eventobj, IVSHMSG_Event_Notifier), 'Bad object' 124 | eventobj.cbdata = cbdata 125 | self.eventobj = eventobj 126 | self.eventobj.last_value = None 127 | self.callback = callback 128 | 129 | def fileno(self): 130 | return self.eventobj.get_fd() # Might as well use it 131 | 132 | def logPrefix(self): 133 | return 'ServerEvent@%d' % self.fileno() 134 | 135 | def doRead(self): 136 | fired, value = self.eventobj.reset() 137 | if fired: 138 | self.eventobj.last_value = value 139 | self.callback(self.eventobj) 140 | 141 | def connectionLost(self, reason): 142 | TIreactor.removeReader(self) # Paranoid? EAGAIN? Use destroy()? 143 | self.eventobj.cleanup() 144 | self.eventobj = None 145 | self.callback = None 146 | 147 | def start(self): 148 | '''Convenience, not in twisted classes.''' 149 | TIreactor.addReader(self) 150 | 151 | def destroy(self): 152 | '''Convenience, not in twisted classes.''' 153 | TIreactor.removeReader(self) 154 | self.loseConnection() 155 | 156 | -------------------------------------------------------------------------------- /ivshmsg_twisted/ivshmsg_mailbox.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | 3 | # The IVSHMEM protocol practiced by QEMU demands a memory-mappable file 4 | # descriptor as part of the initial exchange, so give it one. The mailbox 5 | # is a shared common area split into "slots". slot 0 is a global read-only 6 | # area populated by the server, then "nClients" worth of slots, and a final 7 | # entry "nClients + 1" is the server mailslot. This is contiguous because it 8 | # it aligns with how QEMU expects delivery of peer info, including the twisted 9 | # server (which is an extension beyond stock QEMU IVSHMSG protocol). 10 | # Right now a mailslot is 512 bytes: 128 bytes of metadata (currently about 11 | # 96 used), then 384 of message buffer. 12 | # Go for the max slots in the file to hardwire libvirt domain XML file size. 13 | # VM guest kernel modules read global data to understand the mailbox layout. 14 | 15 | # All numbers are unsigned of an appropriate size. All strings are multiples 16 | # of 32 (including the C terminating NULL) on 32-byte boundaries. Then it 17 | # all looks good in "od -Ad -c" and even better in "od -Ax -c -tu8 -tx8". 18 | 19 | import ctypes 20 | import mmap 21 | import os 22 | import struct 23 | import sys 24 | 25 | from os.path import stat as STAT # for constants 26 | from pdb import set_trace 27 | from time import sleep 28 | from time import time as NOW 29 | 30 | 31 | class IVSHMSG_MailGlobals(ctypes.Structure): 32 | _fields_ = [ # A magic ctypes class attribute. 33 | ('slotsize', ctypes.c_ulonglong), 34 | ('buf_offset', ctypes.c_ulonglong), 35 | ('nClients', ctypes.c_ulonglong), 36 | ('nEvents', ctypes.c_ulonglong), 37 | ('server_id', ctypes.c_ulonglong), 38 | ] 39 | 40 | 41 | class IVSHMSG_MailSlot(ctypes.Structure): 42 | # c_char_p is variable length so force fixed size fields. 43 | 44 | _strsize = 32 45 | _bufsize = 384 46 | 47 | _fields_ = [ # A magic ctypes class attribute. 48 | ('_nodename', ctypes.c_char * _strsize), 49 | ('_cclass', ctypes.c_char * _strsize), 50 | ('buflen', ctypes.c_ulonglong), 51 | ('peer_id', ctypes.c_ulonglong), 52 | ('last_responder', ctypes.c_ulonglong), 53 | ('peer_SID', ctypes.c_ulonglong), 54 | ('peer_CID', ctypes.c_ulonglong), 55 | ('pad', ctypes.c_ulonglong * 3), 56 | ('buf', ctypes.c_char * _bufsize) 57 | ] 58 | 59 | @property 60 | def nodename(self): 61 | return ctypes.string_at(self._nodename).decode() 62 | 63 | @property 64 | def cclass(self): 65 | return ctypes.string_at(self._cclass).decode() 66 | 67 | # This does not blank pad to the end, properly detects overrun, and 68 | # lays down a NUL properly UNLESS the entire space is filled. Help 69 | # keep the length down and always preserve that final NUL. 70 | 71 | @nodename.setter 72 | def nodename(self, instr): 73 | inbytes = instr.encode() 74 | assert self._strsize > len(inbytes), '"%s" too big' % instr 75 | self._nodename = inbytes 76 | 77 | @cclass.setter 78 | def cclass(self, instr): 79 | inbytes = instr.encode() 80 | assert self._strsize > len(inbytes), '"%s" too big' % instr 81 | self._cclass = inbytes 82 | 83 | 84 | class IVSHMSG_MailBox(object): 85 | 86 | # QEMU rules: file size (product of first two) must be a power of two. 87 | MAILBOX_MAX_SLOTS = 16 # Dummy + server leaves 14 actual clients 88 | MAILBOX_SLOTSIZE = 512 89 | FILESIZE = MAILBOX_MAX_SLOTS * MAILBOX_SLOTSIZE 90 | MS_BUF_off = 128 91 | MS_MAX_BUFLEN = 384 92 | assert MAILBOX_SLOTSIZE == MS_BUF_off + MS_MAX_BUFLEN, 'Big oops. Huge!' 93 | 94 | fd = None # There can be only one 95 | mm = None # Then I can access fill() from the class 96 | nClients = None 97 | nEvents = None 98 | server_id = None 99 | slots = None # 0 == MailGlobal, 1 - server_id == MailSlot 100 | 101 | #----------------------------------------------------------------------- 102 | # Slots[] array: Globals at offset 0 (slot 0; each slot (1 through 103 | # nClients holds a peer, and the server is always at end. 104 | 105 | @classmethod 106 | def _initialize_mailbox(cls, args): 107 | cls.nClients = args.nClients 108 | cls.nEvents = args.nEvents 109 | cls.server_id = args.server_id 110 | 111 | # mm[] is bytes and that involves copies and manual indexing. The 112 | # view is an overlay, especially when combined with ctype structures. 113 | cls.mm = mmap.mmap(cls.fd, 0) 114 | cls.view = memoryview(cls.mm) 115 | cls.slots = [ None, ] * cls.nEvents 116 | 117 | # Empty it. Simple code that's never too demanding on size, 118 | # default of 16 slots == now 32k. 119 | data = b'\0' * cls.FILESIZE 120 | cls.mm[0:len(data)] = data 121 | 122 | # Fill in the globals; used by ivshmsg.ko and the C struct globals. 123 | # It's at the start of the memory area so no indexing is needed. 124 | mbg = IVSHMSG_MailGlobals.from_buffer(cls.view) 125 | mbg.slotsize = cls.MAILBOX_SLOTSIZE 126 | mbg.buf_offset = cls.MS_BUF_off 127 | mbg.nClients = cls.nClients 128 | mbg.nEvents = cls.nEvents 129 | mbg.server_id = cls.server_id 130 | cls.slots[0] = mbg 131 | 132 | # Get a data structure overlay for each slot, then set the peer_id 133 | # as a sentinel for other code. Don't forget the server. 134 | 135 | for slot in range(1, cls.nEvents): 136 | cls.slots[slot] = IVSHMSG_MailSlot.from_buffer( 137 | cls.view[mbg.slotsize * slot : mbg.slotsize * (slot + 1)]) 138 | cls.slots[slot].peer_id = slot 139 | 140 | # Server's "hostname" and Base Component Class. Zero-padding occurs 141 | # because it was all zeroed out just above. 142 | name = 'Z-switch' if args.smart else 'Z-server' 143 | cls.slots[cls.server_id].nodename = name 144 | cls.slots[cls.server_id].cclass = 'FabricSwitch' 145 | 146 | #---------------------------------------------------------------------- 147 | # Polymorphic. Someday I'll learn about metaclasses. While initialized 148 | # as an instance, use of the whole file and individual slots is done 149 | # as class methods and attributes, so __init__ is a singleton. 150 | 151 | _beentheredonethat = False 152 | 153 | def __init__(self, args=None, fd=-1, client_id=-1): 154 | '''Server: args with command line stuff from command line. 155 | Client: starts with an fd and id read from AF_UNIX socket.''' 156 | cls = self.__class__ 157 | if cls._beentheredonethat: 158 | return 159 | cls._beentheredonethat = True 160 | 161 | if args is None: 162 | assert fd >= 0 and client_id > 0, 'Bad call, ump!' 163 | cls.fd = fd 164 | cls._init_mailslot(client_id) 165 | return 166 | assert fd == -1 and client_id == -1, 'Cannot assign fd/id to server' 167 | 168 | path = args.mailbox # Match previously written code 169 | gr_gid = -1 # Makes no change. Try Debian, CentOS, other 170 | for gr_name in ('libvirt-qemu', 'libvirt', 'libvirtd'): 171 | try: 172 | gr_gid = grp.getgrnam(gr_name).gr_gid 173 | break 174 | except Exception as e: 175 | pass 176 | 177 | if '/' not in path: 178 | path = '/dev/shm/' + path 179 | oldumask = os.umask(0) 180 | try: 181 | if not os.path.isfile(path): 182 | fd = os.open(path, os.O_RDWR | os.O_CREAT, mode=0o666) 183 | os.posix_fallocate(fd, 0, cls.FILESIZE) 184 | os.fchown(fd, -1, gr_gid) 185 | else: # Re-condition and re-use 186 | lstat = os.lstat(path) 187 | assert STAT.S_ISREG(lstat.st_mode), 'not a regular file' 188 | assert lstat.st_size >= cls.FILESIZE, \ 189 | 'existing size (%d) is < required (%d)' % ( 190 | lstat.st_size, cls.FILESIZE) 191 | if lstat.st_gid != gr_gid and gr_gid > 0: 192 | print('Changing %s to group %s' % (path, gr_name)) 193 | os.chown(path, -1, gr_gid) 194 | if lstat.st_mode & 0o660 != 0o660: # at least 195 | print('Changing %s to permissions 666' % path) 196 | os.chmod(path, 0o666) 197 | fd = os.open(path, os.O_RDWR) 198 | except Exception as e: 199 | raise RuntimeError('Problem with %s: %s' % (path, str(e))) 200 | 201 | os.umask(oldumask) 202 | 203 | cls.path = path # Final absolute path 204 | cls.fd = fd 205 | cls._initialize_mailbox(args) 206 | 207 | #---------------------------------------------------------------------- 208 | # Dig the mail and node name out of the slot for peer_id (1:1 mapping). 209 | # It's not so much (passively) receivng mail as it is actively getting. 210 | 211 | @classmethod 212 | def retrieve(cls, peer_id, asbytes=False, clear=True): 213 | '''Return the message.''' 214 | ms = cls.slots[peer_id] 215 | # This next test seems paranoid, but also validates that id != 0 216 | # (which would have grabbed the MailGlobals). Oh and it is self- 217 | # limiting past server_id. 218 | assert ms.peer_id == peer_id, '%d != %d: this is SO wrong' % ( 219 | ms.peer_id, peer_id) 220 | buf = ms.buf[:ms.buflen] 221 | 222 | # The message is copied so mark the mailslot length zero as handshake 223 | # to the requester that its mailbox has been emptied. 224 | 225 | if clear: 226 | ms.buflen = 0 227 | 228 | return buf if asbytes else buf.decode() 229 | 230 | #---------------------------------------------------------------------- 231 | # Post a message to the indicated mailbox slot but don't kick the 232 | # EventFD. First, this routine doesn't know about them and second, 233 | # keeping it a separate operation facilitates sender spoofing. 234 | 235 | @classmethod 236 | def fill(cls, sender_id, buf): 237 | if isinstance(buf, str): 238 | buf = buf.encode() 239 | assert isinstance(buf, bytes), 'buf must be string or bytes' 240 | buflen = len(buf) 241 | assert buflen < cls.MS_MAX_BUFLEN, 'Message too long' 242 | 243 | # The previous responder needs to clear the msglen to indicate it 244 | # has pulled the message out of the sender's mailbox. 245 | ms = cls.slots[sender_id] 246 | stop = NOW() + 1.05 247 | intime = True 248 | while NOW() < stop and ms.buflen: 249 | sleep(0.1) 250 | if NOW() >= stop: 251 | intime = False 252 | print('pseudo-HW not ready to receive timeout: now stomping') 253 | 254 | ms.buflen = buflen 255 | ms.buf = buf 256 | return intime 257 | 258 | #---------------------------------------------------------------------- 259 | # Called by Python client on graceful shutdowns, and always by server 260 | # when a peer dies. This is mostly for QEMU crashes so the nodename 261 | # is not reused when a QEMU restarts, before loading ivshmsg.ko. 262 | 263 | @classmethod 264 | def clear_mailslot(cls, id): 265 | cls.slots[id].nodename = '' 266 | cls.slots[id].cclass = '' 267 | cls.slots[id].peer_id = id 268 | 269 | @classmethod 270 | def active_ids(cls): 271 | return sorted([ 272 | slot.peer_id for slot in cls.slots[1:cls.server_id + 1] if 273 | slot.nodename]) 274 | 275 | #---------------------------------------------------------------------- 276 | # Called only by client. mmap() the file and retrieve globals. 277 | 278 | @classmethod 279 | def _init_mailslot(cls, id): 280 | if cls.mm is None: 281 | # First time has some extra setup, not all vars need to be kept. 282 | buf = os.fstat(cls.fd) 283 | assert STAT.S_ISREG(buf.st_mode), 'Mailbox FD is not a regular file' 284 | cls.mm = mmap.mmap(cls.fd, 0) 285 | view = memoryview(cls.mm) 286 | mbg = IVSHMSG_MailGlobals.from_buffer(view) 287 | # Convenience 288 | cls.nClients = mbg.nClients 289 | cls.nEvents = mbg.nEvents 290 | cls.server_id = mbg.server_id 291 | 292 | # Create all the peer data structures now as it simplifies 293 | # connection logic removes interplay from twisted_client.py. 294 | cls.slots = [ None, ] * cls.nEvents 295 | cls.slots[0] = mbg 296 | for slot in range(1, cls.nEvents): 297 | cls.slots[slot] = IVSHMSG_MailSlot.from_buffer( 298 | view[mbg.slotsize * slot : mbg.slotsize * (slot + 1)]) 299 | assert cls.slots[slot].peer_id == slot, 'What happened?' 300 | 301 | if id > cls.server_id: # Probably a test run of twisted_restapi 302 | return 303 | assert cls.slots[id].peer_id == id, 'What happened?' 304 | cls.clear_mailslot(id) 305 | 306 | #---------------------------------------------------------------------- 307 | # Typing conveniences. No setters, use the full expression. 308 | 309 | @classmethod 310 | def nodename(cls, index): 311 | return cls.slots[index].nodename 312 | 313 | @classmethod 314 | def cclass(cls, index): 315 | return cls.slots[index].cclass 316 | -------------------------------------------------------------------------------- /ivshmsg_twisted/ivshmsg_sendrecv.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | 3 | # This work is licensed under the terms of the GNU GPL, version 2 or 4 | # (at your option) any later version. See the LICENSE file in the 5 | # top-level directory. 6 | 7 | # Rocky Craig 8 | 9 | # Routine names here mirror those in qemu/contrib/ivshmem-[client|server]. 10 | # The IVSHMEM communications protocol (now christened IVSHMSG) is based on 11 | # 8-byte integers and an optional 4-byte file descriptor. Twisted 12 | # transport.sendFileDescriptor gets the packing sizes wrong for IVSHMSG 13 | # so do it right. 14 | 15 | import socket 16 | import struct 17 | import sys 18 | 19 | ########################################################################### 20 | 21 | 22 | def ivshmsg_send_one_msg(thesocket, data, fd=None): 23 | # On the far side, if no fd is received from here, a helper routine 24 | # returns fd == -1 which is checked in various places. 25 | data = int(data) 26 | bdata_iovec = [ struct.pack('q', data) ] # One item in the vector 27 | if fd is None: 28 | cmsg = [] # Message array of none, defaults to fd == -1 on far side. 29 | else: 30 | fd = int(fd) 31 | cmsg = [ # Message array of one. 32 | (socket.SOL_SOCKET, socket.SCM_RIGHTS, struct.pack('i', fd)) 33 | ] 34 | try: 35 | ret = thesocket.sendmsg(bdata_iovec, cmsg) 36 | except Exception as e: 37 | ret = -1 38 | return ret == 8 39 | 40 | 41 | def ivshmsg_recv_one_msg(thesocket): 42 | print(thesocket.recvmsg(64, 64)) 43 | -------------------------------------------------------------------------------- /ivshmsg_twisted/twisted_client.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | 3 | # This work is licensed under the terms of the GNU GPL, version 2 or 4 | # (at your option) any later version. See the LICENSE file in the 5 | # top-level directory. 6 | 7 | # Rocky Craig 8 | 9 | import argparse 10 | import grp 11 | import mmap 12 | import struct 13 | import sys 14 | 15 | from collections import OrderedDict 16 | 17 | from twisted.internet import stdio 18 | from twisted.internet import error as TIError 19 | from twisted.internet import reactor as TIreactor 20 | 21 | from twisted.internet.endpoints import UNIXClientEndpoint 22 | 23 | from twisted.internet.interfaces import IFileDescriptorReceiver 24 | 25 | from twisted.internet.protocol import ClientFactory as TIPClientFactory 26 | from twisted.internet.protocol import Protocol as TIPProtocol 27 | 28 | from zope.interface import implementer 29 | 30 | try: 31 | from commander import Commander 32 | from ivshmsg_mailbox import IVSHMSG_MailBox as MB 33 | from famez_requests import handle_request, send_payload, ResponseObject 34 | from ivshmsg_eventfd import ivshmsg_event_notifier_list, EventfdReader 35 | except ImportError as e: 36 | from .commander import Commander 37 | from .ivshmsg_mailbox import IVSHMSG_MailBox as MB 38 | from .famez_requests import handle_request, send_payload, ResponseObject 39 | from .ivshmsg_eventfd import ivshmsg_event_notifier_list, EventfdReader 40 | 41 | ########################################################################### 42 | # See qemu/docs/specs/ivshmem-spec.txt::Client-Server protocol and 43 | # qemu/contrib/ivshmem-server.c::ivshmem_server_handle_new_conn() calling 44 | # qemu/contrib/ivshmem-server.c::ivshmem_server_send_initial_info(), then 45 | # qemu/contrib/ivshmem-client.c::ivshmem_client_connect() 46 | 47 | # The UNIX transport in the middle of this, at 48 | # /usr/lib/python3/dist-packages/twisted/internet/unix.py line 174 49 | # will properly glean a file descriptor if present. There must be 50 | # real data to go with this ancillary data. Then, if this protocol 51 | # is recognized as implementing IFileDescriptorReceiver, it will FIRST 52 | # call fileDescriptorReceived before dataReceived. So for the initial 53 | # info exchange, version and my (new) id are put out without an fd, 54 | # then a -1 is put out with the mailbox fd. What triggers here is 55 | # one fileDescriptorReceived, THEN a dataReceived of thre quad words. 56 | # Then it pingpongs evenly between an fd and a single quadword for 57 | # each grouping. 58 | 59 | 60 | @implementer(IFileDescriptorReceiver) # Energizes fileDescriptorReceived 61 | class ProtocolIVSHMSGClient(TIPProtocol): 62 | 63 | CLIENT_IVSHMEM_PROTOCOL_VERSION = 0 64 | 65 | args = None 66 | id2fd_list = OrderedDict() # Sent to me for each peer 67 | id2EN_list = OrderedDict() # Generated from fd_list 68 | 69 | def __init__(self, cmdlineargs): 70 | try: # twisted causes blindness 71 | if self.args is None: 72 | cls = self.__class__ 73 | cls.args = cmdlineargs 74 | cls.logmsg = print 75 | cls.logerr = print 76 | cls.stdtrace = sys.stdout 77 | cls.verbose = cls.args.verbose 78 | 79 | # The state machine major decisions about the semantics of blocks 80 | # of data have one predicate. initial_pass is an extra guard. 81 | self.id = None # Until initial info; state machine key 82 | self._latest_fd = None 83 | self.initial_pass = True 84 | 85 | # Other stuff 86 | self.quitting = False 87 | self.isPFM = False 88 | self.linkattrs = { 'State': 'up' } 89 | self.peerattrs = {} 90 | self.SID0 = 0 91 | self.CID0 = 0 92 | self.nodename = None 93 | self.cclass = None 94 | self.afterACK = [] 95 | except Exception as e: 96 | print('__init__() failed: %s' % str(e)) 97 | # ...and any number of attribute references will fail soon 98 | 99 | @property # For Commander prompt 100 | def promptname(self): 101 | return self.nodename 102 | 103 | @staticmethod 104 | def parse_target(caller_id, instr): 105 | '''Return a list even for one item for consistency with keywords 106 | ALL and OTHERS.''' 107 | try: 108 | tmp = (int(instr), ) 109 | if 1 <= tmp[0] <= MB.server_id: 110 | return tmp 111 | except TypeError as e: 112 | return None 113 | except ValueError as e: 114 | if instr.lower()[-6:] in ('server', 'switch'): 115 | return (MB.server_id,) 116 | active_ids = MB.active_ids() 117 | for id in active_ids: 118 | if MB.slots[id].nodename == instr: 119 | return (id, ) 120 | if instr.lower() == 'all': # Includes caller_id 121 | return active_ids 122 | if instr.lower() == 'others': 123 | return active_ids.remove(caller_id) 124 | return None 125 | 126 | def place_and_go(self, dest, msg, src=None, reset_tracker=True): 127 | '''Yes, reset_tracker defaults to True here.''' 128 | dest_indices = self.parse_target(self.id, dest) 129 | if src is None: 130 | src_indices = (self.id,) 131 | else: 132 | src_indices = self.parse_target(self.id, src) 133 | if self.verbose > 1: 134 | print('P&G dest %s=%s src %s=%s' % 135 | (dest, dest_indices, src, src_indices)) 136 | assert src_indices, 'missing or unknown source(s)' 137 | assert dest_indices, 'missing or unknown destination(s)' 138 | for S in src_indices: 139 | for D in dest_indices: 140 | if self.verbose > 1: 141 | print('P&G(%s, "%s", %s)' % (D, msg, S)) 142 | try: 143 | # First get the list for dest, then the src ("from me") 144 | # doorbell EN. 145 | doorbell = self.id2EN_list[D][S] 146 | 147 | # This repeat-loads the source mailslot D times per S 148 | # but I don't care. 149 | send_payload(msg, S, doorbell, reset_tracker=reset_tracker) 150 | except KeyError as e: 151 | print('No such peer id', str(e)) 152 | continue 153 | except Exception as e: 154 | print('place_and_go(%s, "%s", %s) failed: %s' % 155 | (D, msg, S, str(e))) 156 | return 157 | 158 | def fileDescriptorReceived(self, latest_fd): 159 | assert self._latest_fd is None, 'Latest fd has not been consumed' 160 | self._latest_fd = latest_fd # See the next property 161 | 162 | @property 163 | def nodename(self): 164 | return self._nodename 165 | 166 | @nodename.setter 167 | def nodename(self, name): 168 | self._nodename = name 169 | if name: 170 | MB.slots[self.id].nodename = name 171 | 172 | @property 173 | def cclass(self): 174 | return self._cclass 175 | 176 | @cclass.setter 177 | def cclass(self, name): 178 | self._cclass = name 179 | if name: 180 | MB.slots[self.id].cclass = name 181 | 182 | @property 183 | def latest_fd(self): 184 | '''This is NOT idempotent!''' 185 | tmp = self._latest_fd 186 | self._latest_fd = None 187 | return tmp 188 | 189 | def retrieve_initial_info(self, data): 190 | # 3 longwords: protocol version w/o FD, my (new) ID w/o FD, 191 | # and then a -1 with the FD of the IVSHMEM file which is 192 | # delivered before this. 193 | assert self.initial_pass, 'Internal state error (1)' 194 | assert len(data) == 24, 'Initial data needs three quadwords' 195 | 196 | # Enough idiot checks. 197 | mailbox_fd = self.latest_fd 198 | version, tmpid, minusone = struct.unpack('qqq', data) 199 | assert version == self.CLIENT_IVSHMEM_PROTOCOL_VERSION, \ 200 | 'Unxpected protocol version %d' % version 201 | assert minusone == -1, \ 202 | 'Expected -1 with mailbox fd, got %d' % minusone 203 | 204 | # Initialize my mailbox slot. Get other parameters from the 205 | # globals because the IVSHMSG protocol doesn't allow values 206 | # beyond the intial three. The constructor does some work then 207 | # returns a few attributes pulled out of the globals, but work 208 | # is only actually done on the first call. It's a singleton with 209 | # class variables so there's no reason to keep the instance. 210 | # SI is the object passed to the event callback so flesh it out. 211 | MB(fd=mailbox_fd, client_id=tmpid) 212 | 213 | # Wait for initialized mailbox to finally (re)assign the sentinel. 214 | self.id = tmpid 215 | self.nodename = 'z%02d' % self.id 216 | self.cclass = 'Debugger' 217 | print('This ID = %2d (%s)' % (self.id, self.nodename)) 218 | 219 | # Called multiple times so keep state info about previous calls. 220 | def dataReceived(self, data): 221 | if self.id is None: 222 | self.retrieve_initial_info(data) 223 | return # But I'll be right back :-) 224 | 225 | # Now into the stream of pairs. Unless it's 226 | # a single which is a disconnect notification. 227 | latest_fd = self.latest_fd 228 | assert len(data) == 8, 'Expecting a signed long long' 229 | thisbatch = struct.unpack('q', data)[0] 230 | if self.verbose > 1: 231 | print('Just got index %s, fd %s' % (thisbatch, latest_fd)) 232 | assert thisbatch >= 0, 'Latest data is negative number' 233 | 234 | if latest_fd is None: # "thisbatch" is a disconnect notification 235 | print('%s (%d) has left the building' % 236 | (MB.slots[thisbatch].nodename, thisbatch)) 237 | for collection in (self.id2EN_list, self.id2fd_list): 238 | try: 239 | del collection[thisbatch] 240 | except Exception as e: 241 | pass 242 | return 243 | 244 | # Collect all the fds for this batch, max batch length == nEvents 245 | # (the dummy slot 0, nClients, and the server). This batch may one 246 | # one of many, or one of one for two different reasons: 247 | # 1. 1 of 1: Another new client has joined the cluster. This batch 248 | # is for that new client. 249 | # 2. First contact by this client instance: There will be one batch 250 | # for each existing peer, terminated by a batch of "my" fds. 251 | # That last batch is tagged with "my" self.id AND must come last 252 | # (an assumption of QEMU). 253 | # A. if using the stock QEMU "ivshmem_server" there will NOT be 254 | # a batch for the server itself. Thus it's possible to receive 255 | # only one batch during first_contact, if this is the only peer. 256 | # B. if using twisted_server, on first contact you will always get 257 | # at least the server batch first, so the minimum batch run 258 | # length is two. 259 | # Keep track of all batches then post-process when they stop coming. 260 | 261 | # Am I starting the final batch of my initial connection? This 262 | # paranoia check is on MB and its assignment during _initial_info(). 263 | if thisbatch == self.id and not MB.server_id: 264 | assert MB.server_id == self.prevbatch, 'Then dont assign it' 265 | self.prevbatch = thisbatch # corner case where I am first peer 266 | 267 | # Just save the eventfd now, generate objects later. 268 | try: 269 | tmp = len(self.id2fd_list[thisbatch]) 270 | assert tmp <= MB.server_id, 'fd list is too long' 271 | if tmp == MB.nEvents: # Beginning of client reconnect 272 | assert thisbatch != self.id, \ 273 | 'Updating MY eventfds??? off-by-one' 274 | raise KeyError('Forced update') 275 | self.id2fd_list[thisbatch].append(latest_fd) # order matters 276 | except KeyError as e: 277 | self.id2fd_list[thisbatch] = [latest_fd, ] # first one 278 | 279 | if self.verbose > 1: 280 | print('fd list is now %s' % str(self.id2fd_list.keys())) 281 | for id, eventfds in self.id2fd_list.items(): 282 | print(id, eventfds) 283 | 284 | # Assumes all vector lists are the same length. 285 | batchneeds = MB.nEvents - len(self.id2fd_list[thisbatch]) 286 | if batchneeds > 0: 287 | if self.verbose > 1: 288 | print('Batch for peer id %d expecting %d more fds...\n' % 289 | (thisbatch, batchneeds)) 290 | return 291 | 292 | # Batch is complete, it's either 293 | # 1. one of many, so more to come in this initial contact 294 | # (unless it was me) 295 | # 2. Just a single batch, ie, a new peer 296 | if self.initial_pass: 297 | if thisbatch != self.id: 298 | if self.verbose > 1: 299 | print('%d is not the final batch' % thisbatch) 300 | return 301 | print('Active client list now complete') 302 | else: 303 | print('New client %d complete' % thisbatch) 304 | 305 | # Generate event notifiers from each (new) fd_list for signalling 306 | # to other peers. 307 | for id in self.id2fd_list: # Triggers message pickup 308 | if id not in self.id2EN_list: # already processed? 309 | self.id2EN_list[id] = ivshmsg_event_notifier_list( 310 | self.id2fd_list[id]) 311 | 312 | if not self.initial_pass: # It was just one additional peer 313 | return 314 | 315 | # Finally arm my incoming events and announce readiness. 316 | assert thisbatch == self.id, 'Cuz it\'s not paranoid if you catch it' 317 | for i, N in enumerate(self.id2EN_list[self.id]): 318 | N.num = i 319 | tmp = EventfdReader(N, self.ClientCallback, self) 320 | tmp.start() 321 | print('Ready player %s' % self.nodename) 322 | self.place_and_go('server', 'Link CTL Peer-Attribute') 323 | self.initial_pass = False 324 | 325 | def connectionMade(self): 326 | if self.verbose: 327 | print('Connection made on fd', self.transport.fileno()) 328 | 329 | def connectionLost(self, reason): 330 | print(reason.value) 331 | if reason.check(TIError.ConnectionDone) is None: # Dirty 332 | print('Client was probably interrupted or killed.') 333 | else: 334 | if self.quitting: 335 | print('Last interactive command was "quit".') 336 | else: 337 | print('The server was probably shut down.') 338 | MB.clear_mailslot(self.id) # In particular, nodename 339 | if TIreactor.running: # Stopped elsewhere on SIGINT 340 | TIreactor.stop() 341 | 342 | # The cbdata is precisely the object which can be used for the response. 343 | # In other words, it's directly "me", with "my" identity data. 344 | @staticmethod 345 | def ClientCallback(vectorobj): 346 | requester_id = vectorobj.num 347 | requester_name = MB.nodename(requester_id) 348 | request = MB.retrieve(requester_id) 349 | requester_obj = vectorobj.cbdata 350 | # print('Raw Req ID = %d\n%s' % (requester_id, vars(requester_obj))) 351 | 352 | # [dest][src] 353 | ro = ResponseObject( 354 | this=requester_obj, # has CID0, SID0, and cclass 355 | proxy=None, # I don't manage ever (for now) 356 | from_id=requester_obj.id, 357 | to_doorbell=requester_obj.id2EN_list[requester_id][requester_obj.id], 358 | logmsg=requester_obj.logmsg, 359 | stdtrace=requester_obj.stdtrace, 360 | verbose=requester_obj.verbose, 361 | ) 362 | ret = handle_request(request, requester_name, ro) 363 | 364 | #---------------------------------------------------------------------- 365 | # Command line parsing. 366 | 367 | 368 | def doCommand(self, cmd, args): 369 | cmd = cmd.lower() 370 | if cmd in ('p', 'ping', 's', 'send'): 371 | if cmd.startswith('p'): 372 | assert len(args) == 1, 'Missing dest' 373 | cmd = 'send' 374 | args.append('ping') # Message payload 375 | else: 376 | assert len(args) >= 1, 'Missing dest' 377 | dest = args.pop(0) 378 | msg = ' '.join(args) # Empty list -> empty string 379 | self.place_and_go(dest, msg) 380 | return True 381 | 382 | if cmd in ('sp', 'spoof'): # Like send but specify a src 383 | assert len(args) >= 2, 'Missing src and/or dest' 384 | src = args.pop(0) 385 | dest = args.pop(0) 386 | msg = ' '.join(args) # Empty list -> empty string 387 | self.place_and_go(dest, msg, src) 388 | return True 389 | 390 | if cmd in ('d', 'dump'): # Include the server 391 | if self.verbose > 1: 392 | print('Peer list keys (%d max):' % (MB.nClients + 1)) 393 | print('\t%s' % sorted(self.id2EN_list.keys())) 394 | 395 | print('\nActor event fds:') 396 | for key in sorted(self.id2fd_list.keys()): 397 | print('\t%2d %s' % (key, self.id2fd_list[key])) 398 | print() 399 | 400 | print('Client node/host names:') 401 | for key in sorted(self.id2fd_list.keys()): 402 | print('\t%2d %s' % (key, MB.slots[key].nodename)) 403 | 404 | print('\nMy CID0:SID0 = %d:%d' % (self.CID0, self.SID0)) 405 | print('Link attributes:\n', self.linkattrs) 406 | print('Peer attributes:\n', self.peerattrs) 407 | 408 | return True 409 | 410 | if cmd in ('h', 'help') or '?' in cmd: 411 | print('dest/src can be integer, hostname, or "server"\n') 412 | print('h[elp]\n\tThis message') 413 | print('l[ink]\n\tLink commands (CTL and RFC)') 414 | print('p[ing] dest\n\tShorthand for "send dest ping"') 415 | print('q[uit]\n\tJust do it') 416 | print('r[fc]\n\tSend "Link RFC ..." to the server') 417 | print('s[end] dest [text...]\n\tSend text from this client') 418 | print('sp[oof] src dest [text...]\n\tLike send but fake the src') 419 | print('w[ho]\n\tList all peers') 420 | return True 421 | 422 | if cmd in ('w', 'who'): 423 | print('\nThis ID = %2d (%s)' % (self.id, self.nodename)) 424 | for id in self.id2fd_list.keys(): 425 | if id == self.id: 426 | continue 427 | print('Peer ID = %2d (%s)' % (id, MB.slots[id].nodename)) 428 | return True 429 | 430 | if cmd in ('l', 'link'): 431 | assert len(args) >= 1, 'Missing directive' 432 | msg = 'Link %s' % ' '.join(args) 433 | self.place_and_go('server', msg) 434 | return True 435 | 436 | if cmd in ('r', 'rfc'): 437 | msg = 'Link RFC TTC=27us' 438 | self.place_and_go('server', msg) 439 | return True 440 | 441 | if cmd in ('q', 'quit'): 442 | self.quitting = True 443 | self.transport.loseConnection() 444 | return False 445 | 446 | print('Unrecognized command "%s", try "help"' % cmd) 447 | return True 448 | 449 | ########################################################################### 450 | # Normally the Endpoint and listen() call is done explicitly, 451 | # interwoven with passing this constructor. This approach hides 452 | # all the twisted things in this module. 453 | 454 | 455 | class FactoryIVSHMSGClient(TIPClientFactory): 456 | 457 | _required_arg_defaults = { 458 | 'socketpath': '/tmp/ivshmsg_socket', 459 | 'verbose': 0, 460 | } 461 | 462 | def __init__(self, args=None): 463 | '''Args must be an object with the following attributes: 464 | socketpath, verbose 465 | Suitable defaults will be supplied.''' 466 | 467 | # Pass command line args to ProtocolIVSHMSG, then open logging. 468 | if args is None: 469 | args = argparse.Namespace() 470 | for arg, default in self._required_arg_defaults.items(): 471 | setattr(args, arg, getattr(args, arg, default)) 472 | 473 | self.args = args 474 | 475 | # checkPID looks for .lock which the server sets up 476 | # as a symlink to file named 477 | E = UNIXClientEndpoint( 478 | TIreactor, 479 | args.socketpath, 480 | timeout=1, 481 | checkPID=False) 482 | E.connect(self) 483 | 484 | def buildProtocol(self, addr): 485 | if self.args.verbose > 1: 486 | print('buildProtocol', addr.name) 487 | protobj = ProtocolIVSHMSGClient(self.args) 488 | Commander(protobj) 489 | return protobj 490 | 491 | def startedConnecting(self, connector): 492 | print('Started connecting') 493 | 494 | def clientConnectionFailed(self, connector, reason): 495 | print('Failed connection:', str(reason)) 496 | 497 | def clientConnectionLost(self, connector, reason): 498 | print('Lost connection:', str(reason)) 499 | 500 | def run(self): 501 | TIreactor.run() 502 | 503 | if __name__ == '__main__': 504 | from pdb import set_trace 505 | set_trace() 506 | pass 507 | -------------------------------------------------------------------------------- /ivshmsg_twisted/twisted_restapi.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | 3 | # https://gist.github.com/berlincount/409a631d49c3be210abd 4 | 5 | import attr 6 | import json 7 | import os 8 | import struct 9 | import sys 10 | 11 | from collections import defaultdict, OrderedDict 12 | from pdb import set_trace 13 | from pprint import pformat, pprint 14 | 15 | from klein import Klein # Uses default reactor... 16 | 17 | from twisted.internet import reactor as TIreactor # ...like twisted_server 18 | from twisted.web import server as TWserver 19 | 20 | 21 | class MailBoxReSTAPI(object): 22 | 23 | N = attr.make_class('Nodes', []) 24 | L = attr.make_class('Links', []) 25 | 26 | app = Klein() # Now its decorators can be used on methods below 27 | isLeaf = True # TWserver seems to need this 28 | 29 | mb = None 30 | mm = None 31 | nClients = None 32 | nEvents = None 33 | server_id = None 34 | nodes = None 35 | 36 | @classmethod 37 | def mb2dict(cls): 38 | thedict = OrderedDict(( 39 | ('nClients', cls.nClients), 40 | ('server_ivshmsg_id', cls.server_ivshmsg_id), 41 | )) 42 | 43 | # The D3 Javascript framework refers to a node's name as its "id". 44 | for ivshmsg_id in range(1, cls.nEvents): 45 | this = cls.nodes[ivshmsg_id] 46 | this.ivshmsg_id = ivshmsg_id 47 | this.id = cls.mb.slots[ivshmsg_id].nodename 48 | this.cclass = cls.mb.slots[ivshmsg_id].cclass 49 | this.hardware = cls.cclass_to_hardware_type(this.cclass) 50 | 51 | # Since they all connect to the one switch (no P2P) 52 | this.group = 'port_%s' % ivshmsg_id 53 | 54 | # this.CID = 0 # Later 55 | # this.SID = 0 56 | # this.TXpackets = 0 57 | # this.RXpackets = 0 58 | # this.port = 0 59 | thedict['nodes'] = [ vars(n) for n in cls.nodes[1:] if n.id ] 60 | 61 | links = [] 62 | server_id = cls.nodes[cls.server_ivshmsg_id].id # a string 63 | for node in thedict['nodes']: 64 | id = node['id'] # also a string 65 | if id and id != server_id: 66 | links.append({'source': id, 'target': server_id}) 67 | thedict['links'] = links 68 | return thedict 69 | 70 | def cclass_to_hardware_type(cclass_name): 71 | lower = cclass_name.lower() 72 | for hw in ('qemu', 'debugger', 'adapter', 'switch'): 73 | if hw in lower: 74 | return hw 75 | return lower 76 | 77 | @app.route('/system') 78 | def get_system(self, request): 79 | thedict = self.mb2dict() 80 | 81 | # Twisted "fixes" the case of headers and uses bytearrays. 82 | reqhdrs = dict(request.requestHeaders.getAllRawHeaders()) 83 | request.setHeader('Access-Control-Allow-Origin', '*') 84 | 85 | return json.dumps(thedict) 86 | if b'Apiversion' in reqhdrs: 87 | return json.dumps(thedict) 88 | return('
%s
' % pformat(dict(thedict))) 89 | 90 | @app.route('/') 91 | def home(self, request): 92 | # print('Received "%s"' % request.uri.decode(), file=sys.stderr) 93 | reqhdrs = dict(request.requestHeaders.getAllRawHeaders()) 94 | 95 | return '
\n%s\nUse /system\n
' % '\n'.join( 96 | sorted([k.decode() for k in reqhdrs.keys()])) 97 | 98 | # Must come after all Klein dependencies and @decorators 99 | def __init__(self, already_initialized_IVSHMSG_mailbox, port=1991): 100 | cls = self.__class__ 101 | if cls.mb is not None: 102 | return 103 | cls.mb = already_initialized_IVSHMSG_mailbox 104 | cls.nClients = cls.mb.nClients 105 | cls.nEvents = cls.mb.nEvents 106 | cls.server_ivshmsg_id = cls.mb.server_id # see mb2dict 107 | # Clients/ports are enumerated 1-nClients inclusive 108 | cls.nodes = [ cls.N() for _ in range(cls.nEvents) ] 109 | 110 | # Instead of this.app.run(), break it open and wait for 111 | # twister_server.py to finally invoke TIreactor.run() as all these 112 | # things use the default reactor. Note that self.app was assigned 113 | # durng the class-level scan/eval of this source file. See also 114 | # /usr/lib/python3/dist-packages/klein/app.py::run() 115 | s = TWserver.Site(self.app.resource()) 116 | TIreactor.listenTCP(port, s) 117 | 118 | 119 | if __name__ == '__main__': 120 | 121 | from twisted.python import log as TPlog # Deprecated 122 | 123 | from ivshmsg_mailbox import IVSHMSG_MailBox 124 | 125 | # These things are done explicitly in twisted_server.py 126 | fname = '/dev/shm/ivshmsg_mailbox' if len(sys.argv) < 2 else sys.argv[1] 127 | if not fname or fname[0] == '-': 128 | raise SystemExit('usage: %s [ /path/to/mailbox ]' % sys.argv[0]) 129 | print('Opening', fname) 130 | fd = os.open(fname, os.O_RDWR) 131 | mb = IVSHMSG_MailBox(fd=fd, client_id=99) 132 | tmp = MailBoxReSTAPI(mb) 133 | 134 | # This is done explicitly in twisted_server.py 135 | TPlog.startLogging(sys.stdout, setStdout=False) 136 | 137 | # This is done implicitly after protocol registration in full app. 138 | TIreactor.run() 139 | -------------------------------------------------------------------------------- /ivshmsg_twisted/twisted_server.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | 3 | # This work is licensed under the terms of the GNU GPL, version 2 or 4 | # (at your option) any later version. See the LICENSE file in the 5 | # top-level directory. 6 | 7 | # Rocky Craig 8 | 9 | import argparse 10 | import attr 11 | import functools 12 | import grp 13 | import mmap 14 | import os 15 | import random 16 | import struct 17 | import sys 18 | import time 19 | 20 | from collections import OrderedDict 21 | from pprint import pprint 22 | 23 | # While deprecated, it has the best examples and is the only thing I 24 | # could get working. twisted.logger.Logger() is the new way. 25 | from twisted.python import log as TPlog 26 | from twisted.python.logfile import DailyLogFile 27 | 28 | from twisted.internet import error as TIError 29 | from twisted.internet import reactor as TIreactor 30 | 31 | from twisted.internet.endpoints import UNIXServerEndpoint 32 | 33 | from twisted.internet.protocol import ServerFactory as TIPServerFactory 34 | from twisted.internet.protocol import Protocol as TIPProtocol 35 | 36 | try: 37 | from commander import Commander 38 | from ivshmsg_mailbox import IVSHMSG_MailBox as MB 39 | from famez_requests import handle_request, send_payload, ResponseObject 40 | from ivshmsg_eventfd import ivshmsg_event_notifier_list, EventfdReader 41 | from ivshmsg_sendrecv import ivshmsg_send_one_msg 42 | from twisted_restapi import MailBoxReSTAPI 43 | except ImportError as e: 44 | from .commander import Commander 45 | from .ivshmsg_mailbox import IVSHMSG_MailBox as MB 46 | from .famez_requests import handle_request, send_payload, ResponseObject 47 | from .ivshmsg_eventfd import ivshmsg_event_notifier_list, EventfdReader 48 | from .ivshmsg_sendrecv import ivshmsg_send_one_msg 49 | from .twisted_restapi import MailBoxReSTAPI 50 | 51 | # Don't use peer ID 0, certain docs imply it's reserved. Use its mailslot 52 | # as global data storage, primarily the server command-line arguments. 53 | 54 | IVSHMSG_LOWEST_ID = 1 55 | 56 | PRINT = functools.partial(print, file=sys.stderr) 57 | PPRINT = functools.partial(pprint, stream=sys.stderr) 58 | 59 | ########################################################################### 60 | # Broken; need to get smarter to find actual module loggers. 61 | # See the section below with "/dev/null" with the bandaid fix. 62 | 63 | import logging 64 | 65 | def shutdown_http_logging(): 66 | for module in ('urllib3', 'requests', 'asdfjkl'): 67 | logger = logging.getLogger(module) 68 | logger.addHandler(logging.NullHandler()) 69 | logger.setLevel(logging.CRITICAL) 70 | logger.disabled = True 71 | logger.propagate = False 72 | 73 | ########################################################################### 74 | # See qemu/docs/specs/ivshmem-spec.txt::Client-Server protocol and 75 | # qemu/contrib/ivshmem-server.c::ivshmem_server_handle_new_conn() calling 76 | # qemu/contrib/ivshmem-server.c::ivshmem_server_send_initial_info(), then 77 | # qemu/contrib/ivshmem-client.c::ivshmem_client_connect() 78 | 79 | 80 | class ProtocolIVSHMSGServer(TIPProtocol): 81 | 82 | SERVER_IVSHMEM_PROTOCOL_VERSION = 0 83 | 84 | SI = None # Server Instance, contrived to be "me" not a peer 85 | 86 | def __init__(self, factory, args=None): 87 | '''"self" is a new client connection, not "me" the server. As such 88 | it is a proxy object for the other end of each switch "port". 89 | args is NOT passed on instantiation via connection.''' 90 | assert isinstance(factory, TIPServerFactory), 'arg0 not my Factory' 91 | shutdown_http_logging() 92 | self.isPFM = False 93 | 94 | # Am I one of many peer proxies? 95 | if self.SI is not None and args is None: 96 | self.create_new_peer_id() 97 | # Python client will quickly fix this. QEMU VM will eventually 98 | # modprobe, etc. 99 | self.peerattrs = { 100 | 'CID0': '0', 101 | 'SID0': '0', 102 | 'cclass': 'Driverless QEMU' 103 | } 104 | MB.slots[self.id].cclass = self.peerattrs['cclass'] 105 | return 106 | 107 | # This instance is voodoo from the first manual kick. Originally 108 | # the server had "other" tracking but now it's a "reserved instance" 109 | # which is used as the callback for events (ie, incoming mailslot 110 | # doorbell for each active peer proxy). 111 | 112 | cls = self.__class__ 113 | cls.SI = self # Reserve it and flesh it out. 114 | cls.verbose = args.verbose 115 | 116 | self.id = args.server_id 117 | assert self.id == MB.server_id, 'Server ID mismatch' 118 | self.quitting = False 119 | self.cclass = MB.cclass(self.id) 120 | self.logmsg = args.logmsg 121 | self.logerr = args.logerr 122 | self.stdtrace = sys.stderr 123 | self.smart = args.smart 124 | self.clients = OrderedDict() # Order probably not necessary 125 | self.recycled = {} if args.recycle else None 126 | 127 | # For the ResponseObject/request(). 128 | if self.smart: 129 | self.default_SID = 27 130 | self.SID0 = self.default_SID 131 | self.CID0 = self.id * 100 132 | self.isPFM = True 133 | else: 134 | self.default_SID = 0 135 | self.SID0 = 0 136 | self.CID0 = 0 137 | 138 | # Non-standard addition to IVSHMEM server role: this server can be 139 | # interrupted and messaged to particpate in client activity. 140 | # This variable will get looped even if it's empty (silent mode). 141 | # Usually create eventfds for receiving messages in IVSHMSG and 142 | # set up a callback. This arming is not a race condition as any 143 | # peer for which this is destined has not yet been "listened/heard". 144 | 145 | self.EN_list = [] 146 | if not args.silent: 147 | self.EN_list = ivshmsg_event_notifier_list(MB.nEvents) 148 | # The actual client doing the sending needs to be fished out 149 | # via its "num" vector. 150 | for i, EN in enumerate(self.EN_list): 151 | EN.num = i 152 | tmp = EventfdReader(EN, self.ServerCallback, cls.SI) 153 | if i: # Skip mailslot 0, the globals "slot" 154 | tmp.start() 155 | 156 | @property 157 | def promptname(self): 158 | '''For Commander prompt''' 159 | return 'Z-switch' if self.SI.smart else 'Z-server' 160 | 161 | def logPrefix(self): # This override works after instantiation 162 | return 'ProtoIVSHMSG' 163 | 164 | def dataReceived(self, data): 165 | ''' TNSH :-) ''' 166 | self.SI.logmsg('dataReceived, quite unexpectedly') 167 | raise NotImplementedError(self) 168 | 169 | # If errors occur early enough, send a bad revision to the client so it 170 | # terminates the connection. Remember, "self" is a proxy for a peer. 171 | def connectionMade(self): 172 | recycled = self.SI.recycled # Does it exist? 173 | if recycled: 174 | recycled = self.SI.recycled.get(self.id, None) # Am I there? 175 | if recycled: 176 | del self.SI.recycled[recycled.id] 177 | msg = 'new socket %d == peer id %d %s' % ( 178 | self.transport.fileno(), self.id, 179 | 'recycled' if recycled else '' 180 | ) 181 | self.SI.logmsg(msg) 182 | if self.id == -1: # set from __init__ 183 | self.SI.logmsg('Max clients reached') 184 | self.send_initial_info(False) # client complains but with grace 185 | return 186 | 187 | # The original original code was written around this variable name. 188 | # Keep that convention for easier comparison. 189 | server_peer_list = list(self.SI.clients.values()) 190 | 191 | # Server line 175: create specified number of eventfds. These are 192 | # shared with all other clients who use them to signal each other. 193 | # Recycling keeps QEMU sessions from dying when other clients drop, 194 | # a perk not found in original code. 195 | if recycled: 196 | self.EN_list = recycled.EN_list 197 | else: 198 | try: 199 | self.EN_list = ivshmsg_event_notifier_list(MB.nEvents) 200 | except Exception as e: 201 | self.SI.logmsg('Event notifiers failed: %s' % str(e)) 202 | self.send_initial_info(False) 203 | return 204 | 205 | # Server line 183: send version, peer id, shm fd 206 | if self.verbose: 207 | PRINT('Sending initial info to new peer...') 208 | if not self.send_initial_info(): 209 | self.SI.logmsg('Send initial info failed') 210 | return 211 | 212 | # Server line 189: advertise the new peer to others. Note that 213 | # this new peer has not yet been added to the list; this loop is 214 | # NOT traversed for the first peer to connect. 215 | if not recycled: 216 | if self.verbose: 217 | PRINT('NOT recycled: advertising other peers...') 218 | for other_peer in server_peer_list: 219 | for peer_EN in self.EN_list: 220 | ivshmsg_send_one_msg( 221 | other_peer.transport.socket, 222 | self.id, 223 | peer_EN.wfd) 224 | 225 | # Server line 197: advertise the other peers to the new one. 226 | # Remember "this" new peer proxy has not been added to the list yet. 227 | if self.verbose: 228 | PRINT('Advertising other peers to the new peer...') 229 | for other_peer in server_peer_list: 230 | for other_peer_EN in other_peer.EN_list: 231 | ivshmsg_send_one_msg( 232 | self.transport.socket, 233 | other_peer.id, 234 | other_peer_EN.wfd) 235 | 236 | # Non-standard voodoo extension to previous advertisment: advertise 237 | # this server to the new peer. To QEMU it just looks like one more 238 | # grouping in the previous batch. Exists only in non-silent mode. 239 | if self.verbose: 240 | PRINT('Advertising this server to the new peer...') 241 | for server_EN in self.SI.EN_list: 242 | ivshmsg_send_one_msg( 243 | self.transport.socket, 244 | self.SI.id, 245 | server_EN.wfd) 246 | 247 | # Server line 205: advertise the new peer to itself, ie, send the 248 | # eventfds it needs for receiving messages. This final batch 249 | # where the embedded self.id matches the initial_info id is the 250 | # sentinel that communications are finished. 251 | if self.verbose: 252 | PRINT('Advertising the new peer to itself...') 253 | for peer_EN in self.EN_list: 254 | ivshmsg_send_one_msg( 255 | self.transport.socket, 256 | self.id, 257 | peer_EN.get_fd()) # Must be a good story here... 258 | 259 | # And now that it's finished: 260 | self.SI.clients[self.id] = self 261 | 262 | # QEMU did the connect but its VM is probably not yet running well 263 | # enough to respond. Since there's no (easy) way to tell, this is 264 | # a blind shot... 265 | self.printswitch(self.SI.clients) # default settling time 266 | if not self.SI.isPFM: 267 | send_payload('Link CTL Peer-Attribute', 268 | self.SI.id, 269 | self.EN_list[self.id]) 270 | 271 | def connectionLost(self, reason): 272 | '''Tell the other peers that this one has died.''' 273 | dirty = reason.check(TIError.ConnectionDone) is None 274 | status = 'Dirty' if dirty else 'Clean' 275 | verb = 'server shutdown' if self.SI.quitting else 'disconnect' 276 | self.SI.logmsg('%s %s of peer id %d' % (status, verb, self.id)) 277 | # For QEMU crashes and shutdowns (not the OS guest but QEMU itself). 278 | MB.clear_mailslot(self.id) 279 | 280 | if self.id in self.SI.clients: # Only if everything was completed 281 | del self.SI.clients[self.id] 282 | if self.SI.recycled and not self.SI.quitting: 283 | self.SI.recycled[self.id] = self 284 | else: 285 | try: 286 | for other_peer in self.SI.clients.values(): 287 | ivshmsg_send_one_msg(other_peer.transport.socket, self.id) 288 | for EN in self.EN_list: 289 | EN.cleanup() 290 | except Exception as e: 291 | self.SI.logmsg('Closing peer transports failed: %s' % str(e)) 292 | self.printswitch(self.SI.clients) 293 | if self.SI.quitting and not self.SI.clients: # last one exited 294 | self.SI.logmsg('Final client disconnected after "quit"') 295 | TIreactor.stop() # turn out the lights 296 | 297 | def create_new_peer_id(self): 298 | '''Determine the lowest unused client ID and set self.id.''' 299 | 300 | self.SID0 = 0 # When queried, the answer is in the context... 301 | self.CID0 = 0 # ...of the server/switch, NOT the proxy item. 302 | if len(self.SI.clients) >= MB.nClients: 303 | self.id = -1 # sentinel 304 | return # Until a Link RFC is executed 305 | 306 | # Generate ID sets used by each. The range includes the highest 307 | # client ID. Two modes: dumb == monotonic from 1; smart == random. 308 | all_ids = frozenset((range(IVSHMSG_LOWEST_ID, self.SI.id))) 309 | active_ids = frozenset(self.SI.clients.keys()) 310 | available_ids = all_ids - active_ids 311 | if self.SI.smart: 312 | self.id = random.choice(tuple(available_ids)) 313 | else: 314 | if not self.SI.clients: # empty 315 | self.id = 1 316 | else: 317 | self.id = (sorted(available_ids))[0] 318 | 319 | if self.SI.smart: 320 | self.SID0 = self.SI.default_SID 321 | self.CID0 = self.id * 100 322 | 323 | def send_initial_info(self, ok=True): 324 | thesocket = self.transport.socket # self is a proxy for the peer. 325 | try: 326 | # 1. Protocol version without fd. 327 | if not ok: # Violate the version check and bomb the client. 328 | PRINT('Early termination') 329 | ivshmsg_send_one_msg(thesocket, -1) 330 | self.transport.loseConnection() 331 | self.id = -1 332 | return 333 | if not ivshmsg_send_one_msg(thesocket, 334 | self.SERVER_IVSHMEM_PROTOCOL_VERSION): 335 | PRINT('This is screwed') 336 | return False 337 | 338 | # 2. The client's (new) id, without an fd. 339 | ivshmsg_send_one_msg(thesocket, self.id) 340 | 341 | # 3. -1 for data with the fd of the IVSHMEM file. Using this 342 | # protocol a valid fd is required. 343 | ivshmsg_send_one_msg(thesocket, -1, MB.fd) 344 | return True 345 | except Exception as e: 346 | PRINT(str(e)) 347 | return False 348 | 349 | # The server is the receiver, and the cbdata is the server SI class 350 | # attribute shared by all requester proxy objects. The client instance 351 | # which which made the request (and provides the target for the response) 352 | # must be indirectly looked up. 353 | @staticmethod 354 | def ServerCallback(vectorobj): 355 | requester_id = vectorobj.num 356 | requester_name = MB.nodename(requester_id) 357 | request = MB.retrieve(requester_id) 358 | SI = vectorobj.cbdata 359 | 360 | # Recover the appropriate requester proxy object which can die between 361 | # its interrupt and this callback. 362 | try: 363 | requester_proxy = SI.clients[requester_id] 364 | assert requester_proxy.SI is SI, 'Say WHAT?' 365 | assert requester_proxy.id == requester_id, 'WTF MF?' 366 | requester_proxy.requester_id = requester_id # Pedantic? 367 | # For QEMU/VM, this may be the first chance to grab this (if the 368 | # drivers hadn't come up before). Just get it fresh each time. 369 | requester_proxy.nodename = requester_name 370 | requester_proxy.cclass = MB.cclass(requester_id) 371 | requester_proxy.peerattrs['cclass'] = requester_proxy.cclass 372 | except KeyError as e: 373 | SI.logmsg('Disappeering act by %d' % requester_id) 374 | return 375 | 376 | # The object passed has two sets of data: 377 | # 1. Id/target information on where to send the response 378 | # 2. Peer attributes used in two requests: 379 | # readout to send them 380 | # overwrite if they're being sent by a PFM 381 | # I'm "if blah blah" in handle_request() but I should just strip the 382 | # peer attribute here (the server in this case). for the server, 383 | # requester_name and requester_proxy.nodename are the same 384 | # peer_attributes are from the server, not the proxy. 385 | # it's different for the client. 386 | 387 | ro = ResponseObject( 388 | this=SI, # has "my" (server) CID0, SID0, and cclass 389 | proxy=requester_proxy, # for certain server-only admin 390 | from_id=SI.id, 391 | to_doorbell=requester_proxy.EN_list[SI.id], 392 | logmsg=SI.logmsg, 393 | stdtrace=SI.stdtrace, 394 | verbose=SI.verbose 395 | ) 396 | ret = handle_request(request, requester_name, ro) 397 | 398 | # ret is either True, False, or... 399 | 400 | if ret == 'dump': 401 | # Might be some other stuff, but finally 402 | ProtocolIVSHMSGServer.printswitch(SI.clients) 403 | 404 | #---------------------------------------------------------------------- 405 | # ASCII art switch: Left side and right sider are each half of the ports. 406 | 407 | @staticmethod 408 | def printswitch(clients, delay=0.5): 409 | if int(delay) < 0 or int(delay) > 2: 410 | delay = 1.0 411 | time.sleep(delay) 412 | lfmt = '%s %s [%s,%s]' 413 | rfmt = '[%s,%s] %s %s' 414 | half = (MB.MAILBOX_MAX_SLOTS - 1) // 2 415 | NSP = 34 416 | lspaces = ' ' * NSP 417 | PRINT('\n%s ____ ____' % lspaces) 418 | notch = 'U' 419 | for i in range(1, half + 1): 420 | left = i 421 | right = MB.server_id - left # 74XX TTL 422 | try: 423 | ldesc = lspaces 424 | c = clients[left] 425 | pa = c.peerattrs 426 | pa['cclass'] = MB.cclass(left) 427 | ldesc += lfmt % (pa['cclass'], MB.nodename(left), 428 | pa['CID0'], pa['SID0']) 429 | except KeyError as e: 430 | pass 431 | try: 432 | c = clients[right] 433 | pa = c.peerattrs 434 | pa['cclass'] = MB.cclass(right) 435 | rdesc = rfmt % (pa['CID0'], pa['SID0'], 436 | pa['cclass'], MB.nodename(right)) 437 | except KeyError as e: 438 | rdesc = '' 439 | PRINT('%-s -|%1d %c %2d|- %s' % ( 440 | ldesc[-NSP:], left, notch, right, rdesc)) 441 | notch = ' ' 442 | PRINT('%s =========' % lspaces) 443 | 444 | #---------------------------------------------------------------------- 445 | # Command line parsing, picked up by commander.py. This instance was 446 | # from the original "class-setting" call so it has no transport. 447 | 448 | def doCommand(self, cmd, args=None): 449 | 450 | if cmd in ('h', 'help') or '?' in cmd: 451 | print('h[elp]\n\tThis message') 452 | print('d[ump]\n\tPrint status of all ports') 453 | print('q[uit]\n\tShut it all down') 454 | return True 455 | 456 | if cmd in ('d', 'dump'): 457 | if self.verbose > 1: 458 | PRINT('') 459 | for id, peer in self.SI.clients.items(): 460 | PRINT('%10s: %s' % (MB.nodename(id), peer.peerattrs)) 461 | if self.verbose > 2: 462 | PPRINT(vars(peer), stream=sys.stdout) 463 | self.printswitch(self.SI.clients, 0) 464 | return True 465 | 466 | if cmd in ('q', 'quit'): 467 | self.quitting = True # self == SI, remember? 468 | self.logmsg('Interactive command to "quit"') 469 | if self.clients: # Trigger lostConnection 470 | for c in self.clients.values(): 471 | c.transport.loseConnection() # Final callback exits 472 | else: 473 | TIreactor.stop() 474 | return False 475 | 476 | PRINT('Unrecognized command "%s", try "help"' % cmd) 477 | return True 478 | 479 | ########################################################################### 480 | # Normally the Endpoint and listen() call is done explicitly, interwoven 481 | # with passing this constructor. This approach used here hides all the 482 | # twisted things in this module. 483 | 484 | 485 | class FactoryIVSHMSGServer(TIPServerFactory): 486 | 487 | _required_arg_defaults = { 488 | 'title': 'IVSHMSG', 489 | 'foreground': True, # Only affects logging choice in here 490 | 'logfile': '/tmp/ivshmsg_log', 491 | 'mailbox': 'ivshmsg_mailbox', # Will end up in /dev/shm 492 | 'nClients': 2, 493 | 'recycle': False, # Try to preserve other QEMUs 494 | 'silent': False, # Does participate in eventfds/mailbox 495 | 'socketpath': '/tmp/ivshmsg_socket', 496 | 'verbose': 0, 497 | } 498 | 499 | def __init__(self, args=None): 500 | '''Args must be an object with the following attributes: 501 | foreground, logfile, mailbox, nClients, silent, socketpath, verbose 502 | Suitable defaults will be supplied.''' 503 | 504 | # Pass command line args to ProtocolIVSHMSG, then open logging. 505 | if args is None: 506 | args = argparse.Namespace() 507 | for arg, default in self._required_arg_defaults.items(): 508 | setattr(args, arg, getattr(args, arg, default)) 509 | 510 | # Mailbox may be sized above the requested number of clients to 511 | # satisfy QEMU IVSHMEM restrictions. 512 | args.server_id = args.nClients + 1 513 | args.nEvents = args.nClients + 2 514 | 515 | # It's a singleton so no reason to keep the instance, however it's 516 | # the way I wrote the Klein API server so... 517 | mb = MB(args=args) 518 | MailBoxReSTAPI(mb) 519 | shutdown_http_logging() 520 | 521 | if args.foreground: 522 | if args.verbose > 1: 523 | TPlog.startLogging(sys.stdout, setStdout=False) 524 | else: 525 | TPlog.startLogging(open('/dev/null', 'a'), setStdout=False) 526 | else: 527 | PRINT('Logging to %s' % args.logfile) 528 | TPlog.startLogging( 529 | DailyLogFile.fromFullPath(args.logfile), 530 | setStdout=True) # "Pass-through" explicit print() for debug 531 | args.logmsg = TPlog.msg 532 | args.logerr = TPlog.err 533 | 534 | # By Twisted version 18, "mode=" is deprecated and you should just 535 | # inherit the tacky bit from the parent directory. wantPID creates 536 | # .lock as a symlink to "PID". 537 | E = UNIXServerEndpoint( 538 | TIreactor, 539 | args.socketpath, 540 | mode=0o666, # Deprecated at Twisted 18 541 | wantPID=True) 542 | E.listen(self) 543 | args.logmsg('%s server @%d ready for %d clients on %s' % 544 | (args.title, args.server_id, args.nClients, args.socketpath)) 545 | 546 | # https://stackoverflow.com/questions/1411281/twisted-listen-to-multiple-ports-for-multiple-processes-with-one-reactor 547 | 548 | # Voodoo kick to a) set up one-time SI and b)setup commander. 549 | # Docs mislead, have to explicitly pass something to get persistent 550 | # state across protocol/transport invocations. As there is only 551 | # one server object per process instantion, that's not necessary. 552 | 553 | protobj = ProtocolIVSHMSGServer(self, args) # With "args" 554 | Commander(protobj) 555 | 556 | def buildProtocol(self, useless_addr): 557 | # Unfortunately this doesn't work. Search for /dev/null above. 558 | shutdown_http_logging() 559 | 560 | protobj = ProtocolIVSHMSGServer(self) # Without "args" 561 | return protobj 562 | 563 | def run(self): 564 | TIreactor.run() # and hangs here 565 | 566 | --------------------------------------------------------------------------------