├── .gitignore ├── COPYING ├── Doxyfile ├── MANIFEST.in ├── Makefile ├── doc ├── changelog.md └── todo.md ├── mqspeak.conf ├── mqspeak.service ├── mqspeak ├── __init__.py ├── __main__.py ├── args.py ├── channel.py ├── collecting.py ├── config.py ├── data.py ├── sending.py ├── system.py └── updating.py ├── readme.md └── setup.py /.gitignore: -------------------------------------------------------------------------------- 1 | __pycache__/ 2 | doxygen_sqlite3.db 3 | html/ 4 | latex/ 5 | mqspeak.egg-info/ 6 | dist/ 7 | build/ 8 | -------------------------------------------------------------------------------- /COPYING: -------------------------------------------------------------------------------- 1 | GNU GENERAL PUBLIC LICENSE 2 | Version 3, 29 June 2007 3 | 4 | Copyright (C) 2007 Free Software Foundation, Inc. 5 | Everyone is permitted to copy and distribute verbatim copies 6 | of this license document, but changing it is not allowed. 7 | 8 | Preamble 9 | 10 | The GNU General Public License is a free, copyleft license for 11 | software and other kinds of works. 12 | 13 | The licenses for most software and other practical works are designed 14 | to take away your freedom to share and change the works. By contrast, 15 | the GNU General Public License is intended to guarantee your freedom to 16 | share and change all versions of a program--to make sure it remains free 17 | software for all its users. We, the Free Software Foundation, use the 18 | GNU General Public License for most of our software; it applies also to 19 | any other work released this way by its authors. You can apply it to 20 | your programs, too. 21 | 22 | When we speak of free software, we are referring to freedom, not 23 | price. Our General Public Licenses are designed to make sure that you 24 | have the freedom to distribute copies of free software (and charge for 25 | them if you wish), that you receive source code or can get it if you 26 | want it, that you can change the software or use pieces of it in new 27 | free programs, and that you know you can do these things. 28 | 29 | To protect your rights, we need to prevent others from denying you 30 | these rights or asking you to surrender the rights. Therefore, you have 31 | certain responsibilities if you distribute copies of the software, or if 32 | you modify it: responsibilities to respect the freedom of others. 33 | 34 | For example, if you distribute copies of such a program, whether 35 | gratis or for a fee, you must pass on to the recipients the same 36 | freedoms that you received. You must make sure that they, too, receive 37 | or can get the source code. And you must show them these terms so they 38 | know their rights. 39 | 40 | Developers that use the GNU GPL protect your rights with two steps: 41 | (1) assert copyright on the software, and (2) offer you this License 42 | giving you legal permission to copy, distribute and/or modify it. 43 | 44 | For the developers' and authors' protection, the GPL clearly explains 45 | that there is no warranty for this free software. For both users' and 46 | authors' sake, the GPL requires that modified versions be marked as 47 | changed, so that their problems will not be attributed erroneously to 48 | authors of previous versions. 49 | 50 | Some devices are designed to deny users access to install or run 51 | modified versions of the software inside them, although the manufacturer 52 | can do so. This is fundamentally incompatible with the aim of 53 | protecting users' freedom to change the software. The systematic 54 | pattern of such abuse occurs in the area of products for individuals to 55 | use, which is precisely where it is most unacceptable. Therefore, we 56 | have designed this version of the GPL to prohibit the practice for those 57 | products. If such problems arise substantially in other domains, we 58 | stand ready to extend this provision to those domains in future versions 59 | of the GPL, as needed to protect the freedom of users. 60 | 61 | Finally, every program is threatened constantly by software patents. 62 | States should not allow patents to restrict development and use of 63 | software on general-purpose computers, but in those that do, we wish to 64 | avoid the special danger that patents applied to a free program could 65 | make it effectively proprietary. To prevent this, the GPL assures that 66 | patents cannot be used to render the program non-free. 67 | 68 | The precise terms and conditions for copying, distribution and 69 | modification follow. 70 | 71 | TERMS AND CONDITIONS 72 | 73 | 0. Definitions. 74 | 75 | "This License" refers to version 3 of the GNU General Public License. 76 | 77 | "Copyright" also means copyright-like laws that apply to other kinds of 78 | works, such as semiconductor masks. 79 | 80 | "The Program" refers to any copyrightable work licensed under this 81 | License. Each licensee is addressed as "you". "Licensees" and 82 | "recipients" may be individuals or organizations. 83 | 84 | To "modify" a work means to copy from or adapt all or part of the work 85 | in a fashion requiring copyright permission, other than the making of an 86 | exact copy. The resulting work is called a "modified version" of the 87 | earlier work or a work "based on" the earlier work. 88 | 89 | A "covered work" means either the unmodified Program or a work based 90 | on the Program. 91 | 92 | To "propagate" a work means to do anything with it that, without 93 | permission, would make you directly or secondarily liable for 94 | infringement under applicable copyright law, except executing it on a 95 | computer or modifying a private copy. Propagation includes copying, 96 | distribution (with or without modification), making available to the 97 | public, and in some countries other activities as well. 98 | 99 | To "convey" a work means any kind of propagation that enables other 100 | parties to make or receive copies. Mere interaction with a user through 101 | a computer network, with no transfer of a copy, is not conveying. 102 | 103 | An interactive user interface displays "Appropriate Legal Notices" 104 | to the extent that it includes a convenient and prominently visible 105 | feature that (1) displays an appropriate copyright notice, and (2) 106 | tells the user that there is no warranty for the work (except to the 107 | extent that warranties are provided), that licensees may convey the 108 | work under this License, and how to view a copy of this License. If 109 | the interface presents a list of user commands or options, such as a 110 | menu, a prominent item in the list meets this criterion. 111 | 112 | 1. Source Code. 113 | 114 | The "source code" for a work means the preferred form of the work 115 | for making modifications to it. "Object code" means any non-source 116 | form of a work. 117 | 118 | A "Standard Interface" means an interface that either is an official 119 | standard defined by a recognized standards body, or, in the case of 120 | interfaces specified for a particular programming language, one that 121 | is widely used among developers working in that language. 122 | 123 | The "System Libraries" of an executable work include anything, other 124 | than the work as a whole, that (a) is included in the normal form of 125 | packaging a Major Component, but which is not part of that Major 126 | Component, and (b) serves only to enable use of the work with that 127 | Major Component, or to implement a Standard Interface for which an 128 | implementation is available to the public in source code form. A 129 | "Major Component", in this context, means a major essential component 130 | (kernel, window system, and so on) of the specific operating system 131 | (if any) on which the executable work runs, or a compiler used to 132 | produce the work, or an object code interpreter used to run it. 133 | 134 | The "Corresponding Source" for a work in object code form means all 135 | the source code needed to generate, install, and (for an executable 136 | work) run the object code and to modify the work, including scripts to 137 | control those activities. However, it does not include the work's 138 | System Libraries, or general-purpose tools or generally available free 139 | programs which are used unmodified in performing those activities but 140 | which are not part of the work. For example, Corresponding Source 141 | includes interface definition files associated with source files for 142 | the work, and the source code for shared libraries and dynamically 143 | linked subprograms that the work is specifically designed to require, 144 | such as by intimate data communication or control flow between those 145 | subprograms and other parts of the work. 146 | 147 | The Corresponding Source need not include anything that users 148 | can regenerate automatically from other parts of the Corresponding 149 | Source. 150 | 151 | The Corresponding Source for a work in source code form is that 152 | same work. 153 | 154 | 2. Basic Permissions. 155 | 156 | All rights granted under this License are granted for the term of 157 | copyright on the Program, and are irrevocable provided the stated 158 | conditions are met. This License explicitly affirms your unlimited 159 | permission to run the unmodified Program. The output from running a 160 | covered work is covered by this License only if the output, given its 161 | content, constitutes a covered work. This License acknowledges your 162 | rights of fair use or other equivalent, as provided by copyright law. 163 | 164 | You may make, run and propagate covered works that you do not 165 | convey, without conditions so long as your license otherwise remains 166 | in force. You may convey covered works to others for the sole purpose 167 | of having them make modifications exclusively for you, or provide you 168 | with facilities for running those works, provided that you comply with 169 | the terms of this License in conveying all material for which you do 170 | not control copyright. Those thus making or running the covered works 171 | for you must do so exclusively on your behalf, under your direction 172 | and control, on terms that prohibit them from making any copies of 173 | your copyrighted material outside their relationship with you. 174 | 175 | Conveying under any other circumstances is permitted solely under 176 | the conditions stated below. Sublicensing is not allowed; section 10 177 | makes it unnecessary. 178 | 179 | 3. Protecting Users' Legal Rights From Anti-Circumvention Law. 180 | 181 | No covered work shall be deemed part of an effective technological 182 | measure under any applicable law fulfilling obligations under article 183 | 11 of the WIPO copyright treaty adopted on 20 December 1996, or 184 | similar laws prohibiting or restricting circumvention of such 185 | measures. 186 | 187 | When you convey a covered work, you waive any legal power to forbid 188 | circumvention of technological measures to the extent such circumvention 189 | is effected by exercising rights under this License with respect to 190 | the covered work, and you disclaim any intention to limit operation or 191 | modification of the work as a means of enforcing, against the work's 192 | users, your or third parties' legal rights to forbid circumvention of 193 | technological measures. 194 | 195 | 4. Conveying Verbatim Copies. 196 | 197 | You may convey verbatim copies of the Program's source code as you 198 | receive it, in any medium, provided that you conspicuously and 199 | appropriately publish on each copy an appropriate copyright notice; 200 | keep intact all notices stating that this License and any 201 | non-permissive terms added in accord with section 7 apply to the code; 202 | keep intact all notices of the absence of any warranty; and give all 203 | recipients a copy of this License along with the Program. 204 | 205 | You may charge any price or no price for each copy that you convey, 206 | and you may offer support or warranty protection for a fee. 207 | 208 | 5. Conveying Modified Source Versions. 209 | 210 | You may convey a work based on the Program, or the modifications to 211 | produce it from the Program, in the form of source code under the 212 | terms of section 4, provided that you also meet all of these conditions: 213 | 214 | a) The work must carry prominent notices stating that you modified 215 | it, and giving a relevant date. 216 | 217 | b) The work must carry prominent notices stating that it is 218 | released under this License and any conditions added under section 219 | 7. This requirement modifies the requirement in section 4 to 220 | "keep intact all notices". 221 | 222 | c) You must license the entire work, as a whole, under this 223 | License to anyone who comes into possession of a copy. This 224 | License will therefore apply, along with any applicable section 7 225 | additional terms, to the whole of the work, and all its parts, 226 | regardless of how they are packaged. This License gives no 227 | permission to license the work in any other way, but it does not 228 | invalidate such permission if you have separately received it. 229 | 230 | d) If the work has interactive user interfaces, each must display 231 | Appropriate Legal Notices; however, if the Program has interactive 232 | interfaces that do not display Appropriate Legal Notices, your 233 | work need not make them do so. 234 | 235 | A compilation of a covered work with other separate and independent 236 | works, which are not by their nature extensions of the covered work, 237 | and which are not combined with it such as to form a larger program, 238 | in or on a volume of a storage or distribution medium, is called an 239 | "aggregate" if the compilation and its resulting copyright are not 240 | used to limit the access or legal rights of the compilation's users 241 | beyond what the individual works permit. Inclusion of a covered work 242 | in an aggregate does not cause this License to apply to the other 243 | parts of the aggregate. 244 | 245 | 6. Conveying Non-Source Forms. 246 | 247 | You may convey a covered work in object code form under the terms 248 | of sections 4 and 5, provided that you also convey the 249 | machine-readable Corresponding Source under the terms of this License, 250 | in one of these ways: 251 | 252 | a) Convey the object code in, or embodied in, a physical product 253 | (including a physical distribution medium), accompanied by the 254 | Corresponding Source fixed on a durable physical medium 255 | customarily used for software interchange. 256 | 257 | b) Convey the object code in, or embodied in, a physical product 258 | (including a physical distribution medium), accompanied by a 259 | written offer, valid for at least three years and valid for as 260 | long as you offer spare parts or customer support for that product 261 | model, to give anyone who possesses the object code either (1) a 262 | copy of the Corresponding Source for all the software in the 263 | product that is covered by this License, on a durable physical 264 | medium customarily used for software interchange, for a price no 265 | more than your reasonable cost of physically performing this 266 | conveying of source, or (2) access to copy the 267 | Corresponding Source from a network server at no charge. 268 | 269 | c) Convey individual copies of the object code with a copy of the 270 | written offer to provide the Corresponding Source. This 271 | alternative is allowed only occasionally and noncommercially, and 272 | only if you received the object code with such an offer, in accord 273 | with subsection 6b. 274 | 275 | d) Convey the object code by offering access from a designated 276 | place (gratis or for a charge), and offer equivalent access to the 277 | Corresponding Source in the same way through the same place at no 278 | further charge. You need not require recipients to copy the 279 | Corresponding Source along with the object code. If the place to 280 | copy the object code is a network server, the Corresponding Source 281 | may be on a different server (operated by you or a third party) 282 | that supports equivalent copying facilities, provided you maintain 283 | clear directions next to the object code saying where to find the 284 | Corresponding Source. Regardless of what server hosts the 285 | Corresponding Source, you remain obligated to ensure that it is 286 | available for as long as needed to satisfy these requirements. 287 | 288 | e) Convey the object code using peer-to-peer transmission, provided 289 | you inform other peers where the object code and Corresponding 290 | Source of the work are being offered to the general public at no 291 | charge under subsection 6d. 292 | 293 | A separable portion of the object code, whose source code is excluded 294 | from the Corresponding Source as a System Library, need not be 295 | included in conveying the object code work. 296 | 297 | A "User Product" is either (1) a "consumer product", which means any 298 | tangible personal property which is normally used for personal, family, 299 | or household purposes, or (2) anything designed or sold for incorporation 300 | into a dwelling. In determining whether a product is a consumer product, 301 | doubtful cases shall be resolved in favor of coverage. For a particular 302 | product received by a particular user, "normally used" refers to a 303 | typical or common use of that class of product, regardless of the status 304 | of the particular user or of the way in which the particular user 305 | actually uses, or expects or is expected to use, the product. A product 306 | is a consumer product regardless of whether the product has substantial 307 | commercial, industrial or non-consumer uses, unless such uses represent 308 | the only significant mode of use of the product. 309 | 310 | "Installation Information" for a User Product means any methods, 311 | procedures, authorization keys, or other information required to install 312 | and execute modified versions of a covered work in that User Product from 313 | a modified version of its Corresponding Source. The information must 314 | suffice to ensure that the continued functioning of the modified object 315 | code is in no case prevented or interfered with solely because 316 | modification has been made. 317 | 318 | If you convey an object code work under this section in, or with, or 319 | specifically for use in, a User Product, and the conveying occurs as 320 | part of a transaction in which the right of possession and use of the 321 | User Product is transferred to the recipient in perpetuity or for a 322 | fixed term (regardless of how the transaction is characterized), the 323 | Corresponding Source conveyed under this section must be accompanied 324 | by the Installation Information. But this requirement does not apply 325 | if neither you nor any third party retains the ability to install 326 | modified object code on the User Product (for example, the work has 327 | been installed in ROM). 328 | 329 | The requirement to provide Installation Information does not include a 330 | requirement to continue to provide support service, warranty, or updates 331 | for a work that has been modified or installed by the recipient, or for 332 | the User Product in which it has been modified or installed. Access to a 333 | network may be denied when the modification itself materially and 334 | adversely affects the operation of the network or violates the rules and 335 | protocols for communication across the network. 336 | 337 | Corresponding Source conveyed, and Installation Information provided, 338 | in accord with this section must be in a format that is publicly 339 | documented (and with an implementation available to the public in 340 | source code form), and must require no special password or key for 341 | unpacking, reading or copying. 342 | 343 | 7. Additional Terms. 344 | 345 | "Additional permissions" are terms that supplement the terms of this 346 | License by making exceptions from one or more of its conditions. 347 | Additional permissions that are applicable to the entire Program shall 348 | be treated as though they were included in this License, to the extent 349 | that they are valid under applicable law. If additional permissions 350 | apply only to part of the Program, that part may be used separately 351 | under those permissions, but the entire Program remains governed by 352 | this License without regard to the additional permissions. 353 | 354 | When you convey a copy of a covered work, you may at your option 355 | remove any additional permissions from that copy, or from any part of 356 | it. (Additional permissions may be written to require their own 357 | removal in certain cases when you modify the work.) You may place 358 | additional permissions on material, added by you to a covered work, 359 | for which you have or can give appropriate copyright permission. 360 | 361 | Notwithstanding any other provision of this License, for material you 362 | add to a covered work, you may (if authorized by the copyright holders of 363 | that material) supplement the terms of this License with terms: 364 | 365 | a) Disclaiming warranty or limiting liability differently from the 366 | terms of sections 15 and 16 of this License; or 367 | 368 | b) Requiring preservation of specified reasonable legal notices or 369 | author attributions in that material or in the Appropriate Legal 370 | Notices displayed by works containing it; or 371 | 372 | c) Prohibiting misrepresentation of the origin of that material, or 373 | requiring that modified versions of such material be marked in 374 | reasonable ways as different from the original version; or 375 | 376 | d) Limiting the use for publicity purposes of names of licensors or 377 | authors of the material; or 378 | 379 | e) Declining to grant rights under trademark law for use of some 380 | trade names, trademarks, or service marks; or 381 | 382 | f) Requiring indemnification of licensors and authors of that 383 | material by anyone who conveys the material (or modified versions of 384 | it) with contractual assumptions of liability to the recipient, for 385 | any liability that these contractual assumptions directly impose on 386 | those licensors and authors. 387 | 388 | All other non-permissive additional terms are considered "further 389 | restrictions" within the meaning of section 10. If the Program as you 390 | received it, or any part of it, contains a notice stating that it is 391 | governed by this License along with a term that is a further 392 | restriction, you may remove that term. If a license document contains 393 | a further restriction but permits relicensing or conveying under this 394 | License, you may add to a covered work material governed by the terms 395 | of that license document, provided that the further restriction does 396 | not survive such relicensing or conveying. 397 | 398 | If you add terms to a covered work in accord with this section, you 399 | must place, in the relevant source files, a statement of the 400 | additional terms that apply to those files, or a notice indicating 401 | where to find the applicable terms. 402 | 403 | Additional terms, permissive or non-permissive, may be stated in the 404 | form of a separately written license, or stated as exceptions; 405 | the above requirements apply either way. 406 | 407 | 8. Termination. 408 | 409 | You may not propagate or modify a covered work except as expressly 410 | provided under this License. Any attempt otherwise to propagate or 411 | modify it is void, and will automatically terminate your rights under 412 | this License (including any patent licenses granted under the third 413 | paragraph of section 11). 414 | 415 | However, if you cease all violation of this License, then your 416 | license from a particular copyright holder is reinstated (a) 417 | provisionally, unless and until the copyright holder explicitly and 418 | finally terminates your license, and (b) permanently, if the copyright 419 | holder fails to notify you of the violation by some reasonable means 420 | prior to 60 days after the cessation. 421 | 422 | Moreover, your license from a particular copyright holder is 423 | reinstated permanently if the copyright holder notifies you of the 424 | violation by some reasonable means, this is the first time you have 425 | received notice of violation of this License (for any work) from that 426 | copyright holder, and you cure the violation prior to 30 days after 427 | your receipt of the notice. 428 | 429 | Termination of your rights under this section does not terminate the 430 | licenses of parties who have received copies or rights from you under 431 | this License. If your rights have been terminated and not permanently 432 | reinstated, you do not qualify to receive new licenses for the same 433 | material under section 10. 434 | 435 | 9. Acceptance Not Required for Having Copies. 436 | 437 | You are not required to accept this License in order to receive or 438 | run a copy of the Program. Ancillary propagation of a covered work 439 | occurring solely as a consequence of using peer-to-peer transmission 440 | to receive a copy likewise does not require acceptance. However, 441 | nothing other than this License grants you permission to propagate or 442 | modify any covered work. These actions infringe copyright if you do 443 | not accept this License. Therefore, by modifying or propagating a 444 | covered work, you indicate your acceptance of this License to do so. 445 | 446 | 10. Automatic Licensing of Downstream Recipients. 447 | 448 | Each time you convey a covered work, the recipient automatically 449 | receives a license from the original licensors, to run, modify and 450 | propagate that work, subject to this License. You are not responsible 451 | for enforcing compliance by third parties with this License. 452 | 453 | An "entity transaction" is a transaction transferring control of an 454 | organization, or substantially all assets of one, or subdividing an 455 | organization, or merging organizations. If propagation of a covered 456 | work results from an entity transaction, each party to that 457 | transaction who receives a copy of the work also receives whatever 458 | licenses to the work the party's predecessor in interest had or could 459 | give under the previous paragraph, plus a right to possession of the 460 | Corresponding Source of the work from the predecessor in interest, if 461 | the predecessor has it or can get it with reasonable efforts. 462 | 463 | You may not impose any further restrictions on the exercise of the 464 | rights granted or affirmed under this License. For example, you may 465 | not impose a license fee, royalty, or other charge for exercise of 466 | rights granted under this License, and you may not initiate litigation 467 | (including a cross-claim or counterclaim in a lawsuit) alleging that 468 | any patent claim is infringed by making, using, selling, offering for 469 | sale, or importing the Program or any portion of it. 470 | 471 | 11. Patents. 472 | 473 | A "contributor" is a copyright holder who authorizes use under this 474 | License of the Program or a work on which the Program is based. The 475 | work thus licensed is called the contributor's "contributor version". 476 | 477 | A contributor's "essential patent claims" are all patent claims 478 | owned or controlled by the contributor, whether already acquired or 479 | hereafter acquired, that would be infringed by some manner, permitted 480 | by this License, of making, using, or selling its contributor version, 481 | but do not include claims that would be infringed only as a 482 | consequence of further modification of the contributor version. For 483 | purposes of this definition, "control" includes the right to grant 484 | patent sublicenses in a manner consistent with the requirements of 485 | this License. 486 | 487 | Each contributor grants you a non-exclusive, worldwide, royalty-free 488 | patent license under the contributor's essential patent claims, to 489 | make, use, sell, offer for sale, import and otherwise run, modify and 490 | propagate the contents of its contributor version. 491 | 492 | In the following three paragraphs, a "patent license" is any express 493 | agreement or commitment, however denominated, not to enforce a patent 494 | (such as an express permission to practice a patent or covenant not to 495 | sue for patent infringement). To "grant" such a patent license to a 496 | party means to make such an agreement or commitment not to enforce a 497 | patent against the party. 498 | 499 | If you convey a covered work, knowingly relying on a patent license, 500 | and the Corresponding Source of the work is not available for anyone 501 | to copy, free of charge and under the terms of this License, through a 502 | publicly available network server or other readily accessible means, 503 | then you must either (1) cause the Corresponding Source to be so 504 | available, or (2) arrange to deprive yourself of the benefit of the 505 | patent license for this particular work, or (3) arrange, in a manner 506 | consistent with the requirements of this License, to extend the patent 507 | license to downstream recipients. "Knowingly relying" means you have 508 | actual knowledge that, but for the patent license, your conveying the 509 | covered work in a country, or your recipient's use of the covered work 510 | in a country, would infringe one or more identifiable patents in that 511 | country that you have reason to believe are valid. 512 | 513 | If, pursuant to or in connection with a single transaction or 514 | arrangement, you convey, or propagate by procuring conveyance of, a 515 | covered work, and grant a patent license to some of the parties 516 | receiving the covered work authorizing them to use, propagate, modify 517 | or convey a specific copy of the covered work, then the patent license 518 | you grant is automatically extended to all recipients of the covered 519 | work and works based on it. 520 | 521 | A patent license is "discriminatory" if it does not include within 522 | the scope of its coverage, prohibits the exercise of, or is 523 | conditioned on the non-exercise of one or more of the rights that are 524 | specifically granted under this License. You may not convey a covered 525 | work if you are a party to an arrangement with a third party that is 526 | in the business of distributing software, under which you make payment 527 | to the third party based on the extent of your activity of conveying 528 | the work, and under which the third party grants, to any of the 529 | parties who would receive the covered work from you, a discriminatory 530 | patent license (a) in connection with copies of the covered work 531 | conveyed by you (or copies made from those copies), or (b) primarily 532 | for and in connection with specific products or compilations that 533 | contain the covered work, unless you entered into that arrangement, 534 | or that patent license was granted, prior to 28 March 2007. 535 | 536 | Nothing in this License shall be construed as excluding or limiting 537 | any implied license or other defenses to infringement that may 538 | otherwise be available to you under applicable patent law. 539 | 540 | 12. No Surrender of Others' Freedom. 541 | 542 | If conditions are imposed on you (whether by court order, agreement or 543 | otherwise) that contradict the conditions of this License, they do not 544 | excuse you from the conditions of this License. If you cannot convey a 545 | covered work so as to satisfy simultaneously your obligations under this 546 | License and any other pertinent obligations, then as a consequence you may 547 | not convey it at all. For example, if you agree to terms that obligate you 548 | to collect a royalty for further conveying from those to whom you convey 549 | the Program, the only way you could satisfy both those terms and this 550 | License would be to refrain entirely from conveying the Program. 551 | 552 | 13. Use with the GNU Affero General Public License. 553 | 554 | Notwithstanding any other provision of this License, you have 555 | permission to link or combine any covered work with a work licensed 556 | under version 3 of the GNU Affero General Public License into a single 557 | combined work, and to convey the resulting work. The terms of this 558 | License will continue to apply to the part which is the covered work, 559 | but the special requirements of the GNU Affero General Public License, 560 | section 13, concerning interaction through a network will apply to the 561 | combination as such. 562 | 563 | 14. Revised Versions of this License. 564 | 565 | The Free Software Foundation may publish revised and/or new versions of 566 | the GNU General Public License from time to time. Such new versions will 567 | be similar in spirit to the present version, but may differ in detail to 568 | address new problems or concerns. 569 | 570 | Each version is given a distinguishing version number. If the 571 | Program specifies that a certain numbered version of the GNU General 572 | Public License "or any later version" applies to it, you have the 573 | option of following the terms and conditions either of that numbered 574 | version or of any later version published by the Free Software 575 | Foundation. If the Program does not specify a version number of the 576 | GNU General Public License, you may choose any version ever published 577 | by the Free Software Foundation. 578 | 579 | If the Program specifies that a proxy can decide which future 580 | versions of the GNU General Public License can be used, that proxy's 581 | public statement of acceptance of a version permanently authorizes you 582 | to choose that version for the Program. 583 | 584 | Later license versions may give you additional or different 585 | permissions. However, no additional obligations are imposed on any 586 | author or copyright holder as a result of your choosing to follow a 587 | later version. 588 | 589 | 15. Disclaimer of Warranty. 590 | 591 | THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY 592 | APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT 593 | HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY 594 | OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, 595 | THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR 596 | PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM 597 | IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF 598 | ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 599 | 600 | 16. Limitation of Liability. 601 | 602 | IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING 603 | WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS 604 | THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY 605 | GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE 606 | USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF 607 | DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD 608 | PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), 609 | EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF 610 | SUCH DAMAGES. 611 | 612 | 17. Interpretation of Sections 15 and 16. 613 | 614 | If the disclaimer of warranty and limitation of liability provided 615 | above cannot be given local legal effect according to their terms, 616 | reviewing courts shall apply local law that most closely approximates 617 | an absolute waiver of all civil liability in connection with the 618 | Program, unless a warranty or assumption of liability accompanies a 619 | copy of the Program in return for a fee. 620 | 621 | END OF TERMS AND CONDITIONS 622 | 623 | How to Apply These Terms to Your New Programs 624 | 625 | If you develop a new program, and you want it to be of the greatest 626 | possible use to the public, the best way to achieve this is to make it 627 | free software which everyone can redistribute and change under these terms. 628 | 629 | To do so, attach the following notices to the program. It is safest 630 | to attach them to the start of each source file to most effectively 631 | state the exclusion of warranty; and each file should have at least 632 | the "copyright" line and a pointer to where the full notice is found. 633 | 634 | 635 | Copyright (C) 636 | 637 | This program is free software: you can redistribute it and/or modify 638 | it under the terms of the GNU General Public License as published by 639 | the Free Software Foundation, either version 3 of the License, or 640 | (at your option) any later version. 641 | 642 | This program is distributed in the hope that it will be useful, 643 | but WITHOUT ANY WARRANTY; without even the implied warranty of 644 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 645 | GNU General Public License for more details. 646 | 647 | You should have received a copy of the GNU General Public License 648 | along with this program. If not, see . 649 | 650 | Also add information on how to contact you by electronic and paper mail. 651 | 652 | If the program does terminal interaction, make it output a short 653 | notice like this when it starts in an interactive mode: 654 | 655 | Copyright (C) 656 | This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. 657 | This is free software, and you are welcome to redistribute it 658 | under certain conditions; type `show c' for details. 659 | 660 | The hypothetical commands `show w' and `show c' should show the appropriate 661 | parts of the General Public License. Of course, your program's commands 662 | might be different; for a GUI interface, you would use an "about box". 663 | 664 | You should also get your employer (if you work as a programmer) or school, 665 | if any, to sign a "copyright disclaimer" for the program, if necessary. 666 | For more information on this, and how to apply and follow the GNU GPL, see 667 | . 668 | 669 | The GNU General Public License does not permit incorporating your program 670 | into proprietary programs. If your program is a subroutine library, you 671 | may consider it more useful to permit linking proprietary applications with 672 | the library. If this is what you want to do, use the GNU Lesser General 673 | Public License instead of this License. But first, please read 674 | . 675 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include readme.md 2 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | doc: 2 | doxygen Doxyfile 3 | 4 | clean: 5 | rm -f doxygen_sqlite3.db 6 | rm -rf html 7 | rm -rf latex 8 | 9 | .PHONY: doc clean 10 | -------------------------------------------------------------------------------- /doc/changelog.md: -------------------------------------------------------------------------------- 1 | # mqspeak changelog 2 | 3 | ## v0.1.0 4 | 5 | - Initial release. 6 | - Implemented sending data to [TingSpeak](https://thingspeak.com/) servers. 7 | - Implemented sending data to [phant](http://phant.io/) servers. 8 | - blackout, average and buffered updates. 9 | - Supported MQTT user authentication. 10 | - Supported updating multiple channels. 11 | - Supported receiving data from multiple brokers. 12 | - Configured with .ini style config file. 13 | 14 | ## v0.2.0 15 | 16 | - Mutexes are always released in finally block. Avoiding possible deadlocks. 17 | - Changed setup.py long description to content of README.md file. 18 | - Added channel waiting mechanism. 19 | - Logging to syslog. 20 | - Added `-o` option which instruct mqspeak to log to stdout instead to syslog. 21 | - Average updater now calculates a average value from all MQTT topic updates. 22 | Not only from complete measurement. 23 | - Implemented `onchange` updater. 24 | 25 | ## v0.2.0 26 | 27 | - Fixed python package. 28 | -------------------------------------------------------------------------------- /doc/todo.md: -------------------------------------------------------------------------------- 1 | # mqspeak todo list 2 | 3 | - Implement partial update. Useful when channel update is composed from data of 4 | multiple sensors. When single sensor dies, any channel updates will be stopped. 5 | - Implement some logging. 6 | -------------------------------------------------------------------------------- /mqspeak.conf: -------------------------------------------------------------------------------- 1 | [Brokers] 2 | Enabled = temperature-broker humidity-broker door-broker 3 | 4 | [temperature-broker] 5 | Host = temperatureBrokerHostname 6 | Port = 1883 7 | User = brokerUser 8 | Password = brokerPass 9 | Topic = sensors/temperature sensors/something 10 | 11 | [humidity-broker] 12 | Host = humidityBrokerHostname 13 | Port = 1883 14 | User = brokerUser 15 | Password = brokerPass 16 | Topic = sensors/humidity 17 | 18 | [door-broker] 19 | Host = doorBrokerHostname 20 | Port = 1883 21 | User = brokerUser 22 | Password = brokerPass 23 | Topic = # 24 | 25 | [Channels] 26 | Enabled = channel1 channel2 channel3 channel4 27 | 28 | [channel1] 29 | Id = CHANNELID 30 | Key = CHANNELKEY 31 | Type = thingspeak 32 | UpdateRate = 15 33 | UpdateType = blackout 34 | UpdateFields = dht-update 35 | 36 | [channel2] 37 | Id = CHANNELID 38 | Key = CHANNELKEY 39 | Type = thingspeak 40 | UpdateRate = 15 41 | UpdateType = buffered 42 | UpdateFields = dht-update 43 | 44 | [channel3] 45 | Id = CHANNELID 46 | Key = CHANNELKEY 47 | Type = thingspeak 48 | UpdateRate = 15 49 | UpdateType = average 50 | UpdateFields = dht-update 51 | 52 | [channel4] 53 | Id = CHANNELID 54 | Key = CHANNELKEY 55 | Type = phant 56 | UpdateRate = 15 57 | UpdateType = onchange 58 | UpdateFields = door-update 59 | 60 | [dht-update] 61 | field1 = humidity-broker sensors/humidity 62 | field2 = temperature-broker sensors/temperature 63 | 64 | [door-update] 65 | state = door-broker sensors/door 66 | -------------------------------------------------------------------------------- /mqspeak.service: -------------------------------------------------------------------------------- 1 | [Unit] 2 | Description=MQTT to ThingSpeak bridge 3 | After=network-online.target 4 | Wants=network-online.target 5 | 6 | [Service] 7 | Type=simple 8 | ExecStart=/usr/local/bin/mqspeak 9 | Restart=always 10 | 11 | [Install] 12 | WantedBy=multi-user.target 13 | -------------------------------------------------------------------------------- /mqspeak/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (C) Ivo Slanina 2 | # 3 | # This program is free software: you can redistribute it and/or modify 4 | # it under the terms of the GNU General Public License as published by 5 | # the Free Software Foundation, either version 3 of the License, or 6 | # (at your option) any later version. 7 | # 8 | # This program is distributed in the hope that it will be useful, 9 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 10 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 11 | # GNU General Public License for more details. 12 | # 13 | # You should have received a copy of the GNU General Public License 14 | # along with this program. If not, see . 15 | 16 | __version__ = "0.2.1" 17 | __author__ = "Ivo Slanina" 18 | __email__ = "ivo.slanina@gmail.com" 19 | __url__ = "https://github.com/mqopen/mqspeak" 20 | __project_name__ = "mqopen" 21 | __project_url__ = "http://mqopen.org/" 22 | -------------------------------------------------------------------------------- /mqspeak/__main__.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # Copyright (C) Ivo Slanina 3 | # 4 | # This program is free software: you can redistribute it and/or modify 5 | # it under the terms of the GNU General Public License as published by 6 | # the Free Software Foundation, either version 3 of the License, or 7 | # (at your option) any later version. 8 | # 9 | # This program is distributed in the hope that it will be useful, 10 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 11 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 12 | # GNU General Public License for more details. 13 | # 14 | # You should have received a copy of the GNU General Public License 15 | # along with this program. If not, see . 16 | 17 | from mqreceive.receiving import BrokerThreadManager 18 | from mqspeak.sending import ChannelUpdateDispatcher 19 | from mqspeak.system import System 20 | from mqspeak.updating import ChannnelUpdateSupervisor 21 | 22 | def main(): 23 | System.initialize() 24 | 25 | # Channel update dispatcher object 26 | channelConvertMapping = System.getChannelConvertMapping() 27 | updateDispatcher = ChannelUpdateDispatcher(channelConvertMapping) 28 | 29 | channelUpdateSupervisor = ChannnelUpdateSupervisor(System.getChannelUpdateMapping()) 30 | channelUpdateSupervisor.setDispatcher(updateDispatcher) 31 | 32 | # MQTT cliens 33 | brokerManager = BrokerThreadManager(System.getBrokerListenDescriptors(), channelUpdateSupervisor) 34 | 35 | # run all MQTT client threads 36 | brokerManager.start() 37 | 38 | # run main thread 39 | try: 40 | updateDispatcher.run() 41 | except KeyboardInterrupt as ex: 42 | 43 | # program exit 44 | channelUpdateSupervisor.stop() 45 | brokerManager.stop() 46 | updateDispatcher.stop() 47 | 48 | if __name__ == '__main__': 49 | try: 50 | main() 51 | except KeyboardInterrupt as ex: 52 | pass 53 | -------------------------------------------------------------------------------- /mqspeak/args.py: -------------------------------------------------------------------------------- 1 | # Copyright (C) Ivo Slanina 2 | # 3 | # This program is free software: you can redistribute it and/or modify 4 | # it under the terms of the GNU General Public License as published by 5 | # the Free Software Foundation, either version 3 of the License, or 6 | # (at your option) any later version. 7 | # 8 | # This program is distributed in the hope that it will be useful, 9 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 10 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 11 | # GNU General Public License for more details. 12 | # 13 | # You should have received a copy of the GNU General Public License 14 | # along with this program. If not, see . 15 | 16 | import os 17 | import argparse 18 | import mqspeak 19 | 20 | class HelpFormatter(argparse.ArgumentDefaultsHelpFormatter): 21 | def __init__(self, prog, indent_increment=2, max_help_position=64, width=180): 22 | argparse.ArgumentDefaultsHelpFormatter.__init__(self, prog, indent_increment, max_help_position, width) 23 | 24 | def create_parser(): 25 | parser = argparse.ArgumentParser( 26 | description="MQTT bridge v{}".format(mqspeak.__version__), 27 | epilog="Copyright (C) {} <{}> {} ".format( 28 | mqspeak.__author__, 29 | mqspeak.__email__, 30 | mqspeak.__project_url__), 31 | formatter_class=HelpFormatter) 32 | parser.add_argument('-c', '--config', 33 | help='path to configuration file', 34 | default="/etc/mqspeak.conf") 35 | parser.add_argument('-v', '--verbose', 36 | help='verbose output', 37 | action='store_true') 38 | parser.add_argument('-o', '--log-stdout', 39 | help='log to stdout instead to syslog', 40 | action='store_true') 41 | parser.add_argument('--version', 42 | action='version', 43 | version='{}'.format(mqspeak.__version__)) 44 | return parser 45 | 46 | def parse_args(): 47 | parser = create_parser() 48 | return parser.parse_args() 49 | -------------------------------------------------------------------------------- /mqspeak/channel.py: -------------------------------------------------------------------------------- 1 | # Copyright (C) Ivo Slanina 2 | # 3 | # This program is free software: you can redistribute it and/or modify 4 | # it under the terms of the GNU General Public License as published by 5 | # the Free Software Foundation, either version 3 of the License, or 6 | # (at your option) any later version. 7 | # 8 | # This program is distributed in the hope that it will be useful, 9 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 10 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 11 | # GNU General Public License for more details. 12 | # 13 | # You should have received a copy of the GNU General Public License 14 | # along with this program. If not, see . 15 | 16 | import enum 17 | 18 | class ChannelType(enum.Enum): 19 | """! 20 | Enumeration of channel types. 21 | """ 22 | 23 | thingspeak = 0 24 | phant = 1 25 | 26 | class Channel: 27 | """! 28 | ThingSpeak channel identification object. 29 | """ 30 | 31 | ## @var channelType 32 | # Type of channel. 33 | 34 | ## @var name 35 | # Channel name. 36 | 37 | ## @var channelID 38 | # Channel ID or None. 39 | 40 | ## @var apiKey 41 | # Channel API write key. 42 | 43 | ## @var waiting 44 | # Channel waiting timedelta object or None if waiting is disabled. 45 | 46 | def __init__(self, channelType, name, channelID, apiKey, waiting): 47 | """! 48 | Initiate channel object. 49 | 50 | @param channelType ChannelType enumeration object. 51 | @param name Channel name. 52 | @param channelID Channel identification. 53 | @param apiKey Channel write API key. 54 | @param waiting Channel waiting timedelta object of None 55 | """ 56 | self.channelType = channelType 57 | self.name = name 58 | self.channelID = channelID 59 | self.apiKey = apiKey 60 | self.waiting = waiting 61 | 62 | def __hash__(self): 63 | """! 64 | Calculate hash from channel name and API key. 65 | 66 | @return Hash. 67 | """ 68 | return hash((self.name, self.apiKey)) 69 | 70 | def __str__(self): 71 | """! 72 | Convert Channel to string. 73 | 74 | @return String. 75 | """ 76 | return "{} [Id: {}, Key: {}]".format(self.name, self.channelID, self.apiKey) 77 | 78 | def __repr__(self): 79 | """! 80 | Convert Channel to representation string. 81 | 82 | @return Representation string. 83 | """ 84 | return "<{}>".format(self.__str__()) 85 | 86 | def hasWaiting(self): 87 | """! 88 | Check if channel has waiting enabled. 89 | 90 | @return True if waiting is enabled, False otherwise. 91 | """ 92 | return self.waiting is not None 93 | 94 | class ThingSpeakChannel(Channel): 95 | """! 96 | ThingSpeak channel identification object. 97 | """ 98 | 99 | def __init__(self, name, channelID, apiKey, waiting): 100 | """! 101 | Initiate ThingSpeak channel object. 102 | 103 | @param name Channel name. 104 | @param channelID Channel identification. 105 | @param apiKey Channel write key. 106 | """ 107 | Channel.__init__(self, ChannelType.thingspeak, name, channelID, apiKey, waiting) 108 | 109 | class PhantChannel(Channel): 110 | """ 111 | Phant channel identification object. 112 | """ 113 | 114 | def __init__(self, name, channelID, apiKey, waiting): 115 | """! 116 | Initiate Phant channel object. 117 | 118 | @param name Channel name. 119 | @param channelID Channel identification. 120 | @param apiKey Channel write key. 121 | """ 122 | self.checkChannelIdentification(channelID) 123 | Channel.__init__(self, ChannelType.phant, name, channelID, apiKey, waiting) 124 | 125 | def checkChannelIdentification(self, channelID): 126 | """! 127 | Check if channelID is not None 128 | 129 | @throws ChannelException if channelID is None. 130 | """ 131 | if channelID is None: 132 | raise ChannelException("Phant channel must have identification") 133 | 134 | class ChannelException: 135 | """! 136 | Channel related exceptions. 137 | """ 138 | -------------------------------------------------------------------------------- /mqspeak/collecting.py: -------------------------------------------------------------------------------- 1 | # Copyright (C) Ivo Slanina 2 | # 3 | # This program is free software: you can redistribute it and/or modify 4 | # it under the terms of the GNU General Public License as published by 5 | # the Free Software Foundation, either version 3 of the License, or 6 | # (at your option) any later version. 7 | # 8 | # This program is distributed in the hope that it will be useful, 9 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 10 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 11 | # GNU General Public License for more details. 12 | # 13 | # You should have received a copy of the GNU General Public License 14 | # along with this program. If not, see . 15 | 16 | import logging 17 | import collections 18 | import copy 19 | from mqreceive.data import DataIdentifier 20 | from mqspeak.data import Measurement 21 | 22 | class BaseUpdateBuffer: 23 | """! 24 | Class for buffering required data set before sending them out. 25 | """ 26 | 27 | ## @var dataIdentifiers 28 | # Iterable of DataIdentifier objects. 29 | 30 | def __init__(self, dataIdentifiers): 31 | """! 32 | Initiate UpdateBuffer object. 33 | 34 | @param dataIdentifiers Iterable of DataIdentifier objects. 35 | """ 36 | self.dataIdentifiers = dataIdentifiers 37 | 38 | def isComplete(self): 39 | """! 40 | Check if all required data are buffered. 41 | 42 | @return True if all required data are buffered, False otherwise. 43 | """ 44 | raise NotImplementedError("Override this mehod in sub-class") 45 | 46 | def isUpdateRelevant(self, dataIdentifier): 47 | """! 48 | Check if update is relevant to this channel. 49 | 50 | @param dataIdentifier Update data identifier. 51 | @return True if update is relevant, False otherwise 52 | """ 53 | return dataIdentifier in self.dataIdentifiers 54 | 55 | def updateReceivedData(self, dataIdentifier, value): 56 | """! 57 | Update received data. 58 | 59 | @param dataIdentifier Data identification. 60 | @param value Data content. 61 | @throws TopicException If unwanted topic is updated. 62 | """ 63 | raise NotImplementedError("Override this mehod in sub-class") 64 | 65 | def getData(self): 66 | """! 67 | Get dictionary with buffered data. Override in subclass. 68 | 69 | @return Buffered data. 70 | """ 71 | raise NotImplementedError("Override this mehod in sub-class") 72 | 73 | def getMeasurement(self): 74 | """! 75 | Get current measurement with stored data. 76 | 77 | @return Measureent object. 78 | """ 79 | raise NotImplementedError("Override this mehod in sub-class") 80 | 81 | def getMissingDataIdentifiers(self): 82 | """! 83 | Get iterable of data identifiers which doesn't have stored data. 84 | """ 85 | raise NotImplementedError("Override this mehod in sub-class") 86 | 87 | def hasAnyData(self): 88 | """! 89 | Check if udate buffer has stored any data. 90 | 91 | @return True if there are stored any data, False otherwise. 92 | """ 93 | raise NotImplementedError("Override this mehod in sub-class") 94 | 95 | def reset(self): 96 | """! 97 | Clear buffered data. 98 | """ 99 | raise NotImplementedError("Override this mehod in sub-class") 100 | 101 | def __str__(self): 102 | """! 103 | Convert UpdateBuffer object to string. 104 | 105 | @return String. 106 | """ 107 | return "{}({})".format(self.__class__.__name__, self.dataIdentifiers) 108 | 109 | def __repr__(self): 110 | """! 111 | Convert UpdateBuffer object to representation string. 112 | 113 | @return Representation string. 114 | """ 115 | return "<{}>".format(self.__str__()) 116 | 117 | class SingleValueUpdateBuffer(BaseUpdateBuffer): 118 | """! 119 | Base class for update buffers which are update over time and they store a 120 | single value. 121 | """ 122 | 123 | ## @var dataMapping 124 | # The {DataIdentifier: value} mapping. 125 | 126 | ## @var hasData 127 | # Boolean inicated that buffer stores any data. Private. 128 | 129 | def __init__(self, dataIdentifiers): 130 | BaseUpdateBuffer.__init__(self, dataIdentifiers) 131 | self.dataMapping = {} 132 | for dataIdentifier in dataIdentifiers: 133 | self.dataMapping[dataIdentifier] = None 134 | self.hasData = False 135 | 136 | def updateReceivedData(self, dataIdentifier, value): 137 | if not self.isUpdateRelevant(dataIdentifier): 138 | raise TopicException("Illegal topic update: {}".format(dataIdentifier)) 139 | self.handleUpdateReceivedData(dataIdentifier, value) 140 | self.hasData = True 141 | 142 | def handleUpdateReceivedData(self, dataIdentifier, value): 143 | """! 144 | Update received data. 145 | 146 | @param dataIdentifier Data identification. 147 | @param value Data content. 148 | """ 149 | raise NotImplementedError("Override this mehod in sub-class") 150 | 151 | def isComplete(self): 152 | return not any(x is None for x in self.dataMapping.values()) 153 | 154 | def getMeasurement(self): 155 | return Measurement.currentMeasurement(self.getData()) 156 | 157 | def getMissingDataIdentifiers(self): 158 | for dataIdentifier, value in self.dataMapping.items(): 159 | if value is None: 160 | yield dataIdentifier 161 | 162 | def hasAnyData(self): 163 | return self.hasData 164 | 165 | def reset(self): 166 | for dataIdentifier in self.dataMapping: 167 | self.dataMapping[dataIdentifier] = None 168 | self.hasData = False 169 | 170 | class LastValueUpdateBuffer(SingleValueUpdateBuffer): 171 | """! 172 | Keeps only last value. When some value is updated, the preveous value is lost. 173 | 174 | Updater can hold any kind of data. 175 | """ 176 | 177 | def handleUpdateReceivedData(self, dataIdentifier, value): 178 | self.dataMapping[dataIdentifier] = value 179 | 180 | def getData(self): 181 | return copy.deepcopy(self.dataMapping) 182 | 183 | class AverageUpdateBuffer(SingleValueUpdateBuffer): 184 | """! 185 | Calculate arithmetic average value. Each new value is stored in internal buffer. 186 | getData() method returns data mapping with calculated average value. 187 | """ 188 | 189 | def handleUpdateReceivedData(self, dataIdentifier, value): 190 | try: 191 | value = float(value) 192 | if self.dataMapping[dataIdentifier] is None: 193 | self.dataMapping[dataIdentifier] = [] 194 | self.dataMapping[dataIdentifier].append(value) 195 | except ValueError as ex: 196 | raise ValueError("Can't convert data to number: {}".format(value)) 197 | 198 | def getData(self): 199 | mapping = {} 200 | for dataIdentifier, valueList in self.dataMapping.items(): 201 | if valueList is not None: 202 | mapping[dataIdentifier] = float(sum(valueList)) / len(valueList) 203 | else: 204 | mapping[dataIdentifier] = None 205 | return mapping 206 | 207 | class ChangeValueBuffer(BaseUpdateBuffer): 208 | """! 209 | Store all change updates. 210 | """ 211 | 212 | def __init__(self, dataIdentifiers): 213 | BaseUpdateBuffer.__init__(self, dataIdentifiers) 214 | self.lastValueMapping = {} 215 | for dataIdentifier in dataIdentifiers: 216 | self.lastValueMapping[dataIdentifier] = None 217 | self.measurementBuffer = collections.deque() 218 | 219 | def updateReceivedData(self, dataIdentifier, value): 220 | if self.lastValueMapping[dataIdentifier] is None or self.lastValueMapping[dataIdentifier] != value: 221 | self.lastValueMapping[dataIdentifier] = value 222 | measurement = Measurement.currentMeasurement({dataIdentifier: value}) 223 | self.measurementBuffer.append(measurement) 224 | else: 225 | logging.getLogger().error( 226 | "New data are equals to previous one ({}: {}). Skipping...".format(dataIdentifier, repr(value))) 227 | 228 | def reset(self): 229 | self.measurementBuffer.popleft() 230 | 231 | def getMeasurement(self): 232 | return self.measurementBuffer[0] 233 | 234 | def hasAnyData(self): 235 | return len(self.measurementBuffer) > 0 236 | 237 | def isComplete(self): 238 | return self.hasAnyData() 239 | 240 | class TopicException(Exception): 241 | """! 242 | Update buffer related errors. 243 | """ 244 | -------------------------------------------------------------------------------- /mqspeak/config.py: -------------------------------------------------------------------------------- 1 | # Copyright (C) Ivo Slanina 2 | # 3 | # This program is free software: you can redistribute it and/or modify 4 | # it under the terms of the GNU General Public License as published by 5 | # the Free Software Foundation, either version 3 of the License, or 6 | # (at your option) any later version. 7 | # 8 | # This program is distributed in the hope that it will be useful, 9 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 10 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 11 | # GNU General Public License for more details. 12 | # 13 | # You should have received a copy of the GNU General Public License 14 | # along with this program. If not, see . 15 | 16 | import configparser 17 | import datetime 18 | from mqreceive.broker import Broker 19 | from mqspeak.channel import ThingSpeakChannel, PhantChannel 20 | from mqreceive.data import DataIdentifier 21 | from mqspeak.updating import BlackoutUpdater, BufferedUpdater, AverageUpdater, OnChangeUpdater 22 | 23 | class ProgramConfig: 24 | """! 25 | Program configuration parser. 26 | """ 27 | 28 | ## @var configFile 29 | # Configuration file name. 30 | 31 | ## @var parser 32 | # Parser object. 33 | 34 | def __init__(self, configFile): 35 | """! 36 | Initiate program configuration object. 37 | 38 | @param configFile Path to configuration file. 39 | """ 40 | self.configFile = configFile 41 | self.parser = configparser.ConfigParser() 42 | 43 | def parse(self): 44 | """! 45 | Parse config file. 46 | 47 | @return Configuration object. 48 | """ 49 | self.parser.read(self.configFile) 50 | self.checkForMandatorySections() 51 | configCache = ConfigCache() 52 | for broker, subscriptions in self.getBrokers(): 53 | configCache.addBroker(broker, subscriptions) 54 | for channel, updaterFactory, updateMappingFactory in self.getChannels(): 55 | updateMapping = updateMappingFactory.build(configCache) 56 | updater = updaterFactory.build(channel, updateMapping) 57 | configCache.addChannel(channel, updater, updateMapping) 58 | return configCache 59 | 60 | def checkForMandatorySections(self): 61 | """! 62 | Check if all necessary sections are mandatory. 63 | 64 | @throws ConfigException if some section is missing. 65 | """ 66 | self.checkForSectionList(["Brokers", "Channels"]) 67 | 68 | def getBrokers(self): 69 | """! 70 | Get list of enabled brokers. 71 | 72 | @return Iterable of brokers. 73 | """ 74 | section = "Brokers" 75 | self.checkForEnabledOption(section) 76 | brokerSections = self.parser.get(section, "Enabled").split() 77 | self.checkForSectionList(brokerSections) 78 | for brokerSection in brokerSections: 79 | self.checkForBrokerMandatoryOptions(brokerSection) 80 | broker = self.createBroker(brokerSection) 81 | subscriptions = self.getBrokerSubscribtions(brokerSection) 82 | yield broker, subscriptions 83 | 84 | def checkForBrokerMandatoryOptions(self, brokerSection): 85 | """! 86 | Check for mandatory options of broker section. 87 | 88 | @param brokerSection Broker section name. 89 | @throws ConfigException If some option is missing. 90 | """ 91 | optionList = ["Topic"] 92 | self.checkForOptionList(brokerSection, optionList) 93 | 94 | def createBroker(self, brokerSection): 95 | """! 96 | Create broker object from broker section. 97 | 98 | @param brokerSection Broker section name. 99 | @return Broker object 100 | """ 101 | try: 102 | options = self.parser.options(brokerSection) 103 | host = self.parser.get(brokerSection, "Host", fallback = "127.0.0.1") 104 | port = self.parser.getint(brokerSection, "Port", fallback = 1883) 105 | broker = Broker(brokerSection, host, port) 106 | if "user" in options or "password" in options: 107 | (user, password) = self.getBrokerCredentials(brokerSection) 108 | broker.setCredentials(user, password) 109 | return broker 110 | except ValueError as ex: 111 | raise ConfigException("Invalid broker port number: {}".format(self.parser.get(brokerSection, "Port"))) 112 | 113 | def getBrokerCredentials(self, brokerSection): 114 | """! 115 | Get username and password of broker. 116 | 117 | @param brokerSection Broker section name. 118 | @return Tuple of (username, password). 119 | @throws ConfigException If username or password is missing in configuration file. 120 | """ 121 | user = None 122 | password = None 123 | try: 124 | user = self.parser.get(brokerSection, "User") 125 | except configparser.NoOptionError as ex: 126 | raise ConfigException("Section {}: User option is missing".format(brokerSection)) 127 | try: 128 | password = self.parser.get(brokerSection, "Password") 129 | except configparser.NoOptionError as ex: 130 | raise ConfigException("Section {}: Password option is missing".format(brokerSection)) 131 | return user, password 132 | 133 | def getBrokerSubscribtions(self, brokerSection): 134 | """! 135 | Get list of broker subscribe topics. 136 | 137 | @param brokerSection Broker section name. 138 | @return List of broker subscribe topics. 139 | @throws ConfigException If zero topics are specified. 140 | """ 141 | subscriptions = self.parser.get(brokerSection, "Topic").split() 142 | if len(subscriptions) == 0: 143 | raise ConfigException("At least one topic subscribe has to be defined") 144 | return subscriptions 145 | 146 | def getChannels(self): 147 | """! 148 | Create list of enable channels. 149 | 150 | @return Iterable of tuples of (channel, updaterFactory, updateMappingFactory). 151 | @throws ConfigException 152 | """ 153 | section = "Channels" 154 | self.checkForEnabledOption(section) 155 | channelSections = self.parser.get(section, "Enabled").split() 156 | self.checkForSectionList(channelSections) 157 | for channelSection in channelSections: 158 | self.checkForChannelMandatoryOptions(channelSection) 159 | channel = self.createChannel(channelSection) 160 | updaterFactory = self.getChannelUpdater(channelSection) 161 | updateMappingFactory = self.getDataFieldMapping(channelSection) 162 | yield channel, updaterFactory, updateMappingFactory 163 | 164 | def checkForChannelMandatoryOptions(self, channelSection): 165 | """! 166 | Check if channel section has all mandatoty options. 167 | 168 | @param channelSection Channel section name. 169 | @throws ConfigException If some options are missing. 170 | """ 171 | optionList = ["Key", "Type", "UpdateRate", "UpdateType", "UpdateFields"] 172 | self.checkForOptionList(channelSection, optionList) 173 | 174 | def createChannel(self, channelSection): 175 | """! 176 | Create channel object. 177 | 178 | @param channelSection Channel section name. 179 | @return Channel object. 180 | @throws ConfigException If configuration specifies unknown channel type. 181 | """ 182 | channelID = self.parser.get(channelSection, "Id", fallback = None) 183 | writeKey = self.parser.get(channelSection, "Key") 184 | channelType = self.parser.get(channelSection, "Type") 185 | waitInterval = None 186 | try: 187 | waitInterval = self.parser.getint(channelSection, "WaitInterval", fallback = None) 188 | except ValueError as ex: 189 | raise ConfigException("Channel {} - WaitInterval: {}".format(channelSection, self.parser.get(channelSection, "WaitInterval"))) 190 | if waitInterval is not None: 191 | waitInterval = datetime.timedelta(seconds = waitInterval) 192 | 193 | if channelType == "thingspeak": 194 | return ThingSpeakChannel(channelSection, channelID, writeKey, waitInterval) 195 | elif channelType == "phant": 196 | return PhantChannel(channelSection, channelID, writeKey, waitInterval) 197 | else: 198 | raise ConfigException("Unsupported channel type: {}".format(channelType)) 199 | 200 | def getChannelUpdater(self, channelSection): 201 | """! 202 | Create channel updaterFactory. 203 | 204 | @param channelSection Channel section name. 205 | @return ChannelUpdaterFactory object for that channel. 206 | @throws ConfigException If channel specifies invalid update interval. 207 | """ 208 | try: 209 | updateRate = datetime.timedelta(seconds = self.parser.getint(channelSection, "UpdateRate")) 210 | updaterName = self.parser.get(channelSection, "UpdateType") 211 | updaterCls, updaterArgs = self.createUpdaterFactory(updaterName, updateRate) 212 | return ChannelUpdaterFactory(updaterCls, updaterArgs) 213 | except ValueError as ex: 214 | raise ConfigException("Invalid update rate interval: {}".format(self.parser.get(channelSection, "UpdateRate"))) 215 | 216 | def createUpdaterFactory(self, updaterName, updateRate): 217 | """! 218 | Create updater factory based on updater name. 219 | 220 | @param updaterName Updater name. 221 | @param updateRate Update interval. 222 | @return ChannelUpdaterFactory object. 223 | @throws ConfigException If unknown updater name is specified in config file. 224 | """ 225 | updaterCls = None 226 | if updaterName == "blackout": 227 | updaterCls = BlackoutUpdater 228 | elif updaterName == "buffered": 229 | updaterCls = BufferedUpdater 230 | elif updaterName == "average": 231 | updaterCls = AverageUpdater 232 | elif updaterName == "onchange": 233 | updaterCls = OnChangeUpdater 234 | else: 235 | raise ConfigException("Unknown UpdateType: {}".format(updaterName)) 236 | updaterArgs = (updateRate,) 237 | return updaterCls, updaterArgs 238 | 239 | def getDataFieldMapping(self, channelSection): 240 | """! 241 | Create field mapping for channel. 242 | 243 | @param channelSection Channel section name. 244 | @return Data field mapping. 245 | """ 246 | updaterSection = self.parser.get(channelSection, "UpdateFields") 247 | return self.createDataFieldMapping(updaterSection) 248 | 249 | def createDataFieldMapping(self, updateSection): 250 | """! 251 | Create data field mapping from update section. 252 | 253 | @param updateSection Update section name. 254 | @return Data field mapping. 255 | """ 256 | updateMappingFactory = UpdateMappingFactory() 257 | for mappingOption in self.parser.options(updateSection): 258 | optionValue = self.parser.get(updateSection, mappingOption).split() 259 | if len(optionValue) < 2: 260 | raise ConfigException("{}: {} - option must contain two space separated values".format(updateSection, mappingOption)) 261 | brokerName, topic = optionValue 262 | updateMappingFactory.addMapping(brokerName, topic, mappingOption) 263 | return updateMappingFactory 264 | 265 | def checkForEnabledOption(self, section): 266 | """! 267 | Check for "Enabled" option in given section. 268 | 269 | @param section Section name. 270 | @throws ConfigException If "Enabled" option is missing in section. 271 | """ 272 | self.checkForOption(section, "Enabled") 273 | 274 | def checkForSectionList(self, sectionList): 275 | """! 276 | Check for list of sections in configuration file. 277 | 278 | @param sectionList 279 | @throws ConfigException 280 | """ 281 | for section in sectionList: 282 | self.checkForSection(section) 283 | 284 | def checkForSection(self, section): 285 | """! 286 | Check for section in configuration file. 287 | 288 | @param section 289 | @throws ConfigException 290 | """ 291 | if not self.parser.has_section(section): 292 | raise ConfigException("{} section is missing".format(section)) 293 | 294 | def checkForOptionList(self, section, optionList): 295 | """! 296 | Check for list of options in single section. 297 | 298 | @param section 299 | @param optionList 300 | @throws ConfigException 301 | """ 302 | for option in optionList: 303 | self.checkForOption(section, option) 304 | 305 | def checkForOption(self, section, option): 306 | """! 307 | Check for option in configuration file. 308 | 309 | @param section Section name. 310 | @param option Option name. 311 | @throws ConfigException If given section doesn!t contain specified option. 312 | """ 313 | if not self.parser.has_option(section, option): 314 | raise ConfigException("Section {}: {} option is missing".format(section, option)) 315 | 316 | class ConfigCache: 317 | """! 318 | Cache object for storing app configuration. 319 | """ 320 | 321 | ## @var listenDescriptors 322 | # Listen descriptors. 323 | 324 | ## @var channelUpdateDescribtors 325 | # Update descriptors. 326 | 327 | def __init__(self): 328 | """! 329 | Initiate configuration cache object. 330 | """ 331 | self.listenDescriptors = [] 332 | self.channelUpdateDescribtors = [] 333 | 334 | def addBroker(self, broker, subscriptions): 335 | """! 336 | Add broker to configuration object. 337 | 338 | @param broker 339 | @param subscriptions 340 | """ 341 | listenDescriptor = (broker, subscriptions) 342 | self.listenDescriptors.append(listenDescriptor) 343 | 344 | def addChannel(self, channel, updater, updateMapping): 345 | """! 346 | Add channel to configuration object. 347 | 348 | @param channel 349 | @param updater 350 | @param updateMapping 351 | @throws ConfigException When update mapping contains unknown broker. 352 | """ 353 | channelUpdateDescribtor = (channel, updater, updateMapping) 354 | self.channelUpdateDescribtors.append(channelUpdateDescribtor) 355 | 356 | def check(self): 357 | """! 358 | @todo implement this method 359 | 360 | 1. warning for unused brokers 361 | 2. warning for not feasible topics in udate mappings. 362 | """ 363 | 364 | def getBrokerByName(self, brokerName): 365 | """! 366 | Get broker object identified by its't name. 367 | 368 | @param @brokerName 369 | @throws ConfigException If the name don't match to any stored broker object. 370 | """ 371 | for broker, subscriptions in self.listenDescriptors: 372 | if brokerName == broker.name: 373 | return broker 374 | raise ConfigException("Unknown broker name: {}".format(brokerName)) 375 | 376 | class ChannelUpdaterFactory: 377 | """! 378 | Build channel. 379 | """ 380 | 381 | ## @var updaterCls 382 | # Updater class. 383 | 384 | ## @var updaterArgs 385 | # Arguments to call updater class. 386 | 387 | def __init__(self, updaterCls, updaterArgs): 388 | """! 389 | Initiate ChannelUpdaterFactory object. 390 | 391 | @param updaterCls 392 | @param updaterArgs 393 | """ 394 | self.updaterCls = updaterCls 395 | self.updaterArgs = updaterArgs 396 | 397 | def build(self, channel, updateMapping): 398 | """! 399 | Build ChannelUpdater object. 400 | 401 | @param channel 402 | @return ChannelUpdater object. 403 | """ 404 | return self.updaterCls(channel, updateMapping, *self.updaterArgs) 405 | 406 | class UpdateMappingFactory: 407 | """! 408 | Build UpdateMapping object. 409 | """ 410 | 411 | ## @var mapping 412 | # Mapping. 413 | 414 | def __init__(self): 415 | """! 416 | Initiate UpdateMappingFactory object. 417 | """ 418 | self.mapping = {} 419 | 420 | def addMapping(self, brokerName, topic, field): 421 | """! 422 | Add build mapping. 423 | 424 | @param brokerName 425 | @param topic 426 | @param field 427 | """ 428 | self.checkNewBrokerName(brokerName) 429 | self.mapping[brokerName].append((topic, field)) 430 | 431 | def checkNewBrokerName(self, brokerName): 432 | """! 433 | Check if broker name exist in mapping. Create new if not. 434 | 435 | @param brokerName 436 | """ 437 | if brokerName not in self.mapping: 438 | self.mapping[brokerName] = [] 439 | 440 | def getNeededBrokers(self): 441 | """ 442 | Get list of needed brokers. 443 | 444 | @return List of broker names. 445 | """ 446 | return list(self.mapping.keys()) 447 | 448 | def build(self, brokerNameResolver): 449 | """! 450 | Build UpdateMapping object. 451 | 452 | @param brokerNameResolver Object for resolving broker names into Broker objects. 453 | @return UpdateMapping object. 454 | """ 455 | mapping = {} 456 | for brokerName in self.mapping.keys(): 457 | broker = brokerNameResolver.getBrokerByName(brokerName) 458 | for topic, field in self.mapping[brokerName]: 459 | dataIdentifier = DataIdentifier(broker, topic) 460 | mapping[dataIdentifier] = field 461 | return mapping 462 | 463 | class ConfigException(Exception): 464 | """! 465 | Exception raised during parsing configuration file 466 | """ 467 | -------------------------------------------------------------------------------- /mqspeak/data.py: -------------------------------------------------------------------------------- 1 | # Copyright (C) Ivo Slanina 2 | # 3 | # This program is free software: you can redistribute it and/or modify 4 | # it under the terms of the GNU General Public License as published by 5 | # the Free Software Foundation, either version 3 of the License, or 6 | # (at your option) any later version. 7 | # 8 | # This program is distributed in the hope that it will be useful, 9 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 10 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 11 | # GNU General Public License for more details. 12 | # 13 | # You should have received a copy of the GNU General Public License 14 | # along with this program. If not, see . 15 | 16 | import datetime 17 | 18 | class Measurement: 19 | """! 20 | Measured data mapping in DataIdentifier: value format with corresponding timestamp. 21 | """ 22 | 23 | ## @var fields 24 | # Measurement fields. 25 | 26 | ## @var time 27 | # Measurement timestamp. 28 | 29 | def __init__(self, fields, time): 30 | """! 31 | Initiate measurement object. 32 | 33 | @param fields Maping {dataIdentifier: vaue}. 34 | @param time Measurement timestamp. 35 | """ 36 | self.fields = fields 37 | self.time = time 38 | 39 | @classmethod 40 | def currentMeasurement(cls, fields): 41 | """! 42 | Build measurement object with current time. 43 | 44 | @param fields Maping {dataIdentifier: vaue}. 45 | """ 46 | return cls(fields, datetime.datetime.utcnow()) 47 | 48 | def __str__(self): 49 | """! 50 | Convert object to string. 51 | 52 | @return String. 53 | """ 54 | return "[{}] {}".format(self.time, self.fields) 55 | 56 | def __repr__(self): 57 | """! 58 | Convert object to representation string. 59 | 60 | @return representation string. 61 | """ 62 | return "<{}>".format(self.__str__()) 63 | 64 | def __len__(self): 65 | """! 66 | Get number of measurement fields. 67 | 68 | @return Number of fields. 69 | """ 70 | return len(self.fields) 71 | 72 | class MeasurementParamConverter: 73 | """! 74 | Convert data measurement into ThingSpeak fields for single channel. 75 | """ 76 | 77 | ## @var dataFieldsMapping 78 | # Mapping of data fields. 79 | 80 | def __init__(self, dataFieldsMapping): 81 | """! 82 | Initiate MeasurementParamConverter object. 83 | 84 | @param dataFieldsMapping Object for mapping DataIdentifier object to 85 | ThingSpeak channel field {DataIdentifier: "field"}. 86 | """ 87 | self.dataFieldsMapping = dataFieldsMapping 88 | 89 | def convert(self, measurement): 90 | """! 91 | Convert measurement. 92 | 93 | @param measurement Measurement object. 94 | @throws ConvertException If reguired measurement item is missing. 95 | """ 96 | params = dict() 97 | for topicName in measurement.fields: 98 | if measurement.fields[topicName] is not None: 99 | fieldName = self.dataFieldsMapping[topicName] 100 | params[fieldName] = measurement.fields[topicName] 101 | return params 102 | 103 | def __str__(self): 104 | """! 105 | Convert object to string. 106 | 107 | @return String. 108 | """ 109 | return "Param Converter: {}".format(str(self.dataFieldsMapping)) 110 | 111 | def __repr__(self): 112 | """! 113 | Convert object to representation string. 114 | 115 | @return representation string. 116 | """ 117 | return "<{}>".format(self.__str__()) 118 | 119 | class ConvertException(Exception): 120 | """! 121 | Conversion error. 122 | """ 123 | -------------------------------------------------------------------------------- /mqspeak/sending.py: -------------------------------------------------------------------------------- 1 | # Copyright (C) Ivo Slanina 2 | # 3 | # This program is free software: you can redistribute it and/or modify 4 | # it under the terms of the GNU General Public License as published by 5 | # the Free Software Foundation, either version 3 of the License, or 6 | # (at your option) any later version. 7 | # 8 | # This program is distributed in the hope that it will be useful, 9 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 10 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 11 | # GNU General Public License for more details. 12 | # 13 | # You should have received a copy of the GNU General Public License 14 | # along with this program. If not, see . 15 | 16 | import collections 17 | import datetime 18 | import http.client 19 | import threading 20 | import logging 21 | import urllib.parse 22 | from mqspeak.channel import ChannelType 23 | 24 | class ChannelUpdateDispatcher: 25 | """! 26 | Dispatching new update threads. 27 | """ 28 | 29 | ## @var channelSenders 30 | # Channel sendres mapping. 31 | 32 | ## @var dispatchLock 33 | # Mutual exclusion for update job dispatching. 34 | 35 | ## @var running 36 | # Keep track if dispatcher is running. 37 | 38 | ## @var updateQueue 39 | # Queue of pending updates. 40 | 41 | def __init__(self, channelConvertMapping): 42 | """! 43 | Initiate ChannelUpdateDispatcher object. 44 | 45 | @param channelConvertMapping Mapping {channel: channelParamConverter}. 46 | """ 47 | self.channelSenders = self.createChannelSenders(channelConvertMapping) 48 | self.dispatchLock = threading.Semaphore(0) 49 | self.running = False 50 | self.updateQueue = collections.deque() 51 | 52 | def createChannelSenders(self, channelConvertMapping): 53 | """! 54 | Crate channel senders mapping. 55 | 56 | @param channelConvertMapping ChannelConvertMapping object. 57 | @return Senders mapping. 58 | """ 59 | channelSenders = {} 60 | channelSenders[ChannelType.thingspeak] = ThingSpeakSender(channelConvertMapping) 61 | channelSenders[ChannelType.phant] = PhantSender(channelConvertMapping) 62 | return channelSenders 63 | 64 | def updateAvailable(self, channel, measurement, resultNotify): 65 | """! 66 | Notify main thread when new data is available. 67 | 68 | @param channel 69 | @param measurement 70 | @param resultNotify 71 | """ 72 | self.updateQueue.append((channel, measurement, resultNotify)) 73 | self.dispatchLock.release() 74 | 75 | def sendJobDone(self, result): 76 | """! 77 | Notify updater. 78 | 79 | @param result 80 | """ 81 | (returnCode, updater) = result 82 | updater.notifyUpdateResult(returnCode) 83 | 84 | def run(self): 85 | """! 86 | Start update dispatcher main loop. 87 | """ 88 | self.running = True 89 | while self.running: 90 | self.dispatchLock.acquire() 91 | 92 | # check for stop() method call 93 | if not self.running: 94 | return 95 | 96 | # start send thread 97 | channel, measurement, resultNotify = self.updateQueue.popleft() 98 | self.dispatch(channel, measurement, resultNotify) 99 | 100 | def stop(self): 101 | """! 102 | Stop dispatcher thread. 103 | """ 104 | if self.running: 105 | self.running = False 106 | self.dispatchLock.release() 107 | 108 | def dispatch(self, channel, measurement, updater): 109 | """! 110 | Dispatch new ThingSpeak update thread. 111 | 112 | @param channel Updated channel. 113 | @param measurement Update data. 114 | @param updater Notified object with update results. 115 | """ 116 | sendThread = threading.Thread( 117 | target = SendRunner( 118 | self.channelSenders[channel.channelType], 119 | channel, 120 | measurement, 121 | updater, 122 | self)) 123 | sendThread.start() 124 | 125 | class BaseSender: 126 | """! 127 | Sender base class. 128 | """ 129 | 130 | ## @var channelConvertMapping 131 | # Mapping {channel: channelParamConverter}. 132 | 133 | def __init__(self, channelConvertMapping): 134 | """! 135 | Initiate Sender base class. 136 | 137 | @param channelConvertMapping Mapping {channel: channelParamConverter}. 138 | """ 139 | self.channelConvertMapping = channelConvertMapping 140 | 141 | def send(self, channel, measurement): 142 | """! 143 | Send measurement. 144 | 145 | @param channel Updated channel object. 146 | @param measurement Measured data. 147 | """ 148 | success = False 149 | try: 150 | logging.getLogger().info( 151 | "Sending data to channel {}: {}...".format(channel, measurement)) 152 | status, reason, responseBytes = self.fetch(channel, measurement) 153 | response = self.decodeResponseData(responseBytes) 154 | logging.getLogger().info( 155 | "Channel {} response: {} {}: {}".format(channel, status, reason, response)) 156 | result = (status, reason, response) 157 | success = self.checkSendResult(result) 158 | except BaseException as ex: 159 | logging.getLogger().info("Send exception: {}".format(ex)) 160 | finally: 161 | return UpdateResult(success) 162 | 163 | def decodeResponseData(self, responseBytes): 164 | """! 165 | Decode response data. 166 | 167 | @param responseBytes 168 | @return Decoded data or decode error message. 169 | """ 170 | data = None 171 | try: 172 | data = responseBytes.decode("utf-8").strip() 173 | except UnicodeError as ex: 174 | logging.getLogger().error("Can't decode response data: {}".format(responseBytes)) 175 | data = "" 176 | finally: 177 | return data 178 | 179 | def fetch(self, channel, measurement): 180 | """! 181 | Upload data to channel. 182 | 183 | @param channel Channel identification object. 184 | @param measurement Uploaded data. 185 | """ 186 | raise NotImplementedError("Override this mehod in sub-class") 187 | 188 | def checkSendResult(self, result): 189 | """! 190 | Check if data upload was succcessful. 191 | 192 | @param result Tuple of (status, reason, response). 193 | """ 194 | raise NotImplementedError("Override this mehod in sub-class") 195 | 196 | class ThingSpeakSender(BaseSender): 197 | """! 198 | Class for sending data to ThingSpeak. This class send measurements to URL api.thingspeak.com 199 | using HTTPS method. It also parses send result and checks if transfer was successful. 200 | """ 201 | 202 | def fetch(self, channel, measurement): 203 | """! 204 | @copydoc BaseSender::fetch() 205 | """ 206 | body = self.channelConvertMapping[channel].convert(measurement) 207 | body.update({'created_at': measurement.time.isoformat(sep = ' ')}) 208 | body.update({'api_key': channel.apiKey}) 209 | bodyEncoded = urllib.parse.urlencode(body) 210 | conn = http.client.HTTPSConnection("api.thingspeak.com", timeout = 30) 211 | conn.request("POST", "/update", bodyEncoded) 212 | response = conn.getresponse() 213 | status = response.status 214 | reason = response.reason 215 | responseBytes = response.read() 216 | conn.close() 217 | return status, reason, responseBytes 218 | 219 | def checkSendResult(self, result): 220 | """! 221 | @copydoc BaseSender::checkSendResult() 222 | """ 223 | status, reason, data = result 224 | if status != 200: 225 | logging.getLogger().error("Response status error: {} {} - {}.".format(status, reason, data)) 226 | return False 227 | try: 228 | entries = int(data) 229 | if entries == 0: 230 | logging.getLogger().error("Data send error: ThingSpeak responded with return code 0.") 231 | return False 232 | except ValueError as ex: 233 | logging.getLogger().error("Data send error: ThingSpeak responded with unexpected response: {}".format(repr(data))) 234 | return False 235 | return True 236 | 237 | class PhantSender(BaseSender): 238 | """! 239 | Send data to Phant server. 240 | """ 241 | 242 | def fetch(self, channel, measurement): 243 | """! 244 | @copydoc BaseSender::fetch() 245 | """ 246 | body = self.channelConvertMapping[channel].convert(measurement) 247 | bodyEncoded = urllib.parse.urlencode(body) 248 | headers = {"Phant-Private-Key": channel.apiKey, 249 | "Content-Type": "application/x-www-form-urlencoded"} 250 | conn = http.client.HTTPConnection("data.sparkfun.com", timeout = 30) 251 | conn.request("POST", "/input/{}".format(channel.channelID), bodyEncoded, headers=headers) 252 | response = conn.getresponse() 253 | status = response.status 254 | reason = response.reason 255 | responseBytes = response.read() 256 | conn.close() 257 | return status, reason, responseBytes 258 | 259 | def checkSendResult(self, result): 260 | """! 261 | @copydoc BaseSender::checkSendResult() 262 | """ 263 | (status, reason, data) = result 264 | if status != 200: 265 | logging.getLogger().error("Response status error: {} {} - {}.".format(status, reason, data)) 266 | return False 267 | return True 268 | 269 | class SendRunner: 270 | """! 271 | Callable wrapper class for sending data to ThingSpeak in separate thread. 272 | """ 273 | 274 | ## @var sender 275 | # Sender object. This object must implement send(channel, measurement) method. 276 | 277 | ## @var channel 278 | # Channel description object. 279 | 280 | ## @var measurement 281 | # Measured data. 282 | 283 | ## @var updater 284 | # Updater object which will be called by dispatcher after send job is done. 285 | 286 | ## @var jobNotify 287 | # Listener object called after data send. 288 | 289 | def __init__(self, sender, channel, measurement, updater, jobNotify): 290 | """! 291 | Initiate SendRunner object. 292 | 293 | @param sender Sender object. This object must implement send(channel, measurement) method. 294 | @param channel Channel description object. 295 | @param measurement Measured data. 296 | @param updater Updater object which will be called by dispatcher after send job is done. 297 | @param jobNotify Listener object called after data send. 298 | """ 299 | self.sender = sender 300 | self.channel = channel 301 | self.measurement = measurement 302 | self.updater = updater 303 | self.jobNotify = jobNotify 304 | 305 | def __call__(self): 306 | """! 307 | Thread code. 308 | """ 309 | try : 310 | result = (self.sender.send(self.channel, self.measurement), self.updater) 311 | self.jobNotify.sendJobDone(result) 312 | except Exception as ex: 313 | self.jobNotify.sendJobDone(ex) 314 | 315 | class UpdateResult: 316 | """! 317 | Encapsulate update result. 318 | """ 319 | 320 | ## @var success 321 | # Flag if update was successful. 322 | 323 | def __init__(self, success): 324 | """! 325 | Initiate update result. 326 | 327 | @param success Indicate if update was successful or not 328 | """ 329 | self.success = success 330 | 331 | def wasSuccessful(self): 332 | """! 333 | Check if UpdateResul was successful. 334 | 335 | @return True if was sucessful, False otherwise. 336 | """ 337 | return self.success 338 | -------------------------------------------------------------------------------- /mqspeak/system.py: -------------------------------------------------------------------------------- 1 | # Copyright (C) Ivo Slanina 2 | # 3 | # This program is free software: you can redistribute it and/or modify 4 | # it under the terms of the GNU General Public License as published by 5 | # the Free Software Foundation, either version 3 of the License, or 6 | # (at your option) any later version. 7 | # 8 | # This program is distributed in the hope that it will be useful, 9 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 10 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 11 | # GNU General Public License for more details. 12 | # 13 | # You should have received a copy of the GNU General Public License 14 | # along with this program. If not, see . 15 | 16 | import sys 17 | import logging 18 | import logging.handlers 19 | from mqspeak.config import ProgramConfig, ConfigException 20 | from mqspeak.data import MeasurementParamConverter 21 | from mqspeak import args 22 | 23 | class System: 24 | """! 25 | System initialization object. This object encapsulates program configuration state 26 | defined by command line arguments and parsed configuration file. 27 | """ 28 | 29 | @classmethod 30 | def initialize(cls): 31 | """! 32 | Initiate system configuration. 33 | """ 34 | 35 | l = logging.getLogger() 36 | l.setLevel(logging.INFO) 37 | h = logging.handlers.SysLogHandler(address='/dev/log') 38 | 39 | cls.cliArgs = args.parse_args() 40 | 41 | # Logging destination. 42 | if cls.cliArgs.log_stdout: 43 | h = logging.StreamHandler(stream = sys.stdout) 44 | 45 | # Verbose. 46 | if cls.cliArgs.verbose: 47 | h.setLevel(logging.INFO) 48 | else: 49 | h.setLevel(logging.ERROR) 50 | l.addHandler(h) 51 | 52 | config = ProgramConfig(cls.cliArgs.config) 53 | # TODO: handle config exceptions 54 | try: 55 | cls.configCache = config.parse() 56 | except ConfigException as ex: 57 | logging.getLogger().error("Configuration error: {}".format(ex)) 58 | exit(1) 59 | 60 | @classmethod 61 | def getChannelConvertMapping(cls): 62 | """! 63 | Get mapping for converting measurements for each channel. 64 | 65 | @return {channel: channelParamConverter} 66 | """ 67 | channelConvertMapping = {} 68 | for channel, _, updateMapping in cls.configCache.channelUpdateDescribtors: 69 | channelConvertMapping[channel] = MeasurementParamConverter(updateMapping) 70 | return channelConvertMapping 71 | 72 | @classmethod 73 | def getBrokerListenDescriptors(cls): 74 | """! 75 | Get list of tuples (broker, ["subscribeTopic"]). 76 | 77 | @return (broker, ["subscribeTopic"]) 78 | """ 79 | return cls.configCache.listenDescriptors 80 | 81 | @classmethod 82 | def getUpdateBuffers(cls): 83 | """! 84 | Get list of UpdateBuffer instances. 85 | 86 | @return 87 | """ 88 | updateBuffers = [] 89 | for channel, _, updateMapping in cls.configCache.channelUpdateDescribtors: 90 | dataIdentifiers = list(updateMapping.keys()) 91 | updateBuffer = UpdateBuffer(channel, dataIdentifiers) 92 | updateBuffers.append(updateBuffer) 93 | return updateBuffers 94 | 95 | @classmethod 96 | def getChannelUpdateMapping(cls): 97 | """! 98 | Get mapping {channel: updater}. 99 | 100 | @return {channel: updater} 101 | """ 102 | channelUpdateMapping = {} 103 | for channel, updater, _ in cls.configCache.channelUpdateDescribtors: 104 | channelUpdateMapping[channel] = updater 105 | return channelUpdateMapping 106 | -------------------------------------------------------------------------------- /mqspeak/updating.py: -------------------------------------------------------------------------------- 1 | # Copyright (C) Ivo Slanina 2 | # 3 | # This program is free software: you can redistribute it and/or modify 4 | # it under the terms of the GNU General Public License as published by 5 | # the Free Software Foundation, either version 3 of the License, or 6 | # (at your option) any later version. 7 | # 8 | # This program is distributed in the hope that it will be useful, 9 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 10 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 11 | # GNU General Public License for more details. 12 | # 13 | # You should have received a copy of the GNU General Public License 14 | # along with this program. If not, see . 15 | 16 | import datetime 17 | import threading 18 | import time 19 | import queue 20 | import logging 21 | from mqspeak.collecting import LastValueUpdateBuffer, AverageUpdateBuffer, ChangeValueBuffer 22 | from mqreceive.collecting import DataCollector 23 | from mqspeak.data import Measurement 24 | 25 | class ChannnelUpdateSupervisor(DataCollector): 26 | """! 27 | Manage channel updaters. Object is responsible to delivering channel update event to 28 | correct Updater object. 29 | """ 30 | 31 | ## @var channelUpdaterMapping 32 | # Mapping for {channel: updater}. 33 | 34 | ## @var waitingChannels 35 | # Mapping of channels which has some data wainting. 36 | 37 | def __init__(self, channelUpdaterMapping): 38 | """! 39 | Initiate ChannnelUpdateSupervisor object. 40 | 41 | @param channelUpdaterMapping Mapping for {channel: updater}. 42 | """ 43 | self.channelUpdaterMapping = channelUpdaterMapping 44 | self.waitingChannels = {} 45 | self.waintingUpdater = SchedulerExecutor( 46 | datetime.timedelta(seconds = 1), 47 | self.updateWaitingData) 48 | threading.Thread(target = self.waintingUpdater).start() 49 | 50 | def updateWaitingData(self, executor): 51 | """! 52 | Do partitial update. Clear waiting data. 53 | """ 54 | for updater in self.channelUpdaterMapping.values(): 55 | updater.notifyUpdateWaiting() 56 | threading.Thread(target = self.waintingUpdater).start() 57 | 58 | def setDispatcher(self, dispatcher): 59 | """! 60 | Assign a dispatcher to all updaters. 61 | 62 | @param dispatcher 63 | """ 64 | for updater in self.channelUpdaterMapping.values(): 65 | updater.setDispatcher(dispatcher) 66 | 67 | def stop(self): 68 | """! 69 | Stop execution of all updaters. 70 | """ 71 | self.waintingUpdater.stop() 72 | for updater in self.channelUpdaterMapping.values(): 73 | updater.stop() 74 | 75 | def onNewData(self, dataIdentifier, data): 76 | try: 77 | data = data.decode("utf-8") 78 | except UnicodeError as ex: 79 | logging.getLogger().info("Can't decode received message payload: {}".format(repr(data))) 80 | 81 | for updater in self.channelUpdaterMapping.values(): 82 | if updater.isUpdateRelevant(dataIdentifier): 83 | 84 | # Notify updater in separate thread for case that updater will 85 | # block for some reason. 86 | threading.Thread( 87 | target = updater.updateReceivedData, 88 | args = (dataIdentifier, data)).start() 89 | 90 | class BaseUpdater: 91 | """! 92 | Updater base class. 93 | 94 | Before use of any instance of this class, call setDispatcher() method to 95 | assign a update disatcher. Update dispather is object which runs an update 96 | in its separate thread and notifies back an updater, when update finishes. 97 | """ 98 | 99 | ## @var channel 100 | # Updated channel. 101 | 102 | ## @var updateInterval 103 | # Channel update interval. 104 | 105 | ## @var isUpdateRunning 106 | # Boolean variable to keep track if some update is currently running. 107 | 108 | ## @var lastUpdated 109 | # Last update time. 110 | 111 | ## @var waitingStarted 112 | # Timestamp of started waiting (when updater has some data and is in 113 | # waiting state) or None if updater doesn't wait for any remaning data. 114 | 115 | ## @var updateLock 116 | # Mutual exclusion to running updates. 117 | 118 | ## @var dispatcher 119 | # Update dispatcher object. 120 | 121 | ## @var updateBuffer 122 | # Channel UpdateBuffer object. 123 | 124 | def __init__(self, channel, updateInterval, updateBuffer): 125 | """! 126 | Initiate BaseUpdater object. 127 | 128 | @param channel Update Channel object. 129 | @param updateInterval timedelta object defining update interval. 130 | @param updateBuffer UpdateBuffer object. 131 | """ 132 | self.channel = channel 133 | self.updateInterval = updateInterval 134 | self.isUpdateRunning = False 135 | self.lastUpdated = datetime.datetime.min 136 | self.waitingStarted = None 137 | self.updateLock = threading.Semaphore(1) 138 | self.updateBuffer = updateBuffer 139 | 140 | def setDispatcher(self, dispatcher): 141 | """! 142 | Assign a dispatcher. 143 | 144 | @param dispatcher Dispatcher object. 145 | """ 146 | self.dispatcher = dispatcher 147 | 148 | def stop(self): 149 | """! 150 | Override this method if updater manage some other running thread. 151 | """ 152 | 153 | def isUpdateIntervalExpired(self): 154 | """! 155 | Check if Update interval has expired. 156 | 157 | @return True if update interval has expired, False otherwise. 158 | """ 159 | return (datetime.datetime.now() - self.lastUpdated) > self.updateInterval 160 | 161 | def restartUpdateIntervalCounter(self): 162 | """! 163 | Restart interval counter. 164 | """ 165 | self.lastUpdated = datetime.datetime.now() 166 | 167 | def isUpdateRelevant(self, dataIdentifier): 168 | """! 169 | Check if update is relevant to this channel. 170 | 171 | @param dataIdentifier Update data identifier. 172 | @return True if update is relevant, False otherwise 173 | """ 174 | return self.updateBuffer.isUpdateRelevant(dataIdentifier) 175 | 176 | def updateReceivedData(self, dataIdentifier, value): 177 | """! 178 | Update received data. 179 | 180 | @param dataIdentifier Data identification. 181 | @param value Data content. 182 | @throws TopicException If unwanted topic is updated. 183 | """ 184 | # TODO: execute this code in separate thread. If one channel blocks, all 185 | # other channels will be also blocked. 186 | self.updateLock.acquire() 187 | try: 188 | self.updateBuffer.updateReceivedData(dataIdentifier, value) 189 | if not self.isUpdateRunning: 190 | if self.updateBuffer.isComplete(): 191 | self.dataComplete() 192 | else: 193 | if self.isUpdateIntervalExpired() and \ 194 | self.channel.hasWaiting() and \ 195 | self.waitingStarted is None: 196 | self.waitingStarted = datetime.datetime.now() 197 | except Exception as ex: 198 | logging.getLogger().error("Channel <{}>: {}".format(self.channel, ex)) 199 | finally: 200 | self.updateLock.release() 201 | 202 | def notifyUpdateWaiting(self): 203 | """! 204 | Call this method periodically to chech if waiting interval has been 205 | exceeded. 206 | """ 207 | if self.channel.hasWaiting(): 208 | self.updateLock.acquire() 209 | try: 210 | if not self.isUpdateRunning: 211 | if self.waitingStarted is not None : 212 | delta = datetime.datetime.now() - self.waitingStarted 213 | if self.updateBuffer.hasAnyData() and delta > self.channel.waiting: 214 | logging.getLogger().warning( 215 | "Waiting timeouted, data items {} hasn't any data.".format( 216 | ", ".join(str(x) for x in self.updateBuffer.getMissingDataIdentifiers()))) 217 | self.runUpdate() 218 | self.updateBuffer.reset() 219 | elif self.updateBuffer.hasAnyData() and self.isUpdateIntervalExpired(): 220 | # Update buffer store some data. Start waiting for a case that no 221 | # more data will be received in the future. 222 | self.waitingStarted = datetime.datetime.now() 223 | finally: 224 | self.updateLock.release() 225 | 226 | def dataComplete(self): 227 | """! 228 | Notify that all data needed for update is complete. 229 | """ 230 | raise NotImplementedError("Override this mehod in sub-class") 231 | 232 | def runUpdate(self): 233 | """! 234 | Call this method in sub-class from handleAvailableData method. 235 | 236 | @param measurement 237 | """ 238 | self.isUpdateRunning = True 239 | self.waitingStarted = None 240 | measurement = self.updateBuffer.getMeasurement() 241 | self.updateBuffer.reset() 242 | self.dispatcher.updateAvailable(self.channel, measurement, self) 243 | 244 | def runUpdateLocked(self): 245 | """! 246 | Run update. This method avoids race conditions. Do not call this method 247 | from handleAvailableData() metod - causes dead lock. 248 | 249 | @param measurement 250 | """ 251 | self.updateLock.acquire() 252 | try: 253 | self.runUpdate() 254 | finally: 255 | self.updateLock.release() 256 | 257 | def notifyUpdateResult(self, result): 258 | """! 259 | Callback method with update result 260 | 261 | @param result UpdateResult object. 262 | """ 263 | self.updateLock.acquire() 264 | try: 265 | self.isUpdateRunning = False 266 | if result.wasSuccessful(): 267 | self.restartUpdateIntervalCounter() 268 | self.resolveUpdateResult(result) 269 | finally: 270 | self.updateLock.release() 271 | 272 | def resolveUpdateResult(self, result): 273 | """! 274 | Resolve update result in updater. 275 | 276 | @param result UpdateResult object. 277 | """ 278 | raise NotImplementedError("Override this mehod in sub-class") 279 | 280 | class BlackoutUpdater(BaseUpdater): 281 | """! 282 | Ignore any incomming data during blackout period. Send first data which arriver 283 | after blackout period expires. 284 | """ 285 | 286 | def __init__(self, channel, updateMapping, updateInterval): 287 | BaseUpdater.__init__( 288 | self, 289 | channel, 290 | updateInterval, 291 | LastValueUpdateBuffer(updateMapping.keys())) 292 | 293 | def dataComplete(self): 294 | if self.isUpdateIntervalExpired() and not self.isUpdateRunning: 295 | self.runUpdate() 296 | 297 | def resolveUpdateResult(self, result): 298 | """! 299 | @copydoc BaseUpdater::resolveUpdateResult() 300 | """ 301 | pass 302 | 303 | class SynchronousUpdater(BaseUpdater): 304 | """! 305 | Base class for all updaters which tries to update channel in synchronous fashion. 306 | """ 307 | 308 | ## @var isUpdateScheduled 309 | # Boolean variable to track if some update is already scheduled. 310 | 311 | ## @var scheduleLock 312 | # Mutual exclusion for isUpdateScheduled flag. 313 | 314 | ## @var executors 315 | # Set of running executors. 316 | 317 | def __init__(self, channel, updateInterval, updateBuffer): 318 | """! 319 | Initiate SynchronousUpdater object. 320 | 321 | @param channel 322 | @param updateInterval 323 | """ 324 | BaseUpdater.__init__(self, channel, updateInterval, updateBuffer) 325 | self.isUpdateScheduled = False 326 | self.scheduleLock = threading.Semaphore(1) 327 | # TODO: Check race conditions with this set. 328 | self.executors = set() 329 | 330 | def dataComplete(self): 331 | self.scheduleLock.acquire() 332 | try: 333 | if not self.isUpdateScheduled: 334 | # There is no update sheduled. It is first run or data was unavailable 335 | # for the long time. Also, an another update is running. 336 | if not self.isUpdateRunning: 337 | # No other update is running. Update immidiatelly. 338 | self.runUpdate() 339 | finally: 340 | self.scheduleLock.release() 341 | 342 | def resolveUpdateResult(self, result): 343 | """! 344 | @copydoc BaseUpdater::resolveUpdateResult() 345 | """ 346 | # An update just finished. Wait for configured time and run new update. 347 | self.scheduleLock.acquire() 348 | try: 349 | self.scheduleUpdateJob() 350 | finally: 351 | self.scheduleLock.release() 352 | 353 | def scheduleUpdateJob(self): 354 | """ 355 | Schedule new update job. 356 | """ 357 | self.isUpdateScheduled = True 358 | executor = SchedulerExecutor( 359 | datetime.timedelta(seconds=int(self.updateInterval.total_seconds())), 360 | self.onSchedule) 361 | threading.Thread(target=executor).start() 362 | self.executors.add(executor) 363 | 364 | def onSchedule(self, executor): 365 | """ 366 | Callback method called when scheduler expires. 367 | """ 368 | self.scheduleLock.acquire() 369 | try: 370 | self.executors.discard(executor) 371 | self.isUpdateScheduled = False 372 | if self.updateBuffer.isComplete(): 373 | self.runUpdateLocked() 374 | finally: 375 | self.scheduleLock.release() 376 | 377 | def stop(self): 378 | """! 379 | Stop all executors. 380 | """ 381 | for executor in self.executors: 382 | executor.stop() 383 | self.executors = set() 384 | 385 | class BufferedUpdater(SynchronousUpdater): 386 | """! 387 | Implement some timer to send update is time elapses. Don't wait for incoming data 388 | after time expires. 389 | """ 390 | 391 | def __init__(self, channel, updateMapping, updateInterval): 392 | SynchronousUpdater.__init__( 393 | self, 394 | channel, 395 | updateInterval, 396 | LastValueUpdateBuffer(updateMapping.keys())) 397 | 398 | class AverageUpdater(SynchronousUpdater): 399 | """! 400 | Like BufferedUpdater but keep track all data which wasn't send and calculate 401 | average value while sending them. 402 | """ 403 | 404 | def __init__(self, channel, updateMapping, updateInterval): 405 | SynchronousUpdater.__init__( 406 | self, 407 | channel, 408 | updateInterval, 409 | AverageUpdateBuffer(updateMapping.keys())) 410 | 411 | class OnChangeUpdater(SynchronousUpdater): 412 | """! 413 | Send every value change. 414 | """ 415 | 416 | def __init__(self, channel, updateMapping, updateInterval): 417 | SynchronousUpdater.__init__( 418 | self, 419 | channel, 420 | updateInterval, 421 | ChangeValueBuffer(updateMapping.keys())) 422 | 423 | class SchedulerExecutor: 424 | """! 425 | Execute scheduler object in separate thread. 426 | """ 427 | 428 | ## @var event 429 | # Event object. 430 | 431 | ## @var scheduleTime 432 | # Schedule time. 433 | 434 | ## @var action 435 | # Scheduled action. 436 | 437 | def __init__(self, scheduleTime, action): 438 | """! 439 | Initiate scheduler executor. 440 | 441 | @param scheduleTime Timedelta object. 442 | @param action Callable object executed after schedule time expires. Action takes one argument, 443 | which is reference to this executor. 444 | """ 445 | self.event = threading.Event() 446 | self.scheduleTime = scheduleTime 447 | self.action = action 448 | 449 | def __call__(self): 450 | """! 451 | Run schedule execution. 452 | """ 453 | scheduleExpires = not self.event.wait(self.scheduleTime.total_seconds()) 454 | if scheduleExpires: 455 | self.action(self) 456 | 457 | def stop(self): 458 | """! 459 | Stop scheduler execution. 460 | """ 461 | self.event.set() 462 | -------------------------------------------------------------------------------- /readme.md: -------------------------------------------------------------------------------- 1 | # mqspeak - MQTT bridge 2 | 3 | mqspeak is [MQTT](http://mqtt.org/) client which collect data and transforms 4 | them into [ThingSpeak](https://thingspeak.com/) channel updates or [Phant](http://phant.io/) 5 | data streams. It is able to handle multiple MQTT connections and independetly update 6 | multiple channels. 7 | 8 | This is part of my IoT project. You can 9 | read more about it on my [blog](http://buben19.blogspot.com/). 10 | 11 | ## Install 12 | 13 | Application can be installed with following command: 14 | 15 | $ sudo pip3 install mqspeak 16 | 17 | ## Configuration 18 | 19 | mqspeak is configured using configuration file specified with `-c` or `--config` 20 | option (default `/etc/mqspeak.conf`). This is sample configuration file: 21 | 22 | [Brokers] 23 | Enabled = temperature-broker humidity-broker door-broker 24 | 25 | [temperature-broker] 26 | Host = temperatureBrokerHostname 27 | Port = 1883 28 | User = brokerUser 29 | Password = brokerPass 30 | Topic = sensors/temperature sensors/something 31 | 32 | [humidity-broker] 33 | Host = humidityBrokerHostname 34 | Port = 1883 35 | User = brokerUser 36 | Password = brokerPass 37 | Topic = sensors/humidity 38 | 39 | [door-broker] 40 | Host = doorBrokerHostname 41 | Port = 1883 42 | User = brokerUser 43 | Password = brokerPass 44 | Topic = # 45 | 46 | [Channels] 47 | Enabled = channel1 channel2 channel3 channel4 48 | 49 | [channel1] 50 | Id = CHANNELID 51 | Key = CHANNELKEY 52 | Type = thingspeak 53 | UpdateRate = 15 54 | UpdateType = blackout 55 | UpdateFields = dht-update 56 | 57 | [channel2] 58 | Id = CHANNELID 59 | Key = CHANNELKEY 60 | Type = thingspeak 61 | UpdateRate = 15 62 | UpdateType = buffered 63 | UpdateFields = dht-update 64 | 65 | [channel3] 66 | Id = CHANNELID 67 | Key = CHANNELKEY 68 | Type = thingspeak 69 | UpdateRate = 15 70 | UpdateType = average 71 | UpdateFields = dht-update 72 | 73 | [channel4] 74 | Id = CHANNELID 75 | Key = CHANNELKEY 76 | Type = phant 77 | UpdateRate = 15 78 | UpdateType = onchange 79 | UpdateFields = door-update 80 | 81 | [dht-update] 82 | field1 = humidity-broker sensors/humidity 83 | field2 = temperature-broker sensors/temperature 84 | 85 | [door-update] 86 | state = door-broker sensors/door 87 | 88 | Configuration file has two mandatory sections: `[Brokers]` and `[Channels]`, each with 89 | one `Enabled` option. These options contains space separated broker and channel 90 | section names. 91 | 92 | ### Broker section 93 | 94 | Broker section has to define one mandatory `[Topic]` option, which is space separated 95 | list of MQTT topic subscriptions. Full list of possible options in broker section: 96 | 97 | - `Host` - Broker IP address or hostname (default 127.0.0.1). 98 | - `Port` - Broker port (default 1883). 99 | - `User` - Username. 100 | - `Password` - Password. 101 | - `Topic` - Space separated list of topic subscriptions. Mandatory option. 102 | 103 | ### Channel section 104 | 105 | Each channel section has to define `Key`, `UpdateRate` and `UpdateType` options. 106 | 107 | - `Id` - Channel ID. This field is mandatory for Phant channels. 108 | - `Key` - Channel API write key. Mandatory option. 109 | - `Type` - Specify channel type. Mandatory option. Following types are supported: 110 | - `thingspeak` - [ThinkSpeak](https://thingspeak.com/) channel. 111 | - `phant` - [Phant](http://phant.io/) channel. 112 | - `UpdateRate` - Channel update interval in seconds. Currently, ThinkSpeak allows 113 | interval 15 seconds or greater. Mandatory option. 114 | - `WaitInterval` - Maximum interval to wait for remaining data to arrive. When set to 115 | zero, wait forever (default). See **Update waiting** for more details. 116 | - `UpdateType` - Channel update type. Possible values are `blackout`, `buffered`, 117 | `average` and `onchange`. Mandatory option. 118 | - `blackout` - Until `UpdateRate` interval is expired, any incoming data are 119 | ignored. First data received after interval expiration are sent to ThingSpeak. 120 | - `buffered` - Incoming data are buffered during `UpdateRate` interval. After 121 | this interval expires, most recent values are immediately sent. 122 | - `average` - Similar to `buffered` but mqspeak calculates average value of these 123 | data. Any data which cannot be converted into real numbers are ignored. Channel 124 | is immediately updated after `UpdateRate` interval is expired. 125 | - `onchange` - Data are marked with timestamp and stored in queue. Each item is 126 | sent after `UpdateRate` interval expires. **_Not implemented yet._** 127 | - `UpdateFields` - Specify section which defines updates for this channel. Mandatory option. 128 | 129 | #### Update waiting 130 | 131 | When channel update consists of data from multiple sensors, it may happen that one 132 | sensor die. By default channel never will be updated until data from all sensors arrives. 133 | Inactive sensor causes channel update will be stalled. 134 | 135 | When update waiting enabled, mqspeak will wait defined amount of seconds and then sends 136 | out even incomplete channel update. 137 | 138 | Waiting scenario can be divided into following cases: 139 | 140 | - **`UpdateRate` condition is met but there are no data.** Wait mechanism is not activated 141 | until some data arrives. After it received first part of channel update, mqspeak will wait 142 | defined time and tries collect remaining data. After `WaitInterval` expires, 143 | data will be send. 144 | - **Partial data arrives before `UpdateRate` condition is met.** Waiting is delayed 145 | until `UpdateRate` condition is met. After it expires and there are still not 146 | all required data, waiting is triggered. After `WaitInterval` expires, data 147 | will be send. 148 | - **All required data are collected before `UpdateRate` condition is met.** There is no 149 | need to activate update waiting. Simply send data. 150 | 151 | ### UpdateFields section 152 | 153 | UpdateFields section consists of any number of options. Each option key specifies 154 | field name. Its value must be space separated name of broker section and topic. 155 | 156 | For ThinkSpeak channel, only option keys `Field1` ... `Field8` are valid. 157 | 158 | ## Questions 159 | 160 | - **mqspeak runs in foreground only.** - Yes, there is no double fork combo to run 161 | mqspeak in background. I use systemd init and I prefer to run all services as simple 162 | systemd units, which runs in foreground. Sorry about that. 163 | - **It uses python3. Is python 2.x supported?** - No, I don't plan to support python 2.x. 164 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # 3 | # Copyright (C) Ivo Slanina 4 | # 5 | # This program is free software: you can redistribute it and/or modify 6 | # it under the terms of the GNU General Public License as published by 7 | # the Free Software Foundation, either version 3 of the License, or 8 | # (at your option) any later version. 9 | # 10 | # This program is distributed in the hope that it will be useful, 11 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 12 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 13 | # GNU General Public License for more details. 14 | # 15 | # You should have received a copy of the GNU General Public License 16 | # along with this program. If not, see . 17 | 18 | from setuptools import setup, find_packages 19 | 20 | import mqspeak 21 | 22 | def readme(): 23 | with open('readme.md') as f: 24 | return f.read() 25 | 26 | setup( 27 | name = "mqspeak", 28 | url = mqspeak.__project_url__, 29 | version = mqspeak.__version__, 30 | packages = find_packages(exclude = ['doc']), 31 | install_requires = ['mqreceive>=0.1.1'], 32 | author = mqspeak.__author__, 33 | author_email = mqspeak.__email__, 34 | description = "MQTT bridge", 35 | long_description = readme(), 36 | license = "GPLv3", 37 | classifiers = [ 38 | 'Development Status :: 3 - Alpha', 39 | 'Environment :: Console', 40 | 'Environment :: No Input/Output (Daemon)', 41 | 'Intended Audience :: Customer Service', 42 | 'Intended Audience :: Information Technology', 43 | 'Intended Audience :: Other Audience', 44 | 'Intended Audience :: Telecommunications Industry', 45 | 'License :: OSI Approved :: GNU Lesser General Public License v3 (LGPLv3)', 46 | 'Natural Language :: English', 47 | 'Programming Language :: Python :: 3', 48 | 'Programming Language :: Python :: 3.0', 49 | 'Programming Language :: Python :: 3.1', 50 | 'Programming Language :: Python :: 3.2', 51 | 'Programming Language :: Python :: 3.3', 52 | 'Programming Language :: Python :: 3.4', 53 | 'Programming Language :: Python :: 3.5', 54 | 'Programming Language :: Python :: 3 :: Only', 55 | 'Topic :: Communications', 56 | 'Topic :: Home Automation', 57 | 'Topic :: Internet', 58 | ], 59 | keywords = 'iot internetofthings mqopen mqtt sensors thingspeak phant', 60 | entry_points = { 61 | "console_scripts": [ 62 | "mqspeak = mqspeak.__main__:main" 63 | ] 64 | } 65 | ) 66 | --------------------------------------------------------------------------------