├── .github ├── PULL_REQUEST_TEMPLATE.md └── workflows │ └── autoblack.yml ├── .gitignore ├── LICENSE ├── README.md ├── lib ├── lib_package_linter.py ├── nginxparser │ └── nginxparser.py └── print.py ├── package_linter.py ├── requirements.txt ├── tests ├── test_app.py ├── test_catalog.py ├── test_configurations.py ├── test_manifest.py └── test_scripts.py └── tox.ini /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | ## Problem 2 | 3 | - *Description of why you made this PR, what is its purpose* 4 | 5 | ## Solution 6 | 7 | - *And how do you relevantly fix that problem* 8 | 9 | ## PR checklist 10 | 11 | - [ ] PR finished and ready to be reviewed 12 | -------------------------------------------------------------------------------- /.github/workflows/autoblack.yml: -------------------------------------------------------------------------------- 1 | name: Check / auto apply Black 2 | on: 3 | push: 4 | branches: 5 | - main 6 | 7 | jobs: 8 | black: 9 | name: Check / auto apply black 10 | runs-on: ubuntu-latest 11 | steps: 12 | - uses: actions/checkout@v4 13 | - name: Check files using the black formatter 14 | uses: psf/black@stable 15 | id: black 16 | with: 17 | options: "." 18 | continue-on-error: true 19 | - shell: pwsh 20 | id: check_files_changed 21 | run: | 22 | # Diff HEAD with the previous commit 23 | $diff = git diff 24 | $HasDiff = $diff.Length -gt 0 25 | Write-Host "::set-output name=files_changed::$HasDiff" 26 | - uses: stefanzweifel/git-auto-commit-action@v5 27 | with: 28 | commit_message: ":art: Format Python code with Black" 29 | commit_user_name: yunohost-bot 30 | commit_user_email: yunohost-bot@users.noreply.github.com 31 | commit_author: 'yunohost-bot ' 32 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # ---> Python 2 | # Byte-compiled / optimized / DLL files 3 | __pycache__/ 4 | *.py[cod] 5 | *$py.class 6 | 7 | # YunoHost packages 8 | *_ynh 9 | 10 | #Generated files during linter execution 11 | .apps_git_clone_cache 12 | .spdx_licenses 13 | .apps/ 14 | .manifest.* 15 | .tests.* 16 | 17 | .venv/ 18 | venv/ -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | GNU AFFERO GENERAL PUBLIC LICENSE 2 | Version 3, 19 November 2007 3 | Copyright (C) 2007 Free Software Foundation, Inc. 4 | 5 | Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. 6 | 7 | Preamble 8 | 9 | The GNU Affero General Public License is a free, copyleft license for software and other kinds of works, specifically designed to ensure cooperation with the community in the case of network server software. 10 | 11 | The licenses for most software and other practical works are designed to take away your freedom to share and change the works. By contrast, our General Public Licenses are intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users. 12 | 13 | When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things. 14 | 15 | Developers that use our General Public Licenses protect your rights with two steps: (1) assert copyright on the software, and (2) offer you this License which gives you legal permission to copy, distribute and/or modify the software. 16 | 17 | A secondary benefit of defending all users' freedom is that improvements made in alternate versions of the program, if they receive widespread use, become available for other developers to incorporate. Many developers of free software are heartened and encouraged by the resulting cooperation. However, in the case of software used on network servers, this result may fail to come about. The GNU General Public License permits making a modified version and letting the public access it on a server without ever releasing its source code to the public. 18 | 19 | The GNU Affero General Public License is designed specifically to ensure that, in such cases, the modified source code becomes available to the community. It requires the operator of a network server to provide the source code of the modified version running there to the users of that server. Therefore, public use of a modified version, on a publicly accessible server, gives the public access to the source code of the modified version. 20 | 21 | An older license, called the Affero General Public License and published by Affero, was designed to accomplish similar goals. This is a different license, not a version of the Affero GPL, but Affero has released a new version of the Affero GPL which permits relicensing under this license. 22 | 23 | The precise terms and conditions for copying, distribution and modification follow. 24 | 25 | TERMS AND CONDITIONS 26 | 27 | 0. Definitions. 28 | 29 | "This License" refers to version 3 of the GNU Affero General Public License. 30 | 31 | "Copyright" also means copyright-like laws that apply to other kinds of works, such as semiconductor masks. 32 | 33 | "The Program" refers to any copyrightable work licensed under this License. Each licensee is addressed as "you". "Licensees" and "recipients" may be individuals or organizations. 34 | 35 | To "modify" a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a "modified version" of the earlier work or a work "based on" the earlier work. 36 | 37 | A "covered work" means either the unmodified Program or a work based on the Program. 38 | 39 | To "propagate" a work means to do anything with it that, without permission, would make you directly or secondarily liable for infringement under applicable copyright law, except executing it on a computer or modifying a private copy. Propagation includes copying, distribution (with or without modification), making available to the public, and in some countries other activities as well. 40 | 41 | To "convey" a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying. 42 | 43 | An interactive user interface displays "Appropriate Legal Notices" to the extent that it includes a convenient and prominently visible feature that (1) displays an appropriate copyright notice, and (2) tells the user that there is no warranty for the work (except to the extent that warranties are provided), that licensees may convey the work under this License, and how to view a copy of this License. If the interface presents a list of user commands or options, such as a menu, a prominent item in the list meets this criterion. 44 | 45 | 1. Source Code. 46 | The "source code" for a work means the preferred form of the work for making modifications to it. "Object code" means any non-source form of a work. 47 | 48 | A "Standard Interface" means an interface that either is an official standard defined by a recognized standards body, or, in the case of interfaces specified for a particular programming language, one that is widely used among developers working in that language. 49 | 50 | The "System Libraries" of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an implementation is available to the public in source code form. A "Major Component", in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it. 51 | 52 | The "Corresponding Source" for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work. For example, Corresponding Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication or control flow between those 53 | subprograms and other parts of the work. 54 | 55 | The Corresponding Source need not include anything that users can regenerate automatically from other parts of the Corresponding Source. 56 | 57 | The Corresponding Source for a work in source code form is that same work. 58 | 59 | 2. Basic Permissions. 60 | All rights granted under this License are granted for the term of copyright on the Program, and are irrevocable provided the stated conditions are met. This License explicitly affirms your unlimited permission to run the unmodified Program. The output from running a covered work is covered by this License only if the output, given its content, constitutes a covered work. This License acknowledges your rights of fair use or other equivalent, as provided by copyright law. 61 | 62 | You may make, run and propagate covered works that you do not convey, without conditions so long as your license otherwise remains in force. You may convey covered works to others for the sole purpose of having them make modifications exclusively for you, or provide you with facilities for running those works, provided that you comply with the terms of this License in conveying all material for which you do not control copyright. Those thus making or running the covered works for you must do so exclusively on your behalf, under your direction and control, on terms that prohibit them from making any copies of your copyrighted material outside their relationship with you. 63 | 64 | Conveying under any other circumstances is permitted solely under the conditions stated below. Sublicensing is not allowed; section 10 makes it unnecessary. 65 | 66 | 3. Protecting Users' Legal Rights From Anti-Circumvention Law. 67 | No covered work shall be deemed part of an effective technological measure under any applicable law fulfilling obligations under article 11 of the WIPO copyright treaty adopted on 20 December 1996, or similar laws prohibiting or restricting circumvention of such measures. 68 | 69 | When you convey a covered work, you waive any legal power to forbid circumvention of technological measures to the extent such circumvention is effected by exercising rights under this License with respect to the covered work, and you disclaim any intention to limit operation or modification of the work as a means of enforcing, against the work's users, your or third parties' legal rights to forbid circumvention of technological measures. 70 | 71 | 4. Conveying Verbatim Copies. 72 | You may convey verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice; keep intact all notices stating that this License and any non-permissive terms added in accord with section 7 apply to the code; keep intact all notices of the absence of any warranty; and give all recipients a copy of this License along with the Program. 73 | 74 | You may charge any price or no price for each copy that you convey, and you may offer support or warranty protection for a fee. 75 | 76 | 5. Conveying Modified Source Versions. 77 | You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions: 78 | 79 | a) The work must carry prominent notices stating that you modified it, and giving a relevant date. 80 | 81 | b) The work must carry prominent notices stating that it is released under this License and any conditions added under section 7. This requirement modifies the requirement in section 4 to "keep intact all notices". 82 | 83 | c) You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy. This License will therefore apply, along with any applicable section 7 additional terms, to the whole of the work, and all its parts, regardless of how they are packaged. This License gives no permission to license the work in any other way, but it does not invalidate such permission if you have separately received it. 84 | 85 | d) If the work has interactive user interfaces, each must display Appropriate Legal Notices; however, if the Program has interactive interfaces that do not display Appropriate Legal Notices, your work need not make them do so. 86 | 87 | A compilation of a covered work with other separate and independent works, which are not by their nature extensions of the covered work, and which are not combined with it such as to form a larger program, in or on a volume of a storage or distribution medium, is called an "aggregate" if the compilation and its resulting copyright are not used to limit the access or legal rights of the compilation's users beyond what the individual works permit. Inclusion of a covered work in an aggregate does not cause this License to apply to the other parts of the aggregate. 88 | 89 | 6. Conveying Non-Source Forms. 90 | You may convey a covered work in object code form under the terms of sections 4 and 5, provided that you also convey the machine-readable Corresponding Source under the terms of this License, in one of these ways: 91 | 92 | a) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by the Corresponding Source fixed on a durable physical medium customarily used for software interchange. 93 | 94 | b) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by a written offer, valid for at least three years and valid for as long as you offer spare parts or customer support for that product model, to give anyone who possesses the object code either (1) a copy of the Corresponding Source for all the software in the product that is covered by this License, on a durable physical medium customarily used for software interchange, for a price no more than your reasonable cost of physically performing this conveying of source, or (2) access to copy the Corresponding Source from a network server at no charge. 95 | 96 | c) Convey individual copies of the object code with a copy of the written offer to provide the Corresponding Source. This alternative is allowed only occasionally and noncommercially, and only if you received the object code with such an offer, in accord with subsection 6b. 97 | 98 | d) Convey the object code by offering access from a designated place (gratis or for a charge), and offer equivalent access to the Corresponding Source in the same way through the same place at no further charge. You need not require recipients to copy the Corresponding Source along with the object code. If the place to copy the object code is a network server, the Corresponding Source may be on a different server (operated by you or a third party) that supports equivalent copying facilities, provided you maintain clear directions next to the object code saying where to find the Corresponding Source. Regardless of what server hosts the Corresponding Source, you remain obligated to ensure that it is available for as long as needed to satisfy these requirements. 99 | 100 | e) Convey the object code using peer-to-peer transmission, provided you inform other peers where the object code and Corresponding Source of the work are being offered to the general public at no charge under subsection 6d. 101 | 102 | A separable portion of the object code, whose source code is excluded from the Corresponding Source as a System Library, need not be included in conveying the object code work. 103 | 104 | A "User Product" is either (1) a "consumer product", which means any tangible personal property which is normally used for personal, family, or household purposes, or (2) anything designed or sold for incorporation into a dwelling. In determining whether a product is a consumer product, doubtful cases shall be resolved in favor of coverage. For a particular product received by a particular user, "normally used" refers to a typical or common use of that class of product, regardless of the status of the particular user or of the way in which the particular user actually uses, or expects or is expected to use, the product. A product is a consumer product regardless of whether the product has substantial commercial, industrial or non-consumer uses, unless such uses represent the only significant mode of use of the product. 105 | 106 | "Installation Information" for a User Product means any methods, procedures, authorization keys, or other information required to install and execute modified versions of a covered work in that User Product from a modified version of its Corresponding Source. The information must suffice to ensure that the continued functioning of the modified object code is in no case prevented or interfered with solely because modification has been made. 107 | 108 | If you convey an object code work under this section in, or with, or specifically for use in, a User Product, and the conveying occurs as part of a transaction in which the right of possession and use of the User Product is transferred to the recipient in perpetuity or for a fixed term (regardless of how the transaction is characterized), the Corresponding Source conveyed under this section must be accompanied by the Installation Information. But this requirement does not apply if neither you nor any third party retains the ability to install modified object code on the User Product (for example, the work has been installed in ROM). 109 | 110 | The requirement to provide Installation Information does not include a requirement to continue to provide support service, warranty, or updates for a work that has been modified or installed by the recipient, or for the User Product in which it has been modified or installed. Access to a network may be denied when the modification itself materially and adversely affects the operation of the network or violates the rules and protocols for communication across the network. 111 | 112 | Corresponding Source conveyed, and Installation Information provided, in accord with this section must be in a format that is publicly documented (and with an implementation available to the public in source code form), and must require no special password or key for unpacking, reading or copying. 113 | 114 | 7. Additional Terms. 115 | "Additional permissions" are terms that supplement the terms of this License by making exceptions from one or more of its conditions. Additional permissions that are applicable to the entire Program shall be treated as though they were included in this License, to the extent that they are valid under applicable law. If additional permissions apply only to part of the Program, that part may be used separately under those permissions, but the entire Program remains governed by this License without regard to the additional permissions. 116 | 117 | When you convey a copy of a covered work, you may at your option remove any additional permissions from that copy, or from any part of it. (Additional permissions may be written to require their own removal in certain cases when you modify the work.) You may place additional permissions on material, added by you to a covered work, for which you have or can give appropriate copyright permission. 118 | 119 | Notwithstanding any other provision of this License, for material you add to a covered work, you may (if authorized by the copyright holders of that material) supplement the terms of this License with terms: 120 | 121 | a) Disclaiming warranty or limiting liability differently from the terms of sections 15 and 16 of this License; or 122 | 123 | b) Requiring preservation of specified reasonable legal notices or author attributions in that material or in the Appropriate Legal Notices displayed by works containing it; or 124 | 125 | c) Prohibiting misrepresentation of the origin of that material, or requiring that modified versions of such material be marked in reasonable ways as different from the original version; or 126 | 127 | d) Limiting the use for publicity purposes of names of licensors or authors of the material; or 128 | 129 | e) Declining to grant rights under trademark law for use of some trade names, trademarks, or service marks; or 130 | 131 | f) Requiring indemnification of licensors and authors of that material by anyone who conveys the material (or modified versions of it) with contractual assumptions of liability to the recipient, for any liability that these contractual assumptions directly impose on those licensors and authors. 132 | 133 | All other non-permissive additional terms are considered "further restrictions" within the meaning of section 10. If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term. If a license document contains a further restriction but permits relicensing or conveying under this License, you may add to a covered work material governed by the terms of that license document, provided that the further restriction does not survive such relicensing or conveying. 134 | 135 | If you add terms to a covered work in accord with this section, you must place, in the relevant source files, a statement of the additional terms that apply to those files, or a notice indicating where to find the applicable terms. 136 | 137 | Additional terms, permissive or non-permissive, may be stated in the form of a separately written license, or stated as exceptions; the above requirements apply either way. 138 | 139 | 8. Termination. 140 | 141 | You may not propagate or modify a covered work except as expressly provided under this License. Any attempt otherwise to propagate or modify it is void, and will automatically terminate your rights under this License (including any patent licenses granted under the third paragraph of section 11). 142 | 143 | However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation. 144 | 145 | Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice. 146 | 147 | Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under this License. If your rights have been terminated and not permanently reinstated, you do not qualify to receive new licenses for the same material under section 10. 148 | 149 | 9. Acceptance Not Required for Having Copies. 150 | 151 | You are not required to accept this License in order to receive or run a copy of the Program. Ancillary propagation of a covered work occurring solely as a consequence of using peer-to-peer transmission to receive a copy likewise does not require acceptance. However, nothing other than this License grants you permission to propagate or modify any covered work. These actions infringe copyright if you do not accept this License. Therefore, by modifying or propagating a covered work, you indicate your acceptance of this License to do so. 152 | 153 | 10. Automatic Licensing of Downstream Recipients. 154 | 155 | Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and propagate that work, subject to this License. You are not responsible for enforcing compliance by third parties with this License. 156 | 157 | An "entity transaction" is a transaction transferring control of an organization, or substantially all assets of one, or subdividing an organization, or merging organizations. If propagation of a covered work results from an entity transaction, each party to that transaction who receives a copy of the work also receives whatever licenses to the work the party's predecessor in interest had or could give under the previous paragraph, plus a right to possession of the Corresponding Source of the work from the predecessor in interest, if the predecessor has it or can get it with reasonable efforts. 158 | 159 | You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License. For example, you may not impose a license fee, royalty, or other charge for exercise of rights granted under this License, and you may not initiate litigation (including a cross-claim or counterclaim in a lawsuit) alleging that any patent claim is infringed by making, using, selling, offering for sale, or importing the Program or any portion of it. 160 | 161 | 11. Patents. 162 | 163 | A "contributor" is a copyright holder who authorizes use under this License of the Program or a work on which the Program is based. The work thus licensed is called the contributor's "contributor version". 164 | 165 | A contributor's "essential patent claims" are all patent claims owned or controlled by the contributor, whether already acquired or hereafter acquired, that would be infringed by some manner, permitted by this License, of making, using, or selling its contributor version, but do not include claims that would be infringed only as a consequence of further modification of the contributor version. For purposes of this definition, "control" includes the right to grant patent sublicenses in a manner consistent with the requirements of this License. 166 | 167 | Each contributor grants you a non-exclusive, worldwide, royalty-free patent license under the contributor's essential patent claims, to make, use, sell, offer for sale, import and otherwise run, modify and propagate the contents of its contributor version. 168 | 169 | In the following three paragraphs, a "patent license" is any express agreement or commitment, however denominated, not to enforce a patent (such as an express permission to practice a patent or covenant not to s ue for patent infringement). To "grant" such a patent license to a party means to make such an agreement or commitment not to enforce a patent against the party. 170 | 171 | If you convey a covered work, knowingly relying on a patent license, and the Corresponding Source of the work is not available for anyone to copy, free of charge and under the terms of this License, through a publicly available network server or other readily accessible means, then you must either (1) cause the Corresponding Source to be so available, or (2) arrange to deprive yourself of the benefit of the patent license for this particular work, or (3) arrange, in a manner consistent with the requirements of this License, to extend the patent 172 | license to downstream recipients. "Knowingly relying" means you have actual knowledge that, but for the patent license, your conveying the covered work in a country, or your recipient's use of the covered work in a country, would infringe one or more identifiable patents in that country that you have reason to believe are valid. 173 | 174 | If, pursuant to or in connection with a single transaction or arrangement, you convey, or propagate by procuring conveyance of, a covered work, and grant a patent license to some of the parties receiving the covered work authorizing them to use, propagate, modify or convey a specific copy of the covered work, then the patent license you grant is automatically extended to all recipients of the covered work and works based on it. 175 | 176 | A patent license is "discriminatory" if it does not include within the scope of its coverage, prohibits the exercise of, or is conditioned on the non-exercise of one or more of the rights that are specifically granted under this License. You may not convey a covered work if you are a party to an arrangement with a third party that is in the business of distributing software, under which you make payment to the third party based on the extent of your activity of conveying the work, and under which the third party grants, to any of the parties who would receive the covered work from you, a discriminatory patent license (a) in connection with copies of the covered work conveyed by you (or copies made from those copies), or (b) primarily for and in connection with specific products or compilations that contain the covered work, unless you entered into that arrangement, or that patent license was granted, prior to 28 March 2007. 177 | 178 | Nothing in this License shall be construed as excluding or limiting any implied license or other defenses to infringement that may otherwise be available to you under applicable patent law. 179 | 180 | 12. No Surrender of Others' Freedom. 181 | 182 | If conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot convey a covered work so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may 183 | not convey it at all. For example, if you agree to terms that obligate you to collect a royalty for further conveying from those to whom you convey the Program, the only way you could satisfy both those terms and this License would be to refrain entirely from conveying the Program. 184 | 185 | 13. Remote Network Interaction; Use with the GNU General Public License. 186 | 187 | Notwithstanding any other provision of this License, if you modify the Program, your modified version must prominently offer all users interacting with it remotely through a computer network (if your version supports such interaction) an opportunity to receive the Corresponding Source of your version by providing access to the Corresponding Source from a network server at no charge, through some standard or customary means of facilitating copying of software. This Corresponding Source shall include the Corresponding Source for any work covered by version 3 of the GNU General Public License that is incorporated pursuant to the following paragraph. 188 | 189 | Notwithstanding any other provision of this License, you have permission to link or combine any covered work with a work licensed under version 3 of the GNU General Public License into a single combined work, and to convey the resulting work. The terms of this License will continue to apply to the part which is the covered work, but the work with which it is combined will remain governed by version 3 of the GNU General Public License. 190 | 191 | 14. Revised Versions of this License. 192 | 193 | The Free Software Foundation may publish revised and/or new versions of the GNU Affero General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. 194 | 195 | Each version is given a distinguishing version number. If the Program specifies that a certain numbered version of the GNU Affero General Public License "or any later version" applies to it, you have the option of following the terms and conditions either of that numbered version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the GNU Affero General Public License, you may choose any version ever published by the Free Software Foundation. 196 | 197 | If the Program specifies that a proxy can decide which future versions of the GNU Affero General Public License can be used, that proxy's public statement of acceptance of a version permanently authorizes you to choose that version for the Program. 198 | 199 | Later license versions may give you additional or different permissions. However, no additional obligations are imposed on any author or copyright holder as a result of your choosing to follow a later version. 200 | 201 | 15. Disclaimer of Warranty. 202 | 203 | THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 204 | 205 | 16. Limitation of Liability. 206 | 207 | IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. 208 | 209 | 17. Interpretation of Sections 15 and 16. 210 | 211 | If the disclaimer of warranty and limitation of liability provided above cannot be given local legal effect according to their terms, reviewing courts shall apply local law that most closely approximates an absolute waiver of all civil liability in connection with the Program, unless a warranty or assumption of liability accompanies a copy of the Program in return for a fee. 212 | 213 | END OF TERMS AND CONDITIONS 214 | 215 | How to Apply These Terms to Your New Programs 216 | 217 | If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. 218 | 219 | To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively state the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. 220 | 221 | 222 | Copyright (C) 223 | 224 | This program is free software: you can redistribute it and/or modify 225 | it under the terms of the GNU Affero General Public License as published by 226 | the Free Software Foundation, either version 3 of the License, or 227 | (at your option) any later version. 228 | 229 | This program is distributed in the hope that it will be useful, 230 | but WITHOUT ANY WARRANTY; without even the implied warranty of 231 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 232 | GNU Affero General Public License for more details. 233 | 234 | You should have received a copy of the GNU Affero General Public License 235 | along with this program. If not, see . 236 | 237 | Also add information on how to contact you by electronic and paper mail. 238 | 239 | If your software can interact with users remotely through a computer network, you should also make sure that it provides a way for users to get its source. For example, if your program is a web application, its interface could display a "Source" link that leads users to an archive of the code. There are many ways you could offer source, and different solutions will be better for different programs; see section 13 for the specific requirements. 240 | 241 | You should also get your employer (if you work as a programmer) or school, if any, to sign a "copyright disclaimer" for the program, if necessary. For more information on this, and how to apply and follow the GNU AGPL, see . 242 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # YunoHost apps package linter 2 | 3 | Static analyzer that checks for common issues in Yunohost apps 4 | 5 | ## Usage 6 | Make sure your `python --version` is >=3.11.0 7 | 8 | ```bash 9 | git clone https://github.com/YunoHost/package_linter 10 | cd package_linter 11 | git clone https://github.com//_ynh 12 | python -m venv ./venv # create vritual environment to avoid dependencies' conflict 13 | source venv/bin/activate 14 | pip install -r requirements.txt 15 | ./package_linter.py _ynh 16 | deactivate # if you want to quit the virtual environment 17 | ``` 18 | -------------------------------------------------------------------------------- /lib/lib_package_linter.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | from pathlib import Path 4 | import time 5 | import urllib.request 6 | from typing import Any, Callable, Generator, TypeVar 7 | 8 | import jsonschema 9 | 10 | from lib.print import _print 11 | 12 | # ############################################################################ 13 | # Utilities 14 | # ############################################################################ 15 | 16 | 17 | class c: 18 | HEADER = "\033[94m" 19 | OKBLUE = "\033[94m" 20 | OKGREEN = "\033[92m" 21 | WARNING = "\033[93m" 22 | MAYBE_FAIL = "\033[96m" 23 | FAIL = "\033[91m" 24 | END = "\033[0m" 25 | BOLD = "\033[1m" 26 | UNDERLINE = "\033[4m" 27 | 28 | 29 | class TestReport: 30 | style: str 31 | test_name: str 32 | 33 | def __init__(self, message: str) -> None: 34 | self.message = message 35 | 36 | def display(self, prefix: str = "") -> None: 37 | _print(prefix + self.style % self.message) 38 | 39 | 40 | class Warning(TestReport): 41 | style = c.WARNING + " ! %s " + c.END 42 | 43 | 44 | class Error(TestReport): 45 | style = c.FAIL + " ✘ %s" + c.END 46 | 47 | 48 | class Info(TestReport): 49 | style = " - %s" + c.END 50 | 51 | 52 | class Success(TestReport): 53 | style = c.OKGREEN + " ☺ %s ♥" + c.END 54 | 55 | 56 | class Critical(TestReport): 57 | style = c.FAIL + " ✘✘✘ %s" + c.END 58 | 59 | 60 | def report_warning_not_reliable(message: str) -> None: 61 | _print(c.MAYBE_FAIL + "?", message, c.END) 62 | 63 | 64 | def print_happy(message: str) -> None: 65 | _print(c.OKGREEN + " ☺ ", message, "♥") 66 | 67 | 68 | def urlopen(url: str) -> tuple[int, str]: 69 | try: 70 | conn = urllib.request.urlopen(url) 71 | except urllib.error.HTTPError as e: 72 | return e.code, "" 73 | except urllib.error.URLError as e: 74 | _print("Could not fetch %s : %s" % (url, e)) 75 | return 0, "" 76 | return 200, conn.read().decode("UTF8") 77 | 78 | 79 | def not_empty(file: Path) -> bool: 80 | return file.is_file() and file.stat().st_size > 0 81 | 82 | 83 | def cache_file( 84 | cachefile: Path, ttl_s: int 85 | ) -> Callable[[Callable[..., str]], Callable[..., str]]: 86 | def cache_is_fresh() -> bool: 87 | return cachefile.exists() and time.time() - cachefile.stat().st_mtime < ttl_s 88 | 89 | def decorator(function: Callable[..., str]) -> Callable[..., str]: 90 | def wrapper(*args: Any, **kwargs: Any) -> str: 91 | if not cache_is_fresh(): 92 | cachefile.write_text(function(*args, **kwargs)) 93 | return cachefile.read_text() 94 | 95 | return wrapper 96 | 97 | return decorator 98 | 99 | 100 | @cache_file(Path(".spdx_licenses"), 3600) 101 | def spdx_licenses() -> str: 102 | return urlopen("https://spdx.org/licenses/")[1] 103 | 104 | 105 | @cache_file(Path(".manifest.v2.schema.json"), 3600) 106 | def manifest_v2_schema() -> str: 107 | url = "https://raw.githubusercontent.com/YunoHost/apps/main/schemas/manifest.v2.schema.json" 108 | return urlopen(url)[1] 109 | 110 | 111 | @cache_file(Path(".tests.v1.schema.json"), 3600) 112 | def tests_v1_schema() -> str: 113 | url = "https://raw.githubusercontent.com/YunoHost/apps/main/schemas/tests.v1.schema.json" 114 | return urlopen(url)[1] 115 | 116 | 117 | @cache_file(Path(".config_panel.v1.schema.json"), 3600) 118 | def config_panel_v1_schema() -> str: 119 | url = "https://raw.githubusercontent.com/YunoHost/apps/main/schemas/config_panel.v1.schema.json" 120 | return urlopen(url)[1] 121 | 122 | 123 | def validate_schema( 124 | name: str, schema: dict[str, Any], data: dict[str, Any] 125 | ) -> Generator[Info, None, None]: 126 | v = jsonschema.Draft7Validator(schema) 127 | 128 | for error in v.iter_errors(data): 129 | try: 130 | error_path = " > ".join(error.path) 131 | except TypeError: 132 | error_path = str(error.path) 133 | 134 | yield Info( 135 | f"Error validating {name} using schema: in key {error_path}\n {error.message}" 136 | ) 137 | 138 | 139 | TestResult = Generator[TestReport, None, None] 140 | TestFn = Callable[[Any], TestResult] 141 | 142 | tests: dict[str, list[tuple[TestFn, Any]]] = {} 143 | tests_reports: dict[str, list[Any]] = { 144 | "success": [], 145 | "info": [], 146 | "warning": [], 147 | "error": [], 148 | "critical": [], 149 | } 150 | 151 | 152 | def test(**kwargs: Any) -> Callable[[TestFn], TestFn]: 153 | def decorator(f: TestFn) -> TestFn: 154 | clsname = f.__qualname__.split(".")[0] 155 | if clsname not in tests: 156 | tests[clsname] = [] 157 | tests[clsname].append((f, kwargs)) 158 | return f 159 | 160 | return decorator 161 | 162 | 163 | class TestSuite: 164 | name: str 165 | test_suite_name: str 166 | 167 | def run_tests(self) -> None: 168 | 169 | reports = [] 170 | 171 | for test, options in tests[self.__class__.__name__]: 172 | if "only" in options and self.name not in options["only"]: 173 | continue 174 | if "ignore" in options and self.name in options["ignore"]: 175 | continue 176 | 177 | this_test_reports = list(test(self)) 178 | for report in this_test_reports: 179 | report.test_name = test.__qualname__ 180 | 181 | reports += this_test_reports 182 | 183 | # Display part 184 | 185 | def report_type(report: TestReport) -> str: 186 | return report.__class__.__name__.lower() 187 | 188 | if any(report_type(r) in ["warning", "error", "critical"] for r in reports): 189 | prefix = c.WARNING + "! " 190 | elif any(report_type(r) in ["info"] for r in reports): 191 | prefix = "ⓘ " 192 | else: 193 | prefix = c.OKGREEN + "✔ " 194 | 195 | _print(" " + c.BOLD + prefix + c.OKBLUE + self.test_suite_name + c.END) 196 | 197 | if len(reports): 198 | _print("") 199 | 200 | for report in reports: 201 | report.display(prefix=" ") 202 | 203 | if len(reports): 204 | _print("") 205 | 206 | for report in reports: 207 | tests_reports[report_type(report)].append((report.test_name, report)) 208 | 209 | def run_single_test(self, test: TestFn) -> None: 210 | 211 | reports = list(test(self)) 212 | 213 | def report_type(report: TestReport) -> str: 214 | return report.__class__.__name__.lower() 215 | 216 | for report in reports: 217 | report.display() 218 | test_name = test.__qualname__ 219 | tests_reports[report_type(report)].append((test_name, report)) 220 | -------------------------------------------------------------------------------- /lib/nginxparser/nginxparser.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # type: ignore 3 | """Very low-level nginx config parser based on pyparsing.""" 4 | 5 | # Taken from https://github.com/certbot/certbot (Apache licensed) 6 | # Itself forked from https://github.com/fatiherikli/nginxparser (MIT Licensed) 7 | import copy 8 | import logging 9 | 10 | from pyparsing import ( 11 | Literal, 12 | White, 13 | Forward, 14 | Group, 15 | Optional, 16 | OneOrMore, 17 | QuotedString, 18 | Regex, 19 | ZeroOrMore, 20 | Combine, 21 | ) 22 | from pyparsing import stringEnd 23 | from pyparsing import restOfLine 24 | import six 25 | 26 | logger = logging.getLogger(__name__) 27 | 28 | 29 | class RawNginxParser(object): 30 | # pylint: disable=expression-not-assigned 31 | # pylint: disable=pointless-statement 32 | """A class that parses nginx configuration with pyparsing.""" 33 | 34 | # constants 35 | space = Optional(White()).leaveWhitespace() 36 | required_space = White().leaveWhitespace() 37 | 38 | left_bracket = Literal("{").suppress() 39 | right_bracket = space + Literal("}").suppress() 40 | semicolon = Literal(";").suppress() 41 | dquoted = QuotedString('"', multiline=True, unquoteResults=False, escChar="\\") 42 | squoted = QuotedString("'", multiline=True, unquoteResults=False, escChar="\\") 43 | quoted = dquoted | squoted 44 | head_tokenchars = Regex(r"(\$\{)|[^{};\s'\"]") # if (last_space) 45 | tail_tokenchars = Regex(r"(\$\{)|[^{;\s]") # else 46 | tokenchars = Combine(head_tokenchars + ZeroOrMore(tail_tokenchars)) 47 | paren_quote_extend = Combine(quoted + Literal(")") + ZeroOrMore(tail_tokenchars)) 48 | # note: ')' allows extension, but then we fall into else, not last_space. 49 | 50 | token = paren_quote_extend | tokenchars | quoted 51 | 52 | whitespace_token_group = space + token + ZeroOrMore(required_space + token) + space 53 | assignment = whitespace_token_group + semicolon 54 | 55 | comment = space + Literal("#") + restOfLine 56 | 57 | block = Forward() 58 | 59 | # order matters! see issue 518, and also http { # server { \n} 60 | contents = Group(comment) | Group(block) | Group(assignment) 61 | 62 | block_begin = Group(whitespace_token_group) 63 | block_innards = Group(ZeroOrMore(contents) + space).leaveWhitespace() 64 | block << block_begin + left_bracket + block_innards + right_bracket 65 | 66 | script = OneOrMore(contents) + space + stringEnd 67 | script.parseWithTabs().leaveWhitespace() 68 | 69 | def __init__(self, source): 70 | self.source = source 71 | 72 | def parse(self): 73 | """Returns the parsed tree.""" 74 | return self.script.parseString(self.source) 75 | 76 | def as_list(self): 77 | """Returns the parsed tree as a list.""" 78 | return self.parse().asList() 79 | 80 | 81 | class RawNginxDumper(object): 82 | # pylint: disable=too-few-public-methods 83 | """A class that dumps nginx configuration from the provided tree.""" 84 | 85 | def __init__(self, blocks): 86 | self.blocks = blocks 87 | 88 | def __iter__(self, blocks=None): 89 | """Iterates the dumped nginx content.""" 90 | blocks = blocks or self.blocks 91 | for b0 in blocks: 92 | if isinstance(b0, six.string_types): 93 | yield b0 94 | continue 95 | item = copy.deepcopy(b0) 96 | if spacey(item[0]): 97 | yield item.pop(0) # indentation 98 | if not item: 99 | continue 100 | 101 | if isinstance(item[0], list): # block 102 | yield "".join(item.pop(0)) + "{" 103 | for parameter in item.pop(0): 104 | for line in self.__iter__([parameter]): # negate "for b0 in blocks" 105 | yield line 106 | yield "}" 107 | else: # not a block - list of strings 108 | semicolon = ";" 109 | if ( 110 | isinstance(item[0], six.string_types) and item[0].strip() == "#" 111 | ): # comment 112 | semicolon = "" 113 | yield "".join(item) + semicolon 114 | 115 | def __str__(self): 116 | """Return the parsed block as a string.""" 117 | return "".join(self) 118 | 119 | 120 | # Shortcut functions to respect Python's serialization interface 121 | # (like pyyaml, picker or json) 122 | 123 | 124 | def loads(source): 125 | """Parses from a string. 126 | 127 | :param str source: The string to parse 128 | :returns: The parsed tree 129 | :rtype: list 130 | 131 | """ 132 | return UnspacedList(RawNginxParser(source).as_list()) 133 | 134 | 135 | def load(_file): 136 | """Parses from a file. 137 | 138 | :param file _file: The file to parse 139 | :returns: The parsed tree 140 | :rtype: list 141 | 142 | """ 143 | return loads(_file.read()) 144 | 145 | 146 | def dumps(blocks): 147 | """Dump to a string. 148 | 149 | :param UnspacedList block: The parsed tree 150 | :param int indentation: The number of spaces to indent 151 | :rtype: str 152 | 153 | """ 154 | return str(RawNginxDumper(blocks.spaced)) 155 | 156 | 157 | def dump(blocks, _file): 158 | """Dump to a file. 159 | 160 | :param UnspacedList block: The parsed tree 161 | :param file _file: The file to dump to 162 | :param int indentation: The number of spaces to indent 163 | :rtype: NoneType 164 | 165 | """ 166 | return _file.write(dumps(blocks)) 167 | 168 | 169 | def spacey(x): 170 | return (isinstance(x, six.string_types) and x.isspace()) or x == "" 171 | 172 | 173 | class UnspacedList(list): 174 | """Wrap a list [of lists], making any whitespace entries magically invisible""" 175 | 176 | def __init__(self, list_source): 177 | # ensure our argument is not a generator, and duplicate any sublists 178 | self.spaced = copy.deepcopy(list(list_source)) 179 | self.dirty = False 180 | 181 | # Turn self into a version of the source list that has spaces removed 182 | # and all sub-lists also UnspacedList()ed 183 | list.__init__(self, list_source) 184 | for i, entry in reversed(list(enumerate(self))): 185 | if isinstance(entry, list): 186 | sublist = UnspacedList(entry) 187 | list.__setitem__(self, i, sublist) 188 | self.spaced[i] = sublist.spaced 189 | elif spacey(entry): 190 | # don't delete comments 191 | if "#" not in self[:i]: 192 | list.__delitem__(self, i) 193 | 194 | def _coerce(self, inbound): 195 | """ 196 | Coerce some inbound object to be appropriately usable in this object 197 | 198 | :param inbound: string or None or list or UnspacedList 199 | :returns: (coerced UnspacedList or string or None, spaced equivalent) 200 | :rtype: tuple 201 | 202 | """ 203 | if not isinstance(inbound, list): # str or None 204 | return (inbound, inbound) 205 | else: 206 | if not hasattr(inbound, "spaced"): 207 | inbound = UnspacedList(inbound) 208 | return (inbound, inbound.spaced) 209 | 210 | def insert(self, i, x): 211 | item, spaced_item = self._coerce(x) 212 | slicepos = self._spaced_position(i) if i < len(self) else len(self.spaced) 213 | self.spaced.insert(slicepos, spaced_item) 214 | if not spacey(item): 215 | list.insert(self, i, item) 216 | self.dirty = True 217 | 218 | def append(self, x): 219 | item, spaced_item = self._coerce(x) 220 | self.spaced.append(spaced_item) 221 | if not spacey(item): 222 | list.append(self, item) 223 | self.dirty = True 224 | 225 | def extend(self, x): 226 | item, spaced_item = self._coerce(x) 227 | self.spaced.extend(spaced_item) 228 | list.extend(self, item) 229 | self.dirty = True 230 | 231 | def __add__(self, other): 232 | zzz = copy.deepcopy(self) 233 | zzz.extend(other) 234 | zzz.dirty = True 235 | return zzz 236 | 237 | def pop(self, _i=None): 238 | raise NotImplementedError("UnspacedList.pop() not yet implemented") 239 | 240 | def remove(self, _): 241 | raise NotImplementedError("UnspacedList.remove() not yet implemented") 242 | 243 | def reverse(self): 244 | raise NotImplementedError("UnspacedList.reverse() not yet implemented") 245 | 246 | def sort(self, _cmp=None, _key=None, _Rev=None): 247 | raise NotImplementedError("UnspacedList.sort() not yet implemented") 248 | 249 | def __setslice__(self, _i, _j, _newslice): 250 | raise NotImplementedError( 251 | "Slice operations on UnspacedLists not yet implemented" 252 | ) 253 | 254 | def __setitem__(self, i, value): 255 | if isinstance(i, slice): 256 | raise NotImplementedError( 257 | "Slice operations on UnspacedLists not yet implemented" 258 | ) 259 | item, spaced_item = self._coerce(value) 260 | self.spaced.__setitem__(self._spaced_position(i), spaced_item) 261 | if not spacey(item): 262 | list.__setitem__(self, i, item) 263 | self.dirty = True 264 | 265 | def __delitem__(self, i): 266 | self.spaced.__delitem__(self._spaced_position(i)) 267 | list.__delitem__(self, i) 268 | self.dirty = True 269 | 270 | def __deepcopy__(self, memo): 271 | new_spaced = copy.deepcopy(self.spaced, memo=memo) 272 | zzz = UnspacedList(new_spaced) 273 | zzz.dirty = self.dirty 274 | return zzz 275 | 276 | def is_dirty(self): 277 | """Recurse through the parse tree to figure out if any sublists are dirty""" 278 | if self.dirty: 279 | return True 280 | return any((isinstance(x, UnspacedList) and x.is_dirty() for x in self)) 281 | 282 | def _spaced_position(self, idx): 283 | "Convert from indexes in the unspaced list to positions in the spaced one" 284 | pos = spaces = 0 285 | # Normalize indexes like list[-1] etc, and save the result 286 | if idx < 0: 287 | idx = len(self) + idx 288 | if not 0 <= idx < len(self): 289 | raise IndexError("list index out of range") 290 | idx0 = idx 291 | # Count the number of spaces in the spaced list before idx in the unspaced one 292 | while idx != -1: 293 | if spacey(self.spaced[pos]): 294 | spaces += 1 295 | else: 296 | idx -= 1 297 | pos += 1 298 | return idx0 + spaces 299 | -------------------------------------------------------------------------------- /lib/print.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | from typing import Any 4 | 5 | output = "plain" 6 | 7 | 8 | def _print(*args: Any, **kwargs: Any) -> None: 9 | if not is_json_output(): 10 | print(*args, **kwargs) 11 | 12 | 13 | def set_output_json() -> None: 14 | global output 15 | output = "json" 16 | 17 | 18 | def is_json_output() -> bool: 19 | return output == "json" 20 | -------------------------------------------------------------------------------- /package_linter.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf8 -*- 3 | 4 | import argparse 5 | from pathlib import Path 6 | 7 | from lib.lib_package_linter import c 8 | from lib.print import _print, set_output_json 9 | from tests.test_app import App 10 | 11 | 12 | def main(): 13 | parser = argparse.ArgumentParser() 14 | parser.add_argument("app_path", type=Path, help="The path to the app to lint") 15 | parser.add_argument( 16 | "--json", action="store_true", help="Output json instead of plain text" 17 | ) 18 | args = parser.parse_args() 19 | 20 | if args.json: 21 | set_output_json() 22 | 23 | _print( 24 | """ [YunoHost App Package Linter] 25 | 26 | App packaging documentation - https://yunohost.org/packaging_apps 27 | App package example - https://github.com/YunoHost/example_ynh 28 | Official helpers - https://yunohost.org/packaging_apps_helpers 29 | 30 | If you believe this linter returns false negative (warnings / errors which shouldn't happen), 31 | please report them on https://github.com/YunoHost/package_linter/issues 32 | """ 33 | ) 34 | 35 | app = App(args.app_path) 36 | app.analyze() 37 | 38 | 39 | if __name__ == "__main__": 40 | main() 41 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | jsonschema 2 | toml -------------------------------------------------------------------------------- /tests/test_app.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import copy 4 | import json 5 | import os 6 | from pathlib import Path 7 | import subprocess 8 | import sys 9 | import tomllib 10 | from typing import Generator 11 | 12 | from lib.lib_package_linter import ( 13 | Error, 14 | Info, 15 | Success, 16 | TestResult, 17 | TestSuite, 18 | Warning, 19 | config_panel_v1_schema, 20 | not_empty, 21 | test, 22 | tests_reports, 23 | validate_schema, 24 | ) 25 | from lib.print import _print, is_json_output 26 | from tests.test_catalog import AppCatalog 27 | from tests.test_configurations import Configurations 28 | from tests.test_manifest import Manifest 29 | from tests.test_scripts import Script 30 | 31 | scriptnames = ["_common.sh", "install", "remove", "upgrade", "backup", "restore"] 32 | 33 | 34 | # ############################################################################ 35 | # Helper list 36 | # ############################################################################ 37 | 38 | # Generated May 20 2024 using: 39 | # cat /path/to/yunohost/data/helpers.d/* | grep "^ynh_" | tr -d '(){ ' > helperlist 2>/dev/null 40 | # for HELPER in $(cat helperlist); do REQUIRE=$(grep -whB5 "^$HELPER" /path/to/yunohost/data/helpers.d/* 2>/dev/null | grep "Requires .* or higher\." | grep -o -E "[0-9].[0-9].[0-9]"); echo "'$HELPER': '$REQUIRE'",; done | tr "'" '"' 41 | 42 | official_helpers = { 43 | "ynh_install_apps": "", 44 | "ynh_remove_apps": "", 45 | "ynh_spawn_app_shell": "", 46 | "ynh_wait_dpkg_free": "3.3.1", 47 | "ynh_package_is_installed": "2.2.4", 48 | "ynh_package_version": "2.2.4", 49 | "ynh_apt": "2.4.0", 50 | "ynh_package_update": "2.2.4", 51 | "ynh_package_install": "2.2.4", 52 | "ynh_package_remove": "2.2.4", 53 | "ynh_package_autoremove": "2.2.4", 54 | "ynh_package_autopurge": "2.7.2", 55 | "ynh_package_install_from_equivs": "2.2.4", 56 | "ynh_install_app_dependencies": "2.6.4", 57 | "ynh_add_app_dependencies": "3.8.1", 58 | "ynh_remove_app_dependencies": "2.6.4", 59 | "ynh_install_extra_app_dependencies": "3.8.1", 60 | "ynh_install_extra_repo": "3.8.1", 61 | "ynh_remove_extra_repo": "3.8.1", 62 | "ynh_add_repo": "3.8.1", 63 | "ynh_pin_repo": "3.8.1", 64 | "ynh_backup": "2.4.0", 65 | "ynh_restore": "2.6.4", 66 | "ynh_restore_file": "2.6.4", 67 | "ynh_store_file_checksum": "2.6.4", 68 | "ynh_backup_if_checksum_is_different": "2.6.4", 69 | "ynh_delete_file_checksum": "3.3.1", 70 | "ynh_backup_archive_exists": "", 71 | "ynh_backup_before_upgrade": "2.7.2", 72 | "ynh_restore_upgradebackup": "2.7.2", 73 | "ynh_app_config_get_one": "", 74 | # Commenting out config panel helpers 75 | # that may legitimately be overwritten from config script 76 | # "ynh_app_config_get": "", 77 | # "ynh_app_config_show": "", 78 | # "ynh_app_config_validate": "", 79 | "ynh_app_config_apply_one": "", 80 | # "ynh_app_config_apply": "", 81 | # "ynh_app_action_run": "", 82 | # "ynh_app_config_run": "", 83 | "ynh_add_fail2ban_config": "4.1.0", 84 | "ynh_remove_fail2ban_config": "3.5.0", 85 | "ynh_handle_getopts_args": "3.2.2", 86 | "ynh_go_try_bash_extension": "", 87 | "ynh_use_go": "", 88 | "ynh_install_go": "", 89 | "ynh_remove_go": "", 90 | "ynh_cleanup_go": "", 91 | "ynh_get_ram": "3.8.1", 92 | "ynh_require_ram": "3.8.1", 93 | "ynh_die": "2.4.0", 94 | "ynh_print_info": "3.2.0", 95 | "ynh_print_log": "3.2.0", 96 | "ynh_print_warn": "3.2.0", 97 | "ynh_print_err": "3.2.0", 98 | "ynh_exec_err": "3.2.0", 99 | "ynh_exec_warn": "3.2.0", 100 | "ynh_exec_warn_less": "3.2.0", 101 | "ynh_exec_quiet": "3.2.0", 102 | "ynh_exec_fully_quiet": "3.2.0", 103 | "ynh_exec_and_print_stderr_only_if_error": "", 104 | "ynh_print_OFF": "3.2.0", 105 | "ynh_print_ON": "3.2.0", 106 | "ynh_script_progression": "3.5.0", 107 | "ynh_return": "3.6.0", 108 | "ynh_use_logrotate": "2.6.4", 109 | "ynh_remove_logrotate": "2.6.4", 110 | "ynh_multimedia_build_main_dir": "", 111 | "ynh_multimedia_addfolder": "", 112 | "ynh_multimedia_addaccess": "", 113 | "ynh_mysql_connect_as": "2.2.4", 114 | "ynh_mysql_execute_as_root": "2.2.4", 115 | "ynh_mysql_execute_file_as_root": "2.2.4", 116 | "ynh_mysql_create_db": "2.2.4", 117 | "ynh_mysql_drop_db": "2.2.4", 118 | "ynh_mysql_dump_db": "2.2.4", 119 | "ynh_mysql_create_user": "2.2.4", 120 | "ynh_mysql_user_exists": "2.2.4", 121 | "ynh_mysql_drop_user": "2.2.4", 122 | "ynh_mysql_setup_db": "2.6.4", 123 | "ynh_mysql_remove_db": "2.6.4", 124 | "ynh_find_port": "2.6.4", 125 | "ynh_port_available": "3.8.0", 126 | "ynh_validate_ip": "2.2.4", 127 | "ynh_validate_ip4": "2.2.4", 128 | "ynh_validate_ip6": "2.2.4", 129 | "ynh_add_nginx_config": "4.1.0", 130 | "ynh_remove_nginx_config": "2.7.2", 131 | "ynh_change_url_nginx_config": "11.1.9", 132 | "ynh_use_nodejs": "2.7.1", 133 | "ynh_install_nodejs": "2.7.1", 134 | "ynh_remove_nodejs": "2.7.1", 135 | "ynh_cron_upgrade_node": "2.7.1", 136 | "ynh_permission_create": "3.7.0", 137 | "ynh_permission_delete": "3.7.0", 138 | "ynh_permission_exists": "3.7.0", 139 | "ynh_permission_url": "3.7.0", 140 | "ynh_permission_update": "3.7.0", 141 | "ynh_permission_has_user": "3.7.1", 142 | "ynh_legacy_permissions_exists": "4.1.2", 143 | "ynh_legacy_permissions_delete_all": "4.1.2", 144 | "ynh_add_fpm_config": "4.1.0", 145 | "ynh_remove_fpm_config": "2.7.2", 146 | "ynh_get_scalable_phpfpm": "", 147 | "ynh_composer_exec": "", 148 | "ynh_install_composer": "", 149 | "ynh_psql_connect_as": "3.5.0", 150 | "ynh_psql_execute_as_root": "3.5.0", 151 | "ynh_psql_execute_file_as_root": "3.5.0", 152 | "ynh_psql_create_db": "3.5.0", 153 | "ynh_psql_drop_db": "3.5.0", 154 | "ynh_psql_dump_db": "3.5.0", 155 | "ynh_psql_create_user": "3.5.0", 156 | "ynh_psql_user_exists": "3.5.0", 157 | "ynh_psql_database_exists": "3.5.0", 158 | "ynh_psql_drop_user": "3.5.0", 159 | "ynh_psql_setup_db": "2.7.1", 160 | "ynh_psql_remove_db": "2.7.1", 161 | "ynh_psql_test_if_first_run": "2.7.1", 162 | "ynh_redis_get_free_db": "", 163 | "ynh_redis_remove_db": "", 164 | "ynh_use_ruby": "", 165 | "ynh_install_ruby": "", 166 | "ynh_remove_ruby": "", 167 | "ynh_cleanup_ruby": "", 168 | "ynh_ruby_try_bash_extension": "", 169 | "ynh_app_setting_get": "2.2.4", 170 | "ynh_app_setting_set": "2.2.4", 171 | "ynh_app_setting_delete": "2.2.4", 172 | "ynh_app_setting": "", 173 | "ynh_webpath_available": "2.6.4", 174 | "ynh_webpath_register": "2.6.4", 175 | "ynh_string_random": "2.2.4", 176 | "ynh_replace_string": "2.6.4", 177 | "ynh_replace_special_string": "2.7.7", 178 | "ynh_sanitize_dbid": "2.2.4", 179 | "ynh_normalize_url_path": "2.6.4", 180 | "ynh_add_systemd_config": "4.1.0", 181 | "ynh_remove_systemd_config": "2.7.2", 182 | "ynh_systemd_action": "3.5.0", 183 | "ynh_clean_check_starting": "3.5.0", 184 | "ynh_user_exists": "2.2.4", 185 | "ynh_user_get_info": "2.2.4", 186 | "ynh_user_list": "2.4.0", 187 | "ynh_system_user_exists": "2.2.4", 188 | "ynh_system_group_exists": "3.5.0", 189 | "ynh_system_user_create": "2.6.4", 190 | "ynh_system_user_delete": "2.6.4", 191 | "ynh_exec_as": "4.1.7", 192 | "ynh_exit_properly": "2.6.4", 193 | "ynh_abort_if_errors": "2.6.4", 194 | "ynh_setup_source": "2.6.4", 195 | "ynh_local_curl": "2.6.4", 196 | "ynh_add_config": "4.1.0", 197 | "ynh_replace_vars": "4.1.0", 198 | "ynh_read_var_in_file": "", 199 | "ynh_write_var_in_file": "", 200 | "ynh_render_template": "", 201 | "ynh_get_debian_release": "2.7.1", 202 | "ynh_secure_remove": "2.6.4", 203 | "ynh_read_manifest": "3.5.0", 204 | "ynh_app_upstream_version": "3.5.0", 205 | "ynh_app_package_version": "3.5.0", 206 | "ynh_check_app_version_changed": "3.5.0", 207 | "ynh_compare_current_package_version": "3.8.0", 208 | } 209 | 210 | deprecated_helpers_in_v2 = [ 211 | ("ynh_clean_setup", "(?)"), 212 | ("ynh_abort_if_errors", "nothing, handled by the core, just get rid of it"), 213 | ("ynh_backup_before_upgrade", "nothing, handled by the core, just get rid of it"), 214 | ("ynh_restore_upgradebackup", "nothing, handled by the core, just get rid of it"), 215 | ("ynh_system_user_create", "the system_user resource"), 216 | ("ynh_system_user_delete", "the system_user resource"), 217 | ("ynh_webpath_register", "the permission resource"), 218 | ("ynh_webpath_available", "the permission resource"), 219 | ("ynh_permission_update", "the permission resource"), 220 | ("ynh_permission_create", "the permission resource"), 221 | ("ynh_permission_exists", "the permission resource"), 222 | ("ynh_legacy_permissions_exists", "the permission resource"), 223 | ("ynh_legacy_permissions_delete_all", "the permission resource"), 224 | ("ynh_install_app_dependencies", "the apt resource"), 225 | ("ynh_install_extra_app_dependencies", "the apt source"), 226 | ("ynh_remove_app_dependencies", "the apt resource"), 227 | ("ynh_psql_test_if_first_run", "the database resource"), 228 | ("ynh_mysql_setup_db", "the database resource"), 229 | ("ynh_psql_setup_db", "the database resource"), 230 | ("ynh_mysql_remove_db", "the database resource"), 231 | ("ynh_psql_remove_db", "the database resource"), 232 | ("ynh_find_port", "the port resource"), 233 | ( 234 | "ynh_send_readme_to_admin", 235 | "the doc/POST_INSTALL.md or POST_UPGRADE.md mechanism", 236 | ), 237 | ] 238 | 239 | 240 | class App(TestSuite): 241 | def __init__(self, path: Path) -> None: 242 | 243 | _print(f" Analyzing app {path} ...") 244 | self.path = path 245 | self.manifest_ = Manifest(self.path) 246 | self.manifest = self.manifest_.manifest 247 | self.scripts = { 248 | f: Script(self.path, f, self.manifest.get("id", "")) for f in scriptnames 249 | } 250 | self.configurations = Configurations(self) 251 | self.app_catalog = AppCatalog(self.manifest["id"]) 252 | 253 | self.test_suite_name = "General stuff, misc helper usage" 254 | 255 | _print() 256 | 257 | def analyze(self) -> None: 258 | 259 | self.manifest_.run_tests() 260 | 261 | for script in [self.scripts[s] for s in scriptnames if self.scripts[s].exists]: 262 | script.run_tests() 263 | 264 | self.run_tests() 265 | self.configurations.run_tests() 266 | self.app_catalog.run_tests() 267 | 268 | self.report() 269 | 270 | def report(self) -> None: 271 | 272 | _print(" =======") 273 | 274 | # These are meant to be the last stuff running, they are based on 275 | # previously computed errors/warning/successes 276 | self.run_single_test(App.qualify_for_level_7) 277 | self.run_single_test(App.qualify_for_level_8) 278 | self.run_single_test(App.qualify_for_level_9) 279 | 280 | if is_json_output(): 281 | print( 282 | json.dumps( 283 | { 284 | "success": [test for test, _ in tests_reports["success"]], 285 | "info": [test for test, _ in tests_reports["info"]], 286 | "warning": [test for test, _ in tests_reports["warning"]], 287 | "error": [test for test, _ in tests_reports["error"]], 288 | "critical": [test for test, _ in tests_reports["critical"]], 289 | }, 290 | indent=4, 291 | ) 292 | ) 293 | return 294 | 295 | if tests_reports["error"] or tests_reports["critical"]: 296 | sys.exit(1) 297 | 298 | def qualify_for_level_7(self) -> Generator[Success, None, None]: 299 | 300 | if tests_reports["critical"]: 301 | _print(" There are some critical issues in this app :(") 302 | elif tests_reports["error"]: 303 | _print(" Uhoh there are some errors to be fixed :(") 304 | elif len(tests_reports["warning"]) >= 3: 305 | _print(" Still some warnings to be fixed :s") 306 | elif len(tests_reports["warning"]) == 2: 307 | _print(" Only 2 warnings remaining! You can do it!") 308 | elif len(tests_reports["warning"]) == 1: 309 | _print(" Only 1 warning remaining! You can do it!") 310 | else: 311 | yield Success( 312 | "Not even a warning! Congratz and thank you for keeping this package up to date with good practices! This app qualifies for level 7!" 313 | ) 314 | 315 | def qualify_for_level_8(self) -> Generator[Success, None, None]: 316 | 317 | successes = [test.split(".")[1] for test, _ in tests_reports["success"]] 318 | 319 | # Level 8 = qualifies for level 7 + maintained + long term good quality 320 | catalog_infos = self.app_catalog.catalog_infos 321 | antifeatures = catalog_infos and catalog_infos.get("antifeatures", []) 322 | 323 | if any( 324 | af in antifeatures 325 | for af in [ 326 | "package-not-maintained", 327 | "deprecated-software", 328 | "alpha-software", 329 | "replaced-by-another-app", 330 | ] 331 | ): 332 | _print( 333 | " In the catalog, the app is flagged as not maintained / deprecated / alpha or replaced by another app, therefore does not qualify for level 8" 334 | ) 335 | elif ( 336 | "qualify_for_level_7" in successes 337 | and "is_long_term_good_quality" in successes 338 | ): 339 | yield Success( 340 | "The app is maintained and long-term good quality, and therefore qualifies for level 8!" 341 | ) 342 | 343 | def qualify_for_level_9(self) -> Generator[Success, None, None]: 344 | 345 | if self.app_catalog.catalog_infos.get("high_quality", False): 346 | yield Success("The app is flagged as high-quality in the app catalog") 347 | 348 | ######################################### 349 | # _____ _ # 350 | # | __ \ | | # 351 | # | | \/ ___ _ __ ___ _ __ __ _| | # 352 | # | | __ / _ \ '_ \ / _ \ '__/ _` | | # 353 | # | |_\ \ __/ | | | __/ | | (_| | | # 354 | # \____/\___|_| |_|\___|_| \__,_|_| # 355 | # # 356 | ######################################### 357 | 358 | @test() 359 | def mandatory_scripts(app) -> TestResult: 360 | filenames = ( 361 | "LICENSE", 362 | "README.md", 363 | "scripts/install", 364 | "scripts/remove", 365 | "scripts/upgrade", 366 | "scripts/backup", 367 | "scripts/restore", 368 | ) 369 | 370 | for filename in filenames: 371 | if not not_empty(app.path / filename): 372 | yield Error("Providing %s is mandatory" % filename) 373 | 374 | license = app.path / "LICENSE" 375 | if not_empty(license): 376 | license_content = license.read_text() 377 | if "File containing the license of your package" in license_content: 378 | yield Error("You should put an actual license in LICENSE...") 379 | 380 | @test() 381 | def doc_dir(app) -> TestResult: 382 | 383 | if not (app.path / "doc").exists(): 384 | yield Error( 385 | """Having a doc/ folder is now mandatory in packaging v2 and is expected to contain : 386 | - (recommended) doc/DESCRIPTION.md : a long description of the app, typically around 5~20 lines, for example to list features 387 | - (recommended) doc/screenshots/ : a folder containing at least one .png (or .jpg) screenshot of the app 388 | - (if relevant) doc/ADMIN.md : an admin doc page meant to provide general info about adminstrating this app, will be available in yunohost's webadmin 389 | - (if relevant) doc/SOME_OTHER_PAGE.md : an arbitrarily named admin doc page meant to provide info on a specific topic, will be available in yunohost's webadmin 390 | - (if relevant) doc/PRE_INSTALL.md, POST_INSTALL.md : important informations to display to the user before/after the install (similar mechanism exists for upgrade) 391 | """ 392 | ) 393 | 394 | if (app.path / "doc" / "screenshots").exists(): 395 | du_output = subprocess.check_output( 396 | ["du", "-sb", app.path / "doc" / "screenshots"], shell=False 397 | ) 398 | screenshots_size = int(du_output.split()[0]) 399 | if screenshots_size > 1024 * 1000: 400 | yield Warning( 401 | "Please keep the content of doc/screenshots under ~512Kb. Having screenshots bigger than 512kb is probably a waste of resource and will take unecessarily long time to load on the webadmin UI and app catalog." 402 | ) 403 | elif screenshots_size > 512 * 1000: 404 | yield Info( 405 | "Please keep the content of doc/screenshots under ~512Kb. Having screenshots bigger than 512kb is probably a waste of resource and will take unecessarily long time to load on the webadmin UI and app catalog." 406 | ) 407 | 408 | for file in (app.path / "doc" / "screenshots").rglob("*"): 409 | filename = file.name 410 | if Path.is_dir(file): 411 | continue 412 | if filename == ".gitkeep": 413 | continue 414 | if all( 415 | not filename.lower().endswith(ext) 416 | for ext in [".jpg", ".jpeg", ".png", ".gif", ".webp"] 417 | ): 418 | yield Warning( 419 | "In the doc/screenshots folder, only .jpg, .jpeg, .png, .webp and .gif are accepted" 420 | ) 421 | break 422 | 423 | @test() 424 | def doc_dir_v2(app) -> TestResult: 425 | 426 | if (app.path / "doc").exists() and not ( 427 | app.path / "doc" / "DESCRIPTION.md" 428 | ).exists(): 429 | yield Error( 430 | "A DESCRIPTION.md is now mandatory in packaging v2 and is meant to contains an extensive description of what the app is and does. Consider also adding a '/doc/screenshots/' folder with a few screenshots of what the app looks like." 431 | ) 432 | elif ( 433 | os.system( 434 | rf'grep -inrq "Some long and extensive description\|lorem ipsum dolor sit amet\|Ut enim ad minim veniam" {app.path}/doc/DESCRIPTION.md' 435 | ) 436 | == 0 437 | ): 438 | yield Error("It looks like DESCRIPTION.md just contains placeholder texts") 439 | 440 | if (app.path / "doc" / "DISCLAIMER.md").exists(): 441 | yield Warning( 442 | """DISCLAIMER.md has been replaced with several files in packaging v2 to improve the UX and provide the user with key information at the appropriate step of the app install / upgrade cycles. 443 | 444 | You are encouraged to split its infos into: 445 | 446 | - Integration-related infos (eg. LDAP/SSO support, arch support, resource usage, ...) 447 | -> neant to go in the 'integration' section of the manifest.toml 448 | 449 | - Antifeatures-related infos (eg. alpha/deprecated software, arbitrary limiations, ...) 450 | -> these are now formalized using the 'antifeatures' mechanism in the app catalog directly : cf https://github.com/YunoHost/apps/blob/main/antifeatures.yml and the 'antifeatures' key in apps.json 451 | 452 | - Important infos that the admin should be made aware of *before* or *after* the install 453 | -> infos *before* the install are meant to go in 'doc/PRE_INSTALL.md' 454 | -> infos *after* the install are meant to go in 'doc/POST_INSTALL.md' (mostly meant to replace ynh_send_readme_to_admin, typically tips about how to login for the first time on the app / finish the install, ...). 455 | -> these will be shown to the admin before/after the install (and the post_install notif will also be available in the app info page) 456 | -> note that in these files, the __FOOBAR__ syntax is supported and replaced with the corresponding 'foobar' setting. 457 | 458 | - General admin-related infos (eg. how to access the admin interface of the app, how to install plugin, etc) 459 | -> meant to go in 'doc/ADMIN.md' which shall be made available in the app info page in the webadmin after installation. 460 | -> if relevant, you can also create custom doc page, just create 'doc/WHATEVER.MD' and this will correspond to a specific documentation tab in the webadmin. 461 | -> note that in these files, the __FOOBAR__ syntax is supported and replaced with the corresponding 'foobar' setting. 462 | """ 463 | ) 464 | 465 | @test() 466 | def admin_has_to_finish_install(app) -> TestResult: 467 | 468 | # Mywebapp has a legit use case for this 469 | if app.manifest.get("id") == "my_webapp": 470 | return 471 | 472 | cmd = f"grep -q -IhEr '__DB_PWD__' '{app.path}/doc/'" 473 | if (app.path / "doc").exists() and os.system(cmd) == 0: 474 | yield Warning( 475 | "(doc folder) It looks like this app requires the admin to finish the install by entering DB credentials. Unless it's absolutely not easily automatizable, this should be handled automatically by the app install script using curl calls, or (CLI tools provided by the upstream maybe ?)." 476 | ) 477 | 478 | @test() 479 | def disclaimer_wording_or_placeholder(app) -> TestResult: 480 | if (app.path / "doc").exists(): 481 | if ( 482 | os.system( 483 | r"grep -nr -q 'Any known limitations, constrains or stuff not working, such as\|Other infos that people should be' %s/doc/" 484 | % app.path 485 | ) 486 | == 0 487 | ): 488 | yield Warning( 489 | "In DISCLAIMER.md: 'Any known limitations [...] such as' and 'Other infos [...] such as' are supposed to be placeholder sentences meant to explain to packagers what is the expected content, but is not an appropriate wording for end users :/" 490 | ) 491 | if ( 492 | os.system( 493 | r"grep -nr -q 'This is a dummy\|Ceci est une fausse' %s/doc/" 494 | % app.path 495 | ) 496 | == 0 497 | ): 498 | yield Warning( 499 | "The doc/ folder seems to still contain some dummy, placeholder messages in the .md markdown files. If those files are not useful in the context of your app, simply remove them." 500 | ) 501 | 502 | @test() 503 | def custom_python_version(app) -> TestResult: 504 | 505 | cmd = f"grep -q -IhEr '^[^#]*install_python' '{app.path}/scripts/'" 506 | if os.system(cmd) == 0: 507 | yield Warning( 508 | "It looks like this app installs a custom version of Python which is heavily discouraged, both because it takes a shitload amount of time to compile Python locally, and because it is likely to create complication later once the system gets upgraded to newer Debian versions..." 509 | ) 510 | 511 | @test() 512 | def change_url_script(app) -> TestResult: 513 | 514 | keyandargs = copy.deepcopy(app.manifest["install"]) 515 | for key, infos in keyandargs.items(): 516 | infos["name"] = key 517 | args = keyandargs.values() 518 | 519 | has_domain_arg = any(a["name"] == "domain" for a in args) 520 | 521 | if has_domain_arg and not not_empty(app.path / "scripts" / "change_url"): 522 | yield Info( 523 | "Consider adding a change_url script to support changing where the app can be reached" 524 | ) 525 | 526 | @test() 527 | def config_panel(app) -> TestResult: 528 | 529 | if not_empty(app.path / "config_panel.json"): 530 | yield Error( 531 | "JSON config panels are not supported anymore, should be replaced by a toml version" 532 | ) 533 | 534 | if not_empty(app.path / "config_panel.toml.example"): 535 | yield Warning( 536 | "Please do not commit config_panel.toml.example ... This is just a 'documentation' for the config panel syntax meant to be kept in example_ynh" 537 | ) 538 | 539 | if not not_empty(app.path / "config_panel.toml") and not_empty( 540 | app.path / "scripts" / "config" 541 | ): 542 | yield Warning( 543 | "The script 'config' exists but there is no config_panel.toml ... Please remove the 'config' script if this is just the example from example_ynh, or add a proper config_panel.toml if the point is really to have a config panel" 544 | ) 545 | 546 | if not_empty(app.path / "config_panel.toml"): 547 | check_old_panel = os.system( 548 | "grep -q 'version = \"0.1\"' '%s'" % (app.path / "config_panel.toml") 549 | ) 550 | if check_old_panel == 0: 551 | yield Error( 552 | "Config panels version 0.1 are not supported anymore, should be adapted for version 1.0" 553 | ) 554 | elif (app.path / "scripts" / "config").exists() and os.system( 555 | "grep -q 'YNH_CONFIG_\\|yunohost app action' '%s'" 556 | % (app.path / "scripts" / "config") 557 | ) == 0: 558 | yield Error( 559 | "The config panel is set to version 1.x, but the config script is apparently still using some old code from 0.1 such as '$YNH_CONFIG_STUFF' or 'yunohost app action'" 560 | ) 561 | 562 | yield from validate_schema( 563 | "config_panel", 564 | json.loads(config_panel_v1_schema()), 565 | tomllib.load((app.path / "config_panel.toml").open("rb")), 566 | ) 567 | 568 | @test() 569 | def badges_in_readme(app) -> TestResult: 570 | 571 | id_ = app.manifest["id"] 572 | 573 | readme = app.path / "README.md" 574 | if not not_empty(readme): 575 | return 576 | 577 | content = readme.read_text() 578 | 579 | if "This README was automatically generated" not in content or ( 580 | ( 581 | f"dash.yunohost.org/integration/{id_}.svg" not in content 582 | and f"https://apps.yunohost.org/badge/integration/{id_}" not in content 583 | ) 584 | and ( 585 | f"https://apps.yunohost.org/app/{id_}" not in content 586 | and f"https://raw.githubusercontent.com/YunoHost/apps/main/logos/{id_}.png" 587 | not in content 588 | ) 589 | ): 590 | yield Warning( 591 | "It looks like the README was not generated automatically by https://github.com/YunoHost/apps/tree/main/tools/README-generator. " 592 | "Note that nowadays you are not suppose to edit README.md, the yunohost bot will usually automatically update it if your app is hosted in the YunoHost-Apps org ... or you can also generate it by running the README-generator yourself." 593 | ) 594 | 595 | @test() 596 | def remaining_replacebyyourapp(self) -> TestResult: 597 | if os.system("grep -I -qr 'REPLACEBYYOURAPP' %s 2>/dev/null" % self.path) == 0: 598 | yield Error("You should replace all occurences of REPLACEBYYOURAPP.") 599 | 600 | @test() 601 | def supervisor_usage(self) -> TestResult: 602 | if ( 603 | os.system(r"grep -I -qr '^\s*supervisorctl' %s 2>/dev/null" % self.path) 604 | == 0 605 | ): 606 | yield Warning( 607 | "Please don't rely on supervisor to run services. YunoHost is about standardization and the standard is to use systemd units..." 608 | ) 609 | 610 | @test() 611 | def bad_encoding(self) -> TestResult: 612 | 613 | cmd = ( 614 | "file --mime-encoding $(find %s/ -type f) | grep 'iso-8859-1\\|unknown-8bit' || true" 615 | % self.path 616 | ) 617 | bad_encoding_files = ( 618 | subprocess.check_output(cmd, shell=True).decode("utf-8").strip().split("\n") 619 | ) 620 | for file_ in bad_encoding_files: 621 | if not file_: 622 | continue 623 | file_ = file_.split()[0] 624 | yield Error( 625 | "%s appears to be encoded as latin-1 / iso-8859-1. Please convert it to utf-8 to avoid funky issues. Something like 'iconv -f iso-8859-1 -t utf-8 SOURCE > DEST' should do the trick." 626 | % file_ 627 | ) 628 | 629 | ####################################### 630 | # _ _ _ # 631 | # | | | | | | # 632 | # | |__| | ___| |_ __ ___ _ __ ___ # 633 | # | __ |/ _ \ | '_ \ / _ \ '__/ __| # 634 | # | | | | __/ | |_) | __/ | \__ \ # 635 | # |_| |_|\___|_| .__/ \___|_| |___/ # 636 | # | | # 637 | # |_| # 638 | ####################################### 639 | 640 | @test() 641 | def helpers_now_official(app) -> TestResult: 642 | 643 | cmd = "grep -IhEro 'ynh_\\w+ *\\( *\\)' '%s/scripts' | tr -d '() '" % app.path 644 | custom_helpers = ( 645 | subprocess.check_output(cmd, shell=True).decode("utf-8").strip().split("\n") 646 | ) 647 | custom_helpers = [c.split("__")[0] for c in custom_helpers] 648 | 649 | for custom_helper in custom_helpers: 650 | if custom_helper in official_helpers.keys(): 651 | yield Info( 652 | "%s is now an official helper since version '%s'" 653 | % (custom_helper, official_helpers[custom_helper] or "?") 654 | ) 655 | 656 | @test() 657 | def git_clone_usage(app) -> TestResult: 658 | cmd = ( 659 | f"grep -I 'git clone' '{app.path}'/scripts/install '{app.path}'/scripts/_common.sh 2>/dev/null" 660 | r" | grep -qv 'xxenv\|rbenv\|oracledb'" 661 | ) 662 | if os.system(cmd) == 0: 663 | yield Warning( 664 | "Using 'git clone' is not recommended ... most forge do provide the ability to download a proper archive of the code for a specific commit. Please use the 'sources' resource in the manifest.toml in combination with ynh_setup_source." 665 | ) 666 | 667 | @test() 668 | def helpers_version_requirement(app) -> TestResult: 669 | 670 | cmd = "grep -IhEro 'ynh_\\w+ *\\( *\\)' '%s/scripts' | tr -d '() '" % app.path 671 | custom_helpers = ( 672 | subprocess.check_output(cmd, shell=True).decode("utf-8").strip().split("\n") 673 | ) 674 | custom_helpers = [c.split("__")[0] for c in custom_helpers] 675 | 676 | yunohost_version_req = ( 677 | app.manifest.get("integration", {}).get("yunohost", "").strip(">= ") 678 | ) 679 | 680 | cmd = "grep -IhEro 'ynh_\\w+' '%s/scripts'" % app.path 681 | helpers_used = ( 682 | subprocess.check_output(cmd, shell=True).decode("utf-8").strip().split("\n") 683 | ) 684 | helpers_used = sorted(set(helpers_used)) 685 | 686 | manifest_req = [int(i) for i in yunohost_version_req.split(".")] + [0, 0, 0] 687 | 688 | def validate_version_requirement(helper_req: str) -> bool: 689 | if helper_req == "": 690 | return True 691 | helper_req_list = [int(i) for i in helper_req.split(".")] 692 | for i in range(0, len(helper_req_list)): 693 | if helper_req_list[i] == manifest_req[i]: 694 | continue 695 | return helper_req_list[i] <= manifest_req[i] 696 | return True 697 | 698 | for helper in [h for h in helpers_used if h in official_helpers.keys()]: 699 | if helper in custom_helpers: 700 | continue 701 | helper_req = official_helpers[helper] 702 | if not validate_version_requirement(helper_req): 703 | major_diff = manifest_req[0] > int(helper_req[0]) 704 | message = ( 705 | "Using official helper %s implies requiring at least version %s, but manifest only requires %s" 706 | % (helper, helper_req, yunohost_version_req) 707 | ) 708 | yield Error(message) if major_diff else Warning(message) 709 | 710 | @test() 711 | def helpers_deprecated_in_v2(app) -> TestResult: 712 | 713 | cmd = f"grep -IhEro 'ynh_\\w+' '{app.path}/scripts/install' '{app.path}/scripts/remove' '{app.path}/scripts/upgrade' '{app.path}/scripts/backup' '{app.path}/scripts/restore' || true" 714 | helpers_used = ( 715 | subprocess.check_output(cmd, shell=True).decode("utf-8").strip().split("\n") 716 | ) 717 | helpers_used = sorted(set(helpers_used)) 718 | 719 | deprecated_helpers_in_v2_ = {k: v for k, v in deprecated_helpers_in_v2} 720 | 721 | for helper in [ 722 | h for h in helpers_used if h in deprecated_helpers_in_v2_.keys() 723 | ]: 724 | yield Warning( 725 | f"Using helper {helper} is deprecated when using packaging v2 ... It is replaced by: {deprecated_helpers_in_v2_[helper]}" 726 | ) 727 | 728 | @test() 729 | def helper_consistency_apt_deps(app) -> TestResult: 730 | """ 731 | Check if ynh_install_app_dependencies is present in install/upgrade/restore 732 | so dependencies are up to date after restoration or upgrade 733 | """ 734 | 735 | install_script = app.scripts["install"] 736 | if install_script.contains("ynh_install_app_dependencies"): 737 | for name in ["upgrade", "restore"]: 738 | if app.scripts[name].exists and not app.scripts[name].contains( 739 | "ynh_install_app_dependencies" 740 | ): 741 | yield Warning( 742 | "ynh_install_app_dependencies should also be in %s script" 743 | % name 744 | ) 745 | 746 | cmd = ( 747 | 'grep -IhEr "install_extra_app_dependencies" %s/scripts | grep -v "key" | grep -q "http://"' 748 | % app.path 749 | ) 750 | if os.system(cmd) == 0: 751 | yield Warning( 752 | "When installing dependencies from extra repository, please include a `--key` argument (yes, even if it's official debian repos such as backports - because systems like Raspbian do not ship Debian's key by default!" 753 | ) 754 | 755 | @test() 756 | def helper_consistency_service_add(app) -> TestResult: 757 | 758 | occurences = { 759 | "install": ( 760 | app.scripts["install"].occurences("yunohost service add") 761 | if app.scripts["install"].exists 762 | else [] 763 | ), 764 | "upgrade": ( 765 | app.scripts["upgrade"].occurences("yunohost service add") 766 | if app.scripts["upgrade"].exists 767 | else [] 768 | ), 769 | "restore": ( 770 | app.scripts["restore"].occurences("yunohost service add") 771 | if app.scripts["restore"].exists 772 | else [] 773 | ), 774 | } 775 | 776 | occurences = { 777 | k: [sub_v.replace('"$app"', "$app") for sub_v in v] 778 | for k, v in occurences.items() 779 | } 780 | 781 | all_occurences = ( 782 | occurences["install"] + occurences["upgrade"] + occurences["restore"] 783 | ) 784 | found_inconsistency = False 785 | found_legacy_logtype_option = False 786 | for cmd in all_occurences: 787 | if any( 788 | cmd not in occurences_list for occurences_list in occurences.values() 789 | ): 790 | found_inconsistency = True 791 | if "--log_type systemd" in cmd: 792 | found_legacy_logtype_option = True 793 | 794 | if found_inconsistency: 795 | details = [ 796 | ( 797 | " %s : " % script 798 | + "".join( 799 | "\n " + cmd 800 | for cmd in occurences[script] or ["...None?..."] 801 | ) 802 | ) 803 | for script in occurences.keys() 804 | ] 805 | details_str = "\n".join(details) 806 | yield Warning( 807 | "Some inconsistencies were found in the 'yunohost service add' commands between install, upgrade and restore:\n%s" 808 | % details_str 809 | ) 810 | 811 | if found_legacy_logtype_option: 812 | yield Warning( 813 | "Using option '--log_type systemd' with 'yunohost service add' is not relevant anymore" 814 | ) 815 | 816 | if occurences["install"] and not app.scripts["remove"].contains( 817 | "yunohost service remove" 818 | ): 819 | yield Error( 820 | "You used 'yunohost service add' in the install script, " 821 | "but not 'yunohost service remove' in the remove script." 822 | ) 823 | 824 | @test() 825 | def references_to_superold_stuff(app) -> TestResult: 826 | if any( 827 | script.contains("jessie") 828 | for script in app.scripts.values() 829 | if script.exists 830 | ): 831 | yield Error( 832 | "The app still contains references to jessie, which could probably be cleaned up..." 833 | ) 834 | if any( 835 | script.contains("/etc/php5") or script.contains("php5-fpm") 836 | for script in app.scripts.values() 837 | if script.exists 838 | ): 839 | yield Error( 840 | "This app still has references to php5 (from the jessie era!!) which tends to indicate that it's not up to date with recent packaging practices." 841 | ) 842 | if any( 843 | script.contains("/etc/php/7.0") or script.contains("php7.0-fpm") 844 | for script in app.scripts.values() 845 | if script.exists 846 | ): 847 | yield Error( 848 | "This app still has references to php7.0 (from the stretch era!!) which tends to indicate that it's not up to date with recent packaging practices." 849 | ) 850 | if any( 851 | script.contains("/etc/php/7.3") or script.contains("php7.3-fpm") 852 | for script in app.scripts.values() 853 | if script.exists 854 | ): 855 | yield Error( 856 | "This app still has references to php7.3 (from the buster era!!) which tends to indicate that it's not up to date with recent packaging practices." 857 | ) 858 | 859 | @test() 860 | def conf_json_persistent_tweaking(self) -> TestResult: 861 | if ( 862 | os.system( 863 | "grep -nr '/etc/ssowat/conf.json.persistent' %s | grep -vq '^%s/doc' 2>/dev/null" 864 | % (self.path, self.path) 865 | ) 866 | == 0 867 | ): 868 | yield Error("Don't do black magic with /etc/ssowat/conf.json.persistent!") 869 | 870 | @test() 871 | def app_data_in_unofficial_dir(self) -> TestResult: 872 | 873 | allowed_locations = [ 874 | "/home/yunohost.app", 875 | "/home/yunohost.conf", 876 | "/home/yunohost.backup", 877 | "/home/yunohost.multimedia", 878 | ] 879 | cmd = ( 880 | "grep -IhEro '/home/yunohost[^/ ]*/|/home/\\$app' %s/scripts || true" 881 | % self.path 882 | ) 883 | home_locations = ( 884 | subprocess.check_output(cmd, shell=True).decode("utf-8").strip().split("\n") 885 | ) 886 | 887 | forbidden_locations = set( 888 | [ 889 | location 890 | for location in home_locations 891 | if location and location.rstrip("/") not in allowed_locations 892 | ] 893 | ) 894 | 895 | if forbidden_locations: 896 | yield Warning( 897 | "The app seems to be storing data in the 'forbidden' locations %s. The recommended pratice is rather to store data in /home/yunohost.app/$app or /home/yunohost.multimedia (depending on the use case)" 898 | % ", ".join(forbidden_locations) 899 | ) 900 | -------------------------------------------------------------------------------- /tests/test_catalog.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import json 4 | import os 5 | import subprocess 6 | from pathlib import Path 7 | import sys 8 | import time 9 | import tomllib 10 | from datetime import datetime 11 | from types import ModuleType 12 | from typing import Any, Generator 13 | 14 | from lib.lib_package_linter import ( 15 | Critical, 16 | Error, 17 | Info, 18 | Success, 19 | TestResult, 20 | TestSuite, 21 | Warning, 22 | test, 23 | urlopen, 24 | ) 25 | from lib.print import _print 26 | 27 | PACKAGE_LINTER_DIR = Path(__file__).resolve().parent.parent 28 | APPS_CACHE = PACKAGE_LINTER_DIR / ".apps" 29 | 30 | ######################################## 31 | # _____ _ _ # 32 | # / __ \ | | | | # 33 | # | / \/ __ _| |_ __ _| | ___ __ _ # 34 | # | | / _` | __/ _` | |/ _ \ / _` | # 35 | # | \__/\ (_| | || (_| | | (_) | (_| | # 36 | # \____/\__,_|\__\__,_|_|\___/ \__, | # 37 | # __/ | # 38 | # |___/ # 39 | # # 40 | ######################################## 41 | 42 | 43 | class AppCatalog(TestSuite): 44 | def __init__(self, app_id: str) -> None: 45 | self.app_id = app_id 46 | self.test_suite_name = "Catalog infos" 47 | 48 | self._fetch_app_repo() 49 | 50 | try: 51 | self.app_list = tomllib.load((APPS_CACHE / "apps.toml").open("rb")) 52 | except Exception: 53 | _print("Failed to read apps.toml :/") 54 | sys.exit(-1) 55 | 56 | self.catalog_infos = self.app_list.get(app_id, {}) 57 | 58 | def _fetch_app_repo(self) -> None: 59 | flagfile = PACKAGE_LINTER_DIR / ".apps_git_clone_cache" 60 | if ( 61 | APPS_CACHE.exists() 62 | and flagfile.exists() 63 | and time.time() - flagfile.stat().st_mtime < 3600 64 | ): 65 | return 66 | 67 | if not APPS_CACHE.exists(): 68 | subprocess.check_call( 69 | [ 70 | "git", 71 | "clone", 72 | "https://github.com/YunoHost/apps", 73 | APPS_CACHE, 74 | "--quiet", 75 | ] 76 | ) 77 | else: 78 | subprocess.check_call(["git", "-C", APPS_CACHE, "fetch", "--quiet"]) 79 | subprocess.check_call( 80 | ["git", "-C", APPS_CACHE, "reset", "origin/main", "--hard", "--quiet"] 81 | ) 82 | 83 | flagfile.touch() 84 | 85 | @test() 86 | def is_in_catalog(self) -> TestResult: 87 | if not self.catalog_infos: 88 | yield Critical("This app is not in YunoHost's application catalog") 89 | 90 | @test() 91 | def revision_is_HEAD(self) -> TestResult: 92 | if self.catalog_infos and self.catalog_infos.get("revision", "HEAD") != "HEAD": 93 | yield Error( 94 | "You should make sure that the revision used in YunoHost's apps catalog is HEAD..." 95 | ) 96 | 97 | @test() 98 | def state_is_working(self) -> TestResult: 99 | if ( 100 | self.catalog_infos 101 | and self.catalog_infos.get("state", "working") != "working" 102 | ): 103 | yield Error( 104 | "The application is not flagged as working in YunoHost's apps catalog" 105 | ) 106 | 107 | @test() 108 | def has_category(self) -> TestResult: 109 | if self.catalog_infos and not self.catalog_infos.get("category"): 110 | yield Warning( 111 | "The application has no associated category in YunoHost's apps catalog" 112 | ) 113 | 114 | @test() 115 | def is_in_github_org(self) -> TestResult: 116 | repo_org = "https://github.com/YunoHost-Apps/%s_ynh" % (self.app_id) 117 | repo_brique = "https://github.com/labriqueinternet/%s_ynh" % (self.app_id) 118 | 119 | if self.catalog_infos: 120 | repo_url = self.catalog_infos["url"] 121 | 122 | if repo_url.lower() not in [repo_org.lower(), repo_brique.lower()]: 123 | if repo_url.lower().startswith("https://github.com/YunoHost-Apps/"): 124 | yield Warning( 125 | "The URL for this app in the catalog should be %s" % repo_org 126 | ) 127 | else: 128 | yield Info( 129 | "Consider adding your app to the YunoHost-Apps organization to allow the community to contribute more easily" 130 | ) 131 | 132 | else: 133 | 134 | def is_in_github_org() -> bool: 135 | return urlopen(repo_org)[0] != 404 136 | 137 | def is_in_brique_org() -> bool: 138 | return urlopen(repo_brique)[0] != 404 139 | 140 | if not is_in_github_org() and not is_in_brique_org(): 141 | yield Info( 142 | "Consider adding your app to the YunoHost-Apps organization to allow the community to contribute more easily" 143 | ) 144 | 145 | @test() 146 | def is_long_term_good_quality(self) -> TestResult: 147 | # 148 | # This analyzes the (git) history of apps.json in the past year and 149 | # compute a score according to the time when the app was 150 | # known + flagged working + level >= 5 151 | # 152 | 153 | def git(cmd: list[str]) -> str: 154 | return ( 155 | subprocess.check_output(["git", "-C", APPS_CACHE] + cmd) 156 | .decode("utf-8") 157 | .strip() 158 | ) 159 | 160 | def _time_points_until_today() -> Generator[datetime, None, None]: 161 | 162 | # Prior to April 4th, 2019, we still had official.json and community.json 163 | # Nowadays we only have apps.json 164 | year = 2019 165 | month = 6 166 | day = 1 167 | today = datetime.today() 168 | date = datetime(year, month, day) 169 | 170 | while date < today: 171 | yield date 172 | 173 | day += 14 174 | if day > 15: 175 | day = 1 176 | month += 1 177 | 178 | if month > 12: 179 | month = 1 180 | year += 1 181 | 182 | date = datetime(year, month, day) 183 | 184 | def get_history( 185 | N: int, 186 | ) -> Generator[tuple[datetime, dict[str, Any]], None, None]: 187 | 188 | for t in list(_time_points_until_today())[(-1 * N) :]: 189 | loader: ModuleType 190 | 191 | # Fetch apps.json content at this date 192 | commit = git( 193 | [ 194 | "rev-list", 195 | "-1", 196 | "--before='%s'" % t.strftime("%b %d %Y"), 197 | "main", 198 | ] 199 | ) 200 | if ( 201 | os.system( 202 | f"git -C {APPS_CACHE} cat-file -e {commit}:apps.json 2>/dev/null" 203 | ) 204 | == 0 205 | ): 206 | raw_catalog_at_this_date = git(["show", f"{commit}:apps.json"]) 207 | loader = json 208 | 209 | elif ( 210 | os.system(f"git -C {APPS_CACHE} cat-file -e {commit}:apps.toml") 211 | == 0 212 | ): 213 | raw_catalog_at_this_date = git(["show", f"{commit}:apps.toml"]) 214 | loader = tomllib 215 | else: 216 | raise Exception("No apps.json/toml at this point in history?") 217 | 218 | try: 219 | catalog_at_this_date = loader.loads(raw_catalog_at_this_date) 220 | # This can happen in stupid cases where there was a temporary syntax error in the json.. 221 | except json.decoder.JSONDecodeError: 222 | _print( 223 | "Failed to parse apps.json/toml history for at commit %s / %s ... ignoring " 224 | % (commit, t) 225 | ) 226 | continue 227 | yield (t, catalog_at_this_date.get(self.app_id)) 228 | 229 | # We'll check the history for last 12 months (*2 points per month) 230 | N = 12 * 2 231 | history = list(get_history(N)) 232 | 233 | # Must have been 234 | # known 235 | # + flagged as working 236 | # + level > 5 237 | # for the past 6 months 238 | def good_quality(infos: dict[str, Any]) -> bool: 239 | return ( 240 | bool(infos) 241 | and isinstance(infos, dict) 242 | and infos.get("state") == "working" 243 | and infos.get("level", -1) >= 5 244 | ) 245 | 246 | score = sum([good_quality(infos) for d, infos in history]) 247 | rel_score = int(100 * score / N) 248 | if rel_score > 80: 249 | yield Success("The app is long-term good quality in the catalog!") 250 | -------------------------------------------------------------------------------- /tests/test_configurations.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import json 4 | import os 5 | import re 6 | from pathlib import Path 7 | import subprocess 8 | import tomllib 9 | from typing import Any, Generator 10 | 11 | from lib.lib_package_linter import ( 12 | Error, 13 | Info, 14 | TestReport, 15 | TestResult, 16 | TestSuite, 17 | Warning, 18 | not_empty, 19 | test, 20 | tests_v1_schema, 21 | validate_schema, 22 | ) 23 | from lib.print import _print 24 | 25 | 26 | class Configurations(TestSuite): 27 | def __init__(self, app) -> None: 28 | 29 | self.app = app 30 | self.test_suite_name = "Configuration files" 31 | 32 | ############################ 33 | # _____ __ # 34 | # / ____| / _| # 35 | # | | ___ _ __ | |_ # 36 | # | | / _ \| '_ \| _| # 37 | # | |___| (_) | | | | | # 38 | # \_____\___/|_| |_|_| # 39 | # # 40 | ############################ 41 | 42 | @test() 43 | def tests_toml(self) -> TestResult: 44 | tests_toml_file = self.app.path / "tests.toml" 45 | if not not_empty(tests_toml_file): 46 | yield Error( 47 | "The 'check_process' file that interfaces with the app CI has now been replaced with 'tests.toml' format and is now mandatory for apps v2." 48 | ) 49 | else: 50 | yield from validate_schema( 51 | "tests.toml", 52 | json.loads(tests_v1_schema()), 53 | tomllib.load(tests_toml_file.open("rb")), 54 | ) 55 | 56 | @test() 57 | def encourage_extra_php_conf(self) -> TestResult: 58 | php_conf = self.app.path / "conf" / "php-fpm.conf" 59 | if not_empty(php_conf): 60 | yield Info( 61 | "For the php configuration, consider getting rid of php-fpm.conf " 62 | "and using the --usage and --footprint option of ynh_add_fpm_config. " 63 | "This will use an auto-generated php conf file." 64 | "Additionally you can provide a conf/extra_php-fpm.conf for custom PHP settings " 65 | "that will automatically be appended to the autogenerated conf. " 66 | " (Feel free to discuss this on the chat with other people, the whole thing " 67 | "with --usage/--footprint is legitimately a bit unclear ;))" 68 | ) 69 | 70 | @test() 71 | def misc_source_management(self) -> TestResult: 72 | source_dir = self.app.path / "sources" 73 | 74 | if ( 75 | source_dir.exists() 76 | and len(list(elt for elt in source_dir.iterdir() if elt.is_file())) > 5 77 | ): 78 | yield Error( 79 | "Upstream app sources shouldn't be stored in this 'sources' folder of this git repository as a copy/paste\n" 80 | "During installation, the package should download sources from upstream via 'ynh_setup_source'.\n" 81 | "See the helper documentation. " 82 | "Original discussion happened here: " 83 | "https://github.com/YunoHost/issues/issues/201#issuecomment-391549262" 84 | ) 85 | 86 | @test() 87 | def systemd_config_specific_user(self) -> TestResult: 88 | conf_dir: Path = self.app.path / "conf" 89 | if not conf_dir.exists(): 90 | return 91 | 92 | for file in conf_dir.iterdir(): 93 | # Ignore subdirs or filename not containing nginx in the name 94 | if not file.name.endswith(".service"): 95 | continue 96 | 97 | # Some apps only provide an override conf file, which is different 98 | # from the full/base service config (c.f. ffsync) 99 | if "override" in file.name: 100 | continue 101 | 102 | try: 103 | content = file.read_text() 104 | except UnicodeDecodeError: 105 | yield Info("%s does not look like a text file." % file.name) 106 | continue 107 | except Exception as e: 108 | yield Warning("Can't open/read %s : %s" % (file.name, e)) 109 | continue 110 | 111 | if "[Unit]" not in content: 112 | continue 113 | 114 | Level: type[TestReport] 115 | if re.findall(r"^ *Type=oneshot", content, flags=re.MULTILINE): 116 | Level = Info 117 | else: 118 | Level = Warning 119 | 120 | matches = re.findall(r"^ *(User)=(\S+)", content, flags=re.MULTILINE) 121 | if not any(match[0] == "User" for match in matches): 122 | yield Level( 123 | "You should specify a 'User=' directive in the systemd config !" 124 | ) 125 | continue 126 | 127 | if any(match[1] in ["root", "www-data"] for match in matches): 128 | yield Level( 129 | "DO NOT run the app's systemd service as root or www-data! Use a dedicated system user for this app! If your app requires administrator priviledges, you should consider adding the user to the sudoers (and restrict the commands it can use!)" 130 | ) 131 | 132 | @test() 133 | def systemd_config_harden_security(self) -> TestResult: 134 | conf_dir: Path = self.app.path / "conf" 135 | if not conf_dir.exists(): 136 | return 137 | 138 | for file in conf_dir.iterdir(): 139 | # Ignore subdirs or filename not containing nginx in the name 140 | if not file.name.endswith(".service"): 141 | continue 142 | 143 | if ( 144 | os.system(f"grep -Eqi '^\s*Environment=.*(pass|secret|key)' '{file}'") 145 | == 0 146 | ): 147 | yield Error( 148 | "Systemd configurations are world-readable and should not contain cleartext password/secrets T_T" 149 | ) 150 | 151 | if ( 152 | os.system(f"grep -q '^\s*CapabilityBoundingSet=' '{file}'") != 0 153 | or os.system(f"grep -q '^\s*Protect.*=' '{file}'") != 0 154 | or os.system(f"grep -q '^\s*SystemCallFilter=' '{file}'") != 0 155 | or os.system(f"grep -q '^\s*PrivateTmp=' '{file}'") != 0 156 | ): 157 | yield Info( 158 | f"You are encouraged to harden the security of the systemd configuration {file.name}. You can have a look at https://github.com/YunoHost/example_ynh/blob/main/conf/systemd.service#L14-L46 for a baseline." 159 | ) 160 | 161 | @test() 162 | def php_config_specific_user(self) -> TestResult: 163 | conf_dir: Path = self.app.path / "conf" 164 | if not conf_dir.exists(): 165 | return 166 | 167 | for file in conf_dir.iterdir(): 168 | # Ignore subdirs or filename not containing nginx in the name 169 | if not file.name.startswith("php") or not file.name.endswith(".conf"): 170 | continue 171 | 172 | try: 173 | content = file.read_text() 174 | except UnicodeDecodeError: 175 | yield Info("%s does not look like a text file." % file.name) 176 | continue 177 | except Exception as e: 178 | yield Warning("Can't open/read %s : %s" % (file.name, e)) 179 | continue 180 | 181 | matches = re.findall( 182 | r"^ *(user|group) = (\S+)", content, flags=re.MULTILINE 183 | ) 184 | if not any(match[0] == "user" for match in matches): 185 | yield Error( 186 | "You should at least specify a 'user =' directive in your PHP conf file" 187 | ) 188 | continue 189 | 190 | if any( 191 | match[1] == "root" or match == ("user", "www-data") for match in matches 192 | ): 193 | yield Error( 194 | "DO NOT run the app PHP worker as root or www-data! Use a dedicated system user for this app!" 195 | ) 196 | 197 | @test() 198 | def nginx_http_host(self) -> TestResult: 199 | nginx_conf: Path = self.app.path / "conf" / "nginx.conf" 200 | if not nginx_conf.exists(): 201 | return 202 | 203 | content = nginx_conf.read_text() 204 | if "$http_host" in content: 205 | yield Info( 206 | "In nginx.conf : please don't use $http_host but $host instead. C.f. https://github.com/yandex/gixy/blob/master/docs/en/plugins/hostspoofing.md" 207 | ) 208 | 209 | @test() 210 | def nginx_https_redirect(self) -> TestResult: 211 | conf_dir: Path = self.app.path / "conf" 212 | if not conf_dir.exists(): 213 | return 214 | 215 | for file in conf_dir.iterdir(): 216 | # Ignore subdirs or filename not containing nginx in the name 217 | if not file.is_file() or "nginx" not in file.name: 218 | continue 219 | 220 | content = file.read_text() 221 | if "if ($scheme = http)" in content and "rewrite ^ https" in content: 222 | yield Error( 223 | "Since Yunohost 4.3, the http->https redirect is handled by the core, " 224 | "therefore having an if ($scheme = http) { rewrite ^ https://... } block " 225 | "in the nginx config file is deprecated. (This helps with supporting Yunohost-behind-reverse-proxy use case)" 226 | ) 227 | 228 | @test() 229 | def misc_nginx_add_header(self) -> TestResult: 230 | # 231 | # Analyze nginx conf 232 | # - Deprecated usage of 'add_header' in nginx conf 233 | # - Spot path traversal issue vulnerability 234 | # 235 | conf_dir: Path = self.app.path / "conf" 236 | if not conf_dir.exists(): 237 | return 238 | 239 | for file in conf_dir.iterdir(): 240 | # Ignore subdirs or filename not containing nginx in the name 241 | if not file.is_file() or "nginx" not in file.name: 242 | continue 243 | 244 | content = file.read_text() 245 | if "location" in content and "add_header" in content: 246 | yield Error( 247 | "Do not use 'add_header' in the NGINX conf. Use 'more_set_headers' instead. " 248 | "(See https://www.peterbe.com/plog/be-very-careful-with-your-add_header-in-nginx " 249 | "and https://github.com/openresty/headers-more-nginx-module#more_set_headers )" 250 | ) 251 | 252 | @test() 253 | def misc_nginx_more_set_headers(self) -> TestResult: 254 | conf_dir: Path = self.app.path / "conf" 255 | if not conf_dir.exists(): 256 | return 257 | 258 | for file in conf_dir.iterdir(): 259 | # Ignore subdirs or filename not containing nginx in the name 260 | if not file.is_file() or "nginx" not in file.name: 261 | continue 262 | 263 | content = file.read_text() 264 | 265 | if "location" in content and "more_set_headers" in content: 266 | lines = content.split("\n") 267 | more_set_headers_lines = [ 268 | zzz for zzz in lines if "more_set_headers" in zzz 269 | ] 270 | 271 | def right_syntax(line: str) -> re.Match[str] | None: 272 | return re.search( 273 | r"more_set_headers +[\"\'][\w-]+\s?: .*[\"\'];", line 274 | ) 275 | 276 | lines = [ 277 | line.strip() 278 | for line in more_set_headers_lines 279 | if not right_syntax(line) 280 | ] 281 | if lines: 282 | yield Error( 283 | "It looks like the syntax for the 'more_set_headers' " 284 | "instruction is incorrect in the NGINX conf (N.B. " 285 | ": it's different than the 'add_header' syntax!)... " 286 | "The syntax should look like: " 287 | 'more_set_headers "Header-Name: value"' 288 | f"\nOffending line(s) [{lines}]" 289 | ) 290 | 291 | @test() 292 | def misc_nginx_check_regex_in_location(self) -> TestResult: 293 | conf_dir: Path = self.app.path / "conf" 294 | if not conf_dir.exists(): 295 | return 296 | 297 | for file in conf_dir.iterdir(): 298 | # Ignore subdirs or filename not containing nginx in the name 299 | if not file.is_file() or "nginx" not in file.name: 300 | continue 301 | 302 | cmd = 'grep -q -IhEro "location ~ __PATH__" %s' % file 303 | 304 | if os.system(cmd) == 0: 305 | yield Warning( 306 | "When using regexp in the nginx location field (location ~ __PATH__), start the path with ^ (location ~ ^__PATH__)." 307 | ) 308 | 309 | @test() 310 | def misc_nginx_path_traversal(self) -> TestResult: 311 | conf_dir: Path = self.app.path / "conf" 312 | if not conf_dir.exists(): 313 | return 314 | 315 | for file in conf_dir.iterdir(): 316 | # Ignore subdirs or filename not containing nginx in the name 317 | if not file.is_file() or "nginx" not in file.name: 318 | continue 319 | 320 | # 321 | # Path traversal issues 322 | # 323 | def find_location_with_alias( 324 | locationblock: Any, 325 | ) -> Generator[tuple[str, str], None, None]: 326 | 327 | if locationblock[0][0] != "location": 328 | return 329 | 330 | location = locationblock[0][-1] 331 | for line in locationblock[1]: 332 | instruction = line[0] 333 | if instruction == "alias": 334 | yield (location, line) 335 | elif ( 336 | isinstance(instruction, list) 337 | and instruction 338 | and instruction[0] == "location" 339 | ): 340 | yield from find_location_with_alias(instruction) 341 | else: 342 | continue 343 | 344 | def find_path_traversal_issue( 345 | nginxconf: list[Any], 346 | ) -> Generator[str, None, None]: 347 | 348 | for block in nginxconf: 349 | for location, alias in find_location_with_alias(block): 350 | # Ignore locations which are regexes..? 351 | if location.startswith("^") and location.endswith("$"): 352 | continue 353 | alias_path = alias[-1] 354 | 355 | # Ugly hack to ignore cases where aliasing to a specific file (e.g. favicon.ico or foobar.html) 356 | if "." in alias_path[-5:]: 357 | continue 358 | 359 | # For path traversal issues to occur, both of those are needed: 360 | # - location /foo { (*without* a / after foo) 361 | # - alias /var/www/foo/ (*with* a / after foo) 362 | # 363 | # Note that we also consider a positive the case where 364 | # the alias folder (e.g. /var/www/foo/) does not ends 365 | # with / if __INSTALL_DIR__ ain't used... that probably 366 | # means that the app is not using the standard nginx 367 | # helper, and therefore it is likely to be replaced by 368 | # something ending with / ... 369 | if not location.strip("'").endswith("/") and ( 370 | alias_path.endswith("/") 371 | or "__INSTALL_DIR__" not in alias_path 372 | ): 373 | yield location 374 | 375 | do_path_traversal_check = False 376 | try: 377 | import pyparsing 378 | import six 379 | 380 | do_path_traversal_check = True 381 | except Exception: 382 | # If inside a venv, try to magically install pyparsing 383 | if "VIRTUAL_ENV" in os.environ: 384 | try: 385 | _print("(Trying to auto install pyparsing...)") 386 | subprocess.check_output( 387 | "pip3 install pyparsing six", shell=True 388 | ) 389 | import pyparsing 390 | 391 | _print("OK!") 392 | do_path_traversal_check = True 393 | except Exception as e: 394 | _print("Failed :[ : %s" % str(e)) 395 | 396 | if not do_path_traversal_check: 397 | _print( 398 | "N.B.: The package linter needs you to run 'pip3 install pyparsing six' if you want it to be able to check for path traversal issue in NGINX confs" 399 | ) 400 | 401 | if do_path_traversal_check: 402 | from lib.nginxparser import nginxparser 403 | 404 | try: 405 | nginxconf: list[Any] = nginxparser.load(file.open()) 406 | except Exception as e: 407 | _print(f"Could not parse NGINX conf...: {e}") 408 | nginxconf = [] 409 | 410 | for location in find_path_traversal_issue(nginxconf): 411 | yield Error( 412 | "The NGINX configuration (especially location %s) " 413 | "appears vulnerable to path traversal issues as explained in\n" 414 | " https://www.acunetix.com/vulnerabilities/web/path-traversal-via-misconfigured-nginx-alias/\n" 415 | " To fix it, look at the first lines of the NGINX conf of the example app : \n" 416 | " https://github.com/YunoHost/example_ynh/blob/main/conf/nginx.conf" 417 | % location 418 | ) 419 | 420 | @test() 421 | def bind_public_ip(self) -> TestResult: 422 | conf_dir: Path = self.app.path / "conf" 423 | if not conf_dir.exists(): 424 | return 425 | 426 | for file in conf_dir.rglob("*"): 427 | if not file.is_file(): 428 | continue 429 | 430 | try: 431 | content = file.read_text() 432 | except UnicodeDecodeError: 433 | yield Info("%s does not look like a text file." % file.name) 434 | continue 435 | except Exception as e: 436 | yield Warning("Can't open/read %s: %s" % (file, e)) 437 | continue 438 | 439 | for number, line in enumerate(content.split("\n"), 1): 440 | comment = ("#", "//", ";", "/*", "*") 441 | if ("0.0.0.0" in line or "::" in line) and not line.strip().startswith( 442 | comment 443 | ): 444 | for ip in re.split(r"[ \t,='\"(){}\[\]]", line): 445 | if ip == "::" or ip.startswith("0.0.0.0"): 446 | yield Info( 447 | f"{file.relative_to(self.app.path)}:{number}: " 448 | "Binding to '0.0.0.0' or '::' can result in a security issue " 449 | "as the reverse proxy and the SSO can be bypassed by knowing " 450 | "a public IP (typically an IPv6) and the app port. " 451 | "Please be sure that this behavior is intentional. " 452 | "Maybe use '127.0.0.1' or '::1' instead." 453 | ) 454 | -------------------------------------------------------------------------------- /tests/test_manifest.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import copy 4 | import json 5 | from pathlib import Path 6 | import re 7 | import sys 8 | import tomllib 9 | from typing import Any, Callable 10 | 11 | from lib.lib_package_linter import ( 12 | Critical, 13 | Error, 14 | Info, 15 | TestResult, 16 | TestSuite, 17 | Warning, 18 | c, 19 | manifest_v2_schema, 20 | spdx_licenses, 21 | test, 22 | validate_schema, 23 | ) 24 | 25 | # Only packaging v2 is supported on the linter now ... But someday™ according The Prophecy™, packaging v3 will be a thing 26 | app_packaging_format = 2 27 | 28 | 29 | # Defined in packaging module 30 | # See https://github.com/pypa/packaging/blob/20cd09e00917adbc4afeaa753be831a6bc2740f7/packaging/version.py#L225 31 | VERSION_PATTERN = r""" 32 | v? 33 | (?: 34 | (?:(?P[0-9]+)!)? # epoch 35 | (?P[0-9]+(?:\.[0-9]+)*) # release segment 36 | (?P
                                          # pre-release
 37 |             [-_\.]?
 38 |             (?P(a|b|c|rc|alpha|beta|pre|preview))
 39 |             [-_\.]?
 40 |             (?P[0-9]+)?
 41 |         )?
 42 |         (?P                                         # post release
 43 |             (?:-(?P[0-9]+))
 44 |             |
 45 |             (?:
 46 |                 [-_\.]?
 47 |                 (?Ppost|rev|r)
 48 |                 [-_\.]?
 49 |                 (?P[0-9]+)?
 50 |             )
 51 |         )?
 52 |         (?P                                          # dev release
 53 |             [-_\.]?
 54 |             (?Pdev)
 55 |             [-_\.]?
 56 |             (?P[0-9]+)?
 57 |         )?
 58 |     )
 59 |     (?:\+(?P[a-z0-9]+(?:[-_\.][a-z0-9]+)*))?       # local version
 60 | """
 61 | 
 62 | 
 63 | #############################################
 64 | #   __  __             _  __          _     #
 65 | #  |  \/  |           (_)/ _|        | |    #
 66 | #  | \  / | __ _ _ __  _| |_ ___  ___| |_   #
 67 | #  | |\/| |/ _` | '_ \| |  _/ _ \/ __| __|  #
 68 | #  | |  | | (_| | | | | | ||  __/\__ \ |_   #
 69 | #  |_|  |_|\__,_|_| |_|_|_| \___||___/\__|  #
 70 | #                                           #
 71 | #############################################
 72 | 
 73 | 
 74 | class Manifest(TestSuite):
 75 |     def __init__(self, path: Path) -> None:
 76 | 
 77 |         self.path = path
 78 |         self.test_suite_name = "manifest"
 79 | 
 80 |         manifest_path = path / "manifest.toml"
 81 | 
 82 |         # Taken from https://stackoverflow.com/a/49518779
 83 |         def check_for_duplicate_keys(
 84 |             ordered_pairs: list[tuple[str, Any]],
 85 |         ) -> dict[str, Any]:
 86 |             dict_out = {}
 87 |             for key, val in ordered_pairs:
 88 |                 if key in dict_out:
 89 |                     raise Exception("Duplicated key '%s' in %s" % (key, ordered_pairs))
 90 |                 else:
 91 |                     dict_out[key] = val
 92 |             return dict_out
 93 | 
 94 |         self.raw_manifest = manifest_path.read_text()
 95 |         try:
 96 |             self.manifest = tomllib.loads(self.raw_manifest)
 97 |         except Exception as e:
 98 |             print(
 99 |                 f"{c.FAIL}✘ Looks like there's a syntax issue in your manifest?\n ---> {e}"
100 |             )
101 |             sys.exit(1)
102 | 
103 |     @test()
104 |     def mandatory_fields(self) -> TestResult:
105 | 
106 |         fields = [
107 |             "packaging_format",
108 |             "id",
109 |             "name",
110 |             "description",
111 |             "version",
112 |             "maintainers",
113 |             "upstream",
114 |             "integration",
115 |             "install",
116 |             "resources",
117 |         ]
118 | 
119 |         missing_fields = [
120 |             field for field in fields if field not in self.manifest.keys()
121 |         ]
122 | 
123 |         if missing_fields:
124 |             yield Critical(
125 |                 "The following mandatory fields are missing: %s" % missing_fields
126 |             )
127 | 
128 |         if "license" not in self.manifest.get("upstream", ""):
129 |             yield Error("The license key in the upstream section is missing")
130 | 
131 |     @test()
132 |     def maintainer_sensible_values(self) -> TestResult:
133 |         if "maintainers" in self.manifest.keys():
134 |             for value in self.manifest["maintainers"]:
135 |                 if not value.strip():
136 |                     yield Error("Please don't put empty string as a maintainer x_x")
137 |                 elif "," in value:
138 |                     yield Error(
139 |                         "Please don't use comma in maintainers value, this is supposed to be a list such as ['foo', bar'], not ['foo, bar'] x_x"
140 |                     )
141 | 
142 |     @test()
143 |     def upstream_fields(self) -> TestResult:
144 |         if "upstream" not in self.manifest.keys():
145 |             yield Warning(
146 |                 """READMEs are to be automatically generated using https://github.com/YunoHost/apps_tools/tree/main/readme_generator.
147 |         - You are encouraged to add an 'upstream' section in the manifest, filled with the website, demo, repo, license of the upstream app, as shown here: https://github.com/YunoHost/example_ynh/blob/7b72b7334964b504e8c901637c73ce908204d38b/manifest.json#L11-L18 . (Not all infos are mandatory, you can remove irrelevant entries)"""
148 |             )
149 | 
150 |     @test()
151 |     def upstream_placeholders(self) -> TestResult:
152 |         if "upstream" in self.manifest.keys():
153 |             if "yunohost.org" in self.manifest["upstream"].get("admindoc", ""):
154 |                 yield Error(
155 |                     "The field 'admindoc' should point to the **official** admin doc, not the YunoHost documentation. If there's no official admin doc, simply remove the admindoc key entirely."
156 |                 )
157 |             if "github.com" in self.manifest["upstream"].get("website", ""):
158 |                 yield Warning(
159 |                     "The field 'website' is not meant to point to a code repository ... this is to be handled by the 'code' key ... If the app has no proper website, just remove the 'website' key entirely"
160 |                 )
161 |             if "yunohost.org" in self.manifest["upstream"].get("userdoc", ""):
162 |                 yield Warning(
163 |                     "The field 'userdoc' should point to the **official** user doc, not the YunoHost documentation. (The default auto-generated README already includes a link to the yunohost doc page for this app). If there's no official user doc, simply remove the userdoc key entirely."
164 |                 )
165 |             if "example.com" in self.manifest["upstream"].get(
166 |                 "demo", ""
167 |             ) or "example.com" in self.manifest["upstream"].get("website", ""):
168 |                 yield Error(
169 |                     "It seems like the upstream section still contains placeholder values such as 'example.com' ..."
170 |                 )
171 |             code = self.manifest["upstream"].get("code")
172 |             if code and (
173 |                 code == self.manifest["upstream"].get("userdoc")
174 |                 or code == self.manifest["upstream"].get("admindoc")
175 |             ):
176 |                 yield Warning(
177 |                     "userdoc or admindoc: A code repository is not a documentation x_x"
178 |                 )
179 | 
180 |     @test()
181 |     def FIXMEs(self) -> TestResult:
182 |         if "FIXME" in self.raw_manifest:
183 |             yield Warning("There are still some FIXMEs remaining in the manifest")
184 | 
185 |     @test()
186 |     def yunohost_version_requirement_superold(app) -> TestResult:
187 | 
188 |         yunohost_version_req = (
189 |             app.manifest.get("integration", {}).get("yunohost", "").strip(">= ")
190 |         )
191 |         if yunohost_version_req.startswith("4."):
192 |             yield Critical(
193 |                 "Your app only requires yunohost >= 4.x, which tends to indicate that it may not be up to date with recommended packaging practices and helpers."
194 |             )
195 |         elif yunohost_version_req.startswith("11.0"):
196 |             yield Error(
197 |                 "Your app only requires yunohost >= 11.0, which tends to indicate that it may not be up to date with recommended packaging practices and helpers."
198 |             )
199 | 
200 |     @test()
201 |     def basic_fields_format(self) -> TestResult:
202 | 
203 |         if self.manifest.get("packaging_format") != app_packaging_format:
204 |             yield Error(f"packaging_format should be {app_packaging_format}")
205 |         if not re.match("^[a-z0-9]((_|-)?[a-z0-9])+$", self.manifest.get("id", "")):
206 |             yield Error("The app id is not a valid app id")
207 |         elif self.manifest.get("id", "").endswith("_ynh"):
208 |             yield Warning("The app id is not supposed to end with _ynh :| ...")
209 |         if len(self.manifest["name"]) > 22:
210 |             yield Error("The app name is too long")
211 | 
212 |         keys: dict[str, tuple[Callable[..., bool | re.Match[str] | None], str]] = {
213 |             "yunohost": (
214 |                 lambda v: isinstance(v, str) and re.fullmatch(r"^>=\s*[\d\.]+\d$", v),
215 |                 "Expected something like '>= 4.5.6'",
216 |             ),
217 |             "architectures": (
218 |                 lambda v: v == "all"
219 |                 or (
220 |                     isinstance(v, list)
221 |                     and all(subv in ["i386", "amd64", "armhf", "arm64"] for subv in v)
222 |                 ),
223 |                 "'all' or a list of values in ['i386', 'amd64', 'armhf', 'arm64']",
224 |             ),
225 |             "multi_instance": (
226 |                 lambda v: isinstance(v, bool),
227 |                 "Expected a boolean (true or false, no quotes!)",
228 |             ),
229 |             "ldap": (
230 |                 lambda v: isinstance(v, bool) or v == "not_relevant",
231 |                 "Expected a boolean (true or false, no quotes!) or 'not_relevant'",
232 |             ),
233 |             "sso": (
234 |                 lambda v: isinstance(v, bool) or v == "not_relevant",
235 |                 "Expected a boolean (true or false, no quotes!) or 'not_relevant'",
236 |             ),
237 |             "disk": (lambda v: isinstance(v, str), "Expected a string"),
238 |             "ram": (
239 |                 lambda v: isinstance(v, dict)
240 |                 and isinstance(v.get("build"), str)
241 |                 and isinstance(v.get("runtime"), str),
242 |                 "Expected to find ram.build and ram.runtime with string values",
243 |             ),
244 |         }
245 | 
246 |         for key, validator in keys.items():
247 |             if key not in self.manifest.get("integration", {}):
248 |                 yield Error(f"Missing key in the integration section: {key}")
249 |                 continue
250 |             value = self.manifest["integration"][key]
251 |             if not validator[0](value):
252 |                 yield Error(
253 |                     f"Error found with key {key} in the 'integration' section: {validator[1]}, got: {value}"
254 |                 )
255 | 
256 |         if not self.manifest.get("upstream", {}).get("license"):
257 |             yield Error("Missing 'license' key in the upstream section")
258 | 
259 |     @test()
260 |     def license(self) -> TestResult:
261 | 
262 |         # Turns out there may be multiple licenses... (c.f. Seafile)
263 |         licenses = self.manifest.get("upstream", {}).get("license", "").split(",")
264 | 
265 |         for license in licenses:
266 | 
267 |             license = license.strip()
268 | 
269 |             if "nonfree" in license.replace("-", ""):
270 |                 yield Warning(
271 |                     "'non-free' apps cannot be integrated in YunoHost's app catalog."
272 |                 )
273 |                 return
274 | 
275 |             code_license = '' + license + ""
276 | 
277 |             if code_license not in spdx_licenses():
278 |                 yield Warning(
279 |                     "The license id '%s' is not registered in https://spdx.org/licenses/."
280 |                     % license
281 |                 )
282 |                 return
283 | 
284 |     @test()
285 |     def description(self) -> TestResult:
286 | 
287 |         descr = self.manifest.get("description", "")
288 |         id = self.manifest["id"].lower()
289 |         name = self.manifest["name"].lower()
290 | 
291 |         if isinstance(descr, dict):
292 |             descr = descr.get("en", "")
293 | 
294 |         if len(descr) < 5 or len(descr) > 150:
295 |             yield Warning(
296 |                 "The description of your app is either missing, too short or too long... Please describe in *consise* terms what the app is/does."
297 |             )
298 | 
299 |         if "for yunohost" in descr.lower():
300 |             yield Error(
301 |                 "The 'description' should explain what the app actually does. "
302 |                 "No need to say that it is 'for YunoHost' - this is a YunoHost app "
303 |                 "so of course we know it is for YunoHost ;-)."
304 |             )
305 |         if descr.lower().startswith(id) or descr.lower().startswith(name):
306 |             yield Warning(
307 |                 "Try to avoid starting the description by '$app is' "
308 |                 "... explain what the app is/does directly!"
309 |             )
310 | 
311 |     @test()
312 |     def version_format(self) -> TestResult:
313 |         if not re.match(
314 |             r"^" + VERSION_PATTERN + r"~ynh[0-9]+$",
315 |             self.manifest.get("version", ""),
316 |             re.VERBOSE,
317 |         ):
318 |             yield Error(
319 |                 "The 'version' field should match the format ~ynh. "
320 |                 "For example: 4.3-2~ynh3. It is composed of the upstream version number (in the "
321 |                 "example, 4.3-2) and an incremental number for each change in the package without "
322 |                 "upstream change (in the example, 3). This incremental number can be reset to 1 "
323 |                 "each time the upstream version changes."
324 |             )
325 | 
326 |     @test()
327 |     def custom_install_dir(self) -> TestResult:
328 |         custom_install_dir = (
329 |             self.manifest.get("resources", {}).get("install_dir", {}).get("dir")
330 |         )
331 |         if not custom_install_dir:
332 |             return
333 | 
334 |         if custom_install_dir.startswith("/opt/yunohost"):
335 |             yield Warning(
336 |                 "Installing apps in /opt/yunohost is deprecated ... YunoHost is about standardization, and the standard is to install in /var/www/__APP__ (yes, even if not a webapp, because whatever). Please stick to the default value. the resource system should automatically move the install dir if needed so you don't really need to think about backward compatibility."
337 |             )
338 | 
339 |     @test()
340 |     def install_args(self) -> TestResult:
341 | 
342 |         recognized_types = (
343 |             "string",
344 |             "text",
345 |             "select",
346 |             "tags",
347 |             "email",
348 |             "url",
349 |             "date",
350 |             "time",
351 |             "color",
352 |             "password",
353 |             "path",
354 |             "boolean",
355 |             "domain",
356 |             "user",
357 |             "group",
358 |             "number",
359 |             "range",
360 |             "alert",
361 |             "markdown",
362 |             "file",
363 |             "app",
364 |         )
365 | 
366 |         keyandargs = copy.deepcopy(self.manifest["install"])
367 |         for key, infos in keyandargs.items():
368 |             infos["name"] = key
369 |         args = keyandargs.values()
370 | 
371 |         for argument in args:
372 |             if not isinstance(argument.get("optional", False), bool):
373 |                 yield Warning(
374 |                     "The key 'optional' value for setting %s should be a boolean (true or false)"
375 |                     % argument["name"]
376 |                 )
377 |             if "type" not in argument.keys():
378 |                 yield Warning(
379 |                     "You should specify the type of the argument '%s'. "
380 |                     "You can use: %s." % (argument["name"], ", ".join(recognized_types))
381 |                 )
382 |             elif argument["type"] not in recognized_types:
383 |                 yield Warning(
384 |                     "The type '%s' for argument '%s' is not recognized... "
385 |                     "it probably doesn't behave as you expect? Choose among those instead: %s"
386 |                     % (argument["type"], argument["name"], ", ".join(recognized_types))
387 |                 )
388 |             elif argument["type"] == "boolean" and argument.get(
389 |                 "default", True
390 |             ) not in [True, False]:
391 |                 yield Warning(
392 |                     "Default value for boolean-type arguments should be a boolean... (in particular, make sure it's not a string!)"
393 |                 )
394 |             elif argument["type"] in ["domain", "user", "password"]:
395 |                 if argument.get("default"):
396 |                     yield Info(
397 |                         "Default value for argument %s is superfluous, will be ignored"
398 |                         % argument["name"]
399 |                     )
400 |                 if argument.get("example"):
401 |                     yield Info(
402 |                         "Example value for argument %s is superfluous, will be ignored"
403 |                         % argument["name"]
404 |                     )
405 | 
406 |             if "choices" in argument.keys():
407 |                 choices = [c.lower() for c in argument["choices"]]
408 |                 if len(choices) == 2:
409 |                     if ("true" in choices and "false" in choices) or (
410 |                         "yes" in choices and "no" in choices
411 |                     ):
412 |                         yield Warning(
413 |                             "Argument %s : you might want to simply use a boolean-type argument. "
414 |                             "No need to specify the choices list yourself."
415 |                             % argument["name"]
416 |                         )
417 | 
418 |     @test()
419 |     def obsolete_or_missing_ask_strings(self) -> TestResult:
420 | 
421 |         ask_string_managed_by_the_core = [
422 |             ("domain", "domain"),
423 |             ("path", "path"),
424 |             ("admin", "user"),
425 |             ("is_public", "boolean"),
426 |             ("password", "password"),
427 |             ("init_main_permission", "group"),
428 |         ]
429 | 
430 |         keyandargs = copy.deepcopy(self.manifest["install"])
431 |         for key, infos in keyandargs.items():
432 |             infos["name"] = key
433 |         args = keyandargs.values()
434 | 
435 |         for argument in args:
436 | 
437 |             if (
438 |                 argument.get("ask")
439 |                 and (argument.get("name"), argument.get("type"))
440 |                 in ask_string_managed_by_the_core
441 |             ):
442 |                 yield Warning(
443 |                     "'ask' string for argument %s is superfluous / will be ignored. Since 4.1, the core handles the 'ask' string for some recurring arg name/type for consistency and easier i18n. See https://github.com/YunoHost/example_ynh/pull/142"
444 |                     % argument.get("name")
445 |                 )
446 | 
447 |             elif (
448 |                 not argument.get("ask")
449 |                 and (argument.get("name"), argument.get("type"))
450 |                 not in ask_string_managed_by_the_core
451 |             ):
452 |                 yield Warning(
453 |                     "You should add 'ask' strings for argument %s"
454 |                     % argument.get("name")
455 |                 )
456 | 
457 |     @test()
458 |     def old_php_version(self) -> TestResult:
459 | 
460 |         resources = self.manifest["resources"]
461 | 
462 |         if "apt" in list(resources.keys()):
463 |             packages = resources["apt"].get("packages", "")
464 |             packages = str(packages) if isinstance(packages, list) else packages
465 |             assert isinstance(packages, str)
466 |             if "php7.4-" in packages:
467 |                 yield Warning(
468 |                     "The app currently runs on php7.4 which is pretty old (unsupported by the PHP group since January 2023). Ideally, upgrade it to at least php8.2."
469 |                 )
470 |             elif "php8.0-" in packages:
471 |                 yield Warning(
472 |                     "The app currently runs on php8.0 which is pretty old (unsupported by the PHP group since January 2024). Ideally, upgrade it to at least php8.2."
473 |                 )
474 |             elif "php8.1-" in packages:
475 |                 yield Info(
476 |                     "The app currently runs on php8.1 which is deprecated since January 2024. Ideally, upgrade it to at least php8.2."
477 |                 )
478 | 
479 |     @test()
480 |     def resource_consistency(self) -> TestResult:
481 | 
482 |         resources = self.manifest["resources"]
483 | 
484 |         if "database" in list(resources.keys()):
485 |             if "apt" not in list(resources.keys()):
486 |                 yield Warning(
487 |                     "Having an 'apt' resource is mandatory when using a 'database' resource, to also install postgresql/mysql if needed"
488 |                 )
489 |             else:
490 |                 if list(resources.keys()).index("database") < list(
491 |                     resources.keys()
492 |                 ).index("apt"):
493 |                     yield Warning(
494 |                         "The 'apt' resource should be placed before the 'database' resource, to install postgresql/mysql if needed *before* provisioning the database"
495 |                     )
496 | 
497 |                 dbtype = resources["database"]["type"]
498 | 
499 |                 apt_packages = resources["apt"].get("packages", [])
500 |                 if isinstance(apt_packages, str):
501 |                     apt_packages = [
502 |                         value.strip() for value in re.split(" |,", apt_packages)
503 |                     ]
504 | 
505 |                 if dbtype == "mysql" and "mariadb-server" not in apt_packages:
506 |                     yield Warning(
507 |                         "When using a mysql database, you should add mariadb-server in apt dependencies. Even though it's currently installed by default in YunoHost installations, it might not be in the future !"
508 |                     )
509 |                 if dbtype == "postgresql" and "postgresql" not in apt_packages:
510 |                     yield Warning(
511 |                         "When using a postgresql database, you should add postgresql in apt dependencies."
512 |                     )
513 | 
514 |         main_perm = self.manifest["resources"].get("permissions", {}).get("main", {})
515 |         if (
516 |             isinstance(main_perm.get("url"), str)
517 |             and "init_main_permission" not in self.manifest["install"]
518 |             and not main_perm.get("allowed")
519 |         ):
520 |             yield Warning(
521 |                 "You should add a 'init_main_permission' question, or define `allowed` for main permission to have the app ready to be accessed right after installation."
522 |             )
523 | 
524 |         @test()
525 |         def manifest_schema(self: "Manifest") -> TestResult:
526 |             yield from validate_schema(
527 |                 "manifest", json.loads(manifest_v2_schema()), self.manifest
528 |             )
529 | 


--------------------------------------------------------------------------------
/tests/test_scripts.py:
--------------------------------------------------------------------------------
  1 | #!/usr/bin/env python3
  2 | 
  3 | import os
  4 | from pathlib import Path
  5 | import re
  6 | import shlex
  7 | import statistics
  8 | import subprocess
  9 | from typing import Generator
 10 | 
 11 | from lib.lib_package_linter import (
 12 |     Critical,
 13 |     Error,
 14 |     Info,
 15 |     TestResult,
 16 |     TestSuite,
 17 |     Warning,
 18 |     not_empty,
 19 |     report_warning_not_reliable,
 20 |     test,
 21 | )
 22 | from lib.print import _print
 23 | 
 24 | 
 25 | ##################################
 26 | #   _____           _       _    #
 27 | #  / ____|         (_)     | |   #
 28 | # | (___   ___ _ __ _ _ __ | |_  #
 29 | #  \___ \ / __| '__| | '_ \| __| #
 30 | #  ____) | (__| |  | | |_) | |_  #
 31 | # |_____/ \___|_|  |_| .__/ \__| #
 32 | #                    | |         #
 33 | #                    |_|         #
 34 | ##################################
 35 | class Script(TestSuite):
 36 |     def __init__(self, app: Path, name: str, app_id: str) -> None:
 37 |         self.name = name
 38 |         self.app = app
 39 |         self.app_id = app_id
 40 |         self.path = app / "scripts" / name
 41 |         self.exists = not_empty(self.path)
 42 |         if not self.exists:
 43 |             return
 44 |         self.lines = list(self.read_file())
 45 |         self.test_suite_name = "scripts/" + self.name
 46 | 
 47 |     def read_file(self) -> Generator[list[str], None, None]:
 48 |         lines = self.path.open().readlines()
 49 | 
 50 |         # Remove trailing spaces, empty lines and comment lines
 51 |         lines = [line.strip() for line in lines]
 52 |         lines = [line for line in lines if line and not line.startswith("#")]
 53 | 
 54 |         # Merge lines when ending with \
 55 |         lines = "\n".join(lines).replace("\\\n", "").split("\n")
 56 | 
 57 |         some_parsing_failed = False
 58 | 
 59 |         for line in lines:
 60 | 
 61 |             try:
 62 |                 splitted_line = shlex.split(line, True)
 63 |                 yield splitted_line
 64 |             except Exception as e:
 65 | 
 66 |                 ignore_pattern = [
 67 |                     "/etc/cron",
 68 |                     "admin_panel=",
 69 |                     'echo "',
 70 |                     "__PRE_TAG",
 71 |                     "__URL_TAG",
 72 |                     "maintenance.$app.conf",
 73 |                     "mail_message=",
 74 |                     "maintenance.$app.html",
 75 |                     "> mail_to_send",
 76 |                 ]
 77 |                 if str(e) == "No closing quotation" and any(
 78 |                     pattern in line for pattern in ignore_pattern
 79 |                 ):
 80 |                     continue
 81 | 
 82 |                 if not some_parsing_failed:
 83 |                     _print(
 84 |                         "Some lines could not be parsed in script %s. (That's probably not really critical)"
 85 |                         % self.name
 86 |                     )
 87 |                     some_parsing_failed = True
 88 | 
 89 |                 report_warning_not_reliable("%s : %s" % (e, line))
 90 | 
 91 |     def occurences(self, command: str) -> list[str]:
 92 |         return [
 93 |             line for line in [" ".join(line) for line in self.lines] if command in line
 94 |         ]
 95 | 
 96 |     def contains(self, command: str) -> bool:
 97 |         """
 98 |         Iterate on lines to check if command is contained in line
 99 | 
100 |         For instance, "app setting" is contained in "yunohost app setting $app ..."
101 |         """
102 |         return any(command in line for line in [" ".join(line) for line in self.lines])
103 | 
104 |     def containsregex(self, regex: str) -> bool:
105 |         """
106 |         Iterate on lines to check if command is contained in line
107 | 
108 |         For instance, "app setting" is contained in "yunohost app setting $app ..."
109 |         """
110 |         return any(
111 |             re.search(regex, line) for line in [" ".join(line) for line in self.lines]
112 |         )
113 | 
114 |     @test()
115 |     def error_handling(self) -> TestResult:
116 | 
117 |         if (
118 |             self.contains("ynh_abort_if_errors")
119 |             or self.contains("set -eu")
120 |             or self.contains("set -u")
121 |         ):
122 |             yield Error(
123 |                 "ynh_abort_if_errors or set -eu is now handled by YunoHost core in packaging v2, you should not have to add it to your script !"
124 |             )
125 | 
126 |         return
127 | 
128 |         if self.name in ["backup", "remove", "_common.sh"]:
129 |             present = (
130 |                 self.contains("ynh_abort_if_errors")
131 |                 or self.contains("set -eu")
132 |                 or self.contains("set -u")
133 |             )
134 |         else:
135 |             present = self.contains("ynh_abort_if_errors")
136 | 
137 |         if self.name in ["remove", "_common.sh"]:
138 |             if present:
139 |                 yield Error(
140 |                     "Do not use 'set -eu' or 'ynh_abort_if_errors' in the remove or _common.sh scripts: "
141 |                     "If a single instruction fails, it will stop the script and is "
142 |                     "likely to leave the system in a broken state."
143 |                 )
144 |         elif not present:
145 |             yield Error(
146 |                 "You should add 'ynh_abort_if_errors' in this script, "
147 |                 "c.f. https://github.com/YunoHost/issues/issues/419"
148 |             )
149 | 
150 |     # Skip this in common.sh, sometimes custom not-yet-official helpers need this
151 |     @test(ignore=["_common.sh"])
152 |     def raw_apt_commands(self) -> TestResult:
153 | 
154 |         if (
155 |             self.contains("ynh_package_install")
156 |             or self.contains("apt install")
157 |             or self.contains("apt-get install")
158 |         ):
159 |             yield Error(
160 |                 "You should not use `ynh_package_install` or `apt-get install`, "
161 |                 "use `ynh_install_app_dependencies` instead"
162 |             )
163 | 
164 |         if (
165 |             self.contains("ynh_package_remove")
166 |             or self.contains("apt remove")
167 |             or self.contains("apt-get remove")
168 |         ):
169 |             yield Error(
170 |                 "You should not use `ynh_package_remove` or `apt-get remove`, "
171 |                 "use `ynh_remove_app_dependencies` instead"
172 |             )
173 | 
174 |     @test()
175 |     def obsolete_helpers(self) -> TestResult:
176 |         if self.contains("yunohost app setting"):
177 |             yield Critical(
178 |                 "Do not use 'yunohost app setting' directly. Please use 'ynh_app_setting_(set,get,delete)' instead."
179 |             )
180 |         if self.contains("ynh_detect_arch"):
181 |             yield Warning(
182 |                 "(Requires yunohost 4.3) Using ynh_detect_arch is deprecated, since Yunohost 4.3, an $YNH_ARCH variable is directly available in the global context. Its value directly corresponds to `dpkg --print-architecture` which returns a value among : amd64, i386, armhf, arm64 and armel (though armel is probably not used at all?)"
183 |             )
184 | 
185 |     @test(only=["install"])
186 |     def deprecated_YNH_APP_ARG(self) -> TestResult:
187 |         cmd = f"grep 'YNH_APP_ARG' '{self.path}' | grep -vq 'YNH_APP_ARG_PASSWORD'"
188 |         if os.system(cmd) == 0:
189 |             yield Warning(
190 |                 "Using the YNH_APP_ARG_ syntax is deprecated and will be removed in the future. (Except for password-type question which is a specific case). Questions are saved as settings and are directly available as bash variable $foobar (instead of $YNH_APP_ARG_FOOBAR)"
191 |             )
192 | 
193 |     @test(only=["install", "upgrade"])
194 |     def deprecated_replace_string(self) -> TestResult:
195 |         cmd1 = "grep -Ec 'ynh_replace_string' '%s' || true" % self.path
196 |         cmd2 = "grep -Ec 'ynh_replace_string.*__\\w+__' '%s' || true" % self.path
197 | 
198 |         count1 = int(subprocess.check_output(cmd1, shell=True).decode("utf-8").strip())
199 |         count2 = int(subprocess.check_output(cmd2, shell=True).decode("utf-8").strip())
200 | 
201 |         if count2 > 0 or count1 >= 5:
202 |             yield Info(
203 |                 "Please consider using 'ynh_add_config' to handle config files instead of gazillions of manual cp + 'ynh_replace_string' + chmod"
204 |             )
205 | 
206 |     @test()
207 |     def bad_ynh_exec_syntax(self) -> TestResult:
208 |         cmd = (
209 |             'grep -q -IhEro "ynh_exec_(err|warn|warn_less|quiet|fully_quiet) (\\"|\').*(\\"|\')$" %s'
210 |             % self.path
211 |         )
212 |         if os.system(cmd) == 0:
213 |             yield Warning(
214 |                 "(Requires Yunohost 4.3) When using ynh_exec_*, please don't wrap your command between quotes (typically DONT write ynh_exec_warn_less 'foo --bar --baz')"
215 |             )
216 | 
217 |     @test()
218 |     def ynh_setup_source_keep_with_absolute_path(self) -> TestResult:
219 |         cmd = 'grep -q -IhEro "ynh_setup_source.*keep.*install_dir" %s' % self.path
220 |         if os.system(cmd) == 0:
221 |             yield Info(
222 |                 "The --keep option of ynh_setup_source expects relative paths, not absolute path ... you do not need to prefix everything with '$install_dir' in the --keep arg ..."
223 |             )
224 | 
225 |     @test()
226 |     def ynh_npm_global(self) -> TestResult:
227 |         if self.containsregex(r"ynh_npm.*install.*global"):
228 |             yield Warning(
229 |                 "Please don't install stuff on global scope with npm install --global é_è"
230 |             )
231 | 
232 |     @test()
233 |     def ynh_add_fpm_config_deprecated_package_option(self) -> TestResult:
234 |         if self.containsregex(r"ynh_add_fpm_config .*package=.*"):
235 |             yield Error(
236 |                 "(Requires Yunohost 4.3) Option --package for ynh_add_fpm_config is deprecated : please use 'ynh_install_app_dependencies' with **all** your apt dependencies instead (no need to define a special 'extra_php_dependencies'). YunoHost will automatically install any phpX.Y-fpm / phpX.Y-common if needed."
237 |             )
238 | 
239 |     @test()
240 |     def set_is_public_setting(self) -> TestResult:
241 |         if self.containsregex(r"ynh_app_setting_set .*is_public.*"):
242 |             if self.name == "upgrade":
243 |                 yield Error(
244 |                     "permission system: it should not be needed to save is_public with ynh_app_setting_set ... this setting should only be used during installation to initialize the permission. The admin is likely to manually tweak the permission using YunoHost's interface later."
245 |                 )
246 |             else:
247 |                 yield Warning(
248 |                     "permission system: it should not be needed to save is_public with ynh_app_setting_set ... this setting should only be used during installation to initialize the permission. The admin is likely to manually tweak the permission using YunoHost's interface later."
249 |                 )
250 | 
251 |     @test(only=["_common.sh"])
252 |     def default_php_version_in_common(self) -> TestResult:
253 |         if self.contains("YNH_DEFAULT_PHP_VERSION"):
254 |             yield Warning(
255 |                 "Do not use YNH_DEFAULT_PHP_VERSION in _common.sh ... _common.sh is usually sourced *before* the helpers, which define the version of YNH_DEFAULT_PHP_VERSION (hence it gets replaced with empty string). Instead, please explicitly state the PHP version in the package, e.g. dependencies='php8.2-cli php8.2-imagemagick'"
256 |             )
257 | 
258 |     @test(ignore=["install", "_common.sh"])
259 |     def get_is_public_setting(self) -> TestResult:
260 |         if self.contains("is_public=") or self.contains("$is_public"):
261 |             yield Warning(
262 |                 "permission system: there should be no need to fetch or use $is_public ... is_public should only be used during installation to initialize the permission. The admin is likely to manually tweak the permission using YunoHost's interface later."
263 |             )
264 | 
265 |     @test(only=["upgrade"])
266 |     def temporarily_enable_visitors_during_upgrade(self) -> TestResult:
267 |         if self.containsregex(
268 |             "ynh_permission_update.*add.*visitors"
269 |         ) and self.containsregex("ynh_permission_update.*remove.*visitors"):
270 |             yield Warning(
271 |                 "permission system: since Yunohost 4.3, there should be no need to temporarily add 'visitors' to the main permission. ynh_local_curl will temporarily enable visitors access if needed"
272 |             )
273 | 
274 |     @test()
275 |     def set_legacy_permissions(self) -> TestResult:
276 |         if self.containsregex(
277 |             r"ynh_app_setting_set .*protected_uris"
278 |         ) or self.containsregex(r"ynh_app_setting_set .*skipped_uris"):
279 |             yield Error(
280 |                 "permission system: it looks like the app is still using super-legacy (un)protected/skipped_uris settings. This is now completely deprecated. Please check https://yunohost.org/packaging_apps_permissions for a documentation on how to migrate the app to the new permission system."
281 |             )
282 | 
283 |         elif self.containsregex(
284 |             r"ynh_app_setting_set .*protected_"
285 |         ) or self.containsregex(r"ynh_app_setting_set .*skipped_"):
286 |             yield Warning(
287 |                 "permission system: it looks like the app is still using the legacy permission system (unprotected/protected/skipped uris/regexes setting). Please check https://yunohost.org/packaging_apps_permissions for a documentation on how to migrate the app to the new permission system."
288 |             )
289 | 
290 |     @test()
291 |     def normalize_url_path(self) -> TestResult:
292 |         if self.contains("ynh_normalize_url_path"):
293 |             yield Warning(
294 |                 "You probably don't need to call 'ynh_normalize_url_path'... this is only relevant for upgrades from super-old versions (like 3 years ago or so...)"
295 |             )
296 | 
297 |     @test()
298 |     def safe_rm(self) -> TestResult:
299 |         if (
300 |             self.contains("rm -r")
301 |             or self.contains("rm -R")
302 |             or self.contains("rm -fr")
303 |             or self.contains("rm -fR")
304 |         ):
305 |             yield Error(
306 |                 "You should not be using 'rm -rf', please use 'ynh_secure_remove' instead"
307 |             )
308 | 
309 |     @test()
310 |     def FIXMEs(self) -> TestResult:
311 |         removeme = f"grep -q '#REMOVEME?' '{self.path}'"
312 |         fixme = f"grep -q '# FIXMEhelpers2.1' '{self.path}'"
313 | 
314 |         if os.system(removeme) == 0:
315 |             yield Warning("There are still some REMOVEME? flags to be taken care of")
316 |         if os.system(fixme) == 0:
317 |             yield Warning(
318 |                 "There are still some FIXMEhelpers2.1 flags to be taken care of"
319 |             )
320 | 
321 |     @test()
322 |     def nginx_restart(self) -> TestResult:
323 |         if self.contains("systemctl restart nginx") or self.contains(
324 |             "service nginx restart"
325 |         ):
326 |             yield Error(
327 |                 "Restarting NGINX is quite dangerous (especially for web installs) "
328 |                 "and should be avoided at all cost. Use 'reload' instead."
329 |             )
330 | 
331 |     @test()
332 |     def raw_systemctl_start(self) -> TestResult:
333 |         if self.containsregex(r"systemctl start \"?[^. ]+(\.service)?\"?\s"):
334 |             yield Warning(
335 |                 "Please do not use 'systemctl start' to start services. Instead, you should use 'ynh_systemd_action' which will display the service log in case it fails to start. You can also use '--line_match' to wait until some specific word appear in the log, signaling the service indeed fully started."
336 |             )
337 | 
338 |     @test()
339 |     def bad_line_match(self) -> TestResult:
340 | 
341 |         if self.containsregex(r"--line_match=Started$") or self.containsregex(
342 |             r"--line_match=Stopped$"
343 |         ):
344 |             yield Warning(
345 |                 'Using --line_match="Started" or "Stopped" in ynh_systemd_action is counter productive because it will match the systemd message and not the actual app message ... Please check the log of the service to find an actual, relevant message to match, or remove the --line_match option entirely'
346 |             )
347 | 
348 |     @test()
349 |     def quiet_systemctl_enable(self) -> TestResult:
350 | 
351 |         systemctl_enable = [
352 |             line
353 |             for line in [" ".join(line) for line in self.lines]
354 |             if re.search(r"^\s*systemctl.*(enable|disable)", line)
355 |         ]
356 | 
357 |         if any("-q" not in cmd for cmd in systemctl_enable):
358 |             message = "Please add --quiet to systemctl enable/disable commands to avoid unnecessary warnings when the script runs"
359 |             yield Warning(message)
360 | 
361 |     @test()
362 |     def quiet_wget(self) -> TestResult:
363 | 
364 |         wget_cmds = [
365 |             line
366 |             for line in [" ".join(line) for line in self.lines]
367 |             if re.search(r"^wget ", line)
368 |         ]
369 | 
370 |         if any(
371 |             " -q " not in cmd and "--quiet" not in cmd and "2>" not in cmd
372 |             for cmd in wget_cmds
373 |         ):
374 |             message = "Please redirect wget's stderr to stdout with 2>&1 to avoid unecessary warnings when the script runs (yes, wget is annoying and displays a warning even when things are going okay >_> ...)"
375 |             yield Warning(message)
376 | 
377 |     @test(only=["install"])
378 |     def argument_fetching(self) -> TestResult:
379 | 
380 |         if self.containsregex(r"^\w+\=\$\{?[0-9]"):
381 |             yield Critical(
382 |                 "Do not fetch arguments from manifest using 'variable=$N' (e.g."
383 |                 " domain=$1...) Instead, use 'name=$YNH_APP_ARG_NAME'"
384 |             )
385 | 
386 |     @test(only=["install"])
387 |     def sources_list_tweaking(self) -> TestResult:
388 |         common_sh = self.app / "scripts" / "_common.sh"
389 |         if self.contains("/etc/apt/sources.list") or (
390 |             common_sh.exists()
391 |             and "/etc/apt/sources.list" in common_sh.read_text()
392 |             and "ynh_add_repo" not in common_sh.read_text()
393 |         ):
394 |             yield Error(
395 |                 "Manually messing with apt's sources.lists is strongly discouraged "
396 |                 "and should be avoided. Please use 'ynh_install_extra_app_dependencies' if you "
397 |                 "need to install dependencies from a custom apt repo."
398 |             )
399 | 
400 |     @test()
401 |     def firewall_consistency(self) -> TestResult:
402 |         if self.contains("yunohost firewall allow") and not self.contains(
403 |             "--needs_exposed_ports"
404 |         ):
405 |             yield Info(
406 |                 "You used 'yunohost firewall allow' to expose a port on the outside but did not use 'yunohost service add' with '--needs_exposed_ports' ... If you are ABSOLUTELY SURE that the service needs to be exposed on THE OUTSIDE, then add '--needs_exposed_ports' to 'yunohost service add' with the relevant port number. Otherwise, opening the port leads to a significant security risk and you should keep the damn port closed !"
407 |             )
408 | 
409 |         if self.contains("Configuring firewall") and not self.contains(
410 |             "yunohost firewall allow"
411 |         ):
412 |             yield Warning(
413 |                 "Some message is talking about 'Configuring firewall' but there's no mention of 'yunohost firewall allow' ... If you're only finding an available port for *internal reverse proxy*, this has nothing to do with 'Configuring the firewall', so the message should be changed to avoid confusion... "
414 |             )
415 | 
416 |     @test()
417 |     def exit_ynhdie(self) -> TestResult:
418 | 
419 |         if self.contains(r"\bexit\b"):
420 |             yield Error(
421 |                 "'exit' command shouldn't be used. Please use 'ynh_die' instead."
422 |             )
423 | 
424 |     @test()
425 |     def old_regenconf(self) -> TestResult:
426 |         if self.contains("yunohost service regen-conf"):
427 |             yield Error(
428 |                 "'yunohost service regen-conf' has been replaced by 'yunohost tools regen-conf'."
429 |             )
430 | 
431 |     @test()
432 |     def ssowatconf_or_nginx_reload(self) -> TestResult:
433 |         # Dirty hack to check only the 10 last lines for ssowatconf
434 |         # (the "bad" practice being using this at the very end of the script, but some apps legitimately need this in the middle of the script)
435 |         oldlines = list(self.lines)
436 |         self.lines = self.lines[-10:]
437 |         if self.contains("yunohost app ssowatconf"):
438 |             yield Warning(
439 |                 "You probably don't need to run 'yunohost app ssowatconf' in the app self. It's supposed to be ran automatically after the script."
440 |             )
441 | 
442 |         if self.name not in ["change_url", "restore"]:
443 |             if self.contains("ynh_systemd_action --service_name=nginx --action=reload"):
444 |                 yield Warning(
445 |                     "You should not need to reload nginx at the end of the script ... it's already taken care of by ynh_add_nginx_config"
446 |                 )
447 | 
448 |         self.lines = oldlines
449 | 
450 |     @test()
451 |     def sed(self) -> TestResult:
452 |         if self.containsregex(
453 |             r"sed\s+(-i|--in-place)\s+(-r\s+)?s"
454 |         ) or self.containsregex(r"sed\s+s\S*\s+(-i|--in-place)"):
455 |             yield Info(
456 |                 "You should avoid using 'sed -i' for substitutions, please use 'ynh_replace_string' or 'ynh_add_config' instead"
457 |             )
458 | 
459 |     @test()
460 |     def sudo(self) -> TestResult:
461 |         if self.containsregex(
462 |             r"sudo \w"
463 |         ):  # \w is here to not match sudo -u, legit use because ynh_exec_as not official yet...
464 |             yield Warning(
465 |                 "You should not need to use 'sudo', the script is being run as root. "
466 |                 "(If you need to run a command using a specific user, use 'ynh_exec_as' (or 'sudo -u'))"
467 |             )
468 | 
469 |     @test()
470 |     def chownroot(self) -> TestResult:
471 | 
472 |         # Mywebapp has a legit use case for this >_>
473 |         if self.app_id == "my_webapp":
474 |             return
475 | 
476 |         if self.containsregex(
477 |             r"^\s*chown.* root:?[^$]* .*install_dir"
478 |         ) and not self.contains('chown root:root "$install_dir"'):
479 |             # (Mywebapp has a special case because of SFTP é_è)
480 |             yield Warning(
481 |                 "Using 'chown root $install_dir' is usually symptomatic of misconfigured and wide-open 'other' permissions ... Usually ynh_setup_source should now set sane default permissions on $install_dir (if the app requires Yunohost >= 4.2) ... Otherwise, consider using 'chown $app', 'chown nobody' or 'chmod' to limit access to $install_dir ..."
482 |             )
483 | 
484 |     @test()
485 |     def chmod777(self) -> TestResult:
486 |         if self.containsregex(r"chmod .*777") or self.containsregex(r"chmod .*o\+w"):
487 |             yield Warning(
488 |                 "DO NOT use chmod 777 or chmod o+w that gives write permission to every users on the system!!! If you have permission issues, just make sure that the owner and/or group owner is right..."
489 |             )
490 | 
491 |     @test()
492 |     def random(self) -> TestResult:
493 |         if self.contains("dd if=/dev/urandom") or self.contains("openssl rand"):
494 |             yield Error(
495 |                 "Instead of 'dd if=/dev/urandom' or 'openssl rand', you should use 'ynh_string_random'"
496 |             )
497 | 
498 |     @test(only=["install"])
499 |     def progression(self) -> TestResult:
500 |         if not self.contains("ynh_script_progression"):
501 |             yield Warning(
502 |                 "Please add a few messages for the user using 'ynh_script_progression' "
503 |                 "to explain what is going on (in friendly, not-too-technical terms) "
504 |                 "during the installation. (and ideally in scripts remove, upgrade and restore too)"
505 |             )
506 | 
507 |     @test(only=["backup"])
508 |     def progression_in_backup(self) -> TestResult:
509 |         if self.contains("ynh_script_progression"):
510 |             yield Warning(
511 |                 "We recommend to *not* use 'ynh_script_progression' in backup "
512 |                 "scripts because no actual work happens when running the script "
513 |                 ": YunoHost only fetches the list of things to backup (apart "
514 |                 "from the DB dumps which effectively happens during the script...). "
515 |                 "Consider using a simple message like this instead: 'ynh_print_info \"Declaring files to be backed up...\"'"
516 |             )
517 | 
518 |     @test()
519 |     def progression_time(self) -> TestResult:
520 | 
521 |         # Usage of ynh_script_progression with --time or --weight=1 all over the place...
522 |         if self.containsregex(r"ynh_script_progression.*--time"):
523 |             yield Info(
524 |                 "Using 'ynh_script_progression --time' should only be for calibrating the weight (c.f. '--weight'). It's not meant to be kept for production versions."
525 |             )
526 | 
527 |     @test(ignore=["_common.sh", "backup"])
528 |     def progression_meaningful_weights(self) -> TestResult:
529 |         def weight(line: list[str]) -> int:
530 |             match = re.search(
531 |                 r"ynh_script_progression.*--weight=([0-9]+)", " ".join(line)
532 |             )
533 |             if match:
534 |                 try:
535 |                     return int(match.groups()[0])
536 |                 except Exception:
537 |                     return -1
538 |             else:
539 |                 return 1
540 | 
541 |         script_progress = [
542 |             line for line in self.lines if "ynh_script_progression" in line
543 |         ]
544 |         weights = [weight(line) for line in script_progress]
545 | 
546 |         if not weights:
547 |             return
548 | 
549 |         if len(weights) > 3 and statistics.stdev(weights) > 50:
550 |             yield Warning(
551 |                 "To have a meaningful progress bar, try to keep the weights in the same range of values, for example [1,10], or [10,100]... otherwise, if you have super-huge weight differences, the progress bar rendering will be completely dominated by one or two steps... If these steps are really long, just try to indicated in the message that this will take a while."
552 |             )
553 | 
554 |     @test(only=["install", "_common.sh"])
555 |     def php_deps(self) -> TestResult:
556 |         if self.containsregex("dependencies.*php-"):
557 |             # (Stupid hack because some apps like roundcube depend on php-pear or php-php-gettext and there's no phpx.y-pear phpx.y-php-gettext>_> ...
558 |             if not self.contains("php-pear") or not self.contains("php-php-gettext"):
559 |                 yield Warning(
560 |                     "You should avoid having dependencies like 'php-foobar'. Instead, specify the exact version you want like 'php7.0-foobar'. Otherwise, the *wrong* version of the dependency may be installed if sury is also installed. Note that for Stretch/Buster/Bullseye/... transition, YunoHost will automatically patch your file so there's no need to care about that."
561 |                 )
562 | 
563 |     @test(only=["backup"])
564 |     def systemd_during_backup(self) -> TestResult:
565 |         if self.containsregex("^ynh_systemd_action"):
566 |             yield Warning(
567 |                 "Unless you really have a good reason to do so, starting/stopping services during backup has no benefit and leads to unecessary service interruptions when creating backups... As a 'reminder': apart from possibly database dumps (which usually do not require the service to be stopped) or other super-specific action, running the backup script is only a *declaration* of what needs to be backed up. The real copy and archive creation happens *after* the backup script is ran."
568 |             )
569 | 
570 |     @test()
571 |     def helpers_sourcing_after_official(self) -> TestResult:
572 |         helpers_after_official = subprocess.check_output(
573 |             "head -n 30 '%s' | grep -A 10 '^ *source */usr/share/yunohost/helpers' | grep '^ *source ' | tail -n +2"
574 |             % self.path,
575 |             shell=True,
576 |         ).decode("utf-8")
577 |         helpers_after_official = (
578 |             helpers_after_official.replace("source", "").replace(" ", "").strip()
579 |         )
580 |         if helpers_after_official:
581 |             helpers_after_official_list = helpers_after_official.split("\n")
582 |             yield Warning(
583 |                 "Please avoid sourcing additional helpers after the official helpers (in this case file %s)"
584 |                 % ", ".join(helpers_after_official_list)
585 |             )
586 | 
587 |     @test(only=["backup", "restore"])
588 |     def helpers_sourcing_backuprestore(self) -> TestResult:
589 |         if self.contains("source _common.sh") or self.contains("source ./_common.sh"):
590 |             yield Error(
591 |                 'In the context of backup and restore scripts, you should load _common.sh with "source ../settings/scripts/_common.sh"'
592 |             )
593 | 
594 |     @test(only=["_common.sh"])
595 |     def no_progress_in_common(self) -> TestResult:
596 |         if self.contains("ynh_script_progression"):
597 |             yield Warning(
598 |                 "You should not use `ynh_script_progression` in _common.sh because it will produce warnings when trying to install the application."
599 |             )
600 | 
601 |     @test(only=["remove"])
602 |     def no_log_remove(self) -> TestResult:
603 |         if self.containsregex(r"(ynh_secure_remove|ynh_safe_rm|rm).*(\/var\/log\/)"):
604 |             yield Warning(
605 |                 "Do not delete logs on app removal, else they will be erased if the app upgrade fails. This is handled by the core."
606 |             )
607 | 


--------------------------------------------------------------------------------
/tox.ini:
--------------------------------------------------------------------------------
1 | [flake8]
2 | extend-ignore = E501
3 | exclude = .git,__pycache__


--------------------------------------------------------------------------------