├── .github └── pull_request_template.md ├── FUNDABLES.md └── README.md /.github/pull_request_template.md: -------------------------------------------------------------------------------- 1 | 17 | 18 | **What is the current situation/context?** 19 | 20 | 21 | **What ought to be fixed, made, or implemented?** 22 | 23 | 24 | **What problems would this solve, and what new capabilities would it cause?** 25 | 26 | 27 | **What kinds of work are necessary to make this happen?** 28 | 29 | -------------------------------------------------------------------------------- /FUNDABLES.md: -------------------------------------------------------------------------------- 1 | # Packaging improvements that could be funded 2 | 3 | This page lists specific things that 4 | 5 | 1. the Python packaging community wants 6 | 2. are fairly well-scoped 7 | 3. would happen much faster if the [Packaging Working 8 | Group](https://wiki.python.org/psf/PackagingWG) got funding to achieve them 9 | (through [donations](http://donate.pypi.org/) or grants/directed gifts) 10 | 11 | Please contact the Packaging WG by emailing 12 | [`packaging-wg@python.org`](mailto:packaging-wg@python.org) to ask us to estimate how much one of 13 | these improvements would cost; we'll get back to you within a few business 14 | days. 15 | 16 | This is roughly prioritized by urgency and impact, but is not a roadmap. 17 | 18 | ## Foundational tool improvements 19 | 20 | ### Implement metadata PEPs 21 | 22 | The packaging ecosystem relies heavily on access to project metadata. This data 23 | is currently difficult to access, because traditionally it was calculated "on 24 | demand" and only available by downloading the full distribution file. [PEP 25 | 643](https://www.python.org/dev/peps/pep-0643/) specifies a mechanism that 26 | allows projects to expose static metadata from source distributions where 27 | available (critically, project name and version are required to be static), and 28 | [PEP 658](https://www.python.org/dev/peps/pep-0658/) allows package indexes to 29 | expose metadata without requiring a download of the full distribution file. 30 | 31 | These two PEPs are currently not implemented throughout the packaging ecosystem. 32 | In particular, Warehouse does not yet allow PEP 643 metadata to be uploaded, 33 | and it does not implement PEP 658 for either wheels or source distributions. And 34 | setuptools does not yet implement PEP 643 (while a full implementation involves 35 | complex backward compatibility questions, a simple implementation of static name 36 | and version metadata could be relatively straightforward). 37 | 38 | If these two PEPs were implemented as described above, so that they achieve a 39 | critical mass of being available for a significant proportion of package queries, 40 | tools consuming package metadata, such as pip and poetry, could successfully 41 | use that data to simplify and optimise package handling, and adhoc tools would 42 | be able to reliably access package metadata without needing to implement complex 43 | download and build processes. 44 | 45 | I am writing this proposal from the point of view of potential consumers of the 46 | relevant metadata. I have not reached out to the setuptools or warehouse projects 47 | to establish if they would be interested in having this work funded, but I believe 48 | that we need to find some way of speeding up adoption of these standards to allow 49 | the ecosystem to move forward. 50 | 51 | ### Better specifications, toolchain, and services for building distributions 52 | 53 | PyTorch, TensorFlow, and many other Python packages (especially 54 | science packages) suffer from cross-platform installability problems, 55 | which affect both users and developers. Packagers and users prefer using 56 | built distributions (usually in the wheel format); publishing built 57 | distributions increases convenience for end users because source code is 58 | pre-compiled, which significantly reduces install time (e.g., from 10+ 59 | minutes to several seconds). 60 | 61 | Supporting the multifarious Linux platforms is something we've been lagging on; 62 | we are [still finishing up the rollout of 63 | manylinux2010](https://discuss.python.org/t/where-we-are-on-manylinux2010-and-how-that-relates-to-manylinux2014/1987/) 64 | and [recently approved the new standard 65 | manylinux2014](https://www.python.org/dev/peps/pep-0599/). But even so, 66 | packagers will have to build their own wheels to release packages, which can be 67 | fiddly, brittle, and time-consuming. 68 | 69 | We'd like help to: 70 | 71 | - [Fully implement & maintain conda-press](https://conda-incubator.github.io/conda-press-docs/) 72 | Conda-press is a tool that takes conda packages and turns them into wheels, without 73 | recompiling. This makes it very fast to create a wheel out of an existing package. 74 | It usually works. However, there have been a variety of bug and maintainence issues 75 | that require more development (and perhaps a refactor) to address. 76 | - [Create a generic wheel-building 77 | service](https://github.com/pypa/packaging-problems/issues/25) to make 78 | releases faster and more robust 79 | 80 | We need funding for specification research and writing, backend and 81 | frontend development, testing, DevOps/infrastructure/platform services, 82 | user experience work, technical writing for end users, project 83 | management, and community outreach. 84 | 85 | #### Port auditwheel to Windows and Mac 86 | 87 | The packaging tool [auditwheel](https://github.com/pypa/auditwheel) 88 | "is a command line tool to facilitate the creation of Python wheel 89 | packages for Linux (containing pre-compiled binary extensions) that 90 | are compatible with a wide variety of Linux distributions" and key 91 | standards. It can inspect a wheel, checking whether it is 92 | standards-compliant. It can also repair a wheel. If a wheel depends on 93 | libraries that are not on the system, it can rewrite that wheel and inject 94 | libraries needed, parsing and rewriting ELF data. It can also repair the 95 | relevant `manylinux` tag(s) on a wheel. 96 | 97 | However, no such utility exists on Windows, and so package maintainers 98 | on Windows face trouble creating wheels and debugging their 99 | packages. And [the similar utility for Mac 100 | OS](https://pypi.org/project/delocate/) does not share auditwheel's 101 | code and user interface. 102 | 103 | Therefore, [developers would like to add Windows and Mac support to 104 | auditwheel.](https://discuss.python.org/t/need-auditwheel-like-utility-for-other-platforms/2028/4) 105 | Porting auditwheel to Windows would make it *much* easier to make 106 | Windows wheels, and porting it to macOS would reduce duplication on 107 | the packaging maintainers' side, and reduce the proliferation of 108 | quirky tools that individual package maintainers need to learn about. 109 | 110 | A simpler and more consistent cross-platform workflow will make it 111 | easier for package maintainers to use generic off-the-shelf 112 | automation. More maintainers will be able to leverage available 113 | automation (GitHub Actions, Travis CI, Azure pipelines, and 114 | potentially a future PyPI wheel-building service) to speed up releases 115 | and reduce grunt work. Also, this will especially be useful for 116 | scientific programmers, since they often create Python applications or 117 | libraries that include binaries written in other languages, and wheel 118 | distributions of those packages are prone to complication. 119 | 120 | We need funding for backend development, hardware, testing, continuous 121 | integration platform services, technical writing for end users, 122 | project management, and community outreach. 123 | 124 | ### Robust interoperability testing 125 | 126 | We need funding to ensure core packaging tools work well with each other; 127 | currently they aren't seamlessly interoperable. See [the integration-test 128 | project](https://github.com/pypa/integration-test). This will help us get 129 | faster at testing and rolling out bugfixes and features for **all** [Python 130 | packaging and distribution tools](https://packaging.python.org/key_projects): 131 | well-known projects like `pip`, `virtualenv`, and `wheel`, but also all the 132 | downstream projects that depend on them. 133 | 134 | ### Revamp PyPI API 135 | 136 | The [Python Package Index](https://pypi.org/), a key platform for Python developers, has a browser interface, but most people use PyPI by hitting its [API endpoints](https://warehouse.readthedocs.io/api-reference/#available-apis) with client applications such as `pip`. PyPI has [a minimal download API](https://warehouse.readthedocs.io/api-reference/) that does not implement [many features that users have requested](https://github.com/pypa/warehouse/labels/APIs%2Ffeeds). The lack of a full-featured download API in Warehouse (the PyPI codebase) blocks many improvements, including: 137 | 138 | - [Light-bandwidth metadata-only API 139 | calls](https://github.com/pypa/warehouse/issues/474) and [JSON 140 | standardization](https://github.com/pypa/pip/issues/7406#issuecomment-583891169) 141 | that would enable better downloads, installations, dependency resolution 142 | features, and troubleshooting for `pip` and other clients 143 | - [RSS feeds](https://github.com/pypa/warehouse/pull/7013) that 144 | other platforms could reuse to get PyPI updates in user tooling 145 | - [Security notification feeds](https://github.com/pypa/warehouse/issues/798) 146 | - [Caching for the bandersnatch mirroring 147 | client](https://github.com/pypa/warehouse/issues/284) 148 | 149 | We'd like to [architect and implement a new Warehouse download API](https://github.com/pypa/warehouse/issues/284) to support these features, and deprecate and decommission the old endpoints. This requires backend development work, technical writing, user experience research, and publicity and coordination work within Python's community. 150 | 151 | ### Make setuptools the reference implementation of the distutils API 152 | 153 | There is a part of the Python standard library 154 | [called](https://docs.python.org/3/library/distutils.html) `distutils`, and some 155 | users directly use it. [We want users to instead switch to the supported 156 | toolchain](https://github.com/pypa/packaging-problems/issues/127), which uses 157 | `setuptools`, and we want to move all the functionality from `distutils` into 158 | `setuptools`. This requires backend development work, technical writing, 159 | project management, and publicity work within Python's community. 160 | 161 | ### De-duplicate and reorganize setuptools documentation 162 | 163 | The [documentation for setuptools](https://setuptools.readthedocs.io/) 164 | has grown messily over time and is difficult to browse and 165 | navigate. Also, the [legacy documentation for 166 | distutils](https://docs.python.org/3/distutils/index.html) and the 167 | current setuptools docs heavily overlap in content. These references 168 | thus trip up even experienced developers who want to understand these 169 | fundamental utilities. 170 | 171 | We need funding for several weeks of technical writer work and developer review to: 172 | 173 | * de-duplicate the distutils and setuptools documentation, making the latter independent of the former 174 | * re-organize the setuptools documentation 175 | 176 | ### Add support for reproducible builds to setuptools 177 | 178 | [Reproducible builds](https://en.wikipedia.org/wiki/Reproducible_builds) allow 179 | developers to independently verify that a distributed software package was not 180 | tampered with. Since a considerable number of the Python packages use 181 | `setuptools`, adding support for reproducible builds to this build backend 182 | can help to improve security in Python ecosystem as a whole. 183 | Some preliminary works that can help structuring this activity are available (see 184 | [pypa/setuptools#2133](https://github.com/pypa/setuptools/issues/2133), 185 | [pypa/setuptools#1512](https://github.com/pypa/setuptools/pull/1512) and 186 | [pypa/wheel#362](https://github.com/pypa/wheel/issues/362)), however the 187 | effort was never concluded. Funding would be used finalize the development of 188 | this feature. 189 | 190 | Import tasks are: 191 | - Support `SOURCE_DATA_EPOCH` environment variable for both sdists and wheels. 192 | - Make both sdist and wheels independent of umask. 193 | - Ensure that C/C++-extensions compiled with setuptools are reproducible. 194 | - Document the process of verifying a sdist or wheel. 195 | 196 | This project might require coordination with other tools in the ecosystem 197 | (e.g. `wheel`). 198 | 199 | ### Audit and update package metadata 200 | 201 | If we [audit and update PyPI metadata for existing projects based on 202 | already-uploaded 203 | artifacts](https://github.com/pypa/warehouse/issues/474#issuecomment-370986838), 204 | we can publish information about what packages depend on each other and on 205 | certain environments, and ensure a high-quality API for many tools to reuse and 206 | build upon. The current PyPI upload API relies on the upload client extracting 207 | the metadata and supplying it with the first upload request, and that isn't a 208 | valid assumption for older upload clients. Currently, our constraint is a 209 | combination of developer time, compute resources, and privileged backend 210 | database access; funding would break this bottleneck. 211 | 212 | ### Improve user experience of packaging 213 | 214 | User experience research, and UX and development implementation work, would [make 215 | it easier for packagers to create configuration 216 | files](https://github.com/pypa/packaging-problems/issues/1). We aim to use the 217 | UX research work from [improvements in pip's user 218 | experience](https://wiki.python.org/psf/Fundable%20Packaging%20Improvements#Improve_pip_user_experience) 219 | and build on them to improve the larger experience of packaging for Python in 220 | general. 221 | 222 | ### Improve specificity of license classifiers 223 | 224 | Our packaging ecosystem relies on [a particular structured data format 225 | (classifiers)](https://pypi.org/classifiers/) to indicate a package's legal 226 | license. However, our current system [allows for ambiguity that makes some 227 | downstream data display incoherent or very difficult, and doesn't allow for some 228 | license specificity that downstream consumers 229 | need](https://github.com/pypa/warehouse/issues/2996) 230 | ([Libraries.io](https://libraries.io/) and similar projects). Fixing this is a 231 | fairly small project, involving Python development, public communications, 232 | project management, and potentially a few hours of legal counsel for review. 233 | 234 | ### Standardize and implement a lockfile format 235 | 236 | `pip` currently uses `requirements.txt` to specify dependencies; it can specify 237 | __versions__ of packages but not __hashes__. The [newer pipfile 238 | format](https://github.com/pypa/pipfile) can include hashes, which some users 239 | prefer. But `pip` [doesn't yet 240 | support](https://github.com/pypa/pip/issues/4732) `pipfile`, so many users are 241 | blocked from using hashes to better secure their Python runtimes. We have [made 242 | some progress toward standardizing an interoperable lockfile 243 | format](https://github.com/uranusjr/lock-file), but we need to finish [that 244 | design standardization and consensus-gathering 245 | work](https://discuss.python.org/t/structured-exchangeable-lock-file-format-requirements-txt-2-0/876/) 246 | and implement it in `pip`, `pipenv`, and related tools. Other attempts reached the PEP 247 | stage [^pep-650][^pep-665], but ultimately were rejected. We'd need Python 248 | engineering work and project management to develop and deploy this. 249 | 250 | [^pep-650]: https://peps.python.org/pep-0650/ 251 | [^pep-665]: https://peps.python.org/pep-0665/ 252 | 253 | ### Package preview feature for PyPI 254 | 255 | Right now, there are ways for package maintainers to test and share draft 256 | versions of their upcoming releases, but they cause friction and confusion. So 257 | we want to add [staged releases -- a temporary state that a release can be in, 258 | where PyPI __has__ it and can evaluate it, but hasn't __published__ it 259 | yet](https://github.com/pypa/warehouse/issues/726). 260 | 261 | This will: 262 | 263 | - let project owners/maintainers 264 | preview/[test](https://github.com/pypa/warehouse/issues/720) 265 | how their package metadata displays on the website, and review where 266 | their fresh releases are out of compliance with site and 267 | interoperability requirements (preventing the problem of 268 | [maintainers wanting to re-upload removed files](https://github.com/pypa/packaging-problems/issues/74)) 269 | - help cross-platform package maintainers 270 | [coordinate dozens of wheels built on multiple machines](https://github.com/pypa/warehouse/issues/4056) for simultaneous release 271 | - [Provide an interoperability check for toolchain developers, and a testing site for people learning packaging](https://github.com/pypa/warehouse/issues/2286) 272 | - [Simplify packagers' upload configuration files](https://github.com/takluyver/flit/issues/125) 273 | - reduce complexity that currently forces maintainers to use 274 | [confusing "dev" or prerelease version numbers](https://github.com/pypa/warehouse/issues/5707) 275 | - [Improve security of package uploads, by allowing maintainers to scope upload API tokens to the newly staged package](https://github.com/pypa/warehouse/issues/6378) 276 | - [Prevent package name conflicts](https://github.com/pypa/packaging-problems/issues/114) 277 | - [Streamline infrastructure maintenance and confusing documentation by letting us take down the separate test.pypi.org staging site](https://github.com/pypa/warehouse/issues/918) 278 | - Provide pre-release warnings to maintainers of packages that [fail metadata 279 | checks](https://github.com/pypa/packaging-problems/issues/264) (such as 280 | rejecting or warning for [packages without Python requirements 281 | metadata](https://github.com/pypa/warehouse/issues/3889), or [manylinux 282 | wheels that fail auditwheel 283 | checks](https://github.com/pypa/warehouse/issues/5420) -- as we increase 284 | the packaging ecology's strictness regarding metadata standards compliance, 285 | during the intermediate period where we're warning maintainers/owners about 286 | failing strictness checks but not yet blocking releases on those new 287 | stricter checks, the package preview feature will help us provide soft 288 | warnings. 289 | 290 | We'll need database support for understanding the release state ("is 291 | this published or not"), user experience and developer support, and 292 | testing, security, infrastructure, and project management support. 293 | 294 | ### Feature flag system on PyPI 295 | 296 | It's difficult to roll out new features gradually to PyPI's test site or 297 | to selected test users. A 298 | [feature flag system](https://github.com/pypa/warehouse/issues/5869) would help us do targeted outreach to particular groups of 299 | users, deploy more confidently, and roll back changes when needed. We'd 300 | need user experience, front and backend engineer, data analytics, and 301 | project management support to develop and deploy this. 302 | 303 | ### User support ticket system 304 | 305 | Python packagers who need help currently create 306 | [Sourceforge](https://sourceforge.net/p/pypi/support-requests/) 307 | and [GitHub](https://github.com/pypa/pypi-support/issues) tickets, 308 | email mailing lists, tweet at maintainers, and so on. A 309 | [unified user support ticket system](https://github.com/pypa/warehouse/issues/3231#issuecomment-405561741), integrated into Warehouse, would: 310 | 311 | - help managers, entrepreneurs, and academics 312 | [reserve specific package names](https://github.com/pypa/warehouse/issues/2082) 313 | - [support username changes](https://github.com/pypa/warehouse/issues/1190) 314 | - give users [a reporting system to quickly flag malware and spam](https://github.com/pypa/warehouse/issues/2982) 315 | - provide a [transfer system for abandoned/unmaintained projects](https://github.com/pypa/warehouse/issues/1506) 316 | - reduce work for PyPI's core developers who currently have to sift 317 | through user support issues to find bug reports and feature requests 318 | - enable PyPI admins to better delegate support and moderation work to 319 | volunteers 320 | 321 | We need funding for backend and frontend development, testing and 322 | security checks, DevOps/infrastructure/platform services (including 323 | API/email integration), user experience work, technical writing for end 324 | users, project management, and community outreach. 325 | 326 | ### Architecture to support alternative authentication methods in packaging tools 327 | 328 | Python packaging tools that interact with package indexes, such as pip ([pypa/pip#4475](https://github.com/pypa/pip/issues/4475)) and twine ([pypa/twine#362](https://github.com/pypa/twine/issues/362)), currently only have simple authentication support to secure private sources, such as [basic access authentication](https://en.wikipedia.org/wiki/Basic_access_authentication). Open source tool maintainers acknowledge that, when using third-party indices, sometimes organisational policies require stronger authentication methods, such as single sign-on. We believe it’s beneficial to develop a pluggable Python library that can be depended by the packaging tools to provide additional authentication methods. But we lack both the use case and domain knowledge in the area. We are looking for funding and expertise support from organisations. 329 | 330 | We are interested in developing a shared interface and implementation for various alternative authentication methods. Support can be developed for both tools (and maybe more), so organisations can choose to install them to be able to use e.g. Kerberos to secure their private package indexes. The work involved would include development, research, project management, and technical writing work towards the following tasks: 331 | 332 | * Survey various authentication methods, and how they can be implemented as a pluggable library. 333 | * Develop an interface that tools (e.g. pip) can implement to detect authentication method support, and call into the library that provides it. 334 | * Develop and maintain libraries that implement the various auth methods for users to install when support is needed. 335 | 336 | ### Release wheel 1.0 and provide a public API 337 | 338 | The wheel project, the official binary distribution format for Python, 339 | is [approaching version 340 | 1.0](https://mail.python.org/archives/list/distutils-sig@python.org/thread/X2NNYA25EBAQYTPKRSJASPIBY6YT5VKB/). The 341 | [milestone](https://github.com/pypa/wheel/milestone/1) includes 342 | [Provide a public API: "Since there is clearly a need for wheel to 343 | function as a library as well, a public API should be defined and 344 | documented."](https://github.com/pypa/wheel/issues/262) The stability 345 | implied by the 1.x version number, and the public API, will improve 346 | other tools' ability to call and reuse `wheel`, thus reducing 347 | duplication and improving other toolmakers' ability to move faster and 348 | maintain their codebases more easily. We would like support for design 349 | and implementation, community coordination, and technical writing. 350 | 351 | ## Security improvements and prerequisites 352 | 353 | ### Improved support for Multi-Factor Authentication in Warehouse (PyPI) 354 | 355 | To mitigate account takeover attacks, where attackers upload malicious code in existing popular packages, we need to continue improving our support for MFA and use of API tokens instead of password-based auth. This work would involve research, development, and technical writing to finalise and implement existing proposals including: 356 | 357 | - [Require API Tokens for upload if MFA is enabled](https://github.com/pypa/warehouse/issues/7265) (upload currently bypasses MFA) 358 | - [Improve 2FA Account Recovery request fulfillment process](https://github.com/pypa/pypi-support/issues/796) 359 | - [other issues tagged `tokens`](https://github.com/pypa/warehouse/labels/tokens) 360 | 361 | Once these technical prerequisites are satisfied, we would be able to [revisit discussion of MFA *policy*](https://discuss.python.org/t/require-mfa-on-pypi/12077) - including encouraging MFA requirements for popular packages, or even mandating it for all users as some other package registries are considering. This subsequent work would involve additional development, project management, and community engagement as we determine and implement authentication policy, including ongoing user support to handle a much greater frequency of account recovery requests. 362 | 363 | ### System to label projects on PyPI with administrative statuses/attributes 364 | 365 | To scale up our anti-abuse moderation and help package maintainers with 366 | security response, we need to be able to, for instance, mark a release 367 | as deprecated or a project as unsupported. This means we need a generic 368 | system to add, edit, and remove administrative attributes ("flags" or 369 | "statuses") to individual projects and releases. We need support to do 370 | the architectural design to implement this. (See 371 | [notes from this meeting](https://wiki.python.org/psf/PackagingWG/2019-03-22-Warehouse).) 372 | 373 | ### Security notifications for vulnerable packages 374 | 375 | To keep PyPI's users secure, we want to give them 376 | [an opt-in communication channel to hear about security vulnerabilities for the packages they use](https://github.com/pypa/warehouse/issues/798). Implementing this would also give us 377 | architectural support to __warn or prevent__ `pip` users who try to 378 | install a PyPI package that's been found to be broken or malware. We 379 | need funding for user experience work, development, testing, 380 | infrastructure, potentially platform services (e.g., SMS), and community 381 | outreach. 382 | 383 | ### Check for insecure maintainer email addresses 384 | 385 | [Recent research on weaknesses in the `npm` supply chain](https://arxiv.org/abs/2112.10165) 386 | identified 2,818 maintainer email addresses at expired domains, affecting 8,494 packages. 387 | Such weaknesses can be mitigated by multi-factor authentication, but generally make 388 | targeted account hijacking trivially easy. 389 | 390 | Because registration expiry dates are public via whois records, PyPI could warn maintainers 391 | with email addresses at soon-to-expire domains. As a further enhancement, PyPI could block 392 | password resets via any email address at a domain which has expired and been re-registered 393 | *after* the 30 day renewal grace period. This gives defenders a natural time-based advantage 394 | over attackers. 395 | 396 | Funding would be used for backend development, security engineering, project management, 397 | system administration, outreach to package maintainers, and ongoing operational costs. 398 | 399 | 400 | ### Productionize Malware Detection 401 | Since anyone can upload a package to PyPI, malicious users might upload malware, which would then harm users. To mitigate this risk, PSF [previously obtained funding](https://github.com/python/request-for/blob/master/2019-Q4-PyPI/RFP.md) to add some malware detection in [Warehouse](https://github.com/pypa/warehouse) in late 2019, but the goals for [the relevant milestone](https://github.com/python/request-for/blob/master/2019-Q4-PyPI/RFP.md#milestone-2---systems-for-automated-detection-of-malicious-uploads) were more ambitious than funding allowed for. The malware detection system is currently in limbo: an interesting prototype with limited practical impact because of the astounding number of false-positives. To protect users from malware, we still need to: 402 | - [Make Malware Verdicts Auditable](https://github.com/pypa/warehouse/issues/7421) - Right now, verdicts are removed once the associated project/package is removed. We need to change the backend to retain past verdicts so we can evaluate and improve the efficacy of this system. 403 | - [Detect packages being published with typo'ish names](https://github.com/pypa/warehouse/issues/4998) - Add a typosquatting check. 404 | - [Implement a more robust malware detector](https://github.com/pypa/warehouse/issues/7748) - One current check relies on simple pattern matching with YARA. A better approach requires parsing the package code into an AST. 405 | - [YARA rules for setup.py not ignoring comments](https://github.com/pypa/warehouse/issues/7475) - Related to the above issue. 406 | - Release a canonical dataset for developers to write their checks against, and iterate on to improve detection accuracy. 407 | 408 | We also want to set up a partnership with [VirusTotal](https://www.virustotal.com/) or a similar third-party virus checking service during the check development to scan every uploaded package. Integration with a third-party virus scanner is low-hanging fruit that could move the needle on PyPI package security. 409 | 410 | Funding would be used for backend development, security engineering, project management, system administration, and publicity to stakeholders. Ideally, AV integrations would be donated by the vendors. 411 | 412 | # Items that have now been funded 413 | 414 | Some TODOs that were on this page have now received funding! 415 | 416 | ## Foundational tool improvements 417 | 418 | ### Finish dependency resolver for pip 419 | 420 | __([This is now funded](https://pyfound.blogspot.com/2019/11/seeking-developers-for-paid-contract.html) and 421 | [we hired people to work on this project](https://github.com/python/request-for/blob/master/2020-pip/RFP.md). The [new resolver is in beta](https://blog.python.org/2020/07/upgrade-pip-20-2-changes-20-3.html).)__ 422 | 423 | We're partway through a next-generation rewrite of the dependency 424 | resolver within pip, Python's package download and installation tool. 425 | The project ran into massive technical debt, but the refactoring is 426 | nearly finished and prototype functionality is in alpha now. 427 | ([In-depth explanation by Sebastian Awwad of the problem & our approach](https://docs.google.com/document/d/1x_VrNtXCup75qA3glDd2fQOB2TakldwjKZ6pXaAjAfg/edit), 428 | [lead developer Pradyun Gedam's initial plan](https://gist.github.com/pradyunsg/5cf4a35b81f08b6432f280aba6f511eb), 429 | [2017 status updates](https://pradyunsg.me/gsoc-2017/), and [GitHub issue #988 tracking progress](https://github.com/pypa/pip/issues/988) and [June 2019 status update](https://pradyunsg.me/blog/2019/06/23/pip-update/), and 430 | [issue #6536 for planning rollout](https://github.com/pypa/pip/issues/6536).) 431 | 432 | Funding would support user experience, communications/publicity, and 433 | testing work (including developing robust testing/CI infrastructure) as 434 | well as core feature development and review. 435 | 436 | We need to finish the resolver because so many other improvements are 437 | blocked on it: 438 | 439 | * [adding an "upgrade-all" command to pip](https://github.com/pypa/pip/issues/4551) 440 | * [warning when trying to download or build wheels from incompatible set of packages/requirements](https://github.com/pypa/pip/issues/5497) 441 | * [adding a no-implicit-upgrades strategy](https://github.com/pypa/pip/issues/4745) 442 | * [making PyPI and pip enforce metadata compliance more strictly](https://github.com/pypa/packaging-problems/issues/264) 443 | * [warning the user when uninstalling a package that other packages depend on](https://github.com/pypa/pip/issues/4681) 444 | * [properly respecting constraints](https://github.com/pypa/pip/issues/6495) 445 | * [recording requested and installed extras](https://github.com/pypa/packaging-problems/issues/215) 446 | * [option to show what versions of packages are currently available](https://github.com/pypa/pip/issues/53) 447 | * [listing packages' dependencies and dependents on PyPI](https://github.com/pypa/packaging-problems/issues/54) 448 | * [minimizing duplication of work between pip and pipenv](https://mail.python.org/archives/list/distutils-sig@python.org/thread/2QECNWSHNEW7UBB24M2K5BISYJY7GMZF/#2QECNWSHNEW7UBB24M2K5BISYJY7GMZF) 449 | * [better pipenv functionality](https://github.com/pypa/pipenv/issues?q=is%3Aopen+is%3Aissue+label%3A%22Category%3A+Dependency+Resolution%22) 450 | * [package namespace support](https://discuss.python.org/t/namespace-support-in-pypi/1609/35) 451 | * [moving more code out of Python's standard library so we can release improvements faster](https://discuss.python.org/t/if-python-started-moving-more-code-out-of-the-stdlib-and-into-pypi-packages-what-technical-mechanisms-could-packaging-use-to-ease-that-transition/1738/24) 452 | 453 | and it would fix so many dependency issues for our users: 454 | 455 | * [Django installation conflict](https://github.com/pypa/pip/issues/4907) 456 | * [cherrypy/six/cheroot installation conflict](https://github.com/pradyunsg/zazo/issues/2) 457 | * [Spyder downgrade requirement](https://github.com/pypa/pip/issues/5043) 458 | * [boto3/bravado dependency failure](https://github.com/pradyunsg/zazo/issues/4) 459 | * [Ansible/PyOpenSSL/cryptography failure](https://github.com/pypa/pip/issues/5313) 460 | * [extras installation failure](https://github.com/pypa/pip/issues/4957) 461 | * [extras upgrade failure](https://github.com/pypa/pip/issues/4391) 462 | * [breaking installed packages](https://github.com/pypa/pip/issues/6494) 463 | * [elasticsearch/requests failure](https://github.com/pradyunsg/zazo/issues/14) 464 | * [hatch, another packaging tool](https://github.com/ofek/hatch/issues/47) 465 | 466 | And in our larger ecology, this causes installation problems for: 467 | 468 | * [conda's compatibility with pip](https://github.com/conda/conda/issues/8657) 469 | * [the Servo browser engine](https://github.com/servo/servo/issues/10611) 470 | * [numpy and scipy](https://github.com/pypa/pip/issues/4582) 471 | * [Canonical's DevOps tool Juju](https://github.com/juju/python-libjuju/issues/45) 472 | * [a Cap'n Proto implementation](https://github.com/antocuni/capnpy/issues/16) 473 | * [toil, awscli, and boto3](https://github.com/DataBiosphere/toil/issues/2230) 474 | * [the Mozilla website & icalendar](https://github.com/mozilla/bedrock/issues/5967) 475 | * [certbot, in the past and possibly the future](https://github.com/certbot/certbot/issues/5195) 476 | * [TurboGears](https://github.com/TurboGears/tg2devtools/issues/13) 477 | * [a JIRA API client library](https://github.com/pycontribs/jira/pull/744) 478 | * [a WebSocket protocol test suite](https://github.com/crossbario/autobahn-testsuite/issues/55) 479 | * [Robot Operating System tooling](https://github.com/gerkey/ros1_external_use/issues/7) 480 | 481 | ### Improve pip user experience 482 | 483 | __This is now funded, thanks to 484 | [the Chan Zuckerberg Initiative](https://chanzuckerberg.com/eoss/proposals/improving-user-experience-and-debuggability-of-pip-for-all-python-users/) and 485 | [Mozilla Open Source Support](https://www.mozilla.org/en-US/moss/).__ 486 | 487 | `pip`'s user experience needs to improve by providing 488 | [better error messages](https://github.com/pypa/pip/milestone/25) 489 | and prompts, logs, output, and reporting, and becoming more consistent 490 | across features, to fit the user's mental model better, make hairy 491 | problems easier to untangle, and reduce unintended data loss. `pip`'s 492 | maintainers have [a list of TODOs](https://github.com/pypa/pip/milestone/10) and need funding so that user experience researchers, UX 493 | designers, developers, and technical writers can spend dedicated time 494 | addressing them. 495 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Packaging improvements that could be funded 2 | 3 | This repository holds [FUNDABLES.md](./FUNDABLES.md): the list of tasks that the [Packaging Working Group](https://wiki.python.org/psf/PackagingWG) is seeking funding for. We use this list to prioritize how we use [donations](http://donate.pypi.org/), and to write proposals asking for grants and directed gifts. 4 | 5 | The Packaging WG uses three criteria to add tasks to this list: 6 | 7 | 1. **the Python packaging community wants it**: there's already consensus among project maintainers, and if a PEP is involved it's been accepted. 8 | 2. **fairly well-scoped** 9 | 3. **fundable**: would happen much faster if the [Packaging Working Group](https://wiki.python.org/psf/PackagingWG) got funding to implement the work. (So, it has to be legal and physically possible.) 10 | 11 | See [past funded projects](https://pyfound.blogspot.com/2020/04/sponsoring-python-packaging.html) for examples. All money goes through [the Python Software Foundation](https://www.python.org/psf/). 12 | 13 | The list is roughly prioritized by urgency and impact, but is not a roadmap. Please contact the Packaging WG to ask us to estimate how much one of these improvements would cost; we'll get back to you within a few business days. 14 | 15 | ## Suggest a task 16 | 17 | To suggest a task for the list, make a pull request against this repo. We welcome your suggestion. 18 | 19 | ## Maintainers and contact 20 | 21 | This list is maintained and updated by the [Packaging Working Group](https://wiki.python.org/psf/PackagingWG), a [volunteer work group of the Python Software Foundation](https://www.python.org/psf/workgroups/#packaging-work-group), approximately twice a month. Please [refer to the WG for administration and contact info](https://wiki.python.org/psf/PackagingWG#Administration_and_Contact). 22 | --------------------------------------------------------------------------------