Floating cheeses

[FOSDEM 2022] InterPlanetary Wheels: A resilient approach to distributing software

Floating cheeses — Sun, 06 Feb 2022 15:06:54 GMT

This is the recording of a lightning talk at FOSDEM 2022, titled InterPlanetary Wheels: A resilient approach to distributing software. The original description read:

InterPlanetary Wheels (IPWHL) are platform-unique, singly-versioned Python built distributions backed by IPFS for security and reproducibility. Using the peer-to-peer file system IPFS, the distributions have the advantage of being easily replicated and not having a single point of failure, thus are more resilient. While this project targets at Python package in particular, the idea can be similarly applied to other software distributions such as Linux distributions.

[PackagingCon 2021] IPFS ❤ Python Wheels: Efficient, Secure and Reproducible Repository

Floating cheeses — Thu, 11 Nov 2021 13:49:53 GMT

Python wheel is a beautifully simple format for cross-platform binary distribution. Combining it with the simple repository API, we have the Python Package Index (PyPI) tirelessly serving Pythonistas. PyPI is great as a package index, but in certain ways it is unsuitable for end-user usages: it is subject to multiple supply chain attacks, its centralised nature leads to difficult mirroring while being a single point of failure, and expensive dependency resolution is left for client-side.

The interplanetary wheels (IPWHL) are platform-unique, singly-versioned Python binary distributions backed by IPFS. It does not try to replace PyPI but aims to be a downstream wheel supplier in a fashion similar to GNU/Linux distributions, whilst take advantage of a content-addressing peer-to-peer network to provide a reproducible, easy-to-mirror source of packages.

This talk will first briefly discuss the wheel package format and the current state of PyPI in the Python packaging ecosystem, focusing on a few shortcoming and relevant recent efforts. It will then introduce IPWHL from the motivating philosophy to real-world properties, before showing the current process and a demo usage. As this happens, the upstream-downstream relationship in Python packaging will be analyzed, clarifying the role of each actor in the process. After the talk, the audience will know how (and when) IPWHL can benefit them and the different ways they can directly and indirectly help the project.

[PyData Global 2021] Towards Collaborative Reproducibility: Pinning Repository of Binary Distributions

Floating cheeses — Sun, 31 Oct 2021 07:36:13 GMT

Instead of pinning individual requirements or packaging up the entire environment in a container, how about archiving a repository snapshot of all dependencies? This talk at PyData Global 2021 will explore the characteristics of this approach, as well as what have been done with the InterPlanetary Wheels to realize them.

IPWHL is a fresh take on Python dependency management. It aims to be a downstream wheel supplier (in a similar fashion to GNU/Linux distributions), whilst taking advantage of a content-addressing peer-to-peer network to provide a reproducible, easy-to-mirror source of packages.

While communicating the ideas around IPWHL, I will also discuss and compare it with common packaging and archival practices, highlighting the gap between upstream developers and the users of their libraries, in the following order:

Upstream and downstream development: why do we need both?
PyPI is a package index, not a user-facing repository!
Containers are not as useful as you (might) think.
Is IPWHL one-size-fits-all? No, but…
How to use IPWHL and how can one help?

After the presentation, an attendee shall know the advantages (and shortcomings) of IPWHL and how to get started. Prior knowledge on Python packaging and IPFS can be helpful, but not required to understand the content of the talk.

Slides: https://git.sr.ht/~huyngo/ipwhl-pres/blob/main/pydata-global-2021/slides.pdf