borgbackup/borg

Offer wheels for borg on pypi

Open

#7370 opened on Feb 22, 2023

View on GitHub
 (7 comments) (0 reactions) (0 assignees)Python (10,086 stars) (732 forks)batch import
binarieshelp wantedquestion

Description

Continuing on from https://github.com/borgbackup/borg/issues/6721 with new information and ideas:

Whilst still working with @grantbevis on https://github.com/borgmatic-collective/docker-borgmatic, I have revisited cibuildwheel.

Using github actions, it should be relatively straightforward to build a wheel for x86_64 and arm64, then upload to pypi.

Initial setup would look something like this.

And delivery to pypi can be achieved like this.

I have done a brief test on an Ubuntu VM to generate a x86_64 and aarch64 musllinux_1_1 wheel:

Test steps: Fresh Ubuntu VM Install pipx + git

git clone https://github.com/borgbackup/borg.git
cd borg
nano pyproject.toml

Add the following to the top of the file (I'm sure not all of these packages are required, some are historical for some other packages, so please feel free to optimise):

[tool.cibuildwheel]
before-all = "apk add -Uu acl-dev alpine-sdk attr-dev attr-dev bzip2-dev fuse-dev g++ libcrypto1.1 libffi-dev libssl1.1 libxxhash linux-headers lz4-dev musl-dev ncurses-dev openssl-dev py3-msgpack py3-packaging py3-pip py3-setuptools py3-setuptools_scm py3-wheel py3-xxhash python3-dev readline-dev sqlite-dev tree xxhash-dev xz-dev zlib-dev zstd-dev && python3 -m pip install -U pkgconfig pip setuptools wheel"
build = "cp311-musllinux_*"

[tool.cibuildwheel.linux]
archs = ["x86_64", "aarch64"]

run pipx run cibuildwheel --platform linux

Output:

root@borg:/home/modem7/borg# ls wheelhouse/
borgbackup-2.0.0b5.dev276+g3f2aac8f.d20230222-cp311-cp311-musllinux_1_1_aarch64.whl  
borgbackup-2.0.0b5.dev276+g3f2aac8f.d20230222-cp311-cp311-musllinux_1_1_x86_64.whl

I'm sure much more modifications can be done to better suit the project, but it seems like a quick win to implement, and automatically build and deploy.

@ThomasWaldmann and @grantbevis - I know we've discussed this before to a degree, but cibuildwheel has come quite some distance now, especially with being able to build musllinux.

Do either of you have any thoughts on the above?

Contributor guide

Tech stack
pythongithub actionsdocker
Domain
build systemci cdrelease
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
needs maintainer response
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Python packagingGitHub ActionsDocker basics
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
65
Research direction
The issue proposes adding cibuildwheel to build and upload Python wheels for multiple architectures (x86 64, arm64) on Linux musllinux. The main file to modify is pyproject.toml to add [tool.cibuildwheel] configuration as shown in the test steps. Additionally, a GitHub Actions workflow would need to be created to trigger builds on releases or pushes. The related discussion in issue #6721 and the test steps provided give a concrete starting point. Maintainers should confirm the desired build matrix and the deployment target (PyPI) before implementing.