Build System Support¶
What is it?¶
Python packaging has come a long way.
The traditional setuptools
way of packaging Python modules
uses a setup()
function within the setup.py
script. Commands such as
python setup.py bdist
or python setup.py bdist_wheel
generate a
distribution bundle and python setup.py install
installs the distribution.
This interface makes it difficult to choose other packaging tools without an
overhaul. Because setup.py
scripts allow for arbitrary execution, it
is difficult to provide a reliable user experience across environments
and history.
PEP 517 came to the rescue and specified a new standard for packaging and distributing Python modules. Under PEP 517:
A
pyproject.toml
file is used to specify which program to use to generate the distribution.Two functions provided by the program,
build_wheel(directory: str)
andbuild_sdist(directory: str)
, create the distribution bundle in the specifieddirectory
.The program may use its own configuration file or extend the
.toml
file.The actual installation is done with
pip install *.whl
orpip install *.tar.gz
. If*.whl
is available,pip
will go ahead and copy its files into thesite-packages
directory. If not,pip
will look atpyproject.toml
and decide which program to use to ‘build from source’. (Note that if there is nopyproject.toml
file or thebuild-backend
parameter is not defined, then the fall-back behaviour is to usesetuptools
.)
With this standard, switching between packaging tools is a lot easier.
How to use it?¶
Start with a package that you want to distribute. You will need your source
files, a pyproject.toml
file and a setup.cfg
file:
~/meowpkg/
pyproject.toml
setup.cfg
meowpkg/
__init__.py
module.py
The pyproject.toml
file specifies the build system (i.e. what is
being used to package your scripts and install from source). To use it with
setuptools
the content would be:
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"
build_meta
implements setuptools
’ build system support.
The setuptools
package implements the build_sdist
and
build_wheel
commands.
Use setuptools
’ declarative config to
specify the package information in setup.cfg
:
[metadata]
name = meowpkg
version = 0.0.1
description = a package that meows
[options]
packages = find:
Now generate the distribution. To build the package, use PyPA build:
$ pip install -q build
$ python -m build
And now it’s done! The .whl
file and .tar.gz
can then be distributed
and installed:
dist/
meowpkg-0.0.1.whl
meowpkg-0.0.1.tar.gz
$ pip install dist/meowpkg-0.0.1.whl
or:
$ pip install dist/meowpkg-0.0.1.tar.gz
Dynamic build dependencies and other build_meta
tweaks¶
With the changes introduced by PEP 517 and PEP 518, the
setup_requires
configuration field was deprecated in setup.cfg
and
setup.py
, in favour of directly listing build dependencies in the
requires
field of the build-system
table of pyproject.toml
.
This approach has a series of advantages and gives package managers and
installers the ability to inspect the build requirements in advance and
perform a series of optimisations.
However, some package authors might still need to dynamically inspect the final
user’s machine before deciding these requirements. One way of doing that, as
specified by PEP 517, is to “tweak” setuptools.build_meta
by using an
in-tree backend.
Tip
Before implementing an in-tree backend, have a look at PEP 508. Most of the time, dependencies with environment markers are enough to differentiate operating systems and platforms.
If you put the following configuration in your pyproject.toml
:
[build-system]
requires = ["setuptools"]
build-backend = "backend"
backend-path = ["_custom_build"]
then you can implement a thin wrapper around build_meta
in
the _custom_build/backend.py
file, as shown in the following example:
from setuptools import build_meta as _orig
from setuptools.build_meta import *
def get_requires_for_build_wheel(config_settings=None):
return _orig.get_requires_for_build_wheel(config_settings) + [...]
def get_requires_for_build_sdist(config_settings=None):
return _orig.get_requires_for_build_sdist(config_settings) + [...]
Note
You can override any of the functions specified in PEP 517, not only the ones responsible for gathering
requirements. It is important to import *
so that the hooks that you
choose not to reimplement would be inherited from the setuptools’ backend
automatically. This will also cover hooks that might be added in the future
like the ones that PEP 660 declares.
Important
Make sure your backend script is included in the source distribution, otherwise the build will fail. This can be done by using a SCM/VCS plugin (like setuptools-scm and setuptools-svn), or by correctly setting up MANIFEST.in.
The generated .tar.gz
and .whl
files are compressed archives that
can be inspected as follows:
On POSIX systems, this can be done with tar -tf dist/*.tar.gz
and unzip -l dist/*.whl
.
On Windows systems, you can rename the .whl
to .zip
to be able to
inspect it from File Explorer. You can also use the above tar
command in a
command prompt to inspect the .tar.gz
file. Alternatively, there are GUI programs
like 7-zip that handle .tar.gz
and .whl
files.
In general, the backend script should be present in the .tar.gz
(so the
project can be built from the source) but not in the .whl
(otherwise the
backend script would end up being distributed alongside your package).
See “Package Discovery and Namespace Packages” for more details about package
files.