Extending or Customizing Setuptools¶
Setuptools design is based on the distutils package originally distributed as part of Python’s standard library, effectively serving as its successor (as established in PEP 632).
This means that setuptools
strives to honor the extension mechanisms
provided by distutils
, and allows developers to create third party packages
that modify or augment the build process behavior.
A simple way of doing that is to hook in new or existing
commands and setup()
arguments just by defining “entry points”. These
are mappings from command or argument names to a specification of where to
import a handler from. (See the section on Advertising Behavior for some more background on entry points).
The following sections describe the most common procedures for extending
the distutils
functionality used by setuptools
.
Important
Any entry-point defined in your setup.cfg
, setup.py
or
pyproject.toml
files are not immediately available for use. Your
package needs to be installed first, then setuptools
will be able to
access these entry points. For example consider a Project-A
that
defines entry points. When building Project-A
, these will not be
available. If Project-B
declares a build system requirement on Project-A
, then setuptools
will be able to use Project-A
’ customizations.
Customizing Commands¶
Both setuptools
and distutils
are structured around the command design
pattern. This means that each main action executed when building a
distribution package (such as creating a sdist
or wheel) correspond to the implementation of a Python class.
Originally in distutils
, these commands would correspond to actual CLI
arguments that could be passed to the setup.py
script to trigger a
different aspect of the build. In setuptools
, however, these command
objects are just a design abstraction that encapsulate logic and help to
organise the code.
You can overwrite existing commands (or add new ones) by defining entry
points in the distutils.commands
group. For example, if you wanted to add
a foo
command, you might add something like this to your project:
# setup.cfg
...
[options.entry_points]
distutils.commands =
foo = mypackage.some_module:foo
Assuming, of course, that the foo
class in mypackage.some_module
is
a setuptools.Command
subclass (documented below).
Once a project containing such entry points has been activated on sys.path
,
(e.g. by running pip install
) the command(s) will be available to any
setuptools
-based project. In fact, this is
how setuptools’ own commands are installed: the setuptools project’s setup
script defines entry points for them!
The commands sdist
, build_py
and build_ext
are especially useful
to customize setuptools
builds. Note however that when overwriting existing
commands, you should be very careful to maintain API compatibility.
Custom commands should try to replicate the same overall behavior as the
original classes, and when possible, even inherit from them.
You should also consider handling exceptions such as CompileError
,
LinkError
, LibError
, among others. These exceptions are available in
the setuptools.errors
module.
- class setuptools.Command(dist: Distribution, **kw)¶
Setuptools internal actions are organized using a command design pattern. This means that each action (or group of closely related actions) executed during the build should be implemented as a
Command
subclass.These commands are abstractions and do not necessarily correspond to a command that can (or should) be executed via a terminal, in a CLI fashion (although historically they would).
When creating a new command from scratch, custom defined classes SHOULD inherit from
setuptools.Command
and implement a few mandatory methods. Between these mandatory methods, are listed:initialize_options()
,finalize_options()
andrun()
.A useful analogy for command classes is to think of them as subroutines with local variables called “options”. The options are “declared” in
initialize_options()
and “defined” (given their final values, aka “finalized”) infinalize_options()
, both of which must be defined by every command class. The “body” of the subroutine, (where it does all the work) is therun()
method. Betweeninitialize_options()
andfinalize_options()
,setuptools
may set the values for options/attributes based on user’s input (or circumstance), which means that the implementation should be careful to not overwrite values infinalize_options()
unless necessary.Please note that other commands (or other parts of setuptools) may also overwrite the values of the command’s options/attributes multiple times during the build process. Therefore it is important to consistently implement
initialize_options()
andfinalize_options()
. For example, all derived attributes (or attributes that depend on the value of other attributes) SHOULD be recomputed infinalize_options()
.When overwriting existing commands, custom defined classes MUST abide by the same APIs implemented by the original class. They also SHOULD inherit from the original class.
- abstract finalize_options() None ¶
Set final values for all options/attributes used by the command. Most of the time, each option/attribute/cache should only be set if it does not have any value yet (e.g.
if self.attr is None: self.attr = val
).
Supporting sdists and editable installs in build
sub-commands¶
build
sub-commands (like build_py
and build_ext
)
are encouraged to implement the following protocol:
- class setuptools.command.build.SubCommand(*args, **kwargs)¶
In order to support editable installations (see PEP 660) all build subcommands SHOULD implement this protocol. They also MUST inherit from
setuptools.Command
.When creating an editable wheel,
setuptools
will try to evaluate custombuild
subcommands using the following procedure:setuptools
will set theeditable_mode
attribute toTrue
setuptools
will execute therun()
command.Important
Subcommands SHOULD take advantage of
editable_mode=True
to adequate its behaviour or perform optimisations.For example, if a subcommand doesn’t need to generate an extra file and all it does is to copy a source file into the build directory,
run()
SHOULD simply “early return”.Similarly, if the subcommand creates files that would be placed alongside Python files in the final distribution, during an editable install the command SHOULD generate these files “in place” (i.e. write them to the original source directory, instead of using the build directory). Note that
get_output_mapping()
should reflect that and include mappings for “in place” builds accordingly.setuptools
use any knowledge it can derive from the return values ofget_outputs()
andget_output_mapping()
to create an editable wheel. When relevantsetuptools
MAY attempt to use file links based on the value ofget_output_mapping()
. Alternatively,setuptools
MAY attempt to use import hooks to redirect any attempt to import to the directory with the original source code and other files built in place.
Please note that custom sub-commands SHOULD NOT rely on
run()
being executed (or not) to provide correct return values forget_outputs()
,get_output_mapping()
orget_source_files()
. Theget_*
methods should work independently ofrun()
.- build_lib: str¶
String representing the directory where the build artifacts should be stored, e.g.
build/lib
. For example, if a distribution wants to provide a Python module namedpkg.mod
, then a corresponding file should be written to{build_lib}/package/module.py
. A way of thinking about this is that the files saved underbuild_lib
would be eventually copied to one of the directories insite.PREFIXES
upon installation.A command that produces platform-independent files (e.g. compiling text templates into Python functions), CAN initialize
build_lib
by copying its value from thebuild_py
command. On the other hand, a command that produces platform-specific files CAN initializebuild_lib
by copying its value from thebuild_ext
command. In general this is done inside thefinalize_options
method with the help of theset_undefined_options
command:def finalize_options(self): self.set_undefined_options("build_py", ("build_lib", "build_lib")) ...
- editable_mode: bool = False¶
Boolean flag that will be set to
True
when setuptools is used for an editable installation (see PEP 660). Implementations SHOULD explicitly set the default value of this attribute toFalse
. When subcommands run, they can use this flag to perform optimizations or change their behaviour accordingly.
- finalize_options()¶
(Required by the original
setuptools.Command
interface)
- get_output_mapping() dict[str, str] ¶
Return a mapping between destination files as they would be produced by the build (dict keys) into the respective existing (source) files (dict values). Existing (source) files should be represented as strings relative to the project root directory. Destination files should be strings in the form of
"{build_lib}/destination/file/path"
.
- get_outputs() list[str] ¶
Return a list of files intended for distribution as they would have been produced by the build. These files should be strings in the form of
"{build_lib}/destination/file/path"
.Note
The return value of
get_output()
should include all files used as keys inget_output_mapping()
plus files that are generated during the build and don’t correspond to any source file already present in the project.
- get_source_files() list[str] ¶
Return a list of all files that are used by the command to create the expected outputs. For example, if your build command transpiles Java files into Python, you should list here all the Java files. The primary purpose of this function is to help populating the
sdist
with all the files necessary to build the distribution. All files should be strings relative to the project root directory.
- initialize_options()¶
(Required by the original
setuptools.Command
interface)
- run()¶
(Required by the original
setuptools.Command
interface)
Adding Arguments¶
Warning
Adding arguments to setup is discouraged as such arguments are only supported through imperative execution and not supported through declarative config.
Sometimes, your commands may need additional arguments to the setup()
call. You can enable this by defining entry points in the
distutils.setup_keywords
group. For example, if you wanted a setup()
argument called bar_baz
, you might add something like this to your
extension project:
# setup.cfg
...
[options.entry_points]
distutils.commands =
foo = mypackage.some_module:foo
distutils.setup_keywords =
bar_baz = mypackage.some_module:validate_bar_baz
The idea here is that the entry point defines a function that will be called
to validate the setup()
argument, if it’s supplied. The Distribution
object will have the initial value of the attribute set to None
, and the
validation function will only be called if the setup()
call sets it to
a non-None
value. Here’s an example validation function:
def assert_bool(dist, attr, value):
"""Verify that value is True, False, 0, or 1"""
if bool(value) != value:
raise SetupError(
"%r must be a boolean value (got %r)" % (attr,value)
)
Your function should accept three arguments: the Distribution
object,
the attribute name, and the attribute value. It should raise a
SetupError
(from the setuptools.errors
module) if the argument
is invalid. Remember, your function will only be called with non-None
values,
and the default value of arguments defined this way is always None
. So, your
commands should always be prepared for the possibility that the attribute will
be None
when they access it later.
If more than one active distribution defines an entry point for the same
setup()
argument, all of them will be called. This allows multiple
extensions to define a common argument, as long as they agree on
what values of that argument are valid.
Customizing Distribution Options¶
Plugins may wish to extend or alter the options on a Distribution
object to
suit the purposes of that project. For example, a tool that infers the
Distribution.version
from SCM-metadata may need to hook into the
option finalization. To enable this feature, Setuptools offers an entry
point setuptools.finalize_distribution_options
. That entry point must
be a callable taking one argument (the Distribution
instance).
If the callable has an .order
property, that value will be used to
determine the order in which the hook is called. Lower numbers are called
first and the default is zero (0).
Plugins may read, alter, and set properties on the distribution, but each plugin is encouraged to load the configuration/settings for their behavior independently.
Defining Additional Metadata¶
Some extensible applications and frameworks may need to define their own kinds
of metadata, which they can then access using the importlib.metadata
APIs.
Ordinarily, this is done by having plugin
developers include additional files in their ProjectName.egg-info
directory. However, since it can be tedious to create such files by hand, you
may want to create an extension that will create the necessary files
from arguments to setup()
, in much the same way that setuptools
does
for many of the setup()
arguments it adds. See the section below for more
details.
Adding new EGG-INFO Files¶
Some extensible applications or frameworks may want to allow third parties to
develop plugins with application or framework-specific metadata included in
the plugins’ EGG-INFO directory, for easy access via the pkg_resources
metadata API. The easiest way to allow this is to create an extension
to be used from the plugin projects’ setup scripts (via setup_requires
)
that defines a new setup keyword, and then uses that data to write an EGG-INFO
file when the egg_info
command is run.
The egg_info
command looks for extension points in an egg_info.writers
group, and calls them to write the files. Here’s a simple example of an
extension defining a setup argument foo_bar
, which is a list of
lines that will be written to foo_bar.txt
in the EGG-INFO directory of any
project that uses the argument:
# setup.cfg
...
[options.entry_points]
distutils.setup_keywords =
foo_bar = setuptools.dist:assert_string_list
egg_info.writers =
foo_bar.txt = setuptools.command.egg_info:write_arg
This simple example makes use of two utility functions defined by setuptools for its own use: a routine to validate that a setup keyword is a sequence of strings, and another one that looks up a setup argument and writes it to a file. Here’s what the writer utility looks like:
def write_arg(cmd, basename, filename):
argname = os.path.splitext(basename)[0]
value = getattr(cmd.distribution, argname, None)
if value is not None:
value = "\n".join(value) + "\n"
cmd.write_or_delete_file(argname, filename, value)
As you can see, egg_info.writers
entry points must be a function taking
three arguments: a egg_info
command instance, the basename of the file to
write (e.g. foo_bar.txt
), and the actual full filename that should be
written to.
In general, writer functions should honor the command object’s dry_run
setting when writing files, and use logging
to do any console output.
The easiest way to conform to this requirement is to use
the cmd
object’s write_file()
, delete_file()
, and
write_or_delete_file()
methods exclusively for your file operations.
See those methods’ docstrings for more details.
Adding Support for Revision Control Systems¶
If the files you want to include in the source distribution are tracked using Git, Mercurial or SVN, you can use the following packages to achieve that:
Git and Mercurial: setuptools_scm
SVN: setuptools_svn
If you would like to create a plugin for setuptools
to find files tracked
by another revision control system, you can do so by adding an entry point to
the setuptools.file_finders
group. The entry point should be a function
accepting a single directory name, and should yield all the filenames within
that directory (and any subdirectories thereof) that are under revision
control.
For example, if you were going to create a plugin for a revision control system called “foobar”, you would write a function something like this:
def find_files_for_foobar(dirname):
... # loop to yield paths that start with `dirname`
And you would register it in a setup script using something like this:
# setup.cfg
...
[options.entry_points]
setuptools.file_finders =
foobar = my_foobar_module:find_files_for_foobar
Then, anyone who wants to use your plugin can simply install it, and their local setuptools installation will be able to find the necessary files.
It is not necessary to distribute source control plugins with projects that
simply use the other source control system, or to specify the plugins in
setup_requires
. When you create a source distribution with the sdist
command, setuptools automatically records what files were found in the
SOURCES.txt
file. That way, recipients of source distributions don’t need
to have revision control at all. However, if someone is working on a package
by checking out with that system, they will need the same plugin(s) that the
original author is using.
A few important points for writing revision control file finders:
Your finder function MUST return relative paths, created by appending to the passed-in directory name. Absolute paths are NOT allowed, nor are relative paths that reference a parent directory of the passed-in directory.
Your finder function MUST accept an empty string as the directory name, meaning the current directory. You MUST NOT convert this to a dot; just yield relative paths. So, yielding a subdirectory named
some/dir
under the current directory should NOT be rendered as./some/dir
or/somewhere/some/dir
, but always as simplysome/dir
Your finder function SHOULD NOT raise any errors, and SHOULD deal gracefully with the absence of needed programs (i.e., ones belonging to the revision control system itself. It may, however, use
logging.warning()
to inform the user of the missing program(s).
Final Remarks¶
To use a
setuptools
plugin, your users will need to add your package as a build requirement to their build-system configuration. Please check out our guides on Dependencies Management in Setuptools for more information.Directly calling
python setup.py ...
is considered a deprecated practice. You should not add new commands tosetuptools
expecting them to be run via this interface.