Data Files Support¶

In the Python ecosystem, the term “data files” is used in various complex scenarios and can have nuanced meanings. For the purposes of this documentation, we define “data files” as non-Python files that are installed alongside Python modules and packages on the user’s machine when they install a distribution via wheel.

These files are typically intended for use at runtime by the package itself or to influence the behavior of other packages or systems.

Old packaging installation methods in the Python ecosystem have traditionally allowed installation of “data files”, which are placed in a platform-specific location. However, the most common use case for data files distributed with a package is for use by the package, usually by including the data files inside the package directory.

Setuptools focuses on this most common type of data files and offers three ways of specifying which files should be included in your packages, as described in the following section.

Subdirectory for Data Files¶

A common pattern is where some (or all) of the data files are placed under a separate subdirectory. For example:

project_root_directory
├── setup.py        # and/or setup.cfg, pyproject.toml
└── src
    └── mypkg
        ├── data
        │   ├── data1.rst
        │   └── data2.rst
        ├── __init__.py
        ├── data1.txt
        └── data2.txt

Here, the .rst files are placed under a data subdirectory inside mypkg, while the .txt files are directly under mypkg.

In this case, the recommended approach is to treat data as a namespace package (refer PEP 420). This way, you can rely on the same methods described above, using either 2. package_data or 1. include_package_data. For the sake of completeness, we include below configuration examples for the subdirectory structure, but please refer to the detailed information in the previous sections of this document.

With 2. package_data, the configuration might look like this:

pyproject.toml

# Scanning for namespace packages in the ``src`` directory is true by
# default in pyproject.toml, so you do NOT need to include the
# `tool.setuptools.packages.find` if it looks like the following:
# [tool.setuptools.packages.find]
# namespaces = true
# where = ["src"]

[tool.setuptools.package-data]
mypkg = ["*.txt"]
"mypkg.data" = ["*.rst"]

setup.cfg

[options]
# ...
packages = find_namespace:
package_dir =
    = src

[options.packages.find]
where = src

[options.package_data]
mypkg =
    *.txt
mypkg.data =
    *.rst

setup.py

from setuptools import setup, find_namespace_packages
setup(
    # ...,
    packages=find_namespace_packages(where="src"),
    package_dir={"": "src"},
    package_data={
        "mypkg": ["*.txt"],
        "mypkg.data": ["*.rst"],
    }
)

In other words, we allow setuptools to scan for namespace packages in the src directory, which enables the data directory to be identified, and then, we separately specify data files for the root package mypkg, and the namespace package data under the package mypkg.

Alternatively, you can also rely on 1. include_package_data. Note that this is the default behaviour in pyproject.toml, but you need to manually enable scanning of namespace packages in setup.cfg or setup.py:

pyproject.toml

[tool.setuptools]
# ...
# By default, include-package-data is true in pyproject.toml, so you do
# NOT have to specify this line.
include-package-data = true

[tool.setuptools.packages.find]
# scanning for namespace packages is true by default in pyproject.toml, so
# you need NOT include this configuration.
namespaces = true
where = ["src"]

setup.cfg

[options]
packages = find_namespace:
package_dir =
    = src
include_package_data = True

[options.packages.find]
where = src

setup.py

from setuptools import setup, find_namespace_packages
setup(
    # ... ,
    packages=find_namespace_packages(where="src"),
    package_dir={"": "src"},
    include_package_data=True,
)

To avoid common mistakes with 1. include_package_data, please ensure MANIFEST.in is properly set or use a revision control system plugin (see Controlling files in the distribution).

Accessing Data Files at Runtime¶

Typically, existing programs manipulate a package’s __file__ attribute in order to find the location of data files. For example, if you have a structure like this:

project_root_directory
├── setup.py        # and/or setup.cfg, pyproject.toml
└── src
    └── mypkg
        ├── data
        │   └── data1.txt
        ├── __init__.py
        └── foo.py

Then, in mypkg/foo.py, you may try something like this in order to access mypkg/data/data1.txt:

import os
data_path = os.path.join(os.path.dirname(__file__), 'data', 'data1.txt')
with open(data_path, 'r') as data_file:
     ...

However, this manipulation isn’t compatible with PEP 302-based import hooks, including importing from zip files and Python Eggs. It is strongly recommended that, if you are using data files, you should use importlib.resources to access them. In this case, you would do something like this:

from importlib.resources import files
data_text = files('mypkg.data').joinpath('data1.txt').read_text()

importlib.resources was added to Python 3.7. However, the API illustrated in this code (using files()) was added only in Python 3.9, [2] and support for accessing data files via namespace packages was added only in Python 3.10 [3] (the data subdirectory is a namespace package under the root package mypkg). Therefore, you may find this code to work only in Python 3.10 (and above). For other versions of Python, you are recommended to use the importlib-resources backport which provides the latest version of this library. In this case, the only change that has to be made to the above code is to replace importlib.resources with importlib_resources, i.e.

from importlib_resources import files
...

See Using importlib_resources for detailed instructions.

Tip

Files inside the package directory should be read-only to avoid a series of common problems (e.g. when multiple users share a common Python installation, when the package is loaded from a zip file, or when multiple instances of a Python application run in parallel).

If your Python package needs to write to a file for shared data or configuration, you can use standard platform/OS-specific system directories, such as ~/.local/config/$appname or /usr/share/$appname/$version (Linux specific) [1]. A common approach is to add a read-only template file to the package directory that is then copied to the correct system directory if no pre-existing file is found.

Data Files from Plugins and Extensions¶

You can resort to a native/implicit namespace package (as a container for files) if you want plugins and extensions to your package to contribute with package data files. This way, all files will be listed during runtime when using importlib.resources. Note that, although not strictly guaranteed, mainstream Python package managers, like pip and derived tools, will install files belong to multiple distributions that share a same namespace into the same directory in the file system. This means that the overhead for importlib.resources will be minimum.

Non-Package Data Files¶

Historically, setuptools by way of easy_install would encapsulate data files from the distribution into the egg (see the old docs). As eggs are deprecated and pip-based installs fall back to the platform-specific location for installing data files, there is no supported facility to reliably retrieve these resources.

Instead, the PyPA recommends that any data files you wish to be accessible at run time be included inside the package.

Data Files Support¶

Configuration Options¶

1. `include_package_data`¶

2. `package_data`¶

3. `exclude_package_data`¶

Interplay between these keywords¶

Summary¶

Subdirectory for Data Files¶

Accessing Data Files at Runtime¶

Data Files from Plugins and Extensions¶

Non-Package Data Files¶

Data Files Support¶

Configuration Options¶

1. include_package_data¶

2. package_data¶

3. exclude_package_data¶

Interplay between these keywords¶

Summary¶

Subdirectory for Data Files¶

Accessing Data Files at Runtime¶

Data Files from Plugins and Extensions¶

Non-Package Data Files¶

1. `include_package_data`¶

2. `package_data`¶

3. `exclude_package_data`¶