[BUG] Inconsistent behaviour of providing extension without sources between platforms

[Andrej730]

setuptools version

82.0.1

Python version

3.13

OS

Windows

Additional environment information

No response

Description

Hi! Stumbled upon an inconsistency when building with setuptools. I was using ext_modules and Extension to provide prebuilt extension (building process is complicated and it's simpler to just build it separately). This workflow works OK on Unix - .so is included to .whl and .whl is correctly named repo-0.0.0-cp313-cp313-linux_x86_64.whl, using the platform tag. But when done on Windows it breaks with:

File "C:\Users\XXX\AppData\Local\Temp\build-env-70ukpjvj\Lib\site-packages\setuptools\_distutils\compilers\C\msvc.py", line 537, in link
    build_temp = os.path.dirname(objects[0])
                                 ~~~~~~~^^^
IndexError: list index out of range

ERROR Backend subprocess exited when trying to invoke build_wheel.

Expected behavior

setuptools to include existing binary to the wheel and assing platform tag, as it does on Unix.

How to Reproduce

0. Use Windows.
1. Run setup-env.py (code below)
2. cd setup-test && python -m build

setup-env.py:

from pathlib import Path
import sys

# Create the directory structure
base_dir = Path("setup-test")
base_dir.mkdir(exist_ok=True)

repo_dir = base_dir / "repo"
repo_dir.mkdir(exist_ok=True)

# Create __init__.py in repo
init_file = repo_dir / "__init__.py"
init_file.touch()

# Create setup.py in setup-test
setup_py = base_dir / "setup.py"
setup_py.write_text(
"""from setuptools import setup, find_packages, Extension

setup(
    name="repo",
    packages=find_packages(include=["repo"]),
    ext_modules=[Extension("repo.wrapper", sources=[])],
)
""")

# Create repo/wrapper.so (or .pyd on Windows) using touch
ext = ".pyd" if sys.platform == "win32" else ".so"
wrapper_so = repo_dir / f"wrapper{ext}"
wrapper_so.touch()

print("✓ Created setup-test/")
print("✓ Created setup-test/repo/")
print("✓ Created setup-test/repo/__init__.py")
print("✓ Created setup-test/setup.py")
print(f"✓ Created setup-test/repo/wrapper{ext}")

Output

Full Traceback

File "C:\Users\XXX\AppData\Roaming\Python\Python313\site-packages\pyproject_hooks\_in_process\_in_process.py", line 389, in <module>
    main()
    ~~~~^^
  File "C:\Users\XXX\AppData\Roaming\Python\Python313\site-packages\pyproject_hooks\_in_process\_in_process.py", line 373, in main
    json_out["return_val"] = hook(**hook_input["kwargs"])
                             ~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\XXX\AppData\Roaming\Python\Python313\site-packages\pyproject_hooks\_in_process\_in_process.py", line 280, in build_wheel
    return _build_backend().build_wheel(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        wheel_directory, config_settings, metadata_directory
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "C:\Users\XXX\AppData\Local\Temp\build-env-70ukpjvj\Lib\site-packages\setuptools\build_meta.py", line 438, in build_wheel
    return _build(['bdist_wheel'])
  File "C:\Users\XXX\AppData\Local\Temp\build-env-70ukpjvj\Lib\site-packages\setuptools\build_meta.py", line 429, in _build
    return self._build_with_temp_dir(
           ~~~~~~~~~~~~~~~~~~~~~~~~~^
        cmd,
        ^^^^
    ...<3 lines>...
        self._arbitrary_args(config_settings),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "C:\Users\XXX\AppData\Local\Temp\build-env-70ukpjvj\Lib\site-packages\setuptools\build_meta.py", line 410, in _build_with_temp_dir
    self.run_setup()
    ~~~~~~~~~~~~~~^^
  File "C:\Users\XXX\AppData\Local\Temp\build-env-70ukpjvj\Lib\site-packages\setuptools\build_meta.py", line 520, in run_setup
    super().run_setup(setup_script=setup_script)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\XXX\AppData\Local\Temp\build-env-70ukpjvj\Lib\site-packages\setuptools\build_meta.py", line 317, in run_setup
    exec(code, locals())
    ~~~~^^^^^^^^^^^^^^^^
  File "<string>", line 3, in <module>
  File "C:\Users\XXX\AppData\Local\Temp\build-env-70ukpjvj\Lib\site-packages\setuptools\__init__.py", line 117, in setup
    return distutils.core.setup(**attrs)  # type: ignore[return-value]
           ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "C:\Users\XXX\AppData\Local\Temp\build-env-70ukpjvj\Lib\site-packages\setuptools\_distutils\core.py", line 186, in setup
    return run_commands(dist)
  File "C:\Users\XXX\AppData\Local\Temp\build-env-70ukpjvj\Lib\site-packages\setuptools\_distutils\core.py", line 202, in run_commands
    dist.run_commands()
    ~~~~~~~~~~~~~~~~~^^
  File "C:\Users\XXX\AppData\Local\Temp\build-env-70ukpjvj\Lib\site-packages\setuptools\_distutils\dist.py", line 1000, in run_commands
    self.run_command(cmd)
    ~~~~~~~~~~~~~~~~^^^^^
  File "C:\Users\XXX\AppData\Local\Temp\build-env-70ukpjvj\Lib\site-packages\setuptools\dist_py", line 1107, in run_command
    super().run_command(command)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "C:\Users\XXX\AppData\Local\Temp\build-env-70ukpjvj\Lib\site-packages\setuptools\_distutils\dist.py", line 1019, in run_command
    cmd_obj.run()
    ~~~~~~~~~~~^^
  File "C:\Users\XXX\AppData\Local\Temp\build-env-70ukpjvj\Lib\site-packages\setuptools\command\bdist_wheel.py", line 370, in run
    self.run_command("build")
    ~~~~~~~~~~~~~~~~^^^^^^^^^
  File "C:\Users\XXX\AppData\Local\Temp\build-env-70ukpjvj\Lib\site-packages\setuptools\_distutils\cmd.py", line 341, in run_command
    self.distribution.run_command(command)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "C:\Users\XXX\AppData\Local\Temp\build-env-70ukpjvj\Lib\site-packages\setuptools\dist.py", line 1107, in run_command
    super().run_command(command)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "C:\Users\XXX\AppData\Local\Temp\build-env-70ukpjvj\Lib\site-packages\setuptools\_distutils\dist.py", line 1019, in run_command
    cmd_obj.run()
    ~~~~~~~~~~~^^
  File "C:\Users\XXX\AppData\Local\Temp\build-env-70ukpjvj\Lib\site-packages\setuptools\_distutils\command\build.py", line 135, in run
    self.run_command(cmd_name)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "C:\Users\XXX\AppData\Local\Temp\build-env-70ukpjvj\Lib\site-packages\setuptools\_distutils\cmd.py", line 341, in run_command
    self.distribution.run_command(command)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "C:\Users\XXX\AppData\Local\Temp\build-env-70ukpjvj\Lib\site-packages\setuptools\dist.py", line 1107, in run_command
    super().run_command(command)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "C:\Users\XXX\AppData\Local\Temp\build-env-70ukpjvj\Lib\site-packages\setuptools\_distutils\dist.py", line 1019, in run_command
    cmd_obj.run()
    ~~~~~~~~~~~^^
  File "C:\Users\XXX\AppData\Local\Temp\build-env-70ukpjvj\Lib\site-packages\setuptools\_distutils\command\build_ext.py", line 97, in run
    _build_ext.run(self)
    ~~~~~~~~~~~~~~^^^^^^
  File "C:\Users\XXX\AppData\Local\Temp\build-env-70ukpjvj\Lib\site-packages\setuptools\_distutils\command\build_ext.py", line 367, in run
    self.build_extensions()
    ~~~~~~~~~~~~~~~~~~~~~^^
  File "C:\Users\XXX\AppData\Local\Temp\build-env-70ukpjvj\Lib\site-packages\setuptools\_distutils\command\build_ext.py", line 483, in build_extensions
    self._build_extensions_serial()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "C:\Users\XXX\AppData\Local\Temp\build-env-70ukpjvj\Lib\site-packages\setuptools\_distutils\command\build_ext.py", line 509, in _build_extensions_serial
    self.build_extension(ext)
    ~~~~~~~~~~~~~~~~~~~~^^^^^
  File "C:\Users\XXX\AppData\Local\Temp\build-env-70ukpjvj\Lib\site-packages\setuptools\command\build_ext.py", line 262, in build_extension
    _build_ext.build_extension(self, ext)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "C:\Users\XXX\AppData\Local\Temp\build-env-70ukpjvj\Lib\site-packages\setuptools\_distutils\command\build_ext.py", line 588, in build_extension
    self.compiler.link_shared_object(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        objects,
        ^^^^^^^^
    ...<8 lines>...
        target_lang=language,
        ^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "C:\Users\XXX\AppData\Local\Temp\build-env-70ukpjvj\Lib\site-packages\setuptools\_distutils\compilers\C\base.py", line 805, in link_shared_object
    self.link(
    ~~~~~~~~~^
        Compiler.SHARED_OBJECT,
        ^^^^^^^^^^^^^^^^^^^^^^^
    ...<11 lines>...
        target_lang,
        ^^^^^^^^^^^^
    )
    ^
  File "C:\Users\XXX\AppData\Local\Temp\build-env-70ukpjvj\Lib\site-packages\setuptools\_distutils\compilers\C\msvc.py", line 537, in link
    build_temp = os.path.dirname(objects[0])
                                 ~~~~~~~^^^
IndexError: list index out of range

ERROR Backend subprocess exited when trying to invoke build_wheel
NativeCommandExitException: Program "python.exe" ended with non-zero exit code: 1 (0x00000001).

[abravalheri]

I believe that the behaviour of compiling an extension when there is an empty list of source files is currently undefined, and not supported. We could modify the implementation to always error when the list of sources in empty to make it well defined, but that is something that is likely to require discussion as it is better to avoid backwards incompatible changes...

[Andrej730]

Is there some other way to provide prebuilt binary?
I know that it's also possible using package_data={"repo": "repo.so"}, but then I'd also need to hardcode somewhere wheel tags it should use, instead of delegating it to setuptools.

[Andrej730]

I think I've found a workaround - it works on Windows and makes Unix use of this less hacky - running script below and then cd setup-test && python -m build works on all platforms.
build_ext in setuptools is skipping binaries if they were prebuilt, so if we use package_data to ensure it includes .pyd / .so to package, then when processing `Extension("repo.wrapper", sources=[])`` it skips build process, because build artifact is already there.

Couple caveats:

• I'm using EXT_SUFFIX in the example below (example value - .cp313-win_amd64.pyd), which is important, because build_ext has a fixed suffix that it will expect. If you'd use just .pyd it will not recognize it as a prebuilt artifact. On Unix it will work out, because of the undefined behaviour pointed out above, but you end up having your original .so and empty .cpython-313-x86_64-linux-gnu.so next to it in the wheel.
• Workaround for fixed suffix - you can use custom binary suffix by setting SETUPTOOLS_EXT_SUFFIX env variable. E.g. you can use export SETUPTOOLS_EXT_SUFFIX=".so" to use just .so extension for binaries istead of full .cpython-313-x86_64-linux-gnu.so. It also allows you to do all kind of tomfoolery with packing wheels for other platforms - e.g. by setting it to .cpython-313-wasm32-emscripten.so for pyodide wheels.
• If you want to pack binary for other platform, it's also possible to override platform tag on the final wheel (e.g. python -m build --config-setting="--build-option=--plat-name=pyodide_2025_0_wasm32"), but then you need to manage those tags manually somehow.

from pathlib import Path
from distutils.sysconfig import get_config_var

# Create the directory structure
base_dir = Path("setup-test")
base_dir.mkdir(exist_ok=True)

repo_dir = base_dir / "repo"
repo_dir.mkdir(exist_ok=True)

# Create __init__.py in repo
init_file = repo_dir / "__init__.py"
init_file.touch()

# Create setup.py in setup-test
setup_py = base_dir / "setup.py"

setup_py.write_text(
"""
from setuptools import setup, find_packages, Extension
from distutils.sysconfig import get_config_var
ext = get_config_var("EXT_SUFFIX")
assert isinstance(ext, str)

setup(
    name="repo",
    packages=find_packages(include=["repo"]),
    ext_modules=[Extension("repo.wrapper", sources=[])],
    package_data={
        # "*.ext" is needed to include prebuilt binary extension. Otherwise it would try to build it and fail.
        "repo": ["*"+ext],
    },
)
""")

ext = get_config_var("EXT_SUFFIX")
# Create repo/wrapper.so (or .pyd on Windows) using touch
wrapper_so = repo_dir / f"wrapper{ext}"
wrapper_so.touch()

print("✓ Created setup-test/")
print("✓ Created setup-test/repo/")
print("✓ Created setup-test/repo/__init__.py")
print("✓ Created setup-test/setup.py")
print(f"✓ Created {wrapper_so}")