Skip to content

Commit bd7696a

Browse files
rickeylevaignas
andauthored
feat!: make windows use venvs (#3680)
This makes Windows use a lite venv to better align it with the unixy implementations. A lite venv is one that has the interpreter and pyvenv.cfg file, but site-packages isn't populated. Unfortunately, it has to recreate the venv at runtime because Bazel incorrectly canonicalizes relative symlinks on Windows. This isn't ideal, however, Windows builds were already going through a zip-unzip process, so creating a handful of symlinks seems like an improvement overall. A particularity of venvs on Windows is that various supporting `.dll` files *must* be in the same directory as `python.exe`. A new attribute (and associated provider plumbing) is added to `py_runtime` to capture these extra files. Note that this **requires** symlink support be enabled in Bazel and Windows. If both aren't enabled, then weird errors will occur because junctions (instead of symlinks) are created that point to files (junctions can only point to directories). Zipapp support for these venvs is also added. This required adding `PyExecutableInfo.venv_interpreter_symlinks` to keep track of relative symlinks that need to be created. Tracking them separately is needed because Bazel has a bug where relative symlinks are made absolute on Windows. The internal plumbing for these symlinks is kept relatively generic to support using such symlinks in site-packages in a future change. Along the way... * Hash-based extract support is added for zipapps passed to Python. * Fixed manually passing a zipapp to Python to bootstrap it. This was previously broken because the launcher doesn't know how to look inside the zip file. * Added `RULES_PYTHON_EXTRACT_ROOT` support when manually passing a zipapp to Python. * Fix several issues with forward-slashes being used on Windows. * Cleanup and improve bootstrap scripts in various ways. * Cleanup some doc in local_runtime_repo. * Have repo_utils.which print nicely formatted path entries on failure. * Add --config=fast-tests to make it easier to run small/medium tests * And --config=testone to make it easier to run a specific test --------- Co-authored-by: Ignas Anikevicius <240938+aignas@users.noreply.github.com>
1 parent dd3614e commit bd7696a

29 files changed

Lines changed: 889 additions & 310 deletions

.bazelrc

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,7 @@ common --enable_bzlmod
4242
# Local disk cache greatly speeds up builds if the regular cache is lost
4343
common --disk_cache=~/.cache/bazel/bazel-disk-cache
4444

45+
4546
# Additional config to use for readthedocs builds.
4647
# See .readthedocs.yml for additional flags that can only be determined from
4748
# the runtime environment.
@@ -54,3 +55,6 @@ common --incompatible_no_implicit_file_export
5455

5556
build --lockfile_mode=update
5657

58+
import %workspace%/specialized_configs.bazelrc
59+
60+
try-import user.bazelrc

CHANGELOG.md

Lines changed: 27 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,9 @@ END_UNRELEASED_TEMPLATE
6464
--windows_enable_symlinks` to your `.bazelrc` to enable Bazel using full
6565
symlink support on Windows.
6666
* venv-based binaries are created by default ({obj}`--bootstrap_impl=system_python`)
67-
on supported platforms (Linux/Mac with Bazel 8+).
67+
on supported platforms (Linux/Mac with Bazel 8+, or Windows).
68+
* `--build_python_zip` on Windows is ignored. Use {obj}`py_zipapp_binary` to create
69+
zips of Python programs.
6870

6971
Other changes:
7072
* (pypi) Update dependencies used for `compile_pip_requirements`, building
@@ -73,19 +75,21 @@ Other changes:
7375
we will from now on fetch the lists of available packages on each index. The
7476
used package mappings will be written as facts to the `MODULE.bazel.lock` file
7577
on supported bazel versions and it should be done at most once. As a result,
76-
per-package {obj}`experimental_index_url_overrides` is no longer needed if the index URLs are
77-
passed to the `pip.parse` via `experimental_index_url` and `experimental_extra_index_urls`.
78-
What is more, we start implementing the flags for `--index_url` and `--extra_index_urls` more in
79-
line to how it is used in `uv` and `pip`, i.e. we default to `--index_url` if the package is not
80-
found in `--extra_index_urls`.
81-
Fixes
82-
([#3260](https://github.com/bazel-contrib/rules_python/issues/3260) and
78+
per-package {obj}`experimental_index_url_overrides` is no longer needed if the
79+
index URLs are passed to the `pip.parse` via `experimental_index_url` and
80+
`experimental_extra_index_urls`. What is more, we start implementing the flags
81+
for `--index_url` and `--extra_index_urls` more in line to how it is used in
82+
`uv` and `pip`, i.e. we default to `--index_url` if the package is not found in
83+
`--extra_index_urls`. Fixes
84+
([#3260](https://github.com/bazel-contrib/rules_python/issues/3260) and
8385
[#2632](https://github.com/bazel-contrib/rules_python/issues/2632)).
84-
* (uv) We will now use the download URL specified in the `uv`'s `dist_manifest.json`
85-
file. If you have redirects or blocking rules as part of your downloader setup,
86-
you may need to adjust them. What is more, the default uv version has been bumped
87-
`0.11.2`.
88-
* (runfiles): We are stopping the type annotation testing with `mypy` for Python 3.9.
86+
* (uv) We will now use the download URL specified in the `uv`'s
87+
`dist_manifest.json` file. If you have redirects or blocking rules as part of
88+
your downloader setup, you may need to adjust them. What is more, the default
89+
uv version has been bumped `0.11.2`.
90+
* (runfiles): Type annotations are no longer tested for Python 3.9.
91+
* Windows no longer defaults to creating a zip file and extracting it; a
92+
symlink-based runfiles tree is created, as on unix-like platforms.
8993

9094
{#v0-0-0-fixed}
9195
### Fixed
@@ -122,11 +126,18 @@ Other changes:
122126
{#v0-0-0-added}
123127
### Added
124128
* (pypi) Write SimpleAPI contents to the `MODULE.bazel.lock` file if using
125-
{obj}`experimental_index_url` which should speed up consecutive initializations and should no
126-
longer require the network access if the cache is hydrated.
127-
Implements [#2731](https://github.com/bazel-contrib/rules_python/issues/2731).
129+
{obj}`experimental_index_url` which should speed up consecutive
130+
initializations and should no longer require the network access if the cache is
131+
hydrated. Implements
132+
[#2731](https://github.com/bazel-contrib/rules_python/issues/2731).
128133
* (wheel) Specifying a path ending in `/` as a destination in `data_files`
129134
will now install file(s) to a folder, preserving their basename.
135+
* Various attributes and fields added to support venvs on Windows:
136+
* {obj}`py_runtime.venv_bin_files` and {obj}`PyRuntime.venv_binfiles`
137+
field added to specify additional Python runtime files Windows needs for
138+
venvs.
139+
* {obj}`PyExecutableInfo.venv_interpreter_runfiles`, and
140+
{obj}`PyExecutableInfo.venv_interpreter_symlinks` adde
130141

131142
{#v1-9-0}
132143
## [1.9.0] - 2026-02-21

python/private/hermetic_runtime_repo_setup.bzl

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -240,6 +240,18 @@ def define_hermetic_runtime_toolchain_impl(
240240
_IS_FREETHREADED_YES: "cpython-{major}{minor}t".format(**version_dict),
241241
_IS_FREETHREADED_NO: "cpython-{major}{minor}".format(**version_dict),
242242
}),
243+
# On Windows, a symlink-style venv requires supporting .dll files.
244+
venv_bin_files = select({
245+
"@platforms//os:windows": native.glob(
246+
include = [
247+
"*.dll",
248+
],
249+
# This must be true because glob empty-ness is checked
250+
# during loading phase, before select() filters it out.
251+
allow_empty = True,
252+
),
253+
"//conditions:default": [],
254+
}),
243255
)
244256

245257
py_runtime_pair(

python/private/local_runtime_repo.bzl

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -74,19 +74,19 @@ def _norm_path(path):
7474
def _symlink_libraries(rctx, logger, libraries, shlib_suffix):
7575
"""Symlinks the shared libraries into the lib/ directory.
7676
77+
Individual files are symlinked instead of the whole directory because
78+
shared_lib_dirs contains multiple search paths for the shared libraries,
79+
and the python files may be missing from any of those directories, and
80+
any of those directories may include non-python runtime libraries,
81+
as would be the case if LIBDIR were, for example, /usr/lib.
82+
7783
Args:
7884
rctx: A repository_ctx object
7985
logger: A repo_utils.logger object
8086
libraries: paths to libraries to attempt to symlink.
8187
shlib_suffix: Optional. Ensure that the generated symlinks end with this suffix.
8288
Returns:
8389
A list of library paths (under lib/) linked by the action.
84-
85-
Individual files are symlinked instead of the whole directory because
86-
shared_lib_dirs contains multiple search paths for the shared libraries,
87-
and the python files may be missing from any of those directories, and
88-
any of those directories may include non-python runtime libraries,
89-
as would be the case if LIBDIR were, for example, /usr/lib.
9090
"""
9191
result = []
9292
for source in libraries:

python/private/local_runtime_repo_setup.bzl

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -153,6 +153,19 @@ def define_local_runtime_toolchain_impl(
153153
implementation_name = implementation_name,
154154
abi_flags = abi_flags,
155155
pyc_tag = "{}-{}{}{}".format(implementation_name, major, minor, abi_flags),
156+
venv_bin_files = select({
157+
"@platforms//os:windows": native.glob(
158+
include = [
159+
"lib/*.dll",
160+
# The pdb files just provide debugging information
161+
"lib/*.pdb",
162+
],
163+
# This must be true because glob empty-ness is checked
164+
# during loading phase, before select() filters it out.
165+
allow_empty = True,
166+
),
167+
"//conditions:default": [],
168+
}),
156169
)
157170

158171
py_runtime_pair(

0 commit comments

Comments
 (0)