Skip to content

add documentation for pydefix in virtualenv #365

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
glesur wants to merge 2 commits into
base: develop
Choose a base branch
from
Open

add documentation for pydefix in virtualenv #365

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c588d0d
add documentation for pydefix in virtualenv
glesur Mar 10, 2026
723c5a3
update doc following PR discussion
glesur Mar 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions doc/source/modules/pydefix.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,9 @@ The python script `pydefix_example.py` initializes the initial condition of the
Troubleshooting
---------------

Pybind11 not found by cmake
++++++++++++++++++++++++++++

It during configuration stage, you get::

CMake Error at CMakeLists.txt:122 (find_package):
Expand All @@ -124,3 +127,19 @@ You can then exit the interpreter and set the pybind11_DIR environement variable
export pybind11_DIR=env/lib/python3.10/site-packages/pybind11

you can then run cmake which should be able to find pybind11, and compile the code.

Using pydefix in a virtual environment
+++++++++++++++++++++++++++++++++++++++
If you use pydefix in a python virtual environment (venv), pydefix might not detect your environment and fail to find pybind11 or other
python packages in your venv. This typically results in a python error "xxxx module not found" when you run Idefix with pydefix.

The fix is to force the following environement variables (replace XX by your python version) before launching Idefix:

```bash
export PYTHONPATH=$VIRTUAL_ENV/lib/python3.XX/site-packages:$PYTHONPATH
Copy link
Copy Markdown
Collaborator

@neutrinoceros neutrinoceros Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has much more consequences and can easily create a worse situation for a user, especially if they see the export as a suggestion to hardcode this line into a .bashrc or similar config file.
I'd strongly advise against this.

Copy link
Copy Markdown
Contributor Author

@glesur glesur Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see where it says this should be in .bashrc. My question would be: do you have a better proposition?

Copy link
Copy Markdown
Collaborator

@neutrinoceros neutrinoceros Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to be more constructive, I would suggest making pydefix a proper Python package if it makes sense (and it doesn't, there is a bigger problem to address here)

Copy link
Copy Markdown
Collaborator

@neutrinoceros neutrinoceros Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see where it says this should be in .bashrc.

I'm just saying this is typically where export statements from documentation / modifications to PYTHONPATH end up, and it's never1 the right solution.

Footnotes

  1. from the limited scope of my own experience ↩

Copy link
Copy Markdown
Contributor Author

@glesur glesur Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the problem is not due to pydefix (a c++ class in idefix), but to pybind11, which is already a python package. On Idefix startup, pybind11 does not necessarily pick up the python interpreter from the current venv.

Copy link
Copy Markdown
Collaborator

@svalat svalat Mar 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are the commands which are failing to reproduce with Idefix ?

I wanted to make a ticket for pybind11 but when trying to make a basic reproducer outside for them it looks to work on my setup ?

In the mean time, what I would export is the exact value taken by the python command :

export PYTHONPATH=$(python3 -c "import sys; print(':'.join(x for x in sys.path if x))")

Copy link
Copy Markdown
Collaborator

@neutrinoceros neutrinoceros Mar 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also python3 isn't defined in all envs. I suspect at least some users might follow the documentation to the literal letter and turn out even more confused as a result.
It's becoming abundantly clear to me that trying to work around the problem may be more work than fixing it upstream.

Copy link
Copy Markdown
Collaborator

@svalat svalat Mar 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree.
What is the way to reproduce the problem with Idefix before I make a detailed ticket to pybind11 ?

Copy link
Copy Markdown
Contributor Author

@glesur glesur Mar 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added a patch to #367 in order to detect the virtual env at the cmake stage and hard code it in the executable, so that PYTHONPATH is not needed on startup. I'll keep you posted if this works.

Copy link
Copy Markdown
Contributor Author

@glesur glesur Mar 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Steps to reproduce:
in test/IO/pydefix
create a venv with python_requirements.txt
configure, compile and launch idefix. On my side, it uses the system's python, and fail to find matplotlib (required by the test), because pybind11 doesn't use the venv

```

.. warning::
This is a dirty patch to fix an issue actually caused by pybind11. A better solution would be to fix pybind11 to properly detect virtual environments.
Note that you *should never* set the PYTHONPATH variable globally in your system (i.e. in .bashrc), but only in the terminal where you launch Idefix with pydefix.
The reader may check the dedicated `pydefix PR for details <https://github.com/idefix/idefix/pull/365>`_.
Loading