Skip to content

Commit 188dec5

Browse files
benoitcbenoitc
committed
Clean up CHANGELOG and add missing documentation
- Remove duplicate v2.0.0 entries from v2.2.0 section in CHANGELOG - Add missing Performance section for nif_process_ready_tasks optimization - Add config-based initialization entry for imports/paths - Create docs/imports.md documenting the py_import module API - Add missing docs to hexdoc: buffer, process-bound-envs, preload, owngil_internals, event_loop_architecture, imports - Add new "Internals" group in ex_doc for architecture docs
1 parent 371fbe1 commit 188dec5

3 files changed

Lines changed: 202 additions & 116 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 11 additions & 115 deletions
Original file line numberDiff line numberDiff line change
@@ -61,123 +61,19 @@
6161
This simplifies the architecture by consolidating event handling into a single worker process.
6262
The `py_nif:set_shared_router/1` function has been removed.
6363

64-
### Added
65-
66-
- **Event Loop Pool** - Pool of event loops for parallel Python coroutine execution
67-
- `py_event_loop_pool:get_loop/0` - Get event loop for current process (process affinity)
68-
- `py_event_loop_pool:create_task/3,4` - Submit async task to pool
69-
- `py_event_loop_pool:run/3,4` - Blocking call via pool
70-
- `py_event_loop_pool:spawn_task/3,4` - Fire-and-forget task
71-
- `py_event_loop_pool:await/1,2` - Wait for task result
72-
- Process affinity ensures same PID always routes to same loop (ordered execution)
73-
- Uses `persistent_term` for O(1) loop access
74-
- Configurable via `{event_loop_pool_size, N}` (default: schedulers count)
75-
- Benchmarks: 417k tasks/sec (fire-and-collect), 164k tasks/sec (50 concurrent processes)
76-
77-
- **ByteChannel API** - Raw byte streaming channel without term serialization
78-
- `py_byte_channel:new/0,1` - Create byte channel (with optional backpressure)
79-
- `py_byte_channel:send/2` - Send raw bytes to Python
80-
- `py_byte_channel:recv/1,2` - Blocking receive with optional timeout
81-
- `py_byte_channel:try_receive/1` - Non-blocking receive
82-
- Python `ByteChannel` class with:
83-
- `send_bytes(data)` - Send bytes back to Erlang
84-
- `receive_bytes()` - Blocking receive (GIL released)
85-
- `try_receive_bytes()` - Non-blocking receive
86-
- `async_receive_bytes()` - Asyncio-compatible async receive
87-
- Sync and async iteration (`for chunk in ch`, `async for chunk in ch`)
88-
- Reuses the same `py_channel_t` infrastructure but skips term encoding/decoding
89-
- Suitable for HTTP bodies, file streaming, and binary protocols
90-
91-
- **Automatic Env Reuse for Event Loop Tasks** - Functions defined via `py:exec(Ctx, Code)`
92-
can now be called directly using `py_event_loop:run/3,4`, `create_task/3,4`, and `spawn_task/3,4`
93-
without manual env passing. The process-local environment is automatically detected and used
94-
for function lookup when targeting `__main__` module.
95-
96-
- **PyBuffer API** - Zero-copy WSGI input buffer for streaming HTTP bodies
97-
- `py_buffer:new/0,1` - Create buffer (chunked or with content_length)
98-
- `py_buffer:write/2` - Append data, signals waiting Python readers
99-
- `py_buffer:close/1` - Signal EOF, wake all readers
100-
- Python `PyBuffer` type with file-like interface:
101-
- `read(size)`, `readline()`, `readlines()` - Blocking reads with GIL released
102-
- `read_nonblock(size)` - Non-blocking read for async I/O
103-
- `readable_amount()` - Bytes available without blocking
104-
- `at_eof()` - Check if at EOF with no more data
105-
- `seek(offset, whence)`, `tell()` - Position tracking
106-
- `find(sub)` - Fast substring search via memmem/memchr
107-
- `memoryview(buf)` - Zero-copy buffer protocol
108-
- `for line in buf:` - Line iteration
109-
- Auto-conversion: Passing buffer ref to `py:call`/`py:eval` wraps as `PyBuffer`
110-
- Suitable for `wsgi.input` in WSGI applications
111-
- See [Buffer API docs](docs/buffer.md)
112-
113-
- **Inline Continuation API** - High-performance scheduling without Erlang messaging
114-
- `erlang.schedule_inline(module, func, args, kwargs)` - Chain Python calls via `enif_schedule_nif()`
115-
- ~3x faster than `schedule_py` for tight loops (bypasses gen_server messaging)
116-
- Captures caller's globals/locals for correct namespace resolution with subinterpreters
117-
- `InlineScheduleMarker` type returned, must be returned from handler
118-
- See [Scheduling API docs](docs/asyncio.md#explicit-scheduling-api)
119-
120-
- **Inline Continuation Benchmark** - Performance comparison
121-
- `bench_schedule_inline` in `examples/benchmark.erl`
122-
- Compares `schedule_inline` vs `schedule_py` throughput
123-
124-
- **Process-Bound Python Environments** - Each Erlang process gets an isolated Python namespace
125-
- Variables defined via `py:exec()` persist across calls within the same Erlang process
126-
- Automatic cleanup when the Erlang process exits (no manual deallocation needed)
127-
- Resetting Python state = terminating the Erlang process (follows Erlang's "let it crash")
128-
- Enables "Python actors" - gen_server processes with encapsulated Python state
129-
- Works with both subinterpreter and worker modes
130-
- Memory-safe: environments created inside the correct interpreter's allocator
131-
- See [Process-Bound Environments](docs/process-bound-envs.md) for patterns and examples
132-
133-
- **Docker Test Configs** - Containerized test environment
134-
- `docker/Dockerfile.python312` - Python 3.12 test image
135-
- `docker/Dockerfile.python314` - Python 3.14 test image
136-
- `docker/Dockerfile.asan` - AddressSanitizer build for memory testing
137-
- `docker/docker-compose.yml` - Multi-container test orchestration
138-
- `docker/run-tests.sh` - Automated test runner script
139-
140-
- **Async Task Benchmark** - Performance testing for async operations
141-
- `examples/bench_async_task.erl` - Erlang benchmark runner
142-
- `priv/test_async_task.py` - Python async task implementation
143-
144-
- **OWN_GIL Context Mode** - True parallel Python execution (Python 3.12+)
145-
- `py_context:start_link(Id, owngil)` - Create context with dedicated pthread and GIL
146-
- Each OWN_GIL context runs in its own thread with independent Python GIL
147-
- Enables true CPU parallelism across multiple Python contexts
148-
- Full feature support: channels, buffers, callbacks, PIDs, reactor, async tasks
149-
- `py_context:get_nif_ref/1` - Get NIF reference for low-level operations
150-
- New benchmark: `examples/bench_owngil.erl` comparing SHARED_GIL vs OWN_GIL
151-
- See [OWN_GIL Internals](docs/owngil_internals.md) for architecture details
152-
153-
- **Process-Local Environments for OWN_GIL** - Namespace isolation within shared contexts
154-
- `py_context:create_local_env/1` - Create isolated Python namespace for calling process
155-
- `py_nif:context_exec(Ref, Code, Env)` - Execute with process-local environment
156-
- `py_nif:context_eval(Ref, Expr, Locals, Env)` - Evaluate with process-local environment
157-
- `py_nif:context_call(Ref, Mod, Func, Args, Kwargs, Env)` - Call with process-local environment
158-
- Multiple Erlang processes can share an OWN_GIL context with isolated namespaces
159-
- Interpreter ID validation prevents cross-interpreter env usage
160-
161-
- **Per-Process Event Loop Namespaces** - Process isolation for event loop API
162-
- `py_nif:event_loop_exec/2` - Execute code in calling process's namespace
163-
- `py_nif:event_loop_eval/2` - Evaluate expression in calling process's namespace
164-
- Functions defined via exec callable via `create_task` with `__main__` module
165-
- Automatic cleanup when Erlang process exits
166-
167-
- **OWN_GIL Test Suites** - Feature verification
168-
- `py_context_owngil_SUITE` - Core OWN_GIL functionality (15 tests)
169-
- `py_owngil_features_SUITE` - Feature integration (44 tests covering channels,
170-
buffers, callbacks, PIDs, reactor, async tasks, asyncio, local envs)
64+
- **Config-based initialization** - Import and path configuration via application environment
65+
- Configure imports: `{erlang_python, [{imports, [{json, dumps}]}]}`
66+
- Configure paths: `{erlang_python, [{paths, ["/path/to/modules"]}]}`
67+
- Applied immediately to all running interpreters
68+
- See [Imports documentation](docs/imports.md) for details
17169

172-
### Changed
173-
174-
- **Event Loop Lock Ordering** - GIL acquired before `namespaces_mutex` in cleanup paths
175-
to prevent ABBA deadlocks with normal execution path
70+
### Performance
17671

177-
- **Asyncio Compatibility** - Fixed for Python 3.12+ with subinterpreters
178-
- Thread-local event loop context in `process_ready_tasks`
179-
- Eager task execution handling for Python 3.12+
180-
- Deprecation warning fix: use `erlang.run()` instead of `erlang.install()`
72+
- **nif_process_ready_tasks optimization** - ~15% improvement in async task processing
73+
- Replace `asyncio.iscoroutine()` with `PyCoro_CheckExact` C API
74+
- Use stack buffers for module/func strings
75+
- Cache `asyncio.events` module
76+
- Pool `ErlNifEnv` allocations with mutex protection
18177

18278
## 2.1.0 (2026-03-12)
18379

‎docs/imports.md‎

Lines changed: 176 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,176 @@
1+
# Import and Path Registry
2+
3+
The `py_import` module manages a global registry for Python imports and sys.path
4+
additions that are automatically applied to all Python interpreters.
5+
6+
## Overview
7+
8+
When you call `py:call/3` or similar functions, erlang_python needs to import
9+
the Python module first. The import registry allows you to:
10+
11+
- Pre-register modules that should be imported in all interpreters
12+
- Add paths to `sys.path` in all interpreters
13+
- Configure imports and paths via application environment
14+
15+
Registered imports and paths are applied:
16+
- Immediately to all running interpreters (contexts, event loops, OWN_GIL sessions)
17+
- Automatically to any new interpreters created later
18+
19+
## Configuration
20+
21+
Configure imports and paths in your application environment:
22+
23+
```erlang
24+
%% sys.config or app.src
25+
{erlang_python, [
26+
{imports, [
27+
{json, dumps},
28+
{json, loads},
29+
{os, getcwd}
30+
]},
31+
{paths, [
32+
"/path/to/my/modules",
33+
"/another/path"
34+
]}
35+
]}
36+
```
37+
38+
## API
39+
40+
### Import Registry
41+
42+
#### `ensure_imported/1`
43+
44+
Register a module for import in all interpreters.
45+
46+
```erlang
47+
ok = py_import:ensure_imported(json).
48+
```
49+
50+
#### `ensure_imported/2`
51+
52+
Register a module/function pair for import.
53+
54+
```erlang
55+
ok = py_import:ensure_imported(json, dumps).
56+
ok = py_import:ensure_imported(json, loads).
57+
```
58+
59+
#### `is_imported/1,2`
60+
61+
Check if a module or module/function is registered.
62+
63+
```erlang
64+
true = py_import:is_imported(json).
65+
true = py_import:is_imported(json, dumps).
66+
false = py_import:is_imported(unknown_module).
67+
```
68+
69+
#### `all_imports/0`
70+
71+
Get all registered imports.
72+
73+
```erlang
74+
[{<<"json">>, all}, {<<"math">>, <<"sqrt">>}] = py_import:all_imports().
75+
```
76+
77+
#### `import_list/0`
78+
79+
Get imports as a map grouped by module.
80+
81+
```erlang
82+
{ok, #{<<"json">> => [<<"dumps">>, <<"loads">>],
83+
<<"math">> => []}} = py_import:import_list().
84+
```
85+
86+
#### `clear_imports/0`
87+
88+
Remove all registered imports. Does not affect already-running interpreters.
89+
90+
```erlang
91+
ok = py_import:clear_imports().
92+
```
93+
94+
### Path Registry
95+
96+
#### `add_path/1`
97+
98+
Add a directory to `sys.path` in all interpreters.
99+
100+
```erlang
101+
ok = py_import:add_path("/path/to/my/modules").
102+
```
103+
104+
#### `add_paths/1`
105+
106+
Add multiple directories to `sys.path`.
107+
108+
```erlang
109+
ok = py_import:add_paths(["/path/to/lib1", "/path/to/lib2"]).
110+
```
111+
112+
#### `all_paths/0`
113+
114+
Get all registered paths in insertion order.
115+
116+
```erlang
117+
[<<"/path/to/modules">>] = py_import:all_paths().
118+
```
119+
120+
#### `is_path_added/1`
121+
122+
Check if a path is registered.
123+
124+
```erlang
125+
true = py_import:is_path_added("/path/to/modules").
126+
```
127+
128+
#### `clear_paths/0`
129+
130+
Remove all registered paths. Does not affect already-running interpreters.
131+
132+
```erlang
133+
ok = py_import:clear_paths().
134+
```
135+
136+
## Examples
137+
138+
### Pre-loading Common Modules
139+
140+
```erlang
141+
%% At application startup
142+
ok = py_import:ensure_imported(json),
143+
ok = py_import:ensure_imported(os),
144+
ok = py_import:ensure_imported(datetime).
145+
146+
%% Now all py:call invocations skip the import step for these modules
147+
{ok, Json} = py:call(json, dumps, [[{key, value}]]).
148+
```
149+
150+
### Adding Custom Module Paths
151+
152+
```erlang
153+
%% Add your project's Python modules to sys.path
154+
ok = py_import:add_path("/opt/myapp/python"),
155+
ok = py_import:add_path("/opt/myapp/vendor").
156+
157+
%% Now you can import modules from these directories
158+
{ok, Result} = py:call(mymodule, myfunction, []).
159+
```
160+
161+
### Runtime Configuration
162+
163+
```erlang
164+
%% Check what's registered
165+
Imports = py_import:all_imports(),
166+
Paths = py_import:all_paths(),
167+
io:format("Registered imports: ~p~n", [Imports]),
168+
io:format("Registered paths: ~p~n", [Paths]).
169+
```
170+
171+
## Notes
172+
173+
- The `__main__` module cannot be cached (returns `{error, main_not_cacheable}`)
174+
- Clearing registries does not affect already-running interpreters
175+
- Paths are added in order, maintaining their relative priority in `sys.path`
176+
- All module and path values are normalized to binaries internally

‎rebar.config‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,17 +44,23 @@
4444
<<"docs/type-conversion.md">>,
4545
<<"docs/context-affinity.md">>,
4646
<<"docs/pools.md">>,
47+
<<"docs/imports.md">>,
4748
<<"docs/channel.md">>,
49+
<<"docs/buffer.md">>,
4850
<<"docs/streaming.md">>,
4951
<<"docs/memory.md">>,
5052
<<"docs/logging.md">>,
5153
<<"docs/scalability.md">>,
5254
<<"docs/threading.md">>,
5355
<<"docs/asyncio.md">>,
5456
<<"docs/reactor.md">>,
57+
<<"docs/process-bound-envs.md">>,
5558
<<"docs/security.md">>,
5659
<<"docs/distributed.md">>,
57-
<<"docs/testing-free-threading.md">>
60+
<<"docs/testing-free-threading.md">>,
61+
<<"docs/preload.md">>,
62+
<<"docs/owngil_internals.md">>,
63+
<<"docs/event_loop_architecture.md">>
5864
]},
5965
{groups_for_extras, [
6066
{<<"Guides">>, [
@@ -64,7 +70,9 @@
6470
<<"docs/type-conversion.md">>,
6571
<<"docs/context-affinity.md">>,
6672
<<"docs/pools.md">>,
73+
<<"docs/imports.md">>,
6774
<<"docs/channel.md">>,
75+
<<"docs/buffer.md">>,
6876
<<"docs/streaming.md">>,
6977
<<"docs/memory.md">>,
7078
<<"docs/logging.md">>
@@ -74,9 +82,15 @@
7482
<<"docs/threading.md">>,
7583
<<"docs/asyncio.md">>,
7684
<<"docs/reactor.md">>,
85+
<<"docs/process-bound-envs.md">>,
7786
<<"docs/security.md">>,
7887
<<"docs/distributed.md">>,
7988
<<"docs/testing-free-threading.md">>
89+
]},
90+
{<<"Internals">>, [
91+
<<"docs/preload.md">>,
92+
<<"docs/owngil_internals.md">>,
93+
<<"docs/event_loop_architecture.md">>
8094
]}
8195
]}
8296
]}.

0 commit comments

Comments
 (0)