Skip to content

Commit 912d120

Browse files
Merge pull request #613 from Blosc/opsi-indexes
This provides support for indexing 1D NDArray objects. The API follows the existing OPSI implementation for PyTables, although implementation differs considerable from it. Light indexes are specially useful because they can be created relatively fast, consume little space, and lookups are still pretty fast; this is the default mode for indexing.
2 parents 291cafc + 9856f2f commit 912d120

28 files changed

Lines changed: 15216 additions & 48 deletions

AGENTS.md

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# Repository Guidelines
2+
3+
## Project Structure & Module Organization
4+
The Python package lives in `src/blosc2/`, including the C/Cython extension sources
5+
(`blosc2_ext.*`) and core modules such as `core.py`, `ndarray.py`, and `schunk.py`.
6+
Tests are under `tests/`, with additional doctests enabled for select modules per
7+
`pytest.ini`. Documentation sources are in `doc/` and build output lands in `html/`.
8+
Examples are in `examples/`, and performance/benchmark scripts live in `bench/`.
9+
10+
## Build, Test, and Development Commands
11+
- `pip install .` builds the bundled C-Blosc2 and installs the package.
12+
- `pip install -e .` installs in editable mode for local development.
13+
- `CMAKE_PREFIX_PATH=/usr/local USE_SYSTEM_BLOSC2=1 pip install -e .` builds
14+
against a separately installed C-Blosc2.
15+
- `pytest` runs the default test suite (excludes `heavy` and `network` markers).
16+
- `pytest -m "heavy"` runs long-running tests.
17+
- `pytest -m "network"` runs tests requiring network access.
18+
- `cd doc && rm -rf ../html _build && python -m sphinx . ../html` builds docs.
19+
20+
## Coding Style & Naming Conventions
21+
Use Ruff for formatting and linting (line length 109). Enable pre-commit hooks:
22+
`python -m pip install pre-commit` then `pre-commit install`. Follow Python
23+
conventions: 4-space indentation, `snake_case` for functions/variables, and
24+
`PascalCase` for classes. Pytest discovery expects `tests/test_*.py` and
25+
`test_*` functions. Do not use leading underscores in module-level helper
26+
function names when those helpers are imported from other modules; reserve
27+
leading underscores for file-local implementation details. Avoid leading
28+
underscores in core module filenames under `src/blosc2/`; prefer non-underscored
29+
module names unless there is a strong reason to keep a module private.
30+
31+
For documentation and tutorial query examples, prefer the shortest idiom that
32+
matches the intended result type. Use `expr[:]` or `arr[mask][:]` when showing
33+
values, use `expr.compute()` when materializing an `NDArray`, and use
34+
`expr.compute(_use_index=False)` when demonstrating scan-vs-index behavior.
35+
Avoid `expr.compute()[:]` unless a NumPy array is specifically required.
36+
37+
## Testing Guidelines
38+
Pytest is required; warnings are treated as errors. The default configuration
39+
adds `--doctest-modules`, so keep doctest examples in `blosc2/core.py`,
40+
`blosc2/ndarray.py`, and `blosc2/schunk.py` accurate. Use markers `heavy` and
41+
`network` for slow or network-dependent tests.
42+
43+
## Commit & Pull Request Guidelines
44+
Recent commit messages are short, imperative sentences (e.g., “Add …”, “Fix …”)
45+
without ticket prefixes. For pull requests: branch from `main`, add tests for
46+
behavior changes, update docs for API changes, ensure the test suite passes,
47+
and avoid introducing new compiler warnings. Link issues when applicable and
48+
include clear reproduction steps for bug fixes.

CMakeLists.txt

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -41,11 +41,20 @@ add_custom_command(
4141
DEPENDS "${CMAKE_CURRENT_SOURCE_DIR}/src/blosc2/blosc2_ext.pyx"
4242
VERBATIM)
4343

44+
add_custom_command(
45+
OUTPUT indexing_ext.c
46+
COMMAND Python::Interpreter -m cython
47+
"${CMAKE_CURRENT_SOURCE_DIR}/src/blosc2/indexing_ext.pyx" --output-file indexing_ext.c
48+
DEPENDS "${CMAKE_CURRENT_SOURCE_DIR}/src/blosc2/indexing_ext.pyx"
49+
VERBATIM)
50+
4451
# ...and add it to the target
4552
Python_add_library(blosc2_ext MODULE blosc2_ext.c WITH_SOABI)
53+
Python_add_library(indexing_ext MODULE indexing_ext.c WITH_SOABI)
4654

4755
# We need to link against NumPy
4856
target_link_libraries(blosc2_ext PRIVATE Python::NumPy)
57+
target_link_libraries(indexing_ext PRIVATE Python::NumPy)
4958

5059
# Fetch and build miniexpr library
5160
include(FetchContent)
@@ -63,7 +72,7 @@ endif()
6372

6473
FetchContent_Declare(miniexpr
6574
GIT_REPOSITORY https://github.com/Blosc/miniexpr.git
66-
GIT_TAG feadbc633a887bafd84b2fbc370ef2962d01b7ee
75+
GIT_TAG f2faef741c4c507bf6a03167c72ce7f92c6f0ae8
6776
# SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/../miniexpr
6877
)
6978
FetchContent_MakeAvailable(miniexpr)
@@ -72,6 +81,7 @@ FetchContent_MakeAvailable(miniexpr)
7281
target_link_libraries(blosc2_ext PRIVATE miniexpr_static)
7382

7483
target_compile_features(blosc2_ext PRIVATE c_std_11)
84+
target_compile_features(indexing_ext PRIVATE c_std_11)
7585
if(WIN32 AND CMAKE_C_COMPILER_ID STREQUAL "Clang")
7686
execute_process(
7787
COMMAND "${CMAKE_C_COMPILER}" -print-resource-dir
@@ -119,7 +129,7 @@ else()
119129
include(FetchContent)
120130
FetchContent_Declare(blosc2
121131
GIT_REPOSITORY https://github.com/Blosc/c-blosc2
122-
GIT_TAG b32256fc1287b6e24c22f09ac202265c7054e2bc
132+
GIT_TAG 0568990388e6201240b170947d4c2199572f795d
123133
# SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/../c-blosc2
124134
)
125135
FetchContent_MakeAvailable(blosc2)
@@ -148,7 +158,7 @@ endif()
148158

149159
# Python extension -> site-packages/blosc2
150160
install(
151-
TARGETS blosc2_ext
161+
TARGETS blosc2_ext indexing_ext
152162
LIBRARY DESTINATION ${SKBUILD_PLATLIB_DIR}/blosc2
153163
)
154164

0 commit comments

Comments
 (0)