VER: Release 0.24.0

See release notes.

Author: cjdsellers

.github/workflows/test.yml

@@ -10,7 +10,7 @@ jobs:
       fail-fast: false
       matrix:
         os: [ubuntu-latest, macos-latest, windows-latest]
-        python-version: ["3.8", "3.9", "3.10", "3.11"]
+        python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
     name: build - Python ${{ matrix.python-version }} (${{ matrix.os }})
     runs-on: ${{ matrix.os }}
 

CHANGELOG.md

@@ -1,5 +1,41 @@
 # Changelog
 
+## 0.24.0 - 2023-11-23
+
+This release adds support for DBN v2 as well as Python v3.12.
+
+DBN v2 delivers improvements to the `Metadata` header symbology, new `stype_in` and `stype_out` fields for `SymbolMappingMsg`, and extends the symbol field length for `SymbolMappingMsg` and `InstrumentDefMsg`. The entire change notes are available [here](https://github.com/databento/dbn/releases/tag/v0.14.0). Users who wish to convert DBN v1 files to v2 can use the `dbn-cli` tool available in the [databento-dbn](https://github.com/databento/dbn/) crate. On a future date, the Databento live and historical APIs will stop serving DBN v1.
+
+This release of `databento-python` is fully compatible with both DBN v1 and v2, so this upgrade should be seamless for most users.
+
+In some cases, DBN v1 records will be converted to their v2 counterparts:
+- When iterating a `DBNStore` and with `DBNStore.replay`
+- When iterating a `Live` client and records dispatched to callbacks
+
+#### Enhancements
+- Added support for Python 3.12
+- Improved the performance for stream writes in the `Live` client
+- Upgraded `databento-dbn` to 0.14.2
+- Added `databento.common.types` module to hold common type annotations
+
+#### Bug fixes
+- Fixed an issue where specifying an OHLCV schema in `DBNStore.to_ndarray` or `DBNStore.to_df` would not properly filter records by their interval
+- Fixed an issue where `DBNStore.to_ndarray` and `DBNStore.to_df` with a non-zero count could get stuck in a loop if the DBN data did not contain any records
+
+#### Breaking Changes
+- `DBNStore` iteration and `DBNStore.replay` will upgrade DBN version 1 messages to version 2
+- `Live` client iteration and callbacks upgrade DBN version 1 messages to version 2
+- Moved `DBNRecord`, `RecordCallback`, and `ExceptionCallback` types to them `databento.common.types` module
+- Moved `AUTH_TIMEOUT_SECONDS` and `CONNECT_TIMEOUT_SECONDS` constants from the `databento.live` module to `databento.live.session`
+- Moved `INT64_NULL` from the `databento.common.dbnstore` module to `databento.common.constants`
+- Moved `SCHEMA_STRUCT_MAP` from the `databento.common.data` module to `databento.common.constants`
+- Removed `schema` parameter from `DataFrameIterator` constructor, `struct_type` is to be used instead
+- Removed `NON_SCHEMA_RECORD_TYPES` constant as it is no longer used
+- Removed `DERIV_SCHEMAS` constant as it is no longer used
+- Removed `SCHEMA_COLUMNS` constant as it is no longer used
+- Removed `SCHEMA_DTYPES_MAP` constant as it is no longer used
+- Removed empty `databento.common.data` module
+
 ## 0.23.1 - 2023-11-10
 
 #### Enhancements

README.md

@@ -32,7 +32,7 @@ The library is fully compatible with the latest distribution of Anaconda 3.8 and
 The minimum dependencies as found in the `pyproject.toml` are also listed below:
 - python = "^3.8"
 - aiohttp = "^3.8.3"
-- databento-dbn = "0.13.0"
+- databento-dbn = "0.14.2"
 - numpy= ">=1.23.5"
 - pandas = ">=1.5.3"
 - requests = ">=2.24.0"

databento/__init__.py

@@ -37,9 +37,9 @@
 from databento.common.publishers import Publisher
 from databento.common.publishers import Venue
 from databento.common.symbology import InstrumentMap
+from databento.common.types import DBNRecord
 from databento.historical.api import API_VERSION
 from databento.historical.client import Historical
-from databento.live import DBNRecord
 from databento.live.client import Live
 from databento.version import __version__  # noqa
 

databento/common/constants.py

@@ -1 +1,59 @@
-ALL_SYMBOLS = "ALL_SYMBOLS"
+from __future__ import annotations
+
+from typing import Final
+
+import numpy as np
+from databento_dbn import ImbalanceMsg
+from databento_dbn import InstrumentDefMsg
+from databento_dbn import InstrumentDefMsgV1
+from databento_dbn import MBOMsg
+from databento_dbn import MBP1Msg
+from databento_dbn import MBP10Msg
+from databento_dbn import OHLCVMsg
+from databento_dbn import Schema
+from databento_dbn import StatMsg
+from databento_dbn import TradeMsg
+
+from databento.common.types import DBNRecord
+
+
+ALL_SYMBOLS: Final = "ALL_SYMBOLS"
+
+
+DEFINITION_TYPE_MAX_MAP: Final = {
+    x[0]: np.iinfo(x[1]).max
+    for x in InstrumentDefMsg._dtypes
+    if not isinstance(x[1], str)
+}
+
+INT64_NULL: Final = 9223372036854775807
+
+SCHEMA_STRUCT_MAP: Final[dict[Schema, type[DBNRecord]]] = {
+    Schema.DEFINITION: InstrumentDefMsg,
+    Schema.IMBALANCE: ImbalanceMsg,
+    Schema.MBO: MBOMsg,
+    Schema.MBP_1: MBP1Msg,
+    Schema.MBP_10: MBP10Msg,
+    Schema.OHLCV_1S: OHLCVMsg,
+    Schema.OHLCV_1M: OHLCVMsg,
+    Schema.OHLCV_1H: OHLCVMsg,
+    Schema.OHLCV_1D: OHLCVMsg,
+    Schema.STATISTICS: StatMsg,
+    Schema.TBBO: MBP1Msg,
+    Schema.TRADES: TradeMsg,
+}
+
+SCHEMA_STRUCT_MAP_V1: Final[dict[Schema, type[DBNRecord]]] = {
+    Schema.DEFINITION: InstrumentDefMsgV1,
+    Schema.IMBALANCE: ImbalanceMsg,
+    Schema.MBO: MBOMsg,
+    Schema.MBP_1: MBP1Msg,
+    Schema.MBP_10: MBP10Msg,
+    Schema.OHLCV_1S: OHLCVMsg,
+    Schema.OHLCV_1M: OHLCVMsg,
+    Schema.OHLCV_1H: OHLCVMsg,
+    Schema.OHLCV_1D: OHLCVMsg,
+    Schema.STATISTICS: StatMsg,
+    Schema.TBBO: MBP1Msg,
+    Schema.TRADES: TradeMsg,
+}