Wingfoil

Wingfoil is a blazingly fast, highly scalable stream processing framework designed for latency-critical use cases such as electronic trading and real-time AI systems.

Wingfoil simplifies receiving, processing and distributing streaming data across your entire stack.

Features

• Fast: Ultra-low latency and high throughput with an efficient DAG based execution engine.
• Simple and obvious to use: Define your graph of calculations; Wingfoil manages its execution.
• Multi-language: currently available as a Rust crate and as a beta release, python package with plans to add WASM/JavaScript/TypeScript support.
• Backtesting: Replay historical data to backtest and optimise strategies.
• Async/Tokio: seamless integration, allows you to leverage async at your graph edges.
• Multi-threading: distribute graph execution across cores.
• I/O Adapters: production-ready KDB+ integration for tick data, CSV, etcd key-value store, Kafka (via rdkafka), Fluvio distributed streaming, FIX protocol (FIX 4.4 with TLS), ZeroMQ pub/sub messaging (beta), Prometheus metrics exporter, OpenTelemetry OTLP push, etc.

Quick Start

In this example we build a simple, linear pipeline with all nodes ticking in lock-step.

use wingfoil::*;
use std::time::Duration;
fn main() {
    let period = Duration::from_secs(1);
    ticker(period)
        .count()
        .map(|i| format!("hello, world {:}", i))
        .print()
        .run(RunMode::RealTime, RunFor::Duration(period*3)
    );
}

This output is produced:

hello, world 1
hello, world 2
hello, world 3

Order Book Example

Wingfoil lets you easily wire up complex business logic, splitting and recombining streams, and altering the frequency of data. I/O adapters make it easy to plug in real data sources and sinks. In this example we load a CSV of AAPL limit orders, maintain an order book using the lobster crate, derive trades and two-way prices, and export back to CSV — all in a few lines:

let book = RefCell::new(lobster::OrderBook::default());
let get_time = |msg: &Message| NanoTime::new((msg.seconds * 1e9) as u64);
let (fills, prices) = csv_read("aapl.csv", get_time, true)
    .map(move |chunk| process_orders(chunk, &book))
    .split();
let prices_export = prices
    .filter_value(|price: &Option<TwoWayPrice>| !price.is_none())
    .map(|price| price.unwrap())
    .distinct()
    .csv_write("prices.csv");
let fills_export = fills.csv_write("fills.csv");
Graph::new(vec![prices_export, fills_export], RunMode::HistoricalFrom(NanoTime::ZERO), RunFor::Forever)
    .print()
    .run()
    .unwrap();

This output is produced:

Full example.

More Examples

Short code snippets for each adapter live in the examples README. The examples below are all runnable — see each one's README.md for setup and commands.

Core concepts

Example	Description
order_book	Load NASDAQ AAPL limit orders from CSV, maintain an order book, derive trades and two-way prices, export to CSV.
breadth_first	Why wingfoil's BFS execution avoids the O(2^N) node explosion of naive depth-first DAGs.
run_mode	Swap RunMode::RealTime and RunMode::HistoricalFrom with the same graph wiring for backtesting.
async	Integrate Tokio async/await at graph edges (I/O adapters) while keeping the core graph synchronous.
threading	Distribute graph execution across worker threads with producer() / mapper().
dynamic	Add and remove nodes at runtime. Includes demux, dynamic-group, and dynamic-manual variants.
tracing	Instrumentation modes (log, tracing, instruments) for event and span handling.
latency	Per-hop latency stamping with Traced<T, L> and LatencyReport, transported over iceoryx2.

I/O adapters

Example	Description
kdb	KDB+ integration: time-sliced reads, cached reads (LRU file cache), and round-trip write/read/validate.
kafka	Kafka / Redpanda adapter — subscribe, transform, publish pipeline via rdkafka.
fluvio	Fluvio distributed streaming — subscribe, transform, publish pipeline.
fix	FIX 4.4 protocol: self-contained loopback, client, echo server, and live LMAX market data over TLS.
zmq	ZeroMQ pub/sub with direct addressing or etcd-based service discovery.
etcd	etcd key-value store adapter for sub/pub with transformation.
iceoryx2	Zero-copy IPC over shared memory (spin, threaded, signaled polling modes).
web	WebSocket adapter streaming synthetic prices and receiving UI events.
telemetry	Metrics export via Prometheus scraping (pull) and OpenTelemetry OTLP (push).

Links

• Checkout the examples
• Download from crates.io
• Read the documentation
• Review the benchmarks
• Download the wingfoil Python module from pypi.org

Get Involved!

We want to hear from you! Especially if you:

• are interested in contributing
• know of a project that wingfoil would be well-suited for
• would like to request a feature or report a bug
• have any feedback

Please do get in touch:

• ping us on discord
• email us at hello@wingfoil.io
• submit an issue
• get involved in the discussion