Introduction

In this book, we explore mid-price prediction in financial markets through the combined lens of statistical filtering and machine learning. The mid-price—halfway between the best bid and best ask—captures the evolving consensus of market participants and serves as a natural target for short-term price forecasting.

We begin by implementing a Kalman Filter as a statistical baseline for sequential state estimation. From there, we train and evaluate a range of machine learning models to assess how modern approaches compare with classical inference methods.

Our goals are two-fold:

• Implement classical inference algorithms such as the Kalman Filter in C++ for efficiency and precision, with Python bindings for experimentation. Python bindings will be provided as well.
• Compare these algorithms against machine learning models in terms of predictive accuracy, robustness, and computational performance.

The comparison will be carried out on classical mid-price forecasting datasets, including:

• FI-2010: a publicly available benchmark dataset for mid-price forecasting for limit order book data
• LOBster: a real limit order book dataset with millisecond-level resolution

By the end, we should have a practical understanding of how statistical filters and machine learning can be applied to mid-price prediction.

Model formulation

We consider a linear dynamical system with additive Gaussian noise:

$$x_k=A{x}_{k-1}+B{u}_k+w_k,w_k\sim \mathcal{N}(0,Q)$$ $$z_k=H{x}_k+v_k,v_k\sim \mathcal{N}(0,R)$$

where:

• $x_k\in {\mathbb{R}}^n$ is the state vector at time step $k$,
• $u_k\in {\mathbb{R}}^m$ is an optional control input,
• $z_k\in {\mathbb{R}}^p$ is the measurement vector,
• $A\in {\mathbb{R}}^{n\times n}$ is the state transition matrix,
• $B\in {\mathbb{R}}^{n\times m}$ is the control-input matrix,
• $H\in {\mathbb{R}}^{p\times n}$ is the observation matrix,
• $Q\in {\mathbb{R}}^{n\times n}$ is the process noise covariance,
• $R\in {\mathbb{R}}^{p\times p}$ is the measurement noise covariance.

Kalman filtering

The Kalman filter maintains the mean and covariance of the posterior distribution $p(x_k\mid z_{1:k})$ under the Gaussian assumption.

Prediction step

Given the previous posterior $({\hat{x}}_{k-1},P_{k-1})$:

$${\hat{x}}_k^{-}=A{\hat{x}}_{k-1}+B{u}_k$$ $$P_k^{-}=A{P}_{k-1}{A}^{\top }+Q$$

Here, $({\hat{x}}_k^{-},P_k^{-})$ are the predicted state mean and covariance.

Update step

With a new measurement $z_k$:

• Innovation (measurement residual): $$y_k=z_k-H{\hat{x}}_k^{-}$$
• Innovation covariance: $$S_k=H{P}_k^{-}{H}^{\top }+R$$
• Kalman gain: $$K_k=P_k^{-}{H}^{\top }{S}_k^{-1}$$
• Updated mean and covariance: $${\hat{x}}_k={\hat{x}}_k^{-}+K_k{y}_k$$ $$P_k=(I-K_{k}H)P_k^{-}(I-K_{k}H{)}^{\top }+K_{k}R{K}_k^{\top }$$

The filter proceeds recursively for each time step.