Why Is Your Futures Backtest So Slow?

Why Is Your Futures Backtest So Slow?

A futures backtest has to process recorded market updates, maintain the order book and simulate the strategy's orders. Across months of data and dozens of parameter combinations, that work can become a bottleneck, even at maximum replay speed. Data conversion, logging and communication between applications can add to the processing time.

That can make every strategy change time-consuming: adjust a parameter, rerun the history, then wait to see whether it helped. Nanoconda's Simulator includes a simpler setup for the research stage. The strategy runs inside the backtester as it reads the historical files. For final testing before production, run the feed publisher and simulator as separate processes to check risk limits, order behavior, and communication between components.

What makes futures backtesting slow?

Detailed futures data records individual changes to the order book. To replay it, the system reads and decodes those updates in sequence, maintains the book and calls the strategy when relevant events arrive. When the strategy submits an order, the simulator also has to determine how it behaves against that book, including estimated queue priority and latency.

A recent preprint reports about 2.86 million E-mini S&P 500 (ES) book-update messages in a single 53-minute capture. Those are market-data messages, not traded contracts. Extend that to months of history, then multiply it by dozens of parameter variations, and millions of updates can translate into billions of events to process across runs.

PCAP files contain captured exchange packets that must be decoded. Databento DBN files, such as .mbo.dbn.zst recordings, contain compressed, normalized market-by-order records. Compression saves storage, but the records still need to be decompressed and processed during replay.

A simpler setup for research

The Nanoconda Simulator package includes two command-line backtesters: dbnbacktester for Databento DBN files and cmebacktester for CME PCAP recordings.

Each reads the historical data, runs your compiled strategy plugin and simulates orders in one process. Market updates and order messages stay in memory, without passing between a separate market-data publisher, strategy application and simulator. These backtesters run without a GUI, so the test avoids the overhead of refreshing charts or trading screens.

With dbn-speed="fast", the DBN backtester processes the recording as quickly as the machine and strategy allow. PCAP replay runs at maximum speed by default. Real-time pacing is an option you can enable when needed. Multiple PCAP files can be replayed together, synchronized by their timestamps.

The backtester still has to process every relevant market update and run your strategy’s calculations, so large datasets and complex strategies take longer. Use it to test parameter changes, compare strategy versions, and check that code changes haven’t broken existing behavior.

The backtesters use an estimated exchange delay for simulated orders, configurable with -L up to 1 ms. They do not apply pre-trade risk limits. Those limits need to be tested in the separate simulator application.

Rehearse the final candidates in the production setup

After selecting a strategy and its parameters, test it with the components running separately, communicating through shared memory, and the GUI attached.

For DBN replay, dbnpublisher reads the historical data and publishes it through shared memory. The simulator runs separately and models matching using Globex-style rules. With dbn-speed="realtime", replay is paced according to the recorded timestamps so you can watch the book, orders and fills in the GUI.

The separate simulator lets you test pre-trade risk limits and use market-data-based latency modeling, where the simulated delay varies with the exchange timestamps in the data. This differs from the backtesters' configurable exchange-delay estimate.

You can then check communication between the components, check risk-limit behavior and inspect simulated orders and fills in the GUI. Enable real-time pacing when you want to watch the session at its recorded speed. Use this setup for the final candidates after completing the parameter sweep in the inline backtester.

Your strategy is compiled C++ either way. The backtesters load it as a shared library (.so) with an init_nanoconda_plugin entry point. For the simulator and live trading you can also build it as a standalone application that calls registerApplication and start. Both use the same C++ API as the live environment, allowing strategy logic to be reused across backtesting, simulation and live trading. To try it with your own strategy, follow the Nanoconda Simulator setup guide.

Read more