Using Daily PCAPs for Nanoconda Trade Simulator¶

The Nanoconda Trade Simulator enables you to replay historical trading days at real-time speed, enabling you to practice launching algos, monitor their behavior, and review performance on real tick data - all without the need for live market licenses.

This workflow is especially useful for:

• Practicing strategies that require human operators.
• Training and onboarding new team members.
• Demonstrating trading systems to investors or stakeholders.

Download and Merge Databento PCAPs¶

The following guide explains how to download and merge PCAP files from the previous trading day's market open:

1. Set Up Your Environment¶

Make sure you have the following installed:

• awscli to download PCAP files from your S3 bucket.
• zstd to decompress .zst files.
• mergecap from Wireshark suite to merge multiple PCAPs into one.

Install any missing packages:

sudo dnf install awscli zstd wireshark-cli

sudo apt install awscli zstd tshark

sudo apt install awscli zstd wireshark-common

sudo pacman -S aws-cli zstd wireshark-cli

2. Automation Script¶

#!/bin/bash

yesterday=$(date -d "yesterday" +"%Y%m%d")
base_dir="/home/user/databento/$yesterday/startday"
recording_dir="/home/user/recordings"

mkdir -p "$base_dir"
aws s3 cp s3://yourbucket/databento/$yesterday/ "$base_dir" --recursive

cd "$base_dir"

cp ../dc3-cme-merged-${yesterday}T14*.pcap.zst .
cp ../dc3-cme-merged-${yesterday}T15*.pcap.zst .
cp ../dc3-cme-merged-${yesterday}T16*.pcap.zst .

for f in *.pcap.zst; do
    zstd -d "$f" -o "${f%.zst}"
done

mergecap -w merged_output.pcap $(ls -1v *.pcap)

mv merged_output.pcap "$recording_dir/${yesterday}-databento.pcap"

ln -sf "$recording_dir/${yesterday}-databento.pcap" "$recording_dir/lastday.pcap"

rm *.pcap

Script Notes:

1. Set Variables: Defines ($base_dir) and output recordings directory ($recording_dir) based on yesterday's date.
2. Download PCAPs from S3: Downloads previous day PCAPs into your local staging directory using aws s3 cp --recursive.
3. Copy Relevant Files: Selects PCAPs covering the market open hours (14:00 - 16:59 UTC) for a focused simulation session.
4. Decompress Files: Decompresses all .pcap.zst files into standard .pcap format using zstd.
5. Merge PCAPs: Combines files into a single merged_output.pcap using mergecap, ensuring correct order with ls -1v.
6. Move and Link for Simulator: Moves the final PCAP and updates the lastday.pcap symlink, which is used by Nanoconda xml.
7. Cleanup: Deletes temporary .pcap files from the staging folder to keep the environment clean.

Now your /home/user/recordings/lastday.pcap is always pointing to the previous day ready recording.

02 09 * * 2-6 /home/user/databento/pullYesterdayPcap.sh

Configure Nanoconda Feed Handler¶

Point the Nanoconda Feed Handler to the prepared PCAP file by using the pcap-file option inside your <Ingester> configuration.

Example configuration:

<Ingester cpu="1" pcap-file="/home/user/recordings/lastday.pcap" pcap-speed="realtime">
    <Highway cpu="2" intf-a="lo" intf-b="lo">
        <Lane mc-a="224.0.31.1:14310" mc-b="224.0.32.1:15310" type="inc" name="l1i" group="310"/>
        <Lane mc-a="224.0.31.5:14314" mc-b="224.0.32.5:15314" type="inc" name="l4i" group="314"/>
        <Lane mc-a="224.0.31.9:14318" mc-b="224.0.32.9:15318" type="inc" name="l8i" group="318"/>
    </Highway>
</Ingester>

Notes:

• pcap-file points to /home/user/recordings/lastday.pcap.
• pcap-speed="realtime" ensures the PCAP is replayed at original market speed.
• Once configured, launching the Feed Handler will simulate the selected trading session as if it were live.

Launch the Feed Handler with:

nanoconda-cli -a cmemdp3 -f config/CME-mdp3-prod-pcap-310-314-318-inc-only.xml