Feed Handler

Feed handlers are launched using dedicated applications designed to support specific market data protocols. For example, use the -a cmemdp3 application is used to handle the CME MDP3 feed.

Each feed handler accepts an XML configuration file that defines all necessary runtime parameters.

Feed handlers can operate in two modes:

Live mode: Consumes data directly from the network interface.
Replay mode: Replays previously recorded data from one or more PCAP files.

Feed Node

<Feed> node must be the root node of the XML file that is supplied to a feed handler. It is responsible for global Feed Handler configuration.

Configuration Options

Option	Description	Notes
`entitlement`	Specifies Entitlement Code	Only one Feed Handler per entitlement is allowed on the same machine
`logger`	Specifies Desired Name of the Log File	Logs will be appended to the same file, if the file exists
`logger-cpu`	Specifies Desired CPU for Logger Thread	By default, Logger thread is not pinned, use this option to pin it
`logger-level`	Specifies Desired Log Level	Default level is `info`, possible levels: `error`,`warn`,`info`,`debug`,`trace`
`efvi-enabled`	Enable Solarflare EFVI	EFVI mode can be enabled for best latency. Default: `false`
`efvi-hw-timestamps`	Enable EFVI HW Timestamps	`receivetime` field is populated with EFVI HW Timestamps. Default: `false`
`process-orders`	Enable individual orders pricessing	Default: `false`

Example:

<Feed logger="myLoggerFile.log" logger-cpu="1" logger-level="debug" entitlement="XCME" efvi-enabled="true">

Threading

Threading of the feed handler is dynamic and based on XML configuration. Here is the list of threads types.

Threading Diagram

stateDiagram-v2
    NIC --> Ingester
    Ingester --> Highway_1
    Ingester --> Highway_2

    state Highway_1 {
        Lane_1
        Lane_2
        Lane_3
    }

    state Highway_2 {
        Lane_4
        Lane_5
        Lane_6
    }


    Highway_1 --> SHARED
    Highway_2 --> SHARED

    SHARED: Shared Memory

Ingester

The primary purpose of the <Ingester> thread is to offload administrative and high-overhead tasks from the processing threads.

By default, the Ingester thread receives packets from the NIC and delegates them to the appropriate processing threads.

When EFVI mode is enabled, the Ingester is also responsible for refilling RX buffers on the NIC.

Each configuration must include at least one Ingester thread.

Use the cpu attribute to pin the Ingester thread to a dedicated core for optimal performance.

PCAP Replay

An <Ingester> can be configured to use one or more PCAP files instead of live data from a NIC. To enable this, provide one or more <PCAP> nodes or set pcap-file attribute with location(s) of the PCAP file(s).

By default, PCAP(s) will be replayed at maximum speed. To enable real-time replay, set pcap-speed="realtime"

Note

When multiple PCAP files are configured, they will be synchronized and replayed based on their timestamps.

Example configuration for replaying multiple PCAP files:

<Ingester cpu="1" pcap-speed="realtime">
      <PCAP file="/location/pcapfile-1.pcap"/>
      <PCAP file="/location/pcapfile-2.pcap"/>
      <PCAP file="/location/pcapfile-3.pcap"/>

For convenience, a single PCAP file can be set directly as an attribute of the <Ingester> node:

<Ingester cpu="1" pcap-file="location/myrecording.pcap" pcap-speed="realtime">

Highway

The <Highway> element defines a market data parsing thread. It is responsible for normalizing data from the exchange protocol and writing updates into shared memory.

Each <Ingester> must contain at least one <Highway>, but multiple <Highway> threads can be configured per ingester.

Use the cpu attribute to pin the Highway thread to a dedicated core for optimal performance.

Each Highway can consume data from its own network interfaces. However, the interfaces must remain consistent within a given Highway. Use intf-a and intf-b to specify the A and B side interfaces, respectively.

Example:

<Highway cpu="2" intf-a="eth0" intf-b="eth1">

Lane

The <Lane> element defines a logical pair of IPs representing redundant multicast channels. Packets received on a single Lane are arbitrated and de-duplicated.

All Lane-related processing is executed on the parent <Highway> thread.

Multiple Lanes can be grouped using the group attribute.

Configuration Options:

Option	Description	Notes
`mc-a`	Specifies A Side IP:PORT channel	The multicast channel will be joined on the `intf-a` NIC from Highway
`mc-b`	Specifies B Side IP:PORT channel	The multicast channel will be joined on the `intf-b` NIC from Highway
`type`	Specifies Type of the MC Channel	`inc` - incremental, `snap` - snapshot, `def` - product definition
`group`	Specifies Group of the MC Channel	Example: CME Group Id, which must match CME official channel id, Ex `310` for `ES`
`name`	Specifies Name of the MC Channel	Can be set arbitrarily, useful when investigating logs

Feed Handler Subscriptions

Feed Handler requires a list of subscriptions to be provided in the XML via <Instruments> of <SpreadGroup> Nodes. symbol can contain the final instrument name (ex: ESM5) or an asset/underlying name (ex: ES - will subscribe to all ES futures expirations). Spread Subscriptions are subscribed via the pattern setting.

Example:

  <Instruments>
    <Instrument symbol="ESM5" />
    <Instrument symbol="MESM5" />
    <Instrument symbol="MNQM5" />
    <Instrument symbol="NQM5" />

    <SpreadGroup pattern="SR3:AB" />

  </Instruments>

Entitlements

Entitlements are controlled via <Entitlements> Node.

Note

Passwords in XML are stored as hashed values, which can be obtained via Nanoconda CLI hashstring application. Example: ./nanoconda-cli -a hashstring -i myPassword

Example:

<Entitlements>
   <User username="username1" passwordhash="5411718394350379800" />
   <User username="username2" passwordhash="5421711394150379800" />
</Entitlements>

Please refer to the dedicated feed pages, such as CME MDP3, for examples of complete feed handler configurations.