RapidMarket Latency Comparison Test Solution

Overview

This solution provides three market data receivers for latency comparison testing:

• RapidMarket 1.0: Market data receiver based on Rapidmarket1.0 (ltp_rm_1_0_receiver.cpp)

• RapidMarket 2.0: Market data receiver based on Rapidmarket2.0 (ltp_rm_2_0_receiver.cpp )

• Binance WebSocket: Direct WebSocket receiver connecting to Binance exchange (binance_websocket_receiver.py )

Comparison Principle

Data matching is performed using the seq_num field to compare the receive_ts (receive timestamp) of the same market data across different receiving methods, thereby evaluating latency differences.

Important Note: seq_num Field Processing

RapidMarket 1.0 receiver's seq_num is processed twice:

• Format: channel_id * 1e18 + exchange_seq_num
• Where channel_id is the exchange channel id and exchange_seq_num is the original exchange sequence number

RapidMarket 2.0 and Binance WebSocket seq_num are original exchange sequence numbers:

• No restoration needed, use directly

Only RapidMarket 1.0 requires exchange_seq_num restoration for latency comparison testing:

Python

# Extract exchange_seq_num from RapidMarket 1.0's seq_num
def extract_exchange_seq_num(rapidmarket_1_0_seq_num):
    """Extract original exchange sequence number from RapidMarket 1.0 seq_num"""
    return rapidmarket_1_0_seq_num % int(1e18)

# Example
rapidmarket_1_0_seq_num = 2000000000000000123  # channel_id=2, exchange_seq_num=123
exchange_seq_num = extract_exchange_seq_num(rapidmarket_1_0_seq_num)  # Result: 123

File Description

1. RapidMarket 1.0 Receiver (ltp_rm_1_0_receiver.cpp)

Function: Receive market data through Rapidmarket1.0 and asynchronously write to file

Compilation:

Bash

protoc --cpp_out=. l2_book.proto
g++ -std=c++17 -Wall -O3 -I. -I./cppzmq ltp_rm_1_0_receiver.cpp l2_book.pb.cc -lzmq -lprotobuf -lpthread -o ltp_rm_1_0_receiver

Execution:

Bash

./ltp_rm_1_0_receiver <zmq_path> <main_thread_cpu_id> <print_latency_thread_cpu_id>

Example:

Bash

./ltp_rm_1_0_receiver "tcp://localhost:50000" 0 1

2. RapidMarket 2.0 Receiver (ltp_rm_2_0_receiver.cpp)

Function: Receive market data through Rapidmarket2.0 and asynchronously write to file

Compilation:

Bash

g++ -o ltp_rm_2_0_receiver ltp_rm_2_0_receiver.cpp -lpthread -L/usr/local/lib -lucli_ffi

Execution:

Bash

./ltp_rm_2_0_receiver <topic> <main_thread_cpu_id> <print_latency_thread_cpu_id>

Example:

Bash

./ltp_rm_2_0_receiver "bf_m_pub" 0 1

3. Binance WebSocket Receiver (binance_websocket_receiver.py)

Function: Directly connect to Binance exchange WebSocket API to receive bookTicker data and asynchronously write to file

Dependency Installation:

Bash

# Install Python dependencies
pip install websockets asyncio

Execution:

Bash

python3 binance_websocket_receiver.py <symbol> <main_thread_cpu_id> <print_latency_thread_cpu_id>

Example:

Bash

python3 binance_websocket_receiver.py btcusdt 0 1

Supported Trading Pairs:

• btcusdt, ethusdt, bnbusdt, adausdt, xrpusdt, etc.

Output Data Format

All three receivers generate date-named log files containing the following fields:

• RapidMarket 1.0: rm_1_0_2025-01-15.log
• RapidMarket 2.0: rm_2_0_2025-01-15.log
• Binance WebSocket: binance_2025-01-15.log

symbol,seq_num,exchange_ts,local_ts,receive_ts,bid_price,bid_qty,ask_price,ask_qty

Field Description

• symbol: Trading pair symbol
• seq_num: Sequence number (used for data matching)
• exchange_ts: Exchange timestamp
• local_ts: Local timestamp
• receive_ts: Receive timestamp (key comparison field)
• bid_price: Best bid price
• bid_qty: Best bid quantity
• ask_price: Best ask price
• ask_qty: Best ask quantity

Latency Comparison Test Steps

1. Environment Preparation

1. Ensure RapidMarket service is running normally
2. Prepare client's own WebSocket direct connection program
3. Start all market data receiving services

2. Start Testing

Start all three receivers simultaneously:

Bash

# RapidMarket 1.0
./ltp_rm_1_0_receiver "tcp://localhost:50000" 0 1 &

# RapidMarket 2.0
./ltp_rm_2_0_receiver "bf_m_pub" 0 2 &

# Binance WebSocket Direct Connection
python3 binance_websocket_receiver.py btcusdt 0 3 &

3. Data Collection

After running for a period, stop all programs and collect the following data files:

• RapidMarket 1.0: rm_1_0_2025-01-15.log
• RapidMarket 2.0: rm_2_0_2025-01-15.log
• Binance WebSocket: binance_2025-01-15.log

4. Latency Analysis

Use seq_num for data matching and compare receive_ts of the same market data:

Python

# Example analysis script
import pandas as pd

def extract_exchange_seq_num(rapidmarket_seq_num):
    """Extract original exchange sequence number from RapidMarket seq_num"""
    return rapidmarket_seq_num % int(1e18)

# Read data
rm1_data = pd.read_csv('rm_1_0_2025-01-15.log')
rm2_data = pd.read_csv('rm_2_0_2025-01-15.log')
binance_data = pd.read_csv('binance_2025-01-15.log')

# Restore RapidMarket 1.0's exchange_seq_num (only RM 1.0 needs restoration)
rm1_data['exchange_seq_num'] = rm1_data['seq_num'].apply(extract_exchange_seq_num)

# Compare RapidMarket 1.0 vs Binance WebSocket
rm1_vs_binance = rm1_data.merge(binance_data, left_on='exchange_seq_num', right_on='seq_num', suffixes=('_rm1', '_binance'))
rm1_vs_binance['latency_diff'] = rm1_vs_binance['receive_ts_binance'] - rm1_vs_binance['receive_ts_rm1']

# Compare RapidMarket 2.0 vs Binance WebSocket (RM 2.0's seq_num is the original sequence number)
rm2_vs_binance = rm2_data.merge(binance_data, on='seq_num', suffixes=('_rm2', '_binance'))
rm2_vs_binance['latency_diff'] = rm2_vs_binance['receive_ts_binance'] - rm2_vs_binance['receive_ts_rm2']

# Statistics on latency distribution
print("=== RapidMarket 1.0 vs Binance WebSocket ===")
print(f"Matched data count: {len(rm1_vs_binance)}")
print(f"Average latency difference: {rm1_vs_binance['latency_diff'].mean()} ns")
print(f"Maximum latency difference: {rm1_vs_binance['latency_diff'].max()} ns")
print(f"Minimum latency difference: {rm1_vs_binance['latency_diff'].min()} ns")

print("\n=== RapidMarket 2.0 vs Binance WebSocket ===")
print(f"Matched data count: {len(rm2_vs_binance)}")
print(f"Average latency difference: {rm2_vs_binance['latency_diff'].mean()} ns")
print(f"Maximum latency difference: {rm2_vs_binance['latency_diff'].max()} ns")
print(f"Minimum latency difference: {rm2_vs_binance['latency_diff'].min()} ns")

# Compare RapidMarket 1.0 vs 2.0 (use restored exchange_seq_num for matching)
rm1_vs_rm2 = rm1_data.merge(rm2_data, left_on='exchange_seq_num', right_on='seq_num', suffixes=('_rm1', '_rm2'))
rm1_vs_rm2['latency_diff'] = rm1_vs_rm2['receive_ts_rm2'] - rm1_vs_rm2['receive_ts_rm1']

print("\n=== RapidMarket 1.0 vs 2.0 ===")
print(f"Matched data count: {len(rm1_vs_rm2)}")
print(f"Average latency difference: {rm1_vs_rm2['latency_diff'].mean()} ns")
print(f"Maximum latency difference: {rm1_vs_rm2['latency_diff'].max()} ns")
print(f"Minimum latency difference: {rm1_vs_rm2['latency_diff'].min()} ns")

Performance Optimization Recommendations

CPU Affinity Settings

• Main Thread: Bind to dedicated CPU core to avoid context switching
• Write Thread: Bind to another CPU core to avoid competition with main thread

Important Notes

1. Time Synchronization: Ensure all test machines are time-synchronized
2. Network Latency: Consider the impact of network latency on test results
3. Data Integrity: Ensure all receivers can receive complete data
4. System Load: Avoid running other high-load programs during testing
5. Binance WebSocket Limitations:
   • Binance WebSocket has connection frequency limits, avoid frequent reconnections
   • Recommend using stable network connections
   • Binance data may arrive slightly later than RapidMarket data, which is normal

Troubleshooting

Common Issues

1. Compilation Errors: Ensure necessary dependency libraries are installed
2. Connection Failures: Check network connections and service status
3. Data Loss: Check system resources and network stability

Debug Options

• Use -v parameter to enable verbose output
• Check system log files
• Monitor system resource usage

Quick Start Guide

1. Install Python Dependencies:

Bash

# Run automatic installation script
chmod +x install_python_deps.sh
./install_python_deps.sh

# Or install manually
pip3 install websockets asyncio

2. Run Binance WebSocket Receiver:

Bash

python3 binance_websocket_receiver.py btcusdt 0 1

3. View Output Files:

Bash

ls -la binance_*.log

Quick Test Script

Create a test script to start all three receivers simultaneously:

Bash

#!/bin/bash
# test_latency.sh

echo "Starting latency comparison test..."

# Start RapidMarket 1.0
echo "Starting RapidMarket 1.0..."
./ltp_rm_1_0_receiver "tcp://localhost:50000" 0 1 &
RM1_PID=$!

# Start RapidMarket 2.0
echo "Starting RapidMarket 2.0..."
./ltp_rm_2_0_receiver "bf_m_pub" 0 2 &
RM2_PID=$!

# Start Binance WebSocket
echo "Starting Binance WebSocket..."
./binance_websocket_receiver "btcusdt" 0 3 &
BINANCE_PID=$!

echo "All receivers started. Press Ctrl+C to stop all processes."

# Wait for user interrupt
trap "echo 'Stopping all processes...'; kill $RM1_PID $RM2_PID $BINANCE_PID; exit" INT
wait

Usage:

Bash

chmod +x test_latency.sh
./test_latency.sh

Client Custom WebSocket Implementation Reference

If clients need to implement their own WebSocket receiver (supporting any programming language), they can refer to the implementation of binance_websocket_receiver.py:

Key Implementation Points:

1. Unified Data Format: Ensure output CSV format is consistent with RapidMarket receivers
2. Timestamp Precision: Use nanosecond-level timestamps (time.time() * 1_000_000_000)
3. seq_num Processing: Use original exchange sequence numbers for easy matching with RapidMarket data
4. Asynchronous Writing: Use queues to avoid blocking the main receiving thread
5. Error Handling: Implement reconnection mechanisms and exception recovery
6. Signal Handling: Support graceful exit

Output Format Requirements:

csv

symbol,seq_num,exchange_ts,local_ts,receive_ts,bid_price,bid_qty,ask_price,ask_qty

seq_num Field Description:

• RapidMarket 1.0 Receiver: seq_num = channel_id * 1e18 + exchange_seq_num
  • Example: 2000000000000000123 (channel_id=2, exchange_seq_num=123)
  • Requires restoration: Use seq_num % 1e18 to extract original sequence number
• RapidMarket 2.0 Receiver: seq_num = exchange_seq_num (original exchange sequence number)
  • Example: 123 (original exchange sequence number)
  • No restoration needed: Use directly
• Binance WebSocket: seq_num = exchange_seq_num (original exchange sequence number)
  • Example: 123 (original exchange sequence number)
  • No restoration needed: Use directly
• Client Custom Receiver: Recommend using original exchange sequence numbers for easy data matching

Example Code Structure (Python Reference Implementation):

Python

# 1. Asynchronous file writer
class AsyncFileWriter:
    def enqueue_event(self, event):
        # Add event to queue

    def run(self):
        # Asynchronous write thread main loop

# 2. WebSocket receiver
class CustomWebSocketReceiver:
    def parse_and_write(self, data):
        # Parse data and create event
        event = {
            'symbol': ...,
            'seq_num': ...,
            'receive_ts': int(time.time() * 1_000_000_000),
            # ... other fields
        }
        self.async_writer.enqueue_event(event)

Other Programming Language Implementation Points:

• Java: Use System.nanoTime() to get nanosecond timestamps
• C++: Use std::chrono::high_resolution_clock to get nanosecond timestamps
• Go: Use time.Now().UnixNano() to get nanosecond timestamps
• Node.js: Use process.hrtime.bigint() to get nanosecond timestamps
• C#: Use DateTimeOffset.UtcNow.ToUnixTimeMilliseconds() * 1000000 to get nanosecond timestamps

Technical Support

For any issues, please contact LiquidityTech technical support team.

---

LiquidityTech - The Leading Digital Asset Prime Brokerage for Institutions