High-Performance Data Architecture
Evolution at a Glance (V1 → V4)
V1: Direct API Dependence (Complete) ☑️
State: Hummingbot’s built-in API candle fetching
Bottleneck: rate-limited, no DEX support
V2: Offline Parquet Files (Complete) ☑️
State: Pre-downloaded historical data, local file reads
Benefit: Seconds per backtest, enabling overnight parameter sweeps
Outcome: Rapid iteration, precise profit scenario identification
V3: Incremental Updates for Live Data (Complete) ☑️
State: Hybrid approach, periodic API fetches feeding in-memory databases, not direct strategy queries
Benefit: Up-to-date data without losing speed advantages
Outcome: Both historically rich and live-relevant strategies
V4: Advanced Analytics & ML (In Progress)
State: Fully leveraged fast access for ML-driven adaptation, scenario-based testing
Benefit: Uncover subtle patterns, adapt quickly, confidently showcase data-driven profit claims
Outcome: Cutting-edge strategies that inspire user confidence
The Challenge of API-Based Data Retrieval (V1)
Initially (V1), direct API-based candle fetching was the default approach. While APIs promise deep historical data, they quickly falter under granular requests. Network latency, JSON parsing, and incremental queries accumulate, turning a year-long backtest into an hours-long ordeal.
Offline Data and Parquet Files (V2)
At V2, we adopted offline Parquet files, eliminating the API bottleneck for historical data. This approach laid the foundation for moving from merely waiting on data to actively innovating.
Integrating Fresh Data for Live Trading (V3)
While V2 excelled historically, we still need fresh data for live trading. V3 will introduce a hybrid model:
Incremental Updates: Periodically load fresh data from APIs into a high-speed in-memory database, not directly into the strategy at runtime.
Result: Strategies remain well-informed by historical depth and agile in the present market, all without returning to API-induced slowdowns.
Advanced Analytical Horizons (V4)
V4 leverages the speed and accessibility built in earlier stages:
Fully Automated Parameter Exploration: Rapid local access enables large-scale overnight parameter sweeps, helping us pinpoint top-performing configurations and deploy with confidence.
Machine Learning-Driven Adaptation: Immediate data access makes advanced ML workflows seamless. Models train and retrain quickly, leading to dynamically evolving strategies we can boast about as cutting-edge and data-driven.
Scenario-Based Regime Testing: Automated segmentation of historical data into regimes (bullish, bearish, volatile, etc.) is now trivial. The strategy’s resilience can be documented and presented as proof that it thrives under various conditions.
Last updated