Enhancing Backtesting Capabilities Using Qlib Quant#

Key Components#

• Data Layer: Flexible architecture for handling multiple data sources, both offline and online.
• Modeling Layer: Tools for building alpha factors, machine learning models, and advanced forecasting methods.
• Backtesting Layer: Integrated environment for simulating trades and evaluating performance metrics.
• Deployment Layer: Mechanisms to push successful models into production.

In this guide, we’ll primarily focus on how Qlib handles backtesting, from the basics of setting up a backtestable dataset to advanced features such as custom cost functions and slippage models.

Prerequisites#

• Python 3.7 or above.
• pip for package installation.
• A C compiler (e.g., GCC) if you plan to install advanced dependencies.

Installation Steps#

1. Create a virtual environment (optional but recommended):

   1
   python -m venv qlib_env
   2
   source qlib_env/bin/activate       # On Linux/Mac
   3
   # or
   4
   qlib_env\Scripts\activate.bat      # On Windows
   

2. Install Qlib from PyPI:

   1
   pip install pyqlib
   

3. Verify your installation:

   1
   import qlib
   2
   print(qlib.__version__)
   

   If this runs without errors and prints a version number, you are good to go.

Data Preparation#

Qlib supports both user-provided data and automatically downloaded stock data (e.g., from Yahoo Finance). For initial testing, you can use the built-in data source that Qlib offers. If you prefer to use your own dataset, Qlib provides a data preparation utility to convert raw CSV files into its internal format.

Data Storage Structure#

When you ingest data into Qlib, it creates a folder structure organizing each symbol’s data along with any derived factors or features. The default structure looks like this:

1
qlib_data/
2
├── features
3
│   └── <instrument>/
4
│       ├── day.fa.gz
5
│       ├── 1min.fa.gz
6
│       └── ...
7
├── instruments
8
│   ├── all.txt
9
│   ├── ...
10
├── calendars
11
│   └── day.txt
12
└── features.cache

Each instrument (stock or asset) has a corresponding folder containing feature arrays (.fa.gz files). These arrays store daily or intraday data for open, high, low, close, volume, etc.

Initializing Qlib for Offline Data#

Before you can use Qlib, you need to initialize its environment. This is typically one of the first lines of code in your script or notebook:

1
import qlib
2
from qlib.config import REG_CN
3

4
qlib.init(provider_uri='~/.qlib/qlib_data/cn_data', # path to your data
5
          region=REG_CN)

This command configures Qlib to look for local data (stored in ’~/.qlib/qlib_data/cn_data’) and sets the region to China (REG_CN). If you’re using U.S. stocks or another market, you can configure this accordingly.

Step 1: Define an Alpha Factor#

Alpha factors often attempt to predict stock returns. Here, we’ll define a simple momentum factor based on rolling returns:

1
import pandas as pd
2
import numpy as np
3
from qlib.data import D
4

5
# Momentum factor: today's close / close of N days ago
6
def momentum_factor(instrument, field='$close', window=20):
7
    data = D.features(instruments=instrument, fields=[field], freq='day')
8
    data['mom'] = data[field] / data[field].shift(window) - 1
9
    return data['mom']

Step 2: Generate Trading Signals#

We can generate long signals when momentum is above a certain threshold and generate short signals when momentum is below a threshold.

1
def generate_signals(mom_series, upper=0.02, lower=-0.02):
2
    # If momentum > 2%, go long
3
    # If momentum < -2%, go short
4
    signal = pd.Series(index=mom_series.index, dtype=float)
5
    signal[mom_series > upper] = 1
6
    signal[mom_series < lower] = -1
7
    signal.fillna(0, inplace=True)
8
    return signal

Step 3: Constructing the Strategy Class#

In Qlib, strategies are typically created using dedicated strategy classes. For demonstration, we’ll define a minimal class:

1
from qlib.contrib.strategy.signal_strategy import SignalStrategy
2

3
class MomentumStrategy(SignalStrategy):
4
    def __init__(self, upper=0.02, lower=-0.02, window=20):
5
        super().__init__()
6
        self.upper = upper
7
        self.lower = lower
8
        self.window = window
9

10
    def generate_signal(self, instrument, init_time=None, end_time=None):
11
        # Fetch momentum factor
12
        mom_series = momentum_factor(instrument, window=self.window)
13
        # Generate signals
14
        signal_series = generate_signals(mom_series, upper=self.upper, lower=self.lower)
15
        return signal_series

Here, we extend SignalStrategy, which serves as a base class for signal-based trading. The generate_signal method retrieves momentum values and translates them into buy/sell signals.

Anatomy of a Qlib Backtest#

Qlib uses a configuration-driven approach for backtesting. Each part (data handler, strategy, execution model, cost model, etc.) is definable in a YAML or Python dictionary format.

Below is a simplified backtesting configuration:

1
from qlib.backtest import backtest
2
from qlib.utils import flatten_dict
3

4
backtest_config = {
5
    "start_time": "2019-01-01",
6
    "end_time": "2020-12-31",
7
    "strategy": {
8
        "class": "MomentumStrategy",
9
        "module_path": "<your_module_path>",
10
        "kwargs": {
11
            "upper": 0.02,
12
            "lower": -0.02,
13
            "window": 20
14
        }
15
    },
16
    "instruments": ["SH600519", "SZ000001"],  # Example instruments
17
    "benchmark": "SH000300",
18
    "freq": "day",
19
    "deal_price": "close",
20
    "open_cost": 0.001,
21
    "close_cost": 0.001,
22
    "min_cost": 5,
23
}
24

25
bt_result = backtest(backtest_config)
26
print(flatten_dict(bt_result))

Execution Flow#

1. Data Slicing
   Qlib automatically slices the data required for the specified start/end dates and instruments.

2. Signal Generation
   The momentum strategy’s generate_signal method is called for each instrument.

3. Order Execution
   Based on signals (1, 0, -1), the backtester simulates entries and exits. The “deal_price” is set to “close” in this example, though you can choose other price fields.

4. Cost and Slippage
   Transaction costs and slippage are accounted for based on the given parameters.

5. Performance Metrics
   Qlib computes essential metrics, such as total returns, Sharpe ratio, maximum drawdown, etc.

Retrieving Backtest Results#

The bt_result object provides a wealth of information:

• backtest: Contains trade logs, positions, and daily PnL.
• analysis: Summaries such as annualized return, information ratio, drawdown, etc.

1
import matplotlib.pyplot as plt
2

3
daily_returns = bt_result["backtest"]["daily_returns"]
4
cumulative_returns = (1 + daily_returns).cumprod()
5

6
plt.plot(cumulative_returns.index, cumulative_returns.values, label="Momentum Strategy")
7
plt.legend()
8
plt.title("Cumulative Returns")
9
plt.show()

7.1 Custom Costs and Slippage Models#

Transaction costs can significantly impact a strategy’s bottom line. Qlib makes it straightforward to add custom cost or slippage models:

1
from qlib.backtest.decision import OrderDir
2
from qlib.backtest.cost import BaseCost
3

4
class CustomCost(BaseCost):
5
    def __init__(self, cost_rate=0.0015):
6
        super().__init__()
7
        self.cost_rate = cost_rate
8

9
    def get_trade_cost(self, trade_price, trade_size, direction):
10
        cost = trade_price * trade_size * self.cost_rate
11
        return cost if cost > 1 else 1  # Minimum 1 currency unit

By overriding get_trade_cost, you can apply any formula you desire. The same approach applies to slippage models, which can be integrated into the backtest configuration.

7.2 Factor Research and Analysis#

Qlib provides factor analysis tools that let you research alpha factors, combine them into scoring systems, and evaluate factor returns. Tools such as IC (Information Coefficient) or risk factor decomposition can be conducted using built-in modules.

7.3 Machine Learning and Forecasting#

Beyond simple factors, Qlib supports sophisticated pipelines where a machine learning model (e.g., LightGBM, XGBoost) predicts future returns or alpha values. This advanced usage typically involves:

1. Building a dataset with relevant features (price-based, fundamental, alpha factors).
2. Training a predictive model.
3. Using the predictions as trading signals.

While this topic merits its own deep dive, rest assured Qlib has integrated modules that facilitate model training and nested cross-validation for robust performance estimation.

8.1 Caching Mechanisms#

Qlib automatically caches certain computations, such as feature extraction, to speed up subsequent runs. You can configure the caching behavior in qlib.init() by specifying cache paths and parameters.

8.2 In-Memory Data Loading#

When running large-scale backtests, reading from disk can become a bottleneck. Qlib’s data handlers can store critical data in memory to reduce I/O overhead, at the cost of using more RAM.

8.3 Parallelization#

For multi-instrument or multi-parameter backtests, you can parallelize workloads using Python’s multiprocessing or distributed computing frameworks. Qlib’s architecture allows you to distribute the data loading and backtesting tasks, although this can require additional orchestration.

9.1 Custom Strategy Classes#

We showed a small example of building a momentum strategy class. For more complex or machine learning strategies, you can:

• Inherit from StrategyBase, BaseStrategy, or SignalStrategy.
• Override methods like login_step, generate_signal, finish_step.
• Incorporate custom data processing before generating signals.

9.2 Custom Execution Handler#

For specialized market microstructure modeling, you might want to simulate partial fills or volume constraints. You can override Qlib’s default execution handler:

1
from qlib.backtest.executor import BaseExecutor
2

3
class CustomExecutor(BaseExecutor):
4
    def _execute_trade(self, trade_decision):
5
        # Custom logic for partial fills, volume constraints, etc.
6
        pass

9.3 Integration with External Libraries#

If you rely on advanced libraries (e.g., deep learning frameworks or specialized data analytics tools), Qlib can be integrated by:

1. Creating a custom data handler that reads from your pipeline output.
2. Wrapping your ML model in a class that Qlib can call for inference.
3. Substituting the model or factor code in the backtesting configuration.