Supercharge Your Trading Signals Using Qlib Quant#

Key Highlights#

• Extensibility: Qlib provides a modular design so you can swap data sources, models, and strategies.
• Open Source: Free to use, modify, and integrate.
• Community Support: Qlib is actively maintained, with frequent updates and community-contributed examples.

Prerequisites#

Before you install Qlib, ensure you have the following:

• Python 3.6+
• pip or conda
• Git (optional but recommended for cloning the Qlib repository)

If you have a dedicated virtual environment or a conda environment, it’s recommended to install Qlib there to avoid conflicts with other libraries.

Installation Steps#

1. Install Using pip

   1
   pip install pyqlib
   

   This is the simplest way to get started. PyPI hosts the latest stable release.

2. Clone from GitHub (for development or the latest features)

   1
   git clone https://github.com/microsoft/qlib.git
   2
   cd qlib
   3
   pip install --upgrade .
   

   Cloning the repository allows you to stay at the cutting edge of new Qlib features.

Verifying the Installation#

Open a Python shell or Jupyter notebook and execute:

1
import qlib
2
print(qlib.__version__)

If it prints a version without errors, you have a functional Qlib environment. Next, let’s see how to initialize Qlib’s default configuration.

Initializing Qlib#

Qlib needs to know which market or region you’re trading in (e.g., China’s A-share market or the U.S. market). For instance, to initialize Qlib in the Chinese market mode:

1
import qlib
2
from qlib.config import REG_CN
3

4
qlib.init(provider_uri="~/.qlib/qlib_data/cn_data", region=REG_CN)

If you want to load U.S. stock data, you can set:

1
from qlib.config import REG_US
2

3
qlib.init(provider_uri="~/.qlib/qlib_data/us_data", region=REG_US)

You may also configure other advanced parameters, such as custom data directories, logging levels, and online/offline modes.

Data Directory Structure#

Qlib organizes data in a structured format within a root data directory (e.g., ~/.qlib/qlib_data/). Below is a simplified depiction of the directory:

1
qlib_data/
2
    └── cn_data/ (or us_data/)
3
        ├── day/           # Daily bar data
4
        ├── 1min/          # 1-minute frequency data
5
        ├── features/      # Precomputed factors, signals, or transformations
6
        └── instruments/   # Listings of available securities

Supported Frequencies and Symbols#

You can work with different data frequencies (daily, intraday, etc.) by specifying them when fetching or analyzing data. Qlib also supports different market symbols based on the region you select (e.g., 600519.SHG for a Chinese A-share symbol, or AAPL for a U.S. symbol).

Data Loading#

Qlib provides standardized APIs for loading data:

1
import pandas as pd
2
from qlib.data import D
3

4
# Retrieve daily data for a specific stock
5
df = D.features(['AAPL'],
6
                fields=["$close", "$volume"],
7
                freq="day",
8
                start_time="2020-01-01",
9
                end_time="2021-01-01")
10

11
print(df.head())

This returns a multi-indexed Pandas DataFrame with stock symbols and dates. The features function can also load custom signals, such as moving averages or momentum measures.

Step 1: Define the Factor#

A basic signal might be a 20-day moving average of a stock’s closing price. We can define it in Qlib’s expression system:

1
from qlib.data.dataset import DatasetD, Dataset
2

3
# Expression for a 20-day simple moving average of closing prices
4
MA20 = {
5
    "name": "MA20",
6
    "expression": "Mean($close, 20)"
7
}

Step 2: Create a Dataset#

Qlib uses a dataset object that ties instruments, data range, and feature expressions together. For example:

1
from qlib.data.dataset.processor import TSDatasetProcessor
2

3
dataset_config = {
4
    "class": "DatasetD",
5
    "kwargs": {
6
        "handler": {  # Qlib handler config
7
            "class": "Alpha158",
8
            "kwargs": {
9
                "start_time": "2020-01-01",
10
                "end_time": "2022-01-01",
11
                "freq": "day"
12
            }
13
        },
14
        "processors": [
15
            {"class": "CSRankNorm", "kwargs": {"fields_group": "feature"}},
16
        ]
17
    }
18
}
19
# Create dataset
20
my_dataset = DatasetD(**dataset_config["kwargs"])

In this case, we’re using the built-in Alpha158 handler, which provides many standard factors. The CSRankNorm processor normalizes the features across different stocks to facilitate cross-sectional comparison.

Step 3: Implement a Simple Backtest#

To test our MA20 signal, we need to define a strategy or model to act upon the signal. Here’s a simple hand-coded backtest that buys a stock if the closing price is above its 20-day moving average and sells otherwise.

1
import numpy as np
2

3
def simple_signal(df, symbol):
4
    # Symbol-based DataFrame for the required stock
5
    symbol_data = df.xs(symbol, level='instrument')
6
    # This DataFrame should include columns: $close, MA20, etc.
7
    symbol_data["signal"] = np.where(
8
        symbol_data["$close"] > symbol_data["MA20"], 1, 0
9
    )
10
    return symbol_data["signal"]
11

12
# Retrieve data for a single stock
13
data = D.features(["AAPL"], ["$close", "Mean($close, 20)##MA20"],
14
                  start_time="2020-01-01", end_time="2021-01-01")
15

16
signal_series = simple_signal(data, "AAPL")
17

18
# Now, we can simulate PnL:
19
returns = data.xs("AAPL", level='instrument')["$close"].pct_change()
20
strategy_returns = returns * signal_series.shift(1)
21
cumulative_returns = (1 + strategy_returns).cumprod()
22

23
print(cumulative_returns.tail())

This simplistic strategy is meant to illustrate how trading signals are generated and tested in Qlib. For a more robust backtesting system, you can use Qlib’s built-in offline or online backtest modules.

Introduction to Factor Research#

Factors (or alpha signals) are measurable characteristics that can predict future returns. In Qlib, factor research involves:

1. Defining a Factor: Typically through some mathematical expression of historical price, volume, or fundamental data.
2. Analyzing Factor Efficacy: Using correlation analysis, sharpe ratio, or IC (Information Coefficient) to determine how predictive the factor is.
3. Combining Multiple Factors: Merging factors into a composite signal or alpha score.

Built-In Factors#

Qlib ships with a variety of named factors under the Alpha158 or Alpha360 collection. These sets often include:

• Momentum: Past returns over n-days (e.g., Return($close, 20))
• Volatility: Standard deviation of prices, or intraday volatility
• Volume-Weighted: Volume-based signals, like VWAP or BOP (Balance of Power)
• Technical Indicators: RSI, MACD, and multiple moving average crossovers

These can be accessed easily through Qlib’s config files, or you can define them using Qlib’s expression language.

Evaluating Factor Performance#

To systematically evaluate factors, you might implement a pipeline that computes the daily Factor values for all stocks and compute the IC (Information Coefficient) between the factor values and subsequent returns. A high IC (positively or negatively) implies predictive power.

Below is a simplified code snippet to evaluate factor performance:

1
import pandas as pd
2
from qlib.data.dataset import DatasetD
3
from qlib.data import D
4

5
# 1. Define your factor expression
6
factor_expr = "Mean($close, 20) - Mean($close, 60)"  # For example
7

8
# 2. Pull data for an instrument universe
9
instruments = D.list_instruments(D.features(join=True))
10
data = D.features(instruments, [factor_expr+"##my_factor", "$close"],
11
                  freq="day", start_time="2020-01-01", end_time="2022-01-01")
12

13
# 3. Compute daily returns
14
data["return_1d"] = data.groupby(level='instrument')["$close"].pct_change()
15

16
# 4. Shift the factor by 1 day to avoid lookahead bias
17
shifted_factor = data["my_factor"].groupby(level='instrument').shift(1)
18
shifted_return = data["return_1d"].groupby(level='instrument')
19
ic = shifted_factor.corr(shifted_return)
20
print(f"Information Coefficient (IC) for factor: {ic}")

If the factor is predictive, the IC should be significantly different from zero (positive or negative), indicating you could exploit it with a long-short strategy.

Workflow for ML in Qlib#

1. Prepare a Dataset: Use Qlib’s dataset structure to specify instruments, time range, frequency, feature expressions, and target variables.
2. Configure a Model: Choose from built-in models or custom ones. For instance, LSTM for time-series forecasting.
3. Set Up Trainer: Use Qlib’s R module (research) to define training, validation, and testing splits.
4. Evaluate & Compare Models: Evaluate performance metrics such as prediction accuracy, Sharpe Ratio, or annualized returns from generated signals.

Sample Code for an ML Workflow#

Below is a fully functioning example that trains an LSTM model using Qlib:

1
from qlib.config import REG_CN
2
import qlib
3
from qlib.contrib.model.pytorch_lstm import LSTMModel
4
from qlib.contrib.strategy.signal_strategy import SignalStrategy
5
from qlib.contrib.evaluate import risk_analysis
6

7
# Initialize Qlib
8
qlib.init(provider_uri="~/.qlib/qlib_data/cn_data", region=REG_CN)
9

10
# 1. Prepare Dataset
11
dataset_config = {
12
    "class": "DatasetD",
13
    "module_path": "qlib.data.dataset",
14
    "kwargs": {
15
        "handler": {
16
            "class": "Alpha158",
17
            "kwargs": {
18
                "start_time": "2017-01-01",
19
                "end_time": "2021-01-01",
20
                "freq": "day",
21
            }
22
        },
23
        "segments": {
24
            "train": ("2017-01-01", "2019-12-31"),
25
            "valid": ("2020-01-01", "2020-06-30"),
26
            "test": ("2020-07-01", "2021-01-01"),
27
        }
28
    }
29
}
30
dataset = DatasetD(**dataset_config["kwargs"])
31

32
# 2. Model Configuration
33
model = LSTMModel(
34
    d_feat=6,          # number of input features
35
    hidden_size=64,
36
    num_layers=2,
37
    dropout=0.0,
38
    n_epochs=10,
39
    lr=0.001,
40
    metric="loss",
41
    early_stop=5
42
)
43

44
# 3. Train the Model
45
model.fit(dataset.get_data("train"), dataset.get_data("valid"))
46

47
# 4. Prediction and Backtest
48
pred = model.predict(dataset.get_data("test"))
49
pred.name = "score"
50

51
backtest_strategy = SignalStrategy()  # default strategy
52
backtest_report = backtest_strategy.run(pred, dataset.get_data("test"))
53
analysis = risk_analysis(backtest_report)
54

55
print(analysis)

Key items to note:

• The Alpha158 handler automatically generates multiple factors.
• We define training, validation, and test segments.
• The LSTM model is trained on the train/valid sets and then evaluated on the test set.
• We generate predictions (scores) which are fed into a simple signal strategy. Finally, the results are analyzed using Qlib’s built-in risk_analysis.

Integrating Custom Data#

Qlib can use external CSV files, databases, or APIs as long as you write a custom data handler that transforms raw data into Qlib’s standard format. A typical process is:

1. Fetch from source (e.g., a CSV file with columns: date, open, high, low, close, volume).
2. Transform data to match Qlib’s required structure (instrument, date, fields).
3. Store the data in Qlib’s local directory or keep it in memory.
4. Register your custom handler in Qlib’s configuration.

Merging Alternative Data#

Assume you have a dataset of daily sentiment scores for each stock. You could combine this with price data by:

• Creating a custom field ($sentiment) in Qlib’s dataset.
• Generating a factor expression like Mean($sentiment, 5) to measure weekly average sentiment.

Then you can incorporate it into your backtest or ML model just like any standard factor.

Avoiding Common Bottlenecks#

• Excessive I/O: Ensure your data provider URI is on a fast disk or in memory.
• Unnecessary Re-computations: If you’re repeatedly computing the same factor, use Qlib’s feature caching or precompute them once and store them.
• Large Universe: If you’re working with thousands of instruments, consider a sampling approach or an incremental update strategy.

Development Lifecycle#

Before deploying, your typical development workflow might look like:

1. Local Research: Develop factors and strategies locally on a sample dataset.
2. Paper Trading: Test strategies in a simulated environment to verify performance.
3. Pilot Deployment: Deploy a small share of capital or use a single instrument to confirm real-world viability.
4. Scaling: Gradually allocate more capital or broaden the instrument universe.

Infrastructure Options#

• Cloud Platforms: Host Qlib on AWS, GCP, or Azure. Leverage their managed services like Amazon S3 for data storage or AWS Lambda for scheduled tasks.
• On-Premise Servers: Keep everything in-house for security and regulatory reasons.
• Hybrid: Use the cloud for deep learning model training but keep sensitive data locally.

Continuous Integration and Monitoring#

• Automated Testing: Implement unit tests for your data pipelines and factor computations.
• Scheduled Jobs: Use cron or managed job schedulers to update data daily, recompute signals, and make trades.
• Error Handling: Implement robust logs and alerts to quickly pinpoint data feed disruptions or model errors.

Risk Management and Compliance#

Even the best backtests can fail in live markets if risk controls are not set properly. Consider:

• Stop-Loss and Risk Limits: Incorporate them as part of your trading strategy.
• Regulatory Compliance: Keep track of short-sale constraints, position limits, or futures market regulations.