Unlock Proprietary Alpha Strategies with Qlib Quant#

4.1 Working with Qlib Datasets#

Qlib supports multiple regional datasets out of the box, such as Chinese stock markets and partial support for US stocks. If you want to work with Qlib’s minimal sample data or more comprehensive data, you can do so with a single command line:

1
# For China market data
2
python scripts/get_data.py qlib_data --target_dir ~/.qlib/qlib_data/cn_data --region cn

1
# For minimal sample data
2
python scripts/get_data.py qlib_data --target_dir ~/.qlib/qlib_data/cn_data --interval 1d --region sample

Once the above commands complete, your local data directory will be populated with a standard Qlib structure:

• csv folder containing daily bar data.
• calibration meta files that store information about the data range.

4.2 Custom Datasets#

For advanced use cases or specialized markets, you can ingest custom data. The process involves:

1. Converting your raw CSV or database records into a standard format containing at least the “open,” “close,” “high,” “low,” and “volume” columns.
2. Using Qlib’s data import functions to load and store the data in its backend format.

Below is a simple snippet showing how to use ingesting scripts:

1
import qlib
2
from qlib.data import D
3

4
provider_uri = "~/.qlib/qlib_data/custom_data"
5
qlib.init(provider_uri=provider_uri)
6

7
# Suppose you have a CSV path with columns: date, symbol, open, close, high, low, volume
8
csv_path = "path/to/your/custom_dataset.csv"
9

10
# You can create a custom importer or adapt Qlib's built-in methods
11
importer = D.features(csv_path, start_time="2015-01-01", end_time="2021-12-31")
12

13
# Potential advanced usage:
14
# Parse your CSV into a DataFrame, rename columns as needed, and call the Qlib format storage APIs

4.3 Data Quality Checks#

Before building signals, perform data quality checks:

• Ensure your data is time-stamped correctly, especially if you’re combining multiple data sources.
• Verify that missing or erroneous data points are handled (e.g., fill-forward for certain columns, dropping rows for others).
• Confirm corporate actions like splits, dividends, and mergers are properly adjusted in your dataset.

Clean data dramatically improves the reliability of your backtests and factor evaluations.

5.1 Basic Factor Types#

Below is a table illustrating some standard alpha factor categories and their common definitions:

Factors typically combine a raw data series (e.g., price, volume, or fundamentals) with a transformation (e.g., difference, normalization, ratio). Each factor is tested for its predictive power, usually by analyzing forward returns.

5.2 Factor Pipeline Example#

Here is a minimal code snippet demonstrating how to define and compute alpha factors in Qlib:

1
import qlib
2
import pandas as pd
3
from qlib.data import D
4
from qlib.contrib.data.handler import Alpha360
5

6
# Initialize Qlib with your data provider URI
7
qlib.init(provider_uri="~/.qlib/qlib_data/cn_data")
8

9
# Define a custom DataHandler that loads the daily data
10
handler = Alpha360(start_time="2020-01-01", end_time="2021-01-01")
11

12
data = handler.fetch()
13
# data now contains raw columns like open, close, high, low, volume
14

15
# Example: Momentum Factor = (close - delay(close, N)) / delay(close, N)
16
def momentum_factor(df, window=5):
17
    df['factor_mom'] = (df['AVG']['close'] - df['AVG']['close'].shift(window)) / df['AVG']['close'].shift(window)
18
    return df
19

20
factor_df = momentum_factor(data.copy(), window=5)
21
# factor_df contains your new momentum factor in 'factor_mom'

5.3 Parameter Tuning#

Many factors include free parameters, such as lookback windows for moving averages or thresholds for overbought/oversold levels. Tune these factors through rigorous statistical validation, walk-forward analysis, or hyperparameter optimization. In practice, you may find that certain windows or factor definitions consistently produce stronger alpha signals in particular market regimes.

5.4 Combining Factors#

Rather than relying on a single factor, combining multiple signals into a composite factor often yields more stable and robust performance. You can do this by:

1. Standardizing each alpha factor to ensure they are comparable.
2. Averaging, weighting, or applying a machine learning model to combine signals.
3. Testing correlation among factors to avoid redundancy in your final strategy.

6.1 Qlib’s Strategy Components#

Qlib provides a variety of built-in strategy components to handle the signal-to-portfolio logic and the backtest engine. The general process includes:

1. Define Your Data Handler (loading factors and raw data).
2. Create a Model or Strategy: This can be a rule-based or predictive (ML) approach.
3. Run a Backtest: Evaluate results, including returns, drawdowns, and standard risk metrics.

Here is an example of a simple strategy implementation in Qlib:

1
import qlib
2
from qlib.data.dataset import DatasetD
3
from qlib.data.dataset.handler import DataHandlerLP
4
from qlib.contrib.strategy.signal_strategy import TopkDropoutStrategy
5
from qlib.contrib.evaluate import backtest as normal_backtest
6

7
qlib.init(provider_uri="~/.qlib/qlib_data/cn_data")
8

9
# Create a dataset
10
class SimpleDataset(DatasetD):
11
    def __init__(self, handler_kwargs, segments):
12
        super().__init__(handler=DataHandlerLP(**handler_kwargs), segments=segments)
13

14
dataset = SimpleDataset(
15
    handler_kwargs={
16
        "start_time": "2020-01-01",
17
        "end_time": "2021-01-01",
18
        "frequency": "day",
19
        "instruments": "csi300"
20
    },
21
    segments={"train": ("2020-01-01", "2021-01-01")}
22
)
23

24
# Suppose the dataset includes a 'score' column as our alpha signal
25
strategy = TopkDropoutStrategy(
26
    signal="score",
27
    ndropout=0,
28
    topk=50,
29
    hedge="BB",
30
    stop_loss=None,
31
    take_profit=None
32
)
33

34
backtest_config = {
35
    "start_time": "2020-01-01",
36
    "end_time": "2021-01-01",
37
    "account": 100000000,
38
    "benchmark": "SH000300",  # CSI 300 as benchmark
39
    "verbose": True,
40
}
41

42
# Running backtest
43
report_df, positions_df = normal_backtest(
44
    strategy=strategy,
45
    data=dataset,
46
    backtest_config=backtest_config
47
)
48

49
print(report_df.head())

6.2 Performance Metrics#

Qlib’s backtest engine offers many performance metrics:

• Annualized Return
• Sharpe Ratio
• Max Drawdown
• Information Ratio
• Win-Loss Ratios

These quantitative metrics let you compare strategies across different periods and parameters. Perform multiple backtests to see how stable each strategy is under varied market conditions. That helps weed out overfitted or regime-specific signals.

6.3 Pitfalls to Avoid#

1. Look-Ahead Bias: Make sure you do not use future data (e.g., tomorrow’s closing price) when calculating today’s features.
2. Data Snooping: Over-optimizing hyperparameters on a limited backtest set can lead to illusions of high performance.
3. Liquidity and Slippage: In real markets, position sizes can impact prices. Model transaction costs, market impact, and partial fills as accurately as possible.
4. Survivorship Bias: Ensure you include delisted stocks or earlier tickers that changed. Excluding them can artificially inflate backtest returns.

7.1 The Momentum Hypothesis#

Momentum strategies bet that assets which have outperformed in the recent past will continue to do so in the short to medium term. Conversely, underperformers are expected to keep lagging.

7.2 Factor Construction#

We’ll use a simple rate of change factor, measuring price momentum:

1
import qlib
2
import pandas as pd
3
from qlib.data import D
4
from qlib.contrib.data.handler import Alpha360
5

6
qlib.init(provider_uri="~/.qlib/qlib_data/cn_data")
7

8
handler = Alpha360(start_time="2020-01-01", end_time="2021-01-01")
9

10
data = handler.fetch()
11

12
def momentum_factor(df, window=20):
13
    df['Mom'] = (df['AVG']['close'] - df['AVG']['close'].shift(window)) / df['AVG']['close'].shift(window)
14
    return df
15

16
df = momentum_factor(data.copy(), window=20)

We then use this “Mom” column as our alpha signal.

7.3 Strategy Setup#

1. Convert the “Mom” signal into buy/hold/sell decisions.
2. Allocate capital to the top decile (10%) of stocks with the highest “Mom” factor.
3. Rebalance monthly.

7.4 Backtesting Code#

Below is a conceptual snippet to run a momentum strategy using Qlib’s built-in modules:

1
from qlib.contrib.strategy.signal_strategy import WeightStrategyBase
2
from qlib.contrib.evaluate import backtest as normal_backtest
3
import numpy as np
4

5
class MomentumStrategy(WeightStrategyBase):
6
    def __init__(self, top_percent=0.1):
7
        super().__init__()
8
        self.top_percent = top_percent
9

10
    def generate_weight_position(self, score, current):
11
        # Sort stocks by factor
12
        score = score.sort_values(ascending=False)
13
        # Select top portion
14
        n_top = int(len(score) * self.top_percent)
15
        selected_stocks = score.index[:n_top]
16
        # Assign equal weights among selected stocks
17
        weights = pd.Series(0, index=score.index)
18
        if n_top > 0:
19
            weights[selected_stocks] = 1.0 / n_top
20
        return weights
21

22
strategy = MomentumStrategy(top_percent=0.1)
23

24
dataset_for_backtest = SimpleDataset(
25
    handler_kwargs={
26
        "start_time": "2020-01-01",
27
        "end_time": "2021-01-01",
28
        "frequency": "day",
29
        "instruments": "csi300"
30
    },
31
    segments={"train": ("2020-01-01", "2021-01-01")}
32
)
33

34
report_df, positions_df = normal_backtest(
35
    strategy=strategy,
36
    data=dataset_for_backtest,
37
    backtest_config={
38
        "start_time": "2020-01-01",
39
        "end_time": "2021-01-01",
40
        "account": 10000000,
41
        "benchmark": "SH000300",
42
        "verbose": True
43
    }
44
)
45

46
print("Backtest Report:")
47
print(report_df)

7.5 Observing Results#

Typically, the results will show the strategy’s annualized return, Sharpe ratio, and drawdown. A strong momentum factor strategy can outperform the benchmark in momentum-friendly markets but may fare poorly in mean-reverting environments.

8.1 ML Pipeline Overview#

A typical pipeline for an ML-based alpha model:

1. Feature Engineering: Generate multiple alpha factors and raw features.
2. Label Construction: Often forward returns are used as the target variable.
3. Split Data: Train/validation/test sets.
4. Train Model: Use regression or classification to predict the target.
5. Generate Signals: Convert predicted returns into ranking or weighting signals.
6. Evaluate Performance: Backtest the resulting signals.

8.2 Example with LightGBM#

Below is a code snippet using LightGBM as the predictive model:

1
import lightgbm as lgb
2
import pandas as pd
3
from qlib.contrib.model.gbdt import LGBModel
4
from qlib.data.dataset import DatasetH
5
from qlib.data.dataset.handler import DataHandlerLP
6
from qlib.model.base import Model
7
from qlib.workflow import R
8

9
# Step 1: Define your data handler
10
class MyDataHandler(DataHandlerLP):
11
    def __init__(self, **kwargs):
12
        super().__init__(**kwargs)
13

14
    def feature(self, df):
15
        # Create multiple factors
16
        df["Mom10"] = df["close"].pct_change(10)
17
        df["Volatility20"] = df["close"].rolling(20).std()
18
        # shift label by next day return
19
        df["Label"] = df["close"].shift(-1) / df["close"] - 1
20
        return df
21

22
# Step 2: Build a dataset
23
dataset = DatasetH(
24
    data_handler=MyDataHandler(instruments="csi300", start_time="2020-01-01", end_time="2022-01-01", freq="day"),
25
    segments={"train": ("2020-01-01", "2021-01-01"), "test": ("2021-01-02", "2022-01-01")}
26
)
27

28
# Step 3: Initialize the LGBModel
29
config = {
30
    "learning_rate": 0.01,
31
    "num_leaves": 64,
32
    "n_estimators": 1000
33
}
34
lgb_model = LGBModel(**config)
35

36
# Step 4: Train the model
37
lgb_model.fit(dataset)
38

39
# Step 5: Predict on test set
40
pred = lgb_model.predict(dataset, segment="test")
41
print(pred.head())
42

43
# Step 6: Evaluate
44
# Convert pred to signals and backtest similarly to the factor-based approach

Here, we used Qlib’s wrapper for LightGBM (LGBModel), which simplifies model training and inference in a quantitative context.

8.3 Handling Overfitting#

Machine learning models are prone to overfitting, especially on small or insufficiently diverse datasets. Mitigating overfitting involves:

1. Proper Train/Test Splits: Use future data only for testing.
2. Regularization: Techniques like dropout, L1/L2 regularization, or early stopping.
3. Hyperparameter Tuning: Automated search tools (grid search, Bayesian optimization) to find robust configurations.
4. Extensive Cross-Validation: Rolling window cross-validation simulates real conditions.

8.4 Ensemble Methods#

Ensembling multiple models or factor strategies can enhance robustness. Combining a simple linear model with a more advanced tree-based or deep learning model often yields steadier performance over varying market conditions.

9.1 Position Sizing and Leverage#

Your position sizing method (equal-weight, volatility-based, or risk-parity) can drastically impact your returns. Qlib allows flexible portfolio construction rules to help you systematically size positions, limit maximum weight per instrument, and incorporate leverage.

9.2 Stop-Loss and Take-Profit Logic#

In Qlib, you can integrate stop-loss or take-profit triggers within the strategy module to exit positions upon certain drawdowns or after reaching profit targets. This is especially important in volatile markets, where short-term trends can quickly reverse.

9.3 Factor Risk Models#

Professional quant shops often use factor risk models (e.g., Barra or Northfield) to measure exposures to market, size, momentum, value, and other systematic risk factors. By plotting your portfolio’s exposure along these axes, you can identify unintended bets.

9.4 Portfolio Optimization Approach#

Classical approaches like Markowitz mean-variance optimization or advanced methods like Black-Litterman can be used to build or refine allocations. In Qlib, you could:

1. Generate expected returns from your alpha model.
2. Estimate covariance matrices of returns.
3. Solve an optimization problem to find an efficient frontier portfolio.

Below is a simplified conceptual snippet:

1
import numpy as np
2
import pandas as pd
3

4
expected_returns = pd.Series({  # hypothetical examples
5
    "AAPL": 0.02, "MSFT": 0.015, "TSLA": 0.025
6
})
7

8
cov_matrix = pd.DataFrame(
9
    [[0.01, 0.003, 0.002],
10
     [0.003, 0.02, 0.004],
11
     [0.002, 0.004, 0.03]],
12
    index=expected_returns.index, columns=expected_returns.index
13
)
14

15
# Markowitz function
16
def markowitz_optimize(returns, cov, risk_aversion=1):
17
    # Simplistic approach: w = (1/λ) * Σ^-1 * μ
18
    # where Σ is covariance, μ is expected returns, λ is risk aversion
19
    inv_cov = np.linalg.inv(cov.values)
20
    w = np.dot(inv_cov, returns.values) / risk_aversion
21
    w /= np.sum(np.abs(w))  # normalize
22
    return pd.Series(w, index=returns.index)
23

24
optimal_weights = markowitz_optimize(expected_returns, cov_matrix, risk_aversion=2)
25
print(optimal_weights)

In a real Qlib pipeline, your alpha model could produce daily or weekly expected returns, then feed these calculations into a dynamic optimizer.