Building Alpha-Focused Portfolios with Qlib Quant#

1.1 What is Alpha?#

Alpha is the component of a stock’s (or any asset’s) return that cannot be explained by broader market movements. If the equity market as a whole rises by 5%, and your portfolio gains 7%, then the “excess” 2% could be considered your alpha. Specifically:

• Beta captures the portion of movement explained by overall market movements.
• Alpha focuses on idiosyncratic contributions—returns unique to the security or strategy itself.

1.2 Measuring Alpha in a Quantitative Context#

In a quantitative setting, alpha is often captured by models that forecast future returns or performance indicators. These might be built on:

• Simple signals (e.g., ratio-based fundamental metrics like P/E, P/B).
• Price-derived metrics (e.g., momentum, volatility).
• Machine learning models using alternative data sources (e.g., news sentiment, web data).

Performance is then evaluated through backtesting on historical data, where you compare your predicted returns with actual returns. The more accurate your alpha estimates, the higher your strategy’s probability of outperformance—assuming you also manage risk effectively.

1.3 Why Use Qlib?#

Qlib is an open-source library designed for quantitative researchers who want to easily handle large-scale data, build alpha factors, integrate machine learning models, and perform robust experiments. Some key benefits include:

• Data infrastructure: Efficient data loading, cleaning, and manipulation for large-scale market data.
• Extensive factor library: Built-in factors and utilities for generating new ones.
• ML integration: Facilities for model training, hyperparameter tuning, evaluation.
• Modularity: Each part of the pipeline (data, alpha factors, model evaluation) is loosely coupled, allowing you to adapt or replace individual modules without rewriting everything else.

2.1 Installation#

Qlib is available via PyPI. Most commonly, you can install it using:

1
pip install pyqlib

Alternatively, if you want the latest features, consider installing directly from the GitHub repository:

1
git clone https://github.com/microsoft/qlib.git
2
cd qlib
3
pip install -r requirements.txt
4
python setup.py install

2.2 Setting Up Offline Data#

Qlib needs market data to run its analyses. If you’ve never operated a historical database or data service before, Qlib simplifies this process. You can download built-in data for publicly traded stocks in several markets. For example, to prepare the offline dataset for the Chinese stock market, run:

1
# Inside the Qlib repository
2
python scripts/get_data.py qlib_data_cn --target_dir ~/.qlib/qlib_data/cn_data --interval=1d

Qlib also supports custom data. You can ingest your own CSV files (or another data source) as long as you follow the required data format. This flexibility is particularly powerful if you want to incorporate alternative datasets.

Data Directory Structure (Example):

2.3 Initializing Qlib#

Before accessing data or running any experiment, Qlib needs to be initialized in your notebook or Python script. For example:

1
import qlib
2
from qlib.config import C
3

4
# Initialize Qlib
5
qlib.init(provider_uri='~/.qlib/qlib_data/cn_data',
6
          region='cn',
7
          expression_cache=None,
8
          dataset_cache=None)

Replacing the provider URI with your own data path is often sufficient. Without further configuration, Qlib will load daily data from your local directory.

3.1 Qlib’s Data Abstraction#

Qlib organizes data into a multi-dimensional structure with the following building blocks:

• Instrument (Stock): A ticker or identifier.
• Fields: Data attributes such as “Close,” “Volume,” or “Factor1.”
• Calendar: A trading day index.

The library’s design allows you to slice and dice data easily—fetching, for example, the closing prices for 100 instruments over the last 30 trading days.

3.2 A Quick Data Retrieval Example#

Below is a snippet demonstrating how you can retrieve raw bar data (like OHLCV) for a single stock, for a specific time range:

1
from qlib.data import D
2
import datetime
3

4
# Specify time range
5
start_time = '2020-01-01'
6
end_time = '2020-12-31'
7
symbol = 'SH600519'
8

9
# Fetch data
10
data_df = D.features(
11
    instruments=[symbol],
12
    fields=['$close', '$high', '$low', '$volume'],
13
    start_time=start_time,
14
    end_time=end_time
15
)
16

17
print(data_df.head())

The code requests four fields—close, high, low, and volume—for the instrument “SH600519” (Kweichow Moutai in the Chinese market) from January 1, 2020 to December 31, 2020. Notice the $ prefix in the field names, which indicates raw market data.

4.1 Simple Factors#

A simple ex-post “momentum” factor might be defined as the percentage change in closing price over the previous 5 days. In Qlib, you can express it with a straightforward expression, or you can define your own factor function.

Example of a Momentum Factor (5-day return):

1
import pandas as pd
2

3
# daily_prices is a DataFrame of daily closing prices
4
# We'll shift prices by 5 days to compute 5-day returns
5
daily_prices['momentum_5d'] = (daily_prices['$close'] / daily_prices['$close'].shift(5)) - 1

By itself, this factor simply captures whether the closing price has increased over a short window. In a cross-sectional strategy, one might expect stocks showing higher momentum to continue to outperform over the near term. This is a classic factor from momentum investing research.

4.2 Built-in Expressions and Features#

Qlib comes pre-packaged with expressions like Mean, StdDev, Ref (shift data backward), and many others. You can combine them to define more advanced transformations. For instance, you could create a volatility factor:

1
from qlib.data.dataset.loader import StaticDataLoader
2
from qlib.data import D
3

4
vol_20d = D.features(
5
    instruments=['SH600519'],
6
    fields=['Ref($close, 1) / $close - 1', 'StdDev($close, 20)'],
7
    start_time='2020-01-01',
8
    end_time='2020-12-31'
9
)

This instructs Qlib to compute two columns: a 1-day backward reference ratio and a 20-day standard deviation of closing prices. You can then use these columns in further calculations or as direct alpha signals.

5.1 Setting Up a Prediction Task#

In a typical alpha prediction task, you want to predict a short-term or medium-term return, say 5-day forward return. For each instrument and date in your dataset, your target variable might be something like:

1
future_return_5d = (close_price(t+5) - close_price(t)) / close_price(t)

Then your alpha factors become features in a supervised learning problem, where:

• Features = factor values at time t
• Target = future return (e.g., 5-day forward return)

5.2 Workflow Outline#

1. Feature engineering: Build your library of factors (momentum, volatility, valuation ratios, etc.).
2. Alignment: Align factors and forward returns so each row in your dataset corresponds to the same time index.
3. Model training: Train a regression model to predict forward returns using the factors.
4. Evaluation: Backtest on historical data that the model has not seen.

6.1 Data Preparation#

Qlib supports a dataset concept, which brings together the alpha factors (features) and future returns (labels) for each instrument over time. For example:

1
import qlib
2
from qlib.contrib.data.handler import Alpha158
3
from qlib.contrib.dataset import Alpha158Dataset
4

5
qlib.init(provider_uri='~/.qlib/qlib_data/cn_data', region='cn')
6

7
market = "csi300"
8
benchmark = "SH000300"
9

10
# Specifically, we can use built-in alpha158 dataset
11
dataset = Alpha158Dataset(
12
    handler_kwargs={
13
        "start_time": "2017-01-01",
14
        "end_time": "2020-12-31",
15
        "freq": "day",
16
        "instruments": market
17
    },
18
    segments={
19
        "train": ("2017-01-01", "2019-06-30"),
20
        "valid": ("2019-07-01", "2019-12-31"),
21
        "test": ("2020-01-01", "2020-12-31")
22
    }
23
)

• Alpha158 is one of Qlib’s built-in factor libraries that calculates 158 commonly used factors.
• Segments define training, validation, and testing periods.

6.2 Model Training#

Qlib offers helper modules for training. One approach is to use the built-in machine learning trainers, or you can integrate libraries like scikit-learn, LightGBM, XGBoost, or PyTorch. Here’s an example with a LightGBM model:

1
from qlib.utils import init_instance_by_config
2

3
# Define a LightGBM task configuration
4
task = {
5
    "model": {
6
        "class": "LGBModel",
7
        "module_path": "qlib.contrib.model.gbdt",
8
        "kwargs": {
9
            "num_leaves": 64,
10
            "max_depth": -1,
11
            "learning_rate": 0.01,
12
            "n_estimators": 500,
13
        }
14
    },
15
    "dataset": dataset
16
}
17

18
# Initialize and train the model
19
model = init_instance_by_config(task["model"])
20
model.fit(dataset)

6.3 Evaluating the Model#

After training, you can generate predictions on the validation or test dataset:

1
import pandas as pd
2

3
predictions = model.predict(dataset, segment="test")
4
predictions_df = pd.DataFrame(predictions, columns=["score"])

Here, predictions_df will contain the model’s alpha scores (predicted returns) for each instrument-date pair in the test period. The next step is backtesting these predictions.

7.1 The Concept of Backtesting#

A backtest simulates how your strategy would have performed historically using out-of-sample data. It applies your model’s alpha signals to create daily (or weekly) portfolios and tracks their returns over time.

7.2 Qlib’s Backtest Framework#

Qlib includes a backtesting module that can handle portfolio generation based on predicted scores:

1. Signal: The output from your alpha model (predicted score or expected return).
2. Backtest rules: Including rebalancing frequency, position size, and transaction costs.
3. Performance metrics: Such as annualized return, Sharpe ratio, max drawdown, and turnover.

You can configure these through a simple dictionary:

1
from qlib.contrib.strategy.strategy import TopkDropoutStrategy
2
from qlibBacktest import backtest
3

4
strategy_config = {
5
    "topk": 50,
6
    "dropout_n": 10,
7
    "holding_period": 5,
8
    "signal": "score",  # The column in predictions_df
9
}
10

11
analysis_config = {
12
    "excess_return": False,
13
    "risk_free": 0.02,  # annual risk free rate
14
}
15

16
# Convert predictions to a Qlib signal DataFrame
17
# Format: MultiIndex (datetime, instrument) with "score"
18
pred_scores = predictions_df.copy()
19
pred_scores.index = pd.MultiIndex.from_tuples(
20
    [(row['datetime'], row['instrument']) for idx, row in predictions_df.iterrows()]
21
)
22
pred_scores.columns = [strategy_config['signal']]
23

24
backtest_result = backtest(
25
    pred_scores,
26
    TopkDropoutStrategy(**strategy_config),
27
    **analysis_config
28
)
29

30
print(backtest_result)

In this example, the TopkDropoutStrategy invests in the top 50 instruments by predicted alpha score, dropping 10 from the old positions if they fall out of the top ranks upon rebalancing. The holding_period means you keep each position for 5 days before rebalancing again or exiting.

9.1 Risk Management#

Risk management involves limiting portfolio exposure to undesirable factors, e.g., sector, size, or style biases. In Qlib, you can integrate risk models and constraints during the portfolio construction phase. For instance:

• Volatility targeting: Scale position sizes so your portfolio volatility remains near a target.
• Factor neutralization: Neutralize undesired exposure to factors like sector or market beta.
• Leverage constraints: Control your gross or net positions to mitigate extreme exposures.

9.2 Factor Blending and Orthogonalization#

When building multiple alpha factors, you can:

• Blend: Combine them through weighted averaging or a machine learning model that ingests all factors at once.
• Orthogonalize: Remove overlap between factors by regressing one factor on another and retaining only the residual. This ensures that factors capture distinct alpha sources.

9.3 Hyperparameter Tuning and Cross-Validation#

Quant models benefit from thorough hyperparameter tuning. Techniques like cross-validation or walk-forward analysis ensure your factor or ML model generalizes well. Qlib provides:

• Rolling or expanding window cross-validation to test your alpha signals over different historical regimes.
• Parameter grid or Bayesian optimization to systematically search for the best model parameters.

9.4 Alternative Data#

In addition to price, volume, and fundamental data, you can incorporate alternative datasets (sentiment, satellite imagery, supply chain data, etc.) to achieve unique alpha. The flexible Qlib data handler architecture allows you to fuse new data sources with standard features.

10.1 Environment Setup#

1
# 1. Environment Setup
2
import qlib
3
from qlib.config import C
4
from qlib.data import D
5

6
qlib.init(provider_uri='~/.qlib/qlib_data/cn_data', region='cn')

10.2 Defining Factors#

1
import pandas as pd
2
import numpy as np
3

4
# 2. Factor Engineering
5
def momentum_factor(df, window=5):
6
    return df['$close'] / df['$close'].shift(window) - 1
7

8
def volatility_factor(df, window=5):
9
    return df['$close'].rolling(window).std()

10.3 Creating a Dataset#

We will fetch daily bars for a particular time range and apply our factors:

1
# 3. Create a dataset for multiple instruments
2
instruments = D.list_instruments(D.instruments_d, filter_pipe=[("market", "==", "csi300")])
3
start_date = '2018-01-01'
4
end_date = '2020-12-31'
5

6
data_dict = {}
7
for inst in instruments:
8
    df = D.features(
9
        instruments=[inst],
10
        fields=['$close'],
11
        start_time=start_date,
12
        end_time=end_date
13
    )
14
    # Apply factors
15
    df['momentum_5d'] = momentum_factor(df, 5)
16
    df['volatility_5d'] = volatility_factor(df, 5)
17
    # Drop NaN data at the beginning
18
    df.dropna(inplace=True)
19
    data_dict[inst] = df

10.4 Constructing Training Samples#

We want to predict 5-day forward returns, so let’s add the label column.

1
def future_return_5d(df):
2
    return df['$close'].shift(-5) / df['$close'] - 1
3

4
all_data = []
5
for inst, df in data_dict.items():
6
    df['future_ret_5d'] = future_return_5d(df)
7
    # Shift factor columns to avoid lookahead bias
8
    # The factors at time t should not include price info from t+1 or beyond.
9
    # In many cases, just dealing with daily data is enough if no future overlap is present.
10

11
    # Prepare DataFrame
12
    temp = df[['momentum_5d', 'volatility_5d', 'future_ret_5d']].dropna()
13
    temp['instrument'] = inst
14
    all_data.append(temp.reset_index())
15

16
combined_df = pd.concat(all_data, ignore_index=True).dropna()

Now we have a combined DataFrame with the key factor data and labels for each instrument-date.

10.5 Split into Train/Validation/Test#

We’ll adopt a time-based partition:

1
train_end = '2019-06-30'
2
valid_end = '2019-12-31'
3

4
train_data = combined_df[combined_df['datetime'] <= train_end]
5
valid_data = combined_df[(combined_df['datetime'] > train_end) & (combined_df['datetime'] <= valid_end)]
6
test_data = combined_df[combined_df['datetime'] > valid_end]
7

8
features = ['momentum_5d', 'volatility_5d']
9
label = 'future_ret_5d'

10.6 Training a LightGBM Model#

1
import lightgbm as lgb
2

3
train_x = train_data[features]
4
train_y = train_data[label]
5

6
valid_x = valid_data[features]
7
valid_y = valid_data[label]
8

9
test_x = test_data[features]
10
test_y = test_data[label]
11

12
lgb_train = lgb.Dataset(train_x, train_y)
13
lgb_valid = lgb.Dataset(valid_x, valid_y, reference=lgb_train)
14

15
params = {
16
    'objective': 'regression',
17
    'metric': 'rmse',
18
    'learning_rate': 0.01,
19
    'num_leaves': 32,
20
    'verbose': -1
21
}
22

23
gbm = lgb.train(
24
    params,
25
    lgb_train,
26
    num_boost_round=2000,
27
    valid_sets=[lgb_train, lgb_valid],
28
    early_stopping_rounds=50
29
)

10.7 Generating Predictions#

1
test_data['pred_score'] = gbm.predict(test_x, num_iteration=gbm.best_iteration)

10.8 Backtesting the Predictions#

To backtest, we convert test_data into a signal table for each date-instrument pair. Then, we feed it into a strategy akin to a top-k approach:

1
# 1. Sort predictions by date, then by score
2
test_data_sorted = test_data.sort_values(by=['datetime', 'pred_score'], ascending=[True, False])
3

4
# 2. For each date, pick the top 50 instruments
5
top_k = 50
6
portfolio_records = []
7

8
for date, group in test_data_sorted.groupby('datetime'):
9
    group_topk = group.head(top_k)
10
    # Simple approximation: average returns of these top picks
11
    avg_return = group_topk[label].mean()
12
    portfolio_records.append([date, avg_return])
13

14
results_df = pd.DataFrame(portfolio_records, columns=['date', 'strategy_return'])
15
results_df.set_index('date', inplace=True)
16

17
# 3. Calculate cumulative performance
18
results_df['cumulative_return'] = (1 + results_df['strategy_return']).cumprod()

While this is not leveraging Qlib’s built-in backtest suite directly, it demonstrates a simplified conceptual approach. For a more robust test, integrate Qlib’s modules for weighting, cost accounting, rebalancing, and risk constraints.

10.9 Reviewing Performance#

1
import matplotlib.pyplot as plt
2

3
plt.figure(figsize=(10,6))
4
results_df['cumulative_return'].plot()
5
plt.title('Strategy Cumulative Return')
6
plt.xlabel('Date')
7
plt.ylabel('Cumulative Return')
8
plt.show()
9

10
# Evaluate final performance
11
cagr = results_df['cumulative_return'].iloc[-1] ** (252.0 / len(results_df)) - 1
12
print("Approx. Annualized Return:", cagr)
13

14
# Additional metrics can be integrated here

11.1 Custom Docker Environments#

For institutional usage, encapsulate your entire Qlib environment and dependencies in Docker containers. This improves reproducibility and consistency across different servers or cloud platforms.

11.2 CI/CD for Quant Research#

You can set up continuous integration (CI) workflows that automatically run tests and backtests when you modify alpha factors or update your code. This fosters a systematic, incremental approach to developing trading strategies.

11.3 Deployment and Live Trading#

While Qlib’s main focus is research, you can adapt your pipeline for live trading by:

• Scheduling factor/model calculations to run at specific times (e.g., pre-market).
• Pushing generated signals or recommended trades to an execution system.
• Monitoring real-time risk exposures and performance metrics.