Benchmark¶

Introduction¶

Benchmarking the capabilities of R&D is a crucial research problem in this area. We are continuously exploring methods to benchmark these capabilities. The current benchmarks are listed on this page.

Development Capability Benchmarking¶

Benchmarking is used to evaluate the effectiveness of factors with fixed data. It mainly includes the following steps:

read and prepare the eval_data
declare the method to be tested and pass the arguments
declare the eval method and pass the arguments
run the eval
save and show the result

Configuration¶

pydantic settings rdagent.components.benchmark.conf.BenchmarkSettings¶

Show JSON schema

{
   "title": "BenchmarkSettings",
   "type": "object",
   "properties": {
      "bench_data_path": {
         "default": "example.json",
         "format": "path",
         "title": "Bench Data Path",
         "type": "string"
      },
      "bench_test_round": {
         "default": 10,
         "title": "Bench Test Round",
         "type": "integer"
      },
      "bench_test_case_n": {
         "anyOf": [
            {
               "type": "integer"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Bench Test Case N"
      },
      "bench_method_cls": {
         "default": "rdagent.components.coder.factor_coder.FactorCoSTEER",
         "title": "Bench Method Cls",
         "type": "string"
      },
      "bench_method_extra_kwargs": {
         "additionalProperties": true,
         "title": "Bench Method Extra Kwargs",
         "type": "object"
      },
      "bench_result_path": {
         "default": "result",
         "format": "path",
         "title": "Bench Result Path",
         "type": "string"
      }
   },
   "additionalProperties": false
}

Config:

env_prefix: str = BENCHMARK_

Fields:

bench_data_path (pathlib.Path)
bench_method_cls (str)
bench_method_extra_kwargs (dict)
bench_result_path (pathlib.Path)
bench_test_case_n (int | None)
bench_test_round (int)

field bench_data_path: Path = PosixPath('example.json')¶: data for benchmark

field bench_method_cls: str = 'rdagent.components.coder.factor_coder.FactorCoSTEER'¶: method to be used for test cases

field bench_method_extra_kwargs: dict [Optional]¶: extra kwargs for the method to be tested except the task list

field bench_result_path: Path = PosixPath('result')¶: result save path

field bench_test_case_n: int | None = None¶: how many test cases to run; If not given, all test cases will be run

field bench_test_round: int = 10¶: how many rounds to run, each round may cost 10 minutes

class Config¶

env_prefix = 'BENCHMARK_'¶: Use BENCHMARK_ as prefix for environment variables

Example¶

The default value for bench_test_round is 10, which takes about 2 hours to run. To modify it from 10 to 2, adjust the environment variables in the .env file as shown below.

BENCHMARK_BENCH_TEST_ROUND=2

Data Format¶

The sample data in bench_data_path is a dictionary where each key represents a factor name. The value associated with each key is factor data containing the following information:

description: A textual description of the factor.
formulation: A LaTeX formula representing the model’s formulation.
variables: A dictionary of variables involved in the factor.
Category: The category or classification of the factor.
Difficulty: The difficulty level of implementing or understanding the factor.
gt_code: A piece of code associated with the factor.

Here is an example of this data format:

{
    "Turnover_Rate_Factor": {
        "description": "A traditional factor based on 20-day average turnover rate, adjusted for market capitalization, which is further improved by applying the information distribution theory.",
        "formulation": "\\text{Adjusted Turnover Rate} = \\frac{\\text{mean}(20\\text{-day turnover rate})}{\\text{Market Capitalization}}",
        "variables": {
            "20-day turnover rate": "Average turnover rate over the past 20 days.",
            "Market Capitalization": "Total market value of a company's outstanding shares."
        },
        "Category": "Fundamentals",
        "Difficulty": "Easy",
        "gt_code": "import pandas as pd\n\ndata_f = pd.read_hdf('daily_f.h5')\n\ndata = data_f.reset_index()\nwindow_size = 20\n\nnominator=data.groupby('instrument')[['TurnoverRate_30D']].rolling(window=window_size).mean().reset_index(0, drop=True)\n# transfer to series\nnew=nominator['TurnoverRate_30D']\ndata['Turnover_Rate_Factor']=new/data['TradableACapital']\n\n# set the datetime and instrument as index and drop the original index\nresult=pd.DataFrame(data['Turnover_Rate_Factor']).set_index(data_f.index)\n\n# transfer the result to series\nresult=result['Turnover_Rate_Factor']\nresult.to_hdf(\"result.h5\", key=\"data\")" 
    },
    "PctTurn20": {
        "description": "A factor representing the percentage change in turnover rate over the past 20 trading days, market-value neutralized.",
        "formulation": "\\text{PctTurn20} = \\frac{1}{N} \\sum_{i=1}^{N} \\left( \\frac{\\text{Turnover}_{i, t} - \\text{Turnover}_{i, t-20}}{\\text{Turnover}_{i, t-20}} \\right)",
        "variables": {
            "N": "Number of stocks in the market.",
            "Turnover_{i, t}": "Turnover of stock i at day t.",
            "Turnover_{i, t-20}": "Turnover of stock i at day t-20."
        },
        "Category": "Volume&Price",
        "Difficulty": "Medium",
        "gt_code": "import pandas as pd\nfrom statsmodels import api as sm\n\ndef fill_mean(s: pd.Series) -> pd.Series:\n    return s.fillna(s.mean()).fillna(0.0)\n\ndef market_value_neutralize(s: pd.Series, mv: pd.Series) -> pd.Series:\n    s = s.groupby(\"datetime\", group_keys=False).apply(fill_mean)\n    mv = mv.groupby(\"datetime\", group_keys=False).apply(fill_mean)\n\n    df_f = mv.to_frame(\"MarketValue\")\n    df_f[\"const\"] = 1\n    X = df_f[[\"MarketValue\", \"const\"]]\n\n    # Perform the Ordinary Least Squares (OLS) regression\n    model = sm.OLS(s, X)\n    results = model.fit()\n\n    # Calculate the residuals\n    df_f[\"residual\"] = results.resid\n    df_f[\"norm_resi\"] = df_f.groupby(level=\"datetime\", group_keys=False)[\"residual\"].apply(\n        lambda x: (x - x.mean()) / x.std(),\n    )\n    return df_f[\"norm_resi\"]\n\n\n# get_turnover\ndf_pv = pd.read_hdf(\"daily_pv.h5\", key=\"data\")\ndf_f = pd.read_hdf(\"daily_f.h5\", key=\"data\")\nturnover = df_pv[\"$money\"] / df_f[\"TradableMarketValue\"]\n\nf = turnover.groupby(\"instrument\").pct_change(periods=20)\n\nf_neutralized = market_value_neutralize(f, df_f[\"TradableMarketValue\"])\n\nf_neutralized.to_hdf(\"result.h5\", key=\"data\")"
    },
    "PB_ROE": {
        "description": "Constructed using the ranking difference between PB and ROE, with PB and ROE replacing original PB and ROE to obtain reconstructed factor values.",
        "formulation": "\\text{rank}(PB\\_t) - rank(ROE_t)",
        "variables": {
            "\\text{rank}(PB_t)": "Ranking PB on cross-section at time t.",
            "\\text{rank}(ROE_t)": "Ranking single-quarter ROE on cross-section at time t."
        },
        "Category": "High-Frequency",
        "Difficulty": "Hard",
        "gt_code": "#!/usr/bin/env python\n\nimport pandas as pd\n\ndata_f = pd.read_hdf('daily_f.h5')\n\ndata = data_f.reset_index()\n\n# Calculate the rank of PB and ROE\ndata['PB_rank'] = data.groupby('datetime')['B/P'].rank()\ndata['ROE_rank'] = data.groupby('datetime')['ROE'].rank()\n\n# Calculate the difference between the ranks\ndata['PB_ROE'] = data['PB_rank'] - data['ROE_rank']\n\n# set the datetime and instrument as index and drop the original index\nresult=pd.DataFrame(data['PB_ROE']).set_index(data_f.index)\n\n# transfer the result to series\nresult=result['PB_ROE']\nresult.to_hdf(\"result.h5\", key=\"data\")"
    }
}

Ensure the data is placed in the FACTOR_COSTEER_SETTINGS.data_folder_debug. The data files should be in .h5 or .md format and must not be stored in any subfolders. LLM-Agents will review the file content and implement the tasks.

Run Benchmark¶

Start the benchmark after completing the Installation and Configuration.

dotenv run -- python rdagent/app/benchmark/factor/eval.py

Once completed, a pkl file will be generated, and its path will be printed on the last line of the console.

Show Result¶

The analysis.py script reads data from the pkl file and converts it to an image. Modify the Python code in rdagent/app/quant_factor_benchmark/analysis.py to specify the path to the pkl file and the output path for the png file.

dotenv run -- python rdagent/app/benchmark/factor/analysis.py <log/path to.pkl>

A png file will be saved to the designated path as shown below.