Quick Start

Install tsbenchmark with command pip:

pip install tsbenchmark

The document describes how to define a player and run the benchmark. An example of training a prophet model task is shown below.

1.Firstly, create a directory named prophet_player. Then create the subfile player.yaml which describes the way to build an operating environment. The example below says conda virtual environment:

env:
  venv:
    kind: conda  # use conda to create a virtual environment
  requirements:
    kind: conda_yaml  # use the yaml file generated by conda to configure the virtual environment
    config:
      file_name: env.yaml 

tasks:  # state that the player only supports univariate forecast task
  - univariate-forecast

2.Create another subfile env.yaml which defines the configurations of the virtual environment, under directory prophet_player.

name: tsb_prophet_player
channels:
  - defaults
  - conda-forge
dependencies:
  - prophet
  - pip:
      - tsbenchmark

3.Create the third subfile exec.py under directory prophet_player to perform the training task.

from prophet import Prophet

import tsbenchmark as tsb
import tsbenchmark.api


def main():
    task = tsb.api.get_task()
    print(task)
    m = Prophet()
    m.fit(df)
    future = m.make_future_dataframe(periods=365)
    report_data = {'reward': 0.7}
    tsb.api.report_task(report_data=report_data)

if __name__ == "__main__":
    main()

NOTE: The operating environment of players are created by conda, please firstly install conda to /opt/miniconda3.

4.Create the configuration file benchmark.yaml parallel to the main directory:

name: 'benchmark_example'
desc: 'local benchmark run prophet'

kind: local

players:
  - ./prophet_player

datasets:
  filter:
    tasks:
      - univariate-forecast
    data_sizes:
      - small

random_states: [ 23163 ]

constraints:
  task:
    reward_metric: rmse

venv:
  conda:
    home: /opt/miniconda3

So far, the directory structure looks like below:

.
├── benchmark.yaml
└── prophet_player
    ├── env.yaml
    ├── exec.yaml
    └── player.yaml

5.Run this benchmark by the command below:

$ tsb run --config ./benchmark.yaml

When the benchmark execution ends, an experiment report is generated under directory ./report.