Quick Start
Install tsbenchmark
with command pip
:
pip install tsbenchmark
The document describes how to define a player and run the benchmark. An example of training a prophet model task is shown below.
1.Firstly, create a directory named prophet_player
. Then create the subfile player.yaml
which describes the way to build an operating environment. The example below says conda virtual environment:
env:
venv:
kind: conda # use conda to create a virtual environment
requirements:
kind: conda_yaml # use the yaml file generated by conda to configure the virtual environment
config:
file_name: env.yaml
tasks: # state that the player only supports univariate forecast task
- univariate-forecast
2.Create another subfile env.yaml
which defines the configurations of the virtual environment, under directory prophet_player
.
name: tsb_prophet_player
channels:
- defaults
- conda-forge
dependencies:
- prophet
- pip:
- tsbenchmark
3.Create the third subfile exec.py
under directory prophet_player
to perform the training task.
from prophet import Prophet
import tsbenchmark as tsb
import tsbenchmark.api
def main():
task = tsb.api.get_task()
print(task)
m = Prophet()
m.fit(df)
future = m.make_future_dataframe(periods=365)
report_data = {'reward': 0.7}
tsb.api.report_task(report_data=report_data)
if __name__ == "__main__":
main()
NOTE: The operating environment of players are created by conda
, please firstly install conda to /opt/miniconda3
.
4.Create the configuration file benchmark.yaml
parallel to the main directory:
name: 'benchmark_example'
desc: 'local benchmark run prophet'
kind: local
players:
- ./prophet_player
datasets:
filter:
tasks:
- univariate-forecast
data_sizes:
- small
random_states: [ 23163 ]
constraints:
task:
reward_metric: rmse
venv:
conda:
home: /opt/miniconda3
So far, the directory structure looks like below:
.
├── benchmark.yaml
└── prophet_player
├── env.yaml
├── exec.yaml
└── player.yaml
5.Run this benchmark by the command below:
$ tsb run --config ./benchmark.yaml
When the benchmark execution ends, an experiment report is generated under directory ./report
.