Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

feat: enabling associating experiment results to a benchmark instance when number of instance properties exceeds experiment properties

クローズ
  • 何か問題がありますか？
#282 コメント 2 件 リアクション 0 件 担当者 1 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

@christian-pinto がすでに取り組んでいます。

2026年9月29日 から。

評価

この issue はまだ評価されていません。

説明

enhancement

What is the problem

In the current implementation if a benchmark problem (logical benchmark) defines more instance properties than experiments need for executing a benchmark instance of that problem, the experiment results cannot be associated to the correct benchmark instance. Example

num_assets: 10
num_periods: 10
budget: 4
scenario: s00
risk_aversion: "1e-4"
market_data:
  artifacts_location: market_data
models:
  artifacts_location: models

For this problem instance experiments only need to take one of the "models" to run. However all the keys are needed for matching.

The reason is that it is

  • we define a unique "key" for each benchmark problem instance that is based on its {property-id:value} pairs
  • we use the experiment binding to form a similar key from each run of the experiment
  • we can then find all experiments which have same key as an instance

However this only works if the experiments have input properties that map to every instance property. If they don't one property is left out, the key is incomplete and matching fails.

Note: we can't set a static value for those properties in the experiment binding. This is because the binding describes how to create the key for all problem instances. Any static value for a property set their will only match instances with that exact value for the property - all others will never be matched.

Proposed solution

In many problems there is a split between descriptive instance properties and the instance formulation. The former are some properties that give information about the characteristics of the problem w.r.t the problem definition e.g. number of graph nodes, number of graph edges. The later are actual files defining a formulation of a problem from which the other properties can be read e.g. a json graph.

One solution is to allow defining which instance properties are descriptive (related to the problem class) and which contain the formulation (required for running a benchmark). Then the key can be created just on the formulation properties. i.e. we define the keys use for "routing".

descriptive_properties:
  num_assets: 10
  num_periods: 10
  budget: 4
  scenario: s00
  risk_aversion: "1e-4"
  market_data:
    artifacts_location: market_data
routing_properties
  models:
    artifacts_location: models

Some issues:

  • in certain situations, like above, this would reduce the key to a single file, which then has to be unique.
  • we have to pre-decide what is descriptive - perhaps some descriptive properties could be taken as inputs.

Additional solutions considered

The underlying problem is how to map experiment runs to problem instances.

  1. Benchmark submission specify the target problem instance by id (instead of keys)

In this way it's known what the instance is for filtering and association purposes. We still need a binding but then only needed to show the correct values for the problem instance fields for each instance. If a value is not present in the binding it takes the instance default. We don't need to use keys

  1. Bind to benchmark instances not to benchmark problems

Instead of binding experiments to logical benchmarks they bind to problem instances.

  • This binding can be at same level as logical benchmark -> one binding per instance - we can keep key approach as we are able to make it directly.
  • This binding can be in each submission.

So we have to choose between 1 binding per problem class -> 1 binding per problem instance -> 1 binding per submission

Additional Context

In all cases we need a binding file. A question is if this should be used to create a key for identifying runs for an instance OR we use directly an identifier.

The need for "descriptive properties" is one cause of this issue -> previously we understood all the logical benchmark properties would be inputs. Now it's not clear cut.

主要言語
Python
スター
4
フォーク
7
平均マージ
17時間 4分
マージ済み PR（30日）
29

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

IBM/algorithm-nexus のほかの issue

IBM/algorithm-nexus の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。