Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

feat: enabling associating experiment results to a benchmark instance when number of instance properties exceeds experiment properties

Đang mở
  • Có gì không ổn?
#282 1 bình luận 0 reaction 1 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

@christian-pinto đang làm issue này rồi.

Từ ngày 29/9/2026.

Đánh giá

Issue này chưa được đánh giá.

Mô tả

enhancement

What is the problem

In the current implementation if a benchmark problem (logical benchmark) defines more instance properties than experiments need for executing a benchmark instance of that problem, the experiment results cannot be associated to the correct benchmark instance. Example

num_assets: 10
num_periods: 10
budget: 4
scenario: s00
risk_aversion: "1e-4"
market_data:
  artifacts_location: market_data
models:
  artifacts_location: models

For this problem instance experiments only need to take one of the "models" to run. However all the keys are needed for matching.

The reason is that it is

  • we define a unique "key" for each benchmark problem instance that is based on its {property-id:value} pairs
  • we use the experiment binding to form a similar key from each run of the experiment
  • we can then find all experiments which have same key as an instance

However this only works if the experiments have input properties that map to every instance property. If they don't one property is left out, the key is incomplete and matching fails.

Note: we can't set a static value for those properties in the experiment binding. This is because the binding describes how to create the key for all problem instances. Any static value for a property set their will only match instances with that exact value for the property - all others will never be matched.

Proposed solution

In many problems there is a split between descriptive instance properties and the instance formulation. The former are some properties that give information about the characteristics of the problem w.r.t the problem definition e.g. number of graph nodes, number of graph edges. The later are actual files defining a formulation of a problem from which the other properties can be read e.g. a json graph.

One solution is to allow defining which instance properties are descriptive (related to the problem class) and which contain the formulation (required for running a benchmark). Then the key can be created just on the formulation properties. i.e. we define the keys use for "routing".

descriptive_properties:
  num_assets: 10
  num_periods: 10
  budget: 4
  scenario: s00
  risk_aversion: "1e-4"
  market_data:
    artifacts_location: market_data
routing_properties
  models:
    artifacts_location: models

Some issues:

  • in certain situations, like above, this would reduce the key to a single file, which then has to be unique.
  • we have to pre-decide what is descriptive - perhaps some descriptive properties could be taken as inputs.

Additional solutions considered

The underlying problem is how to map experiment runs to problem instances.

  1. Benchmark submission specify the target problem instance by id (instead of keys)

In this way it's known what the instance is for filtering and association purposes. We still need a binding but then only needed to show the correct values for the problem instance fields for each instance. If a value is not present in the binding it takes the instance default. We don't need to use keys

  1. Bind to benchmark instances not to benchmark problems

Instead of binding experiments to logical benchmarks they bind to problem instances.

  • This binding can be at same level as logical benchmark -> one binding per instance - we can keep key approach as we are able to make it directly.
  • This binding can be in each submission.

So we have to choose between 1 binding per problem class -> 1 binding per problem instance -> 1 binding per submission

Additional Context

In all cases we need a binding file. A question is if this should be used to create a key for identifying runs for an instance OR we use directly an identifier.

The need for "descriptive properties" is one cause of this issue -> previously we understood all the logical benchmark properties would be inputs. Now it's not clear cut.

Ngôn ngữ chính
Python
Star
3
Fork
7
Merge trung bình
15 giờ 34 phút
Pull request đã merge (30 ngày)
18

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của IBM/algorithm-nexus

Tất cả issue của IBM/algorithm-nexus

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.