`get_one_exchange_neighbourhood` raises instead of falling back to enumeration
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 86/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- python
- Domain
- machine-learning
Research direction
Start in ConfigSpace/hyperparameters/distributions.py at quantized_neighborhood and reproduce the Integer example from the issue with seed 3426. Verify that the sampling fallback returns eight distinct neighbours instead of raising, including for the listed edge-of-range cases, and add or run regression coverage if the repository provides a suitable test location.
Written by the indexing model from the issue text.
Description
Description
get_one_exchange_neighbourhood raises a ValueError when it cannot sample the requested number of distinct neighbours, although every neighbour it needs is available and could be enumerated cheaply.
This happens for a quantized hyperparameter whose current value sits at the edge of its range and that offers exactly one more neighbour than requested.
Steps to reproduce
from ConfigSpace import ConfigurationSpace, Configuration, Integer
from ConfigSpace.util import get_one_exchange_neighbourhood
space = ConfigurationSpace(seed=0)
space.add([Integer("n", (1, 10), default=1)]) # 10 values -> 9 possible neighbours
config = Configuration(space, values={"n": 1}) # at the lower edge of the range
# Asking for 8 of the 9 possible neighbours
list(get_one_exchange_neighbourhood(config, seed=3426, stdev=0.025, num_neighbors=8))
Actual result
ValueError: Failed to find enough neighbors with 8 retries. Given n=8 neighbors to
generate, we only found 7. The normal's for sampling neighbors were
Normal(0.0, [0.025, 0.146875, ..., 1.0]) which were meant to find neighbors of 0.0.
in the range (0.0, 1.0).
It is seed dependent: with this space it occurs for roughly 0.03 % of seeds (58 out of 200000 for seeds 0..199999). Failing seeds include 3426, 4096, 4389, 15390, 22771.
Expected result
Eight distinct neighbours. All nine are available, so the request is satisfiable.
Cause
quantized_neighborhood in ConfigSpace/hyperparameters/distributions.py chooses between enumerating and sampling:
n_available = bins - 1 # 10 values -> 9
if n >= n_available: # 8 >= 9 -> False
... # enumerate everything, cannot fail
# otherwise: repeated sampling from a normal, then
raise ValueError(...) # if not enough distinct values after the retries
Just below that cutoff the sampler has to collect almost every available value by drawing from a normal centred on the current one. When that value is at the edge of the range, about half of every sample batch falls outside [lower, upper] and is discarded, and with a small std the remainder concentrates on the nearest bins. After the retries one value is still missing.
The effect is limited to a narrow window — measured at the lower edge of Integer(1, c) with num_neighbors=8, 30000 seeds each:
| cardinality | available | branch | failures |
|---|---|---|---|
| 8, 9 | 7, 8 | enumerate | 0 |
| 10 | 9 | sample | 6 |
| 11 … 18 | 10 … 17 | sample | 0 |
It also only affects quantized hyperparameters: a Categorical with ten choices and a continuous Float both show 0 failures under the same conditions.
Suggested fix
Return what was found instead of raising. The data is already there, one line above the raise:
return neighbors[1:offset]
Suggested fix
Falling back to enumeration when sampling runs out of retries. Replacing the raise at the end of quantized_neighborhood:
# Sampling could not find `n` distinct neighbors within the retries. This happens when `n`
# is just below `n_available`, especially when `vector` sits at the edge of the range, where
# half of each sample batch falls outside the bounds. Fall back to enumerating every
# available neighbor, which is cheap in exactly this regime.
all_neighbors = np.setdiff1d(
np.arange(0, bins, dtype=f64) / (bins - 1),
np.array([qvector], dtype=f64),
)
seed.shuffle(all_neighbors)
return all_neighbors[:n]
Environment
- ConfigSpace 1.2.2 (note:
ConfigSpace.__version__still reports1.2.0) - Python 3.10, Linux
- Dominant language
- Python
- Stars
- 225
- Forks
- 95
- PR merge metrics
- No merged PRs in 30d
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from automl/ConfigSpace
-
Sigma erroneously scaling with lower bound.Possibly taken @angrinord claimed this 31 days ago. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
automl/ConfigSpace#431 ·
-
Equal values do not hash the samePossibly taken @angrinord claimed this 18 days ago. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 25/100
automl/ConfigSpace#434 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 58/100
automl/ConfigSpace#430 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 52/100
automl/ConfigSpace#429 · 3 comments ·
-
Configuration Space does not allow UpdatesMay be free again @thijssnelleman claimed this 250 days ago, and no pull request is open. Openenhancement
automl/ConfigSpace#412 · 7 comments · 1 assignee ·
All issues in automl/ConfigSpace
Similar issues
-
link-rot
Difficulty 1/5 Under an hour Newbie friendliness 67/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
developmentseed/titiler-cmr#265 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
Insecure Brotli version <1.2.1Possibly taken @jamescw19 claimed this today. Open
Difficulty 1/5 Under an hour Newbie friendliness 85/100
stac-utils/stac-fastapi-pgstac#438 · 1 reaction ·
-
bug triage
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
Maintainers usually reply within 1 day