Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

CategoricalDtype can be lost if addressing several values

Open
  • Is something wrong?
#349 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
30/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
pandas, python
Domain
data

Research direction

Start by reproducing the filewise and segmented examples at the db['files'].get() and db['segments'].get() entry points, then compare the multi-value assignment with the single-value assignments. The issue is resolved when assigning several values no longer silently removes CategoricalDtype and forbidden labels remain rejected, with behavior confirmed for both table types.

Written by the indexing model from the issue text.

Description

This is similar to #324, but the underlying problem seems to be a pandas issue.

Let's start with a filewise and a segmented table, both using a 'spk' scheme with the labels 'a' and 'b', and both containing two entries labeled as 'a'.

import audformat


db = audformat.Database('db')

db.schemes['spk'] = audformat.Scheme('str', labels=['a', 'b'])
index = audformat.filewise_index(['f1', 'f2'])
db['files'] = audformat.Table(index)
db['files']['spk'] = audformat.Column(scheme_id='spk')
db['files']['spk'].set(['a', 'a'])

db.schemes['label'] = audformat.Scheme('int')
index = audformat.segmented_index(['f1', 'f1'], [0, 1], [1, 2])
db['segments'] = audformat.Table(index)
db['segments']['spk'] = audformat.Column(scheme_id='spk')
db['segments']['spk'].set(['a', 'a'])

The following behaves as expected:

>>> df = db['files'].get()
>>> df.spk.cat.categories
Index(['a', 'b'], dtype='object')
>>> df.loc['f1', 'spk'] = 'c'
...
TypeError: Cannot setitem on a Categorical with a new category (c), set the categories first
>>> df.iloc[0, 0] = 'c'
...
TypeError: Cannot setitem on a Categorical with a new category (c), set the categories first

>>> df = db['segments'].get()
>>> df.spk.cat.categories
 Index(['a', 'b'], dtype='object')
>>> df.loc[audformat.segmented_index(['f1'], [0], [1]), 'spk'] = 'c'
...
TypeError: Cannot setitem on a Categorical with a new category (c), set the categories first
>>> df.iloc[0, 0] = 'c'
...
TypeError: Cannot setitem on a Categorical with a new category (c), set the categories first

But we can still force to set a forbidden label and remove CategoricalDtype by addressing several values at once:

>>> df = db['files'].get()
>>> df.loc[:, 'spk'] = 'c'
>>> df.spk.cat.categories
...
AttributeError: Can only use .cat accessor with a 'category' dtype
>>> df
     spk
file    
f1     c
f2     c

>>> df = db['segments'].get()
>>> df.loc[:, 'spk'] = 'c'
>>> df.spk.cat.categories
...
AttributeError: Can only use .cat accessor with a 'category' dtype
>>> df
                                     spk
file start           end                
f1   0 days 00:00:00 0 days 00:00:01   c
     0 days 00:00:01 0 days 00:00:02   c

I'm not sure yet if this is considered a feature or a bug in pandas.

There is no upstream issue that matches directly, but related issues: https://github.com/pandas-dev/pandas/issues/46820, https://github.com/pandas-dev/pandas/issues/40080

Dominant language
Python
Stars
13
Forks
1
Avg merge
50m
Merged PRs (30d)
2

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from audeering/audformat

All issues in audeering/audformat

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.