Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

How to use mutli-column functions in Patsy which return categorical output?

Open
  • Is something wrong?
#79 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
25/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
numpy, pandas, python
Domain
data

Research direction

Start with the patsy.dmatrix call and the custom multival function shown in the issue, then review the linked duplicate in statsmodels/statsmodels#2843. The work is complete when custom multi-column categorical output exposes meaningful level-based column names instead of numeric indices.

Written by the indexing model from the issue text.

Description

I am dealing with a situation where each item can have 1 or 2 labels both of which come from the same set. In this case I only want to have 1 set of categorical variables which encode if the values of each level are present or absent with respect to the reference. I also want to inversely weight the values of these categorical variables based on how many labels are present.
E.g. Let my categories be {A, B, C}

data = pd.DataFrame({"col1": ["A", "A", "B"], "col2": ["O", "B", "C"], "num_vals": [1,2,2]})

def inverse_val(x):
    return 1.0/x

# Using categorical coding will not give the correct output
X = patsy.dmatrix("(C(col1) + C(col2)):inverse_val(num_vals)", data, return_type="dataframe")

I tried to solve this issue using the following code:

def multival(*x, **kwargs):
  #raise Exception("Not Implemented")
  levels, reference = kwargs.get("levels", None), kwargs.get("reference", None)
  weights = kwargs.get("weights", None)
  if len(x[0].shape) != 1:
    raise Exception("Mismatching Shapes. All arrays should be 1d and should have the same shape")
  for k in x:
    if k.shape != x[0].shape:
      raise Exception("Mismatching Shapes. All arrays should be 1d and should have the same shape")
  if levels is None:
    levels = np.sort(np.unique(np.hstack(x))) # Sort the unique values and then use this ordering as levels
  if reference is None:
    reference = levels[0]
  #print "Levels: %s, reference: %s" % (levels, reference)
  levels = levels[levels != reference] # Remove reference from levels
  level_len = len(levels)
  #print x[0].shape[0], level_len
  out = np.zeros((x[0].shape[0], level_len))
  for i, v in enumerate(levels):
    # print i, v
    for col in x:
      out[np.where(np.array(col) == v), i] = 1
  #print "Created matrix with shape: ", out.shape
  colnames = ["T.%s" % k for k in levels]
  if weights is not None:
    weights = weights.values
    return pd.DataFrame(out, columns=colnames).divide(weights, axis=0)

  return pd.DataFrame(out, columns=colnames)

X = patsy.dmatrix("multival(col1, col2, weights=num_vals)", data, return_type="dataframe")

There is no way of identifying the column names generated by patsy:

E.g. The names are like this:

multival(col1, col2, weights=num_vals, reference='O')[0]
multival(col1, col2, weights=num_vals, reference='O')[1]
multival(col1, col2, weights=num_vals, reference='O')[2]

I would much rather prefer column names like the following:

multival(col1, col2, weights=num_vals, reference='O')[T.0]
multival(col1, col2, weights=num_vals, reference='O')[T.1]
multival(col1, col2, weights=num_vals, reference='O')[T.2]

OR if I am passing the levels variable:

multival(col1, col2, weights=num_vals, reference='O')[T.A]
multival(col1, col2, weights=num_vals, reference='O')[T.B]
multival(col1, col2, weights=num_vals, reference='O')[T.C]

Is there a way to achieve this in patsy ?

Duplicate of statsmodels/statsmodels#2843

Dominant language
Python
Stars
990
Forks
106
Avg merge
7d 34m
Merged PRs (30d)
1

Getting set up

We have not checked this project's setup files yet. Start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from pydata/patsy

All issues in pydata/patsy

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.