next-exp/IC

Keep file information when loading many files

Open

#731 创建于 2020年7月9日

在 GitHub 查看
 (2 评论) (0 反应) (0 负责人)Python (77 fork)auto 404
easygood first issuehelp wanted

仓库指标

Star
 (10 star)
PR 合并指标
 (PR 指标待抓取)

描述

Currently, our concatenation operation on dataframes doesn't remember which rows came from which file. It would be nice to keep this information. This is easy to do in by just modifying load_dsts. However, the design of the output is not obvious. We could

  1. Simply add a column with the file name. This is the simplest solution, but it adds a lot of noise and unnecessary data to the dst. Large dsts may become to heavy to be held in memory.
  2. Add a smart file indexer that maybe adds just the index of the file being loaded. This relies heavily on the file being read to be in the standard filename format and on this format not changing. Might be a pain in the future.
  3. Make the output more complicated. Add only file_index column to the main dataframe, but return also a second dataframe that associates file_index to the actual file name. This seems the most robust solution, but also the most annoying for the user, maybe.
  4. Do something else

People interested in this should comment what their preference is or propose other schemes.

贡献者指南