[Task/Feature/Issue] Use AI to generate better tags/keywords
#154 opened on May 16, 2025
Repository metrics
- Stars
- (16 stars)
- PR merge metrics
- (PR metrics pending)
Description
Is there an existing issue for this?
- I have searched the existing issues
Task/Feature/Issue Details
The metadata in HF datasets is (presumably) added manually by the owners. This means it is often going to be 1) wrong, 2) incomplete, 3) use less useful synonyms that the most common keyword for a concept. For the current static catalog, I manually associated keywords that were related into the same tables. (see: https://github.com/The-AI-Alliance/open-trusted-data-initiative/blob/main/static-catalog/src/scripts/write-category-files.py)
This issue proposes analyzing datasets with LLM classifiers to categorize them using a standard set of keywords, which we can either define or adopt from another source. (For example, the advertising industry defines categories and terms like this.) The metadata keywords should still be displayed to the user, but the derived keywords can be used for searching, hierarchical displays, etc.
Related: should we create a knowledge graph or at least a topic hierarchy using this approach, not just a flat set of canonical keywords?
Additional Context
No response