The-AI-Alliance/open-trusted-data-initiative

[Task/Feature/Issue] Use AI to generate better tags/keywords

オープン

#154 opened on 2025/05/16

 (0 件のコメント) (0 件のリアクション) (2 人の担当者)Jupyter Notebook (7 件のフォーク)auto 404
data pipelinesenhancementhelp wanted

Repository metrics

Stars
 (16 個のスター)
PR merge metrics
 (PR metrics pending)

説明

Is there an existing issue for this?

  • I have searched the existing issues

Task/Feature/Issue Details

The metadata in HF datasets is (presumably) added manually by the owners. This means it is often going to be 1) wrong, 2) incomplete, 3) use less useful synonyms that the most common keyword for a concept. For the current static catalog, I manually associated keywords that were related into the same tables. (see: https://github.com/The-AI-Alliance/open-trusted-data-initiative/blob/main/static-catalog/src/scripts/write-category-files.py)

This issue proposes analyzing datasets with LLM classifiers to categorize them using a standard set of keywords, which we can either define or adopt from another source. (For example, the advertising industry defines categories and terms like this.) The metadata keywords should still be displayed to the user, but the derived keywords can be used for searching, hierarchical displays, etc.

Related: should we create a knowledge graph or at least a topic hierarchy using this approach, not just a flat set of canonical keywords?

Additional Context

No response

コントリビューターガイド