alibaba/GraphScope

Add more popular datasets to graphscope built-in datasets

Open

Aperta il 17 nov 2021

Vedi su GitHub
 (0 commenti) (0 reazioni) (1 assegnatario)HTML (2401 star) (301 fork)batch import
good first issue

Descrizione

We have several built-in datasets that can be easily loaded in one-line, located in the dataset directory of Aliyun OSS bucket graphscope, and the corresponding utility function to load them, located in python/graphscope/dataset/. We are planning to enrich the datasets continuously.

There's the procedure to add new datasets:

  1. Find a popular and appropriate dataset, adapt the format to property graph if necessary,
  2. Put all data files inside a folder, give the folder a meaningful name,
  3. Compress the folder, then upload the compressed file together with the original folder to the dataset folder of the OSS bucket. Assume you have a folder named foo/, and two files foo/nodes.csv and foo/edge.csv, after this step, you will have the following file structure in the bucket:
dataset
|-- foo.tar.gz
|-- foo
    |-- nodes.csv
    |-- edge.csv
  1. Write the loading function load_foo in a new file named python/graphscope/dataset/foo.py.
  2. A corresponding unit test is appreciated!

Guida contributor

Tech stack
python
Dominio
data
Tipo issue
feature
DifficoltàQuanto dovrebbe essere impegnativa per un nuovo contributor.
3
Tempo stimatoTempo stimato per completare e verificare uno scope piccolo.
1-2 days
Stato attivitàQuanto è probabile che la issue sia ancora attiva e reviewable.
stale
ChiarezzaQuanto chiaramente la issue descrive problema, scope e risultato atteso.
clear
Prerequisiti
basic Pythonfamiliarity with graph datasets
Adatta ai principiantiQuanto la issue è adatta a contributor alla prima esperienza.
75
Direzione di ricerca
The issue provides a clear step by step procedure for adding a new dataset. First, find a popular graph dataset and adapt it to property graph format if necessary. Then, create a folder with the dataset name containing CSV or other files, compress it as .tar.gz, and upload both the compressed file and the original folder to the dataset folder in the Aliyun OSS bucket. Next, write a Python loading function in a new file under python/graphscope/dataset/, following the pattern of existing loaders. Finally, add a unit test for the new loader. Look at existing dataset loaders in the repository, such as those in the same directory, for reference.