turicas/brasil.io

Coletar dados automaticamente

Open

#86 opened on Jun 6, 2018

View on GitHub
 (4 comments) (0 reactions) (0 assignees)Python (150 forks)github user discovery
dataenhancementhelp wanted

Repository metrics

Stars
 (1,032 stars)
PR merge metrics
 (PR metrics pending)

Description

Pré-requisito: #85.

A etapa final para que os dados de todos os datasets sejam coletados automaticamente será criar um ambiente no backend que agenda e executa tarefas.

Tarefas (ainda necessita de melhor detalhamento):

  • Criar versão inicial de script que executará os crawlers (data-worker)
  • Testar script acima nos datasets já libertados (e adequar datasets ao padrão)
  • Alterar modelos Django para incluir metadados sobre a execução da tarefa
  • Implementar/configurar agendador de tarefas (Celery?)
  • Criar worker para rodar o script e atualizar a plataforma com os novos dados

Contributor guide

Research direction
Set up Celery with Django, create a periodic task to run the data worker script, update Django models to store execution metadata, and ensure the worker updates the platform with new data.
Tech stack
pythondjango
Domain
backendapi
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
3-5 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Mostly clear
Prerequisites
PythonDjangoGitCelery basics
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
35