Enhancement: Inversion of control scraper base class
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 5/5
- Tiempo estimado
- Más de una semana
- Aptitud para principiantes
- 25/100
Línea de trabajo
Comienza revisando las abstracciones de scraper existentes y el trabajo del scraper de DevDocs descrito en la issue; no se mencionan archivos ni tests específicos. Define una clase base reutilizable que cubra la configuración compartida, como la caché, los flags de CLI, los metadatos de Zim, el logging, el seguimiento del progreso y el acceso HTTP, y verifica después que un scraper pueda usarla con menos configuración y tests duplicados.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
When recently building the DevDocs scraper, I realized there are a ton of things that I was relying on @benoit74's expertise for to make the scraper sustainable for ZimFarm but weren't core to building a functional scraper. These include:
- Caching if S3 is available.
- Expected CLI flags.
- Zim file parameters/naming/destinations.
- ZimFarm operator friendly logging setup.
- Progress tracker.
- HTTP client setup.
The scraper still isn't fully there, but I've already spent a lot of time implementing and testing some of these things. I've written Logstash plugins and Elastic Beats (which are more smiilar to Zim scrapers and those were dramatically easier because that common logic was abstracted away.
I'd love to see something like the following (example only!) as what I had to build while knowing most of the above would be taken care of:
# New type ZimMetadata contains the properties to populate config_metadata() on a Zim.
# There are specific types based on whether the scraper is for a single Zim or multiple (in which case
# it supports placeholders).
# Methods can be overridden for fine-grained control e.g. to add additional formatting parameters.
M = TypeVar('M', bound=ZimMetadata)
class MyScraper(MultiZimScraper):
# Parent class includes a logger, HTTP client, potentially other items.
def add_flags(parser: argparse.ArgumentParser):
'''Add custom flags to the program'''
pass
def setup(namespace: argparse.Namespace):
'''Parse flags and set up resources for execution.
After this call, MultiZimScraper may have additional internal variables set up
e.g. an HTTP client that automatically caches to S3 if running in ZimFarm and with
retries/delay.
'''
pass
def list_zims() -> M:
'''Called after setup to list all Zims to be created. '''
pass
def add_contents(creator: Creator, metadata: M):
'''Called for each item in list_zims().
The JSON progress file is updated between calls, logs for progress/next ZIM/timing are written
and a scraper check utility could be asserted after.
'''
pass
I don't think all scrapers would need to use this format, but something like it would have dramatically cut down on the amount of testing and knowledge needed for me to produce a quality Zim scraper.
- Lenguaje dominante
- Python
- Estrellas
- 31
- Forks
- 27
- Merge medio
- 3 d 7 h
- PR fusionados (30 d)
- 2
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de openzim/python-scraperlib
-
bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
openzim/python-scraperlib#339 ·
-
enhancement
Dificultad 2/5 1-3 horas Aptitud para principiantes 68/100
openzim/python-scraperlib#292 ·
-
bug
Dificultad 4/5 3-5 días Aptitud para principiantes 35/100
openzim/python-scraperlib#341 ·
-
bug
Dificultad 3/5 1-2 días Aptitud para principiantes 45/100
openzim/python-scraperlib#340 ·
-
bug
openzim/python-scraperlib#334 · 1 asignado ·
Todos los issues de openzim/python-scraperlib
Issues similares
-
bug confirmed issue
Dificultad 2/5 1-3 horas Aptitud para principiantes 75/100
open-webui/open-webui#30750 · 1 comentario ·
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 75/100
-
enhancement
Dificultad 2/5 1-3 horas Aptitud para principiantes 75/100
OpenwaterHealth/openmotion-bloodflow-app#604 · 1 comentario ·
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 70/100
-
good first issue
Dificultad 1/5 Menos de una hora Aptitud para principiantes 90/100