onaio/onadata

Improve prefetch by the Project model

Open

#1,497 opened on Nov 7, 2018

View on GitHub
 (5 comments) (0 reactions) (0 assignees)Python (143 forks)auto 404
APIErrorHelp WantedModule: ProjectsPerformancePriority: MediumRefactorSize: Medium (2-3)

Repository metrics

Stars
 (187 stars)
PR merge metrics
 (PR metrics pending)

Description

class PrefetchManager(models.Manager):
    def get_queryset(self):
        from onadata.apps.logger.models.xform import XForm
        from onadata.apps.api.models.team import Team
        return super(PrefetchManager, self).get_queryset().select_related(
            'created_by', 'organization'
        ).prefetch_related(
            Prefetch('xform_set',
                     queryset=XForm.objects.filter(deleted_at__isnull=True)
                     .select_related('user')
                     .prefetch_related('user')
                     .prefetch_related('dataview_set')
                     .prefetch_related('metadata_set')
                     .only('id', 'user', 'project', 'title', 'date_created',
                           'last_submission_time', 'num_of_submissions',
                           'downloadable', 'id_string', 'is_merged_dataset'),
                     to_attr='xforms_prefetch')

The above prefetch code will cause serious memory issues when dealing with a project that has a lot of dataviews.

In the case of one user who had over 400,000 dataviews in one project, she was never able to load the projects endpoint for that specific project as the onadata application would always get OOM killed.

Contributor guide