graphql-python/graphene-django

Performance issues with large data sets and pagination.

Open

#829 opened on Dec 19, 2019

View on GitHub
 (3 comments) (6 reactions) (0 assignees)Python (4,173 stars) (754 forks)batch import
help wanted✨enhancement

Description

It is known that GraphQL is not the fastest API when you have many objects, see: https://github.com/graphql-python/graphene/issues/268

If you have many objects, you want to use pagination with DjangoConnectionField.

I have run some benchmarks for this:

  1. There are 1000 items and i fetch 100 of them:
django v1.11.26
graphene v2.1.8
graphene-django v2.7.1
Use timeit statement: graphene.Schema(query=Query).execute('{Items(first: 100) {edges {node {id}}}}')
Run one timeit call... takes: 13.3 ms
timeit... use 5 * 75 loop...
max...: 10.84 ms
median: 10.66 ms
min...: 10.61 ms
cProfile stats for one request: 30148 function calls (28688 primitive calls) in 0.019 seconds
  1. There are only 100 items and i didn't use a DjangoConnectionField:
django v1.11.26
graphene v2.1.8
graphene-django v2.7.1
Use timeit statement: graphene.Schema(query=Query).execute('{Items {id}}')
Run one timeit call... takes: 13.3 ms
timeit... use 5 * 106 loop...
max...: 7.54 ms
median: 7.48 ms
min...: 7.29 ms
cProfile stats for one request: 18556 function calls (17830 primitive calls) in 0.012 seconds

Graphene already makes a lot of calls anyway. But with these two variants there is also a very clear difference to be noticed.

Contributor guide

Tech stack
pythondjango
Domain
backendapiperformance
Issue type
performance
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
Django ORMGraphQL pagination
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30
Research direction
Investigate the overhead of DjangoConnectionField in graphene django, focusing on the number of SQL queries executed per page. Compare with raw Django queries and consider implementing prefetch related or select related optimizations. Review the benchmark code and identify if the issue is in graphene core or graphene django. Possible fix: optimize the DjangoConnectionField resolver to reduce query counts, or add documentation on best practices for large datasets.