jazzband/django-silk

Problem with silk in child threads

Open

#169 opened on Mar 14, 2017

View on GitHub
 (13 comments) (7 reactions) (0 assignees)Python (3,958 stars) (319 forks)batch import
enhancementhelp wanted

Description

Hi,

I'm working on synchronous API that is processing some data. In order to speed things up I have decided to spawn multiple threads in Django request and do processing there. The problem occurs when one of the threads tries to execute sql queries. Is there any way to run silk and spawn child threads?

The self.register_objects(TYP_QUERIES, *args) could just give a warning in that case and carry on.

File "/zzzz/service/yyyy/xxxx.py", line 883, in get_negative_info
    p_1 = Node.objects.get(id=node)
  File "/Library/Python/2.7/site-packages/django/db/models/manager.py", line 122, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/Library/Python/2.7/site-packages/django/db/models/query.py", line 381, in get
    num = len(clone)
  File "/Library/Python/2.7/site-packages/django/db/models/query.py", line 240, in __len__
    self._fetch_all()
  File "/Library/Python/2.7/site-packages/django/db/models/query.py", line 1074, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/Library/Python/2.7/site-packages/django/db/models/query.py", line 52, in __iter__
    results = compiler.execute_sql()
  File "/Library/Python/2.7/site-packages/silk/sql.py", line 52, in execute_sql
    DataCollector().register_query(query_dict)
  File "/Library/Python/2.7/site-packages/silk/collector.py", line 122, in register_query
    self.register_objects(TYP_QUERIES, *args)
  File "/Library/Python/2.7/site-packages/silk/collector.py", line 109, in register_objects
    self.ensure_middleware_installed()
  File "/Library/Python/2.7/site-packages/silk/collector.py", line 43, in ensure_middleware_installed
    raise_middleware_error()
  File "/Library/Python/2.7/site-packages/silk/collector.py", line 24, in raise_middleware_error
    'Silk middleware has not been installed correctly. Ordering must ensure that Silk middleware can '
RuntimeError: Silk middleware has not been installed correctly. Ordering must ensure that Silk middleware can execute process_request and process_response. If an earlier middleware returns from either of these methods, Silk will not have the chance to inspect the request/response objects.

Thanks, Marcin

Contributor guide

Tech stack
pythonsql
Domain
backenddatabase
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
DjangoPythonthreadingdjango silk basics
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
65
Research direction
Investigate the silk/collector.py file, specifically the register objects and ensure middleware installed functions. Consider modifying ensure middleware installed to return a boolean or handle the missing middleware in child threads gracefully, perhaps by logging a warning instead of raising an error. Review the 13 comments for any additional context or proposed solutions from the community.