Python3WebSpider/ProxyPool

tester可能存在内存泄漏

Closed

#119 opened on May 12, 2021

View on GitHub
 (6 comments) (0 reactions) (1 assignee)Python (1,877 forks)batch import
bughelp wanted

Repository metrics

Stars
 (5,195 stars)
PR merge metrics
 (No merged PRs in 30d)

Description

运行一段时间之后，tester占用的内存越来越多，必须手动重启才行，请问有人能帮忙看一下可能是哪里内存泄漏了么？

我使用docker-compose运行程序，并且修改了tester相关的配置，如下：

TEST_URL: <test-api-url>
TEST_TIMEOUT: 3
TEST_BATCH: 100

截图如下：

QQ截图20210512122509

可以看到，程序运行两个多月，getter和server的内存占用都比较正常，但是tester占用600+MB内存

Contributor guide

Research direction
Profile the tester process to identify objects causing memory growth, check for unreleased references or accumulating data structures.
Tech stack
python
Domain
backend
Issue type
Bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
Pythonmemory profiling
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40