Slave API 404s can fill up log files

Open
#326 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
python
Domain
api, backend

Research direction

Start with app/web_framework/cluster_slave_application.py at the get handler and app/common/cluster_service.py at get_console_output, then trace how ItemNotFoundError becomes a 404 and is logged. Confirm the handling for missing console output and the CORS message, and add or update focused tests if the surrounding code provides them; done means repeated 404s no longer flood the slave log.

Written by the indexing model from the issue text.

Description

enhancement help wanted minor

We've created a dashboard for viewing console output data and if for any reason the console output is not available, the slave log file fills up with tons of errors. CR probably shouldn't log anything on a 404. We could probably also remove the line about the Access-Control-Allow-Origin header.

Example (hundreds of copies of this):

[2017-02-02 12:49:54.280] 19480 WARNING SlaveTornadoThr tornado.access  404 GET /v1/build/141/subjob/590/atom/10/console?max_lines=50 (10.8.189.29) 1.08ms
[2017-02-02 12:49:54.281] 19480 DEBUG   SlaveTornadoThr cluster_base_ha Origin "null" did not match cors_allowed_origins_regex conf value of "^http://[^.]*\.((inside-box)|(pod\.box)|(dev\.box))\.net". Not setting Access-Control-Allow-Origin header.
[2017-02-02 12:49:54.282] 19480 ERROR   SlaveTornadoThr cluster_base_ha Exception occurred during request to /v1/build/141/subjob/590/atom/12/console?max_lines=50.
Traceback (most recent call last):
  File "/home/jenkins/.virtualenvs/clusterrunner/lib/python3.4/site-packages/tornado/web.py", line 1346, in _when_complete
  File "/home/jenkins/.virtualenvs/clusterrunner/lib/python3.4/site-packages/tornado/web.py", line 1367, in _execute_method
  File "/home/jenkins/ClusterRunnerBuild/app/web_framework/cluster_slave_application.py", line 156, in get
  File "/home/jenkins/ClusterRunnerBuild/app/common/cluster_service.py", line 49, in get_console_output
app.util.exceptions.ItemNotFoundError: Output file doesn't exist for build_id: 141 subjob_id: 590 atom_id: 12
[2017-02-02 12:49:54.282] 19480 WARNING SlaveTornadoThr tornado.access  404 GET /v1/build/141/subjob/590/atom/12/console?max_lines=50 (10.8.189.29) 1.10ms
[2017-02-02 12:49:54.283] 19480 DEBUG   SlaveTornadoThr cluster_base_ha Origin "null" did not match cors_allowed_origins_regex conf value of "^http://[^.]*\.((inside-box)|(pod\.box)|(dev\.box))\.net". Not setting Access-Control-Allow-Origin header.
[2017-02-02 12:49:54.283] 19480 ERROR   SlaveTornadoThr cluster_base_ha Exception occurred during request to /v1/build/141/subjob/590/atom/3/console?max_lines=50.
Traceback (most recent call last):
  File "/home/jenkins/.virtualenvs/clusterrunner/lib/python3.4/site-packages/tornado/web.py", line 1346, in _when_complete
  File "/home/jenkins/.virtualenvs/clusterrunner/lib/python3.4/site-packages/tornado/web.py", line 1367, in _execute_method
  File "/home/jenkins/ClusterRunnerBuild/app/web_framework/cluster_slave_application.py", line 156, in get
  File "/home/jenkins/ClusterRunnerBuild/app/common/cluster_service.py", line 49, in get_console_output
app.util.exceptions.ItemNotFoundError: Output file doesn't exist for build_id: 141 subjob_id: 590 atom_id: 3
Dominant language
Python
Stars
184
Forks
41
PR merge metrics
No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from box/ClusterRunner

All issues in box/ClusterRunner

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.