YaleComputerSociety/Yalies-Legacy

Better error handling for scraper

开放

#222 创建于 2024年2月5日

 (1 条评论) (0 个反应) (0 位负责人)Python (23 个派生)auto 404
good first issuescraperwill be fixed during rewrite

仓库指标

星标
 (12 个星标)
PR 合并指标
 (30 天内没有已合并 PR)

描述

When the scraper encounters any sort of error (e.g. a blank HTTP response), the whole script hangs. Add some exception handling so the script can continue and just skip over that one page.

Also, from #223

because the different parts of the scraper mostly run asynchronously, errors are not being properly thrown from the appropriate thread. This leads to errors that can be difficult to debug, because the apparent cause of a crash will be the fact that one thread doesn't return a students list for example, when in fact the actual root cause is something more specific that happened in that thread several thousand lines of logs ago.

贡献者指南