YaleComputerSociety/Yalies-Legacy

Better error handling for scraper

開放

#222 建立於 2024年2月5日

 (1 則留言) (0 個反應) (0 位負責人)Python (23 個分叉)auto 404
good first issuescraperwill be fixed during rewrite

倉庫指標

星標
 (12 顆星)
PR 合併指標
 (30 天內沒有已合併 PR)

描述

When the scraper encounters any sort of error (e.g. a blank HTTP response), the whole script hangs. Add some exception handling so the script can continue and just skip over that one page.

Also, from #223

because the different parts of the scraper mostly run asynchronously, errors are not being properly thrown from the appropriate thread. This leads to errors that can be difficult to debug, because the apparent cause of a crash will be the fact that one thread doesn't return a students list for example, when in fact the actual root cause is something more specific that happened in that thread several thousand lines of logs ago.

貢獻者指南