Persistent `httpx.ConnectTimeout` when activating and downloading large volumes of UDM files
まだ誰も着手していません。
評価
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 初心者へのやさしさ
- 35/100
- issue の種類
- バグ
- 明瞭さ
- 説明が足りない
- 活発さ
- 停滞
- 技術スタック
- python
- 領域
- api, backend, networking
調査の方向性
Start with the retry configuration in planet/http.py and trace how DataClient.wait_asset and DataClient.download_asset handle concurrent requests. Review related issue #580 and determine whether the completed work should address ConnectTimeout and PoolTimeout failures, document concurrency limits and errors, or both.
索引モデルが issue の本文から書いたものです。
説明
Expected behavior
I expect the SDK to be able to handle a large number of concurrent requests under typical usage patterns, e.g. activating and downloading about a thousand UDM files.
Actual behavior (describe the problem)
We are trying to download a week's worth of PSScene UDMs for 150 different small AOIs, all in parallel.
When running many UDM activations in parallel, we see persistent httpx.ConnectTimeout failures during DataClient.wait_asset and DataClient.download_asset. Rarely, I instead see httpx.PoolTimeout, though it seems like this error can come from any DataClient interaction.
Based on the discussion in #580, it appears httpx.ConnectTimeout is not currently retried, because it didn't occur frequently during that set of testing (which was mainly focused on the orders API). I expect we are seeing a higher incidence of connect failures here because UDMs are small and quick to download.
However, just adding httpx.ConnectTimeout to RETRY_EXCEPTIONS is not sufficient to fix this issue - when I tried this, I reliably got the httpx.PoolTimeout, that I was more rarely seeing before. That failure appears more difficult to resolve, as it is coming from the session as a whole rather than an individual request. Once more revisiting #580, the PoolTimeout error seems correlated to the total volume of concurrent requests (which, for this use case, should be no greater than 1050).
Obviously, resolving the underlying issues would be optimal, but barring that there is another problem here - these sorts of failures produce errors that are difficult or impossible to troubleshoot and solve by the end-user.
If there is in fact an upper limit on the number of concurrent requests the SDK can handle (as seems to be implied by this failure, and the discussion in #580), some documentation describing those limits, and the kinds of errors they cause, would be beneficial.
Related Issues
- #580
Workaround
The only workaround we've found has been to not use the SDK.
Minimum, Complete, Viable Code Sample
None at the moment, but I can provide a link to the project this error was encountered on.
Environment Information
- MacOS 14.5
- Python 3.11
- planet SDK 2.10.0
Installation Method
- pip
- 主要言語
- Python
- スター
- 299
- フォーク
- 100
- 平均マージ
- 9日 22時間
- マージ済み PR（30日）
- 3
環境構築
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
planetlabs/planet-client-python のほかの issue
-
better-testing
難易度 2/5 1〜3時間 初心者へのやさしさ 65/100
planetlabs/planet-client-python#1110 ·
-
proposal
難易度 3/5 1〜2日 初心者へのやさしさ 65/100
planetlabs/planet-client-python#1221 ·
-
proposal
難易度 3/5 1〜2日 初心者へのやさしさ 57/100
planetlabs/planet-client-python#1214 ·
-
bug
難易度 3/5 1〜2日 初心者へのやさしさ 48/100
planetlabs/planet-client-python#1197 · コメント 1 件 · リアクション 1 件 ·
-
Quota API再び着手できるかも @asonnenschein が 410 日前に担当しましたが、オープン中のプルリクエストはありません。 オープンai-claude
planetlabs/planet-client-python#1191 · コメント 1 件 · 担当者 1 名 ·
planetlabs/planet-client-python の issue をすべて見る
似ている issue
-
docs pydanty:is-working
難易度 2/5 1〜3時間 初心者へのやさしさ 75/100
pydantic/pydantic-ai#8863 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 68/100
run-llama/llama_index#23278 ·
メンテナーはふだん 2 日以内に返信
-
documentation from-review-extraction github-actions priority: low severity:nit
難易度 1/5 1時間未満 初心者へのやさしさ 92/100
LearningCircuit/local-deep-research#6946 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
oracle/langchain-oracle#323 ·
メンテナーはふだん 1 日以内に返信
-
難易度 1/5 1時間未満 初心者へのやさしさ 88/100
tenstorrent/tt-metal#58057 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信