StoryMaps very slow to clone due to encoding check
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 55/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Stale
- Tech stack
- python
- Domain
- authentication
Research direction
Start in Lib/site-packages/arcgis/auth/_auth/_token.py at handle_401, then reproduce the clone_items() StoryMap example with resources. Check the response handling for resources/export?f=zip and verify that binary ZIP responses no longer trigger encoding detection while token errors remain handled correctly.
Written by the indexing model from the issue text.
Description
Describe the bug
When trying to clone a StoryMap from one organisation to another, the clone_items() method takes a very long time to clone a StoryMap, specifically its resources.
I've identified the cause of this issue. The response from resources/export?f=zip is passed to the method handle_401 in Lib\site-packages\arcgis\auth_auth_token.py.
This method checks the response .text which causes the requests module to try to identify the encoding of the response content. This takes a long time as it is attempting to identify the encoding of a zip file... several hours, on my machine.
To Reproduce
Steps to reproduce the behavior:
Clone a StoryMap with resources (images) from one organisation to another
gis1 = GIS()
gis2 = GIS()
storymap = gis1.content.get("<id>")
copy_list = [storymap]
clone_results=gis2.content.clone_items(copy_list, copy_data=True, search_existing_items=True)
error:
2024-11-13 09:28:35,588,588 DEBUG [mbcharsetprober.py:65] EUC-KR Korean prober hit error at byte 10
2024-11-13 09:28:35,589,589 DEBUG [mbcharsetprober.py:65] CP949 Korean prober hit error at byte 60
2024-11-13 09:28:35,590,590 DEBUG [mbcharsetprober.py:65] Big5 Chinese prober hit error at byte 10
2024-11-13 09:28:35,591,591 DEBUG [mbcharsetprober.py:65] EUC-TW Taiwan prober hit error at byte 10
2024-11-13 09:29:08,463,463 DEBUG [sbcharsetprober.py:129] windows-1251 confidence = 0.046932830793816153, below negative shortcut threshhold 0.05
2024-11-13 09:29:39,105,105 DEBUG [sbcharsetprober.py:129] KOI8-R confidence = 0.04048963086780929, below negative shortcut threshhold 0.05
2024-11-13 09:30:09,992,992 DEBUG [sbcharsetprober.py:129] ISO-8859-5 confidence = 0.04340445793941901, below negative shortcut threshhold 0.05
2024-11-13 09:30:32,587,587 DEBUG [sbcharsetprober.py:129] MacCyrillic confidence = 0.04770257721175075, below negative shortcut threshhold 0.05
Screenshots
Timing warning from VS Code
Debug logs showing multiple minutes trying to identify file encoding
Stack trace
Results of printing r.text, showing that r.text is not empty for zip files.
Expected behavior
For binary files, the response content is not checked by this method.
A not-very-elegant approach I used to successfully mitigate this issue was to update the handle_401 method manually to bypass any response where the encoding was not provided:
def handle_401(self, r, **kwargs):
"""
handles the issues in the response where token might be rejected
"""
parsed = parse_url(r.url)
if r.encoding is None:
return r
elif (
r.text.lower().find("invalid token") > -1
or r.text.find("Token is valid but access is denied.") > -1
or (parsed.scheme, parsed.netloc, parsed.path) in self._no_go_token
):
Platform (please complete the following information):
- OS: Windows
- Browser: N/A Visual Studio Code
- Python API Version: 2.4.0
Additional context
N/A
- Dominant language
- Python
- Stars
- 2.2k
- Forks
- 1.1k
- Avg merge
- 2h 40m
- Merged PRs (30d)
- 2
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Esri/arcgis-python-api
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Esri/arcgis-python-api#2513 · 1 comment ·
-
bug
Difficulty 1/5 Under an hour Newbie friendliness 85/100
Esri/arcgis-python-api#2453 · 3 comments ·
-
documentation enhancement
Difficulty 1/5 Under an hour Newbie friendliness 76/100
Esri/arcgis-python-api#2097 · 1 comment ·
-
bug
Difficulty 4/5 3-5 days Newbie friendliness 58/100
Esri/arcgis-python-api#2537 · 3 comments ·
-
bug
Difficulty 4/5 3-5 days Newbie friendliness 56/100
Esri/arcgis-python-api#2531 · 1 comment ·
All issues in Esri/arcgis-python-api
Similar issues
-
documentation help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
simonw/sqlite-utils#872 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100