StoryMaps very slow to clone due to encoding check

Open
#2,159 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
55/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Stale
Tech stack
python

Research direction

Start in Lib/site-packages/arcgis/auth/_auth/_token.py at handle_401, then reproduce the clone_items() StoryMap example with resources. Check the response handling for resources/export?f=zip and verify that binary ZIP responses no longer trigger encoding detection while token errors remain handled correctly.

Written by the indexing model from the issue text.

Description

bug

Describe the bug
When trying to clone a StoryMap from one organisation to another, the clone_items() method takes a very long time to clone a StoryMap, specifically its resources.

I've identified the cause of this issue. The response from resources/export?f=zip is passed to the method handle_401 in Lib\site-packages\arcgis\auth_auth_token.py.

This method checks the response .text which causes the requests module to try to identify the encoding of the response content. This takes a long time as it is attempting to identify the encoding of a zip file... several hours, on my machine.

To Reproduce
Steps to reproduce the behavior:

Clone a StoryMap with resources (images) from one organisation to another

gis1 = GIS()
gis2 = GIS()
storymap = gis1.content.get("<id>")
copy_list = [storymap]
clone_results=gis2.content.clone_items(copy_list, copy_data=True, search_existing_items=True)

error:

2024-11-13 09:28:35,588,588 DEBUG    [mbcharsetprober.py:65] EUC-KR Korean prober hit error at byte 10
2024-11-13 09:28:35,589,589 DEBUG    [mbcharsetprober.py:65] CP949 Korean prober hit error at byte 60
2024-11-13 09:28:35,590,590 DEBUG    [mbcharsetprober.py:65] Big5 Chinese prober hit error at byte 10
2024-11-13 09:28:35,591,591 DEBUG    [mbcharsetprober.py:65] EUC-TW Taiwan prober hit error at byte 10
2024-11-13 09:29:08,463,463 DEBUG    [sbcharsetprober.py:129] windows-1251 confidence = 0.046932830793816153, below negative shortcut threshhold 0.05
2024-11-13 09:29:39,105,105 DEBUG    [sbcharsetprober.py:129] KOI8-R confidence = 0.04048963086780929, below negative shortcut threshhold 0.05
2024-11-13 09:30:09,992,992 DEBUG    [sbcharsetprober.py:129] ISO-8859-5 confidence = 0.04340445793941901, below negative shortcut threshhold 0.05
2024-11-13 09:30:32,587,587 DEBUG    [sbcharsetprober.py:129] MacCyrillic confidence = 0.04770257721175075, below negative shortcut threshhold 0.05

Screenshots
image
Timing warning from VS Code

image
Debug logs showing multiple minutes trying to identify file encoding

image
Stack trace

image
Results of printing r.text, showing that r.text is not empty for zip files.

Expected behavior
For binary files, the response content is not checked by this method.

A not-very-elegant approach I used to successfully mitigate this issue was to update the handle_401 method manually to bypass any response where the encoding was not provided:

    def handle_401(self, r, **kwargs):
        """
        handles the issues in the response where token might be rejected
        """
        parsed = parse_url(r.url)
        if r.encoding is None:
            return r
        elif (
            r.text.lower().find("invalid token") > -1
            or r.text.find("Token is valid but access is denied.") > -1
            or (parsed.scheme, parsed.netloc, parsed.path) in self._no_go_token
        ):

Platform (please complete the following information):

  • OS: Windows
  • Browser: N/A Visual Studio Code
  • Python API Version: 2.4.0

Additional context
N/A

Dominant language
Python
Stars
2.2k
Forks
1.1k
Avg merge
2h 40m
Merged PRs (30d)
2

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Esri/arcgis-python-api

All issues in Esri/arcgis-python-api

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.