"UnicodeEncodeError: 'latin-1' codec can't encode character, ordinal not in range" on GCS upload_object_via_stream

Ouverte
  • Quelque chose ne va pas ?
#1,802 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
32/100
Type d'issue
Bug
Clarté
À clarifier
Activité
À l'abandon
Stack technique
google-cloud, python
Domaine
cloud

Piste de recherche

Commencez par upload_object_via_stream dans libcloud/storage/base.py et le chemin du driver S3 dans libcloud/storage/drivers/s3.py, puis suivez la préparation de la requête à travers libcloud/common/base.py et libcloud/http.py. Reproduisez l’échec avec un CSV contenant des caractères non-encodables en Latin-1 dans l’environnement Python et conteneur indiqué. C’est terminé lorsque l’upload gère ce contenu CSV sans le UnicodeEncodeError et inclut une couverture de régression pour le chemin en échec.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Summary

I'm trying to upload some CSV files to GCS using libcloud, today I this error:

UnicodeEncodeError: 'latin-1' codec can't encode character '\u201c' in position 628: ordinal not in range(256)

Detailed Information

The encode character and position change with each run, even though the files don't change. I'm not sure what's causing that, CSV files are basically plaintext.

Here's the full stack (some internal code ommitted):

Traceback (most recent call last):
  [...]
    self.__container.upload_object_via_stream(stream, name)  # type: ignore
  File "/usr/local/lib/python3.8/site-packages/libcloud/storage/base.py", line 227, in upload_object_via_stream
    return self.driver.upload_object_via_stream(
  File "/usr/local/lib/python3.8/site-packages/libcloud/storage/drivers/s3.py", line 843, in upload_object_via_stream
    return self._put_object(
  File "/usr/local/lib/python3.8/site-packages/libcloud/storage/drivers/s3.py", line 1007, in _put_object
    result_dict = self._upload_object(
  File "/usr/local/lib/python3.8/site-packages/libcloud/storage/base.py", line 874, in _upload_object
    response = self.connection.request(
  File "/usr/local/lib/python3.8/site-packages/libcloud/common/base.py", line 659, in request
    return request_to_be_executed(
  File "/usr/local/lib/python3.8/site-packages/libcloud/common/base.py", line 678, in _retryable_request
    self.connection.prepared_request(
  File "/usr/local/lib/python3.8/site-packages/libcloud/http.py", line 260, in prepared_request
    self.response = self.session.send(
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/lib/python3.8/http/client.py", line 1256, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1302, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1251, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1040, in _send_output
    for chunk in chunks:
  File "/usr/local/lib/python3.8/http/client.py", line 999, in _read_readable
    datablock = datablock.encode("iso-8859-1")
UnicodeEncodeError: 'latin-1' codec can't encode character '\u2019' in position 5251: ordinal not in range(256)

I'm running this from inside a k8s pod in a GKE cluster, trying to write to a GCS bucket.

apache-libcloud version: 3.5.1
python version: 3.8.15
pod image: python:3.8-slim

I'm not sure what are the steps to reproduce it. It's a CSV file generated by pulling some BigQuery schema metadata. It works locally, but when I ran it in the pod it raised this error.

Langage dominant
Python
Étoiles
2.1k
Forks
935
Merge moyen
5 j 3 h
PR mergées (30 j)
15

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de apache/libcloud

Toutes les issues de apache/libcloud

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.