Python 3: make decoding shapefile fields more lenient...

Open
#187 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
35/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Stale
Tech stack
python
Domain
data

Research direction

Start in shapefile.py at the Python 3 field-decoding helper and reproduce the failure with the linked network.zip shape. Verify that fields containing undecodable bytes no longer prevent the shape from being drawn, while existing string and non-byte handling remains intact.

Written by the indexing model from the issue text.

Description

In python 3, field values of shapefile are parsed in shapefile.py, as shown below:

def u(v):
    if PYTHON3:
        if isinstance(v, bytes):
            # For python 3 decode bytes to str.
            return v.decode('utf-8')
        elif isinstance(v, str):
            # Already str.
            return v
        else:
            # Error.
            raise Exception('Unknown input type')
    else:
        # For python 2 assume str passed in and return str.
        return v

But in the case v.decode fails in the first if (when v is an instance of bytes), then the file won't be drawn. I propose to change it as follows (to make it work):

def u(v):
    if PYTHON3:
        if isinstance(v, bytes):
            # For python 3 decode bytes to str.
            return v.decode('utf-8', errors='ignore')
        elif isinstance(v, str):
            # Already str.
            return v
        else:
            # Error.
            raise Exception('Unknown input type')
    else:
        # For python 2 assume str passed in and return str.
        return v

Here is the link to the shape on which the original version fails and the changed one works:
https://dl.dropboxusercontent.com/u/4629759/network.zip

Cheers

Dominant language
Python
Stars
817
Forks
395
No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from matplotlib/basemap

All issues in matplotlib/basemap

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.