Cannot access file and directory _names_ that are invalid UTF-8

Open
#919 11 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
20/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale

Research direction

The affected entry points are the file-system functions listed in Section 9.16 of the Chez Scheme User’s Guide; the issue does not name source files or tests. Start by tracing how those functions convert names on POSIX systems and review the linked UTF-8b references. Done means invalid UTF-8 names can be accessed without loss across the listed functions.

Written by the indexing model from the issue text.

Description

The functions

(current-directory) (cd) (directory-list)
(file-exists?) (file-regular?) (file-directory?) (file-symbolic-link?) (file-access-time) (file-change-time)
(file-modification-time) (mkdir) (delete-file) (delete-directory) (rename-file) (chmod) (get-mode)

described in Section 9.16. File System Interface https://cisco.github.io/ChezScheme/csug10.0/io.html#./io:h16 operate on file names and directory names represented as Scheme strings.

When actually accessing the file system on POSIX systems, such strings are automatically converted from/to UTF-8.

This has the side effect that existing files and directories whose names are invalid UTF-8 cannot be accessed by the functions listed above.

Example:

$ mkdir example
$ cd example
$ touch $(printf 'AAA\xffzzz')
$ ls -l aaa*
-rw-r--r-- 1 user users 0 Feb 20 10:15 'AAA'$'\377''zzz'
$ chezscheme
> (define x (car (sort string<? (directory-list "."))))

> x
"AAA�zzz"

> (char->integer (string-ref x 3))
65533

> (delete-file x)
#f

> (delete-file x #t)
Exception in delete-file: failed for AAA�zzz: no such file or directory
Type (debug) to enter the debugger.

The problem is: byte #xff is not a valid UTF-8 sequence,
and (directory-list) converts it to replacement character #xFFFD as per UTF-8 error-handling rules.

As a consequence, the file created with shell command touch $(printf 'AAA\xffzzz')
and all other files or directories whose names are invalid UTF-8
cannot be accessed with Chez Scheme functions listed above.

Since POSIX file system specifications do not require that files or directory names are valid UTF-8,
this leaves the above Chez Scheme functions in the uncomfortable position of failing on some valid POSIX file and directory names.

A solution could be to convert file and directory names from/to UTF-8b (note the 'b') instead of UTF-8,
because UTF-8b is an extension of UTF-8 designed exactly to losslessly convert any byte or byte sequence.

For a definition of UTF-8b, see
https://peps.python.org/pep-0383
https://web.archive.org/web/20090830064219/http://mail.nl.linux.org/linux-utf8/2000-07/msg00040.html

Dominant language
Scheme
Stars
7.4k
Forks
1k
Avg merge
1d 20h
Merged PRs (30d)
19

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from cisco/ChezScheme

All issues in cisco/ChezScheme

Similar issues

More Operating Systems issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.